Key the URL dataset cache on the uprating inputs, and let parameter changes that reach them re-uprate the data

changelog.d/uprating-inputs-reach-data.fixed.md

@@ -0,0 +1 @@
+Dataset caching no longer couples a macro scenario to other simulations: a scenario built first left its projected incomes for its own baseline and every later simulation from the same `hf://` or `gs://` dataset, and one built after a baseline got current law's data. Dated parameter changes that change what the dataset projection reads, such as a reform dict setting an OBR growth rate for a year, are now written before the data is built, so incomes, rents and council tax follow them.

docs/book/usage/scenarios.md

@@ -415,7 +415,7 @@ Some parameters hold one value per calendar year, dated 1 January, and are read
 
 Processing also builds some parameters from others: the triple lock rate (`gov.economic_assumptions.yoy_growth.triple_lock`), the lagged and private pension series, the September CPI uprating rate and the indices. A change to one is written as soon as it is built, so everything built from it follows. `{"gov.economic_assumptions.yoy_growth.triple_lock": {"2028": 0.05}}` uprates the State Pension by 5% in April 2028, exactly as `gov.dwp.state_pension.triple_lock.outturn` set for that April does. A change to an index uprates whatever is uprated by it. The deprecated `gov.simulation.labor_supply_responses` paths change the `labour_supply_responses` parameters they copy.
 
-Changes applied after the data is loaded (the default) leave the data as built, so a changed economic assumption does not re-uprate incomes. Dated changes (`parameter_changes` and `"start.stop"` or bare-year keys) still come before the steps that read parameters while the simulation is built: the Universal Credit rebalancing awards and the structural reforms that parameters switch on. Single-day keys (other than a 1 January key on a calendar-year series, which is a dated change) and a scenario's own simulation modifier run after those steps, as they always have. Set `applied_before_data_load=True` on a scenario to project the data with changed economic assumptions too. A sum with a part applied before data load applies all its parameter changes before the data is built (see "Combining scenarios" above).
+A dataset's later years are projected from its first by the economic assumptions as the simulation is built. Dated changes that change what that projection reads (a growth rate, say, through `parameter_changes`, `"start.stop"` or bare-year keys, or a 1 January key on a calendar-year series) are written before the data is built, so incomes, rents and council tax follow them, as with `applied_before_data_load=True`. Other dated changes come just after the data is built, before the steps that read parameters while the simulation is built: the Universal Credit rebalancing awards and the structural reforms that parameters switch on. Other single-day keys and a scenario's own simulation modifier run after those steps, as they always have, and leave the data as built. Datasets loaded from `hf://` or `gs://` are cached once projected, under current law only: a simulation whose projection inputs differ builds its own, so a scenario never reaches its baseline or a later simulation. A sum with a part applied before data load applies all its parameter changes before the data is built (see "Combining scenarios" above).
 
 All of this applies when the scenario or reform is passed to `Simulation(...)`. `Scenario.apply(simulation)` on a simulation that is already built edits its annualised values instead, as it always has: a `"2027-04-06.2028-04-05"` key there overrides those dates of the fiscal-year values, so it does not set fiscal year 2027, and the model logs a warning saying so. Pass the scenario when constructing the simulation for dated measures.
 

policyengine_uk/data/economic_assumptions.py

@@ -4,6 +4,8 @@
     CGT_INDEX_PATH,
 )
 
+import hashlib
+
 import yaml
 from policyengine_core.parameters import ParameterNode
 from policyengine_core.simulations.simulation import _uprating_index_value
@@ -103,6 +105,73 @@ def _unmapped_uprating_indices(uprating):
     }
 
 
+COUNCIL_TAX_GROWTH = "gov.economic_assumptions.yoy_growth.obr.council_tax"
+SOCIAL_RENT_GROWTH = "gov.economic_assumptions.yoy_growth.obr.social_rent"
+PRIVATE_RENT_GROWTH = "gov.economic_assumptions.yoy_growth.ons.private_rental_prices"
+
+
+def _uprating_index_map() -> dict:
+    with open(Path(__file__).parent / "uprating_indices.yaml", "r") as f:
+        return yaml.safe_load(f)
+
+
+def uprating_input_paths() -> list:
+    """Every parameter path the dataset uprating reads.
+
+    ``apply_single_year_uprating`` and the functions it calls read these, and
+    no other parameters: the growth rates in ``uprating_indices.yaml``, the
+    council tax and rent series, the indices of the input variables that
+    file leaves out (after ``LEGACY_PROJECTION_END_YEAR``) and the capital
+    gains projection's metadata (``CGT_INDEX_PATH``). A path may name a node
+    (the regional private rent series), which covers its parameters.
+    """
+    uprating = _uprating_index_map()
+    series = {value for value in _COUNCIL_TAX_GROWTH_SERIES.values() if value}
+    return sorted(
+        set(uprating)
+        | {f"{COUNCIL_TAX_GROWTH}.{name}" for name in series}
+        | {SOCIAL_RENT_GROWTH, PRIVATE_RENT_GROWTH, CGT_INDEX_PATH}
+        | set(_unmapped_uprating_indices(uprating).values())
+    )
+
+
+def uprating_input_fingerprint(parameters: ParameterNode) -> str:
+    """A digest of the parameter values the dataset uprating reads.
+
+    Two parameter trees with the same fingerprint uprate a dataset
+    identically, so an uprated dataset can be shared between them. It covers
+    each read parameter's full timeline and the capital gains projection's
+    metadata.
+    """
+    from policyengine_core.parameters import Parameter
+
+    parts = []
+    for path in uprating_input_paths():
+        try:
+            node = parameters.get_child(path)
+        except ValueError:
+            parts.append((path, None))
+            continue
+        leaves = (
+            [node]
+            if isinstance(node, Parameter)
+            else [p for p in node.get_descendants() if isinstance(p, Parameter)]
+        )
+        for leaf in sorted(leaves, key=lambda leaf: leaf.name):
+            parts.append(
+                (
+                    leaf.name,
+                    tuple(
+                        (entry.instant_str, repr(entry.value))
+                        for entry in leaf.values_list
+                    ),
+                )
+            )
+        if path == CGT_INDEX_PATH:
+            parts.append((path, repr((node.metadata or {}).get("projection"))))
+    return hashlib.sha256(repr(parts).encode()).hexdigest()
+
+
 def extend_single_year_dataset(
     dataset: UKSingleYearDataset,
     tax_benefit_system_parameters: ParameterNode,
@@ -184,8 +253,7 @@ def apply_single_year_uprating(
 
     # First, apply standard variable-YoY growth based uprating.
 
-    with open(Path(__file__).parent / "uprating_indices.yaml", "r") as f:
-        uprating = yaml.safe_load(f)
+    uprating = _uprating_index_map()
     for index_name, variables in uprating.items():
         index_rel_change = parameters.get_child(index_name)(current_year.time_period)
         for variable in variables:
@@ -455,8 +523,7 @@ def reset_uprating(
 def reset_growthfactor_uprating(
     dataset: UKMultiYearDataset,
 ):
-    with open(Path(__file__).parent / "uprating_indices.yaml", "r") as f:
-        uprating = yaml.safe_load(f)
+    uprating = _uprating_index_map()
 
     growthfactor_uprated_variables = {
         variable for variables in uprating.values() for variable in variables

policyengine_uk/simulation.py

@@ -34,6 +34,7 @@
 from policyengine_uk.data.economic_assumptions import (
     extend_single_year_dataset,
     reset_growthfactor_uprating,
+    uprating_input_fingerprint,
 )
 from policyengine_uk.data.dataset_sources import materialize_gcs_dataset_url
 from policyengine_uk.data.input_dataframe import (
@@ -71,6 +72,32 @@
 )
 
 
+def _projects_data(situation, dataset) -> bool:
+    """Whether building from this source projects a dataset's later years.
+
+    ``extend_single_year_dataset`` projects a single-year dataset from the
+    parameters: one read from a file or URL (including the default dataset),
+    a ``UKSingleYearDataset`` or a policyengine-core ``Dataset``. A situation,
+    a DataFrame and a multi-year dataset are taken as given.
+    """
+    if situation is not None:
+        return False
+    return not isinstance(dataset, (pd.DataFrame, UKMultiYearDataset))
+
+
+def _scenario_assignments(scenario) -> list:
+    """A scenario's statutory-timeline changes: ``parameter_changes``, then
+    each of its ``dated_reforms``, in order."""
+    assignments = []
+    if scenario.parameter_changes:
+        assignments += read_parameter_changes(
+            _canonical_parameter_paths(scenario.parameter_changes),
+            calendar_years=parameter_reads_calendar_years(),
+        )
+    assignments += scenario.dated_assignments(order_start=len(assignments))
+    return assignments
+
+
 def _canonical_parameter_paths(changes: dict) -> dict:
     """Changes keyed by canonical parameter paths (labour supply aliases)."""
     return canonicalize_parameter_changes(changes)
@@ -83,6 +110,30 @@ def _canonical_parameter_paths(changes: dict) -> dict:
 # Avoids repeating HDF5 reading (0.84s), uprating (0.69s) and enum
 # encoding (0.67s) on every Simulation() call after the first.
 _url_dataset_cache: dict = {}
+# The uprating fingerprint of current law, for each CGT projection selection
+# (None: no projection), computed once. Only datasets uprated under current
+# law are cached: a simulation whose uprating inputs differ builds its own.
+_current_law_uprating_fingerprints: dict = {}
+
+
+def _current_law_uprating_fingerprint(capital_gains_projection) -> str:
+    key = (
+        None
+        if capital_gains_projection is None
+        else capital_gains_projection.fingerprint
+    )
+    if key not in _current_law_uprating_fingerprints:
+        system = (
+            CountryTaxBenefitSystem()
+            if capital_gains_projection is None
+            else CountryTaxBenefitSystem(
+                capital_gains_projection=capital_gains_projection
+            )
+        )
+        _current_law_uprating_fingerprints[key] = uprating_input_fingerprint(
+            system.parameters
+        )
+    return _current_law_uprating_fingerprints[key]
 
 
 def get_default_dataset_url() -> str:
@@ -226,11 +277,23 @@ def __init__(
 
         # A scenario's parameter changes rebuild the parameters from the
         # statutory tree, so they come before its modifier, whose edits a
-        # rebuild would discard.
+        # rebuild would discard. A sum's parameter changes all come with its
+        # before-load part. Without one, they come before the data is built
+        # when they change what the dataset's uprating reads (a growth rate,
+        # say), so the data's projected years follow them; otherwise just
+        # after.
+        after_parameters_before_data = (
+            before_data_load is None
+            and after_data_load is not None
+            and _projects_data(situation, dataset)
+            and self._scenario_changes_uprating_inputs(after_data_load)
+        )
         if before_data_load is not None:
             self.apply_scenario_parameters(before_data_load)
             if before_data_load.simulation_modifier is not None:
                 before_data_load.simulation_modifier(self)
+        elif after_parameters_before_data:
+            self.apply_scenario_parameters(after_data_load)
         self._refresh_capital_gains_projection()
         # Build simulation from appropriate source
         if situation is not None:
@@ -250,12 +313,12 @@ def __init__(
         else:
             raise ValueError(f"Unsupported dataset type: {dataset.__class__}")
 
-        # Parameter changes applied after data load leave the data as built
-        # (a changed economic assumption does not re-uprate it), but come
+        # Parameter changes that leave the data's uprating alone come here,
         # before the steps below that read the parameters: the Universal
         # Credit rebalancing inputs and the structural reforms switched on by
-        # parameters.
-        if after_data_load is not None:
+        # parameters. Changes a scenario's modifier makes after data load
+        # (including single-day reform-dict keys) leave the data as built.
+        if after_data_load is not None and not after_parameters_before_data:
             self.apply_scenario_parameters(after_data_load)
 
         # Universal Credit reform (July 2025). Needs closer integration in the baseline,
@@ -606,14 +669,22 @@ def capital_gains_projection_metadata(self) -> dict:
     def _dataset_cache_key(self, source: str):
         """The URL dataset cache key for a source, or None not to cache.
 
-        A cached dataset is already projected to later years, with current
-        law's economic assumptions unless a scenario applied before data load
-        changed them, so a simulation with such a scenario neither reads nor
-        writes the cache.
+        A cached dataset is already projected to later years, so it is shared
+        only between simulations that project it identically: none with a
+        scenario applied before data load (whose modifier can change how the
+        data is read, as a structural reform re-ordering an enum does), and
+        only those whose uprating inputs are current law's. A simulation with
+        changed inputs (a macro scenario, ``no_economic_assumptions``, or a
+        reform that changes them) neither reads nor writes the cache.
         """
         if self._scenario_before_data_load:
             return None
-        if self.tax_benefit_system.capital_gains_projection is None:
+        system = self.tax_benefit_system
+        if uprating_input_fingerprint(
+            system.parameters
+        ) != _current_law_uprating_fingerprint(system.capital_gains_projection):
+            return None
+        if system.capital_gains_projection is None:
             return source
         metadata = self.capital_gains_projection_metadata
         return (
@@ -658,14 +729,22 @@ def apply_scenario_parameters(self, scenario: Scenario) -> None:
         statutory tree does not define) are applied by the scenario's
         modifier instead.
         """
-        assignments = []
-        if scenario.parameter_changes:
-            assignments += read_parameter_changes(
-                _canonical_parameter_paths(scenario.parameter_changes),
-                calendar_years=parameter_reads_calendar_years(),
-            )
-        assignments += scenario.dated_assignments(order_start=len(assignments))
-        self.apply_parameter_assignments(assignments)
+        self.apply_parameter_assignments(_scenario_assignments(scenario))
+
+    def _scenario_changes_uprating_inputs(self, scenario: Scenario) -> bool:
+        """Whether a scenario's parameter changes change the dataset uprating.
+
+        Compares the uprating fingerprint of the parameters with the changes
+        (rebuilt on a copy of the system, a rebuild the memo then serves to
+        this one) with that of the parameters without them.
+        """
+        assignments = _scenario_assignments(scenario)
+        if not assignments:
+            return False
+        system = self.tax_benefit_system
+        return uprating_input_fingerprint(
+            system.parameters_with(assignments)
+        ) != uprating_input_fingerprint(system.parameters)
 
     def apply_parameter_assignments(self, assignments: list) -> None:
         """Rebuild the parameters with changes on the statutory timeline."""

policyengine_uk/tax_benefit_system.py

@@ -354,6 +354,23 @@ def apply_parameter_assignments(
         finally:
             self.reset_parameter_caches()
 
+    def parameters_with(self, assignments: List[ParameterAssignment]) -> ParameterNode:
+        """The parameters this system would have with more dated changes.
+
+        Rebuilds them on a shallow copy, which shares the variables and
+        entities but holds its own copy of the parameters, so this system is
+        left as it is, its parameter caches included, even when the rebuild
+        fails. A rebuild only reassigns the copy's attributes. When this
+        system was processed from the YAML and the changes' values can be
+        memoised (plain numbers, booleans and strings), the memo then serves
+        the same rebuild to this system if it is given the changes. The tree
+        returned is never this system's own.
+        """
+        probe = copy.copy(self)
+        probe.parameters = self.parameters.clone()
+        probe.apply_parameter_assignments(assignments)
+        return probe.parameters
+
     def add_legacy_overrides(self, overrides: List[LegacyOverride]) -> None:
         """Apply changes to the annualised parameters and keep them.
 

policyengine_uk/tests/test_simulation_dataset_sources.py

@@ -57,10 +57,18 @@ def fake_build_from_file(dataset_file, *, cache_key=None):
     assert captured == {"url": url}
 
 
+def _current_law_system():
+    """A stand-in system with current law's parameters: the URL cache is used
+    only by simulations whose dataset uprating inputs are current law's."""
+    from policyengine_uk.tax_benefit_system import system
+
+    return SimpleNamespace(capital_gains_projection=None, parameters=system.parameters)
+
+
 def test_dataset_source_routes_gcs_urls_to_materialized_file(monkeypatch):
     captured = {}
     simulation = Simulation.__new__(Simulation)
-    simulation.tax_benefit_system = SimpleNamespace(capital_gains_projection=None)
+    simulation.tax_benefit_system = _current_law_system()
     url = "gs://policyengine-uk-data-private/enhanced_frs_2023_24.h5@1.55.10"
 
     def fake_materialize_gcs_dataset_url(dataset_url):
@@ -91,7 +99,7 @@ def test_dataset_source_reuses_cached_gcs_dataset_before_materializing(monkeypat
     captured = {}
     cached_dataset = object()
     simulation = Simulation.__new__(Simulation)
-    simulation.tax_benefit_system = SimpleNamespace(capital_gains_projection=None)
+    simulation.tax_benefit_system = _current_law_system()
     url = "gs://policyengine-uk-data-private/enhanced_frs_2023_24.h5@1.55.10"
 
     def fake_materialize_gcs_dataset_url(dataset_url):
@@ -161,7 +169,7 @@ def fake_build_from_multi_year_dataset(dataset):
     )
 
     simulation = Simulation.__new__(Simulation)
-    simulation.tax_benefit_system = SimpleNamespace(capital_gains_projection=None)
+    simulation.tax_benefit_system = _current_law_system()
     monkeypatch.setattr(
         simulation,
         "build_from_multi_year_dataset",