Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
d5f9cb6
Keep stored household ISA totals after the projection horizon
MaxGhenis Oct 30, 2026
1f16eca
Document how stored ISA totals project past 2032
MaxGhenis Oct 30, 2026
8a2ab9d
Check stored totals stay flat with savings without economic growth
MaxGhenis Oct 30, 2026
ac21e3c
Let newer person balances replace an older stored ISA total
MaxGhenis Oct 31, 2026
3b458e9
Check that a reform's formula for an ISA total runs past the stored y…
MaxGhenis Oct 31, 2026
8991161
Rewrap the ISA capital stocks paragraph
MaxGhenis Oct 31, 2026
38d4005
Merge main into fix-2740-stored-isa-totals
MaxGhenis Oct 31, 2026
2b7ccc1
Leave a reform's parameter sum for an ISA total to Core
MaxGhenis Oct 31, 2026
10b19a5
Leave a reform sum whose parts share no index to Core
MaxGhenis Oct 31, 2026
c31d62c
Say when a stored ISA total grows and when its parts' sum stands
MaxGhenis Oct 31, 2026
7373879
Leave a reform sum that subtracts to Core, and tighten the ISA wording
MaxGhenis Oct 31, 2026
95cfa9d
Rewrap the ISA capital stocks paragraph
MaxGhenis Oct 31, 2026
309c112
Merge main into fix-2740-stored-isa-totals
MaxGhenis Oct 31, 2026
ced0005
Merge main into fix-2740-stored-isa-totals
MaxGhenis Oct 31, 2026
db6aa6c
Grow a stored ISA total only while it stands for its stored parts
MaxGhenis Oct 31, 2026
f6850ab
Say when a stored ISA total grows and when a reform gets the sum
MaxGhenis Oct 31, 2026
7c2b43d
Grow the several-parts test total from its last stored year
MaxGhenis Oct 31, 2026
e421eeb
Say which reforms leave a stored total to Core in its comment
MaxGhenis Oct 31, 2026
6cf11d4
Set balances only on a named branch in the stored-totals property test
MaxGhenis Oct 31, 2026
2003d18
Leave a stored total to Core once a reform abolishes its part
MaxGhenis Oct 31, 2026
bbab680
Leave a total to Core when its part stops reading annual inputs
MaxGhenis Oct 31, 2026
9def367
Leave a total to Core when its part casts, waits or holds another unit
MaxGhenis Nov 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/2740.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Keep the household ISA totals a dataset stores after the years it holds, grown like the person balances they sum, so means-tested capital no longer drops by those holdings from 2033.
16 changes: 10 additions & 6 deletions docs/book/assumptions/isa-inputs-and-capital-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,16 @@ established by the household tests.
`cash_isa_balance`, and `stocks_and_shares_isa` sums their
`stocks_and_shares_isa_balance`, as `household_lifetime_isa_balance` sums
`lifetime_isa_balance`. A dataset or situation that stores a household total
keeps it: the total is recomputed only when it is not stored, and a stored
total is projected with `savings` or `corporate_wealth`. A household input is
used only for the year it is given for; the person balances carry the
variable-level uprating to later years. The means tests read the household
totals, so a person's balance is pooled with the household's other financial
capital rather than attributed to their own benefit unit.
keeps it: the model sums the person balances only for a year with no stored
total. In each later year the model grows the latest total that a dataset
stores by the index of `savings`, `corporate_wealth` or `lifetime_isa_balance`
respectively, including years after the dataset projection ends, unless the
dataset holds the person balances for a later year than the total, or a reform
changes what the total sums or stops a person balance reading its inputs. A
household input in a situation applies only to its own year; the person
balances carry the variable-level uprating to later years. The means tests read
the household totals, so they pool a person's balance with the household's
other financial capital rather than attribute it to their own benefit unit.

Ordinary accessible cash ISA balances enter the capital sources for Universal
Credit, Housing Benefit, Pension Credit, Income Support, income-based JSA and
Expand Down
18 changes: 14 additions & 4 deletions docs/engineering/skills/dataset-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,10 +156,20 @@ Producers can export per-person `cash_isa_balance` and
`stocks_and_shares_isa_balance`. The household `cash_isa` and
`stocks_and_shares_isa` sum them, but a stored household column takes
precedence, so existing datasets that store only the household totals keep
them, projected with `savings` and `corporate_wealth` through
`uprating_indices.yaml`. A producer that exports the person balances should
either stop exporting the household totals or keep them equal to the sums;
the model does not reconcile the two.
them in every year. `uprating_indices.yaml` projects them with `savings` and
`corporate_wealth` through the years the dataset projection covers. In each
year a data source holds no total for, the simulation grows the latest
earlier stored total by the index its person balances declare
(`STORED_HOUSEHOLD_TOTALS` in `data/economic_assumptions.py`, which also covers
`household_lifetime_isa_balance`, with the person balance each total sums).
Person balances that the data hold for a later year than the total replace it
with their sum, and so does any reform that changes what a total sums or
stops one of its person balances reading its inputs. A situation differs: a
household total given in one applies only to its own year. An input frame is
data, so a situation reloaded from its input frame grows such a total in
later years. A producer that exports the person balances should either stop
exporting the household totals or keep them equal to the sums; the model does
not reconcile the two.

`cash_isa_subscriptions` and `stocks_and_shares_isa_subscriptions` are
annual person flows, summed over all of a person's accounts of each type, that
Expand Down
39 changes: 39 additions & 0 deletions policyengine_uk/data/economic_assumptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
)

import hashlib
from types import MappingProxyType

import yaml
from policyengine_core.parameters import ParameterNode
Expand Down Expand Up @@ -165,10 +166,29 @@ def growth_rate(
# which the Enhanced FRS does not store, so it falls to -1); one whose
# variable has an `uprating` attribute is scaled by that index; any other,
# rent and council tax among them, is held at its last projected value.
# Simulation grows the household totals in STORED_HOUSEHOLD_TOTALS itself.
# A finite CGT scenario can shorten the whole dataset horizon; it never
# extends beyond the evidence declared by that scenario.
PROJECTION_END_YEAR = 2032

# Household totals that sum person inputs (`adds`) but that a dataset can
# store itself (#2740), each with the parts it stands for. For a year with no
# stored total, Core runs the sum, which falls to zero in data that store
# only the total, and Core does not let a variable declare both `adds` and
# `uprating`. So in a simulation built from data, Simulation grows the latest
# earlier stored total by the index its parts declare, as Core grows the
# parts, while the total still means what the data stored: a sum of exactly
# these parts, each a plain input. Core sums the parts where a reform changes
# that sum or a part, where the data hold no total for that year or an
# earlier one, or where they hold the parts for a later year than the total.
STORED_HOUSEHOLD_TOTALS = MappingProxyType(
{
"cash_isa": frozenset({"cash_isa_balance"}),
"stocks_and_shares_isa": frozenset({"stocks_and_shares_isa_balance"}),
"household_lifetime_isa_balance": frozenset({"lifetime_isa_balance"}),
}
)

# Before this horizon was extended, Core uprated unmapped inputs from 2030.
# Preserve that behavior in the newly materialized years without changing
# projections for earlier years. Correcting historical mapping omissions is
Expand All @@ -194,6 +214,25 @@ def growth_rate(
YEAR_SPECIFIC_INPUTS = frozenset({"unused_pension_annual_allowance"})


def stored_total_uprating(tax_benefit_system, name: str) -> str | None:
"""The index that grows a stored household total: its parts' ``uprating``.

Growing the total like its parts keeps a stored total equal to the sum it
replaces. ``None`` holds the total flat, as Core holds parts with no
``uprating``. Parts that declare different indices raise ``ValueError``.
"""
indices = {
tax_benefit_system.variables[part].uprating
for part in tax_benefit_system.variables[name].adds
}
if len(indices) != 1:
raise ValueError(
f"The parts of {name} declare different uprating indices: "
f"{sorted(map(str, indices))}."
)
return indices.pop()


def _unmapped_uprating_indices(uprating):
from policyengine_uk.system import system

Expand Down
4 changes: 3 additions & 1 deletion policyengine_uk/data/uprating_indices.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,9 @@ gov.economic_assumptions.yoy_growth.obr.per_capita.gdp:
- benunit_reported_capital
# cash_isa and stocks_and_shares_isa sum the person balances listed with
# them; they are listed so a dataset that stores the household totals
# grows them like their parts.
# grows them like their parts. Past the projected years, Simulation grows
# each stored total in STORED_HOUSEHOLD_TOTALS, household_lifetime_isa_balance
# among them, the same way.
- cash_isa
- cash_isa_balance
- corporate_wealth
Expand Down
160 changes: 136 additions & 24 deletions policyengine_uk/simulation.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,10 @@
from policyengine_uk.utils.scenario import ConstructionReformState, Scenario
from policyengine_uk.data.economic_assumptions import (
BEREAVEMENT_SPLIT_INPUTS,
STORED_HOUSEHOLD_TOTALS,
extend_single_year_dataset,
reset_growthfactor_uprating,
stored_total_uprating,
uprating_input_fingerprint,
)
from policyengine_uk.data.dataset_sources import materialize_gcs_dataset_url
Expand Down Expand Up @@ -83,6 +85,7 @@
from policyengine_uk.utils.data_source import built_from_data
from policyengine_uk.utils.input_dataframe_export import input_dataframe
from policyengine_uk.utils.measurement_branches import (
drop_calculated_totals,
drop_calculated_values,
retained_input_variables,
)
Expand Down Expand Up @@ -706,7 +709,135 @@ def _isa_stored_exemption(self, period):
# anchor when translating that stored amount in either direction.
return self.calculate(TAX_FREE_INTEREST, period_(period).start.period("year"))

def _record_derived(self, variable_name):
# The value depends on which other periods hold inputs, as an uprated
# one does in Core, so an input set for any period can change it.
history = getattr(self, "_get_store_history", None)
if callable(history):
# Store-history tracking is available in newer Core releases;
# the declared minimum already caches formula results without it.
from policyengine_core.data_storage.store_history import (
next_sequence_number,
)

history().record_derived(variable_name, next_sequence_number())

def _index_growth(self, index_name, source, period):
# Core's uprating factor: the ratio of the index at the two starts,
# or 1 where the index has no value or is zero at the source.
if index_name is None:
return 1
index = self.tax_benefit_system.parameters.get_child(index_name)
source_index = _uprating_index_value(index, source.start)
target_index = _uprating_index_value(index, period.start)
return (
target_index / source_index
if source_index not in (None, 0) and target_index is not None
else 1
)

def _latest_source(self, name, period):
source = self._isa_latest_input(name, period, earlier=True)
if source is None:
# Core first uprates strictly earlier annual inputs, then carries
# an input starting on the query date if there was none earlier.
source = self._isa_latest_input(name, period)
return source

def _abolished(self, name, period) -> bool:
# Core's own test for gov.abolitions, which a reform may supply and
# the baseline does not: without the node, nothing is abolished.
try:
return bool(self.tax_benefit_system.parameters(period).gov.abolitions[name])
except Exception:
return False

def _carries_over(self, variable) -> bool:
# Core's own test for carrying over an input with no index.
return (
self.tax_benefit_system.auto_carry_over_input_variables
and variable.calculate_output is None
)

def _stands_for_stored_parts(self, variable, period) -> bool:
# A stored total means the sum of the parts STORED_HOUSEHOLD_TOTALS
# names for it, each a plain input. Grow it only while the variable
# still sums exactly those parts, each once, with nothing subtracted,
# and each part still reads its inputs: the total's own period, a
# float Core doesn't round, no computation prerequisite, no formula,
# sum or subtraction for this period, no defined_for mask, not
# neutralized or abolished, and an index or Core's carry-over to bring
# an earlier input forward. Anything a reform changes there leaves the
# total to Core, as in data with no stored total.
parts = STORED_HOUSEHOLD_TOTALS.get(variable.name)
if (
parts is None
or not built_from_data(self)
or variable.get_formula(period) is not None
or variable.subtracts
or sorted(variable.adds or ()) != sorted(parts)
):
return False
for name in parts:
part = self.tax_benefit_system.variables.get(name)
if (
part is None
or part.definition_period != variable.definition_period
or part.value_type is not float
or part.requires_computation_after
or part.get_formula(period) is not None
or part.adds
or part.subtracts
or part.defined_for is not None
or part.is_neutralized
or self._abolished(name, period)
or not (part.uprating is not None or self._carries_over(part))
):
return False
return True

def _stored_household_total(self, variable, period):
# A dataset that stores a household total keeps it, projected like
# its parts, in every year (#2740). Core serves the stored total for
# each year the data hold. For a later year it runs the variable's
# sum of person balances, which such data need not hold, so grow the
# latest stored total instead, as Core grows an input with `uprating`.
# Which years hold the total or its parts decides the result.
if not self._stands_for_stored_parts(variable, period):
return None
self._record_derived(variable.name)
source = self._latest_source(variable.name, period)
if source is None:
return None
for part in variable.adds:
part_source = self._isa_latest_input(part, period)
if part_source is None:
continue
native = self.tax_benefit_system.variables[part].definition_period
if part_source.start > source.start or part_source.unit != native:
# Person balances stored for a later year than the total are
# the newer observation, and Core carries a balance stored in
# another unit flat rather than uprating it, so their sum
# stands.
return None
try:
index_name = stored_total_uprating(self.tax_benefit_system, variable.name)
except ValueError:
# Parts that grow by different indices give the total no index of
# its own, so their sum stands.
return None
# The grown total reads none of the parts, so record them: a part
# input that a branch sets later for a later year then replaces it.
for part in variable.adds:
self._record_derived(part)
stored = self.get_holder(variable.name).get_array(source, self.branch_name)
return stored * self._index_growth(index_name, source, period)

def _run_formula(self, variable, population, period):
if variable.name in STORED_HOUSEHOLD_TOTALS:
stored = self._stored_household_total(variable, period)
if stored is not None:
return stored
result = super()._run_formula(variable, population, period)
if (
result is not None
Expand All @@ -720,20 +851,8 @@ def _run_formula(self, variable, population, period):
# This is fallback, after Core's explicit input/abolition checks and
# policy formulas. Preserve the old gross history even past the last
# materialized dataset year; new additive records use Core normally.
history = getattr(self, "_get_store_history", None)
if callable(history):
# Store-history tracking is available in newer Core releases;
# the declared minimum already caches formula results without it.
from policyengine_core.data_storage.store_history import (
next_sequence_number,
)

history().record_derived(variable.name, next_sequence_number())
source = self._isa_latest_input(variable.name, period, earlier=True)
if source is None:
# Core first uprates strictly earlier annual inputs, then carries
# an input starting on the query date if there was none earlier.
source = self._isa_latest_input(variable.name, period)
self._record_derived(variable.name)
source = self._latest_source(variable.name, period)
holder = self.get_holder(variable.name)
if source is None:
gross = holder.default_array()
Expand All @@ -744,16 +863,7 @@ def _run_formula(self, variable, population, period):
== "additive"
):
gross = gross + self._isa_stored_exemption(source)
if variable.uprating is not None:
index = self.tax_benefit_system.parameters.get_child(variable.uprating)
source_index = _uprating_index_value(index, source.start)
target_index = _uprating_index_value(index, period.start)
factor = (
target_index / source_index
if source_index not in (None, 0) and target_index is not None
else 1
)
gross = gross * factor
gross = gross * self._index_growth(variable.uprating, source, period)
return gross - self.calculate(TAX_FREE_INTEREST, period)

@staticmethod
Expand Down Expand Up @@ -1009,6 +1119,8 @@ def reset_calculations(self):
for variable in self.tax_benefit_system.variables:
if variable not in retained:
self.delete_arrays(variable)
elif variable in STORED_HOUSEHOLD_TOTALS:
drop_calculated_totals(self, variable)

def get_known_variables(self):
variables = []
Expand Down
Loading
Loading