Repository navigation
Keep stored household ISA totals after 2032 - #2781
Conversation
A dataset that stores cash_isa, stocks_and_shares_isa or household_lifetime_isa_balance lost the total from 2033 (#2740). Past PROJECTION_END_YEAR, policyengine-core runs a variable's adds instead of reading a stored input, and the person balances it sums are absent from such data. Core does not let a variable declare both adds and uprating. In a simulation built from data, Simulation._run_formula now grows the latest earlier stored total by the index its parts declare, the index Core grows the parts by, and runs the sum only where no total is stored. A situation's household total still applies to its own year only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stored household total now grows only while no person balances are stored for a later year; balances stored after the total are the newer observation, so their sum stands, as on main. The override also leaves a variable to Core when a reform gives it a formula or drops its sum, and records the parts as well as the total as derived. Tests check this through a branch edit instead of Core's store history, pin that an input frame counts as data, and build fewer, larger property examples. The docs name the Lifetime ISA total and the projection's capital gains horizon, and use active voice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ears Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Core lets `adds` name a parameter that lists the parts, or list a parameter beside variables. The stored-total path read each part as a variable, so a reform that summed cash_isa that way raised VariableNotFoundError (for 's', the first character of the path) once a dataset stored the total. Simulation now grows a stored total only when `adds` lists variables, and otherwise runs Core's sum, as it does for a reform formula. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reform that gives cash_isa a part with another uprating index, or none, left stored_total_uprating no single index, and its ValueError stopped the simulation once a dataset stored the total. Simulation now runs Core's sum in that case, as for a reform that sums a parameter. Simulation also records the parts as derived only when it returns a grown total, the one result that reads none of them, and a test checks that a balance set on a branch for a later year than the stored total replaces a grown total the branch has already calculated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The variable documentation, the ISA and dataset-source pages and the changelog now say that the model grows the latest stored total after the last year the data hold, input frames and multi-year datasets included, and that balances held for a later year replace it. The changelog no longer counts stocks-and-shares ISA holdings that corporate_wealth already covers as lost capital, and the edited comments use the active voice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reform that gave a stored ISA total `subtracts` took the grown-total path, which ignores what it subtracts. Simulation now grows a stored total only for a plain sum of variables, and a test covers the subtracting reform beside the parameter and different-index sums, and the ValueError that stored_total_uprating raises for parts that share no index. The variable documentation now says that a reform formula for the balances leaves a stored total unchanged, the docs say the model grows the latest earlier total in each year a source holds none for, the changelog covers frames and multi-year datasets that hold the balances too, and the edited test comments use the active voice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
STORED_HOUSEHOLD_TOTALS now maps each total to the person balances it sums. Simulation grows a stored total only while the variable sums exactly those parts, each once, with nothing subtracted and no formula, and while each part reads its inputs in that year: no formula, sum, subtraction, defined_for mask or neutralization. Any other reform gets Core's own calculation, as on main. Resets that follow an input change now delete the years of a stored total that the simulation calculated, through Core's own input records, and keep every year the data or a caller supplied. A total set for another year, or a part set for a later year, then reaches a year the default branch had already grown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the default branch Core keeps a value it has uprated for a person balance after a balance for an earlier year changes, as on main, so a total that sums the balances reads it. The property now sets totals on either branch and balances on a named branch, and runs two fixed sequences: the grown totals after inputs for 2030 and 2036, and a balance changed after the sum of balances was calculated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reform can abolish a part for a period through gov.abolitions, which the stored-total gate didn't read: the total kept growing while its part and Core's sum fell to zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reform can give a part another definition period, so the gate read a period Core doesn't, or leave a part with no index and no carry-over, so Core reads nothing for it. Either way the total kept growing while the part and Core's sum gave zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Core casts each uprated part to its declared type, refuses a part whose prerequisite hasn't run, and carries a balance stored in another unit flat. Growing the total did none of these. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed 9def3670d166dbcfafc9995809bd3eb56db63b7e. Complete review; no findings or material evidence gaps.
Stored totals, explicit zero, newer observations, situations, reform redeclarations and reset/branch provenance follow the documented contract. Restored totals substitute for their person balances without adding both; capital-source rules remain unchanged. The separate person-input cache limitation in #2774 remains documented.
Validation: all 61 repository tests passed, including all 42 new tests. A two-household native control matched totals-only and person-balance data through final UC/income in 2032 and 2033, including tariff income and the high-capital exclusion. Disabling only the new fallback reproduced the lost-holdings error. The corrected fixture and its initial setup failure are retained separately. Local Core 3.33.2 and locked-Core 3.33.12 hosted CI are distinguished; exact-head checks passed. Private exports and population effects were outside this review.
Fixes #2740
What and why
A dataset that stores the household
cash_isaandstocks_and_shares_isatotals, and not the person balances they sum, loses both totals from 2033. Universal Credit, Housing Benefit, Pension Credit, Income Support, income-based JSA and income-related ESA then count no cash ISA holdings. They also lose stocks-and-shares ISA holdings thatcorporate_wealthdoesn't already cover:share_like_capitalcountscorporate_wealthunless its itemised components, the stocks-and-shares ISA among them, add up to more. #2729 notes that all current Microcosm exports store only the household totals.What policyengine-core does after the last projected year
extend_single_year_dataset(policyengine_uk/data/economic_assumptions.py) copies a single-year dataset into every year up toPROJECTION_END_YEAR(2032, or sooner if a finite capital gains projection ends sooner). It grows each column thatuprating_indices.yamllists, both totals among them, so the dataset stores them through 2032.For a later year, policyengine-core 3.33.12 (the version
uv.lockpins) calculates a variable in this order (Simulation._calculateinpolicyengine_core/simulations/simulation.py):Simulation._run_formulatreatsaddsas a formula and sums the listed variables.adds, does it fall back to inputs from other years. It scales the latest earlier input this branch reads by the ratio of the variable'supratingindex between the two years (_uprating_index_value), or else carries the latest input over unchanged.Variable.check_computation_modes(policyengine_core/variables/variable.py) rejects a variable that declares more than one of a formula,adds/subtractsanduprating. #2712 replaced the totals'upratingwithadds. So from 2033 Core sumscash_isa_balanceandstocks_and_shares_isa_balance, which a totals-only dataset doesn't hold, and the totals fall to zero.The change
STORED_HOUSEHOLD_TOTALS(policyengine_uk/data/economic_assumptions.py) maps each household total to the person balance it sums:cash_isatocash_isa_balance,stocks_and_shares_isatostocks_and_shares_isa_balance, andhousehold_lifetime_isa_balancetolifetime_isa_balance. A test pins the mapping to theaddsof each variable. In a simulation built from data,Simulation._run_formulagrows a stored total before Core's sum runs, while the total still stands for the sum the data stored:addsorsubtracts, nodefined_formask, no neutralization and no abolition for that year, and anupratingindex or Core's carry-over to bring an earlier input forward.The simulation then takes the latest stored total from an earlier year and grows it by the index its parts declare (
stored_total_uprating), with Core's own index ratio.savings,corporate_wealthandlifetime_isa_balance, the variables each total's documentation names, declare the same index,gov.economic_assumptions.indices.obr.per_capita.gdp. Parts with no index hold the total flat, as Core holds such parts.The reset that follows an input change (
drop_calculated_values, whichSimulation.calculateruns after aset_input, andSimulation.reset_calculations) keeps every variable the data supply, all its years included. For a stored total it now deletes the years the simulation calculated and keeps every year the data or a caller supplied, so a total set for another year reaches a total the simulation has already grown for a later year, on the default branch as on a named one. A person balance set for another year does the same, within the limit below. Core's own input records decide which years go:Holder._drop_computedreads_user_input_keysthroughHolder._stores_user_input, asdrop_computed_arraysdoes. Simulation also records a grown total and its parts in Core's store history as derived from other years, as Core does for an uprated input, so Core drops a grown total from a branch when a calculation sets the total or a part there.The reset still keeps every year of any other input, as on
main, including a year Core grew from an earlier input. On the default branch, a person balance set for an earlier year therefore misses a later year Core already grew, and a total summed from that balance takes the old value. Withcash_isa_balanceset to £100 for 2034 andcash_isacalculated for 2040, setting £200 for 2035 leavescash_isaat £117.74 where a named branch and a new simulation give £227.91.maingives the same values, and #2774 tracks the limit.Core's sum still runs:
main;calculate_outputstops it;defined_formask, neutralization, an abolition, another definition period, another value type, a computation prerequisite, or no index and no carry-over);Beyond the two totals in the issue, the change makes these choices:
household_lifetime_isa_balancehas the same shape and loses a stored total after 2032 in the same way, so the change covers it. No means test reads it.built_from_dataalready treats an input frame as data for take-up and random draws, and the change does the same. A total stored in an input frame or a multi-year dataset therefore grows in the years after those it holds, wheremainsummed the person balances. For these sources the change reaches years up to 2032, as the differential below shows. A situation reloaded from its input frame therefore grows a household total that the situation itself applied to one year only.mainswitched to the sum. With equal amounts the two agree within float32 rounding. A stored total of zero stays zero beside positive balances for the same year, as it does up to 2032.maingives zero from 2033 under such a reform, as it does without one.The legacy savings-interest fallback in
_run_formulanow uses the extracted_record_derived,_index_growthand_latest_sourcehelpers. Its behaviour doesn't change.Effect on the issue's example
The issue's reproduction script stores household totals for 2025: £1,000 savings, a £3,000 cash ISA and a £2,000 stocks-and-shares ISA.
cash_isaonmaincash_isaon this branchuc_assessable_capitalonmainuc_assessable_capitalon this branchInvariants
These hold for any non-negative stored amounts. Hypothesis checks the first five, and example tests check the rest:
total(Y) = total(2032) × index(Y) / index(2032).lifetime_isa_balancein both, they also give the same assessable capital for Universal Credit, Housing Benefit, Pension Credit, Income Support, income-based JSA, income-related ESA and Council Tax Reduction in 2025, 2032, 2033 and 2040.savings, tocorporate_wealthor to a household's singlelifetime_isa_balanceequals it, with no tolerance, in every year from 2025 to 2040, under current law and underno_economic_assumptions. A total stored beside its balances for the same year equals their sum in every year, within float32 rounding, with the balances' own index, with the CPI index a reform gives them, and with no index.replace_variableor byupdate_variable, leaves every total in every year from 2025 to 2040, and Universal Credit capital in 2025, 2032, 2033 and 2040, the same as without it, with no tolerance.STORED_HOUSEHOLD_TOTALSnames the parts each total'saddslists, no more and no fewer, each a plain input, and covers every variable thatuprating_indices.yamllists and that sums uprated person inputs.Test plan
make format:ruff 0.17.2 format --checkandruff checkpass on the changed Python files,ruff check --select F811,F821 policyengine_ukpasses, andgit diff --checkreports nothing.changelog.d/2740.fixed.mdResults
The branch merges
mainat 2da212a, the Budget 2026 measures (#2719) included.New module
policyengine_uk/tests/test_stored_household_totals.pypasses 42 of 42 on policyengine-core 3.33.12 and on 3.34.1. Onmainat 2da212a, with stand-ins forSTORED_HOUSEHOLD_TOTALSandstored_total_uprating, whichmaindoesn't define, six of these tests run alone and all six fail. The reproducer reads a £0.00 cash ISA in 2033 where it expects £3,932.42. The others cover a stored zero total, a total set for another year on each branch, resets and a total of several parts.Mutants
Each mutant changes one thing in a scratch copy of the code, and the module kills all 33:
built_from_datacheck)household_lifetime_isa_balanceleft out ofSTORED_HOUSEHOLD_TOTALS>=for>)subtractson the totaldefined_formask on a partaddson a partsubtractson a partreset_calculationskeeps a grown totalcalculate_outputThe first 25 mutants change the code at 7c2b43d, which differs from this head only in comments and the abolition, period, carry-over, value type, prerequisite and stored-unit checks. Seventeen of them, from the top of the table through the reset that deletes supplied years, the third excepted, ran with the module at that commit, whose stateful property test lacked its two fixed sequences. The other eight ran with the module at 6cf11d4, which lacks only the abolition test and three rows of the reform table. The runs from the tenth row through the 25th, and the third row's run, limit both Hypothesis properties to the
explicit,reuseandgeneratephases, which turns off targeting, shrinking and explanation; their strategies, examples and assertions match the committed tests.A time cap stopped the three runs marked * before their last one or two tests, after the failures shown. The run marked † covered only the four tests that read Core's records or set a total for another year, the run marked ‡ covered only the abolition test, and the runs marked § covered only the eight reform-table tests. The Hypothesis test that compares the two data forms kills the four growth mutants on its own, and the stateful one kills the reset that deletes supplied years. The three runs marked ¶ covered only the tests for a cast, a prerequisite and a balance in another unit, at the head.
Regression
Each file ran alone at 6cf11d4, which lacks only the abolition, period, carry-over, value type, prerequisite and stored-unit checks and their tests, on policyengine-core 3.33.12:
policyengine_uk/tests/policy/baseline/pass 108 of 108:gov/dwp/cash_isa_capital.yaml(8),finance/benefit/family/universal_credit/claims_as_single_person.yaml(33),gov/dwp/pension_credit/pension_credit_reported_capital.yaml(9),gov/dwp/lifetime_isa_capital.yaml(24),gov/hmrc/isa/isa_income.yaml(7),gov/hmrc/isa/isa_subscription_limits.yaml(10),gov/hmrc/isa/lifetime_isa_bonus.yaml(4),gov/hmrc/isa/lifetime_isa_withdrawal_charge.yaml(11),household/wealth/isa_balances.yaml(2).policyengine_uk/tests/pass 173 of 173:test_savings_income_uprating.py(134),test_projection_horizon.py(14),test_isa_accounting.py(12),test_isa_balances.py(2),test_measurement_branches.py(5),test_isa_income_consumers.py(3),test_lifetime_isa_capital.py(3).Differential against
mainA script built eight households, two of them pensioner couples, for each dataset and input frame below, and one household of two people for the situation. It compared 17 outputs on
main(2da212a) and on 6cf11d4 in every year from 2025 to 2040: the three totals and their parts,savings,corporate_wealth,share_like_capital,gross_financial_wealthand the seven assessable-capital variables.A second script applied 14 reforms, in seven simulations, each on totals-only data and on an input frame. Six reforms stop a part reading its inputs: they give
cash_isa_balancea formula or a mask, givestocks_and_shares_isa_balancea sum, a subtraction or a mask, and neutralizelifetime_isa_balance. Eight change what a total sums, by giving it a formula, a parameter list, a parameter among its parts, an extra part, a part listed twice, a part with no index, another part, or a subtraction. Every total a reform changes, and every part, matchesmainin every year from 2025 to 2040: 5,280 values, none different. The totals that a simulation's reforms leave alone grow from their stored values, as without a reform.Reproducer
Reproducer: policyengine_uk/tests/test_stored_household_totals.py::test_stored_household_totals_grow_past_the_projection_horizon
axiom: n/a: this restores a data projection and changes no rule.
🤖 Generated with Claude Code