Repository navigation
Conversation
The SPI income model predicts from age, gender and region only; applying it to the FRS half gave Universal Credit claimants six-figure dividends unrelated to anything they reported (policyengine-uk#1948). Only the SPI-donor copy is now imputed, as in the microcosm UK build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frs.py summed investment-account dividends to person.index while account.person_id holds household_id * 1e3 + person, so only 8 people got any (about £40m against £8bn). Extract the calculation into a tested helper and annualise with WEEKS_IN_YEAR like the savings lines beside it. Correct the income.py comments that relied on the near-zero FRS dividends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
UK hub (2026-10-12 12:05Z): the hub adopted this PR under d1280's fallback, because it has no live owner session. A Subfleet build job ( |
Retain both dividend and Pension Credit capital helpers; combine person_id dividend keying with main's property-income helper. Keep SPI gainful-self-employment updates while removing the FRS dividend overwrite. Keep the 1.58.0 frozen lock (policyengine-uk 2.122.2, core 3.32.13).
Add a tiny account-to-output regression using real income-draw application and ID-remapping stack output. Property tests conserve eligible annual gross dividends and FRS-half reported amounts. Ignore workspace-local hub job artifacts. Dividend regressions and repository formatting pass; merge-neighbour checks continue in the locked environment.
|
UK hub adoption hand-off — #498 Final head: Mechanism: account dividends now join to Tests: 1,353 passed, 8 absent-data fixture skips across seven targeted files, including all eight dividend regressions. New tiny fixture goes through actual income draws and real ID-remapping stack output; two properties conserve eligible gross account amounts (100 examples) and FRS-half reports/person mappings (50 examples). Worktree venv reconciled with frozen all-extras sync; source/versions verified; serial pytest, one numerical thread, bytecode writes disabled. Impact remains pending. No dataset build or impact microsimulation ran. Hub next: dispatch the independent GPT-6.1 Sol review at this head; copy the explicit Workspace: |
Fold in the investigation patch on top of Round 1 without rewriting history. Keep the account-to-output and conservation regressions, add create_frs holder-keying coverage and assert stage 2 learns from FRS-reported dividends. Correct earnings-group and join documentation, hoist the gross-up constant, and remove the redundant annualisation definition. Conflict resolutions retain the existing gift-aid explanation and all Round 1 tests. Keep precise historical aggregates per d932/notice 9. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
UK hub adoption hand-off — #498 Round 2 Final head: Folded in the investigation patch. Mechanism: Tests: 39 passed, 5 explicit absent-dataset skips across four targeted files. Frozen worktree venv: Python 3.13.9 / UK 2.122.2 / core 3.32.13; serial pytest and one numerical thread. Impact pending: the corrected body marks b45c373/c9b203e results stale before #529. It explains historical UC +£0.38bn with old PE main versus −£0.66bn under #1950, alongside wealth re-imputation, fresh stage-2 training, CGT donors/downstream income and the base FRS H5/tiny change. Precise SPI-donor aggregates remain; the small corrections are 0.34%, 9 bands and last logged epoch 510. No dataset build, impact microsimulation or in-job review ran. Workspace: |
|
Independent code review (GPT-6.1 Sol, UK hub): APPROVE at VERDICT: APPROVE
No blocker or should-fix findings. The code is approved for the batch; keep the PR draft until the queued build is measured. The mechanism is correct. datasets.py:26 groups by account-holder ID, reindexes to the supplied keys and zero-fills misses. The old positional keys lost almost all account dividends. frs.py:1350 now supplies Stage 2 receives the untouched FRS dataset at income.py:531. Dividends enter its predictor frame and model fit. test_imputation_source_flags.py:199 explicitly asserts the reported training vector. The conservation property test:355 checks FRS totals and person mappings through real income assignment and stacking. No assertions or validations were weakened, and no skips or xfails were introduced. Prior findings 1, 2 and 10 are addressed. test_cgt_band_donors.py:230 contains the inherited sixfold reduced-build guard; full builds retain the strict 50% bound. job-r2/PR_BODY.md:28 honestly reports the absent-dataset skip. Lines 34–55 mark b45c373 figures historical, correct the impact channels and numerical errors, and disclose SPI-donor outcomes. Current impact remains pending. Finding 3 is excluded under d932. The build plan:5 supports one seed-0, 512-epoch branch build against the specified saved 1.58.0 artifact. Its baseline-model reuse and fresh stage-2 training decisions have evidence and hash checks. measure_498.py:137 covers FRS/SPI dividends, dividend income tax, UC caseload and spend, and weighted/unweighted UC-unit counts above £50k of member dividends, with source-additivity assertions. Read-only static review; I ran no tests, builds or microsimulations. Existing job logs record 39 passes, five absent-dataset skips and both expected mutation failures. |
This branch has not been deployed
1 participant
WIP — keep this PR a draft. Impact against the published 1.58.0 dataset is pending the hub's queued post-batch build and independent review. This changes a published dataset, including the base FRS H5 files, and lands only in a batch on Max's go (d833).
Part of #495; fixes the data cause of PolicyEngine/policyengine-uk#1948. Original author and commits retained. Round 1 merged main
2b9c1c2c, including #544 / 1.58.0 and the frozen policyengine-uk 2.122.2 / core 3.32.13 lock. The engine already contains uk#1950. Round 2 folds in the investigation session's review patch on top ofa0d0e697, with no history rewrite or relock.Mechanism
FRS account and person keys are
household_id * 1000 + person. The old dividend calculation passedperson.indextosum_to_entity; unmatched grouped keys became zero.frs_dividend_incomeand itscreate_frscall key onperson.person_id, preserving account eligibility/gross-up rules. Raw 2024-25 checks reproduce £0.0398bn → £8.015bn, across 1,717 people, withgross4weights and no orphan accounts. Annualisation uses365.25 / 7rather than 52: a 0.34% difference (0.343%).The second loss was an SPI draw overwriting every original FRS person's reported dividends.
impute_incomenow imputes incomes only on the SPI-donor copy and stacks it with unchanged FRS reports. Since #529, the SPI model predicts from age, gender and region within earnings groups; the former “alone” description is stale. Conservation tests cover FRS reports before calibration; SPI draws and calibrated weights can change total weighted dividends.Conflict resolutions and review changes
Round 1's main merge kept both the dividend and Pension Credit helpers in
frs.py; combined dividend person-ID keying with main's property-income helper; and kept main's SPI gainful-self-employment update while removing the FRS dividend overwrite.Round 2's patch conflicts were resolved as follows:
imputations/income.py: retained Round 1's corrected gift-aid initialisation comment, removed the stale “Impute wealth” comment, and applied the earnings-group description.test_imputation_source_flags.py: added capture/assertion of the stage-2 FRS training dividends before the appended Round 1 tiny-fixture and Hypothesis tests; all tests from both sources are retained.The
create_frsfixture now uses the real nonempty grouped join and a type-8 account.sum_to_entitydocuments the required key space and ndarray return.INVERTED_BASIC_RATEis defined once at module level; the duplicateWEEKS_IN_YEARdefinition is removed. Precise small-cell aggregates remain permitted by Max's d932 / hub notice 9; no revision deletion or suppression is requested.Tests at the Round 2 head
39 passed, 5 skipped across four targeted files (44 cases):
test_frs_dividend_income.py,test_imputation_source_flags.py,test_legacy_benefit_proxies.pyandtest_cgt_band_donors.py. The five skips explicitly require absent FRS/enhanced-FRS files; no dataset build was run. Frozen worktree venv: Python 3.13.9, UK 2.122.2/core 3.32.13/data 1.58.0. Serial pytest (xdist is absent from the lock), one numerical thread and bytecode writes disabled.uvx ruff format --check .passes for all 293 files. Both targeted regression checks reject deliberately incorrect code: reverting the call site toperson.indexfails the holder assertion; training stage 2 on the SPI copy fails the reported-dividend assertion. The source files were restored after each check.Tests retain the real income-draw application and ID-remapping stack path. Hypothesis checks eligible annual gross account dividends over 100 examples and FRS-half totals/person mappings over 50 examples. New coverage pins the actual
create_frscall site (adult account holder gets dividends; child gets zero) and proves stage 2 trains on FRS-reported dividends rather than SPI draws.#535 is resolved: main's
d39371d5, included in the 10/8 batch (#544), bounds reduced-build gains within a factor of 6 in either direction, covering #498's historical 3.6–3.9× values. Full builds retain the strict 50% bound. The old local “main fails too” claim used two OA clones rather than CI's one and was incorrect for CI. The explicitTESTING=1 PE_UK_DATA_OA_CLONES=1 test_built_total_gainsattempt produced 1 skip: Enhanced FRS dataset not available; its gains assertion was not exercised. No reduced dataset build was run by this job.Impact pending the queued post-batch build
.job/BUILD_PLAN.md,.job/QUEUE_ROW.mdand the aggregate-only measurement helper pin 7a9174a for the hub. Update existing queue row 13, currently QUEUED / NO GO, then run one seeded branch build viasetup_build_worktree.shand seed patch 348dd49, 512 epochs, one OA clone, frozen baseline inputs/cache and cached survey-trained models. Compare with the saved 1.58.0 artifact~/PolicyEngine/_worktrees/uk-data-combined-1008/_out/efrs_combined.h5. Measure 2025/2026/2030 dividends split FRS/SPI, dividend income tax, UC caseload/spending, and weighted/unweighted counts of UC units with member dividends above £50k. No build or impact microsimulation has started.The figures below are historical, comparing main b45c373 (before #529) with c9b203e, one unseeded production build per side. They establish mechanisms to measure again, not current-release impact:
Historical UC changed +£0.38bn under old PE-UK main versus −£0.66bn under #1950, where dividends never enter UC unearned income. The difference, about +£1.04bn, is the direct means-test channel from removing imputed dividends. The total movements also include these channels, alongside calibration:
capital_incomeas a predictor; it sees roughly £8.0bn rather than £0.04bn of base FRS dividends. At historical main weights, 1,018 FRS households (about 6%) had changed wealth; gross financial wealth rose £424bn (+7.8%), corporate wealth £131bn (+2.0%), and households above £16k savings plus corporate wealth were 13.72m → 13.71m. Corporate wealth also gates the old UC dividend term.impute_frs_only_variablestrains afresh on the FRS half's reported dividends during the build. Cached.pklmodels do not freeze stage 2. It re-imputes benefit reports, pension contributions and tax-free savings income on SPI donors, while donor wealth and take-up remain separate coherence work.frs_2024_25.h5and its tiny companion change, from roughly £0.04bn → £8.0bn of weighted dividends, separately from enhanced-FRS income preservation.The historical metric named
dividends_in_frs_uc_reporter_units_bncovers both halves (£7.81bn → £16.34bn). Small movements such as UC +0.5%, net income +0.7% or poverty ±0.2pp are not established from one unseeded build per side. Historical calibration comparisons use epoch 510, the last logged epoch, from 512 total epochs; the £12,570–£200k range contains 9 bands. The historical £300k–£500k dividend amount fit worsened 1.021 → 0.510 and banded dividends moved £60.6bn → £59.8bn against £71.8bn; the queued comparison must reassess these trade-offs.Evidence remains in
~/reviews/uc-dividends-1948-2026-09-30/; public text contains aggregates only. Director dividends/net-to-gross conversion, owner-manager inputs, SPI donor wealth/take-up coherence and the matching microcosm change remain outside this PR (#495). The hub dispatches the independent Sol review at the final pushed head. Keep this WIP draft pending review, queued impact and d833 batch go.