Skip to content

UK: policyengine-uk 2.130.1 and the fourth batch of #1095 ports, with the release's weight year, identifiers and birth months fixed - #1271

Merged
juaristi22 merged 20 commits into
mainfrom
uk-data-ports-1095-4
Oct 15, 2026
Merged

juaristi22 merged 20 commits into
mainfrom
uk-data-ports-1095-4

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

This is the fourth #1095 ports PR. It moves the engine lock to policyengine-uk 2.130.1, ports every item on #1095 that waits for no later engine release, fixes the release's weight year (#796 A19) and identifiers (#1194), and stores each person's birth month in the spine so the release reads the dates of birth its own solve fitted.

All 7 calibrated-seam gates pass on the head, with dwp.uc.households_children_5_or_more held out until 26 November on María's ruling (below). The head's calibration writes a release file, and the weight year, the identifiers and the stored birth months are checked on it.

Before any national build: the renewals of the measure exclusions that lapse from 15 October. After merge, as for the earlier ports PRs: the signed-difference entries for the new and changed columns, whose drafts stay outside the repository until signed, then the national build and the certifier.

Summary

Engine (B1–B3)

  • The lock moves from policyengine-uk 2.123.0 to 2.130.1. The branch first locked 2.127.0. 2.130.1 is the first release published after it (2.128.0 to 2.130.0 never reached PyPI), and María ruled to take it here. 2.131.0, published at 12:29 UTC on 14 October (trading losses), is left for the next bump. policyengine-core moves to 3.33.3 on the UK side, which is what the engine's own floor resolves, and stays at 3.32.5 on the US side; a test pins both.
  • Nothing the UK build writes is renamed, removed or re-typed. Six parts of the build follow what the engine now returns:
  • The other changes the review named need nothing from the build, and the bump arm measures them: statutory sick, maternity and paternity pay as Universal Credit earned income (Count statutory sick, maternity and paternity pay as Universal Credit earned income policyengine-uk#2365), the benefit cap's exceptions (2.129.0) and its earnings test with statutory pay (Fix benefit-cap statutory-pay earnings exemption policyengine-uk#2399), and income ESA and JSA ending in remunerative work (2.129.1). Capital gains get their own projection path in 2.130.0, but with no scenario passed it copies GDP per head exactly, so the build's gains are projected as before.
  • Re-derived with no change to data: the uprating pins, the concept coverage (288 inputs to 299) and its mapping, the road-fuel ratios in the donor-uprating audit, the H2 fixture, the coverage manifest's digests and the approved lock hash. The target references and the compile-parity receipt regenerate identically.

Spine fixes (R1–R4)

Two of these port merged incumbent code; the others go first, so a divergence shows if the incumbent lands differently.

Targets (T1)

Export (E1–E3) and the birth months (M)

  • E1. Weights at the dataset's year (WS-E spine deferrals: known bugs and signed inconsistencies awaiting their increment #796 A19). Goes first: Runtime calibration-year weights are 0.44% below the calibrated ones (base-year materialization and policyengine-uk's uprating use different weight indices) policyengine-uk-data#538 is the incumbent's open issue. policyengine-uk multiplies household_weight by ONS population growth for each year after the dataset's own, so weights solved at 2025 and stored on the 2024-25 dataset read 0.72% high at 2025 on every weighted count and total. The release write now divides by the engine's own factor. The rule-1 scorer reads each side with the weights the engine gives it at the scored year, so the incumbent's estimates are read 0.72% higher than before. Scores are not comparable with earlier ones.
  • E2. Identifiers inside 32 bits (UK clone stages can number identifiers past the 32-bit range policyengine-uk stores #1194). Goes first: OA clones number identifiers past the 32-bit range policyengine-uk stores policyengine-uk-data#558 is an open issue. The clone stages number copies past 2,147,483,647 and the engine stores identifiers as 32-bit integers, so on main's file more than half of them wrap. The release boundary renumbers, one plus the rank of the build identifier per entity, and keeps the build identifiers as person_build_id, benunit_build_id and household_build_id.
  • E3. The calibration measures the engine on the release's identifiers. Microcosm only. Measured on 2.127.0, E2 alone made the release differ from its solve on two-child-limit and Universal Credit award rows, because the engine dealt birth months by a hash of person_id.
  • M. The birth months are stored in the spine. Microcosm only; María asked for it in this PR. policyengine-uk places each date of birth from the age and months_since_last_birthday, and left unset it spreads each age and sex over the year by weight, in an order set by a hash of person_id. So a release, at the calibrated weights, read different dates of birth from those its solve fitted at the spine's weights, on main too: the two-child limit, the State Pension scheme and the Pension Credit qualifying age depend on them. The root stage now draws the months once, the engine's way, at the FRS weights, and every later read takes the stored value.
    • Read as the engine reads it, a release file without M differs from its own solve on 55 of 1,217 rows, 8 of them by more than a percentage point; the State Pension rows by scheme and amount move most. With M, 3 rows differ, and no State Pension, two-child-limit or Pension Credit row moves. What is left (private-school pupils and two land rows) depends on the weights themselves, and is main's too.

The row held out (X)

dwp.uc.households_children_5_or_more is held out of the calibration until 26 November 2026, on María's ruling. DWP counts about 21,000 more Universal Credit households with five or more children than its two-child-limit counts of families with five and six or more. On main the frame tells the two counts apart only through fewer than 10 source households, where a dependant in training counted as a child in the first row but not in the two-child-limit rows. R2 fixes that, the two counts then agree on every family, and the calibration cannot meet both. The two-child-limit rows stay bound. With the row held out, the head passes all 7 gates.

Measurement

Four arms on feed 942a1fa, each a spine build and a national calibration, all with --review-date 2026-10-14 for measurement only. Effective sample sizes are not given.

  • control (main 65d7d153a, policyengine-uk 2.123.0): loss 0.00583; 98.8% of 1,214 rows within 10%; all 7 calibrated-seam gates pass.
  • bump (main plus only the three lock commits, policyengine-uk 2.130.1; measurement only, not in this PR): loss 0.00593; 98.6% of 1,214; all 7 gates pass. Before calibration 169 rows move by more than 1%. On the rows the review named, at the spine's weights: Universal Credit +0.4%, 0.8% fewer capped households, ESA −3.0% with 5.1% fewer income-related claimants; Pension Credit is +6.5% and Carer's Allowance −5.5%, as on 2.127.0.
  • head without M (the head less the stored birth months; measurement only): loss 0.00588; 98.7% of 1,217; all 7 gates pass.
  • head (1da8d681f, the branch before its rebases): loss 0.00590; 98.8% of 1,217; all 7 gates pass. The input-mass gate passes on its calibrated weights.

On the current main. The branch is now rebased onto main 121fd4dd6 (#1256, #1279). #1256 draws the SPI channel's incomes afresh, which changes the spine, so a control and the head were rebuilt on it:

  • control (121fd4dd6): loss 0.00584; 98.4% of 1,214 rows within 10%; all 7 calibrated-seam gates pass.
  • head (0912388b6): loss 0.00599; 98.6% of 1,217; all 7 gates pass. On its written file the weight year, the identifiers and the stored birth months check out as before, the input-mass gate passes, and the file differs from its own solve on 3 rows (private-school pupils and two land rows).

The decomposition into the bump, the spine fixes and the birth months was measured on 65d7d153a and is not repeated. Rebasing onto #1242 changed one test, which now tells this PR's 2026-10-14 exclusion apart from #1242's housing-element renewal of the same date.

Receipts: experiments/1095-uk-data-ports-4-receipts.md. Survey support is given in bands of distinct source households (<10, 10–29, 30+). A cell under 10 has no weighted figure, and row figures are given only for rows with 30+ households behind them.

Not in this PR

Tests

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22

Needs changes before merge. The ports themselves check out and every source change is pinned by a test that fails when it is reverted. Two things block: the receipts commit effective sample sizes, and the head fails its own uk_target_fit gate until you rule on the row. I also think the 2.127.0 lock deserves a decision before merge, because the engine releases after it fix rules this build calibrates on.

  1. must. experiments/1095-uk-data-ports-4-receipts.md:62, 64, 65, 67, 70, 123, 174 give each arm's effective sample size. That is on the never-publish list for this microdata. Please remove them or replace them with a band or a ratio to the control. The PR body correctly leaves them out, so the receipts should match.

  2. should: the 2.127.0 lock misses engine fixes that reach this calibration. policyengine-uk 2.130.1 has been on PyPI since 01:58 UTC on 14 Oct. The 2.128.x to 2.130.0 releases never reached PyPI because the release job failed on the benefit-cap earnings property from 12 Oct, so 2.130.1 is the first published version after 2.127.0. I ran synthetic households on 2.127.0 (the lock) and on 2.130.1:

    • Benefit cap earnings test (pe-uk#2399, 2.130.1). In 2.127.0, is_benefit_cap_exempt_earnings takes pay less all income tax and NI, so it deducts the tax and NI on statutory pay without counting the pay. Take a London lone parent with three children, £10,600 pay and £4,000 SSP. On 2.127.0 she loses the exemption and is capped by about £10,250 a year. On 2.130.1 she is exempt. The spine stores SSP and SMP, and dwp.benefit_cap.capped_households and the up-to-£100 band stay in the objective (calibration_measure_exclusions.json). So part of the fit to DWP's capped count comes from families that 2.127.0 caps wrongly. Main's 2.123.0 has the same formula, so this PR does not make it worse. But it is the fix that unblocked the release, and this PR is the point where the lock moves.
    • SSP and SMP in Universal Credit (pe-uk#2365, 2.128.0). In 2.127.0 the NI and tax on statutory pay come off UC earnings but the pay itself is not counted. In the same household with £13,000 pay, adding £3,000 SSP raises UC by about £460 on 2.127.0 and lowers it by about £1,190 on 2.130.1. That affects every SSP and SMP record on the spine and the UC rows they feed.
    • 2.129.0 rewrites the cap's ESA and disability exceptions (claimant and partner only, ESA support component). 2.128.0 moves legacy tax-credit families onto UC from 2025-26 (pe-uk#2119), and later versions uprate capital gains by their own index instead of GDP per head. All three will move the UC, tax-credit, benefit-cap and CGT rows, so a bump needs its own arm, not a quiet lock edit.

    Which way to go is your scope call. One option is to move to 2.130.1 here and re-derive the pins, concept coverage, H2 fixture, lock hash and arms. The other is to merge at 2.127.0 and say in the body and the changelog fragment that the benefit-cap and UC rows are calibrated on an engine with these known gaps, with 2.130.1 as the next bump. Either way, the body's "the latest release on PyPI" (Summary, Engine) is no longer true and should change.

  3. should: R3 lowers the income of the households it touches on the locked engine. In 2.127.0, statutory_paternity_pay is only a Class 1 NI base. It is not in household_benefits, HBAI income or UC earned income. Those were added in 2.128.0 by pe-uk#2365, which is not an ancestor of the 2.127.0 tag. In a synthetic household, £5,000 of SPP raises NI by £400 and lowers both household_net_income and hbai_household_net_income by £400. On 2.130.1 the same SPP raises net income by £4,600. So at frs_spine.py:700, the new column moves these households' income the wrong way until the engine catches up. Support is under 10 households, so the effect is small, but the column ships in the release. Either hold R3 until the lock reaches 2.128.0 or later, or say this in changelog.d/1095-statutory-paternity-pay.added.md. The fragment says now which engine variables read SPP, but not that it reduces net income. Separately, uk-data#575 leaves open whether SPPADJ × 52 is the right annual amount for a payment that lasts at most two weeks. Matching SSP and SMP is a fair default, but the receipts should record that the question is still open.

  4. should: the failing uk_target_fit row needs a recorded ruling, not only an arm. Until you rule, the head cannot write a release file. My view, as a calibration judgement rather than a code point: the R2 fix is right, and it matches uk-data#561 (trainee == 1 on a child-table record is the same rule as is_trainee & ~is_adult_record). Under 10 source households were what separated DWP's two counts, so the frame has no real support for the roughly 21,000 large UC families outside the two-child limit. That makes holding dwp.uc.households_children_5_or_more as a diagnostic defensible, as is a lower loss weight. Whichever you choose, it should go into calibration_measure_exclusions.json with a reason and an expiry in this PR, so that the head passes its gates as merged rather than only in the measurement-only local commit.

  5. nit. The body calls every R and E item a port, but only uk-data#526 and Rebase stale HT2 taxable-interest rows onto the Pub 1304 national actual (#489 adjudication) #518 are merged. Finish #515: per-record E19200 decomposition by published SOI 2.1 AGI-band components #561 is an open PR, and calibration_diagnostics.json has no declared schema and is not validated on write #575, Staging telemetry silently disabled since 2026-07-02: empty POPULACE_STAGING_REPO_ID defeats the on-by-default staging repo #563, QBI v3 wiring: simulation consumes SCF/SOI/JCT evidence (shape from SCF, level from JCT) #538 and Read HMRC's joint distribution of capital gains by taxable income #558 are open issues. It would help to say which items port merged incumbent code and which go first, so any divergence is visible if the incumbent lands differently.

  6. nit. The test counts differ between the body and the receipts. The body has the engine-free job list as 17,384 tests on 9629176f0. The receipts (Tests) have 17,381 on ffa13ec29.

What I checked

  • Worktree at ed33feb76 with a fresh uv venv (uv sync --all-packages --locked --extra uk, which resolves policyengine-uk 2.127.0).
  • ruff check . passes. ruff format --check would reformat 45 files, none of them touched here, and CI does not run it. tools/ci_test_plan.py verify and tools/graph_acceptance_burndown.py --verify pass.
  • All 29 touched test files, engine-free and engine-uk, pass. Five tests skip because the Chronicle feed is not present here.
  • Reverts and mutations, each of which fails the new tests:
    • R1: code 11 back to LONG_TERM_DISABLED.
    • R2: the base frs_education.py.
    • R3: the base frs_spine.py.
    • R4: three mutations (the pay limit dropped, the half-pay cap dropped, and training on unpaid respondents).
    • E1: the factor set to 1, and the full-build write without the division, which test_uk_rowwise_national_role catches.
    • E2: no renumbering at the export.
    • E3: the scratch file left on build identifiers, which the engine birth-month test catches.
  • R4 against the merged uk-data#518 diff, and R2 against the Finish #515: per-record E19200 decomposition by published SOI 2.1 AGI-band components #561 diff.
  • policyengine-uk 2.127.0 against 2.130.1 and main: the changelog, is_benefit_cap_exempt_earnings, where statutory_paternity_pay is read, uprating_indices.yaml, and the ONS population growth that E1 divides by, which is unchanged after 2.127.0, so the factor holds on 2.130.1. I ran household probes on both versions.
  • Disclosure scan of the body, the receipts, the changelog fragments, the signed differences and the spine fixture. The fixture is generated by tools/graph_uk_spine_fixture.py and is synthetic.
  • The PR is mergeable. main's commits since the base are US and NZ only.

I couldn't build a spine, run a calibration or check any figure in the receipts, because I have no FRS or feed access. I also couldn't run the full engine-uk or engine-free CI job lists or the feed-gated tests here.

🤖 Generated with Claude Code

juaristi22 and others added 19 commits October 14, 2026 21:16
The lock's UK side moves exactly to 2.127.0, the latest release on PyPI
(uv lock --upgrade-package policyengine-uk==2.127.0), and the UK extras
of microcosm-build, microcosm-data and microcosm-frame require
>=2.127.0. policyengine-core moves to 3.33.3 on the UK side, which is
what policyengine-uk's own floor (>=3.32.27) resolves, and stays at
3.32.5 on the US side (#1086's per-country pins). Nothing else in the
lock moves. A new test pins the engine and core every UK install path
resolves, so a later move of either is deliberate.

Nothing the UK build writes is renamed, removed or re-typed between
2.123.0 and 2.127.0. One engine test changes its expectation:
PolicyEngine/policyengine-uk#2084 assesses a claimant whose partner
cannot claim as a single person (UC Regs 2013 reg 3(3)), so the test of
recorded claimant roles now expects the single young allowance for a
24-year-old with a 17-year-old partner. The recorded roles still reach
the engine unchanged.

Re-derived with no data change: the uprating pins (the version stamp,
and the employer Class 1 rate at 2022, 0.138 to 0.1505; the target
references regenerate identically on the pinned feed), the concept
coverage (still 288 inputs: brma, uc_reported_capital,
pension_credit_reported_capital and uc_is_ineligible_partner leave the
input list as formulas, and benunit_reported_capital,
uc_receives_new_claimant_health_element, uses_energy_prepayment_meter
and ebr_energy_bills_credit_eligible join it uncovered), the mapping
reviewed against 2.127.0 (the liquid-asset reason now says
uc_reported_capital is formula-owned), the two engine road-fuel ratios
in the donor-uprating audit (0.852 and 1.032 to 0.946 and 0.927, after
PolicyEngine/policyengine-uk#2186), the spec's declared dependency and
the H2 fixture's, the coverage manifest's source digests, and
APPROVED_UV_LOCK_SHA256.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four texts described the locked engine as 2.123.0: the frs_take_up and
uc_capital_coherence stage notes, the comment on the Universal Credit
take-up population and the comment on the property capital sources,
plus one test docstring. Each claim was re-read against 2.127.0 and
holds: is_uc_eligible still needs a claimant who meets UC's minimum age
and has not reached Pension Credit qualifying age (it now takes the
single or joint claimants from is_uc_claimant), and the Universal Credit
and Pension Credit capital source lists are unchanged.

Left as they are: texts that name 2.123.0 as the release that added the
landlord inputs, and the signed entries.

Regenerated: the rendered stages, the coverage manifest's source
digests and the H2 fixture's stage notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EMPSTATI 11 is "Other Inactive" in the UKDS FRS 2024-25 data dictionary
(adult table; 9 is "Permanently sick/disabled"). The UK runtime kept the
incumbent truncated map's artifact and sent 11 to LONG_TERM_DISABLED, plus
a post-map fillna that sent any code above the domain there too. That put
other-inactive adults into the ESA health-condition and support-group
proxies.

FRS_EMPSTATI_EMPLOYMENT_STATUS now maps codes 1-11 one data-dictionary
label per line, matching policyengine-uk-data#526. People outside adult.tab
stay CHILD; an adult with a blank or unknown code refuses the build, with
the error naming the codes and suppressing adult counts under 10.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 602fae2)
The stage notes in uk/source_stages.json and uk/spec/sources.yaml (and the
charter-H2 fixture's copy) described code 11 as the incumbent
truncated-map artifact. They now state the data-dictionary mapping, the
CHILD rule for people outside adult.tab and the refusal for blank or
unknown adult codes, citing uk-data#526 the way other stage notes cite the
incumbent (the source manifest refuses the package name).

release_input_coverage_manifest.json pins sha256(source_stages.json) in
all 15 families; regenerated with
tools/build_uk_release_input_coverage_manifest.py (--check passes). The
notes are outside stage_contract_sha256, so uk_spine.json does not move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit a2c57ab)
- The changelog fragment cites uk-data#526 instead of the incumbent
  package name, which test_no_incumbent_data_package_references_in_live_tree
  refuses (CI engine-free's only failure).
- The ESA support-group property asserts equality with reported, working
  age, code 9 and no hours.

The original commit also updated the receipts, the differential against
uk-data#526 and the mutation check under
experiments/uk-frs-empstati-other-inactive/. That folder is not carried
on this branch (María's instruction of 2026-10-14), so those parts of
the commit are left out; they remain on #1097.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c376188)
Adults are not survey households, so even a count of 10 or more adults
could describe fewer than 10 households in a build log. The refusal now
names the offending codes only (uk-data#526 made the same change at
1832adeb). A Hypothesis test checks that no digit appears outside the code
list.

The original commit also re-pinned the differential against uk-data#526
and updated the mutation check and the receipts under
experiments/uk-frs-empstati-other-inactive/. That folder is not carried
on this branch (María's instruction of 2026-10-14), so those parts of
the commit are left out; they remain on #1097.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 8c51a1c)
Ports uk-data#561. is_in_approved_training was TRAIN2 in the scheme
codes 1 to 8 or 10, a reading #1063 added. Only the adult table has
TRAIN2, so the stage flagged adults on DWP employment schemes (Work
Programme, Kickstart and others) and never a child-table record.

The FRS counts a 16- to 19-year-old as a dependent child only in
full-time non-advanced education or unwaged government training. So a
child-table record with TRAINEE = 1 is a dependant in approved
training, and that is now the rule. No adult-table record is flagged:
the FRS has already decided it is not a dependant, and its TRAINEE
answer does not say the training is unwaged (UC Regs 2013 reg 5(3)).

Each tab is marked with its table before the two are concatenated, as
frs_relationships does, and a tab without TRAINEE refuses the build
rather than reading as nobody in training. The entry age and the age-19
terminal-date flag read the training flag, so they follow.

The synthetic FRS tabs in the tests and the H2 fixture carry TRAINEE in
place of TRAIN, with the same flags as before. Regenerated: the rendered
stages, the coverage manifest's source digests and the H2 fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uk-data#575. The root stage set statutory sick and maternity pay from
SSPADJ and SMPADJ and nothing set statutory_paternity_pay, so every
adult had none. It is now SPPADJ times the weeks in a year. The FRS
2024-25 variable listing has SPPADJ on the adult table as "SPP amount",
a weekly amount for the adult's main job, beside the other two.

policyengine-uk reads the input in Class 1 National Insurance earnings
and in the Universal Credit and Tax-Free Childcare paid-work tests.

A tab without SPPADJ refuses the build rather than set nobody's. The
column is net-new on the export: it joins the root's outputs and
non-negative outputs, the reviewed extra columns of the export-surface
gate, and the synthetic FRS tabs. Its signed net-new-column entry stays
outside the repository until signed, and gives survey support as a band
only: few adults report paternity pay.

Regenerated: the rendered stages, the coverage manifest's source
digests, the H2 fixture (the root's person schema and rows) and the
gate-battery digests with their mirrors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the final rule of uk-data#518. A salary sacrifice is pay given up
for a pension contribution, and three parts of that rule were missing.

Every amount, reported or predicted, is now limited to the pay the
person still receives, and a person without pay has none. #1089 had
done this only for people not asked the question; respondents kept
their reported amount whatever their pay. The limit bites above all on
SPI-channel rows, which keep their FRS twin's answer beside pay redrawn
from the SPI.

The forest trains only on respondents with pay.

The headcount conversion moves a contribution into salary sacrifice up
to half the pay and leaves the rest as an employee contribution. It
moved the whole contribution before and floored pay at zero, so a
contribution at or above the pay left a sacrifice with no pay behind
it, which policyengine-uk 2.127.0 gives no relief
(PolicyEngine/policyengine-uk#2063). Pay plus sacrifice still equals
the pre-conversion pay on the carrier, exactly. The stage refuses if
any sacrifice ends above the pay that remains.

The stage's manifest declares the training population with pay, the
limit as clamp_maximum and the half-pay cap as move_pay_share_cap (two
optional keys in the sources schema), and the closed-world assert binds
each to its constant. The evidence records what the limit and the cap
touched. The identity tool restores a converted record's contribution
as what it kept plus what moved, and refuses an artifact whose pay and
sacrifice do not add up to the carrier.

Regenerated: the rendered stages, the coverage manifest's source
digests and the H2 fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uk-data#563. The two national rows for ages 15 to 19 by sex fit a
five-year band, which leaves the ages inside it free to move: uk-data
found weights that fit the band while holding 35% too many 16-year-olds
and about half the 18- and 19-year-olds. Nothing in this contract bound
the mix inside the band either.

Each row becomes three: age 15, ages 16-17 and ages 18-19 (María's
ruling of 2026-10-13). The ages 16 and 18 are where the child and
qualifying-young-person rules change. The six rows read the same ONS
mid-year estimate by single year of age, so each sex's three rows sum
to the band they replace; the five-year rows are dropped, since kept
beside the new ones they would be an exact linear combination of them.

Registry accounting: the incumbent's ons/female_15_19 and
ons/male_15_19 move from mapped to excluded, because no single row has
their definition, and the six rows are contract-only declarations. The
contract has 400 targets (396) and 1,283 active references (1,279).

Regenerated on the pinned feed 942a1fa: the target references, the
membership report and the 2025 compile-parity receipt (1,283 compiled;
the two incumbent rows signed fixture-only and the six signed
ledger-only, each with its rationale).

Effect on the fit: every other ons_population row's loss weight falls
to 149/153 of what it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#796 A19; uk-data#538 is the incumbent's side of the same
mechanism. policyengine-uk projects a single-year dataset forward: from
the dataset's own year it multiplies household_weight by one plus the
ONS population growth of each later year. Weights solved at 2025 and
stored on the 2024-25 dataset therefore read 0.72% high at 2025, on
every weighted count and total.

The release write now divides the solved weights by the engine's own
factor, on a derived frame, so the engine returns the solved weights at
the solve year. New module uk_runtime/weight_year.py; called at the
national write, in the national readback's expectation, and in the
local export's descriptor and write. The division is the last record of
the file's mass log. The solve, its gates, its diagnostics, the spine
checkpoints and the engine scratch file keep the solved weights.

The growth is vendored in a new dataset_projection block of
uk/hmrc_uprating_engine_pins.json, written by the pins tool. Engine
tests hold it against the installed engine, check that household_weight
is the only weight the engine projects, and load a written file to read
the solved weights back at 2025 (with the mutation: a file written as
solved reads high by the factor).

Readers of a written file follow the same projection:
- the rule-1 scorer reads each side with the weights the engine gives
  it at the scored year, and its score block gains weight_year. Scores
  are not comparable with earlier ones;
- the release-cut certifier runs its gates on the solved weights and
  refuses a file divided for another year;
- tools/evaluate_uk_incumbent_surface.py reads the candidate the same
  way;
- the assembler's build manifest says which year the weights NPZ stands
  at. The NPZ pairs with the H5 row by row, so it carries the file's
  own weights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1194; uk-data#558 is the incumbent's side. The clone stages
number each copy by adding a power of ten to every identifier, so a
built frame's identifiers run far past 2,147,483,647. policyengine-uk
keeps its five identifier variables as 32-bit integers: the stored
values wrapped, calculate("household_id") did not return the file's
identifiers, and every engine draw keyed on one read the wrapped number.

The release boundary now renumbers. Per entity the identifier is one
plus the rank of the build identifier; the person table's two
memberships map through the same arrays; row order does not change.
national_frame.uk_release_identifier_tables does it, called by
uk_release_export_frame and by the local export. The build identifiers
stay in the file as person_build_id, benunit_build_id and
household_build_id, which the existing lineage columns describe, and
are allow-listed on the export surface (gate digests re-pinned).

The build, its checkpoints and the engine scratch file keep the build
identifiers, because every draw in the build is keyed on them.

Both readbacks refuse a written identifier that repeats, is below 1 or
is above the 32-bit range, naming the column and a count. The release
assembler pairs a release with its spine on the build identifiers; the
size evaluation and tools/verify_uk_identity_stability.py read them too.

Engine tests hold both halves against the installed engine: the
renumbered file's identifiers read back as written, and the build's own
do not.

Consequence, measured in the receipts: policyengine-uk spreads each
person's birth month over the year by a hash of person_id
(months_since_last_birthday, which the State Pension age and Pension
Credit qualifying-age tests read). It deals those months on the
release's identifiers, where the calibration's scratch file dealt them
on the wrapped build identifiers. The Universal Credit deduction draws
do not move: the build stores them.

The full-gates tests now build a real national frame: their stand-in
had no person table for the release boundary to renumber.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The readbacks' identifier check said how many rows were out of range.
A refusal is read in logs and reports, and a count of rows can be a
small one, so it now names the column and nothing else, as the EMPSTATI
refusal does: never an identifier, never a count.

Wording only. The check and what it refuses are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the measurement of the renumbering (microcosm#1194).
policyengine-uk spreads each person's birth month over the year by a
hash of person_id, and the date of birth decides the two-child limit,
the State Pension scheme and the Pension Credit qualifying age. The
calibration measured the engine on a scratch file that kept the build
identifiers, so it fitted one deal of birth dates and the renumbered
release was read with another. On a calibrated file that moved
two-child-limit and Universal Credit award rows by up to a tenth or
more against the solve's own estimates.

The scratch file now carries the release's identifiers: one plus the
rank of the build identifier, per entity, with the memberships mapped.
A per-clone engine block takes the ranks its rows have in the whole
frame (uk_release_identifier_tables gains universe=), so the local
solve numbers its rows as the local release does. No build-identifier
column goes to the engine. The resolver's own frame keeps the build
identifiers, and nothing joins engine outputs to it by identifier.

This replaces the plan's rule that the scratch file keeps the build
identifiers. The spine stages still read the engine on the build
identifiers, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1095)

An engine test for the commit before: on forty people of one age and
sex, the engine file the calibration measures on and the released file
give the same months_since_last_birthday, and a file on the build's
identifiers gives others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lock moved to 2.127.0 earlier on this branch. 2.130.1 is the first
release published after it: 2.128.0 to 2.130.0 never reached PyPI. The
review asked for it because those releases fix rules the calibration
reads, and María ruled to take it here (2026-10-14). uv moved only
policyengine-uk; the UK extras now require >=2.130.1.

What 2.128.0 to 2.130.1 change that the build reads, and what follows:
- Gift Aid extends the basic rate band instead of reducing allowances
  (pe-uk#2271). The CGT taxable-income measure is net income less the
  allowances, as the engine now stacks gains on it; its hand-computed
  test and the engine inversion test follow. earned_income_tax extends
  a donor's bands by the grossed-up gift (ITA 2007 s.414(2)(b)), so
  the salary-sacrifice relief by rate class extends them too, as the
  engine's calc_income_tax_with_gift_aid does; without it the national
  calibration refused, the classes no longer adding up to the engine's
  tax for donors. The three relief rows declare the extension
  (gift_aid_extends_bands) and the measure refuses a binding without
  it; hand-computed donor cases, engine-free and against the engine.
- A child on the working parent childcare entitlement keeps the
  universal hours (pe-uk#2177). The universal-only row now leaves the
  extended entitlement out itself, as the published difference does,
  and the lockstep test checks both halves.
- Families move onto Universal Credit when DWP closes a legacy benefit
  they receive, tax credits from 2025-26 (pe-uk#2119). The UC take-up
  lockstep test names a third case the population does not mirror (a
  pension-age family moved at the closure, SI 2014/1230 reg 60A). The
  concept mapping declares would_claim_uc_at_legacy_closure left at the
  engine's default, true: uk-data#534 draws it at DWP's Move to UC
  claim rates, which are not yet a Chronicle fact.
- Statutory sick, maternity and paternity pay count as UC earned
  income, and paternity pay in household benefits and HBAI income
  (pe-uk#2365). The paternity-pay fragment now says so, and that the
  annualisation of a two-week payment is an open question (uk-data#575).
- The liquid-asset reason names the three company-capital inputs of
  UC's regulation 77 (2.130.0); they default to no company and the
  build sets none.
- The benefit cap's exceptions follow the claimant and partner (2.129.0)
  and its earnings exception counts statutory pay (pe-uk#2399); income
  ESA and JSA end in remunerative work (2.129.1). Nothing in the build
  mirrors those; the arms measure their effect.

Re-derived: the uprating pins (only the version stamp moves; the ONS
population growth E1 divides by is unchanged), the concept coverage
(288 inputs to 299), the stage texts citing the locked engine, the
rendered stages, the coverage manifest's digests, the H2 fixture and
APPROVED_UV_LOCK_SHA256. The target references and the compile-parity
receipt regenerate identically on feed 942a1fa.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

policyengine-uk places a date of birth from the whole age and
months_since_last_birthday, and the date decides the two-child limit,
the State Pension scheme and the Pension Credit qualifying age. Left
unset, the engine spreads each age and sex over the year by weight, in
an order set by a hash of person_id. So a file's birth months moved
with the weights and identifiers it carried, and the release, at the
calibrated weights, read a different spread from the one the solve
fitted.

The root stage now draws the months once, the engine's way: at the FRS
weights, in an order set by an identity-keyed draw on the person's
stable identifier (uk_runtime/birth_month.py). Every later engine read
takes the stored value, and a copy of a person keeps their months. The
column is an output of frs_spine, allow-listed on the export surface;
the stage notes, coverage manifest, H2 fixture and gate digests are
regenerated.

Tests: engine-free tests of the spread (order-free, even within each
age and sex, keyed on the person) and engine tests that the spread
equals the engine's own stratified_uniform and that stored months do
not move with the weights while unset ones do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
María's ruling of 2026-10-14 on Vahid's review of microcosm#1271
(item 4). Once dependants in approved training qualify as young people
(the TRAINEE fix, uk-data#561), fewer than 10 FRS source households
separate DWP's count of Universal Credit households with five or more
children from its two-child-limit counts of five and of six or more.
The row passed the fit gate before only through the dependants the
missing flag miscounted. The calibration cannot meet both counts, and
the two-child-limit rows carry six times the loss weight, so
dwp.uc.households_children_5_or_more fails uk_target_fit and a blocked
calibration writes no release.

The row is held out with a reason, the tracking issue and the
sparse-band expiry (2026-11-26), and the two-child-limit rows stay
bound. The packaged-exclusions test pins the entry and its ruling, and
tells it apart from main's housing-element renewal of the same date
(#1242) by tracking issue; the R2 changelog fragment says what the fix
leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebase onto main 121fd4d (#1256, #1279) settled the generated
files with main's side. Regenerated on the rebased tree: the gate
battery digests (main's export allow-list line beside this branch's),
the coverage manifest's source digests and the H2 fixture, whose
synthetic CGT asset-type facts follow the locked engine. The rendered
stages are unchanged. uv.lock merged cleanly (microcosm-emitter 0.2.1
from PyPI beside policyengine-uk 2.130.1), and the lock commits pin
its new digest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-data-ports-1095-4 branch from db2b824 to 779577a Compare October 14, 2026 20:25
Four arms on feed 942a1fa: main, the 2.130.1 lock alone, the PR
without the stored birth months, and the PR. What the bump moves and
how the two fit; what the spine fixes move; the row held out and why;
the age pieces measured on main's file; the weight year, the
identifiers and the stored months checked on the head's written file
with the engine; the written file against its own solve with and
without the stored months; household SDLT by part; and the tests.

Survey support is given in bands of distinct source households, a
row's figure only where 30+ stand behind it, and no effective sample
size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-data-ports-1095-4 branch from 779577a to fcb6ff2 Compare October 14, 2026 21:19
@juaristi22 juaristi22 changed the title UK: policyengine-uk 2.127.0 and the next #1095 ports, with the release's weight year and identifiers fixed UK: policyengine-uk 2.130.1 and the fourth batch of #1095 ports, with the release's weight year, identifiers and birth months fixed Oct 14, 2026
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@vahid-ahmadi thanks, all six are addressed. The branch is force-pushed: it is rebased onto main 121fd4dd6 (#1256, #1279), so every SHA has changed, and the old receipts commit is replaced rather than edited, so no commit on the branch carries an effective sample size. Head: fcb6ff23e. The four arms in the receipts ran on main 65d7d153a; #1256 changes the SPI channel's draws, so a control and the head were rebuilt on 121fd4dd6: the control fits at loss 0.00584 (98.4% of rows within 10%) and the head at 0.00599 (98.6%), both with all 7 gates, and the head's written file checks out as before.

  1. Effective sample sizes. Removed. The new receipts (fcb6ff23e) compare the arms on loss, rows within 10% and gates only, and give no effective sample size, absolute or relative.

  2. The lock. María ruled to move to 2.130.1 here, with its own arm. 3623ca57d locks it and re-derives the pins, concept coverage (288 to 299 inputs), H2 fixture, lock hash, stage texts and coverage manifest; the target references and compile-parity receipt regenerate identically. The bump arm is main plus only the three lock commits, so the engine's effect is measured on its own. Against main it fits alike (loss 0.00593 against 0.00583, 98.6% of rows within 10% against 98.8%, all 7 gates on both). Before calibration 169 rows move by more than 1%; on the rows you named, at the spine's weights, Universal Credit is +0.4%, capped households −0.8%, and ESA −3.0% with 5.1% fewer income-related claimants. The legacy closure and statutory pay as earned income both reach Universal Credit, and the arm measures them together. The body no longer says "the latest release on PyPI"; 2.131.0 (trading losses, published at 12:29 UTC today) is left for the next bump.

    • The bump surfaced one thing the build had to follow. With Gift Aid extending the bands (pe-uk#2271), the salary-sacrifice relief by rate class no longer added up to earned_income_tax for donors, and the first head calibration on 2.130.1 refused. The allocation now extends a donor's bands by the grossed-up gift as calc_income_tax_with_gift_aid does, the three relief rows declare it, and hand-computed donor cases hold it against the engine (test_a_donors_relief_bands_follow_the_engines_extended_bands).
    • Legacy closure (pe-uk#2119): the build leaves would_claim_uc_at_legacy_closure at the engine's default, so every family moved off a legacy benefit claims Universal Credit. uk-data#534 draws it at DWP's Move to UC claim rates, which are not a Chronicle fact yet; the concept mapping, the changelog and the receipts say the 2025 Universal Credit rows rest on that default.
    • The CGT index: 2.130.0 gives capital gains their own projection path, but with no scenario passed it copies GDP per head exactly (legacy_gdp_per_capita), so the build's gains are projected as before.
  3. R3. Kept, on María's ruling, and documented. On 2.130.1 statutory paternity pay is UC earned income and in household benefits and HBAI income, so the column now raises net income. changelog.d/1095-statutory-paternity-pay.added.md says that before 2.128.0 it was only a National Insurance base and lowered net income, and the fragment and the receipts record the SPPADJ annualisation (weekly amount × 365.25/7) as open on uk-data#575.

  4. The failing row. Recorded: e3342783a holds dwp.uc.households_children_5_or_more out in calibration_measure_exclusions.json with the reason, tracking microcosm#1095 and the sparse-band expiry, 26 November 2026, signed as María's ruling of today; the packaged-exclusions test pins it. The two-child-limit rows stay bound, and the head passes all 7 gates as merged and writes a release file.

  5. Ports and go-firsts. The body and the receipts now say which: R1 (uk-data#526) and R4 (uk-data#518, merged on 14 October) port merged code; R2 (Finish #515: per-record E19200 decomposition by published SOI 2.1 AGI-band components #561, an open PR), R3 (calibration_diagnostics.json has no declared schema and is not validated on write #575), T1 (Staging telemetry silently disabled since 2026-07-02: empty POPULACE_STAGING_REPO_ID defeats the on-by-default staging repo #563), E1 (QBI v3 wiring: simulation consumes SCF/SOI/JCT evidence (shape from SCF, level from JCT) #538) and E2 (Read HMRC's joint distribution of capital gains by taxable income #558) go first; E3 and the stored birth months have no incumbent counterpart.

  6. Test counts. The body and the receipts now quote the same runs. On 1da8d681f (X before the rebases): engine-uk 205 tests, all pass; engine-free 17,475 tests, 17,318 pass, 155 skip and two fail on this machine only (the US scorer test that needs Python 3.14.7, and the live Hugging Face pointer); feed-gated 99 tests, 97 pass, 2 skip. On the head rebased onto Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 (0912388b6): engine-uk 207, all pass, and the engine-free files the rebases touch, Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256's SPI tests among them, 966 tests, 964 pass and two skip.

Also new: 84ce97a2a stores months_since_last_birthday in the spine, which María asked to fold in here. The engine spreads each age and sex over the year by weight when the column is unset, so a release at calibrated weights read different dates of birth from those its solve fitted, on main too. Read as the engine reads it, a file without the fix differs from its own solve on 55 of 1,217 rows, 8 by more than a point (the State Pension rows by scheme and amount most); with it, on 3, and no State Pension, two-child-limit or Pension Credit row moves. What is left (private-school pupils and two land rows) depends on the weights themselves. With the months stored, E3 is the backstop for any other engine read keyed on an identifier.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22

Good to merge. CI is green, and the two shoulds below can land here or in a follow-up. All six round-1 points are fixed. The engine bump has its own measured arm, and on 2.130.1 R3 now raises the income it adds.

Round 1, at fcb6ff23e

  1. Effective sample sizes: fixed on the branch. No effective sample size appears in the receipts, the PR body, any of the 20 commit messages or any added line on the branch. The old commit is still public, though (new item 1).
  2. The lock: fixed. uv sync --locked --extra uk resolves policyengine-uk 2.130.1 and core 3.33.3. The bump arm is reported as shares and rates, with no counts. The rows I named move the expected way before calibration: 0.8% fewer capped households, with ESA down 3.0%, which is what the 2.129.1 remunerative-work rule should do. After calibration the bump fits about as well as the control. The CGT claim holds. With no projection passed, 2.130.1's add_capital_gains_projection copies the GDP-per-head series exactly, and nothing in microcosm passes one. The new cgt_taxable_income definition (net_income less allowances, with Gift Aid moved to the band) matches 2.130.1's capital_gains_tax. "Latest release on PyPI" is gone from the body.
  3. R3: fixed. I re-ran the probe on the locked 2.130.1. £5,000 of SPP now raises household_benefits by £5,000, NI by £400, and both household_net_income and hbai_household_net_income by £4,600. The fragment says what happened before 2.128.0, and the fragment and the receipts both record the SPPADJ annualisation as open on uk-data#575.
  4. The failing row: fixed. calibration_measure_exclusions.json:590 has the reason, the tracking issue, the ruling and the expiry (2026-11-26). If I delete the entry, test_packaged_exclusions_load fails.
  5. Ports against go-firsts: fixed, in both the body and the receipts.
  6. Test counts: fixed. The body and the receipts quote the same runs, and the parts add up.

New

  1. should: the old receipts commit is still public. ed33feb76 is still served by GitHub by its SHA. I fetched it through the API today. Replacing the commit on the branch does not remove it, so the effective sample sizes from round 1 can still be read. Whether that needs a removal request to GitHub Support (their sensitive-data process clears unreferenced commits and cached PR views) is a disclosure judgement for you, not a code point.

  2. should: no test pins how frs_spine stores the birth months. frs_spine.py:513 is the only place the spine gets the engine-way spread at FRS weights. Storing 0 for everyone there passes all 34 touched test files and the H2 parity test, which builds both sides from frs_raw and so moves with the mutation. Spreading with equal weights instead of gross4 also passes both test_uk_birth_month.py files, test_uk_frs_spine.py, test_uk_release_identifiers.py and H2. The engine tests cover uk_months_since_last_birthday itself but not the call. A small test on the synthetic tabs would close the gap: check that the spine's column equals the function at the FRS weights, and that a clone keeps its source person's months.

  3. nit: PyPI has moved past 2.131.0. 2.131.1, 2.131.2 and 2.132.0 to 2.132.5 are all out (2.132.5 at 07:53 UTC today). None of them changes an input the build writes:

    • the build writes no trading_loss and clips self-employment income at zero (frs_spine.py:542), so 2.131.0 has little to act on;
    • the HBAI-based rows in uk_population_targets.json are pending.

    But the ETB services stage reads hbai_household_net_income from the engine (etb_services.py:59). So 2.132.4, which resets negative BHC income to zero, would move that matching for the households it resets. 2.131.2 (AHC now deducts ground rent and service charges) and 2.132.4 also change what users read from a release, including the bottom decile. "Not in this PR" could name 2.132.x rather than only 2.131.0.

  4. nit: the receipts don't show the legacy closure landing. On the bump, UC spending rises 0.4% while UC households fall 0.1%. That is the one move I couldn't square with tax-credit families moving onto UC from 2025-26. There is no tax-credit target row, so the calibration can't see it either. A line giving CTC and WTC on both arms would show the closure took effect, at the same share-level detail as the rest.

What I checked

  • New worktree at fcb6ff23e with a fresh uv venv (uv sync --all-packages --locked --extra uk).
  • ruff check . passes. All 34 touched test files pass, with 6 skips. CI is green on all 10 checks.
  • git range-diff from ed33feb76 to fcb6ff23e. I also diffed every non-test source file whose change differs from round 1.
  • Read the three new commits (B3, M, X), the regenerated-surfaces commit, the receipts, the body, every commit message and the nine changelog fragments.
  • Mutations:
    • Gift Aid band extension switched off: test_a_donors_classes_add_up_to_the_engines_extended_schedule fails.
    • X entry removed: fails as above.
    • Birth months: two mutations, both pass (item 2).
  • Checked the band-extension arithmetic in tax_by_rate_class by hand. It stays exact when a gift spans more than one band.
  • Engine probes on 2.130.1 (SPP, and a legacy tax-credit family). Read the pe-uk changelog from 2.130.1 to 2.132.5, and grepped microcosm for the inputs those releases touch.
  • Disclosure scan of the body, the receipts, the fragments, the exclusion entry and all commit messages. The changed H2 fixture files are generated by tools/graph_uk_spine_fixture.py.

I couldn't build a spine or run a calibration, because I have no FRS or feed access. So none of the arm figures is checked, including the 55-to-3 rows result for the stored birth months. I also couldn't run the full CI job lists or the feed-gated tests.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 71b9b8b into main Oct 15, 2026
10 checks passed
juaristi22 pushed a commit that referenced this pull request Oct 15, 2026
Rebased onto main 71b9b8b (#1271): the release-cut part and full-battery
digests move again; the spine and calibration-seam digests are unchanged.

From the review: the #1124 changelog fragment no longer sends readers to
#1277 for exclusions this change deletes, and the national release runbook
gains an interim check, until #1276's replacement lands, that compares the
SPI stage-1 columns' weighted totals and carrier counts with the promoted
cut using the kept baselines tool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 15, 2026
describe_uk_export takes solve_year as a required keyword since #1271
(0f0bb869e). The C6 test was written before that and called it without one,
so it failed on the rebased base. It now passes solve_year=2025, as the
neighbouring export tests do. Vahid's review round 2, finding 8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants