Skip to content

Port the uk-data wave to policyengine-uk 2.132.6 (fifth #1095 PR) - #1299

Merged
juaristi22 merged 30 commits into
mainfrom
uk-data-ports-1095-5
Oct 16, 2026
Merged

juaristi22 merged 30 commits into
mainfrom
uk-data-ports-1095-5

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

The fifth #1095 ports PR. It locks policyengine-uk 2.132.6 (frozen for this PR) and ports the uk-data wave up to that release, with #868's borrower stocks, #1284's council-tax basis and Vahid's points from his review of #1271. #1287 is left out, on María's ruling. Rulings applied are María's of 15 and 16 October; each commit names the one it applies.

Every item is in, and the head passes all 7 calibrated-seam gates. The directors' dividends (O2) are taken out on María's ruling of 16 October: they made the solve fail the weight-ratio gate, and they stay open in #1095 with what we found (see "Not in this PR"). The receipts are in experiments/1095-uk-data-ports-5-receipts.md.

Fixes #868. Fixes #1284.

Summary

The build stopped on main (fix)

Vahid's review of #1271

  • The stored birth months are pinned by a test (his second should). Both fixture adults share one stratum; the test checks the stored column against the engine-way spread at each household's GROSS4, that it differs from the equal-weights spread, that no month is zero and that an SPI copy keeps its source person's months. It fails under both of his mutations.
  • The engine moves past 2.131.0, and the ETB services stage reads HBAI income before the reset (his third nit): B1 and B2 below.
  • CTC and WTC on both arms (his fourth nit): in the receipts, as shares, with the head's results.

Engine (B1–B3)

  • B1. The UK lock moves from policyengine-uk 2.130.1 to 2.132.6. Only that package moves; policyengine-core stays at 3.33.3 (UK) and 3.32.5 (US). Nothing the build writes is renamed, removed, re-typed or turned into a formula. 2.131.0 adds trade losses, which join the engine's HBAI subtractions; the survey-concept gross income leaves them subtracted. The uprating pins, concept coverage, cited texts, H2 fixture, coverage manifest digests and APPROVED_UV_LOCK_SHA256 re-derive; the target references and both compile-parity receipts regenerate identically.
  • B2. HBAI income before the reset to zero. 2.132.4 resets negative HBAI BHC income to zero (pe-uk#2402). The LCFS predictors and NEED band, the NTS income band, the ETB VAT rate and the ETB services predictors now read hbai_household_net_income_before_reset_to_zero under the predictor name they had.
  • B3. The ESA support group from the FRS ESA group. The root stage reads the benefits tab's VAR3 (ESAGROUP) on ESA rows: esa_includes_support_component is false only where a person's ESA rows record the work-related activity group or the assessment phase and none records the support group. SPI-redrawn rows return to true.

Self-employment losses (L1–L3; uk-data#528)

  • L1. FRS SEINCAM2 splits by sign: profits to self_employment_income, losses to trading_loss (read by the engine from 2.131.0). Before, every loss was dropped.
  • L2. The SPI stage's FRS-only fill draws trading_loss last for the rows it redraws; a row with a drawn profit keeps no loss.
  • L3. The tape's losses brought forward (LOSSBF) move out of self_employment_income into loss_relief and ni_class_4_losses_brought_forward, which the engine relieves (ITA 2007 s.83; SSCBA 1992 Sch 2 para 3); they move, they are not copied. On María's ruling the HMRC self-employment rows measure the profit after that relief, so they keep their values at fixed weights. Three follow-ups make every reader agree: the HMRC total-income bands read total_income_after_loss_relief; the CGT stages and the area income read the profit after the relief; and the measure reads the stored profit, as the rows did before.

Childcare (U2; #1126, #1127)

  • U2a. Vahid's UK: source the extended-hours usage draw from DfE additional-hours PTEs (#1126) #1127 commit, cherry-picked with his authorship: the extended-hours draw's mean moves from 15.019 to 31.141 (DfE's January 2025 additional-hours PTEs). This supersedes UK: source the extended-hours usage draw from DfE additional-hours PTEs (#1126) #1127.
  • U2b. The fitter (tool version 4) follows pe-uk#2177: targeted and universal hours are allocated from working-parent hours, and the universal row counts universal-only children.
  • U2c. The four rates are refitted on the 2.132.6 spine: Tax-Free Childcare 0.9797 (was 0.88), extended hours 0.6579 (0.6054), universal 0.4353 (0.4539), targeted 0.7769 (0.597). No rate reached its bound. The two HMRC Tax-Free Childcare rows still cannot both be met (spend 0.86, children 1.11): a residual of the frame's spend per child, not of the rate.

Targets

  • U1. dwp.uc.households_children_5_or_more becomes a diagnostic beside dwp.uc.two_child_limit.households_5_children, pinned to DWP's April 2025 count, the month of the publication it sits beside. The count includes families outside the two-child limit, and fewer than 10 FRS source households carry that difference. Its exclusion entry retires.
  • E1. obr.esa binds every benefit on OBR's incapacity line the engine carries: esa_income + esa_contrib + incapacity_benefit + sda (uk-data#582; OBR table 4.9 footnote 2). The incapacity part of Income Support has no engine variable and stays a residual of the row with no declared difference.
  • D1a. The student-loan borrowers below the threshold (UK spine: SLC plan-2 borrower stocks at a third of SLC's — the FRS flag captures repayers, not the liable stock #868). frs_education stores SLOS (reports_outstanding_student_loan) and EDUCLEFT (age_left_full_time_education). A reported borrower repays, or, in England or Wales, reports an outstanding loan and is not in full-time education. A course starts three years before its borrower left full-time education (at 18 without an answer), and its start year sets the plan; the Plan 2 walk takes nobody in full-time education. The four SLC rows bind academic year 2024/25, whose reference date falls in calendar 2025, with a signed in-year-snapshot hold; they now match the incumbent fixture. Its own commit, found on the head's calibration: nobody repays a Plan 5 loan before April 2026, so before 2026 a repayer never takes Plan 5; a repayer whose dates point to a course from 2023/24 takes Plan 2, and SLC's published zero of Plan 5 borrowers above the threshold holds. The cohort operation declares the year (plan_5_repayments_from: 2026); María confirmed the rule on 16 October. The year is fixed, not derived from the cohort, so it is to be revisited when the build year moves (Vahid's review).
  • C1. The council-tax stock on the household basis of its area (UK national register: council-tax dwelling stock and the official household controls bind the same household count at different levels #1284). A national-only household_basis_partitions declaration rescales each English region's, Wales's and Scotland's dwelling total onto its household control, and its bands by the same factor: the area's published households per chargeable dwelling, 0.980 to 1.023 on the 735820c feed. A tripwire refuses a factor further than 0.035 from one (the measured 0.0227 with a margin), and the closure check requires each rescaled total to equal its control.
  • U3. The sparse-band support floor. See its own section below.
  • P1. The parity fixture's Scottish constituency rows sit in their seats (uk-data#589). Five unchanged constituencies keep their 2010 codes in the incumbent's source; the extraction tool maps them to S14000107–S14000111 and takes the rows by code into the PCON24 roster's order. 798 values move, no row is added or lost; the local compile parity is regenerated.

Inputs

  • O1. Six owner-manager inputs from the FRS job tab (pe-uk#1965; UC Regs 2013 reg 77). A director owning at least half of the company (DIRPERC 1–3) stands as its sole owner or partner and works in it; the company is the main employment for the first job, and carries on a property business when the main job's SIC division is 68. The income share is the job's dividends after tax; it reads job DIVIDEND only, and dividend_income is unchanged. On SPI-redrawn rows the stage-2 fill draws the six inputs last.
  • U4. Private-school attendance at the spine's weights (uk-data#568). A new last spine stage materializes the engine's attends_private_school once, at the calibration year and the spine's design weights, and stores it, so the solve and the release read the same flags. Stored attendance no longer responds to income changes or to reforms of the attendance parameters.

U3. The sparse-band support floor

The measure-exclusion register moves to schema 3 and declares the floor once, signed by María on 16 October (expires 8 January 2027). Over the four dwp.uc.payment_distribution_* targets, a band whose compiled matrix row reaches fewer than 10 distinct source households (household_source_id) with a positive prior weight is dropped from the solve, not pooled. It runs on the compiled problem before the loss weights, families and metadata are drawn: in the national problem node and, after the stitch, in the full build's measures node (so before #1207's family loss-share floors are computed; those floors cover other families). Its receipt gives each family's dropped published total and no support count. The run refuses outside the window, past the ceilings, and when a per-row exclusion names a band the floor would drop. The drops join the problem's measure exclusions as kind: support_floor, and the national gates, the diagnostics, the full build's target-surface and reference-coverage gates and the certifier read them back.

Ceilings: 24 rows and 5% of the bands' published total. On the head spine the floor drops 19 bands, 3.1% of that total, so the ceilings leave a margin of five rows and about two points; which bands sit under the floor can move from one spine to the next. María confirmed the ceilings on 16 October, with a little headroom.

The 21 per-row band entries, sorted on the head spine (no support count is given here or anywhere):

  • (a) Below the floor, so the floor drops them and the entry retires (9): COUPLE_NO_CHILDREN 22,800–24,000, 24,000–25,200, 26,400–27,600, 27,600–28,800 and 28,800–30,000; SINGLE 21,600–22,800, 26,400–27,600, 27,600–28,800 and 28,800–30,000.
  • (b) At or above the floor but out of reach at the 10× weight cap: none.
  • (c) Neither, so the entry stays and needs your renewal before it expires on 26 November (12): LONE_PARENT 12,000–13,200, 13,200–14,400, 14,400–15,600, 15,600–16,800, 16,800–18,000, 18,000–19,200, 19,200–20,400 and 21,600–22,800; SINGLE 8,400–9,600, 9,600–10,800, 13,200–14,400 and 14,400–15,600.

The floor also drops ten bands the solve bound before: COUPLE_NO_CHILDREN 10,800–12,000, 19,200–20,400, 20,400–21,600, 21,600–22,800 and 25,200–26,400; COUPLE_WITH_CHILDREN 28,800–30,000; SINGLE 20,400–21,600, 22,800–24,000, 24,000–25,200 and 25,200–26,400.

Vahid's review of this PR

  • The ESA group test now pins each code on its own (1 and 3 outside the support group, 2 the support group, any other code no group), and an engine test pins the SPI stage's ESA reset and trading-loss clearing on the stage's output; his mutations fail them.
  • The ESA group codes cite the FRS 2024-25 variable listing (UK Data Service SN 9563, sheet BENEFITS_table): on an ESA row VAR3 holds the questionnaire's ESAGroup, 1 work-related activity group, 2 support group, 3 assessment phase.
  • The receipts say why the Tax-Free Childcare rate (0.98) is not a bound hit and why the universal rate is low; the fitter's docstring on the universal row's allocated term matches the code; the UK: data fixes to model the High Value Council Tax Surcharge (mansion tax) correctly, and the related housing and council-tax gaps #1240 fragment says a started NEED cell replaces the spend its households drew; a comment marks the Plan 5 year for revisiting.
  • His two O2 follow-ups are in UK: track the defects uk-data is fixing that microcosm still has #1095's directors' dividends item.
  • The band names under the support floor are María's call under the UK Data Service rules, before merge.

Deviations and open points

  • L3 uses two named measures (self_employment_income_after_loss_relief, total_income_after_loss_relief) that the UK resolver computes, not a new term in the evaluator grammar the plan named. They give the same values; the grammar change can follow.
  • E1 also binds incapacity_benefit, beyond the recommendation: OBR's footnote names it, and it closed to new claims in 2008, so the term carries only what the FRS still reports.
  • U1 is pinned to April 2025, the month of the two-child-limit publication it sits beside.
  • U4's pupil series stays ISC's member-school census. DfE's count of all independent-school pupils, the series María chose, is not yet a Chronicle fact; switching needs that fact first.
  • D1b is out of reach. The head spine's Plan 2 England stock is about 41% of SLC's at design weights, and the walk's whole pool would give about 47%, so the plan's 25% acceptance cannot be met by the cohort rule. The two UK spine: SLC plan-2 borrower stocks at a third of SLC's — the FRS flag captures repayers, not the liable stock #868 exclusions (slc.borrowers.plan_2_liable, plan_2_above_threshold, expiring 3 November) need a ruling: renew, or turn them into diagnostics.
  • The per-row refusal runs on the national line only. The full build drops below-floor bands too, but does not compile the per-row-excluded bands to measure them; the national run on the same spine does.

Measurement

Each arm builds its spine locally on feed 735820c and runs the national role's calibration on it. Details are in the receipts.

  • control (main 5d4eb37b3 plus the NEED fix, policyengine-uk 2.130.1): loss 0.29148 to 0.00607; 98.6% of 1,226 rows within 10%; all 7 calibrated-seam gates pass.
  • head with O2 (before O2 left): loss 0.26546 to 0.00583; 98.4% of 1,217 rows within 10%; 6 of 7 gates pass, uk_weight_ratio fails.
  • head: loss 0.26817 to 0.00584; 98.4% of 1,216 rows within 10%; all 7 gates pass. On its written file the weight year, the identifiers and the stored birth months check out, the input-mass gate passes, and of the 1,216 solved rows only the two ONS land rows (declared weight-share formulas) read differently from the solve; the private-school pupils row now matches it.
  • C1 on the head: the 97 council-tax stock rows miss by 0.13% on average (control 0.64%) and none by more than 2% (control: 10 rows); the household controls miss by 0.06% (control 0.07%).
  • Tax credits (Vahid's fourth nit): child and working tax credit are zero from 2025 on both the control and the head, so the legacy closure takes effect on both engines; Universal Credit at 2025 is £77.7bn on both at their calibrated weights.

Not in this PR

Disclosure

Checked against the UK Data Service rules before pushing: the commit messages, every added line, the fragments and this body. No record-level value, extreme, effective sample size or support count appears; the one small cell is given only as "fewer than 10"; obr.esa has no before-and-after figure; the ESA group is given only as support group or not.

Tests

  • CI: the checks on this PR's head.
  • After the review: the FRS spine, SPI income (engine-free and engine), student-loan, take-up tool and contract, energy pricing and graph tests, 1,073 with 2 skips, all pass.
  • Locally, at the head's code: the touched surfaces, 1,479 tests with 1 skip; the SPI engine tests, 88 with 1 skip; CI's engine-uk job list, 213, all pass.
  • The feed-gated tests on feed 735820c, at the first push: 103 with 2 skips, all pass.

🤖 Generated with Claude Code

juaristi22 and others added 18 commits October 16, 2026 00:11
…1240)

Main's spine build stopped at the energy_rake gate after #1270: a NEED
cell had a zero current mean, and the rake, which multiplies, cannot move
it. Mobile homes have no NEED property-type row, and the rake takes them to
the all-dwellings mean, the incumbent's translation. A household whose
drawn spend is at or below the standing charge has no kWh, so a
mobile-home cell can be left with none once #1270's copies change the
frame and with it the draws.

price_domestic_energy declares zero_kwh_cells_at_need_mean (accommodation:
all_dwellings). Before the first sweep, the rake starts each declared cell
in which no household drew kWh at the cell's NEED mean, electricity over
every household and gas over the gas-connected ones. A cell with a
positive household is left to the rake. The receipt lists the cells it
started. The energy_rake gate fails on a started cell the stage does not
declare and on any other cell with a zero mean, and its details count the
started cells without naming them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vahid's review of #1271 found that no test pinned the spine's call to
uk_months_since_last_birthday: storing 0 for everyone, or spreading at
equal weights instead of GROSS4, passed every touched test. The fixture
gave each person an age-and-sex stratum of their own, so the months
never depended on the weights.

The new engine-free test puts both fixture adults in one stratum
(households of weight 10 and 20), checks that the stored column equals
the engine-way spread at each household's GROSS4, that it differs from
the equal-weights spread, that no month is zero, and that an SPI support
copy keeps its source person's months. It fails under both mutations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lock's UK side moves from policyengine-uk 2.130.1 to 2.132.6 (only
that package moves; policyengine-core stays at 3.33.3 for the UK and
3.32.5 for the US), and the UK extras of microcosm-build, microcosm-data
and microcosm-frame require >=2.132.6. Nothing the build writes is
renamed, removed, re-typed or turned into a formula.

2.131.0 adds trade losses: trading_loss joins the engine's HBAI
subtractions. The survey-concept gross income leaves it subtracted, since
the surveys count self-employment income net of losses, and the engine
lockstep test now checks the engine's list against the deductions plus a
declared not-added-back tuple.

Re-derived with no change to data: the uprating pins (version stamp
only), the concept coverage (299 inputs to 300: loss_relief joins,
uncovered), the texts that cite the locked engine (each claim re-read at
2.132.6), the H2 fixture, the coverage manifest's source digests and
APPROVED_UV_LOCK_SHA256. The target references and both compile-parity
receipts regenerate identically on feed 735820c. CI's engine-uk job
list: 207 tests, all pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk 2.132.4 resets negative HBAI net income before housing
costs to zero (pe-uk#2402). The stages that band or match households on
HBAI income now read the figure before the reset,
hbai_household_net_income_before_reset_to_zero, under the predictor name
they had: the LCFS consumption predictors and NEED energy band, the NTS
income band, the ETB VAT rate's disposable income and the ETB services
predictors. The surveys they match to do not reset their incomes, and
the gross income the stages build (net income plus the HBAI deductions)
holds only before the reset. On the build's data the figure equals the
HBAI net income 2.130.1 returned. Answers Vahid's note on the ETB
services read in his review of #1271.

The four materialize lists in the spec name the new variable; the QRF
predictor lists and the donor uprating keys keep the predictor name. An
engine test pins the reset and the gross income of a household with
negative income; the engine-free fakes return the new variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk reads esa_includes_support_component (since 2.129.0) for
the benefit cap's ESA exemption and UC's LCWRA test, and without it
assumes the support group for anyone with an ESA award of their own. The
root stage now reads the FRS benefits tab's VAR3 (ESAGROUP) on ESA rows:
a stored column replaces the engine's default for everyone, so it is
false only where a person's ESA rows record the work-related activity
group or the assessment phase and none records the support group, and
true everywhere else. A benefits tab without VAR3 refuses.

On SPI-redrawn rows the stage-2 fill takes the ESA receipts from other
FRS donors, so the column returns to true there. It is a net-new export
column: allow-listed on both lists, typed in the graph, the digests and
the H2 fixture (now with VAR3 on its benefits tab) regenerated. Tests:
the VAR3 mapping and refusal on the spine fixture, the SPI reset, and an
engine test that the benefit cap exemption follows the stored group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SEINCAM2 carries a loss as a negative amount. policyengine-uk takes
profits in self_employment_income, never below zero, and the year's
losses in trading_loss as a positive amount (pe-uk#2085, from 2.131.0),
so the root stage splits the amount by its sign, as uk-data#528 does.
Before this, every loss was dropped.

The concept mapping gains a signed_part transform for the two inputs.
trading_loss joins the reviewed extra export columns, because the pinned
incumbent does not carry it. A hand-computed case shows that the
salary-sacrifice rate classes follow the loss-relieved income.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPI income stage's FRS-only stage 2 now also draws trading_loss,
which it reads no tape value for, from the FRS rows and conditioned on
the drawn incomes (uk-data#528). It comes last in the stage-2 chain, so
the earlier draws do not move. A redrawn row whose drawn
self_employment_income is positive keeps no loss, so profit and loss
never both hold an amount, as on the FRS rows.

The tail-concentration surface checks one more declared QRF output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPI channel held self_employment_income net of the tape's losses
brought forward (LOSSBF). policyengine-uk relieves earlier losses
itself: income tax against loss_relief (ITA 2007 s.83) and Class 4
against ni_class_4_losses_brought_forward, while Universal Credit, HBAI
and market income read the year's profit (uk-data#528).

The stage-1 chain still draws the net profit. Last of all, so that no
earlier draw moves, it draws the year's profit before the losses and the
losses themselves. After the FRS-only fill and the HMRC aggregates, a
drawn row's profit becomes the year's profit and both inputs take its
losses, held so that the profit after the relief is the drawn net profit
on every row. The losses move; they are not copied.

The HMRC aggregates and the four HMRC self-employment rows read the
profit after the relief. The rows bind two measures that the UK
resolver computes from the engine, so at fixed weights they keep their
values and their band membership. The target references and both
compile-parity receipts regenerate unchanged. FRS rows carry no losses
brought forward, and the two inputs join the reviewed extra export
columns. The engine carries both losses into each projected year at
their nominal value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HMRC's total income holds the profit after losses brought forward. The
engine's total_income adds the year's profit, which since the SPI
channel moved the tape's losses into loss_relief is the profit before
that relief. So the SPI and Income Tax Liabilities rows by total-income
band now band on total_income_after_loss_relief, and the regional rows
read and filter on it, as the self-employment rows already do. Without
this, a row with losses brought forward would move up a band.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CGT stages band people and draw their gains on the sum of the
stored income columns, which stands in for HMRC's total income, and
the area target matrix reads the engine's self_employment_income
directly. Since the SPI channel moved the tape's losses brought forward
into loss_relief, both read the year's profit before that relief. That
put some SPI rows in a higher income band and changed where the CGT
stages placed gains. Both now read the profit after the relief, so
their income bands are as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The HMRC self-employment rows read self_employment_income from the
frame's stored column, and the target adapter reads a stored column as
it is. The profit-after-relief measure took the profit from the engine
at the measurement year instead, which uprates it, so the rows moved by
the uprating. The measure now reads the stored profit and losses, as
the rows did before the move. The band measure stays on the engine's
total_income, as the band did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1126)

The frozen clip(N(15.019, 4.972), 0, 30) gave a working-parent 3- or
4-year-old about 15 funded hours a week, the universal offer alone.
DfE's January 2025 census funds 341,988.09 additional-hours PTEs for
379,000 registered 3-4-year-olds: 28.535 weekly hours in total. The mean
becomes 31.141 (sd and bounds held), status sourced_administrative_derived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1095)

Since pe-uk#2177 universal and targeted hours stay beside the
working-parent hours, and a family that does not claim them on their
own still receives them, allocated first from a participating child's
working-parent hours (DfE guidance A1.11). The fitter's expectation
basis gains a third engine run, extended on with the targeted and
universal flags off, so a targeted row adds that allocation at the
extended rate times one minus its own rate. The universal row counts
universal-only children, so each run leaves out a child who receives
extended hours. The fit runs on a spine built before the extended-hours
anchor moved (#1126), so it sets the hours the take-up stage draws under
the current contract. Tool version 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1095)

The take-up stage draws maximum_extended_childcare_hours_usage from the
contract's distribution, so the anchor #1126 moves changes the H2 fixture's
output digest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DWP's count of Universal Credit households with five or more children
includes families outside the two-child limit, which the bound
two-child-limit rows leave out, and fewer than 10 FRS source households
carry that difference. The row, held out since 2026-10-14 (#1271), leaves
the targets on María's ruling of 2026-10-16: it becomes a diagnostic
reference beside dwp.uc.two_child_limit.households_5_children, pinned to
DWP's April 2025 count, the month of the two-child-limit publication it
sits beside. Its exclusion entry retires, the registry-parity mapping
moves the incumbent row to the excluded rows, and the frozen fixture's
fitted row is fixture-only on the compile-parity surface. Target
references and compile parity regenerated on the 735820c feed: 414
targets, 1,291 active references; the local parity surface is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OBR table 4.9 footnote 2: the "Incapacity benefits" line includes
Incapacity Benefit, ESA, Severe Disablement Allowance and the incapacity
part of Income Support, and excludes Universal Credit's health element
(uk-data#582). The binding becomes esa_income + esa_contrib +
incapacity_benefit + sda. Income Support has no incapacity part in the
engine, so that part stays a residual of the row with no declared
difference (María's ruling of 2026-10-16). SDA is a small part of the line
(DWP: GBP 53.8m for England and Wales in 2024-25, against OBR's GBP
12.41bn); the Scottish Government pays the Scottish part (GBP 5.0m) outside
DWP's line, well under 0.1 percent of the row, so it stays inside the
row's Great Britain scope with no per-term mask. Incapacity Benefit closed
to new claims in 2008, so its term carries only what the FRS still reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…education (#868)

The student_loans stage assigned plans to PAYE repayers alone and dated
every course from age 18, so a borrower below the repayment threshold
carried no plan and the Plan 2 top-up walk ran out of people (#868). On
María's ruling of 2026-10-16, frs_education stores two FRS adult-table
answers: reports_outstanding_student_loan (SLOS, which adults who make no
repayment answer) and age_left_full_time_education (EDUCLEFT, 0 for no
answer). A reported borrower now repays, as before, or, in England or
Wales, reports an outstanding loan and is not in full-time education. A
course starts three years before its borrower left full-time education,
or at 18 without an answer, and its start year sets the plan. The Plan 2
walk takes nobody in full-time education, since a student is not yet
liable to repay; the Plan 5 walk keeps the age rule.

The four SLC borrower rows bind the stock the stage tops up to: SLC
academic year 2024/25, whose reference date falls inside calendar 2025,
stored by Chronicle under its opening year 2024. Each row pins that period
and a signed in-year-snapshot hold declares it; before, the rows bound
academic year 2025/26 while the stage topped up to 2024/25. Both new
columns join the export surface's reviewed extra columns; a tab without
SLOS or EDUCLEFT refuses. Target references and compile parity
regenerated on the 735820c feed: the four rows now match the frozen
incumbent fixture, which already read academic year 2024/25.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk (pe-uk#1965) treats a company's owner as self-employed in
Universal Credit (UC Regs 2013 reg 77) through eleven owned-company inputs.
The root stage sets six of them from the FRS job tab. A director (DIRECTOR
1, or NATURE 1 "Director") who owns at least half of the company (DIRPERC
100%, more than 50% or 50%; María's ruling of 2026-10-16) stands as its
sole owner or partner and works in it. The company is their main
employment when the job is their first; it carries on a property business
when the main job's SIC division is 68, real estate, and a trade
otherwise. A company held in another job carries on neither, since the FRS
codes the main job's industry only, so the engine's company-owner rules
stay off for it. The income share is the dividends the job records over
12 months, after tax: the part of the company's profit the survey sees.
The stocks and the intermediaries inputs keep the engine's defaults.

On SPI-redrawn rows the stage-2 FRS fill draws the six inputs last, the
flags as 0 or 1 through its zero regime gate, so they follow the drawn
incomes; the root stage's rules then hold on those rows. The six columns
join the export surface's reviewed extra columns, the graph's cells and
the H2 fixture; a job tab without the director answers refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-data-ports-1095-5 branch from 591cbac to 522d453 Compare October 16, 2026 04:01
juaristi22 and others added 6 commits October 16, 2026 09:37
policyengine-uk imputes attends_private_school with a formula that ranks
households by net income at their weights, so every simulation
re-imputed it: a reform to income tax, National Insurance or benefits
switched pupils in or out and moved the fee VAT, a population under one
million people of weight imputed none (pe-uk#1905), and the calibration
and the release read different flags once the weights changed
(uk-data#568). A new last spine stage, private_school_attendance,
materializes the engine's attendance once, at the calibration year and
the spine's design weights, from the stored random draw, and stores it, as
the root stage stores months_since_last_birthday; no stage after it copies
or re-keys a person. Stored attendance no longer responds to income
changes or to reforms of the attendance parameters.

The stage records its mass-conservation receipt; the roster, the kernel
tables, the coverage manifest, the operation vocabulary, the gate digests
and the H2 fixture (42 stages) move with it, and the column joins the
export surface's reviewed extra columns. The pupil series stays ISC's
member-school census: DfE's count of all independent-school pupils, the
series María chose on 2026-10-16, is not yet a Chronicle fact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The version-4 fitter (targeted hours allocated from working-parent hours,
the universal-only mask and the DfE extended-hours anchor in the
simulation) refits the four rates on the bump arm's spine (policyengine-uk
2.132.6, feed 735820c, 2024 base year, the previous run's arguments). The
fit converges with no rate at its bound: Tax-Free Childcare 0.9797 (was
0.88), extended hours 0.6579 (0.6054), universal 0.4353 (0.4539) and
targeted 0.7769 (0.597). The three DfE rows match their targets; the two
HMRC Tax-Free Childcare rows still cannot both be met (spend 0.86,
children 1.11), a residual of the frame's spend per child, not of the
rate. Each rate's fitting receipt names the spine, feed, engine and
contract digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The council-tax dwelling stock and each nation's official household
estimate bind the same weighted household count of an area at two
published levels, and nothing related them, so the solve traded one family
against the other (#1284). A new national-only declaration,
household_basis_partitions in uk/cross_grain_declarations.json, rescales
each English region's, Wales's and Scotland's dwelling total onto its
household control, after the exact-signature pass, and its bands by the
same factor: the area's published households per chargeable dwelling,
0.980 to 1.023 across the eleven areas on the 735820c feed. The bands keep
their published mix and the level follows the household estimate.

María signed the assumption on 2026-10-16: households per chargeable
dwelling are the same in every band of an area. A tripwire refuses a
factor further than 0.035 from one (the measured 0.023 with a margin), and
the national closure check requires each rescaled total to equal its
control. The local band cells follow the national rows by signature.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1095)

Five unchanged Scottish constituencies keep their 2010 codes in the
incumbent's boundary-change source, so the mapping's sorted 2024 rows put
them, and every Scottish row between them, out of the PCON24 roster's
order (found through uk-data#589). The extraction tool maps the five codes
to S14000107 to S14000111 as uk-data#589 does, asserts that the grain is
the mapping's 2010 order and that the mapped rows are the 2024 roster once
each, and takes the rows by code into the roster's order; the provenance
text signs the fix (María, 2026-10-16). Re-extracted at uk-data 12a1e028:
798 Scottish constituency values move, no row is added or lost. The local
compile parity is regenerated on the re-extracted fixture: the same 798
values move in its calibration-drift rows, and the counts by kind are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UC payment-distribution band that the spine supports with only a few
source households can be fitted only by stretching those few weights, and
a fit that rests on fewer than 10 source households is a cell the release
must not report. Each such band was held out by its own per-row exclusion.

The measure-exclusion register moves to schema 3 and declares a support
floor once, signed by María on 2026-10-16 (expires 2027-01-08): over the
four dwp.uc.payment_distribution_* targets, a band whose compiled matrix
row reaches fewer than 10 distinct household_source_ids with a positive
prior weight is dropped from the solve, not pooled. It runs on the
compiled problem before the loss weights, families and metadata are
drawn: in the national problem node and, after the per-block stitch, in
the full build's measures node. Its receipt gives each family's dropped
published total and no support count. The run refuses outside the window,
past the ceilings on dropped rows and published share, and when a per-row
exclusion names a band the floor would drop; the national node compiles
those rows beside the register only to measure them.

The drops join the problem's measure exclusions as kind support_floor.
The national gates and the driver's diagnostics, the full build's
target-surface and reference-coverage gates and the release certifier read
them back and check them against the declared floor, since none of them
can recompute support. The register written as the scoring surface stays
the approved one, so a frozen-register check still re-derives it.

On the head spine the floor drops 19 bands, 3.1% of the bands' published
total; the ceilings, 24 rows and 5%, leave a margin of five rows and about
two points. Nine of the 21 per-row band entries are among them and retire;
the other 12 bands are well supported and stay, each on its own renewal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nobody repays a Plan 5 loan before April 2026, so SLC publishes no Plan 5
borrower above the threshold in academic year 2024/25, the stock the SLC
borrower rows bind since the first #868 commit of this branch. The stage
still gave Plan 5 to a few young repayers whose answers date a course
from 2023, and the solve could not meet the published zero: the row's
miss alone held the head's loss near 0.30.

Before 2026 a repayer never takes Plan 5: a repayer whose dates point to a
course from 2023/24 takes Plan 2, since a borrower who repays started a
course before Plan 5 existed. From 2026 the rule lapses. The cohort
operation declares the year (plan_5_repayments_from: 2026); the stage
roster, the coverage manifest and the H2 fixture are regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22 thanks, this is my review at 2dcb5619b.

Fine once the B3 tests are tightened and CI is green. O2 is out and the receipts are in. The engine bump, the HBAI swap and the loss split are right, and most new tests fail when I revert the code they cover. Three code paths are still not pinned (items 1 and 2), and I couldn't confirm the ESA group coding from the documents I have (item 3).

Points from my review of #1271

  • Round 1, items 1 to 6: still fixed.
  • Round 2, item 1 (old receipts commit): still public. ed33feb76 is still served by its SHA (I fetched it through the API today). This PR can't fix that. It stays a disclosure call for you.
  • Round 2, item 2 (birth months): fixed. test_birth_months_are_spread_at_the_frs_weights_and_kept_by_copies fails if the spine spreads the months with equal weights instead of GROSS4. It also checks that no month is zero and that copies keep their months, which covers my store-zero mutation.
  • Round 2, item 3 (engine past 2.131): fixed by B1 and B2.
  • Round 2, item 4 (CTC and WTC on both arms): fixed. The receipts (Tax credits) give child and working tax credit as zero from 2025 on both arms and UC at 2025 as £77.7bn on both. These are aggregates rather than shares, but they are weighted totals over the whole frame, so that's fine.

Findings

  1. should: the B3 test doesn't pin each ESA group code. The fixture adult has one work-related activity group row (VAR3 1) and one assessment-phase row (VAR3 3). If I change frs_spine.py:1245 to group.isin((3,)), test_esa_support_group_is_read_from_the_esa_rows still passes, so a WRAG-only person going to true would not be caught. One case per code (1 alone, 3 alone, 2 alone) would close this.

  2. should: two SPI-stage call sites are not pinned. If I delete spi_income.py:1241 (_refresh_esa_support_component) and spi_income.py:1242 (_clear_trading_loss_beside_profit), nothing fails in test_uk_spi_income.py (engine-free or engine), test_uk_spi_spine.py or test_uk_spi_support.py. The helpers have unit tests, but nothing checks that impute_uk_spi_income_support calls them. By contrast, deleting the _move_losses_brought_forward_to_loss_relief or _owner_manager_flags_from_draws call does fail tests. An assertion on the stage output would cover both: a redrawn row's flag is true, and a redrawn row with a profit has no trading_loss.

  3. should: please cite where the ESAGROUP codes come from. _esa_includes_support_component (frs_spine.py:1222) reads VAR3 1 as WRAG, 2 as support group and 3 as assessment phase. The UKDS data dictionary I have labels VAR3 only "Code 0" to "Code 17", and DWP doesn't post the questionnaire. The engine's note that about nine in ten ESA recipients who answer say support group fits code 2 being support, but that is not an independent check. If the codes were swapped, the benefit cap ESA exemption and UC's LCWRA test would flip for most ESA recipients. A reference in the docstring (questionnaire block or DWP variable spec) would settle it.

  4. Resolved: O2 is out. Taking it out of this PR is what I would have suggested. Your local branch without it passes all 7 gates, and the uk_weight_ratio fence (gates.json:823) is not something to loosen to pass. The O2 section of the receipts records the finding without any record-level figure. Two points for the follow-up in UK: track the defects uk-data is fixing that microcosm still has #1095:

    • The removed docstring said DWP add DIVIDGRO beside investment income in total income, which means DWP treat the two as separate, while the larger-of rule assumes they overlap. That is worth settling first, because it changes the dividend mass.
    • Two candidates to measure: adding DIVIDGRO only for owner-managers (DIRPERC 1 to 3, as O1 does), or leaving directors' dividends to the SPI channel at the incomes it redraws.

    O1 stays. On 2.132.6, UC unearned income leaves out dividends (income_definitions/unearned.yaml), so its owned_company_income_share is not counted twice in UC.

  5. should: the Tax-Free Childcare rate is close to its bound. U2c fits 0.9797, and the fitter's bound is 1.0 (tools/fit_uk_childcare_takeup.py:49). Both TFC rows still miss (spend 0.86, children 1.11), so the rate is mostly absorbing the frame's spend per child. Your U2c ruling, as the receipts record it, is that a bound hit keeps the bound. Please say in the receipt whether 0.98 counts as a hit for that rule. Separately, the universal rate falls to 0.4353. A sentence on why the frame needs so low a standalone rate to meet DfE's universal-only count would help a reader.

  6. nit: the fitter docstring says the universal row's allocated term is zero (tools/fit_uk_childcare_takeup.py:291). The code computes that term anyway. Under pe-uk#2177, a child with no standalone take-up who uses 15 hours or less gets base hours and no extended hours. On the allocation side, that child counts as universal-only. At the new usage mean of 31.141 that case is very rare, so the effect is negligible, but the docstring should match the code.

  7. nit: UK: data fixes to model the High Value Council Tax Surcharge (mansion tax) correctly, and the related housing and council-tax gaps #1240 doesn't bias the rake, but it overwrites what the donors drew. Starting a declared all-zero cell at its NEED mean doesn't bias the rake: the accommodation margin holds that cell's mean wherever it starts, and the cell's households are a negligible share of the other margins. Their drawn spend (at or below the standing charge) is replaced by the all-dwellings mean, so their energy spend rises above what they drew. That matches the incumbent translation for mobile homes, so it's fine. The changelog fragment could say it in one line. The gate does fail closed: if I disable the undeclared-cell check at stage_health.py:564, test_the_energy_rake_gate_takes_only_the_declared_held_cells fails, and if I drop the start at energy_pricing.py:1125, its test fails too.

  8. nit: the Plan 5 rule (you asked for a confirmation). For the 2024/25 stock, the rule at student_loans.py:286 is sound. A repayer whose dates point to a course from 2023/24 can't be repaying Plan 5 in 2025. One caveat: the date is fixed (plan_5_repayments_from: 2026), not derived from the cohort. With the three-year course assumption, a 2023/24 starter leaves in 2026 and first repays in April 2027. So a 2026 build would let repayers take Plan 5 a year before the assumed cohort repays. That is worth revisiting when the build year moves.

  9. Disclosure question (a judgement for you). No support count appears anywhere, and "fewer than 10" was accepted on UK: policyengine-uk 2.130.1 and the fourth batch of #1095 ports, with the release's weight year, identifiers and birth months fixed #1271. The rewritten U3 commit (7f862f94e) changes only the measured drop, from 18 bands at 2.8% to 19 bands at 3.1% of the published total, and the margin text. The question is unchanged, and the new receipts widen it: 19 UC payment bands whose matrix rows reach fewer than 10 source households are now named in the PR body, in the receipts (U3 section), in the floor receipt's dropped list and in the exclusion entries. Each named band is in effect a small-cell marker. The receipts give DWP's published households for the dropped bands by family, which is fine on its own. Please confirm the band names are acceptable under the UKDS rules before merge.

    The rest of the receipts are clean. I found no exact count, effective sample size, minimum, maximum, unit-record value or seat-level value in them, nor in the body, the commit messages, the added lines or the fragments. The receipts say weight ratios and the largest weight stay local. The one range given (0.980 to 1.023, C1) is of published households per dwelling across areas. The "at most 0.22 points" is of target-row differences, not microdata.

What looks right

  • B1: in uv.lock only policyengine-uk moves (2.130.1 to 2.132.6, PyPI sdist and wheel hashes). The three requires-dist floors match it, core stays 3.33.3 (UK) and 3.32.5 (US), and shasum -a 256 uv.lock equals APPROVED_UV_LOCK_SHA256. uv sync --locked --extra uk resolves 2.132.6.
  • B2: every engine read of HBAI net income in the LCFS, NTS, ETB VAT and ETB services stages now uses hbai_household_net_income_before_reset_to_zero. The predictor names on the donor side are unchanged. local_targets.py:383,386 still read the reset equiv_hbai_household_net_income(_ahc), which is right for the published HBAI poverty and decile concepts. Leaving trading_loss subtracted from the survey gross income is right too.
  • L1 to L3: the engine (2.132.6) deducts loss_relief from self-employment income only, never below zero. Class 4 starts from self_employment_income before loss_relief (ni_class_4_profits_before_losses), so setting both inputs is not a double deduction. ITA 2007 s.83 and SSCBA 1992 Sch 2 para 3 are the right provisions. On SPI rows, the profit after relief equals the profit the chain drew in both branches of spi_income.py:2741. derive_hmrc_income_auxiliaries runs before the move, while loss_relief is still zero, so nothing is subtracted twice. The HMRC rows, the regional and ITL bands, CGT and the area income all read the profit after relief. The engine's total_income adds the profit before relief, so the swap in total_income_after_loss_relief is correct. band_filter_dimension: total_income is a Ledger filter key, not a variable read. ledger_targets.py mentions self-employment income only in a docstring.
  • U2a: the commit keeps your authorship and date, and its changes are identical to UK: source the extended-hours usage draw from DfE additional-hours PTEs (#1126) #1127's head (b91ef34f8). The clipped normal at mean 31.141, sd 4.972, bounds 0 to 30 gives 28.535 hours, which matches 15 + 341,988.09 × 15 / 379,000.
  • U2b follows pe-uk#2177's allocation (base hours first from working-parent hours, flags following the allocated hours).
  • O1: the DIRPERC codes match the UKDS job dictionary (1 = 100%, 2 = more than 50%, 3 = 50%, 4 = less than 50%).
  • The O2 removal is clean: git diff 522d453d5 2dcb5619b touches only the O2 code and test, its fragment, the fixture and generator columns, the regenerated digests and the receipts.

What I checked

  • New worktree at 2dcb5619b with a fresh venv (uv sync --all-packages --locked --extra uk).
  • git range-diff from 522d453d5: commits 1 to 18 are unchanged, the O2 commit is dropped, and the last six are rebased, with 7f862f94e changed only in its text and the receipts commit added.
  • ruff check . passes, and ruff format --check passes on the 92 touched Python files. tools/ci_test_plan.py verify passes.
  • At 522d453d5: all 45 touched test files passed, 1,613 tests with 10 skipped.
  • At 2dcb5619b: the files covering the rewritten commits pass, 554 tests with 1 skipped. These were the frs_spine, support-floor, student-loan, private-school, take-up, source-stage, coverage-manifest, country-spec, population-target, CGT and spine receipt and parity tests, plus the H parity test.
  • Mutations that fail the new tests:
    • L1 losses dropped;
    • L3 copy instead of move;
    • HMRC measure without the relief;
    • birth months at equal weights;
    • the NEED gate's undeclared-cell check removed;
    • the NEED start removed;
    • O1 without the DIRPERC test;
    • the _move_losses… and _owner_manager_flags_from_draws call sites deleted.
  • Mutations that pass, so the tests miss them: items 1 and 2, which I rechecked at this head.
  • Read in policyengine-uk 2.132.6: the loss relief, Class 4, total_income, esa_includes_support_component, owned_company_income_share and UC unearned income. Also read the changelog from 2.130.1 to 2.132.6 and pe-uk#2177.
  • Disclosure scan of the body, the new receipts, all commit messages, the added lines outside the generated fixtures, and the fragments.
  • CI at this head was still running when I checked: lint, wheels, select-countries and integration-uk pass; engine-free, engine-uk and engine-us are pending.

I couldn't build a spine or run a calibration (no FRS or feed access), so none of the arm figures, the refitted rates or the receipts' measurements are checked. I also couldn't check the ESAGROUP codes (item 3). I only skimmed U3 (support floor: I checked that its receipt carries no support counts), U4, C1, P1, E1, U1 and D1a beyond the Plan 5 commit, so I have no findings on them.

🤖 Generated with Claude Code

juaristi22 and others added 2 commits October 16, 2026 10:19
Vahid's review of #1299 found three code paths no test pinned. The ESA
group test's fixture adult had a work-related activity row and an
assessment-phase row together, so reading only code 3 as outside the
support group still passed. A new test keeps one ESA row per case: codes 1
and 3 give false, 2 gives true, and any other code records no group.

The SPI stage's calls to the ESA group reset and the trading-loss clearing
had unit tests on the helpers but none on the stage. An engine test runs
the stage with fake forests that draw a positive value for every target:
each redrawn row's ESA group returns to true, FRS rows keep theirs, and a
redrawn row with a profit keeps no trading loss. Deleting either call now
fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ESA group reads VAR3 on ESA rows as 1 the work-related activity
group, 2 the support group and 3 the assessment phase. The FRS 2024-25
variable listing gives that coding (UK Data Service SN 9563,
9563_frs2425_variable_listing_eul.xlsx, sheet BENEFITS_table): on an ESA
row VAR3 holds the questionnaire's ESAGroup, formatted 1 Work-related
activity group, 2 Support group, 3 Assessment phase. The docstring and
the changelog fragment now cite it (Vahid's review of #1299).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 4 commits October 16, 2026 10:19
)

The fitter's docstring said the universal row's allocated term is zero;
the code computes it like the other rows'. Under pe-uk#2177 a child whose
family takes no universal hours on their own and who uses 15 hours or
fewer gets the base hours from the allocation and no extended hours, so
counts as universal-only. That is rare at the DfE usage mean of 31.141
hours, so the term is near zero, not zero (Vahid's review of #1299). No
fitted value changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1240)

Starting a declared all-zero cell at its NEED mean does not bias the
rake, since the accommodation margin holds the cell's mean wherever it
starts, but the cell's households lose the spend they drew, at or below
the standing charge, and take the mean, as the incumbent's translation for
mobile homes has it. The changelog fragment now says so (Vahid's review of
#1299).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plan_5_repayments_from is a fixed year, not derived from the cohort. With
three-year courses a 2023/24 starter leaves in 2026 and first repays in
April 2027, so a 2026 build would let a repayer take Plan 5 a year before
the assumed cohort repays. For the 2025 build the rule holds; the comment
asks for a revisit when the build year moves (Vahid's review of #1299).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The arms (control, the head with and without the directors' dividends),
the head's written-file checks, the support floor's receipt and the sort
of the 21 per-row band entries, C1's acceptance, the plan 2 stock, the
tax-credit lines Vahid asked for and the test runs. Aggregates and
publisher figures only: no support count, effective sample size or weight
ratio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the uk-data-ports-1095-5 branch from 2dcb561 to 20c5298 Compare October 16, 2026 09:22
@juaristi22

Copy link
Copy Markdown
Collaborator Author

@vahid-ahmadi thanks. Pushed at 20c52989c. The commits up to f57c50091 are unchanged; five review commits follow it, and the receipts commit moved to the end, so git range-diff 2dcb5619b...20c52989c shows five new commits and the receipts rewritten.

From #1271

  • Round 2, item 1 (old receipts commit): María's disclosure call, outside this PR.

Findings

  1. ESA group codes: fixed in 968a9f945. test_each_esa_group_code_sets_the_support_component_on_its_own keeps one ESA row per case: codes 1 and 3 give false, 2 gives true, and 0 records no group. Your group.isin((3,)) mutation now fails the work-related activity case.
  2. SPI-stage call sites: fixed in 968a9f945. The engine test test_the_stage_resets_the_esa_group_and_clears_a_loss_beside_a_profit runs impute_uk_spi_income_support with the fake forests, which draw a positive value for every target, and asserts on the stage's output: each redrawn row's esa_includes_support_component is true, FRS rows keep theirs, and a redrawn row with a profit keeps no trading_loss. Deleting either call now fails it; I checked both.
  3. ESAGROUP codes: cited in d918388e2. The FRS 2024-25 variable listing (UK Data Service SN 9563, 9563_frs2425_variable_listing_eul.xlsx, sheet BENEFITS_table) maps VAR3 on ESA rows (BENEFIT 16) to the questionnaire's ESAGroup and gives its format: 1 Work-related activity group, 2 Support group, 3 Assessment phase. The "Code 0" to "Code 17" labels in the data dictionary are VAR3's generic range across benefits; the per-benefit formats are in that sheet. The docstring and the changelog fragment cite it.
  4. O2: both follow-up points are now in UK: track the defects uk-data is fixing that microcosm still has #1095's directors' dividends item, under "Next". Thanks for checking that UC unearned income leaves out dividends, so O1's income share is not counted twice.
  5. The Tax-Free Childcare rate: in the receipts' new U2c section. 0.9797 is an interior optimum (L-BFGS-B stopped inside 0 to 1), so the bound-hit rule does not apply. Even at a rate of 1 the frame's Tax-Free Childcare spend is 0.88 of HMRC's while its children with used accounts are 1.13 of HMRC's, so the fit balances the two misses at 0.86 and 1.11: the residual is the frame's spend per child, about 0.78 of HMRC's, not the rate. On the universal rate: with every scheme at a rate of 1 the frame already gives 1.77 times DfE's universal-only children, and at the fitted extended-hours rate about a third of the children eligible for extended hours take none and join the universal-only pool, so 0.44 meets DfE's count.
  6. The fitter's docstring: fixed in 84e1ffece. It now says the universal row's allocated term is computed and near zero, for the case you describe. No fitted value changes.
  7. The UK: data fixes to model the High Value Council Tax Surcharge (mansion tax) correctly, and the related housing and council-tax gaps #1240 fragment: 09a2c8f1b adds the line. A started cell's households take its NEED mean in place of the spend they drew, as the incumbent translation for mobile homes has it, and the rake is not biased.
  8. The Plan 5 year: María confirmed the rule for the 2025 build. 18be8d205 puts your caveat beside PLAN_5_REPAYMENTS_FROM: the year is fixed, not derived from the cohort, and is to be revisited when the build year moves.
  9. The band names: María's call under the UK Data Service rules; she will confirm before merge.

At the new head the files the review touches pass: 1,073 tests with 2 skipped (FRS spine, SPI income engine-free and engine, student loans, the take-up tool and contract, energy pricing, and the graph tests). CI is running.

🤖 Generated with Claude Code

@juaristi22
juaristi22 marked this pull request as ready for review October 16, 2026 09:31
@vahid-ahmadi

Copy link
Copy Markdown
Contributor

@juaristi22 thanks, this is my round-2 pass at 20c52989c.

Good to merge once CI is green and you have made the band-names call (item 9). Items 1 and 2 are now pinned, and my mutations fail the new tests. I checked the ESA group codes against the SN 9563 variable listing myself and they match. The two new findings below are nits.

Status of my round-1 points

  1. ESA group codes pinned one by one: fixed (968a9f945).
  2. SPI-stage call sites pinned: fixed (968a9f945).
  3. ESAGROUP codes cited: fixed and verified (d918388e2).
  4. O2 follow-ups: done (now in UK: track the defects uk-data is fixing that microcosm still has #1095).
  5. Tax-Free Childcare rate near its bound: answered in the receipts' new U2c section. I agree with the answer.
  6. Fitter docstring on the universal row: fixed (84e1ffece).
  7. UK: data fixes to model the High Value Council Tax Surcharge (mansion tax) correctly, and the related housing and council-tax gaps #1240 fragment: fixed (09a2c8f1b).
  8. Plan 5 year: fixed (18be8d205). The comment at student_loans.py:66 records the caveat and that the rule is confirmed.
  9. Small-cell band names: still open. It is your call, and the body says you will make it before merge. Nothing has changed here: the 19 dropped bands and the 12 bands under renewal are still named in the body and the receipts.

Evidence for 1 to 3 and 5

  • Item 1. test_each_esa_group_code_sets_the_support_component_on_its_own (test_uk_frs_spine.py:1430) keeps one ESA row per case. I ran five mutations of frs_spine.py:1249 and each one fails it:

    • isin((3,)) fails the WRAG case;
    • isin((1,)) fails the assessment case;
    • isin((2, 3)) fails the WRAG case;
    • support read as group == 3 fails it, along with the older test;
    • treating every code other than 2 as outside fails the no-group case.

    isin((1, 2, 3)) survives, but it is equivalent: a code-2 row also sets support, so the result can't change.

  • Item 2. test_the_stage_resets_the_esa_group_and_clears_a_loss_beside_a_profit (tests/engine/uk/test_uk_spi_income.py:941) fails if I delete spi_income.py:1241 or spi_income.py:1242. I ran it with the four SPI test files, as in round 1.

  • Item 3. I fetched 9563_frs2425_variable_listing_eul.xlsx from the UKDS documentation site (it is open documentation, not data).

    • On sheet BENEFITS_table, the ESA row (BENEFIT 16, questionnaire key 54, WageBen[7]) puts Busamtdk, ESAType and ESAGroup in the Var1, Var2 and Var3 columns.
    • The decode block gives ESAGroup as 1 Work-related activity group, 2 Support group, 3 Assessment phase.
    • The FRS 2024-25 data dictionary agrees: it labels the derived ESAGROUPHD and ESAGROUPSP "Multi-purpose variable No3".
  • Item 5. The fitter minimises the sum of squared relative misses (fit_uk_childcare_takeup.py:179). Both TFC rows scale linearly with the rate, so the optimum is (0.88 + 1.13) / (0.88² + 1.13²) = 0.980. That matches 0.9797, so it is an interior optimum, and the bound-hit rule doesn't apply. The residual being spend per child (0.88 / 1.13 = 0.78) follows from the same figures. I can't check the universal-rate explanation (1.77 times DfE, about a third joining the pool) without a spine, but it is consistent with 0.4353.

New findings

  1. nit: the body still opens with "Status: draft." (line 3 of the body). The PR is now out of draft.
  2. nit: the docstring rewrite left a ragged line at tools/fit_uk_childcare_takeup.py:296. "The expectation is the quantity a take-up rate" now breaks early before "controls". This is cosmetic, and ruff doesn't reflow docstrings.

No new disclosure problems. The new U2c section gives only ratios to HMRC and DfE totals, plus the fitted rates. The new body section and the receipt bullets carry no counts. The only other new numbers are test totals and target-row counts.

What I checked

  • Range-diff. git range-diff 2dcb5619b...20c52989c: the merge base is f57c50091, which is 2dcb5619b^, so commits 1 to 24 are identical. The five review commits are new and the receipts commit is rewritten. git diff 2dcb5619b 20c52989c touches 8 files and changes no build code apart from the two docstrings and the PLAN_5_REPAYMENTS_FROM comment.
  • Setup. A new worktree with a fresh venv (uv sync --all-packages --locked, then --extra uk), on policyengine-uk 2.132.6.
  • Tests. The FRS spine, SPI income (engine-free and engine), student-loan, energy pricing and stochastic-tools (childcare fitter) test files: 299 passed, 1 skipped.
  • Lint. ruff check . and ruff format --check on the touched Python files both pass. tools/ci_test_plan.py verify is ok.
  • Mutations. The eight listed above.
  • Citation. The SN 9563 variable listing, sheet BENEFITS_table, and the FRS 2024-25 data dictionary.
  • Disclosure. A scan of the new receipts diff, the body diff and the commit messages.
  • CI at 20c52989c. lint, wheels, select-countries, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free (3.13, 3.14) and engine-us (3.13, 3.14) were still pending when I checked.

I couldn't build a spine or run the fitter, so the U2c ratios and the universal-pool share are not checked against data.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants