Repository navigation
UK #1095 A/B fixes: the SPI age band (uk-data#628), households' HRAD from Table 8a receipts, and the SPI pay floor (uk-data#629) - #1426
Conversation
…628) The SPI tape gives age only as a band (AGERANGE), so the age drawn for each donor inside its band carries no income pattern: the stage-1 forest learned none, and twins in one band faced conditionals that differed only by that noise. The tape and reference forests now read the band code instead: the tape's own code, composite records (-1) as a band of their own, and the band of a frame person's whole-year age. at_or_over_spa keeps the State Pension split inside a band. The cross-fit fold cells key on the band, since they key on the predictor values. The drawn ages stay on the donor for the band pools. Regenerated: source_stages.json (in step with sources.yaml), the release input coverage manifest and the H2 parity fixture. The SPI engine tests read the band and count the reference forest once per cross-fit fold in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e 8a receipts The 2024-25 residential nil-rate band was GBP 250,000 all year, so a purchase at or below it owes no standard-rate SDLT and other buyers' Table 8a receipts there are surcharge. At Table 6b's gross HRAD receipts per transaction and per pound they imply about 48,100 HRAD transactions and GBP 7.05bn for other buyers, where individuals' share of all transactions, uk-data#573's rule, left them about 13,400. Households now take Table 6b's band less that; above it the shares stay (solving from receipts there goes negative above GBP 1m). The household bands become the price mix. Households' target: 151,928 transactions and GBP 63.71bn (was 186,589 and GBP 68.81bn), above uk-data#573's lower bounds; the lowest band carries 43% of households' purchases (was 54%). Published HMRC figures only. Regenerated: source_stages.json (in step with sources.yaml) and the release input coverage manifest. The H2 parity fixture is regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 of hmrc_spi_income_spine draws a recipient's annual tape pay on age, sex, region and the bridge predictors, not on hours, and the tape holds part-year workers. A copy whose FRS twin reports usual hours in an employee job could therefore draw a year's pay below what those hours earn at the minimum wage, and fail the 30 hours' and Tax-Free Childcare minimum-income tests more often than FRS rows. A new floor_spi_employment_pay_at_minimum_wage operation lifts the drawn hmrc_spi_pay leaf to weekly usual hours (hours_worked read back at 365.25/7) times 52 weeks times policyengine-uk's gov.hmrc.minimum_wage.non_apprentice rate for the twin's age, read at 30 April of the build year as the engine reads it (2024-25: 11.44 from 21, 8.60 from 18, 6.40 below). It runs after the stage-1 draw, the band-donor resample and the State Pension leaf guard, and before employment_income is derived from the leaves and stage 2 conditions on it; the transport receipt keeps the unfloored draws' uniforms. Only drawn recipients other than band carriers, whose twin has an employee main job, no trade and usual hours, and whose drawn pay is positive, are floored. Tape PAY is taxable pay; PAY alone is compared with the gross minimum because it is the leaf of the engine's gross employment_income input. The operation is declared in sources.yaml, its source_stages.json projection, the sources schema, the operation vocabulary and the HMRC income contract, and checked against a reviewed declaration. The stage evidence gains an employment_pay_floor receipt: source households rounded, weighted figures "<10" on one to nine source households, no row's value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The employment pay floor receipt gave the floored weight, the share floored and the amount added beside the eligible weight, each suppressed only on the floored side's support. Any of them with the eligible weight gives back the weight of the eligible rows the floor left alone, which could rest on one to nine source households. They are now "<10" whenever either side rests on one to nine, as the Carer's Allowance take-up receipt gives its share; the mean uplift alone does not give a side back and keeps the floored side's rule. The receipt declaration in sources.yaml, source_stages.json and the reviewed spine mapping says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, gate hourly pay uk-data#629: drawn SPI employees keep their FRS twin's hours while the tape redraws their pay, and the tape's pay is the tax year's, so part-year tape pay landed on current twins at very low hourly rates. On top of the floor from uk-cc-spipay: - The floor rate is the lower of the minimum wage for the twin's age and the twin's own hourly pay (its employment income before the draw over its annual hours), so no copy is held to a higher rate than the FRS person it copies; hours now read back at 52 a year (uk-data#531). - The floored pay is the row's current pay only: its annual share becomes the unfloored over the floored employment income, so the annual measures (the HMRC income rows) still read the tape's pay. - The receipt adds an hourly-pay check: the drawn employees' weighted share below the minimum wage before and after the floor, beside the FRS employees' share, and the eligible rows still below their floor; a share is withheld beside a side on one to nine source households. - A release-blocking stage gate, uk_stage_hmrc_spi_income_spine_hourly_pay, holds the floor invariant and the drawn share to the FRS share plus two points; it joins the spine gate scope, and the battery and contract digests move with it. Regenerated: source_stages.json (in step with sources.yaml), the sources schema, the release input coverage manifest and the H2 parity fixture. The floor's engine test gives each twin and its copy one age and pay, reads hours at 52 weeks, and checks the twin cap, the kept year of pay and the employer contribution that follows the floored pay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
María's ruling of 2026-10-29 (#1095). The SPI employment pay floor gives fewer low-paid SPI copies Universal Credit, so single people's caseload before the take-up redraw falls close to DWP's, and the redraw's usual downward move reads as about twice the 0.055 allowance; after the redraw single people end a little below the spine without the floor, all in the middle award range. The shortfall of middle and higher awards that the redraw cannot fill is the award fix's (#1095's UC awards below the top band), and the tolerance returns to 0.055 when it lands. The one tolerance also bounds the reachable pools, which realize DWP's count far more closely. The contract re-pins the gate battery's policy, manifest and fingerprint digests and the release certification's spine part; the stage-health test pins 0.12 and adds the floor's pattern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@juaristi22 thanks, this is my review of Verdict: the three items look sound, with four should items and a nit. The age band is the right fix for #628. The nil-band arithmetic reproduces the body's numbers: £265m over £625m/113,500 is about 48,100 transactions, and the target falls by about 34,700. The pay floor runs where it should, after stage 1 and the uprating and before Findings
What I ran. I read the full diff at 🤖 Generated with Claude Code |
Vahid's review of #1422 (its second and third points). - Since the move to a floor of three, rounded_count printed three to nine exactly, while extreme_count and support_band print "3-9" and the module's own rule is that no exact count narrows a small group by difference. It now reads "3-9" there too. Two tests that read an incidental exact count follow. - The module docstring and the changelog say that the SPI Public Use Tape's two conditions (SN 9422) are unchanged by the move to three. - A guard test fails when a UK runtime module, the UK spec, the gate notes or a tool still writes the old "<10" label, so that a receipt left on the floor of ten cannot land unnoticed. #1426's SPI pay-floor receipt writes it today; whichever of the two lands second moves it onto UK_DISCLOSURE_FLOOR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1426 now builds on #1422, which moves every UK disclosure floor to three (receipt_counts.UK_DISCLOSURE_FLOOR) and regenerates the national surfaces on Chronicle 53815b2 with policyengine-uk 2.165.1. - The SPI pay-floor receipt withholds through UK_DISCLOSURE_FLOOR and UK_WITHHELD: a figure on one or two source households reads "<3" where it read "<10" (Vahid's review of #1426). Its description in spi_spine.py, sources.yaml and source_stages.json, the hourly-pay gate's note and docstring, and its tests follow. Two test cases shrink their small side to two households, so they still test the withholding. - Conflicts: the SPI income imports (both sides kept), the stage-1 fold key (#1426's age band) beside #1422's savings-interest method text, and the UC take-up gate test. - Regenerated: the release input coverage manifest. The gate battery and spine-part digests are re-pinned. - The H2 parity fixture takes #1422's side here; a later commit regenerates it on the merged tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hours / 52 * weeks_in_year is a year's paid hours, not weeks (Vahid's review of #1426). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uk_stage_uc_take_up's one tolerance bounded two checks: how far a family type's caseload may move from DWP's across the redraw, which the SPI pay floor needs at 0.12, and how closely the reachable pools realize DWP's count, which the draw meets within 0.0015. The pools now keep 0.055 as maximum_reachable_pool_deviation while maximum_caseload_deviation stays at 0.12 until the UC award fix of microcosm#1095 (Vahid's review of #1426). The gate battery and spine-part digests are re-pinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spi_age_band was an ordinal predictor, so any split that did not separate the composite records' code (-1) from 16-24 put the tape's high-income amalgams in the leaves 16-24 queries land in. The band is now one of the stage-1 categorical predictors, one-hot encoded with gender, region and main source, so composites are a category of their own (Vahid's review of #1426). The predictor rule, the spec's categorical list, the changelog and the coverage manifest follow; the cross-fit fold cells still key on the band's value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table 8a's receipts do not split out the 2% non-resident surcharge, which a non-resident buyer pays on top at or below GBP 250,000. Read as HRAD surcharge, it raises other buyers' implied HRAD and lowers households', so households' nil-band purchases are, if anything, understated (Vahid's review of #1426). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stage-1 queries carry the age band's one-hot columns, not the band itself, and the pay floor's four floored households read "3-9". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No conflicts. The stricter floor guard passes on #1426's SPI and HRAD code, as do the ledger, reference, stage and graph tests. The gate digests and the coverage manifest are unchanged, and the H2 fixture regenerates byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No conflicts. The broader floor guard passes on #1426's SPI and HRAD code, as do the ledger, gate, reference, stage and graph tests. The gate digests and the coverage manifest are unchanged, and the H2 fixture regenerates byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1422 landed (11c1f4e), followed by #1430's company cars and #1432's default load. Only pinned and generated files conflicted. The gate battery and spine-part digests are re-pinned on the merged gates spec (five move), and the release input coverage manifest and the charter-H2 fixture are regenerated; the fixture checks byte-identical. policyengine-uk stays at 2.165.1, the lock main and #1428 share, and the feed pin moves to main's 08962f1. The H2 parity test and 86 related engine-free files pass (5,682 tests, 53 skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@juaristi22 Round 2 at Earlier points
On your point-3 follow-up. I agree the caseload check can go back to 0.055 and nothing is left to track. One caveat on where the evidence comes from. The gate note cites the T1426 spine at New findings
Checked and fine
CI at Verdict: my earlier points are all fixed and I agree with the 0.055 return. B10's JSA limit is the one item to settle before the JSA row's movement is quoted. 🤖 Generated with Claude Code |
…ts household share (#1095 B3: PR 2b J2 + K) The LCFS consumption stage now reads the National Travel Survey's cars (UKDS SN 5340, End User Licence; the vehicle and household tabs, survey years 2022-2024, at the interview weight W3), as the 16 October plan designed J2 and K. - J2: each car is zero-emission at VEH1103's licensed-car share times its household income fifth's relative risk among the cars NTS households keep, normalised over the frame's cars so the national share is unchanged (rule per_car_zero_emission_share_by_income_quintile). The frame ranks household_gross_income over the engine's before-housing-costs equivalence scale, as the NTS ranks gross income McClements-equivalised. A car is zero-emission in the DVLA-linked 0 g/km band; a car without a linked band is left out. The uniforms are unchanged, so only each household's threshold moves. - K: level_road_fuel keeps ONS 07.2.2's household level but levels each fuel to its share of household-paid spend (mix rule household_paid_litres): HMRC's litres times DESNZ's cars share times the share of car miles a household pays for (a kept car's private and commuting miles), at the 2024 pump prices. A car whose miles the release does not split is left out; a blank is missing, never a zero. The road_fuel_level gate rechecks each fuel against the mix it recomputes from the vendored rows. - Spec: both operations declare the same nts_cars block, which the code refuses to see differ; the NTS household tab joins lcfs_consumption under the bus stage's pin, and the vehicle tab is a new pinned private input (--nts-vehicle-tab). - obr.fuel_duties_cars stays held, its reason restated with the household-paid shares (María's ruling of 29 October 2026). - Surfaces: the schema, the source-stage projection, the graph's split sources, the spine tool, the fixture kernels, the H2 fixture (a synthetic vehicle table, W3 on its households), the coverage manifest, and the kept-tens register (the pooling floor of 10 is statistical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spi_support_channel stage draws its 10,000 FRS households uniformly and gave every copy in a region x pension-age stratum the same prior weight, which reproduced the FRS respondent mix rather than the design's population (young families, single adults and one-earner couples over- represented, households of three or more adults under-represented). Copy j of stratum s now takes M_s * share_s * f_j / sum_k f_k, with f the copy's source-household design weight (the incoming raw FRS GROSS4) over the stratum copies' mean, damped to the square root. Each stratum's channel mass stays exactly its declared share and the pension-age share stays 0.2; the hosts and their incomes do not move. The shape is declared on allocate_zero_weight_prior_mass (prior_shape: design_damped, prior_shape_exponent: 0.5; design_bounded with prior_shape_bounds is also understood), validated by the schema and refused by the stage parser if it drifts from the reviewed shape. A design shape requires DESIGN input weights, so shaping by fitted weights (microcosm#630) is refused, and the #420 guard that every copy's source household has a positive design weight is restored. The square root was chosen by re-solving the release calibration on the same hosts: the [0.5, 2] clip put more rows at the weight-ratio cap than equal weights and widened the GBP 125,000-150,000 band's standard error past the damped shape's; the damped shape kept fewer cap rows, the highest Kish ESS and loss within 2.2 percent of equal weights. The stage receipt records the shape and its adjustment's 10th, 50th and 90th percentiles with banded counts (no minimum or maximum), and the mass-change record names the shape, so the release input-coverage manifest needs its pin-pass regeneration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d6c0f25)
…ed reason The prior-shape receipt gave its percentiles and floored clipped shares whatever the channel's size, so a small scale-ladder rung of one to nine copies would report statistics resting on fewer than ten source households. It now writes "<10" in their place, as the take-up receipts do. The coverage-manifest test pinned the old allocation sentence, so it would have failed even after the pin pass regenerates the manifest; it now expects the sentence naming the design-damped shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1d22dd5)
…#1095 residual 12) d6c0f25 and 1d22dd5 left #1338 before it merged (they broke that build's property_purchases guard on its older base) and stayed on its pre-rebase branch; María's ruling of 3 November 2026 brings them back here, to be measured against B-5a. - The prior-shape receipt withholds its spread and shares for a channel of one or two copies ("<3"), on the shared floor of 3 (María's ruling of 29 October 2026), where it had used ten; its banded counts follow receipt_counts. - The pin pass: the release input coverage manifest (the support stage's mass-change reason now names the design-damped shape) and the H2 fixture regenerate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1338 review, item 4) tools/diagnose_uk_spi_prior_concentration.py reports, on any UK spine or calibrated dataset, the cell the SPI prior shape is meant to help: households with a benefit unit that has a child aged 0 to 4 and an adult whose adjusted net income is above £100,000. It gives the cell's weighted households, its Kish effective size over households and over FRS source households (a source and its copies as one unit), the share of its weight on its ten largest source households, its share on SPI-channel copies and its support in source households, rounded; a cell on one or two source households gives none of them. Development diagnostic, aggregate JSON only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s overflow the engine's On a spine, household and benefit-unit ids run past the engine's 32-bit entity ids, so the engine's household_id wrapped and the cell's households were not found in the household table. The receipt now takes memberships, source households and the SPI flag from the dataset's own tables and only values from the engine (age, adjusted net income, household weight), in the tables' row order, which it checks on the stored ages. On a calibrated dataset the release ids are 1 to N and in table order, so there the engine's ids and the tables' agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1095 B3: B-5b, VED) No stage wrote policyengine-uk's car inputs, so every household's vehicle excise duty was zero. The lcfs_consumption stage gains a declared assign_car_attributes_from_nts step (nts_car_attributes.py) on the same NTS kept cars as J2 and K: - Which cars: a household's vehicles less its members' company cars (an employer pays their duty) go one to each other adult, oldest first. A car beyond the adults is left out and counted (policyengine-uk holds one car a person). - Which fuel: a household the fuel flag leaves without road fuel keeps zero-emission cars only; one that buys fuel holds a fuel-burning car first, and each further car is zero-emission at its own per-car share. - Which car: a kept NTS car of its class in the household's income fifth, drawn by keyed uniforms at its interview weight scaled to its fifth and age band's kept cars, since fewer of the newest and oldest cars carry the registration record the CO2 bands come from. A fifth and age band with kept cars but no linked one refuses. - The inputs: car_fuel_type (a diesel registered from 2021 meets RDE2), car_co2_emissions (the band's midpoint on the scale its duty regime reads), car_engine_size (none for a zero-emission car), car_first_registration_date (1 October 2025 less the car's age, drawn within its NTS band, as an ISO date) and car_is_ved_exempt (DLA's higher-rate or PIP's enhanced mobility component exempts the member's own car, or the oldest car-holder's in their benefit unit). car_list_price keeps the engine's default. - Surfaces: the spec operation and notes (sources.yaml and source_stages.json), the schema branch and allowed kinds, the graph's person cells, the stage's person outputs (UK_LCFS_CONSUMPTION_PERSON_OUTPUT_COLUMNS, as frs_take_up's), the release export's allow-list in terminal_gates and gates.json (so the battery's policy, manifest and fingerprint and the release-cut part digests move in the microcosm-data contract), the coverage manifest, and the H2 fixture (synthetic vehicle columns; the audited string columns gain the fuel type and the date, and the doc's count is computed). - An engine test holds the step to policyengine-uk: the inputs and their defaults, the regimes' start dates, each band midpoint inside its engine band, and each regime's duty charged on the step's ISO dates. VED is measured, never bound: policyengine-uk 2.165.1 does not add it into household tax, and no target carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of 3 November) Reverts 0702f7a, 0f5be84 and 674b8a1. The square-root shape left the fit and the over-£100,000 young-child cell's concentration where they were (b5sp against b5a), and moved HICBC liability by about 3%. The SPI stages, their tests, the stage's schema branch and the spec operation return to 6fca272; the vehicle excise duty step stays. The release input coverage manifest and the H2 spine fixture are regenerated (the fixture regenerates byte-identically on --check). The concentration tool stays: it sizes the cell on any spine or calibrated dataset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' hours (Vahid's review of #1426, round 2) B10 gave contributory JSA ESA's permitted-work limit, about twice JSA's own. A copy now keeps its twin's contributory JSA only under 16 hours a week (JSA Regs 1996 reg 51(1)(a)) and below the age-related amount plus the earnings disregard (reg 56's prescribed amount is that sum less 1p), at the lower of the twin's award and the age-related amount less the earnings above the disregard (Jobseekers Act 1995 s. 4(1)(b), reg 80). ESA keeps permitted work and gains its 16-hour test (ESA Regs 2008 reg 45(4)). The amounts are policyengine-uk's gov.dwp.JSA.contrib and remunerative-work parameters and ESA's hours limit at the frame's year, beside esa_exempt_work_higher_earnings_limit; the hours are the twin's usual hours, which the copy keeps and the SPI pay floor reads. The within-limit receipt now counts support in distinct source households, banded, and withholds a weighted figure that rests on one or two of them, or whose before and after would give back a change on one or two (receipt_counts). It gave row counts. Also from the review: B7's flag is called an upper bound on the exception, since para 4(2)'s conditions are not in the FRS grid, and the UC take-up gate's note names t1426m (2b49a00), where single people, the family type that moves least, end 0.016 of DWP's count closer after the redraw. The stage spec and its schema branch (hours, weekly_hours_limit, award), the gate notes, the battery's manifest and fingerprint and the spine part digest (mirrored in microcosm-data), the release input coverage manifest and the H2 fixture follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main was red on the transport bindings' module count (#1342, counted by #1495) and the US spec-engine seed pins (re-pinned by #1496), which is what failed #1426's engine-free and engine-us jobs. The merge is clean. On the merged tree the US seed protocol and seed map match main's pins, the gate battery and certification pins match their mirrors, and the UK release input coverage manifest, the spec-engine coverage and the CI test plan are current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests read (#1095 B10) The node gets only its declared cells, and 0cddc84's work tests read the twin's hours_worked and its receipt counts by household_source_id, so the b5j spine stopped at the node. The stage now declares both, and a test holds its declaration to the work tests' columns, as the property_components test does since the 13 October twin build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…reads 8607310 gave the node hours_worked and household_source_id, so the fixture's spi_benefit_coherence inputs gain those two columns; no other node moves, and main's merge moves none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@vahid-ahmadi thanks for round 2. Your finding, both nits and the gate-note caveat are addressed, and pushed at Your point-3 follow-up (the gate note). The UC take-up gate's note now names 1. Contributory JSA now has JSA's own limits (
2. Hours. Both awards now read hours. The copy keeps its twin's usual hours ( 3. B7's flag is now called an upper bound on the exception in the rule's comment ( Also in B10: the within-limit receipt now counts support by distinct source household and withholds a weighted figure that rests on one or two of them, as the other receipts do ( Since your review (details in the body): the SPI prior shape came onto the branch and came off again (María's ruling of 3 November), and B4's HICBC diagnostics, PR 2b's J2 and K, and the VED inputs are on the branch. The HICBC gap is #1508. The builds measure each item on CI at |
|
@juaristi22 Round 3 at Earlier points
New findings
Checked and fine
CI at Verdict: all four round-2 items are fixed. The two new points are Nits: a docstring line or a receipt figure would cover each, and neither blocks. 🤖 Generated with Claude Code |
… payments (microcosm#1426 review) Vahid's third round on #1426 left two nits on spi_benefit_coherence, and #1426 merged without them. Neither contributory award is reduced for the copy's pension payments, though the copies carry the SPI tape's (private_pension_income, from PENSION). JSA Regs 1996 reg 81 takes every pound above GBP 50 a week off contribution-based JSA, and ESA Regs 2008 reg 74 half of every pound above GBP 85 a week off contributory ESA. B10 tests the copy's work alone, so the awards stand. The receipt now gives, for each award, the support and weighted award of the copies whose pension payments are above its threshold, the most a reduction could take off. A figure is withheld where one or two source households carry it or the rest of the award. JSA's threshold is read from policyengine-uk's gov.dwp.JSA.contrib.pension_disregard with the other JSA limits. ESA's is a constant, because policyengine-uk has no parameter for it. The node declares private_pension_income, so the H2 fixture moves. The docstring also says that the weekly work tests average the tape's annual pay over 52 weeks, which understates part-year claimants, above all of contribution-based JSA, which lasts at most 182 days (Jobseekers Act 1995 s. 5(1)). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@vahid-ahmadi thanks for round 3. #1426 merged before the two nits went in, so they are in #1391, in its last commit (70bd560):
🤖 Generated with Claude Code |
This PR's CI passed at 01:5xZ against main f8f543a. Main has since taken six PRs (#1345, #1493, #1467, #1426, #1465, #1391), 178 files, none of them this PR's. #1493 changes how the target-compilation kernels hash their code, so CI reruns here on today's main before the PR lands. The merge is clean over the single base f8f543a, and against main it changes only this PR's own lines, file for file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR's CI passed at 01:5xZ against main f8f543a. Main has since taken six PRs (#1345, #1493, #1467, #1426, #1465, #1391), 178 files, none of them this PR's. #1493 changes how the target-compilation kernels hash their code, so CI reruns here on today's main before the PR lands. The merge is clean over the single base f8f543a, and against main it changes only this PR's own lines, file for file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main brings #1426 (SPI age bands and #1095's B3 to B10 items, with HMRC's two High Income Child Benefit Charge rows), #1391 (QRF gate calibration) and the rest merged since 3 November. Resolved by hand: - uk_population_targets.json keeps both sides' rows: the five company car rows, then the two HICBC rows, in the targets and the registry scope, and both sides' unmapped declarations; - uprating_holds.json keeps both sides' holds; - the counts in test_uk_population_targets.py (467 targets, 405 in the registry scope, 220 unmapped declarations) and test_uk_target_references.py (1,425 active references) add both sides' rows. Regenerated on the merged tree, after syncing the venv to the merged lock: - the target references and their membership, on Chronicle 08962f1. Against main only the five car rows are new; no existing row changes; - the national compile-parity receipt; - the release input coverage manifest: the source-manifest hash, and company_cars' two pay columns; - the H2 UK spine fixture: company_cars' declaration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven PRs reached main after this PR's last CI run, which tested 52fc34c on main f8f543a: #1345, #1493, #1467, #1426, #1465, #1391 and #1473. None of them changes a file this PR changes, and git merge-tree writes tree ed52268 with no conflict. This merge starts CI on main as it now stands, including the new installed-emitter job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part of #1095. This PR collects the #1095 items that pass their measurement, and since 31 October the Phase B items too. Each item is measured in its own national build against the one before it. Every figure that rests on the SPI channel is given as a class only (SN 9422 and the disclosure rule).
A/B items (measured against the 28 October build, then against the 2 November control):
Phase B items (each measured against the build before it):
Phase B items that change nothing in the build:
Also since the first review:
paid_hours;t1426m.tools/diagnose_uk_spi_prior_concentration.py.79a533c72. It brings main's own fixes for the transport bindings' module count (Count the two modules #1342 added to the transport bindings' identity, and merge main's US re-pin (#1496) #1495) and the US spec-engine seed pins (Repin the US spec-engine contracts after the ACS local SSI and immigration stages #1496), whose absence failed this PR's engine-free and engine-us checks.How the items were measured
Control: María's national build of 2 November, cut f6e33b93 at SPI donor savings rebased to SPI's 2023-24 bands (savings follow-up to #1418) #1428's
329a99e71(policyengine-uk 2.165.1, feed 08962f1b), staged on populace-uk-private.T1426: this branch with the A/B items and Vahid's fixes, built on this machine with the control's review date (2 November) and feed.
Phase B: one build per group of items, in a chain: B-1 against T1426, B-2 against B-1, B-3 against B-2. All use the same review date, feed and lock, and all 1,339 rows are active in every build.
Main merged in: the branch now merges main at
312dc4d6b, which adds SPI donor savings rebased to SPI's 2023-24 bands (savings follow-up to #1418) #1428's last commit (2dbf7cf, SPI banding on total income net of expenses). A control rebuilt on main and this branch's merged tree measure the whole PR on the current base (below).Control (f6e33b93): 98.5% of rows within 10%, final loss 0.00529.
T1426 (A/B items and Vahid's fixes):
B-1 (B5): 98.8% within 10%, final loss 0.00527. No row moves.
B-2 (AC3, B7): 98.7% within 10%, final loss 0.00532. No row moves more than 5 points.
B-3 (B10, B9): 98.7% within 10%, final loss 0.00522. No row moves more than 5 points.
Control on main (
ctl-312d):This branch merged with main (
t1426m):frs_relationships(B7).obr.jobseekers_allowance's shortfall halves (B10).dwp.esa_claimantscomes inside 10%. The UC households-by-children rows stay near the 10% line, as they do in both controls.B-5a (B4, J2 and K) against
t1426m:lcfs_consumptiondiffers.dwp.uc.households_single_with_childrenslips outside 10%. It sits at the line in every build.b5sp(the SPI prior shape) against B-5a: 98.7% within 10%, final loss 0.00524. No row moves more than 5 points. The shape is off the branch (above).B-5b (VED) against
b5sp: onlylcfs_consumptiondiffers, and the calibration is identical: no target reads the new inputs.b5j(JSA's own limits, with VED and without the shape) against B-5a:ccc18c08f:0cddc8464with the node's declared reads, on the312dc4d6bbase.lcfs_consumption(VED),spi_benefit_coherenceanduc_take_updiffer.dwp.uc.households_children_3,dwp.uc.households_single_with_childrenandhmrc.spi_region.taxpayers_by_region_50000_70000@E12000008. None goes outside.Every build above passes every spine gate and all seven calibration gates. They measure the items on main at
312dc4d6b. Main is now merged in at79a533c72, so a control on that main and a build of the merged head are owed before a whole-PR comparison on the new base.1. The SPI stage-1 forests on the SPI age band (uk-data#628)
Why
The 2022-23 SPI public-use tape gives age only as a band (AGERANGE). The SPI income stage drew an age for each donor inside its band and trained the stage-1 tape forest on it, so the forest learned no real income pattern within a band. Twins of one band were then drawn from conditionals that differed only by that noise. uk-data#628 reports the same defect in uk-data.
What changes
age(Vahid's review). On the tape it is the AGERANGE code, with composite records (-1) as a band of their own. On a frame person it is the band of their whole-year age, through one helper shared by both frames.at_or_over_spastill carries the State Pension split inside the 65-74 band.Measurement
2. Households' HRAD at or below £250,000 from Table 8a receipts
Why
The property purchases stage (#1308, porting uk-data#573) allocates households' HRAD purchases by individuals' share of all Table 8a residential transactions (0.881) and value (0.882). #1308's review found this probably overstates households' purchases.
The residential nil-rate band was £250,000 for all of 2024-25. A purchase at or below it owes no standard-rate SDLT, so other buyers' receipts in that band are surcharge.
What changes
All figures in this section are HMRC's published Annual Stamp Tax Statistics 2024 to 2025, Tables 6b and 8a, as the pinned feed carries them.
Measurement
Households' purchases fit their new target, and no row moved by more than 5 points against the 28 October build.
obr.sdltfits within 0.25% after calibration in every build.3. Drawn SPI employees' current pay at their twin's hours (uk-data#629)
Why
The SPI income stage keeps each drawn copy's FRS twin's hours and job, while stage 1 redraws the copy's pay from the 2022-23 tape. The tape's pay covers the tax year and holds part-year workers, so part-year pay could land on a current full-time twin, at an hourly rate far below the minimum wage. Such a copy fails the minimum-income tests of the 30 hours' childcare offer, Tax-Free Childcare and the benefit cap more often than the FRS rows it copies.
What changes
floor_spi_employment_pay_at_minimum_wagelifts a drawn copy's pay to its twin's usual weekly paid hours (paid_hours, Vahid's review) × 52 × the lower of the minimum wage for the twin's age and the twin's own hourly pay.employment_income_annual_sharekeeps the HMRC income rows on the tape's pay.employment_pay_floorreceipt and an hourly-pay check. A figure resting on one or two source households is withheld as "<3", on the shared floor of 3 (Vahid's review).uk_stage_hmrc_spi_income_spine_hourly_payrefuses a spine where an eligible copy stays below its floor, or where drawn employees sit below the minimum wage more often than FRS employees by more than two points.Measurement
After the floor, drawn employees sit below the minimum wage for their age less often than FRS employees, within the gate's margin, and no eligible row stays below its floor. The HMRC employment-income bands end as without the floor, since their annual measures read the tape's pay.
B5: Carer's Allowance after salary sacrifice
salary_sacrifice, so its earnings test reads pay after sacrifice. That puts it after the CGT copies, so its draws are keyed on lineage: a copy draws as the member it copies, and a source and its copies in one stratum count as one record of their joint weight. Original rows keep their draws.B12: Scottish owners' council tax bands
uk/scotgov_chargeable_dwellings_by_band.json. Renters and the per-copy bill stay with the bill-level issue.AC3: two-child-limit births by the engine's cutoff test
uc_is_child_born_before_child_limit(a date of birth before 6 April 2017) instead of a birth year of 2017 or later. A child born from January to 5 April 2017 no longer counts as affected.B7: non-parental care and the two-child limit
frs_relationshipssetschild_limit_non_parental_caring_exceptionfor a child whom no adult of their own benefit unit claims as a natural, adopted or step child (household grid codes 3 and 4 from the child, 7 and 8 from the adult), and whom no local authority placed there. This covers kinship and friends care.B10: SPI copies' contributory JSA and ESA within each benefit's own work limits
spi_benefit_coherencezeroed contributory JSA on every SPI copy and left contributory ESA with the stage-2 forest's draw, which knows nothing of the unit's claim history. A contributory award follows the person's National Insurance record and incapacity, which the copy keeps; what the copy changes is its work.esa_exempt_work_higher_earnings_limitat the frame's year, £183.50 a week in 2024-25.obr.jobseekers_allowance's shortfall halves. The ESA rows start further below DWP's and OBR's: without the forest's extra ESA, the FRS base's own shortfall shows.dwp.esa_claimantscomes inside 10%.b5jagainst B-5a, with JSA's own limits):obr.jobseekers_allowancestarts where it did and B-3's halving stands.B9: student-loan balances to borrowers
was_wealthsplits the household's WAS student-loan balance before any plan exists, so it could sit on a member with no loan while the household's borrower held none. policyengine-uk caps a borrower's repayment at a positive balance.student_loanssets the plans, each household's balance moves to its members who hold one: by repayments where any of them repays, equally otherwise. A household with no borrower keepswas_wealth's split, and no household's total moves. The stage opens a reader-isolation boundary in the UK graph, since it rewrites a cell an earlier open-surface stage reads.B4: HICBC as measured diagnostics
obr.child_benefitfit, so it is not the number of claims. The fix belongs in the SPI channel and is UK: HICBC runs well above HMRC's count, mostly on the SPI copies (too many high-income parents in payment) #1508. Neither the SPI prior shape nor PR 2b moved it.B3, PR 2b (J2 and K): the NTS cars
lcfs_consumptionreads the National Travel Survey's cars (SN 5340, survey years 2022 to 2024, at the interview weight W3).obr.fuel_duties_carsstays held (María's ruling of 29 October), its reason restated with the household-paid shares.t1426m): onlylcfs_consumptiondiffers, and no row moves more than 5 points. The top income fifth now holds the most households whose cars are all zero-emission, and household road fuel moves from petrol to diesel at the same total.B3: vehicle excise duty inputs
b5sp): onlylcfs_consumptiondiffers, and the calibration is identical: no calibration target reads VED. policyengine-uk counts VED in household tax, so household net income falls by each household's duty; HBAI income does not change. An engine test checks each duty regime's rate on the step's own dates.Tests
CI at
1ac980d1d: all ten checks pass.Local runs on the commits since
22703ee0d:test_uk_spi_benefit_coherence.py(34 tests: JSA's prescribed amount, award cut and hours, ESA's hours, the receipt's floor), the engine test of the limits at 2024 and 2025, and the node's declared reads intest_uk_graph.py.microcosm-data's contract, the transport bindings' identity, the country spec and the spec-engine inventory and field usage (629 tests), and B10's other readers (1,004 tests). None fail.test_uk_nts_car_attributes.py, the LCFS and source-stage tests, and the engine test of the duty regimes), and the full engine-free suite at2b49a00ac(351 files, 13,226 tests; two fail only on this machine: one expects the UK extra to be absent, and the Orrery reconstruction test meets a local Node error).New and changed test files, against main:
tests/engine/uk/test_uk_spi_permitted_work_limit.pyreads ESA's and JSA's limits from policyengine-uk at 2024 and 2025;tests/engine/uk/test_uk_vehicle_excise_duty_inputs.pycharges each VED regime on the step's own dates;test_uk_nts_car_attributes.pyandtest_uk_spi_prior_concentration.py.test_uk_spi_income.py.test_uk_carers_allowance_take_up.py,test_uk_frs_relationships.py,test_uk_frs_spine.pyandtest_uk_graph.py;test_uk_lcfs_consumption.py,test_uk_measure_simulation.py,test_uk_measurement_period.pyandtest_uk_population_targets.py;test_uk_property_localisation.py,test_uk_property_purchases.py,test_uk_receipt_counts.pyandtest_uk_source_stages.py;test_uk_spi_benefit_coherence.py,test_uk_spi_income.py,test_uk_spi_pension_age.py,test_uk_spi_rank_transport.pyandtest_uk_spi_spine.py;test_uk_spine_acceptance_receipt.py,test_uk_stage_health.py,test_uk_student_loans.pyandtest_uk_target_references.py;test_uk_uc_capital_coherence.py,test_uk_uc_target_measurements.pyandtest_uk_weighted_integrity.py.test_country_spec.py, andmicrocosm-data'stest_contract.py.uk_atomic_support_fixtures.pyanduk_student_loans.py.Follow-up
79a533c72and a build of the merged head (above).Generated surfaces:
source_stages.json(in step withsources.yaml), the sources schema, the release input coverage manifest, the H2 parity fixture, B4's target references and membership, and the gate digests mirrored inmicrocosm-data's contract.🤖 Generated with Claude Code