Repository navigation
UK SPI channel after #840: residuals, post-merge measurements and stale pointers #1288
Description
Summary
#1256 merged on 14 October and closed #840. It delivered two things:
- the bridge predictors: main source, has pay, has trade, and at or over State Pension age;
- the twin-conditional draw.
#840 closed with its acceptance list unticked. Several residuals recorded in #1256's receipts (experiments/840-spi-draw-receipts.md) had #840 as their only home. This issue collects them.
The full local build on the merged head (R2 in the receipts note) is running. Its results will go on #840.
Weighted aggregates and shares only. Figures marked "design weights" come from the spine twins in the receipts note. Figures marked "calibrated" come from the first national built on 121fd4dd6. That build is not a matched twin: the build it is compared with, 21416b03e, also lacks #1242, #1260 and #1279.
#840's acceptance list
| # | Criterion | State |
|---|---|---|
| 1 | Bridge fields with adjudicated mappings | Done in #1256 |
| 2 | Draw quantiles from the donor's within-cell rank; declared; twin-deterministic | Done in #1256, as a cross-fitted twin-conditional transport rather than a within-cell rank |
| 3 | Before/after by family type and channel: the screened share and the no-award lone-parent mass | Screened share done at design weights: SPI lone parents 0.539 → 0.646, FRS 0.644. No-award mass not measured |
| 4 | Diff confined to SPI-channel columns; FRS rows byte-unchanged | Stage 17 leaves FRS rows byte-identical. End to end, FRS rows move through frame-level stages: the CGT lane, the landlord ranking, the energy rakes and salary sacrifice (receipts R1f) |
| 5 | Band replay and distributional-mass gates green | Done |
| 6 | Rebuilt candidate, with the with-children receipts and the sibling watch | R2; results on #840 |
| 7 | Every moved column signed | Open, below |
What the first calibrated national after #1256 shows
Final errors against target, 121fd4dd6, with 21416b03e in brackets:
- UC households by family: single with children −15.0% (−15.2%); couple with children +11.7% (+9.5%); one child −7.6% (−9.8%); two children −12.9% (−12.4%).
- UC spend (OBR) at initialization: −8.0% (−31.8%). Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 changed who carries the UC mass. The calibrated with-children cells barely moved.
- Childless cells at initialization: couples without children +99% (+53%); singles without children +14.5% (−11.6%). Both fit after calibration, which gets there by down-weighting.
- Bands:
- State Pension £20–30k: +12.8% (+11.0%).
- State Pension £30–40k: +13.3% (+7.5%).
- Self-employment £20–30k: +10.1% (+0.3%).
- All are inside the 25% fence.
- Top-1% weight share: 0.283 (0.280).
- Certification: the release cut is blocked by
uk_qrf_tail_concentrationonhmrc_spi_incapacity_benefit_income, which passed before, and thehmrc_spi_other_incomeexclusion is now stale. Both belong to UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276.
To measure on R2 (post on #840)
- The childless UC cells at constituency and local-authority level (the
children_0buckets), and the award bands. - Lone-parent UC carriers per seat: the number of seats under the floor, never a seat's value. Also the with-children ESS against the pre-Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 local candidate. UK local: rerun the 2025 Autumn Budget constituency analysis on the local candidate to find data gaps and per-seat support before Budget day #1241 covers the generic seat pass.
- The UC taper reform estimate. Align UK UC claim, child and period definitions #882 had −£35bn against −£17bn to −£23bn.
- The no-award SPI lone-parent mass, before and after (criterion 3).
- Housing Benefit reporters by channel, and the JSA and ESA rows. These stay on UK spine: housing benefit £2.7bn against OBR £12.1bn — pensioner and supported-accommodation HB missing after the UC migration #867 and UK spine: JSA claimants 2.8× the DWP caseload — legacy-benefit reporting after the UC migration #869.
Residuals with no other home
- Lone-parent structural shortfall.
- UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840 declared it out of scope: lone-parent families concealed inside larger FRS households.
- With Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 in, it is the main candidate explanation for the lone-parent UC cell sitting at −15% after calibration.
- First step: split the cell's shortfall into benefit-unit composition and receipt.
- The tape's low-pay tail on earning copies.
- The tape's pay is an annual tax-record amount. 22.5% of earning copies sit under £12,570, against 10.4% of their twins (design weights).
- Childless units and couples with children pass the UC screen 17–37% more often than their twins.
- The SPI channel's UC recipients per unit are 1.58× the FRS channel's for singles, and 1.35× for childless couples.
- Calibration absorbs the national counts. The local cells are R2's.
- Candidate fix: extend the no-record rule so twins at or below the personal allowance keep their own values. Measure it on twins.
- Another candidate, which also bears on residual 8: a minimum-wage floor on drawn pay (branch
uk-cc-spipay): comment.
- Stage 2 on drawn copies.
- Stage 2 sees no children and no disability.
- Drawn copies report PIP at 0.035 against their twins' 0.025, and carers at 0.012 against 0.006.
- Lone parents' copies report Child Benefit at 0.40 against 0.64. 5.7% of childless singles' copies get Child Benefit.
- Candidate fix: restore these reports from the twin in
spi_benefit_coherence. Watch the stage's disability-category assert and the tail-gate surface that the stage defines.
- Pension Credit and council-tax-benefit undershares on copies. The Housing Benefit share is on UK spine: housing benefit £2.7bn against OBR £12.1bn — pensioner and supported-accommodation HB missing after the UC migration #867; these two have no home.
- Undrawn copies rewritten downstream.
- Undrawn copies are 18% of the domain's weight.
spi_benefit_coherence, the take-up stage, the UC reporter redraw and the housing shell rewrite them. Against their twins: UC 0.322 vs 0.351, benefits in own right 0.348 vs 0.459, council tax benefit 0.082 vs 0.123.- The UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840 plan held this exemption for review (step 6).
- Candidate fix: skip benefit units and households with no drawn adult.
- State-Pension-only twins are drawn against a tape that holds mostly taxpayers (
sources.yaml, stagehmrc_spi_income_spine). There is no receipt of the lift yet. It may be behind the State Pension band movement above. - Investment and wealth on SPI rows.
- At design weights: interest +32%, dividends +9%, property income +18%. The WAS wealth draws rise with them. SPI income tax is +8.5%.
- The national band targets still fit, so this is distributional. It matters for savings, dividend, property and CGT reforms.
- Hours.
- The bridge conditions on main source, pay and trade, but not hours, and SPI copies keep their twin's hours.
- policyengine-uk reads hours: for Income Support from 2.123.10, and for ESA and JSA from 2.129.1. So drawn pay and kept hours can disagree.
- Moved from WS-E spine deferrals: known bugs and signed inconsistencies awaiting their increment #796.
- Chain order. Put the analogue targets ahead of the identity-uniform leaves; pay has the weakest within-cell rank fidelity. The manifest's note still points at UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840.
- High income × has children (HICBC). The vendored Child Benefit statistics hold opt-outs but no HICBC liability. Pick the target's definition before any Chronicle request.
- Lower priority:
- the INDUSTRY07 ↔ SIC bridge, which needs the Annex A coding check;
- the stage-2 predictors (plan D6);
- an engine test that
has_paycopies draw positive pay on the synthetic tape.
- SPI copy prior. Copies take equal prior weights within a stratum, so the channel carries the FRS respondent mix. Candidate on branch
uk-cc-spiprior: comment.
Registers and pointers
- Signed differences for the columns Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 moved (criterion 7).
spine_swap_signed_differences.jsonhas no UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840 entry; the receipts note lists the inventory. It is not a certification input. - Re-measure the
spi-channel-qrf-incidenceentry's savings-interest delta on a post-Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 spine (tools/verify_uk_spine_parity.py). - Re-point text that names UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840 when it is next touched:
- the chain-order note in
uk/spec/sources.yaml; editing it movessource_stages.jsonand the coverage digests; - the
obr.housing_benefitanddwp.jsa_claimantsreasons incalibration_measure_exclusions.json; both entries expire 2026-12-08.
- the chain-order note in
Not here
- The stage-1 donor pool, the QRF tail gate and the incapacity failure: UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276 and Re-arm the UK QRF tail-concentration gate on independent draws #1277.
- uk-data#529's LFS employee and self-employed levels, and uk-data#551: UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095.
- The employment-indicator diagnostic and the childless-couple bands: Align UK UC claim, child and period definitions #882.
Relates #840, #1256, #882, #867, #869, #1095, #796, #1276, #1241, #1131.
Activity
juaristi22 commentedon Oct 23, 2026
One more residual, and a candidate for residuals 2 and 8
Both came up while reconciling the £100k childcare costing (#1339), but neither is about childcare. They are choices in how the SPI channel builds its copies, and both came out of #1338 before it merged.
12. Copies take equal prior weights within a stratum
Cause
spi_support_channeldraws its 10,000 FRS hosts uniformly from the whole FRS.allocate_zero_weight_prior_massthen gives every copy in a region × pension-age stratum the same weight: the stratum's channel mass over its number of copies.Mechanism
Proposed change (branch
uk-cc-spiprior: d6c0f25, 1d22dd5)prior_shape: design_damped, exponent 0.5).<10under ten copies.Evidence so far
6cba5d71) on the same hosts, against a rule set in advance. The square-root shape improved the fit slightly against equal weights, put fewer rows at the weight-ratio cap, and lowered the £100k free-hours costing modestly.Before review
gates.jsondoes not move.Residuals 2 and 8: a candidate floor on drawn pay
Residual 8 is the cause: stage 1 draws tape PAY without hours, and copies keep their twin's hours. Residual 2's low-pay tail is what shows. PAY is the tax year's taxable pay, so the tape includes part-year workers. #1095's PolicyEngine/policyengine-uk-data#629 item is the same defect from uk-data's side, with why it matters and a diagnosis plan; neither is repeated here.
Candidate change (branch
uk-cc-spipay: ecb2032, 3c5d57a)hmrc_spi_income_spine,floor_spi_employment_pay_at_minimum_wage. It runs after the stage-1 draw, the band-donor resample and the State Pension guard, and beforeemployment_incomeand stage 2.gov.hmrc.minimum_wage.non_apprenticefor the twin's age. FRS rows and undrawn copies keep their values.<10.Evidence so far
property_purchasescapacity below its 80% guard, which is why it came out of UK: childcare take-up at the spring term, under-2 take-up and single-age targets behind the £100k childcare costing (policyengine-uk 2.147.2) #1338. Draw UK QRF stages from uniforms keyed on each record's id, and tie pension contributions to pay #1332 has since re-keyed every UK QRF draw, so re-measure that margin on main.Before review
spi_income.pywith Draw UK QRF stages from uniforms keyed on each record's id, and tie pension contributions to pay #1332's pension coherence. Keep both fields.Both
uk-childcare-100k-fixes-pre-main-rebase.🤖 Generated with Claude Code