Skip to content

UK SPI channel after #840: residuals, post-merge measurements and stale pointers #1288

Open

Description

@juaristi22

Summary

#1256 merged on 14 October and closed #840. It delivered two things:

  • the bridge predictors: main source, has pay, has trade, and at or over State Pension age;
  • the twin-conditional draw.

#840 closed with its acceptance list unticked. Several residuals recorded in #1256's receipts (experiments/840-spi-draw-receipts.md) had #840 as their only home. This issue collects them.

The full local build on the merged head (R2 in the receipts note) is running. Its results will go on #840.

Weighted aggregates and shares only. Figures marked "design weights" come from the spine twins in the receipts note. Figures marked "calibrated" come from the first national built on 121fd4dd6. That build is not a matched twin: the build it is compared with, 21416b03e, also lacks #1242, #1260 and #1279.

#840's acceptance list

# Criterion State
1 Bridge fields with adjudicated mappings Done in #1256
2 Draw quantiles from the donor's within-cell rank; declared; twin-deterministic Done in #1256, as a cross-fitted twin-conditional transport rather than a within-cell rank
3 Before/after by family type and channel: the screened share and the no-award lone-parent mass Screened share done at design weights: SPI lone parents 0.539 → 0.646, FRS 0.644. No-award mass not measured
4 Diff confined to SPI-channel columns; FRS rows byte-unchanged Stage 17 leaves FRS rows byte-identical. End to end, FRS rows move through frame-level stages: the CGT lane, the landlord ranking, the energy rakes and salary sacrifice (receipts R1f)
5 Band replay and distributional-mass gates green Done
6 Rebuilt candidate, with the with-children receipts and the sibling watch R2; results on #840
7 Every moved column signed Open, below

What the first calibrated national after #1256 shows

Final errors against target, 121fd4dd6, with 21416b03e in brackets:

  • UC households by family: single with children −15.0% (−15.2%); couple with children +11.7% (+9.5%); one child −7.6% (−9.8%); two children −12.9% (−12.4%).
  • UC spend (OBR) at initialization: −8.0% (−31.8%). Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 changed who carries the UC mass. The calibrated with-children cells barely moved.
  • Childless cells at initialization: couples without children +99% (+53%); singles without children +14.5% (−11.6%). Both fit after calibration, which gets there by down-weighting.
  • Bands:
    • State Pension £20–30k: +12.8% (+11.0%).
    • State Pension £30–40k: +13.3% (+7.5%).
    • Self-employment £20–30k: +10.1% (+0.3%).
    • All are inside the 25% fence.
  • Top-1% weight share: 0.283 (0.280).
  • Certification: the release cut is blocked by uk_qrf_tail_concentration on hmrc_spi_incapacity_benefit_income, which passed before, and the hmrc_spi_other_income exclusion is now stale. Both belong to UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276.

To measure on R2 (post on #840)

Residuals with no other home

  1. Lone-parent structural shortfall.
  2. The tape's low-pay tail on earning copies.
    • The tape's pay is an annual tax-record amount. 22.5% of earning copies sit under £12,570, against 10.4% of their twins (design weights).
    • Childless units and couples with children pass the UC screen 17–37% more often than their twins.
    • The SPI channel's UC recipients per unit are 1.58× the FRS channel's for singles, and 1.35× for childless couples.
    • Calibration absorbs the national counts. The local cells are R2's.
    • Candidate fix: extend the no-record rule so twins at or below the personal allowance keep their own values. Measure it on twins.
    • Another candidate, which also bears on residual 8: a minimum-wage floor on drawn pay (branch uk-cc-spipay): comment.
  3. Stage 2 on drawn copies.
    • Stage 2 sees no children and no disability.
    • Drawn copies report PIP at 0.035 against their twins' 0.025, and carers at 0.012 against 0.006.
    • Lone parents' copies report Child Benefit at 0.40 against 0.64. 5.7% of childless singles' copies get Child Benefit.
    • Candidate fix: restore these reports from the twin in spi_benefit_coherence. Watch the stage's disability-category assert and the tail-gate surface that the stage defines.
  4. Pension Credit and council-tax-benefit undershares on copies. The Housing Benefit share is on UK spine: housing benefit £2.7bn against OBR £12.1bn — pensioner and supported-accommodation HB missing after the UC migration #867; these two have no home.
  5. Undrawn copies rewritten downstream.
  6. State-Pension-only twins are drawn against a tape that holds mostly taxpayers (sources.yaml, stage hmrc_spi_income_spine). There is no receipt of the lift yet. It may be behind the State Pension band movement above.
  7. Investment and wealth on SPI rows.
    • At design weights: interest +32%, dividends +9%, property income +18%. The WAS wealth draws rise with them. SPI income tax is +8.5%.
    • The national band targets still fit, so this is distributional. It matters for savings, dividend, property and CGT reforms.
  8. Hours.
  9. Chain order. Put the analogue targets ahead of the identity-uniform leaves; pay has the weakest within-cell rank fidelity. The manifest's note still points at UK spine: composition-faithful SPI income draw — bridge predictors and rank-preserving replacement (with-children UC residual) #840.
  10. High income × has children (HICBC). The vendored Child Benefit statistics hold opt-outs but no HICBC liability. Pick the target's definition before any Chronicle request.
  11. Lower priority:
    • the INDUSTRY07 ↔ SIC bridge, which needs the Annex A coding check;
    • the stage-2 predictors (plan D6);
    • an engine test that has_pay copies draw positive pay on the synthetic tape.
  12. SPI copy prior. Copies take equal prior weights within a stratum, so the channel carries the FRS respondent mix. Candidate on branch uk-cc-spiprior: comment.

Registers and pointers

Not here

Relates #840, #1256, #882, #867, #869, #1095, #796, #1276, #1241, #1131.

Activity

  1. juaristi22 commented on Oct 23, 2026

    @juaristi22
    CollaboratorAuthor

    One more residual, and a candidate for residuals 2 and 8

    Both came up while reconciling the £100k childcare costing (#1339), but neither is about childcare. They are choices in how the SPI channel builds its copies, and both came out of #1338 before it merged.

    12. Copies take equal prior weights within a stratum

    Cause

    Mechanism

    • A uniform draw with equal weights carries the FRS respondent mix, not the population's:
      • household types with low FRS design weights (young families, single adults, one-earner couples) are over-represented in the channel;
      • households of three or more adults, which carry high design weights, are under-represented.
    • Calibration starts from this prior and does not undo it. UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069 corrected the same bias for pension age only.
    • It reaches every output read on the channel. In the childcare costing it showed as too many high-income parents of young children.

    Proposed change (branch uk-cc-spiprior: d6c0f25, 1d22dd5)

    Evidence so far

    Before review

    Residuals 2 and 8: a candidate floor on drawn pay

    Residual 8 is the cause: stage 1 draws tape PAY without hours, and copies keep their twin's hours. Residual 2's low-pay tail is what shows. PAY is the tax year's taxable pay, so the tape includes part-year workers. #1095's PolicyEngine/policyengine-uk-data#629 item is the same defect from uk-data's side, with why it matters and a diagnosis plan; neither is repeated here.

    Candidate change (branch uk-cc-spipay: ecb2032, 3c5d57a)

    • Where: a new operation in hmrc_spi_income_spine, floor_spi_employment_pay_at_minimum_wage. It runs after the stage-1 draw, the band-donor resample and the State Pension guard, and before employment_income and stage 2.
    • Rows: drawn recipients other than the band carriers, whose twin has an employee main job, no trade and positive usual hours, and whose drawn PAY is positive.
    • Floor: usual weekly hours × 52 × gov.hmrc.minimum_wage.non_apprentice for the twin's age. FRS rows and undrawn copies keep their values.
    • Receipt: support by source household, rounded. Where the floored or the unfloored side rests on one to nine source households, the floor's figures read <10.

    Evidence so far

    Before review

    Both

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions