UK SPI stage 1 keeps 4–17% of the tape's donors; replace the removed QRF tail gate with a donor-fidelity check #1276

Status (16 October 2026)

• The gate is gone. Remove the UK QRF tail-concentration gate (#1276) #1281 (merged 15 October) removed the QRF tail-concentration gate (uk_qrf_tail_concentration) and its exclusion register. So the original part B (re-arm the gate) and part D (retire UK local release: nation-grain doctrine at K=15 and 2,000 epochs, a 100,000-household release path, line-scoped gate exclusions, and the QRF tail surface #1242's two exclusions and the charitable_investment_gifts entry) no longer apply. Re-arm the UK QRF tail-concentration gate on independent draws #1277 (count independent draws) closed into this issue.
• Nothing replaces it yet. Until a replacement lands, no release-blocking check covers imputed-column tails. tools/measure_uk_weighted_integrity_baselines.py and uk_qrf_tail_concentration_columns stay as the measurement surface. docs/uk-national-release-assembly-runbook-806.md carries an interim check "until microcosm#1276 lands".
• Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 doesn't change the pool. Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 (merged 14 October) changes which quantile each SPI recipient draws, not the pool the forest is fitted on, so the resample below survives it.
• Not in the Budget release (UK local-area release: publish a K=15, 150,000-household dense dataset in at most two builds #1301).

The original write-up is in this issue's edit history. Weighted aggregates and shares only below; counts stay in local receipts.

Why the draws have little support

1. Some components are rare on the tape itself. TAXTERM has a value on 0.024% of the tape's weighted records and OTHERINC on 0.12%. At the SPI channel's size, that prevalence gives a few dozen carriers at most. So no check that requires more than 100 independent carriers can be met honestly for these columns.

2. The stage-1 resample discards most donors. uk_runtime/spi_income.py resamples the 2022-23 tape with replacement, in proportion to FACT, at DEFAULT_SPI_DONOR_SAMPLE_SIZE = 100_000, and then fits on uniform weights, as the enhanced-FRS pipeline does. The forest sees only part of each component's tape carriers (expected under the resample's design):

   stage-1 output	share of the tape's carriers kept
   hmrc_spi_pay	11.1%
   savings_interest_income	11.7%
   private_pension_income	13.4%
   dividend_income	6.2%
   self_employment_income	6.4%
   hmrc_spi_state_pension_income	13.4%
   hmrc_spi_employment_expenses	11.0%
   property_income	5.6%
   hmrc_spi_employment_benefits	7.0%
   gift_aid	4.8%
   other_investment_income	4.4%
   hmrc_spi_miscellaneous_employment_income	5.5%
   hmrc_spi_incapacity_benefit_income	17.3%
   hmrc_spi_unemployment_benefit_income	14.8%
   hmrc_spi_other_social_security_income	11.1%
   hmrc_spi_other_income	16.0%
   hmrc_spi_taxable_termination_pay	4.5%
   charitable_investment_gifts	5.0%

   For common components the loss doesn't matter, since thousands of donors survive. For rare ones it sets the shape of the column. The QRF module protects rare sign classes from its own n-of-n bootstrap, but this pre-fit resample sits upstream of that protection, and every forest then runs its own bootstrap on top (QRF amount forests lose donor design weights after the n-of-n bootstrap: extreme-tail draws are a seed lottery #481).

3. Copy rows inflate row counts and spread the mass. CGT clones copy their parent's values at split, small weights, and SPI income-band donor households carry small weights by design. Any tail check has to count independent draws, not rows.

4. Termination pay sits low against the tape. At calibrated weights the build carries roughly half the tape's weighted total, before uprating; at design weights it sits much closer. These SPI concepts are structurally zero on FRS rows, so part of the gap may be coverage rather than the draw.

5. Incapacity-benefit income dropped with Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256. Its prevalence among SPI recipients fell to about a sixth of the tape's weighted share. Most of the tape's carriers have no pay, trade or pension, and Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 leaves those twins' copies undrawn.

Remaining work

• A. The stage-1 donor pool.
  • Measure stage 1 fitted on the full tape with FACT as design weights (donor_sample_size=None is already supported), or a larger or stratified resample that keeps rare components' donors.
  • Record fit time and memory for each option, and report the per-output donor-pool shares, prevalence and weighted totals against the tape, on twin spines.
  • Decide jointly with QRF amount forests lose donor design weights after the n-of-n bootstrap: extreme-tail draws are a seed lottery #481.
  • Stage-1 parity pins and every SPI draw move, so this needs a re-pin and twin receipts. Measure on the merged base, now that Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 is in.
• B. A replacement for the removed gate, per the request of 14 October and the requirement of 15 October (comments below):
  • compare each imputed column with its donor (stage-1 outputs with the SPI tape, stage-2 outputs with the FRS rows they train on) on weighted prevalence and weighted total, with tolerances armed from a measured baseline plus recorded headroom;
  • compare the tail's shape with the donor's on independent draws: collapse CGT clones onto their parent draw, and treat each non-clone SPI row as a draw;
  • block on columns the engine reads or that feed an engine input, and on unbound columns no target would see; report the rest as diagnostics;
  • keep a concentration backstop for the large engine-read columns, re-measured on a current baseline and low enough to fail the Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462 signature;
  • acceptance cases: a replay of the Build M live default: non_sch_d_capital_gains lands at $74.6B vs its $10.2B SOI target (+635%); Schedule-D CGD route is empty #462 signature fails, the Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256 incapacity-benefit drop fails, and the clone-copy arithmetic of the 14 October build passes; each column's donor-pool share shows in the details;
  • committed notes and shipped reports carry shares, ratios and verdicts only; counts stay in local receipts, with cells under 10 suppressed.
• C. The termination-pay level. Decompose the gap against the tape into coverage (FRS rows can't carry the concept), the draw and calibration, and decide whether a bound is warranted.
• D. The incapacity-benefit draw. Decide whether Draw SPI-channel incomes at the FRS twin's conditional rank (#840) #1256's no-record rule should draw twins who carry incapacity-benefit income, or accept the drop with a receipt. The replacement check (B) must catch it either way.
• E. Retire the interim pointers once B lands: the runbook's interim check, and the notes citing this issue in microcosm-data's contract.py and in uk_runtime/weighted_integrity.py.

Sequencing

1. A runs on the merged base.
2. B can land independently of A. If A changes the pool, arm B's tolerances after A.
3. Neither is in the Budget release (UK local-area release: publish a K=15, 150,000-household dense dataset in at most two builds #1301).

Done when

• A receipt under experiments/ gives the stage-1 donor-pool shares and the chosen sampling, with cost and twin measurements.
• The replacement check blocks release on the material columns and meets its three acceptance cases.
• The termination-pay gap is decomposed, and the incapacity-benefit draw is decided.
• A national candidate built from main certifies with the replacement check in place.

Relates #481 (per-forest bootstrap), #462 and #464 (the incident and the old gate), #840 and #1256 (the stage-1 draw), #1095 and #1100 (spi_benefit_coherence), #1063, #1242, #1277, #1281, #1288.