Take SPI Scottish taxpayer status from SCOT_TXP and keep unknown regions simulable

[MaxGhenis]

Fixes #505.

What and why

create_spi labels SPI records whose GORCODE is not a UK region as UNKNOWN. HMRC's codebook (SN 9422, Annex A) gives these codes as 13 "Address abroad" and 14 "Address unknown or not available", and composite records carry -1. On the 2022-23 public use tape that is 18,859 of 836,850 records: 16,194 abroad, 353 unknown and 2,312 composite. Two things were wrong:

1. The output could not be simulated. Microsimulation(dataset=create_spi(tab, 2022)) failed while the simulation was being built, with ParameterNotFoundError: ... private_rental_prices.UNKNOWN. policyengine-uk 2.104.5 (released 2026-10-02) is the first release with Uprate rent for unknown-region households by the UK-wide index policyengine-uk#1985, which uprates rent for Region.UNKNOWN by the UK-wide index. With it, UNKNOWN simulates as built. UNKNOWN stays the right label: none of these records has a UK region, and relabelling them would distort regional totals.
2. Scottish taxpayer status ignored HMRC's own flag. The tape has SCOT_TXP ("Indicates whether the record is treated as Scottish, for tax purposes"), and its liabilities (TOTTAX_DEVO_TXP, TAX_ST_*_DEVO_TXP) are "based on SCOT_TXP or WELSH_TXP". The dataset didn't carry the flag, so policyengine-uk derived pays_scottish_income_tax from the region. The two disagree for 2,000 records. HMRC notes that the region reflects residence at the end of the tax year, which can differ from Scottish taxpayer status. Records with an address abroad and composite records can also be Scottish taxpayers. HMRC built composites from Scottish, Welsh or other taxpayers only, never a mix.

Changes:

• datasets/spi.py:
  • Set pays_scottish_income_tax from SCOT_TXP. HMRC documents the flag as . or 1, but the 2022-23 tape holds 0 or 1; both parse. WELSH_TXP is not used, because policyengine-uk has no separate Welsh rates of income tax (no Welsh parameters under gov.hmrc.income_tax in 2.93.0 or 2.121.0).
  • The REGION_MAP comment now gives the codes correctly.
  • Add model_simulates_unknown_region(). It checks that the imported policyengine-uk can build a simulation of one SPI-shaped household labelled UNKNOWN, once per process (a few seconds). A failure means "cannot simulate" only if the same household labelled SOUTH_EAST builds, which is exactly when relabelling helps. Otherwise the SOUTH_EAST household's error is raised.
  • The probe establishes that the representative household can be built, not that every SPI dataset will simulate. The dataset-shaped income-tax tests check the real behaviour on the supported releases and years.
  • The household mirrors the fixture that policyengine-uk's own test_rent_uprating.py simulates with UNKNOWN, so the model keeps supporting that shape.
  • The probe checks the imported code rather than the installed version, because the two can differ: make data-local puts a policyengine-uk checkout on PYTHONPATH ahead of the installed release.
• utils/incomes_projection.py:
  • load_spi_dataset relabels UNKNOWN as SOUTH_EAST only when the imported model cannot simulate UNKNOWN. This replaces create_spi output cannot be simulated: unknown regions crash policyengine-uk, and Scottish taxpayer status ignores SCOT_TXP #505's to-do ("remove the relabelling once the lock reaches #1985"): the relabel switches itself off at the relock, with no follow-up PR.
  • ensure_spi_dataset also rebuilds a cached spi_2022_23.h5 that predates the flag. Without that, a stale cache would silently fall back to deriving the flag from the region.

Horizon. policyengine-uk carries a single-year dataset's inputs forward to 2030 (extend_single_year_dataset, end_year=2030, in 2.93.0 and 2.121.0). From 2031 it computes pays_scottish_income_tax from the region again, as it does for any input variable that has a formula. A two-person synthetic check on both releases confirmed this: the input held in 2029 and 2030, and the region rule replaced it in 2031 and 2032. The tests cover 2022, 2026 and 2030.

Evidence

Everything here is aggregate. No count under 10 records is shown. Each count is at least 10, and so is its complement within its group. A count of zero counts as under 10, so no figure here implies one by differencing.

• HMRC tape documentation for 2022-23 (SN 9422) and 2020-21 (SN 9121) covers:

  • GORCODE codes 1-14;
  • the SCOT_TXP and WELSH_TXP definitions;
  • how composite records are built;
  • the note that Scottish taxpayer status can differ from residence.

  SCOT_TXP is on both tapes.

• Tape counts (2022-23):

  • GORCODE 11 but not a Scottish taxpayer: 1,394 records (90,000 weighted).
  • Scottish taxpayers outside GORCODE 11: 606 records (33,000 weighted).

• Historical income-tax evidence against HMRC's own TOTTAX_DEVO_TXP, on the full 2022-23 tape, collected before this refresh. Both datasets ran under policyengine-uk 2.104.5, because the pre-batch create_spi output could not be simulated under the then-locked 2.93.0. "main" in this historical table is create_spi at b45c373. Groups overlap. The within-£10 column is a weighted share (by FACT), not a share of records.

Records	Count	Weighted	Weighted share within £10 of HMRC: main → this PR	PolicyEngine tax: main → this PR	HMRC tax
GORCODE 11, not a Scottish taxpayer	1,394	0.090m	65.9% → 98.2%	£0.227bn → £0.215bn	£0.215bn
Scottish taxpayer, GORCODE not 11	606	0.033m	39.1% → 81.3%	£1.479bn → £1.519bn	£1.528bn
Flag disagrees with region (the two rows above)	2,000	0.123m	58.7% → 93.7%	£1.705bn → £1.733bn	£1.743bn
GORCODE 13, 14 or -1	18,859	1.024m	87.1% → 87.5%	£42.635bn → £42.670bn	£42.911bn
Known region, flag agrees with region	816,231	49.581m	96.69% → 96.69%	£201.167bn → £201.167bn	£201.765bn
Flag agrees with region, any region	834,850	50.591m	96.52% → 96.52%	£242.408bn → £242.408bn	£243.238bn
All	836,850	50.714m	96.43% → 96.51%	£244.114bn → £244.142bn	£244.981bn

• Of the 2,000 records where the flag and the region disagree, 1,074 have different income tax under this PR in 2022, and 1,166 in 2026 (same tape and release).
• policyengine-uk releases: #1985 merged as 1c78908e on 2026-10-02. git tag --contains puts it first in 2.104.5, and 2.104.4 does not contain it. PyPI shows 2.104.4 uploaded at 11:12 UTC and 2.104.5 at 11:15 UTC that day. The latest release when these tests ran (2026-10-05) was 2.122.0.

Invariants

1. The region follows GORCODE alone. Codes 1-12 map to HMRC's regions, and every other code becomes UNKNOWN. Every label is a Region member, and Scottish taxpayers with codes 13, 14 or -1 are not relabelled SCOTLAND.
2. pays_scottish_income_tax is true exactly when SCOT_TXP parses to 1, whatever the GORCODE and however "no" is written (., 0 or blank).
3. With equal incomes, income tax depends only on SCOT_TXP. This holds in the data year, after uprating (2026), and in 2030, the last year policyengine-uk carries dataset inputs to.
4. The region label does not move income tax. Building with UNKNOWN or with the legacy SOUTH_EAST gives equal income tax record for record, in 2022, 2026 and 2030.
5. Records whose SCOT_TXP agrees with their region get exactly the income tax of the old region-derived rule, so only records where the two disagree can change. This is tested in 2022, 2026 and 2030.
6. model_simulates_unknown_region():
   • is false exactly when the UNKNOWN household fails and the same household labelled SOUTH_EAST simulates;
   • raises the SOUTH_EAST household's error when both fail;
   • for a final release not installed from a direct URL, whose imported economic_assumptions.py is the installed file and matches its RECORD, is true exactly when that release is 2.104.5 or later.
7. load_spi_dataset changes a region label only when the imported model cannot simulate UNKNOWN. A cached H5 is reused only if it has the release year and the flag.

Tests

All in tests/test_spi_build.py. SCOT_TXP is added to the fake-tape columns there and in tests/test_spi_allowance_deductions.py.

• test_create_spi_region_and_scottish_taxpayer_invariants[0|.|blank] (invariants 1-2): exhaustive over GORCODE ∈ {-1, 1…14, 99} × SCOT_TXP ∈ {no, 1}, for each encoding of "no". It checks against HMRC's codebook mapping written out in the test, not REGION_MAP.
• test_create_spi_scottish_taxpayer_status_survives_h5_round_trip: the flag and UNKNOWN survive save and load.
• test_spi_income_tax_follows_scottish_taxpayer_flag (invariant 3): uses the legacy label, so it also runs under the historical 2.93.0 lock.
• test_spi_income_tax_changes_only_where_scottish_flag_and_region_disagree (invariant 5): compares against the same dataset with the flag column dropped, which is main's rule. It uses UNKNOWN where the model can simulate it, and SOUTH_EAST otherwise.
• test_create_spi_output_with_unknown_region_can_be_simulated (invariant 4): uses the default UNKNOWN label. Where the imported model cannot simulate UNKNOWN, the build must raise ParameterNotFoundError for private_rental_prices.UNKNOWN and nothing else, and the test reports an xfail. Otherwise it runs in full.
• Invariant 6:
  • test_unknown_region_probe_reads_the_simulation[…]: three cases. It simulates; the rent index is missing; or a non-parameter error that relabelling cures. In each case the test checks which households were simulated.
  • test_unknown_region_probe_raises_failures_relabelling_would_not_cure[…]: an unrelated parameter ending in .UNKNOWN, an unrelated parameter, and a schema error, each also hitting the SOUTH_EAST household. Each must be raised as the SOUTH_EAST household's own error, with the UNKNOWN household's distinct error as its context.
  • test_unknown_region_probe_agrees_with_policyengine_uk_release. It skips:
    • pre-release, dev and local versions;
    • direct-URL and editable installs;
    • an imported tree that is not the installed one;
    • an economic_assumptions.py (the module #1985 changed) that does not match the release's RECORD.
• Invariant 7:
  • test_income_projection_loads_local_h5_dataset[False|True]
  • test_income_projection_rebuilds_spi_dataset_without_scottish_flag, which uses real H5 files.

Historical results from before this refresh (both files, TESTING=1, old PR head 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c; these were not rerun in this refresh session):

policyengine-uk imported	Installed metadata	Result
2.93.0 (historical lock)	2.93.0	32 passed, 1 xfailed
2.104.4 (last release without #1985)	2.104.4	32 passed, 1 xfailed
2.104.5 (first with #1985)	2.104.5	33 passed
2.121.0	2.121.0	33 passed
2.122.0 (latest on 2026-10-05)	2.122.0	33 passed
2.104.5 checkout on PYTHONPATH	2.93.0	32 passed, 1 skipped (the release test, by design)
2.104.4 checkout on PYTHONPATH	2.121.0	31 passed, 1 skipped (the same), 1 xfailed

The last two rows are the version skew the first review found; the old gate read the metadata there and got the answer wrong both ways. The RECORD check was also exercised directly on a 2.121.0 install. It reports a mismatch after a one-line edit to economic_assumptions.py, and a match once the edit is removed.

Historical mutation checks from before this refresh. Each had to fail at least one test; all ran under the then-locked 2.93.0 unless noted. These mutation checks were not rerun in this refresh session.

Mutation	Tests that fail
Scottish status from GORCODE == 11 (main's rule)	6
Probe always true	9
Probe always false (run on 2.104.5)	8
Probe without the known-region control	5
Control household labelled UNKNOWN	5
Probe reads only parameter errors	1
Cache ignores the missing flag	1
Relabel unconditionally	1
Relabel Scottish taxpayers as SCOTLAND	5
Treat any non-zero SCOT_TXP as Scottish	2
Compare SCOT_TXP without parsing it	1
REGION_MAP[8] set to LONDON (survived round 1)	3
Unknown codes labelled with a non-region (round 1's xfail accepted the resulting error)	6
Raise the UNKNOWN household's error instead of the control's (survived round 3)	3

policyengine-uk lock

The 10/8 release batch landed on 10/7 as uk-data 1.58.0. Its main head 4cbedbecca352f52dffe47e849553c2759a00711 pins policyengine-uk 2.122.2 and policyengine-core 3.32.13. This refresh inherits those lock and dependency changes by merging main. A separate one-line fix aligns the lock's local-project version with pyproject.toml at 1.58.0; no dependency version changes.

2.122.2 is newer than the first UNKNOWN-capable release, 2.104.5. The refreshed validation below confirmed that the unknown-region simulation test passes in full, model_simulates_unknown_region() is true, and load_spi_dataset keeps UNKNOWN without the legacy SOUTH_EAST relabel. The compatibility fallback remains for imported models that cannot simulate UNKNOWN.

Impact: completed real post-refresh build

Measured from the completed files on 9 October 2026. All results are aggregates.

Baseline: uk-data 1.58.0 release tip 57ce68ac14ebe2bb6da1965b30555a5a1a919488, with the seeded-dropout patch 348dd493c7cba66b329a384410e4bb21de97cd4d, built at 1fc2a372317b19c59ee6ce4d309a2fc6e7b489d2. After: PR head 4fbd8be43b7dc550d9f6c2871ddc7270fec38c01, with the same seed patch, built at b3ea202266bf695f6d6d8371ef7fe4cedcd18612. The completed build finished at 2026-10-09T17:45:08Z with exit 0. No dataset was rebuilt to finish this report.

Both builds and all reported simulations use policyengine-uk 2.122.2 and policyengine-core 3.32.13, Python 3.13.9. Build settings: 512 epochs per calibration, dropout seed 0, PE_UK_DATA_OA_CLONES=1, TESTING unset, the same prerequisites and frozen HTTP responses. Calibration year 2025, final FRS input year 2024; simulation years 2025 and 2026. The seed patch IDs match. Relative to the release tip, implementation changes are confined to #540’s SPI/loader files and tests; other changes are release bookkeeping.

National amounts from fresh Microsimulation runs on efrs_combined.h5 and efrs_ud540.h5 (£bn; baseline → completed #540 build):

Measure	2025	2026
Income tax	297.803997 → 297.803997	317.626600 → 317.626600
Personal National Insurance	53.144141 → 53.144141	55.516100 → 55.516100
Employee NI	50.529990 → 50.529990	52.771899 → 52.771899
Employer NI	153.997092 → 153.997092	160.587468 → 160.587468
VAT	317.488795 → 317.488795	326.025257 → 326.025257
Universal Credit	79.431270 → 79.431270	83.541821 → 83.541821
Housing Benefit	8.869765 → 8.869765	9.224673 → 9.224673
Pension Credit	6.539324 → 6.539324	7.232308 → 7.232308
Child Benefit	16.245777 → 16.245777	16.877560 → 16.877560
Council Tax Benefit	1.893456 → 1.893456	1.938145 → 1.938145
Council Tax Reduction	1.893456 → 1.893456	1.938145 → 1.938145
State Pension	123.945813 → 123.945813	128.789549 → 128.789549
PIP	27.127360 → 27.127360	28.269706 → 28.269706
Household net income	1,661.190443 → 1,661.190443	1,709.528738 → 1,709.528738

Positive-value counts, in millions of each variable’s own entity. Population and household totals include all rows. All counts use direct simulation weights.

Measure (entity)	2025	2026
Income tax (person)	39.979124 → 39.979124	41.116678 → 41.116678
Personal National Insurance (person)	28.965553 → 28.965553	29.386863 → 29.386863
Employee NI (person)	26.865583 → 26.865583	27.213810 → 27.213810
Employer NI (person)	29.586656 → 29.586656	29.714479 → 29.714479
VAT (household)	31.208496 → 31.208496	31.327089 → 31.327089
Universal Credit (benunit)	6.370731 → 6.370731	6.363323 → 6.363323
Housing Benefit (benunit)	1.308782 → 1.308782	1.307325 → 1.307325
Pension Credit (benunit)	1.477347 → 1.477347	1.484096 → 1.484096
Child Benefit (benunit)	8.150683 → 8.150683	8.166759 → 8.166759
Council Tax Benefit (benunit)	3.054764 → 3.054764	3.082889 → 3.082889
Council Tax Reduction (household)	3.049103 → 3.049103	3.077207 → 3.077207
State Pension (person)	11.512814 → 11.512814	11.427332 → 11.427332
PIP (person)	3.580184 → 3.580184	3.593789 → 3.593789
Household net income (household)	30.996711 → 30.996711	31.112865 → 31.112865
All people	69.666618 → 69.666618	69.931358 → 69.931358
All households	31.243052 → 31.243052	31.361776 → 31.361776

Poverty rates use person weights; BHC/AHC means before/after housing costs.

Measure	2025 rate %	2025 people m	2026 rate %	2026 people m
Absolute poverty BHC	15.635956 → 15.635956	10.893041 → 10.893041	14.858885 → 14.858885	10.391019 → 10.391019
Absolute poverty AHC	20.719246 → 20.719246	14.434397 → 14.434397	20.167177 → 20.167177	14.103179 → 14.103179
Relative poverty BHC	17.810271 → 17.810271	12.407813 → 12.407813	16.766823 → 16.766823	11.725266 → 11.725266
Relative poverty AHC	21.356477 → 21.356477	14.878334 → 14.878334	20.957320 → 20.957320	14.655737 → 14.655737

Scottish status is pays_scottish_income_tax. Scottish income tax here means all income_tax of flagged people, including tax on savings and dividends. NSND is earned_income_tax + property_income_tax before downstream reliefs. Status counts include people with no positive liability; positive taxpayer counts also require income_tax > 0. All amounts and counts below use person weights. The enhanced FRS source flag household_is_spi_synthetic identifies FRS donor households with SPI-imputed incomes, rather than raw HMRC SPI tape records. Neither enhanced-FRS file contains an explicit pays_scottish_income_tax input, so the installed engine derives its status from the FRS donor household’s country.

Dataset / year / source	Scottish tax £bn	Scottish NSND £bn	Scottish-status people m	Positive Scottish taxpayers m
EFRS / 2025 / all	22.442166 → 22.442166	19.474620 → 19.474620	5.825043 → 5.825043	3.394593 → 3.394593
EFRS / 2025 / FRS	19.421628 → 19.421628	16.564754 → 16.564754	4.336106 → 4.336106	2.601537 → 2.601537
EFRS / 2025 / SPI_synthetic	3.020538 → 3.020538	2.909866 → 2.909866	1.488936 → 1.488936	0.793056 → 0.793056
EFRS / 2026 / all	23.875370 → 23.875370	20.671026 → 20.671026	5.847178 → 5.847178	3.452596 → 3.452596
EFRS / 2026 / FRS	20.631071 → 20.631071	17.557409 → 17.557409	4.352584 → 4.352584	2.651222 → 2.651222
EFRS / 2026 / SPI_synthetic	3.244299 → 3.244299	3.113617 → 3.113617	1.494594 → 1.494594	0.801374 → 0.801374
FRS / 2025 / all	20.904550 → 20.904550	20.614001 → 20.614001	5.467110 → 5.467110	3.019287 → 3.019287
FRS / 2026 / all	22.225309 → 22.225309	21.915725 → 21.915725	5.487885 → 5.487885	3.110664 → 3.110664

UNKNOWN-region rows in the finished files, using direct year-specific simulation weights. SPI-synthetic rows retain donor geography; the counts here describe generated FRS rows, not raw HMRC tape records. Public counts below 10 rows and their weights are suppressed to preserve the existing PR disclosure convention; exact computed aggregates remain in the local impact report.

Dataset / year / source	Person rows	Person weight m	Household rows	Household weight m
EFRS / 2025 / all	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed
EFRS / 2025 / SPI_synthetic	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed
EFRS / 2026 / all	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed
EFRS / 2026 / SPI_synthetic	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed
FRS / 2025 / all	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed
FRS / 2026 / all	Suppressed (<10 rows)	Suppressed	Suppressed (<10 rows)	Suppressed

National target fit from the gzipped calibration logs, at the last saved-weight logged epoch 510 of each 512-epoch solve. name=UK selects the national matrix, which contains UK, country and region constraints. Within 10% means abs(estimate-target)/abs(target) < 0.1. There are 645 targets; the one zero target counts outside that share and is omitted from the median over 644 nonzero targets. These log fits precede final public-service/fuel scaling and differ from the logged dropout training metric.

Calibration stage	Targets within 10%	Median absolute percentage error
Constituency	560/645 (86.821705%) → 560/645 (86.821705%)	4.034333 → 4.034333%
Local authority	562/645 (87.131783%) → 562/645 (87.131783%)	4.080685 → 4.080685%

Both decompressed calibration logs match their baseline counterparts exactly. All four HDF5 files open successfully and match their respective checksum manifests; the two completed-build file hashes differ from the baseline. The detailed HDF5 array-content audit is retained with the local impact report.

Across 2025 and 2026, the enhanced FRS has 0 changed national totals, counts or poverty results; income tax is £297.803997bn → £297.803997bn in 2025 and £317.626600bn → £317.626600bn in 2026. Scottish-flagged income tax is £22.442166bn → £22.442166bn in 2025 and £23.875370bn → £23.875370bn in 2026, with 5.825043m → 5.825043m Scottish-status people in 2025. The constituency calibration has 86.821705% → 86.821705% of national targets within 10%, with median error 4.034333% → 4.034333%.

The expectation of an essentially unchanged enhanced FRS holds: the national measurements and Scottish measurements are exactly equal in both years, including the FRS/SPI source split.

Why the effect is zero: the enhanced-FRS pipeline trains its income imputation through generate_spi_table (datasets/imputations/income.py), not create_spi. #540 changes only comments in the SPI mapping constants that the imputation imports (REGION_MAP, AGE_RANGES and the release constants keep their values). Calibration reads the committed incomes_projection.csv, which this PR leaves unchanged. create_income_projections builds that file from HMRC's official income-band tables and uprating factors, not from SPI microdata or Scottish taxpayer flags, and it holds national income bands only. So #540 reaches the standalone SPI dataset (create_spi), and regenerating the projection would not carry it into the calibration targets.

Separately audited earlier standalone SPI evidence, retained from the interrupted job. These numbers come from its saved real Microsimulation runs on the full 2022–23 tape at 57ce68a and 4fbd8be4, age-imputation seed 0, under the same engine. They agree exactly with the saved run log. They describe standalone SPI materializations, not the finished enhanced FRS files, and were not rerun or rebuilt in this reporting job.

Standalone SPI measure	2025	2026
All income tax (£bn)	325.961675 → 326.050872	347.201824 → 347.294259
Scottish-flagged income tax (£bn)	20.894128 → 22.450020	22.368019 → 23.983622
Scottish-flagged NSND (£bn)	20.221507 → 21.502210	21.578518 → 22.909246
Scottish-status people (m)	4.163813 → 4.105313	4.179636 → 4.120913
Positive Scottish taxpayers (m)	3.120414 → 3.104630	3.169463 → 3.153536

Earlier standalone SPI UNKNOWN aggregate	Before → after
Raw 2022–23 tape records	18,859 → 18,859
Raw 2022–23 FACT weight (m)	1.023644 → 1.023644

Limitation: no standalone SPI HDF5 was retained. Fresh 2025/2026 UNKNOWN-region weights for raw SPI could therefore not be measured from a saved file within the no-rebuild scope. The draft scripts reconstructed those weights from population growth; those extrapolated results are excluded. The year-specific UNKNOWN weights above are fresh measurements of the finished enhanced FRS/FRS files. The raw tape count and FACT weight are earlier measured input aggregates, not year-specific simulated weights.

The completed build logged 22 HTTP requests and 0 cache misses. The response log matches the baseline after normalising its one HMRC fetch to a cache hit. Both build logs show exit 0. Failed to download/parse OBR, failed target sources and errors/tracebacks each have no matches in either log. The 10/8 baseline log was available and confirms the same Class 2 and Class 4 NICs “no row matched” warnings: six of each per build. These warnings mean those component targets were unavailable in both builds.

The remaining shared warnings are constant-zero employment/self-employment imputations (two each), constant-zero incapacity benefit (one), skipped maternity_allowance_reported (one), no Northern Ireland OA crosswalk (one), and skipped OA assignment for 5,660 Northern Ireland households (one). Both logs contain 20 warnings in total. The warnings are unchanged.

Post-batch refresh (10/7)

• New head: 4fbd8be43b7dc550d9f6c2871ddc7270fec38c01, a fast-forward descendant of 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c, with main 4cbedbecca352f52dffe47e849553c2759a00711 as the merge commit's other parent.
• Resolved the single end-of-file conflict in tests/test_spi_build.py by keeping both the PR's region/flag invariants and unknown-region simulation tests and Supply is_claimant_or_partner from the FRS adult table #524's test_create_spi_marks_every_taxpayer_as_their_benefit_units_claimant. The shared fake-tape helper supplies both SCOT_TXP and main's SEINC_NUM.
• Supply is_claimant_or_partner from the FRS adult table #524's SPI claimant input is retained alongside the independent SCOT_TXP-based Scottish flag. Draw SPI incomes within earnings groups set by FRS employment status #529's income draws and caches retain their earnings-group semantics. Stop SPI-synthetic rows carrying benefit claims nobody observed #514's SPI-synthetic reported-benefit rules remain intact. These downstream imputations preserve the supplied Scottish flag and household geography; none re-derives Scottish status from region. The documented model input horizon remains 2030.
• Every merge-touched file was audited at merge commit a22fb5daffa29d0ff28dc27d70775285202ff3f2: all 70 main-only paths matched main exactly, all three PR-only paths matched the old PR head, and both shared files preserved each branch's changes. No main or PR hunk was dropped; no conflict markers remain. The only follow-up changes uv.lock's local-project version from 1.57.4 to main's released 1.58.0, fixing uv lock --check without changing any dependency pins.
  Ran the following under policyengine-uk 2.122.2 / policyengine-core 3.32.13, with TESTING=1, sequentially in the foreground, one file per pytest process, -p no:cacheprovider and no xdist. All paths are under policyengine_uk_data/tests/.

Test file	Passed	Skipped	Failed	Xfailed
test_spi_build.py	33	0	0	0
test_spi_allowance_deductions.py	1	0	0	0
test_spi_donor_benefit_rules.py	13	0	0	0
test_spi_income_earnings_groups.py	15	1	0	0
test_claimant_or_partner.py	11	2	0	0

Total: 73 passed, 3 skipped, 0 failed, 0 xfailed. The three skips need a built FRS/enhanced FRS. The unknown-region simulation test passed in full, rather than xfailing. A separate synthetic-H5 check used the real capability probe: it returned true, load_spi_dataset kept UNKNOWN without relabelling to SOUTH_EAST, and the Scottish flag was preserved. uv lock --check passed after the one-line metadata fix; the four PR Python files pass ruff format --check.

The initial harness attempt had 15 passes and 18 setup errors because the configured temporary-directory parent was missing. Creating it fixed the harness; the final SPI run above has no errors. No source changes were needed for that rerun.

• The enhanced-FRS build completed on 9 October 2026; finished-file measurements and separately audited earlier standalone SPI simulations are recorded in the impact section above. No additional dataset build was run to finish this report.

Review

Independent GPT-6.1 Sol reviews.

Round 1 (9fc0e29): REQUEST CHANGES.

1. The version gate could misread the imported model. It is now a capability probe on the imported code.
2. The tests could accept broken behaviour. They now:
   • carry HMRC's mapping in the test;
   • xfail only on the missing private_rental_prices.UNKNOWN index;
   • include an invariant 5 test.
3. The agreement row was narrower than its label. It is relabelled, and an any-region row has been added.
4. The flag input holds only through 2030. This is documented, and the tests run to 2030.
5. The supporting cross-tab's suppression was incomplete. That local output has been deleted.

Round 2 (3c37ef8): REQUEST CHANGES. Round 1 findings 2-4 resolved; 1 and 5 partly.

1. Blocking: the exact full-tape equality statements implied complementary counts of zero. They are removed from this body, which keeps only the aggregate table and counts whose complements are at least 10. The validation script now applies that secondary suppression to its own output. Invariants 4 and 5 are claimed from the synthetic tests only.
2. The probe's error filter matched any .UNKNOWN' in the message. It now uses a known-region control instead of matching text.
   • The reviewer suggested matching error.name, but that cannot work: AttributeError.__init__ resets name to None, so every ParameterNotFoundError carries name=None (checked on Python 3.13).
   • The control also covers the reviewer's schema concern. The household is the shape policyengine-uk itself tests with UNKNOWN, and a failure that the SOUTH_EAST household shares is raised.
3. The differential test treated any matching path as a final release. It now skips pre-release, dev, local, direct-URL and edited installs, as listed under Tests.
4. Nit: the within-£10 column is labelled as weighted, and "latest" is dated.

Round 3 (c65d381): APPROVE WITH CHANGES. Round 2 findings 1, 2 and 4 resolved; 3 partly. All seven result rows were independently reproduced. The disclosure audit passed (93 assertions). Three nits remained, all fixed at 4f86cee:

1. The release-provenance wording claimed more than the checks show. The test and this body now describe the checks exactly: final release, not a direct-URL install, and economic_assumptions.py installed and matching RECORD.
2. The probe's guarantee is now scoped to building the representative household.
3. The shared-failure test now gives the two households distinct errors and requires the control's. The reviewer's surviving mutant (re-raise the UNKNOWN error) now fails 3 tests.

Round 4 (4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c): APPROVE. This was the last independent Sol review before the batch refresh, as recorded by the hub; CI was green at that head. The two delta reviews below cover the refreshed head.

Delta review 1 (refreshed head 4fbd8be43b7dc550d9f6c2871ddc7270fec38c01): REQUEST CHANGES. No code defect. The one blocker was that the impact had not come from a real build of this head.

Delta review 2 (same head, after the real build): APPROVE. The reviewer independently:

• compared the completed files with the baseline: all 1,854 enhanced-FRS arrays and 1,284 FRS arrays match exactly, and the two files are distinct (different paths, inodes and SHA-256);
• reproduced income tax, Universal Credit and Scottish-flagged income tax for 2025 and 2026 on both files;
• confirmed from the code the reason for the zero effect given in the impact section.

Its one nit, stating that reason in this body, is done.

Release

Held out of the 10/8 batch, which landed on 10/7 as uk-data 1.58.0. This PR is for the next uk-data batch, on Max's go (d833). The real build and both delta reviews are complete at this head, so the PR is ready for that batch.

This PR was rescued from an interrupted session's uncommitted patch and finished here. Changes from that patch:

• The minimum model version is 2.104.5 rather than 2.104.1. The patch's version would have run, and failed, the test on 2.104.1-2.104.4. It is now checked through what the imported model can do.
• The relabel in the loader is gated on that check, rather than left as a to-do.
• Stale-cache handling is new.
• The .-encoding test is folded into the exhaustive invariant test.
• The income-tax tests are new: they run under the lock and in uprated years.

axiom: n/a: SPI dataset construction, no policy rule changes

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub

• Head: 4f86ceeeb844d6fcd76f8da7e68e5d2a8062525c.
• Independent review: four rounds of GPT-6.1 Sol on Subfleet. Rounds 1 and 2 requested changes, round 3 approved with changes (three nits), and round 4 APPROVE at this head, with all three nits resolved and nothing else changed. The PR body's "Review" section summarises each round. The reviews are kept with the hub's records.
• Passed at this head:
  • gh pr checks, all green. CI ran 646 passed, 3 skipped and 2 xfailed; one xfail is test_create_spi_output_with_unknown_region_can_be_simulated under the 2.93.0 lock, as designed.
  • mergeable is MERGEABLE.
  • The targeted SPI tests pass on policyengine-uk 2.93.0, 2.104.4, 2.104.5, 2.121.0 and 2.122.0, and in both version-skew directions.
  • All 14 mutations fail at least one test.
• Pending: nothing on this PR. It waits for the uk-data batch, d833: not in the 10/8 batch, and it merges after it on the uk-data release rules.
• Depends on: nothing open. Uprate rent for unknown-region households by the UK-wide index policyengine-uk#1985 is merged and first released in 2.104.5.
• Follow-ups owed:
  1. The relock to policyengine-uk 2.104.5 or later belongs to the batch, not this PR. At that relock the unknown-region test runs in full, and load_spi_dataset stops relabelling with no code change.
  2. After the 10/8 batch, run the rebuild command in the PR body. More than 60 GB must be free, and only one build may run at a time. It should confirm that the enhanced FRS is unchanged, and it rebuilds storage/spi_2022_23.h5.
  3. FYI, not this PR: the CI Test job failed once at 9fc0e29, in test_cgt_band_donors.py::test_built_total_gains. That test reads the enhanced_frs fixture, which this PR does not touch, and it passed on the three later heads.
  4. FYI for policyengine-uk: the model carries a single-year dataset's inputs only to 2030 (extend_single_year_dataset, end_year=2030). From 2031, any input variable that has a formula falls back to that formula. That applies to this PR's pays_scottish_income_tax and to any other dataset that sets such an input.