Repository navigation
Local-area UC calibration targets are positionally misaligned: every constituency and local authority is assigned another area's UC count #468
Description
Summary
The local-area Universal Credit targets are joined to their areas by row position against orderings they do not have. The parsed Stat-Xplore extracts carry area names but no codes, and are returned in the spreadsheet's own row order; the loss builders then assign them positionally to code files in a different order. The result:
- Local authorities: 8 of 360 rows land on the right area. Birmingham is calibrated to West Oxfordshire's UC count (5,828 instead of ~175,600), Hackney to the Isles of Scilly's (70 instead of ~38,200), Glasgow to Moray's (6,728 instead of ~84,100).
- Constituencies: 0 of 650 rows land on the right area, and the misordered vector — whose data is already on 2024 boundaries — is then pushed through the 2010→2024 boundary-change matrix a second time. The final target surface has correlation 0.118 with the true per-constituency values; only 44 of 632 GB constituencies are within 5% of truth (median error 36%).
- The four
uc_hh_*_childrenconstituency columns are derived from the same totals and inherit the misalignment (verified: the splits sum back to the misaligned totals).
National totals are preserved (both parsers rescale to the national payment-distribution total), which is why nothing aggregate-level catches it.
Affected columns: uc_households at both grains + the 4 child-split columns — 5 of 21 constituency loss-matrix columns and 1 of 30 LA columns, in every release since the constituency UC targets landed (2025-10-21, 8fad6d5; LA 2025-11-07, c61798e; child splits 2026-02-19, bd90d28). Verified unfixed at current main (12a1e02, v1.56.16). No test references the alignment.
Mechanism
Local authorities. utils/uc_data.py::_parse_uc_la_households reads uc_la_households.xlsx row-by-row into {la_name, household_count} (GB rows in spreadsheet order, NI rows appended from the DfC ODS), rescales to the national total, and returns — no sort, no merge, no codes. targets/sources/local_uc.py::get_la_uc_targets returns uc_la_households.household_count with a docstring claiming it is "aligned to the same ordering as the LA age.csv"; the construction contains no step that could make that true. local_authorities/loss.py then does y["uc_households"] = get_la_uc_targets().values against local_authorities_2021.csv order. Positional name agreement: 8/360 (the name sets do not even fully coincide — 22 rows differ only by bilingual Welsh labels, but the orderings are unrelated).
Constituencies. _parse_uc_pc_households reads uc_pc_households.xlsx the same way. Its names are 2024 Westminster constituencies — all 650 match constituencies_2024.csv, and only 416 coincide with the 2010 vocabulary of age.csv, which defines the y-frame's row order. Positional agreement with the 2010 order: 0/650 (and 0/650 with constituencies_2024.csv order too — the spreadsheet is in Stat-Xplore's own row order, which matches nothing in the repo). The y frame, built on 2010 rows, is then transformed by boundary_changes/mapping_matrix.py — so PCON24-native data is boundary-mapped as if it were 2010 data, compounding the permutation.
Every other family in the same y frames is code-joined (age.csv carries codes in file order, the ONS/tenure/rent families merge on code/la_code); UC is the only column with no key.
Evidence
1. Same data, permuted — the name-join control. Joining the parsed UC tables to the code files by name (splitting bilingual Welsh labels) aligns 360/360 and 650/650. So joined, the values reproduce the published Stat-Xplore May-2025 statistics (independently extracted in PolicyEngine/chronicle) at the parser's uniform national rescale:
| grain | rows compared | ratio (uk-data ÷ published) | spread |
|---|---|---|---|
| local authority | 350 | 0.8985 | max deviation from uniform 0.46% (extract-revision noise) |
| constituency | 632 | 0.8965 | p5–p95: 0.8904–0.9018 |
2. Positional assignment — what the loss builder actually calibrates to:
| area | positional target | that row actually is | true value (by name) |
|---|---|---|---|
| Birmingham (E08000025) | 5,828 | West Oxfordshire | 175,630 |
| Hackney (E09000012) | 70 | Isles of Scilly | 38,243 |
| Glasgow City (S12000049) | 6,728 | Moray | 84,123 |
3. End-to-end constituency surface (post boundary-mapping, versus published values at the same national scale): correlation 0.118; 44/632 areas within 5%; median |relative error| 0.36; max 5.6×.
What we checked before raising this
- The parsers' deliberate national rescale (×~0.90) is not the discrepancy — it is uniform and survives the name join; the misassignment is orthogonal to it.
- The residual ≤0.46% per-area spread against the Chronicle extraction is publication-revision noise between download vintages, not structure.
origin/mainequals the pinned release (12a1e02); neitherutils/uc_data.pynortargets/sources/local_uc.pynor the loss builders changed since — the defect is live in the current release.- No file under
tests/referencesuc_pc_households/uc_la_households/ the UC target getters.
Possibly related
#452 (UC caseload ~40% below administrative counts in calibrated results). Not claimed as caused by this, but a per-area UC surface that is noise with respect to the household weights gives the solver no coherent gradient toward the UC totals it is simultaneously asked to hit — worth re-checking #452 after a fix.
Suggested fix
Both correct join keys already exist in storage: the constituency names are PCON24 and match constituencies_2024.csv 650/650; the LA names match local_authorities_2021.csv 360/360 after splitting the bilingual Welsh labels. Merge on those (or better, carry codes out of the parse), place the constituency UC column on 2024 rows after the boundary mapping rather than through it (its data is already PCON24), and add a regression test asserting a handful of known areas (e.g. Birmingham ≈ 176k) so a future re-extract cannot silently regress.
How this was found
Found while building the microcosm local target surface (microcosm#759): a compile-parity fixture extracted from this repo's loss builders at 12a1e028 was compared against the same Stat-Xplore publication ported independently as Chronicle facts; the UC family was the only one whose matched rows diverged by more than publication-vintage noise, and the divergence pattern (uniform ratio under a name join, garbage under the positional join) isolated the alignment.
Verification run 2026-08-26 against the policyengine-uk-data-12a1e028 checkout with its own parsers and storage files; comparison facts from the Chronicle UK bundle at 33ca98a (feed sha 4395a4e7…).
🤖 Generated with Claude Code
Activity
vahid-ahmadi commentedon Aug 26, 2026
I've gone through this against
mainrather than taking the write-up at face value, and the core mechanism reproduces on inspection. Confirming the parts I could verify directly from the code, and flagging where I stopped short.The positional join is real and is visible in three places.
targets/sources/local_uc.py::get_constituency_uc_targetsreturnsuc_pc_households.household_countwith a docstring stating it is "aligned to the same ordering as the constituency age.csv". Nothing inutils/uc_data.py::_parse_uc_pc_householdsestablishes that ordering: it appends rows in spreadsheet order (for idx in range(8, len(df_gb)), takingiloc[idx, 1]as the name andiloc[idx, 3]as the count), then appends NI rows, with no sort, no merge and no code column anywhere in the construction. The docstring asserts an invariant the code never creates.datasets/local_areas/constituencies/loss.py:100then doesy["uc_households"] = get_constituency_uc_targets().values, andlocal_authorities/loss.py:113does the same for the LA grain. Both are bare.valuesassignments against a frame whose row order comes fromage.csv/local_authorities_2021.csv.The contrast in the same files is the strongest evidence: the ONS, tenure and rent families a few lines below both of those assignments are built with
la_codes.merge(...).merge(...)and readons_merged[...].valuesoff the merged frame. UC is the only family that skips the key it needs, in a file that otherwise demonstrates the correct pattern throughout.The child splits inherit it, and there is a second inconsistency there.
get_constituency_uc_by_children_targetsreadsconstituencies_2024.csv["code"]andzip(totals, codes)positionally to pick each row's country prefix. So the country bucket applied to a row comes from the 2024 code file while the total comes from Stat-Xplore order — the two are aligned only by coincidence. Even setting the main defect aside, that pairing is unsound on its own terms.What I did not verify. I have not run the parsers, reproduced the 8/360 and 0/650 positional agreement counts, or checked the correlation-0.118 end-to-end figure — those rest on your verification run, not mine. What I can say is that the mechanism you describe is exactly what the code does, that no key exists anywhere in the UC path, and that the failure mode follows necessarily from that. The specific numbers are consistent with the mechanism; I just haven't re-derived them.
On the fix. Merging on the existing names is right, and I'd push slightly further on two points.
Carry codes out of the parse rather than merging on names downstream. Name joins are what got us here — the bilingual Welsh labels already require special handling, and a future re-extract with a renamed area would silently drop back to a partial join. If the parse emits a code column, the join key stops being a formatting question.
Second, the fix should make the wrong thing loud rather than merely making the right thing happen. A regression test pinning Birmingham ≈ 176k is worth having, but the more durable guard is asserting the join is total — that every row matched a code, and that the resulting frame has the expected row count — so a partial or reordered extract raises instead of silently calibrating to a permutation. The reason this survived ten months across three releases is that nothing anywhere asserted the alignment, and national totals were preserved by the rescale. Any fix that leaves those two properties in place leaves the same blind spot.
On #452. Agreed it's worth re-checking after a fix, and I'd go further than "possibly related" as a working hypothesis: a per-area surface uncorrelated with truth doesn't just fail to help the solver, it actively pulls weights toward a permuted target while the national UC total is simultaneously being asked for. That's a plausible mechanism for a systematic shortfall, not just noise. Worth measuring the caseload gap before and after as part of validating the fix — if #452 moves, that's strong confirmation; if it doesn't, we've learned the two are independent and #452 stays open on its own merits.
Good find, and the isolation method — uniform ratio under a name join, garbage under the positional one — is what makes this immediately actionable rather than a vague discrepancy report.
MaxGhenis commentedon Oct 14, 2026
Still unfixed on main at daad71c, so the 1.58.0 and 1.58.1 builds are affected. I re-checked it on 2026-10-14 by parsing the committed workbooks the same way
uc_data.pydoes:Two more places have the same problem:
calibration/matrix_builder.pyL233-L248 calls the same functions and writes the results by position. For constituencies, it then applies the 2010→2024 mapping.get_constituency_uc_by_children_targetstakes each row's country fromconstituencies_2024.csvby position, so the split also uses the wrong country's shares.The income targets (
spi_by_constituency.csv,spi_by_la.csv) do followage.csv's code order, so only UC is affected.The policyengine-model docs audit found this again independently. The UK hub has queued a fix build for this issue, for the next uk-data batch.
MaxGhenis commentedon Oct 15, 2026
Correction to my 2026-10-14 comment above. I wrote that "the income targets … follow
age.csv's code order, so only UC is affected". That is wrong for Scotland.The independent review of #589 (GPT-6.1 Sol, executed) found a second misalignment on main. After the 2010→2024 mapping, the constituency rows come out in sorted boundary-table code order, which includes five Scottish 2010 aliases, while the country mask and the code file use
constituencies_2024.csvorder. Only 593 of 650 positions agree. So each of the 57 Scottish constituency rows carries another Scottish seat's income, age and housing targets (mapping_matrix.py:8-19,constituencies/loss.py:140,matrix_builder.py:249).Over those 57 rows, main → #589 head moves:
Scotland's totals are unchanged.
#589 fixes this as a pure reorder: main's rows, relabelled by their true code, equal the head's rows exactly. But it has no test or changelog entry for it yet. A follow-up round on #589 adds both.