Weight UK BRMA assignment by census private-rented households

[MaxGhenis]

What changes

The UK frs_brma stage drew each benefit unit's Broad Rental Market Area in proportion to the row counts of the 2019-20 LHA list of rents (brma_rent_counts.json). That list's Scottish, Welsh and Northern Ireland lists are byte-identical copies of English BRMAs' lists, each matched to the English list whose 2019 30th percentile is nearest the nation's April 2019 rate, so those nations' draws ignored their rental markets. PolicyEngine/policyengine-uk-data#515 has the evidence. For example, Greater Glasgow had 2% of the Scottish entries; the genuine Scottish list gives it 18%, and Scotland's Census 2022 16% of the nation's private renters.

This PR replaces the resource with brma_private_rented_households.json: census private-rented households by region and BRMA, using the bedroom band that matches each LHA category.

• Category mapping: A and B use one bedroom, C two, D three, E four or more.
• Sources: England and Wales Census 2021, Scotland's Census 2022 and Northern Ireland Census 2021.
• Northern Ireland has no bedrooms question, so every category gets the same NI weights. They come from NISRA households by postcode district, mapped to NIHE's BRMAs and scaled by NI's private-rented share. The private-rented split by area would need ONS Postcode Directory NI records, whose LPS licence doesn't clearly allow publishing derived figures.
• The JSON keeps the existing cells[region][LHA_category][brma] shape, so frs_brma.py's sampler is unchanged.
• tools/build_uk_brma_household_table.py replaces build_uk_brma_count_table.py. It pins the households CSV by sha256 (fd40dae0…) and row count (936). Weight FRS BRMA draws by census private-rented households policyengine-uk-data#516 adds the same CSV, with sources, method, validation and a byte-for-byte rebuild pipeline.

The stage's source text and every resource reference (country package, source stages, spec, tests, graph parity fixture) are renamed to match.

Validation of the weights

The check is the within-region correlation of BRMA shares with DWP's Universal Credit households assessed under LHA, averaged over April 2019 to November 2020:

	England	Scotland	Wales
Census private renters (new)	0.94	0.82	0.95
List-of-rents counts (old)	0.82	0.21	0.23

Invariants

Tested in test_uk_brma_resource.py:

1. The resource is the generator's output for the pinned CSV. This regeneration test runs wherever the CSV is staged; I ran it locally and it passed.
2. There are 12 regions × 5 categories = 60 cells and 200 BRMAs. Every count is positive, and every cell totals at least 18,000 households.
3. The cells are pinned in CI by a digest of their canonical JSON and by each nation's census total: England 4,795,158, Wales 228,601, Scotland 323,001, Northern Ireland 132,449.
4. Categories A and B have identical cells (both use one-bedroom homes).
5. Northern Ireland's five category cells are identical and hold all 8 BRMAs.
6. Lothian and Greater Glasgow are Scotland's two largest BRMAs (Census 2022: 19.3% and 15.9%). This guards against the copied counts, which put Greater Glasgow last.
7. A benefit unit whose region × category cell is missing makes assign_brma_by_cell raise KeyError.

sample_categorical_from_counts and the identity-keyed draws are unchanged.

Not in this PR

• efrs_parity_reference.json compares against an incumbent build. It needs regenerating once the incumbent dataset is rebuilt with the same weights, which happens outside PR CI.
• The BRMA→local-authority bridge that cross_grain_private_rent_scotland_brma is waiting for could come from the same Scottish output-area crosswalk. That is a separate change.

Review

An independent Opus review (round 1) requested changes, and all of them are addressed:

• NI licensing: the NI weights now come only from NISRA and NIHE sources.
• Cells pinned in CI: a digest of the cells and each nation's total.
• Fail-closed test: now exercises the code.
• Generator: one default staged-CSV path, and the error message names the band actually used.

The UK spine graph is re-pinned for the new resource. Round 2 approved after rerunning the resource tests. A one-count mutation of the JSON fails the CI digest test.

Commands run

• pytest on engine_free/uk/test_uk_brma_resource.py, test_uk_take_up_contract.py, test_uk_frs_spine.py, engine_free/shared/test_country_spec.py, test_spec_engine_country_bundles.py and test_spec_engine_identity_contracts.py: all passed. The regeneration test ran with UK_BRMA_HOUSEHOLDS_CSV set. engine/uk/test_uk_release_input_coverage.py and microcosm-graph engine/uk/test_acceptance_h_parity.py pass after regenerating the coverage manifest and the pinned spine graph.
• ruff check and ruff format: clean.
• python3 tools/ci_test_plan.py verify: ok.

Update 2026-10-05: main merged, parity with uk-data#516, review

• Merged origin/main (fcbda41, no rebase), then formatting and a changelog fragment (766df94). Two files had conflicts: release_input_coverage_manifest.json and the uk_spine.json parity fixture. Both keep main's content; only the source-manifest hashes and the frs_brma contract hash change, and the review rebuilt the latter independently. All 15 of this PR's changes survive, and every file that changed only on main keeps main's contents and modes.
• Parity with policyengine-uk-data#516 at 29e1bd7a: exact.
  • The same 200 BRMAs, 12 regions and 240 region–BRMA pairs.
  • All 1,200 region/category/BRMA counts match, with no missing or extra keys.
  • The same category mapping: A and B on one bedroom, C on two, D on three, E on four or more.
  • The same NI treatment.
  • The same source SHA-256 (fd40dae0…).
• Review: GPT-6.1 Sol gave APPROVE WITH NITS at 766df94. Its one P3 nit, a stale docstring in tools/emit_uk_brma_distribution.py, is fixed in d23bd3e; that docstring is the only change.
  • Targeted tests: 152 passed and 1 skipped. Another 137 errored at setup, and every one of those errors was the review sandbox having no temporary directory; no test body failed. CI covers them.
• Merge timing (decision d830): this merges right after the uk-data batch release that carries QRF donor contains aggregate-scale PUF rows (single-record home_mortgage_interest up to $236B); support_clip donor_realized is vacuous against them #516, on its own gates.

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub: ready to land on its gates (d830)

Head: 62645d71de9bbcecc8d1c37178ea90b9858e96b2. It is a merge of main at 75167a688 on top of 965e6e5a8, which merged main at 4cb43524a into d23bd3e62. Both are merge commits, with no rebase and no force-push.

Merge resolution. Both merges conflicted in the same two generated files. Each file was regenerated with the repo's own tool, not hand-resolved:

• release_input_coverage_manifest.json came from tools/build_uk_release_input_coverage_manifest.py. Only the 17 source_manifest_sha256 values changed. All of them equal the SHA-256 of the merged uk/source_stages.json, which is 5c1c7040… at the head. The output was the same whichever conflict side was the placeholder, and --check passes.
• uk_spine/uk_spine.json came from tools/graph_uk_spine_fixture.py. It equals main's graph plus this branch's frs_brma stage_contract_sha256, which is the only node field the branch changes. The other 36 fixture files the generator writes matched the merged fixture byte for byte.

Parity with policyengine-uk-data 1.58.0 (#516).

• The released brma_private_rented_households.csv at tag 1.58.0 has SHA-256 fd40dae0…e1f. It is the same blob as at QRF donor contains aggregate-scale PUF rows (single-record home_mortgage_interest up to $236B); support_clip donor_realized is vacuous against them #516's head 29e1bd7a.
• tools/build_uk_brma_household_table.py run on that file reproduces the committed brma_private_rented_households.json byte for byte.
• test_committed_brma_resource_matches_regenerator passes with that CSV staged.
• The review also confirmed all 1,200 region × category × BRMA counts match, with zero differences. uk-data's load_brma_weights() at 1.58.0 yields the same weights.

Independent review. A GPT-6.1 Sol delta review (subfleet run --task review --tier hard, job 20261008-110407-mc1087-delta-62645d7) returned VERDICT: APPROVE at 62645d71d, with no findings. It checked both merges and the five auto-merged files changed on both sides. It also regenerated the artifacts in memory (37/37 matched) and checked parity with 1.58.0. One semantic difference was already known: Microcosm uses identity-keyed draws, uk-data uses a sequential generator. The conditional assignment distributions agree, but row-level draws can differ.

Tests.

• CI passes at 62645d71d, all 10 checks including engine-uk and integration-uk.
• Local targeted run at 62645d71d (CPython 3.13, UK extra, -n 2 under the suite lock): 357 passed, 2 skipped. The run covered the BRMA resource, country spec, FRS spine, take-up, the engine-free and engine release-input coverage and manifest tests, graph test_h2_uk_spine_parity (it ran and passed), and the BRMA stochastic suites. The skips were the staged-CSV test, which then passed with the 1.58.0 CSV staged, and a test that needs a Chronicle feed not present locally.
• Trial merge into current main d50220d1f (not pushed): clean, both generated files still current, and 127 passed in the overlapping test_uk_frs_spine.py, the BRMA resource tests, the manifest tests and uk_spine parity.

State. MERGEABLE, CLEAN, not a draft, and no CHANGES_REQUESTED review. Nothing is pending on this session's side, and it depends on no other PR. Per d830, uk-data 1.58.0 was published on 2026-10-07, so this can land on its gates.

Evidence: ~/reviews/uk-hub/jobs/mc1087-delta-1008/ (review.md, prompt.md, evidence/).

🤖 Generated with Claude Code

[MaxGhenis]

Landed by the UK hub on the gates, per Max's d830: it lands right after the uk-data batch release, and policyengine-uk-data 1.58.0 was published 10/7. Handed off by session local_8586070c.

• Head 62645d71de9bbcecc8d1c37178ea90b9858e96b2. Merged with a merge commit, pinned to this head.
• CI: all 10 checks ran and passed at this head, including engine-uk and integration-uk.
• Mergeable: MERGEABLE, not a draft.
• Review: an independent GPT-6.1 Sol delta review (20261008-110407-mc1087-delta-62645d7) APPROVES this head with no findings. It covers both main merges and the regenerated release_input_coverage_manifest.json and uk_spine.json.
• Parity: the BRMA private-rented household counts match policyengine-uk-data 1.58.0 (QRF donor contains aggregate-scale PUF rows (single-record home_mortgage_interest up to $236B); support_clip donor_realized is vacuous against them #516) exactly (1,200 counts; same source SHA-256).