UK: re-pin the Chronicle feed to chronicle#359 (paid-claims UC area cells, OBR table 4.9 geography, DESNZ gas years and Northern Ireland energy) #1300

Summary

PolicyEngine/chronicle#359 answers five Microcosm requests: PolicyEngine/chronicle#326, PolicyEngine/chronicle#339, PolicyEngine/chronicle#356, PolicyEngine/chronicle#357 and PolicyEngine/chronicle#358. It merged on 16 October as abf0921, which uk/chronicle_feed.json (now chronicle 735820c) moves to.

The merge breaks nothing in Microcosm by itself: every reader checks the pinned feed. But this re-pin is more than the usual regenerate-and-diff, because several things Microcosm binds today change identity in the new feed:

• UC area references: 3,508 stop matching one fact per month. The child-band references match none, and the households-by-area references match two.
• OBR table 4.9: 12 of the 14 references, and the obr.state_pension diagnostic, match no fact.
• DESNZ energy, in three places:
  • Two vendor selections, including Bind local-authority energy and regional bus journeys before the Budget (#1207) #1286's new local-authority one, raise on row counts.
  • The gas-connected share starts dividing a gas year by a calendar year, and nothing fails.

A target whose selector matches nothing can simply leave the active set rather than fail (see _dimension_values_match in ledger_targets.py). So the membership reports and the compiled target diff are where this shows.

This issue covers the re-pin and those fixes. The new facts' own uses stay on the issues that asked for them (listed at the end).

What changes at the re-pin

1. UC households by area split into paid and nil-award claims (PolicyEngine/chronicle#339)

• The area cells now come in two parts. Chronicle retires the three all-claims area packages. Instead, each area-month carries Payment Indicator "Yes" (an award paid in the month) and "No" (a nil award), January 2025 to May 2026, extracted 15 October 2026. Yes + No matches the retired cells to within 9 households per area-month.

• None of the selectors names a payment indicator. Six families read these cells, 3,508 references in uk/local_target_references.json:

  • dwp.uc.households_by_area, at 632 constituencies and 348 local authorities (980 references). Its contract target names no dimensions, so each reference would meet a Yes and a No fact for every month.
  • dwp.uc.households_by_area_children_{0,1,2,3plus}, at 632 constituencies each (2,528 references). Their contract targets in uk/uk_population_targets.json list dimensions: ["number_of_children"], which must equal a fact's dimension names (ledger_targets.py, _dimensions_match). They would match no fact, silently.

  Add payment_indicator to those dimensions lists, pin payment_indicator: Yes on all five targets as UC households by area: re-extract the constituency, local-authority and by-children series with Payment Indicator as a dimension (PolicyEngine/microcosm#1198) chronicle#339 planned, and regenerate.

• The bridge's factor moves to about 1. national_uc_caseload_vs_uc_households_by_area in uk/cross_grain_declarations.json will relate paid claims to paid claims. At constituency level its factor moves from 0.921 (the GB paid share, applied to every area alike) to about 1.001: GB paid claims average 6,197,311 over 2025, against 6,190,821 summed over constituencies. The gap is about the size of DWP's "Unknown" area, which carries no area fact. Rewrite the bridge's reason, and refuse a factor far from one, as the SPI bridges do.

• The child-band skew should go. The child bands then carry paid-claims shares. That should remove the skew UC households by area: re-extract the constituency, local-authority and by-children series with Payment Indicator as a dimension (PolicyEngine/microcosm#1198) chronicle#339 measured against the national paid rows: +2.8% (no children), -1.5% (one), -3.5% (two) and -5.1% (three or more).

• Legacy selector families retire. The May 2025 legacy families (uk.local_geography.uc_households.*) leave the feed. Nothing on main uses them.

2. OBR table 4.9's DWP lines carry DWP's geography (PolicyEngine/chronicle#357 item 4)

Chronicle now stamps each line with the geography DWP counts it over, from DWP's benefit expenditure and caseload tables, Spring 2026. Values don't change. All fourteen table 4.9 references in uk/target_references.json pin geography_id: K02000001, so twelve would match no fact:

• Regenerating alone won't fix them. tools/generate_uk_target_references.py sets each reference's geography by rule (_geography_id_for_target, with TARGET_ID_GEOGRAPHY_PINS for exceptions). These lines need entries there.
• The hand-written obr.state_pension diagnostic in uk/uk_population_targets.json also pins K02000001; State Pension is now stamped Great Britain.
• The scopes then agree. UK: port the unblocked uk-data fixes tracked in #1095 #1100 (1d1c10da3) already limits the modelled side to the households DWP counts. After the re-pin, the fact and the measure state the same scope. Consider checking that agreement in the compile rather than restating it.
• Northern Ireland's rows are new. NI's two social security rows are newly ported (N92000002, 2024-25 to 2030-31), for when NI benefits are bound.

3. DESNZ: GB gas moves to gas years, and Northern Ireland joins (PolicyEngine/chronicle#356)

• What changes in the feed:
  • GB gas period type. The GB subnational gas rows move from calendar_year to a new gas_year period type. Their values and year labels stay: DESNZ's 2024 gas year runs from mid-May 2024 to mid-May 2025, and DESNZ publishes no exact dates. Electricity stays calendar_year.
  • Northern Ireland's rows share GB's concepts. NI's electricity and gas rows share the GB concepts. NI electricity files three domestic_scope values (standard, Economy 7, all domestic) under the domestic concepts, where GB files all_domestic only. DESNZ says NI is not directly comparable with GB (methodology booklet, July 2026, sections 4.3 and 5.3), and each NI fact's notes say so.
• The region vendor selection raises. subnational_domestic_consumption_and_meters_regions (uk/ledger_fact_vendor_selections.json) selects the four domestic concepts at geography_level: [country, region] and expects 112 rows. NI adds 116 country-level rows at N92000002, so it would match 228 and ledger_fact_vendoring.py refuses. Pin the GB geographies or domestic_scope.
• Bind local-authority energy and regional bus journeys before the Budget (#1207) #1286's local-authority selection raises too. subnational_domestic_meters_local_authorities_2024 pins calendar_year 2024 and expects 697 rows, but its 347 GB gas rows become gas_year.
  • Simply dropping the period pin readmits NI's three electricity rows per district, and domestic_energy_targets.py:368 then raises "two electricity meter rows".
  • Select gas by gas_year, pin domestic_scope: all_domestic, and pin the GB areas.
• The energy stage would mix the two years without failing.
  • uk/desnz_domestic_energy_facts.json vendors the GB, England and Wales, country and region consumption and meter rows for 2023 and 2024.
  • energy_pricing.published_gas_connected_shares selects them by period value alone, since rows_matching filters period type only when asked.
  • After re-vendoring, the LCFS energy stage's gas-connected share would divide gas-year 2024 meters by calendar-2024 electricity meters, and so would the release-blocking gate that rechecks it.
• Microcosm owns the alignment. Decide the gas-year alignment rule and declare it on the stage; UK local energy, area classification and Welsh bus facts: DESNZ Northern Ireland and sub-local consumption, rural-urban lookups, BUS04 country rows (PolicyEngine/microcosm#1207) chronicle#356 left it to Microcosm. Pass period_type in each selection, so a change of type fails by name next time.
• Minor. _normalize_period_value in ledger_targets.py has no gas_year entry, so a gas_year value compares as a plain year. That's enough for value matching. Add it to the annual types if a target ever selects on the declared type.

4. Pin bookkeeping

• Procedure. Follow docs/uk-chronicle-feed-repin.md. The UK feed grows from 950,242 rows to about 1.5 million.

• Only UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age chronicle#359 moves the UK side. Between 735820c and UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age chronicle#359's base, Chronicle merged only an NZ package (Stats NZ territorial-authority resident population, 30 June 2025 (NZ, G8-d2) chronicle#355), so every UK change in the new feed is UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age chronicle#359's.

• R2-only workbooks. Rebuilding the feed needs DESNZ's two LSOA workbooks (50 to 56 MB), which Chronicle keeps in R2 only. Build with LEDGER_SOURCE_ARTIFACT_FETCH=1, or run scripts/fetch_r2_only_source_artifacts.py first; both fetch from gov.uk and check sha256.

• Schema hash, in three places. UK facts for chronicle#357, #356, #339, #326 and #358: stamp taxes, HBAI, Move to UC, OBR geography, UC by payment indicator, NI and LSOA energy, gas years, rural-urban lookups, UV404 by output area, residence type by age chronicle#359 adds gas_year to the v4 schema's period-type description, which moves the v4 digest from cc9efb90… to d953306d6f01fbc64db4372d0262d3012ee590269f8a25b276b18dca0626c6a0. The schema still types period.type as a free string. The old digest is pinned in:

  • uk/chronicle_feed.json, updated with this re-pin;
  • nz/chronicle_feed.json: every NZ bundle built after the merge carries the new digest, so the NZ feed's next re-pin moves it too;
  • tests/engine_free/uk/test_uk_chronicle_feed.py:30, which hard-codes it.

  Microcosm checks the digest against its own pins and doesn't read bundles through the policyengine_chronicle wheel, so nothing else breaks.

• Vendored resources first. All 25 vendored resources name 735820c as their source feed, so re-vendor them first (step 5 of the procedure). The two DESNZ selections above have to change before that step can pass.

• Fixtures. Regenerate the fixtures that carry the old rows. For example, tests/fixtures/uk_target_reference_feed_rows.jsonl holds 123 dwp.uc_households rows.

What the new facts unlock, tracked where they were asked

• UK: track the defects uk-data is fixing that microcosm still has #1095:
  • HMRC stamp tax Tables 6b, 8a and 8b: additional-dwelling and non-residential purchases, and households' SDLT net of non-natural persons. The EFO publishes no split by buyer, so HMRC's outturn share carries forward under Microcosm's own rule.
  • Move to UC claim rates by legacy-benefit combination, for would_claim_uc_at_legacy_closure. Claimed over claimed plus did not claim is 86.74% for all households; DWP's 87% divides by every notice (86.71%).
  • HBAI FYE 2025 with its 95% intervals, for the validation check (UK scorecard: populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks (2026) #731). The FYE 2021 and FYE 2022 facts carry DWP's Note 5 caution on pandemic interviewing.
• UK consumption-tax levels: open items after #1167 #1207 / Bind local-authority energy and regional bus journeys before the Budget (#1207) #1286:
  • DESNZ Northern Ireland electricity and gas by district, with the comparability caveats above;
  • domestic meters and kWh by LSOA and data zone, 2023 and 2024;
  • rural-urban lookups (raw-only);
  • BUS04 operating revenue for Scotland, Wales and GB.
• Scottish private-rent targets: PIPR is BRMA-grain, so council cells cannot bind by translation #1090 / Measure Scotland's BRMA × council overlap; keep BRMA rents out of council targets #1092: NRS UV404 tenure by Scottish output area: all occupied households and private rented.
• UK local surface, consumption and high-value homes: remaining items (consolidation) #1278: residence type by single year of age, with 90 and over split out:
  • ONS for England, Wales and the English regions;
  • NISRA for NI, LGD14 and PCON24;
  • NRS UV101a for Scotland, which goes only to 85 and over.

Done when

• uk/chronicle_feed.json pins abf0921 and the new schema digest, and every vendored resource and generated surface is regenerated from it.
• test_uk_chronicle_feed.py expects the new digest, and the NZ pin moves at its next re-pin, or now if that comes first.
• The six UC area families bind payment_indicator: Yes (the child bands with it in dimensions), and the bridge declares a factor near one.
• The twelve table 4.9 references and the obr.state_pension diagnostic resolve at their new geographies, with no value change.
• Both DESNZ vendor selections re-vendor at declared counts, with the period type and scope they mean.
• The energy stage declares its gas-year alignment, and its gate passes on the re-vendored rows.
• The membership reports show every UC area, table 4.9 and energy reference resolving, none dropped.
• The compiled target diff on both surfaces is recorded in the procedure's history and the changelog fragment. Values move only for the UC area targets (and the energy stage, if its alignment changes); the table 4.9 targets change geography, not value.

Related: PolicyEngine/chronicle#359; #1198 (the 0.921 bridge), #1100 (the DWP household scopes), #1286 (the local-authority energy binding), #1095, #1207, #1090, #1092, #1278, #1167.

🤖 Filed with Claude Code