Re-pin the UK Chronicle feed to abf0921 (#1300)

changelog.d/1300-uk-chronicle-repin-abf0921.changed.md

@@ -0,0 +1,9 @@
+The UK Chronicle feed moves to PolicyEngine/chronicle `abf0921` (PolicyEngine/chronicle#359, which answers chronicle#326, #339, #356, #357 and #358; microcosm#1300). Values move only for the UC area rows.
+
+- **UC households by area count paid claims.** Chronicle splits each area-month by Stat-Xplore's Payment Indicator and retires the all-claims cells. `dwp.uc.households_by_area` and its four child bands add `payment_indicator` to their dimensions and bind `Yes` (the three-plus band's summed operands name it too), so the area cells and the GB caseload that controls them both count paid claims. Against the all-claims cells, the GB constituency total falls 8.0% (households with no children 10.5%, one 6.6%, two 4.6%, three or more 3.0%), so the childless share falls from 53.6% to 52.1%. Each cell takes its own area's paid share where it took the GB one: after the bridge, the middle 90% of constituencies move by between -1.7% and +1.3%. The bridge's factor moves from 0.921 to about 1.001, and a new `CrossGrainBridge.max_factor_shift` (2 per cent on this bridge) refuses a factor far from one. Each lower grain is bounded on its own input total, whichever group rescales it first, and a surface with off-surface cells is bounded on the pass that carries them.
+- **OBR table 4.9 lines carry DWP's geography.** Twelve DWP lines are stamped Great Britain or England and Wales (`TARGET_ID_GEOGRAPHY_PINS` in `tools/generate_uk_target_references.py`), and the `obr.state_pension` diagnostic, which `dwp.state_pension.amount` requires, reads its Great Britain fact; no value changes.
+- **DESNZ gas runs on gas years.** Northern Ireland's rows now share Great Britain's concepts, so both sub-national vendor selections pin Great Britain (`geography_vintage: publication_2025_nspl`); the local-authority one accepts `gas_year` and `calendar_year` rows at `domestic_scope: all_domestic`, and its expected row count (697) and the per-fuel reads pair gas with gas years and electricity with calendar years. The energy stage and the local energy appliers name each fuel's period type (`energy_pricing.SUBNATIONAL_PERIOD_TYPES`): gas year Y pairs with calendar year Y, as the rows were read before Chronicle stamped the gas period, so the gas-connected shares and the Great Britain levels do not move.
+- **The Northern Ireland energy interim stays for now.** Great Britain's share of UK households still stands in for its share of the Energy Trends level: the Northern Ireland facts are in the feed on a basis DESNZ calls not directly comparable, and a change stacked on this re-pin derives Great Britain's part from them (María's ruling of 2026-10-16). The adjudication, its census fence and the code say so.
+- **Schema digest.** The v4 consumer-fact schema digest moves to `d953306d…` (the `gas_year` period type); the NZ pin moves at its own next re-pin.
+- **The local compile-parity receipt moves with the pin.** `uk/ledger_compile_parity_local_incumbent_2025_signed_differences.json` is regenerated on `abf0921`: the release-blocking preflight gate `uk_ledger_compile_parity_local_incumbent_2025` compares its signed values with the live compile, and only the 3,508 UC cells' values move.
+- **Follow-up.** The twelve restamped OBR lines move the national register version with no value moved, so the uk-candidate-eval scoring register is re-frozen before the next national solve.

docs/uk-chronicle-feed-repin.md

@@ -19,7 +19,13 @@ regeneration tests accept a `CHRONICLE_UK_FACTS` override and skip only when
 neither is present.
 
 Verify both SHA-256 digests and the manifest's `facts_sha256`, row count, and
-schema version, then update `uk/chronicle_feed.json` and regenerate. Vendor
+schema version, then update `uk/chronicle_feed.json` and the two tests that
+restate it: `tests/engine_free/uk/test_uk_chronicle_feed.py` (the pin and the
+consumer-fact schema digest) and `tests/engine_free/uk/test_uk_full_targets.py`
+(`test_current_national_and_local_pins_share_one_reviewed_identity`). A schema
+digest moves for every country whose feed is built after the Chronicle change,
+so the NZ pin (`nz/chronicle_feed.json`) moves at its own next re-pin. Then
+regenerate. Vendor
 the per-concern fact resources (step 5) before the national generator: the
 generator refuses to compile against a vendored resource whose feed identity
 differs from the pin, so on a fresh pin step 5 has to run first. The rest go in
@@ -43,9 +49,24 @@ this order:
    `tools/build_uk_release_input_coverage_manifest.py`: it hashes the CGT
    conditioning resource, whose vendored header records the feed identity.
 
+Both compile parity receipts are read at build time: the release-blocking
+preflight gates `uk_ledger_compile_parity_*` compare each signed value with the
+live compile, so a receipt left on the old pin blocks every full or dense build
+and the release-cut certifier. Regenerate them on the head that will be built.
+
+The high-value home stock (`uk/high_value_home_stock.json`) is built from feed
+facts too: the Census 2021 TS054 tenure record sets, the OBR non-owner-occupied
+share and the ONS HPSSA seat cells. Check that those facts are unchanged on the
+new pin, and rebuild the stock with `tools/build_uk_high_value_home_stock.py`
+if they are not; its provenance names the feed it was built from.
+
 Verify the complete compiled target diff on both surfaces, including targets
 outside the intended policy area, and record the value moves in the changelog
-fragment.
+fragment. A national reference whose metadata moves (a fact key or a
+geography stamp) moves the national register version even when no value
+does, so the frozen scoring register of the candidate evaluation
+(uk-candidate-eval) must be re-frozen before the next national solve; record
+that in the history entry.
 
 A re-pin also moves the charter-H2 parity fixture: every stage that declares a
 vendored resource as an artifact binds that resource's digest into its
@@ -171,3 +192,42 @@ Paid yearly files, the UK HPI full file and the NSPL) and are read by
 `tools/build_uk_high_value_home_stock.py` against the registered sha256 pins (950,242 rows, up
 23,003). No existing package moved, so no compiled target changed; one vendor selection
 (`obr_hvcts_costing_facts.json`) pins the OBR note for the benchmarks gate.
+
+The `abf0921` re-pin (microcosm#1300, PolicyEngine/chronicle#359 for chronicle#326, #339, #356, #357
+and #358, on top of `735820c`) brought Universal Credit households by area split by Stat-Xplore's
+Payment Indicator, HMRC stamp tax statistics, DWP's Move to UC claim rates, HBAI with its vintages,
+the DWP geography of OBR table 4.9's lines, DESNZ Northern Ireland and LSOA domestic energy,
+rural-urban lookups, the BUS04 country rows, Scotland's UV404 tenure by output area and census
+residence type by age (1,499,301 rows, up 549,059). The v4 schema digest moves to `d953306d…` for
+the new `gas_year` period type. Three selector families moved with the data, and one interim stays:
+
+- the UC area families bind `payment_indicator: Yes` (`dwp.uc.households_by_area` at both grains
+  and the four child bands, whose summed operands name it too), because Chronicle retired the
+  all-claims cells. The cells now count paid claims, as the GB caseload that controls them does, so
+  the bridge's factor moves from 0.921 to about 1.001, and the bridge refuses a factor more than 2%
+  from one (`max_factor_shift`), each grain read on its own input total and on the pass that
+  carries the off-surface cells;
+- twelve OBR table 4.9 lines are stamped Great Britain or England and Wales
+  (`TARGET_ID_GEOGRAPHY_PINS`), and the `obr.state_pension` diagnostic reads its Great Britain
+  fact; no value moved;
+- the DESNZ vendor selections pin Great Britain (Northern Ireland's rows share its concepts), the
+  local-authority one reads gas by `gas_year`, and every sub-national lookup names its period type
+  (`energy_pricing.SUBNATIONAL_PERIOD_TYPES`, gas year Y with calendar year Y), so the 56 regional
+  and 347 local-authority gas rows come back under new keys with the same values;
+- the local energy rows keep Great Britain's share of UK households as the interim for its share of
+  the Energy Trends level. The Northern Ireland facts are now in the feed, on a basis DESNZ calls
+  not directly comparable, and María ruled on 2026-10-16 that the interim stands until a change
+  stacked on this re-pin derives Great Britain's part from them.
+
+Values move only for the UC area rows: each cell is its area's paid claims, where it was its
+all-claims count scaled by the GB paid share, so the cells keep the GB caseload's total and each
+takes its own area's paid share, as each child band takes its own. The membership reports show it:
+every national reference resolves the fact it resolved before except the twelve restamped OBR
+lines, and every local one except the 3,508 UC area cells and the 346 local-authority gas
+consumption cells, which Chronicle restamped as gas years with the same values. The vendored
+resources carry the same values, and the charter-H2 fixture moves twelve stage contract hashes. The
+local compile-parity receipt is regenerated on this pin, and only the 3,508 UC cells' signed values
+move; the full surface was not recompiled for this entry. The twelve restamped OBR lines move the
+national register version with no value moved, so the uk-candidate-eval scoring register is
+re-frozen before the next national solve. The high-value home stock still names `735820c`: its
+TS054 tenure, OBR share and HPSSA facts are the same on `abf0921`, so it was not rebuilt.

docs/uk-local-cross-grain-reconciliation-802.md

@@ -132,8 +132,10 @@ country-spec fingerprint covers:
   the lower side's. Each nation's household estimate controls its census
   household cells, each region's and nation's higher rate taxpayer row
   controls its area cells, and the GB UC caseload controls UC households by
-  area. The UC sides are not yet one concept (paid claims against every
-  household on UC), so that bridge applies the GB paid share to every area.
+  area. Both UC sides count paid claims since the abf0921 re-pin
+  (microcosm#1300), so that bridge's factor sits near one, and it refuses a
+  factor more than 2% from one at either grain, read on the grain's own
+  input total.
   Eight age bridges join the region-tier ONS age bands (inclusive integer
   ages) to the local bands (half-open). The 80-89 band has no local cell.
 - **partitions**: member targets that divide a parent at one geography,

packages/microcosm-build/src/microcosm/build/cross_grain.py

@@ -42,13 +42,19 @@ class CrossGrainBridge:
     ``per_geography`` declares instead that each higher target is the control
     of the geography it sits at (each nation's household estimate over its
     own legs, microcosm#1123): a geography then carries some of them only.
+    ``max_factor_shift``, when set, refuses a leg whose factor on the input
+    totals (the control over its lower rows as given, before any group
+    rescales them) is further than that from one: the two sides would no
+    longer measure one quantity. Each lower grain is bounded on its own input
+    total, so a grain another group rescales first is still bounded.
     """
 
     bridge_id: str
     concept: str
     higher_target_ids: tuple[str, ...]
     lower_side: str
     per_geography: bool = False
+    max_factor_shift: float | None = None
 
 
 @dataclass(frozen=True)
@@ -272,6 +278,7 @@ def _reconcile_cross_grain_surface_with_control_receipts(
     rule: CrossGrainRule,
     *,
     licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
+    check_factor_bounds: bool = True,
 ) -> tuple[
     pd.DataFrame,
     list[dict[str, Any]],
@@ -293,6 +300,7 @@ def _reconcile_cross_grain_surface_with_control_receipts(
     controls_without_lower_rows: list[dict[str, Any]] = []
     delegated_legs: list[dict[str, Any]] = []
     control_grains = _control_grains(rule)
+    bridges_by_id = {bridge.bridge_id: bridge for bridge in rule.bridges}
     # Rows already rescaled by a nearer control, per group identity and lower
     # grain, so a farther control never rescales them again; and the rows no
     # control has covered yet, checked once every pair has had its turn.
@@ -544,6 +552,32 @@ def _reconcile_cross_grain_surface_with_control_receipts(
                     f"cross-grain inconsistency {group.inconsistency_id!r} "
                     "cannot reconcile opposite-signed lower and control targets."
                 )
+            bridge = bridges_by_id.get(group.bridge_id or "")
+            bound = None if bridge is None else bridge.max_factor_shift
+            input_factor = None
+            if bound is not None:
+                # The bound reads the lower rows as given, not as an earlier
+                # group left them: an exact-signature group that rescales this
+                # grain onto a nearer tier first would leave a factor of one
+                # here whatever the input (microcosm#1302 review).
+                input_sum = float(
+                    local_frame.iloc[positions][columns["value"]]
+                    .to_numpy(dtype=np.float64)
+                    .sum()
+                )
+                if input_sum != 0.0:
+                    input_factor = parent_value / input_sum
+                else:
+                    input_factor = 1.0 if parent_value == 0.0 else float("inf")
+                if check_factor_bounds and not abs(input_factor - 1.0) <= bound:
+                    raise ValueError(
+                        f"cross-grain bridge {bridge.bridge_id!r} would scale the "
+                        f"{group.lower_grain} rows of leg "
+                        f"{'+'.join(control['covered_legs'])!r} by {input_factor!r} "
+                        f"on their input totals, further than its declared "
+                        f"max_factor_shift {bound} from one: the two sides no "
+                        "longer measure one quantity."
+                    )
             reconciled.iloc[positions, reconciled.columns.get_loc(columns["value"])] = (
                 raw_values * factor
             )
@@ -581,6 +615,15 @@ def _reconcile_cross_grain_surface_with_control_receipts(
                         0.0 if raw_sum == 0.0 else (new_total - raw_sum) / raw_sum
                     ),
                     "declared_factor": factor,
+                    **(
+                        {}
+                        if bound is None
+                        else {
+                            "max_factor_shift": bound,
+                            "factor_on_input_totals": input_factor,
+                            "factor_bound_checked": check_factor_bounds,
+                        }
+                    ),
                     "reason": (
                         "standing cross-grain rule: "
                         f"{group.winning_grain} controls {group.lower_grain}"
@@ -628,6 +671,7 @@ def reconcile_cross_grain_surface(
     rule: CrossGrainRule,
     *,
     licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
+    check_factor_bounds: bool = True,
 ) -> tuple[pd.DataFrame, list[dict[str, Any]]]:
     """Return a copied surface with every detected lower grain rescaled."""
 
@@ -636,6 +680,7 @@ def reconcile_cross_grain_surface(
         groups,
         rule,
         licensed_empty_legs=licensed_empty_legs,
+        check_factor_bounds=check_factor_bounds,
     )
     return reconciled, receipts
 
@@ -648,8 +693,14 @@ def apply_cross_grain_reconciliation(
     *,
     reviewed_unbound_higher_targets: Mapping[str, Mapping[str, object]] | None = None,
     licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
+    check_factor_bounds: bool = True,
 ) -> tuple[pd.DataFrame, dict[str, Any]]:
-    """Detect, reconcile, and return the always-present pass receipt."""
+    """Detect, reconcile, and return the always-present pass receipt.
+
+    ``check_factor_bounds=False`` records each bounded bridge's factor without
+    refusing it, for a pass that is not the caller's last (a surface whose
+    off-surface cells join a second pass is bounded on that one).
+    """
 
     bound = tuple(str(target_id) for target_id in bound_higher_targets)
     groups = detect_cross_grain_inconsistencies(
@@ -677,9 +728,11 @@ def apply_cross_grain_reconciliation(
         groups,
         rule,
         licensed_empty_legs=licensed_empty_legs,
+        check_factor_bounds=check_factor_bounds,
     )
     receipt = {
         "bound_higher_targets": list(bound),
+        "factor_bounds_checked": check_factor_bounds,
         "inconsistencies_in_force": [group.inconsistency_id for group in groups],
         "groups": group_receipts,
         "unbound_bridges": unbound_bridges,
@@ -971,6 +1024,17 @@ def _validate_rule(rule: CrossGrainRule) -> None:
                 f"cross-grain bridge {bridge.bridge_id!r} uses forbidden "
                 f"external side(s) {external_sides}; all sides must be contract targets."
             )
+        bound = bridge.max_factor_shift
+        if bound is not None and (
+            isinstance(bound, bool)
+            or not isinstance(bound, int | float)
+            or not np.isfinite(bound)
+            or bound <= 0.0
+        ):
+            raise ValueError(
+                f"cross-grain bridge {bridge.bridge_id!r} declares max_factor_shift "
+                f"{bound!r}; a bound must be a finite positive number."
+            )
     for geography_id, legs in rule.parent_geography_legs.items():
         if not str(geography_id) or not legs or any(not str(leg) for leg in legs):
             raise ValueError(

packages/microcosm-build/src/microcosm/build/uk/ashe_employer_pension_contribution_rates.json

@@ -15,12 +15,12 @@
   },
   "source_fact_feed": {
     "source_repo": "PolicyEngine/chronicle",
-    "source_commit": "735820cf76684aae1dfee4cffb48224d59bb7aab",
+    "source_commit": "abf0921b896d29ee0745ed2aefde8d6337ed7164",
     "build": "build-bundle --suite uk -> build-consumer-artifact",
     "artifact_schema_version": "policyengine_ledger.consumer_artifact.v2",
-    "fact_row_count": 950242,
-    "facts_sha256": "3033e437ed61d7d5e970f4fef34a2a5523fe068770a7798e5768af656f2a48a4",
-    "manifest_sha256": "2777dc33969a3aff449896aad3dcb0b2492c18896d6ab656918c04cd2e18551c"
+    "fact_row_count": 1499301,
+    "facts_sha256": "2b64fecfe746164203dac1726aae6ffdb3e5d8e478bd3d64d118ac76cb00780b",
+    "manifest_sha256": "b90fb1b8f1a910458fac41425360209d101796e6de0b134f952ad93323ca6e21"
   },
   "selections": [
     {

packages/microcosm-build/src/microcosm/build/uk/chronicle_feed.json

@@ -3,14 +3,14 @@
   "country": "uk",
   "scope": "uk_calibration",
   "source_repo": "PolicyEngine/chronicle",
-  "source_commit": "735820cf76684aae1dfee4cffb48224d59bb7aab",
+  "source_commit": "abf0921b896d29ee0745ed2aefde8d6337ed7164",
   "build": "build-bundle --suite uk -> build-consumer-artifact",
   "artifact_schema_version": "policyengine_ledger.consumer_artifact.v2",
   "consumer_fact_schema_versions": [
     "chronicle.consumer_fact.v4"
   ],
-  "consumer_fact_schema_sha256": "cc9efb90e9c73913bc76a89e9df435a9ca6ff49942e63dc8d2bf183c65a70224",
-  "fact_row_count": 950242,
-  "facts_sha256": "3033e437ed61d7d5e970f4fef34a2a5523fe068770a7798e5768af656f2a48a4",
-  "manifest_sha256": "2777dc33969a3aff449896aad3dcb0b2492c18896d6ab656918c04cd2e18551c"
+  "consumer_fact_schema_sha256": "d953306d6f01fbc64db4372d0262d3012ee590269f8a25b276b18dca0626c6a0",
+  "fact_row_count": 1499301,
+  "facts_sha256": "2b64fecfe746164203dac1726aae6ffdb3e5d8e478bd3d64d118ac76cb00780b",
+  "manifest_sha256": "b90fb1b8f1a910458fac41425360209d101796e6de0b134f952ad93323ca6e21"
 }