Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/1301-uk-dense-signoff-2026-10-19.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Sign the measured local release build's gate-only failures for the local line (microcosm#1301; the second dense build, attempt uk-local-candidate-f100-s42-20261018T161007Z-35fb4c89, after the first failed `uk_export_surface`, which microcosm#1310 fixed): 15 national target-fit rows, 31 local target-fit cells and 2 areas below the support floor, approved 2026-10-19; no entry that the reuse build would fail needed retiring. Every signed row stays bound in the solve, each entry suppresses only the local line's release fence, and a build in which a signed row passes fails its entry as stale. The sign-off draft's pin round-trip test now restores a committed pin's local-line scope, which it lost once every entry of a register was local-only.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The UK export-surface gate's allow-list carries the high-value homes support split's three household columns (`high_value_support_copies`, `high_value_support_copy_index` and `household_is_high_value_support_copy`, microcosm#1240), beside the CGT support split's precedent, in `gates.json` and `UK_ALLOWED_EXTRA_EXPORT_COLUMNS` alike. They are the stage's declared coverage-manifest outputs. Without them every UK build since microcosm#1270 fails `uk_export_surface` at the calibrated gates, and the first dense release build (microcosm#1301) did; that gate has no gate-only register, so the fix changes the solve's key and the next build re-solves. The full gate battery's policy, gates-manifest and spec digests and the release-cut part's digests are re-pinned; the spine, calibration-seam and dense digests do not move.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The reviewed incumbent-unresolvable register (`uk/incumbent_unresolvable_measures.json`) signs `household.bus_in_london_boardings` and `household.other_local_bus_boardings` (approved by juaristi22 on 2026-10-19, expiring 2027-01-19). The pinned enhanced FRS carries no bus travel, so it cannot materialize the NTS bus-travel stage's boardings (microcosm#930), and the 9 `dft_bus_journeys` rows bound since microcosm#1286 refused the national rule-1 evaluation (microcosm#578, microcosm#823) of the first national candidate built after them. Those rows are now pruned from both arms of the evaluation, as the other unresolvable families are; they stay bound in the national solve.
2 changes: 1 addition & 1 deletion changelog.d/1301-uk-local-solve-registers.changed.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The registers that change the UK solve are in place before the first local relea

**High-value homes.** The family's region rows control it on the local line, so:
- `tools/generate_uk_local_target_references.py` defers its constituency level whole (`high_value_homes_constituency_grain_deferred`, `defer_if_compiles`), after the English band-H precedent, with a roster guard of 575 England and Wales constituencies;
- a new signed small-cell rule, `hvcts_la_household_floor` in `uk/local_small_cell_rule.json` (prefix `microcosm.high_value_homes.`, floor 400 homes, about two household weights at 150,000 households, expires 2027-04-01), defers the local-authority cells under the floor and keeps 192, 101 and 57 cells over £1m, £1.5m and £2m, whose deferred homes still count in their region rows. The two region rows below the floor, `microcosm.high_value_homes.over_2m_by_region` for the North East (`E12000001`) and Wales (`W92000004`), are signed out of the region fan-out by two row exclusions in `uk/target_reference_signed_exclusions.json` (María's ruling of 17 October 2026): their published stock is about one average household weight each at the ruled 150,000 households (an expectation from the stock, not a count of frame records), so their fit would be noise, and a public gate-only sign-off would narrow an estimate expected to rest on fewer than 10 records. A row-level signed exclusion can now name a geography cell (`row: {"dimension": "geography_id", ...}`). The same areas' £1m and £1.5m rows, which count those homes, stay bound;
- a new signed small-cell rule, `hvcts_la_household_floor` in `uk/local_small_cell_rule.json` (prefix `microcosm.high_value_homes.`, floor 400 homes, about two household weights at 150,000 households, expires 2027-04-01), defers the local-authority cells under the floor and keeps 192, 101 and 57 cells over £1m, £1.5m and £2m, whose deferred homes still count in their region rows. The two region rows below the floor, `microcosm.high_value_homes.over_2m_by_region` for the North East (`E12000001`) and Wales (`W92000004`), are signed out of the region fan-out by two row exclusions in `uk/target_reference_signed_exclusions.json` (María's ruling of 17 October 2026): their published stock is about one average household weight each at the ruled 150,000 households (an expectation from the stock, not a count of frame records), so their fit would be noise, and a public gate-only sign-off would narrow an estimate expected to rest on fewer than 10 records. A row-level signed exclusion can now name a geography cell (`row: {"dimension": "geography_id", ...}`). The same areas' £1m and £1.5m rows, which count those homes, stay bound. Their £2m authority cells are all deferred, so the cross-grain reconciliation now lets a leg that holds only the small-cell rule's deferred cells, of targets licensed empty there, stand without a control, receipted under `deferred_legs_without_control`; a bound row on such a leg is still refused as an orphan;
- the `high_value_home_stock_derived_estimate` adjudication records where the family binds, and that the seat rank correlations, now out of sample, decide its renewal.

**Islands.** Orkney (`local_authority/S12000023`) and Shetland (`local_authority/S12000027`) are pre-signed gate-only exclusions of the area-support floor in `uk/local_area_support_exclusions.json`, scoped to the local line and expiring 2027-01-16. Their cells stay bound.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The `ons.savings_interest_income` measure exclusion is renewed to 2027-01-08 (microcosm#866; renewed by juaristi22 on 2026-10-16): the ONS D.41 interest row stays a measured diagnostic and is never fitted, as ruling R3 of 2026-09-22 set, because its concept differs from the model's cash interest. Without the renewal it lapsed on 2026-10-22, and no national calibration could run after that date.
93 changes: 85 additions & 8 deletions packages/microcosm-build/src/microcosm/build/cross_grain.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,13 +284,19 @@ def _reconcile_cross_grain_surface_with_control_receipts(
*,
licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
check_factor_bounds: bool = True,
frame_only_positions: frozenset[int] = frozenset(),
) -> tuple[
pd.DataFrame,
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
]:
"""Return a copied surface with every detected lower grain rescaled."""
"""Return a copied surface with every detected lower grain rescaled.

``frame_only_positions`` are rows that join the frame to take a share of
a control and bind nothing (a small-cell rule's deferred cells): a leg
holding only those, for targets licensed empty on it, needs no control.
"""

columns = _surface_columns(local_frame)
_validate_rule(rule)
Expand All @@ -313,6 +319,7 @@ def _reconcile_cross_grain_surface_with_control_receipts(
unparented_rows: dict[tuple[str, str], dict[int, str]] = {}
first_pair_id: dict[tuple[str, str], str] = {}
absent_middle_tier_legs: list[dict[str, Any]] = []
deferred_legs_without_control: list[dict[str, Any]] = []
pairs_per_key: dict[tuple[str, str], int] = {}
for group in materialized_groups:
key = (group.inconsistency_id.rsplit(":", 1)[0], group.lower_grain)
Expand Down Expand Up @@ -386,12 +393,24 @@ def _reconcile_cross_grain_surface_with_control_receipts(
if last_pair_for_key and group.lower_grain not in control_grains:
# No farther control can still claim these leaf rows: refuse now,
# ahead of the empty-leg licence check, as the standing rule does.
# A leg only licensed-empty targets hold carries deferred cells
# alone and needs no control (_wholly_deferred_legs).
open_rows = {
position: leg
for position, leg in pending.items()
if position not in already and leg not in assigned_controls
}
unparented = sorted(
{
leg
for position, leg in pending.items()
if position not in already and leg not in assigned_controls
}
set(open_rows.values())
- set(
_wholly_deferred_legs(
open_rows,
reconciled,
columns,
licensed_empty_legs,
frame_only_positions,
)
)
)
if unparented:
raise ValueError(
Expand Down Expand Up @@ -649,27 +668,78 @@ def _reconcile_cross_grain_surface_with_control_receipts(
# A control-tier row with no control above it is the top of its
# own leg, not an orphan.
continue
still = {
leg
open_rows = {
position: leg
for position, leg in pending.items()
if position not in claimed[(identity, lower_grain)]
}
deferred = _wholly_deferred_legs(
open_rows, reconciled, columns, licensed_empty_legs, frame_only_positions
)
still = set(open_rows.values()) - set(deferred)
if still:
raise ValueError(
f"cross-grain inconsistency "
f"{first_pair_id[(identity, lower_grain)]!r} has "
f"unparented lower-grain leg(s) {sorted(still)}."
)
deferred_legs_without_control.extend(
{
"inconsistency_id": first_pair_id[(identity, lower_grain)],
"lower_grain": lower_grain,
"leg": leg,
"lower_target_ids": list(target_ids),
}
for leg, target_ids in sorted(deferred.items())
)
return (
reconciled,
receipts,
empty_legs_licensed,
controls_without_lower_rows,
delegated_legs,
absent_middle_tier_legs,
deferred_legs_without_control,
)


def _wholly_deferred_legs(
rows: Mapping[int, str],
frame: pd.DataFrame,
columns: Mapping[str, str],
licensed_empty_legs: Mapping[str, frozenset[str]] | None,
frame_only_positions: frozenset[int],
) -> dict[str, tuple[str, ...]]:
"""The legs of ``rows`` that hold deferred cells alone.

A leg qualifies when every row on it is frame-only (it binds nothing and
joins the frame only to take its share of a control) and every target on
it is licensed empty there, so no cell of it binds on the leg. With no
control above them such rows have nothing to reconcile, and the leg is
no orphan (microcosm#1301: a region row signed out below the small-cell
floor leaves its wholly deferred authorities without one). A bound row,
or a row of a target the leg is not licensed for, keeps it an orphan.
"""

targets_by_leg: dict[str, set[str]] = {}
bound_legs: set[str] = set()
for position, leg in rows.items():
targets_by_leg.setdefault(leg, set()).add(
str(frame.iloc[position][columns["target_id"]])
)
if position not in frame_only_positions:
bound_legs.add(leg)
return {
leg: tuple(sorted(target_ids))
for leg, target_ids in targets_by_leg.items()
if leg not in bound_legs
and all(
leg in _licensed_legs_for_target(target_id, licensed_empty_legs)
for target_id in target_ids
)
}


def reconcile_cross_grain_surface(
local_frame: pd.DataFrame,
groups: Iterable[CrossGrainInconsistency],
Expand Down Expand Up @@ -699,12 +769,16 @@ def apply_cross_grain_reconciliation(
reviewed_unbound_higher_targets: Mapping[str, Mapping[str, object]] | None = None,
licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
check_factor_bounds: bool = True,
frame_only_positions: Iterable[int] = (),
) -> tuple[pd.DataFrame, dict[str, Any]]:
"""Detect, reconcile, and return the always-present pass receipt.

``check_factor_bounds=False`` records each bounded bridge's factor without
refusing it, for a pass that is not the caller's last (a surface whose
off-surface cells join a second pass is bounded on that one).
``frame_only_positions`` names the rows that bind nothing (deferred
cells); a leg holding only those needs no control
(``deferred_legs_without_control``).
"""

bound = tuple(str(target_id) for target_id in bound_higher_targets)
Expand All @@ -728,12 +802,14 @@ def apply_cross_grain_reconciliation(
controls_without_lower_rows,
delegated_legs,
absent_middle_tier_legs,
deferred_legs_without_control,
) = _reconcile_cross_grain_surface_with_control_receipts(
local_frame,
groups,
rule,
licensed_empty_legs=licensed_empty_legs,
check_factor_bounds=check_factor_bounds,
frame_only_positions=frozenset(int(p) for p in frame_only_positions),
)
receipt = {
"bound_higher_targets": list(bound),
Expand All @@ -745,6 +821,7 @@ def apply_cross_grain_reconciliation(
"controls_without_lower_rows": controls_without_lower_rows,
"delegated_legs": delegated_legs,
"absent_middle_tier_legs": absent_middle_tier_legs,
"deferred_legs_without_control": deferred_legs_without_control,
"absence": (
None
if groups
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,9 +285,9 @@
"reason": "Concept, not a weight question: the ONS UKEA HAXV row is households' (S.14) D.41 interest resources in the national accounts (FISIM-adjusted, including the unincorporated-business part of the sector; GBP 95.5bn at 2025) while the model's savings_interest_income is cash interest, so the row is unreachable by design (-82 percent on the v20 national candidate). From 2026-09-22 the fitting anchor for interest is the SPI 2023-24 Table 3.7 interest rows by total-income band, uprated to the calibration year (hmrc.spi.savings_interest_income.*, PolicyEngine/chronicle#280 lane); this row is measured on every evaluation as the D.41 diagnostic and never fitted. Retire only with a declared D.41-to-cash bridge.",
"tracking": "microcosm#866",
"approved_by": "juaristi22",
"adjudication": "PolicyEngine/microcosm#1006 (issue comment 5812206367, the register rulings María gave in session: ruling R3 of 2026-09-22, the SPI interest rows bound and the ONS row a measured diagnostic; income-anchor lane, PolicyEngine/chronicle#280); previously microcosm#762 A16 (2026-09-03).",
"approved_on": "2026-09-22",
"expires_on": "2026-10-22"
"adjudication": "PolicyEngine/microcosm#1006 (issue comment 5812206367, the register rulings María gave in session: ruling R3 of 2026-09-22, the SPI interest rows bound and the ONS row a measured diagnostic; income-anchor lane, PolicyEngine/chronicle#280); previously microcosm#762 A16 (2026-09-03); renewed on 2026-10-16 to 2027-01-08 by juaristi22 in session (folded into the sixth microcosm#1095 ports PR): the gap is one of concept, not of weights, so the row stays a measured diagnostic until a D.41-to-cash bridge is declared.",
"approved_on": "2026-10-16",
"expires_on": "2027-01-08"
},
{
"name": "obr.housing_benefit",
Expand Down
3 changes: 3 additions & 0 deletions packages/microcosm-build/src/microcosm/build/uk/gates.json
Original file line number Diff line number Diff line change
Expand Up @@ -953,6 +953,9 @@
"household.cash_isa",
"household.cgt_support_copies",
"household.cgt_support_copy_index",
"household.high_value_support_copies",
"household.high_value_support_copy_index",
"household.household_is_high_value_support_copy",
"household.household_is_cgt_residential_clone",
"household.cgt_residential_clone_index",
"household.household_clone_index",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,27 @@
"approved_on": "2026-09-22",
"expires_on": "2026-12-22"
},
"household.bus_in_london_boardings": {
"reason": "NTS bus-travel stage (microcosm#930): the enhanced FRS carries no bus travel, so it cannot materialize household boardings on London buses; the dft.bus_journeys London row (microcosm#1207, bound since microcosm#1286) binds through it.",
"approved_by": "juaristi22",
"adjudication": "microcosm#823 (rule-1 evaluation on the common surface; microcosm#578); the bus journeys family was bound after the incumbent pin (microcosm#1286).",
"approved_on": "2026-10-19",
"expires_on": "2027-01-19"
},
"household.ons_household_type": {
"reason": "ONS census household type (microcosm#887): the enhanced FRS carries no household-type classification; 10 ons_household_composition rows bind through it.",
"approved_by": "juaristi22",
"adjudication": "microcosm#823 (rule-1 evaluation on the common surface; microcosm#578)",
"approved_on": "2026-09-22",
"expires_on": "2026-12-22"
},
"household.other_local_bus_boardings": {
"reason": "NTS bus-travel stage (microcosm#930): the enhanced FRS carries no bus travel, so it cannot materialize household boardings on local buses outside London; the 8 dft.bus_journeys rows for the English regions outside London (microcosm#1207, bound since microcosm#1286) bind through it.",
"approved_by": "juaristi22",
"adjudication": "microcosm#823 (rule-1 evaluation on the common surface; microcosm#578); the bus journeys family was bound after the incumbent pin (microcosm#1286).",
"approved_on": "2026-10-19",
"expires_on": "2027-01-19"
},
"person.capital_gains_asset_type": {
"reason": "CGT asset type (microcosm#939, the E8 spine stage): the enhanced FRS carries gains amounts only; 2 hmrc_cgt rows bind through it.",
"approved_by": "juaristi22",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,26 @@
"release_lines": [
"local"
]
},
"constituency/S14000027": {
"reason": "Gate-only sign-off of the constituency support floor for S14000027 on the local line (microcosm#1301). On dense build-1 attempt uk-local-candidate-f100-s42-20261018T161007Z-35fb4c89 (full gate report sha256 46d42e6abf5fa8eaf08e3301a982703f113838c41cb7869340a3a41bafc6407f) the area missed the ESS-50 floor. It stays in the solve with its measured support and is flagged support_below_floor in the build's area-support table; consumers must treat its estimates as unsupported. Re-measured on every candidate: a build in which the area meets every floor fails the entry as stale.",
"approved_by": "juaristi22",
"adjudication": "Gate-only sign-off of the local release's measured build, drafted from its run (microcosm#1301, microcosm#1310).",
"approved_on": "2026-10-19",
"expires_on": "2027-01-19",
"release_lines": [
"local"
]
},
"local_authority/S12000013": {
"reason": "Gate-only sign-off of the local-authority support floor for Na h-Eileanan Siar (S12000013) on the local line (microcosm#1301). On dense build-1 attempt uk-local-candidate-f100-s42-20261018T161007Z-35fb4c89 (full gate report sha256 46d42e6abf5fa8eaf08e3301a982703f113838c41cb7869340a3a41bafc6407f) the area missed the ESS-50 floor. It stays in the solve with its measured support and is flagged support_below_floor in the build's area-support table; consumers must treat its estimates as unsupported. Re-measured on every candidate: a build in which the area meets every floor fails the entry as stale.",
"approved_by": "juaristi22",
"adjudication": "Gate-only sign-off of the local release's measured build, drafted from its run (microcosm#1301, microcosm#1310).",
"approved_on": "2026-10-19",
"expires_on": "2027-01-19",
"release_lines": [
"local"
]
}
},
"bound_despite_support_floor": {
Expand Down
Loading
Loading