Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The UK export-surface gate's allow-list carries the high-value homes support split's three household columns (`high_value_support_copies`, `high_value_support_copy_index` and `household_is_high_value_support_copy`, microcosm#1240), beside the CGT support split's precedent, in `gates.json` and `UK_ALLOWED_EXTRA_EXPORT_COLUMNS` alike. They are the stage's declared coverage-manifest outputs. Without them every UK build since microcosm#1270 fails `uk_export_surface` at the calibrated gates, and the first dense release build (microcosm#1301) did; that gate has no gate-only register, so the fix changes the solve's key and the next build re-solves. The full gate battery's policy, gates-manifest and spec digests and the release-cut part's digests are re-pinned; the spine, calibration-seam and dense digests do not move.
2 changes: 1 addition & 1 deletion changelog.d/1301-uk-local-solve-registers.changed.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The registers that change the UK solve are in place before the first local relea

**High-value homes.** The family's region rows control it on the local line, so:
- `tools/generate_uk_local_target_references.py` defers its constituency level whole (`high_value_homes_constituency_grain_deferred`, `defer_if_compiles`), after the English band-H precedent, with a roster guard of 575 England and Wales constituencies;
- a new signed small-cell rule, `hvcts_la_household_floor` in `uk/local_small_cell_rule.json` (prefix `microcosm.high_value_homes.`, floor 400 homes, about two household weights at 150,000 households, expires 2027-04-01), defers the local-authority cells under the floor and keeps 192, 101 and 57 cells over £1m, £1.5m and £2m, whose deferred homes still count in their region rows. The two region rows below the floor, `microcosm.high_value_homes.over_2m_by_region` for the North East (`E12000001`) and Wales (`W92000004`), are signed out of the region fan-out by two row exclusions in `uk/target_reference_signed_exclusions.json` (María's ruling of 17 October 2026): their published stock is about one average household weight each at the ruled 150,000 households (an expectation from the stock, not a count of frame records), so their fit would be noise, and a public gate-only sign-off would narrow an estimate expected to rest on fewer than 10 records. A row-level signed exclusion can now name a geography cell (`row: {"dimension": "geography_id", ...}`). The same areas' £1m and £1.5m rows, which count those homes, stay bound;
- a new signed small-cell rule, `hvcts_la_household_floor` in `uk/local_small_cell_rule.json` (prefix `microcosm.high_value_homes.`, floor 400 homes, about two household weights at 150,000 households, expires 2027-04-01), defers the local-authority cells under the floor and keeps 192, 101 and 57 cells over £1m, £1.5m and £2m, whose deferred homes still count in their region rows. The two region rows below the floor, `microcosm.high_value_homes.over_2m_by_region` for the North East (`E12000001`) and Wales (`W92000004`), are signed out of the region fan-out by two row exclusions in `uk/target_reference_signed_exclusions.json` (María's ruling of 17 October 2026): their published stock is about one average household weight each at the ruled 150,000 households (an expectation from the stock, not a count of frame records), so their fit would be noise, and a public gate-only sign-off would narrow an estimate expected to rest on fewer than 10 records. A row-level signed exclusion can now name a geography cell (`row: {"dimension": "geography_id", ...}`). The same areas' £1m and £1.5m rows, which count those homes, stay bound. Their £2m authority cells are all deferred, so the cross-grain reconciliation now lets a leg that holds only the small-cell rule's deferred cells, of targets licensed empty there, stand without a control, receipted under `deferred_legs_without_control`; a bound row on such a leg is still refused as an orphan;
- the `high_value_home_stock_derived_estimate` adjudication records where the family binds, and that the seat rank correlations, now out of sample, decide its renewal.

**Islands.** Orkney (`local_authority/S12000023`) and Shetland (`local_authority/S12000027`) are pre-signed gate-only exclusions of the area-support floor in `uk/local_area_support_exclusions.json`, scoped to the local line and expiring 2027-01-16. Their cells stay bound.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The `ons.savings_interest_income` measure exclusion is renewed to 2027-01-08 (microcosm#866; renewed by juaristi22 on 2026-10-16): the ONS D.41 interest row stays a measured diagnostic and is never fitted, as ruling R3 of 2026-09-22 set, because its concept differs from the model's cash interest. Without the renewal it lapsed on 2026-10-22, and no national calibration could run after that date.
93 changes: 85 additions & 8 deletions packages/microcosm-build/src/microcosm/build/cross_grain.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,13 +284,19 @@ def _reconcile_cross_grain_surface_with_control_receipts(
*,
licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
check_factor_bounds: bool = True,
frame_only_positions: frozenset[int] = frozenset(),
) -> tuple[
pd.DataFrame,
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
]:
"""Return a copied surface with every detected lower grain rescaled."""
"""Return a copied surface with every detected lower grain rescaled.

``frame_only_positions`` are rows that join the frame to take a share of
a control and bind nothing (a small-cell rule's deferred cells): a leg
holding only those, for targets licensed empty on it, needs no control.
"""

columns = _surface_columns(local_frame)
_validate_rule(rule)
Expand All @@ -313,6 +319,7 @@ def _reconcile_cross_grain_surface_with_control_receipts(
unparented_rows: dict[tuple[str, str], dict[int, str]] = {}
first_pair_id: dict[tuple[str, str], str] = {}
absent_middle_tier_legs: list[dict[str, Any]] = []
deferred_legs_without_control: list[dict[str, Any]] = []
pairs_per_key: dict[tuple[str, str], int] = {}
for group in materialized_groups:
key = (group.inconsistency_id.rsplit(":", 1)[0], group.lower_grain)
Expand Down Expand Up @@ -386,12 +393,24 @@ def _reconcile_cross_grain_surface_with_control_receipts(
if last_pair_for_key and group.lower_grain not in control_grains:
# No farther control can still claim these leaf rows: refuse now,
# ahead of the empty-leg licence check, as the standing rule does.
# A leg only licensed-empty targets hold carries deferred cells
# alone and needs no control (_wholly_deferred_legs).
open_rows = {
position: leg
for position, leg in pending.items()
if position not in already and leg not in assigned_controls
}
unparented = sorted(
{
leg
for position, leg in pending.items()
if position not in already and leg not in assigned_controls
}
set(open_rows.values())
- set(
_wholly_deferred_legs(
open_rows,
reconciled,
columns,
licensed_empty_legs,
frame_only_positions,
)
)
)
if unparented:
raise ValueError(
Expand Down Expand Up @@ -649,27 +668,78 @@ def _reconcile_cross_grain_surface_with_control_receipts(
# A control-tier row with no control above it is the top of its
# own leg, not an orphan.
continue
still = {
leg
open_rows = {
position: leg
for position, leg in pending.items()
if position not in claimed[(identity, lower_grain)]
}
deferred = _wholly_deferred_legs(
open_rows, reconciled, columns, licensed_empty_legs, frame_only_positions
)
still = set(open_rows.values()) - set(deferred)
if still:
raise ValueError(
f"cross-grain inconsistency "
f"{first_pair_id[(identity, lower_grain)]!r} has "
f"unparented lower-grain leg(s) {sorted(still)}."
)
deferred_legs_without_control.extend(
{
"inconsistency_id": first_pair_id[(identity, lower_grain)],
"lower_grain": lower_grain,
"leg": leg,
"lower_target_ids": list(target_ids),
}
for leg, target_ids in sorted(deferred.items())
)
return (
reconciled,
receipts,
empty_legs_licensed,
controls_without_lower_rows,
delegated_legs,
absent_middle_tier_legs,
deferred_legs_without_control,
)


def _wholly_deferred_legs(
rows: Mapping[int, str],
frame: pd.DataFrame,
columns: Mapping[str, str],
licensed_empty_legs: Mapping[str, frozenset[str]] | None,
frame_only_positions: frozenset[int],
) -> dict[str, tuple[str, ...]]:
"""The legs of ``rows`` that hold deferred cells alone.

A leg qualifies when every row on it is frame-only (it binds nothing and
joins the frame only to take its share of a control) and every target on
it is licensed empty there, so no cell of it binds on the leg. With no
control above them such rows have nothing to reconcile, and the leg is
no orphan (microcosm#1301: a region row signed out below the small-cell
floor leaves its wholly deferred authorities without one). A bound row,
or a row of a target the leg is not licensed for, keeps it an orphan.
"""

targets_by_leg: dict[str, set[str]] = {}
bound_legs: set[str] = set()
for position, leg in rows.items():
targets_by_leg.setdefault(leg, set()).add(
str(frame.iloc[position][columns["target_id"]])
)
if position not in frame_only_positions:
bound_legs.add(leg)
return {
leg: tuple(sorted(target_ids))
for leg, target_ids in targets_by_leg.items()
if leg not in bound_legs
and all(
leg in _licensed_legs_for_target(target_id, licensed_empty_legs)
for target_id in target_ids
)
}


def reconcile_cross_grain_surface(
local_frame: pd.DataFrame,
groups: Iterable[CrossGrainInconsistency],
Expand Down Expand Up @@ -699,12 +769,16 @@ def apply_cross_grain_reconciliation(
reviewed_unbound_higher_targets: Mapping[str, Mapping[str, object]] | None = None,
licensed_empty_legs: Mapping[str, frozenset[str]] | None = None,
check_factor_bounds: bool = True,
frame_only_positions: Iterable[int] = (),
) -> tuple[pd.DataFrame, dict[str, Any]]:
"""Detect, reconcile, and return the always-present pass receipt.

``check_factor_bounds=False`` records each bounded bridge's factor without
refusing it, for a pass that is not the caller's last (a surface whose
off-surface cells join a second pass is bounded on that one).
``frame_only_positions`` names the rows that bind nothing (deferred
cells); a leg holding only those needs no control
(``deferred_legs_without_control``).
"""

bound = tuple(str(target_id) for target_id in bound_higher_targets)
Expand All @@ -728,12 +802,14 @@ def apply_cross_grain_reconciliation(
controls_without_lower_rows,
delegated_legs,
absent_middle_tier_legs,
deferred_legs_without_control,
) = _reconcile_cross_grain_surface_with_control_receipts(
local_frame,
groups,
rule,
licensed_empty_legs=licensed_empty_legs,
check_factor_bounds=check_factor_bounds,
frame_only_positions=frozenset(int(p) for p in frame_only_positions),
)
receipt = {
"bound_higher_targets": list(bound),
Expand All @@ -745,6 +821,7 @@ def apply_cross_grain_reconciliation(
"controls_without_lower_rows": controls_without_lower_rows,
"delegated_legs": delegated_legs,
"absent_middle_tier_legs": absent_middle_tier_legs,
"deferred_legs_without_control": deferred_legs_without_control,
"absence": (
None
if groups
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,9 +285,9 @@
"reason": "Concept, not a weight question: the ONS UKEA HAXV row is households' (S.14) D.41 interest resources in the national accounts (FISIM-adjusted, including the unincorporated-business part of the sector; GBP 95.5bn at 2025) while the model's savings_interest_income is cash interest, so the row is unreachable by design (-82 percent on the v20 national candidate). From 2026-09-22 the fitting anchor for interest is the SPI 2023-24 Table 3.7 interest rows by total-income band, uprated to the calibration year (hmrc.spi.savings_interest_income.*, PolicyEngine/chronicle#280 lane); this row is measured on every evaluation as the D.41 diagnostic and never fitted. Retire only with a declared D.41-to-cash bridge.",
"tracking": "microcosm#866",
"approved_by": "juaristi22",
"adjudication": "PolicyEngine/microcosm#1006 (issue comment 5812206367, the register rulings María gave in session: ruling R3 of 2026-09-22, the SPI interest rows bound and the ONS row a measured diagnostic; income-anchor lane, PolicyEngine/chronicle#280); previously microcosm#762 A16 (2026-09-03).",
"approved_on": "2026-09-22",
"expires_on": "2026-10-22"
"adjudication": "PolicyEngine/microcosm#1006 (issue comment 5812206367, the register rulings María gave in session: ruling R3 of 2026-09-22, the SPI interest rows bound and the ONS row a measured diagnostic; income-anchor lane, PolicyEngine/chronicle#280); previously microcosm#762 A16 (2026-09-03); renewed on 2026-10-16 to 2027-01-08 by juaristi22 in session (folded into the sixth microcosm#1095 ports PR): the gap is one of concept, not of weights, so the row stays a measured diagnostic until a D.41-to-cash bridge is declared.",
"approved_on": "2026-10-16",
"expires_on": "2027-01-08"
},
{
"name": "obr.housing_benefit",
Expand Down
3 changes: 3 additions & 0 deletions packages/microcosm-build/src/microcosm/build/uk/gates.json
Original file line number Diff line number Diff line change
Expand Up @@ -953,6 +953,9 @@
"household.cash_isa",
"household.cgt_support_copies",
"household.cgt_support_copy_index",
"household.high_value_support_copies",
"household.high_value_support_copy_index",
"household.household_is_high_value_support_copy",
"household.household_is_cgt_residential_clone",
"household.cgt_residential_clone_index",
"household.household_clone_index",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1997,13 +1997,17 @@ def apply_uk_cross_grain_reconciliation(
extra_bridges: Sequence[CrossGrainBridge] = (),
extra_signatures: Mapping[str, Mapping[str, Any]] | None = None,
check_factor_bounds: bool = True,
frame_only_positions: Iterable[int] = (),
) -> tuple[pd.DataFrame, dict[str, Any]]:
"""Apply the standing UK rule to a bound mixed-grain target surface.

``extra_bridges`` and ``extra_signatures`` carry the summed fan-out
controls the joint surface adds for its declared fan-out sum bridges.
``check_factor_bounds=False`` records the bridges' factor bounds without
refusing, for a pass the caller reruns with more rows.
``frame_only_positions`` names the rows that bind nothing (the small-cell
rule's deferred cells), so a wholly deferred leg with no control above it
is receipted rather than refused (microcosm#1301).

Increment #762 may extend the grains in the UK rowwise solve only through
this front door, so detection, reconciliation, and the manifest receipt
Expand Down Expand Up @@ -2040,6 +2044,7 @@ def apply_uk_cross_grain_reconciliation(
reviewed_unbound_higher_targets=reviewed_unbound_higher_targets,
licensed_empty_legs=licences,
check_factor_bounds=check_factor_bounds,
frame_only_positions=frame_only_positions,
)


Expand Down Expand Up @@ -2741,6 +2746,14 @@ def reconcile(
columns=["grain", "geography_id", "target_id", "value", "_output_position"],
)
raw = frame[["grain", "geography_id", "target_id", "value"]]
# The area rows that bind nothing (deferred and off-surface cells): a
# leg holding only deferred ones needs no control (microcosm#1301).
frame_only = [
position
for position, row in enumerate(reconciliation_rows)
if row["_output_position"] is None
and row["grain"] in level_to_area_type.values()
]
result, pass_receipt = apply_uk_cross_grain_reconciliation(
raw,
bound_control_ids,
Expand All @@ -2750,6 +2763,7 @@ def reconcile(
extra_bridges=fanout_bridges,
extra_signatures=fanout_signatures,
check_factor_bounds=check_factor_bounds,
frame_only_positions=frame_only,
)
return frame, raw, result, pass_receipt

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,13 @@ def __post_init__(self) -> None:
# #1045: the CGT support split's family copy count.
"household.cgt_support_copies",
"household.cgt_support_copy_index",
# microcosm#1240: the high-value homes support split's family copy count,
# index and flag, the CGT split's precedent. The stage's coverage-manifest
# outputs; the first dense release build failed this gate on them
# (microcosm#1301).
"household.high_value_support_copies",
"household.high_value_support_copy_index",
"household.household_is_high_value_support_copy",
# #1063: the residential split's arm flag and index.
"household.household_is_cgt_residential_clone",
"household.cgt_residential_clone_index",
Expand Down
Loading
Loading