Compare UK cuts with the current build of their own line

changelog.d/1095-certify-against-current-build.changed.md

@@ -0,0 +1 @@
+UK cuts check and certify their candidate against the current build of their own line, as the Hub's `releases.json` names it, instead of the enhanced FRS (microcosm#1095, now that the migration in microcosm#665 is complete): national against national, dense against dense. A cut resolves its reference once, refusing a build that is withdrawn or not certified, and keeps its record, the dataset and the evidence the gates read from it with the run (`tools/run_uk_cut.py`'s new `reference` step, or `tools/resolve_uk_reference.py` outside it). The comparison of the candidate's target fit with the reference's is a report with no verdict (`score_vs_reference.json`), built by one country-neutral head-to-head module that the US comparison tool now shares. The input-mass gate compares each column's total with the reference's at the reference's own solved weights, within the largest relative drift measured between the last two national builds, and no column may lose more than 90% of the reference's total (the largest drop between those builds, rounded up for headroom), since a symmetric tolerance above one fails no drop short of zero; it withholds the figures of the columns fewer than three source households carry. The export-surface gate compares with the reference's exported columns, so the declared extras shrink from 250 to the 32 the current national build does not export (the dense line's geography and clone columns, and six that microcosm#1550 adds), and an entry the reference now carries is reported as redundant rather than failed. The input-coverage gate requires every loader input the reference carries, measured by the gate's own rule, less the coverage manifest's reviewed exclusions. The gross receipts that microcosm#1550 stops writing, which the current builds still carry, are excluded against those builds in all three gates until a promotion moves the reference past them. The trading allowance full-relief carrier that five SPI rows filter on since microcosm#1439, which the current national build does not carry, is signed as unresolvable on that build for the comparison, as microcosm#1572 signed it on the enhanced FRS. A cut can pin a listed build as its reference (`--reference-build`) for a line with no current build, recorded as pinned, and each national certification records the change since a fixed baseline, chained from its reference's record without loading the baseline, listing every column that has left the chain and recording a chain that restarts. The certification and the release contract move to schema 2 with a `reference` block, the dense build takes `--input-mass-reference` and `--reference-surface` from its own line's current build, and the frozen enhanced-FRS facts stay only as a historical record.

docs/gate-battery-contract.md

@@ -126,7 +126,8 @@ Canonical artifact keys:
 | `coverage_engine`, `coverage_manifest` (optional override) | the UK national build | `release_input_coverage`, both phases |
 | `build_stage_names` | the UK national build preflight | `source_coverage` |
 | `parity_evidence` | the UK national build | `export_surface`, `target_surface`, `target_fit` |
-| `input_mass_reference` | the UK national build, when the licensed sidecar is supplied | `input_mass_parity` (UK override) |
+| `reference_surface` | the UK release cut (`--reference-surface`), and the dense build when given one | `export_surface` (the reference's exported columns), `release_input_coverage` (the inputs it carries), microcosm#1095 |
+| `input_mass_reference` | the UK release cut and the dense build: the current build's totals and carriers, from the cut's reference step | `input_mass_parity` (UK override) |
 | `reviewed_input_mass_exclusions` (optional override) | the UK national build | `input_mass_parity` (UK override) |
 | `uk_area_support_summary`, `exclusions_evaluated_on` | the UK local rowwise build | `area_support` (UK binding; reviewed exclusions state the area's figures beside the floors it misses, and stale entries fail; they are gate-only and defer none of the area's cells, microcosm#1301) |
 | `local_target_diagnostics`, `exclusions_evaluated_on` | the UK local rowwise build | `per_family_fit` (the diagnostics only); `target_fit` on `surface: local_candidate` (UK binding; the local cells stand on `local_target_fit_reviewed_exclusions.json`, keyed `family/area_code/metric`, which lifts the fence only and is evaluated on the build's review date, microcosm#1301) |
@@ -229,7 +230,7 @@ stretches on declaration alone:
 | --- | --- | --- |
 | phases | `terminal` | `preflight`, `assembled`, `transferred`, `terminal` |
 | entries | 9 | 73 (64 national-build entries, 7 local-candidate entries and 2 local diagnostics) |
-| incumbent posture | none — incumbent-comparison gates deliberately not selected; external oracles replace self-parity | full — export/target parity against the pinned enhanced-FRS incumbent |
+| incumbent posture | none — incumbent-comparison gates deliberately not selected; external oracles replace self-parity | full — export surface, input coverage and input mass against the current build of the cut's own line (microcosm#1095; until then the pinned enhanced-FRS incumbent) |
 | criticality mix | 8 blocking + 1 diagnostic | 71 blocking + 2 local diagnostics |
 
 The differences are entirely in the two country inputs — the spec file
@@ -257,15 +258,21 @@ inputs — never implicit module state.** A country picks its reference on
 purpose and cannot silently inherit a mis-referenced comparison.
 
 The UK input-mass entry is the worked case. Its `parameters` declare the
-active `reference`, the closed `reference_registry` of named identities plus
-`totals_sha256` pins, the numeric thresholds, and the
-`reviewed_exclusions_resource` for the schema-3 per-reference register. The
-UK binding holds the runtime-supplied reference sidecar to the active named
-descriptor, failing closed on identity or totals-digest drift; an exclusion
-suppresses only for the active reference block. The export-surface entry declares its
-reviewed comparison registers the same way (the allowed-extra column list
-and the reviewed reference-side exclusions), with the hard-required,
-never-waivable columns remaining a code-level guard noted in the entry.
+active `reference` kind, `current_build` (microcosm#1095): the current build
+of the cut's own line, which the cut's reference step resolves from the
+Hub's `releases.json` and records, so the spec commits to the rule and the
+cut's record to the build. They also declare the numeric thresholds and the
+`reviewed_exclusions_resource` for the schema-3 per-reference register, whose
+blocks are keyed by the reference's `<release>@<build>` label; an exclusion
+suppresses only for the reference it names. The export-surface entry declares
+its reviewed comparison registers the same way (the allowed-extra column
+list, now only what the current national build does not export, and the
+reviewed reference-side exclusions), with the hard-required, never-waivable
+columns remaining a code-level guard noted in the entry. The input-coverage
+entry requires what the reference carries, less the coverage manifest's
+reviewed exclusions. Until microcosm#1095 all three compared with the pinned
+enhanced FRS, through a closed registry of named identities with
+`totals_sha256` pins.
 
 ## The onboarding checklist: every input a country supplies
 

docs/uk-cut-anyone.md

@@ -48,14 +48,18 @@ with `agent-secret get microcosm/uk-gate-signing-key-team maxghenis`, and
 
 1. `fetch`
 2. `preflight`
-3. spine
-4. national dry run, then the national calibration
-5. re-score, only when the in-build evaluation did not pass
-6. certify
-7. stage-1 comparison (with `--previous-h5`)
-8. assemble
-9. publish preflight
-10. publish (with `--publish`)
+3. `reference`: the current national build the cut compares with
+   ([The reference build](#the-reference-build))
+4. spine
+5. national dry run, then the national calibration
+6. re-score, only when the build's comparison with the reference is missing
+   or names another candidate or reference
+7. certify
+8. stage-1 comparison, against the reference (`--previous-h5` names another
+   file)
+9. assemble
+10. publish preflight
+11. publish (with `--publish`)
 
 Every step first checks whether its outputs already exist and verify, and
 skips itself if they do. So when a run stops, fix the cause and run the same
@@ -85,6 +89,7 @@ derive from licensed data:
 | Path | Contents |
 |---|---|
 | `inputs/` | shared verified inputs |
+| `cuts/<label>-<sha9>/reference/` | the build the cut compares with: its dataset, its record (`reference.json`) and the evidence the gates read from it |
 | `cuts/<label>-<sha9>/spine/` | the spine |
 | `cuts/<label>-<sha9>/national/` | the calibration run |
 | `cuts/<label>-<sha9>/release/` | the assembled release |
@@ -150,11 +155,10 @@ MICROCOSM_UK_CUT_MODAL_RUN=1 modal run --detach tools/modal_uk_cut.py --commit <
 `tools/uk_cut_inputs.json` says where each input lives. Every expected digest
 is read at run time from the pins the build already checks:
 `uk/source_stages.json`, `uk/frs_release.json`, `uk/chronicle_feed.json` and
-`uk/gates.json`. The one exception is the input-mass reference: the build
-binds it by a canonical digest of its contents, so the manifest also carries
-the hosted file's byte sha256. A file is installed only if it
-is byte-identical to what the build will accept, and nothing unverified is
-ever left at an input's path.
+`uk/gates.json`. A file is installed only if it is byte-identical to what the
+build will accept, and nothing unverified is ever left at an input's path.
+The build the cut compares with is not a fixed input: the `reference` step
+resolves it ([The reference build](#the-reference-build)).
 
 **Who can read them.** `policyengine/microcosm-uk-build-inputs`,
 `policyengine/populace-uk-private` and `policyengine/populace-uk-staging` are
@@ -168,8 +172,6 @@ is what admits a builder to the licensed inputs.
 | FRS 2024-25 (14 tabs) | `policyengine/policyengine-uk-data-private` `frs_2024_25.zip` @ `a2039519` (gated) | fetched, every member matches its pin |
 | SPI 2022-23, WAS round 8, ETB 1977-2024 | the same repo's `spi_2022_23.zip`, `was_2006_22.zip`, `etb_1977_24.zip` | fetched, matches |
 | HMRC personal incomes ODS | gov.uk URL pinned in `source_stages.json` | fetched, matches |
-| Incumbent enhanced FRS | `policyengine-uk-data-private` `enhanced_frs_2024_25.h5` @ `a9e52499` | fetched, matches |
-| Input-mass reference | `policyengine/microcosm-uk-build-inputs` (private) `input-mass-reference/fd41cb5f…/` | fetched, matches |
 | Chronicle UK consumer artifact (about 8 GB) | `policyengine/microcosm-uk-build-inputs` `chronicle-uk/<the feed's source commit>/`, read at main | fetched, both files match the feed pin. The path follows `chronicle_feed.json`, so each re-pin needs its artifact hosted (below). 64753803's (main's pin since PolicyEngine/chronicle#411) was uploaded on 2026-11-06 from a local build that matches both digests; 272c987's on 2026-10-26 from the build #1359 pinned; abf0921's was rebuilt on Modal and reproduced the pin byte for byte |
 | NTS 2002-2024 (6 tabs) | `policyengine-uk-data-private` `nts_2002_24.zip` @ `559b69e9` | fetched, every member matches its pin |
 | LCFS 2024-25 (2 tabs) | `policyengine-uk-data-private` `lcfs_2024_25.zip` @ `11051052` (a repack of the extracted files) | fetched, every member matches its pin |
@@ -232,22 +234,79 @@ the private repositories, is attached only with
 ephemeral secret for that run; no named workspace secret keeps it. A rebuild
 without `--upload` sends no credential at all.
 
-### Regenerating the input-mass reference
+### The reference build
 
-The certifier binds the reference by a canonical digest of its identity and
-totals. The emitted column set follows the installed policyengine-uk.
-Today's engine adds `bus_fare_spending` and `sic_industry_division`. That
-changes the digest, though all 131 shared totals are identical. Reproduce the
-pinned file (`fd41cb5f…`) at the commit that pinned it:
+A cut checks and certifies its candidate against the current build of the
+national line, as `releases.json` in `policyengine/populace-uk-private` names
+it (microcosm#1095). Until then every cut compared with the enhanced FRS.
+
+The `reference` step resolves that build once per cut. It refuses a build
+that is withdrawn or whose `release_certification.json` is not shippable,
+names another file or was signed by an unregistered key. It downloads the
+dataset, checks its sha256 and writes four files to
+`cuts/<label>-<sha9>/reference/`:
+
+- the dataset;
+- `reference.json`, the record every later step reads, so a promotion while
+  the cut runs does not change what it is compared with;
+- `input_mass_reference.json`, the build's input-mass totals and carriers at
+  its own solved weights, for the input-mass gate;
+- `reference_surface.json`, the columns it exports and the loader inputs it
+  carries, names and flags only, for the export-surface and input-coverage
+  gates.
+
+The totals are licensed-data aggregates and stay in the work directory like
+the dataset. The candidate's score against the reference is a report with no
+verdict (`score_vs_reference.json`); the certification records the reference
+it was checked against.
+
+To see what a cut would compare with now, without downloading anything:
 
 ```bash
-git worktree add --detach ../mc-0eac9bbbd 0eac9bbbd && cd ../mc-0eac9bbbd
-uv sync --all-packages --locked --extra uk        # policyengine-uk 2.89.0
-uv run --no-sync python tools/build_uk_efrs_parity_reference.py \
-  --input-h5 <enhanced_frs_2024_25.h5 @ a9e52499> --emit-weighted-totals <outside the repo>.json
+uv run --no-sync python tools/resolve_uk_reference.py --release microcosm-uk-2024-25-national --check
 ```
 
-The hosted copy's receipt records this recipe.
+`--re-resolve-reference` sets a cut's reference aside (as
+`reference.superseded-<time>`) and resolves the current build again; the
+score and the certification then run again against it.
+
+It refuses to move to the cut's own candidate, which becomes the current build
+once the cut is promoted. A build withdrawn after the fetch stops the certify
+and publish steps, even once the cut is certified, and `status` says so; the
+cut keeps its record until `--re-resolve-reference` moves it, which certifies
+the cut again.
+
+A line with no current build (a new line, or one whose only current build
+was withdrawn, `current: null`) refuses at the reference step. Pin one of its
+listed builds instead, with `--reference-build <build>` here or `--build` on
+`tools/resolve_uk_reference.py`. A pinned build passes the same certification
+checks as the current one and is refused if withdrawn. The record and the
+certification mark it `pinned`. A cut that already compares with another
+build keeps it unless `--re-resolve-reference` is given too. The release ids
+and the registry year are code constants, so the next FRS year needs a code
+change.
+
+Every check compares a cut with the build before it, so the certification
+also records the change since a fixed baseline:
+- each input column's total as a ratio of the baseline's;
+- each column that has left that chain, with why and its last ratio;
+- the inputs the baseline carried;
+- which of those this build no longer carries.
+
+The chain starts at the build each line compared with first (for the national
+line, eeb6093e). Each cut continues its reference's record, multiplying in
+the ratios its own input-mass gate measures, so no baseline dataset is ever
+loaded. It continues from the reference's certification kept beside it, which
+a resumed cut keeps again if the copy is missing.
+
+A column leaves the chain when it is absent on a side, when the reference's
+total is zero, or when one or two source households carry it on a side (its
+figures are withheld). It stays listed in every later certification, with the
+reason and the last ratio the chain recorded for it. A national chain that
+starts at any build but eeb6093e (a pinned build certified before the record
+began, say) has restarted, and the certification records `restarted_from`.
+The record carries no verdict. The dense line's published evidence carries
+none of these gates, so its chain restarts at each reference.
 
 ## Signing keys
 

docs/uk-dense-release-assembly-runbook-762.md

@@ -47,12 +47,26 @@ the gates stood on. See [Optional diagnostics](#optional-diagnostics-microcosm13
 - Optionally, the incumbent extraction
   (`tools/extract_uk_local_incumbent_surface.py`), only for the opt-in
   incumbent comparison.
-- The licensed input-mass reference the national line keeps with its runs
-  (`--input-mass-reference`). `uk_input_mass_parity` is release-blocking on
-  this line too and records `evidence_absent` without it. The sidecar's totals
-  digest must match the registry's `totals_sha256`; the totals depend on the
-  installed policyengine-uk, so re-emitting them under a newer engine does not
-  reproduce the pinned file.
+- The reference: the current build of the dense line, as `releases.json` in
+  `policyengine/populace-uk-private` names it (microcosm#1095). Resolve it
+  once, into a directory the run keeps:
+
+  ```bash
+  uv run --no-sync python tools/resolve_uk_reference.py \
+    --release microcosm-uk-2024-25-dense --output-dir <reference-dir>
+  ```
+
+  This refuses a build that is withdrawn or has a failed gate in its
+  `gate_summary.json`, and writes the dataset, its record (`reference.json`),
+  `input_mass_reference.json` (its input-mass totals and carriers at its own
+  solved weights, licensed-data aggregates that stay with the run) and
+  `reference_surface.json` (the columns it exports and the inputs it carries,
+  names and flags only). The build takes the last two
+  (`--input-mass-reference`, `--reference-surface`): `uk_input_mass_parity`,
+  `uk_export_surface` and `uk_release_input_coverage` are release-blocking on
+  this line too and record `evidence_absent` without them, and the build
+  refuses the two files if they name different builds. Until microcosm#1095
+  the dense line was held to the enhanced FRS's input totals and inputs.
 
 ## 1. Pre-flight the environment
 
@@ -73,7 +87,8 @@ uv run --no-sync python tools/build_uk_full.py --release-role dense \
   --ladder build/uk/uk_oa_ladder_2021.npz --ladder-sha256 <ladder> \
   --ledger-facts <chronicle-uk-artifact-dir> --ledger-facts-sha256 <facts> \
   --ledger-manifest-sha256 <manifest> --seed 42 \
-  --input-mass-reference <licensed-input-mass-reference> \
+  --input-mass-reference <reference-dir>/input_mass_reference.json \
+  --reference-surface <reference-dir>/reference_surface.json \
   --review-date <YYYY-MM-DD> --graph-store <graph-store-dir> \
   --logbook-prev-row-digest <previous row> --out <candidate-dir>
 ```
@@ -458,7 +473,7 @@ records a `rotated_holdout` limitation with `claim: report_only` and the
 coverage's `holdout` block (`run: true`); without it, `no_out_of_sample_claim`
 and `holdout: {run: false, claim: none}`.
 
-**Incumbent comparison.** Score the candidate, then measure the pinned
+**Incumbent comparison.** This local score still compares with the uk-data incumbent's area weights: it is a different comparison from the gates' reference build (microcosm#1095), and that change leaves it as it was. Score the candidate, then measure the pinned
 incumbent surface:
 
 ```bash