Allow a UK local-area release on an accepted model mismatch

changelog.d/699.fixed.md

@@ -0,0 +1,3 @@
+Require a byte-bound input-surface audit when accepting a UK regional model mismatch. The audit compares the regional H5's stored columns with the regional release's build model as well as the target model: it refuses columns the build model defines and the target does not, records columns neither model defines as never modelled, and reports as defaulted only target inputs the file does not store that were not build-model inputs. Certification reads both models again rather than trusting the evidence's lists, records that the stored columns are the auditor's (`stored_columns: auditor_attested`), and adds a limitation naming the defaulted inputs. The bundle and TRACE record the result and limitations and preserve the regional manifest digest and area build model; an area run's execution receipt carries a summary bound to the record's digest and stays uncertified. Dropping the regional release removes its exception record.
+
+Refuse regional releases that explicitly name a different model package, and forward empty CLI reasons so certification rejects them, as it rejects reasons with no visible character. Document that the UK loader silently skips unknown columns.

changelog.d/accept-regional-model-mismatch.added.md

@@ -0,0 +1 @@
+`certify-data --accept-regional-model-mismatch REASON --regional-input-surface-evidence AUDIT.json` (UK only) keeps a local-area release routed when it was built on another policyengine-uk version, and records the reviewed mismatch as `regional_certification` in the bundle instead of refusing; area receipts are uncertified. `scripts/audit_regional_input_surface.py` writes the audit, reading the release's build model in a throwaway `uv run` environment, and `verify_regional_certification` checks an area receipt's record of the exception.

docs/engineering/skills/data-certification.md

@@ -79,6 +79,134 @@ Microcosm national release with one of:
   claim first. Its `frs_2024_25` is not carried, so keep it loadable with a
   `dataset_overlays.uk` entry.
 
+- **An older local-area release, on an accepted mismatch.** When the national
+  cut moves to a new policyengine-uk before a local-area cut on that version
+  exists, a reviewer can accept retaining the older regional route. A reason
+  and a byte-bound input-surface audit are both required. First audit the
+  actual local H5 with the exact target model installed. The audit also needs
+  `uv` and access to PyPI (see below):
+
+  ```bash
+  .venv/bin/python scripts/audit_regional_input_surface.py \
+    --regional-manifest-uri "<the routed local-area release_manifest.json URI>" \
+    --artifact "microcosm_uk_2024_25_dense=/local/dense.h5" \
+    --model-version 2.147.2 \
+    --audited-by "<reviewer>" \
+    --limitation "the dense cut's VAT predates the policyengine-uk VAT factor fix (#2187, 2.135.7); the dashboard holds a VAT limit" \
+    --output regional-input-surface.json
+
+  .venv/bin/python scripts/bundle.py certify-data --country uk --data-producer populace \
+    --manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
+    --regional-manifest-uri "<the routed local-area release_manifest.json URI>" \
+    --accept-regional-model-mismatch "<who accepted it and why>" \
+    --regional-input-surface-evidence regional-input-surface.json \
+    --model-version 2.147.2
+  ```
+
+  The audit compares the entity-table columns the H5 actually stores, in each
+  period, with two models: the build model and core the regional manifest
+  records (`build.built_with_model_package` and
+  `build.built_with_core_package`), and the target. As in Microcosm #1348's
+  `tools/certify_uk_cut_in_policyengine_py.py`, the build model is the
+  baseline:
+
+  - **Dropped columns** are stored columns the build model defines and the
+    target does not. The target's loaders skip them. Certification refuses
+    **any dropped column**; there is no override.
+  - **Never-modelled columns** are stored columns neither model defines:
+    Microcosm's own area codes (among them `constituency_code_oa` and
+    `la_code_oa`), build and source ids, clone indices and SPI donor fields.
+    No model ever loaded them, so they are recorded, not refused.
+  - **Defaulted inputs** are target inputs the file does not store, after the
+    target's input aliases, that were not inputs of the build model. Area runs
+    default them where the build model did not. Inputs absent under both
+    models are not changes.
+
+  The audit reads the build model's variables and inputs in a throwaway
+  `uv run --isolated --no-project --no-config --default-index
+  https://pypi.org/simple --exclude-newer <build.built_at> --with
+  policyengine-uk==<build> --with policyengine-core==<core>` environment, so
+  it needs `uv` and access to PyPI. `--isolated` makes the environment fresh:
+  without it, uv reuses an environment that already meets both pins, such as
+  a `.venv` in the working directory, and neither the index nor the cutoff
+  applies. `--exclude-newer` is the manifest's build time, so the build
+  model's other dependencies resolve as they did then, and every `UV_*`
+  setting except the cache, Python-install and connection ones is removed. A
+  regional manifest that does not record both build versions has no
+  baseline, and one without `build.built_at` cannot be read as of its build;
+  the audit refuses both. The evidence binds the regional release identity,
+  original manifest bytes, actual H5 bytes, both build versions and the exact
+  target model version, and keeps both models' full variable and input lists.
+
+  Certification does not trust those lists. It reads the target model in
+  process, so the environment that runs `certify-data` must have the target
+  policyengine-uk installed, as the audit's does, and reads the build model
+  again with `uv`; any list or input alias that differs refuses. Only the
+  stored columns rest on the auditor's word: they cannot be re-read without
+  the H5, and the evidence binds them to its SHA-256. The record says so,
+  with `stored_columns: auditor_attested`. Certification then
+  recomputes all three lists and refuses missing or mismatched evidence,
+  including evidence without `schema_version: 2`, from before the comparison
+  was relative to the build model. It records the never-modelled columns,
+  every defaulted input and the free-text limitations in the validated
+  `regional_certification`, and adds one limitation naming the defaulted
+  inputs; the bundle hash binds that record.
+
+  The policyengine-uk 2.147.2 dataset loaders silently skip columns absent
+  from the model's variables: both the
+  [generic dataset loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1113-L1127)
+  and the
+  [UK multi-year loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1186-L1196)
+  use `continue` without warning. A removed or renamed input can therefore
+  disappear silently at runtime; the audit must catch that column loss before
+  certification.
+
+  The national certificate never widens. Area status remains `uncertified`:
+  the certificate still covers the national dataset only (#685). TRACE
+  carries the whole regional record, evidence included, separately from the
+  national certification. An area run's execution receipt carries a summary
+  of it (both model versions, the reason, the three lists and the
+  limitations) and `record_sha256`, the digest of the whole record; check it
+  with `verify_regional_certification`. A national receipt carries none: its
+  `certificate_sha256` already digests the record. The flag remains UK only;
+  an empty or invisible reason, a release that already covers the target
+  model, or a missing regional release refuses.
+  The regional core checks still apply. Re-certifying with
+  `--drop-regional-release` removes the regional record, routes and artifact.
+
+  For the Budget route, the audit of dense cut
+  `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`, built with
+  policyengine-uk 2.132.6 and policyengine-core 3.33.3, measured in #699:
+
+  - **83 never-modelled columns**, and **no dropped column** under either
+    2.147.2 or 2.151.0. The first version of this audit counted those 83 as
+    dropped and refused the cut, even against its own build model.
+  - **16 new defaults under 2.147.2:**
+    `additional_residential_property_purchased`,
+    `defined_benefit_pension_share`, `directly_held_shares`,
+    `early_years_school_start_deferred`, `is_full_year_category_a_director`,
+    `local_bus_uncapped_single_fare`, `non_residential_property_purchased`,
+    `non_residential_property_secured_debt`,
+    `other_residential_property_secured_debt`,
+    `pension_annual_allowance_taxable_lump_sum_death_benefits`,
+    `pension_contributions_via_salary_sacrifice_pre_2015`, `pip_dl_points`,
+    `pip_m_points`, `salary_sacrifice_returned_to_income`,
+    `unit_and_investment_trusts`, and `war_widows_pension_reported`.
+  - **37 new defaults under 2.151.0:** those 16 and 21 more. The 21 include
+    the gambling duty inputs, liquid and solid fuel consumption, and bonus
+    sacrifice (`pension_contributions_via_bonus_sacrifice`), so area
+    breakdowns of those measures use defaults on 2.151.0. Certification
+    names every defaulted input in the limitations.
+  - Another **123 inputs already defaulted under 2.132.6**. They are absent
+    under both models, so they are not changes; the first version listed
+    them too, giving 139 defaults under 2.147.2 and 160 under 2.151.0.
+
+  Record the pre-fix VAT limitation shown above for this dense cut.
+
+  After the receipt schema changes, refresh the Budget dashboard's vendored
+  `tests/fixtures/w4/execution_receipt.v1.schema.json` from
+  `src/policyengine/data/schemas/execution_receipt.v1.schema.json`.
+
 Either way, certification checks that the regional release serves the pinned
 model by its own build or compatibility claims (a `--qualified-pair-id` names
 the primary release only), checks its core claims as for the primary release

docs/receipts.md

@@ -11,6 +11,7 @@ Schema v1 covers UK results and is defined by `policyengine.provenance.receipt.E
 | `wrapper`, `model`, `core` | The `policyengine`, `policyengine-uk` and `policyengine-core` versions that actually ran, read from the installed distributions rather than from the bundle manifest. `install_origin` records a PEP 610 direct URL as `archive`, `directory`, `editable` or `vcs`; it is absent for an ordinary index install with no `direct_url.json`. `wheel_sha256` appears only when the receipt builder verified it. One way is `wheel_paths=` (evidence `installed_record_verified`): the installed files, and any bytecode cache the interpreter would load instead of a source file, are checked against that wheel's RECORD. The other is the installer's recorded archive hash (`installer_archive_hash`), taken only when the install came from a `.whl` URL; a source archive's hash is never recorded as a wheel hash. |
 | `data` | The data release and build, and the model version the data was built with (`build_model_version`, distinct from `model.version`). Also the dataset's logical name, URI, revision and the SHA-256 of the bytes that were simulated. The dataset is resolved exactly as `materialize_dataset` resolves it (overlays, per-dataset repository, package and revision, and the release manifest revision when an artifact has none of its own), and the bytes must be the ones the bundle pins for it: a receipt never gives an artifact's name to other bytes. |
 | `certification_status`, `certificate` | `certified` only when the bundle certifies all of: the installed model and core versions, that dataset name and those bytes, and the running `policyengine` version. The UK certificate names one dataset, the national default, so a run on any other dataset, the local-area file that constituency and local-authority regions load included, gets an uncertified receipt. Each package must also have an absent or `archive` install origin, or a `directory` or `vcs` origin with `installed_record_verified` wheel evidence. A directory or VCS install without that evidence produces an `uncertified` receipt, which carries no certificate. An editable install is never certified, whatever the evidence: its wheel holds a pointer to a source tree (a `.pth` file or an import hook) rather than the source, so checking the installed files against it does not bind the code that ran. `certificate_source` records whether the builder read the certificate from the installed bundle (`packaged`) or from a manifest the caller passed (`caller_supplied`). It is the producer's statement, not evidence: trust a certificate only after `verify_certificate` (below). With the packaged bundle, the composition fingerprint of its TRACE TRO is recorded as well. |
+| `regional_certification` | Present only on an area run: a run on the regional release that the bundle routes constituency and local-authority regions to on an accepted model mismatch (`data_releases.uk.regional_certification`). Such a run is uncertified. The member summarises that exception: the model the release was built with and the one area runs use, the reviewer's reason, the dropped, never-modelled and defaulted lists, how the stored columns are known (`stored_columns: auditor_attested`), and the limitations. `record_sha256` binds the whole record, its evidence included; the evidence's model surfaces stay in the bundle and its TRACE TRO. A national receipt carries none, because its `certificate_sha256` already digests the record. Check it with `verify_regional_certification` (below). |
 | `reform`, `baseline` | Both are required. Each is `current_law` or a digest of the reform bound to its representation (see below). |
 | `fiscal_year` | What a year label means: year 2027 is the fiscal year from 6 April 2027 to 5 April 2028, shown as `2027-28`. v1 allows only this convention. |
 | `years` | The reported years, sorted and unique. |
@@ -149,6 +150,8 @@ A reader who holds the members' identities can recompute the composition's diges
 
 To check a certificate, call `verify_certificate(receipt, bundle)` with the policyengine manifest for `certificate.bundle_version`. It rebuilds the certificate from that manifest's certification record, the same way the builder does. Then it compares every identity member: the record digest, the basis, the certified model, core and dataset digest, the certifier and the bundle version. It also checks that the receipt's whole `data` identity is the certified dataset with the certified bytes. An empty list means the bundle certifies exactly what the receipt claims, and through the receipt's own rules that pins `model.version`, `core.version` and `wrapper.version` too. `certificate_source`, `qualification_receipt_sha256`, `tro_composition_fingerprint` and wheel hashes are not determined by the bundle, so it does not check them.
 
+To check an area receipt's regional exception, call `verify_regional_certification(receipt, bundle)` with the policyengine manifest the run used. It rebuilds the summary from that manifest's `regional_certification` record, the same way the builder does, and compares every member, `record_sha256` included. It also checks that the receipt's `data` is the regional artifact that record covers, with the identity the bundle determines. An empty list means the receipt names exactly the bundle's exception; the area run stays uncertified.
+
 Pass the release artifact that was loaded as `dataset_path`. A derived per-year file such as `enhanced_frs_2024_25_year_2026.h5` has different bytes from the artifact the bundle pins, so a receipt for it is refused.
 
 ## Digests
@@ -195,7 +198,8 @@ Consumers that validate with the JSON Schema rather than the Python model must a
 - a `current_law` reform has no digest, while any other reform has both a representation and a digest;
 - a wheel digest always comes with its evidence;
 - a certified receipt has no `editable` package, and no `directory` or `vcs` package without `installed_record_verified` wheel evidence;
-- a certificate is trusted only after it, and the receipt's `data`, have been checked against the published manifest it names (`verify_certificate`).
+- a certificate is trusted only after it, and the receipt's `data`, have been checked against the published manifest it names (`verify_certificate`);
+- `regional_certification` appears only on an uncertified area run, and is checked against the manifest the run used (`verify_regional_certification`).
 
 The schema itself fixes the package names (`policyengine`, `policyengine-uk`, `policyengine-core`) and the UK fiscal-year convention, which starts on 6 April.
 

scripts/audit_regional_input_surface.py

@@ -0,0 +1,70 @@
+"""Write a byte-bound audit of regional H5 columns for an installed UK model.
+
+Run in an environment with the exact requested policyengine-uk version and
+PyTables installed. Supply the H5 files from the immutable regional manifest;
+the audit verifies their hashes and downloads no microdata. The baseline is the
+build model and core the regional manifest records: the audit reads their
+variables in a throwaway ``uv run --with`` environment, resolved from PyPI as
+of the manifest's build time, so it needs ``uv`` and PyPI. Certification reads
+both models again. Columns the build model defines and the target does not
+refuse; columns neither model defines are recorded as never modelled; and
+target inputs the files do not store that were not build inputs are recorded
+as defaulted.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+# The target model must be auditable before the current bundle certifies it.
+os.environ.setdefault("POLICYENGINE_SKIP_COUNTRY_IMPORTS", "1")
+sys.path.insert(0, str(REPO_ROOT / "src"))
+
+from policyengine.provenance.regional_input_surface import (  # noqa: E402
+    measure_regional_input_surface,
+)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--regional-manifest-uri", required=True)
+    parser.add_argument(
+        "--artifact", required=True, action="append", metavar="NAME=PATH"
+    )
+    parser.add_argument("--model-version", required=True)
+    parser.add_argument("--audited-by", required=True)
+    parser.add_argument("--limitation", required=True, action="append")
+    parser.add_argument("--output", required=True, type=Path)
+    args = parser.parse_args(argv)
+    paths: dict[str, Path] = {}
+    for entry in args.artifact:
+        name, separator, path = entry.partition("=")
+        if not separator or not name.strip() or not path.strip() or name in paths:
+            parser.error("Each --artifact must be a unique nonblank NAME=PATH.")
+        paths[name] = Path(path)
+    evidence = measure_regional_input_surface(
+        regional_manifest_uri=args.regional_manifest_uri,
+        artifact_paths=paths,
+        model_version=args.model_version,
+        audited_by=args.audited_by,
+        limitations=args.limitation,
+        token=os.environ.get("HUGGING_FACE_TOKEN") or os.environ.get("HF_TOKEN"),
+    )
+    args.output.write_text(evidence.model_dump_json(indent=2) + "\n")
+    print(
+        f"Saved {len(evidence.artifacts)} artifact input audits to {args.output}: "
+        f"policyengine-uk {evidence.model_version} against build model "
+        f"{evidence.build_model_version} (core {evidence.build_core_version}); "
+        f"{len(evidence.dropped_columns)} dropped, "
+        f"{len(evidence.never_modelled_columns)} never-modelled, "
+        f"{len(evidence.defaulted_inputs)} defaulted."
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())