Certify the 2036 US bundle on policyengine-us 2.2.2, with the ACS-local release on an audited mismatch

changelog.d/618.changed.md

@@ -0,0 +1 @@
+Document and test the 2022–2036 SPM forecast horizon: 2036 joins the supported household years and 2037 becomes the out-of-horizon example.

changelog.d/749.added.md

@@ -0,0 +1 @@
+`certify-data --accept-regional-model-mismatch` and `scripts/audit_regional_input_surface.py` now cover a US local-area release as well as a UK one. The audit reads the US file's entity tables (or PolicyEngine core's variable/period groups), compares them with the release's build policyengine-us and the installed target, and counts a stored legacy input as supplying its live input where .py renames it. The bundle records the accepted mismatch and its evidence; .py writes no US execution receipts.

changelog.d/certify-us-populace-us-2024-spm-20260915.changed.md

@@ -0,0 +1 @@
+Certify the US populace data release `populace-us-2024-spm-20260915` (populace_us_2024) for policyengine-us 2.2.2 with spm-calculator 1.1.0, whose SPM forecast covers 2022–2036 (`335e6755…`), read from its build folder in `policyengine/populace-us`. State and district regions keep the ACS-local release `populace-us-2024-buildo-acs-local-766b7c7-20261017T030317Z`, built with 2.2.1, on an audited accepted mismatch: policyengine-us 2.2.1 and 2.2.2 define the same variables and inputs, and no stored column is dropped.

docs/engineering/skills/data-certification.md

@@ -225,9 +225,10 @@ Microcosm national release with one of:
   of it (both model versions, the reason, the three lists and the
   limitations) and `record_sha256`, the digest of the whole record; check it
   with `verify_regional_certification`. A national receipt carries none: its
-  `certificate_sha256` already digests the record. The flag remains UK only;
-  an empty or invisible reason, a release that already covers the target
-  model, or a missing regional release refuses.
+  `certificate_sha256` already digests the record. The flag applies to UK and
+  US local-area releases (see [US](#us) for the US audit); an empty or
+  invisible reason, a release that already covers the target model, or a
+  missing regional release refuses.
   The regional core checks still apply. Re-certifying with
   `--drop-regional-release` removes the regional record, routes and artifact.
 
@@ -313,25 +314,29 @@ their `_tiny` files.
 
 ### US
 
-For US Populace certification, certify the Populace release manifest directly:
+For US Populace certification, certify the Populace release manifests
+directly. `policyengine/populace-us` publishes each build in its own release
+folder (it has `releases.json`), so give each manifest's folder at the commit
+that published it:
 
 ```bash
 python scripts/bundle.py certify-data --country us --data-producer populace \
-  --manifest-uri "hf://dataset/policyengine/populace-us@<tag>/releases/<tag>/release_manifest.json" \
-  --regional-manifest-uri "hf://dataset/policyengine/populace-us@<local-area-tag>/releases/<local-area-tag>/release_manifest.json" \
+  --manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-national/<build>/release_manifest.json" \
+  --regional-manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-local/<build>/release_manifest.json" \
   --model-version "<policyengine-us-version>"
 ```
 
 US state and congressional-district regions are row filters over one shared,
 certified ACS-local Populace dataset. Its release manifest must declare
 `dataset_role: non_default_local_area`, `is_default: false`, no default
-datasets, and exactly one pinned H5 microdata artifact. Certification writes:
+datasets, and exactly one pinned H5 microdata artifact. Certification writes
+each dataset's path in its build folder:
 
 ```json
 "region_datasets": {
-  "national": {"path_template": "populace_us_2024.h5"},
-  "state": {"path_template": "populace_us_2024_acs_local.h5"},
-  "congressional_district": {"path_template": "populace_us_2024_acs_local.h5"}
+  "national": {"path_template": "releases/microcosm-us-2024-national/<build>/populace_us_2024.h5"},
+  "state": {"path_template": "releases/microcosm-us-2024-local/<build>/populace_us_2024_acs_local.h5"},
+  "congressional_district": {"path_template": "releases/microcosm-us-2024-local/<build>/populace_us_2024_acs_local.h5"}
 }
 ```
 
@@ -343,6 +348,64 @@ files for compatibility checks, certification omits them from the runtime
 bundle. The country-wide default and ACS-local dataset are each single shared
 files; region registries select one and then filter its rows.
 
+**An older ACS-local release, on an accepted mismatch.** When the bundle
+moves to a policyengine-us the ACS-local release was neither built with nor
+claims, route it on an accepted mismatch as for the UK: audit the H5 with the
+exact target installed, then certify with the reason and the evidence.
+
+```bash
+.venv/bin/python scripts/audit_regional_input_surface.py \
+  --regional-manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-local/<build>/release_manifest.json" \
+  --artifact "populace_us_2024_acs_local=/local/populace_us_2024_acs_local.h5" \
+  --model-version "<policyengine-us-version>" \
+  --audited-by "<reviewer>" \
+  --limitation "<what the build model and the target differ in>" \
+  --output regional-input-surface.json
+
+.venv/bin/python scripts/bundle.py certify-data --country us --data-producer populace \
+  --manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-national/<build>/release_manifest.json" \
+  --regional-manifest-uri "hf://dataset/policyengine/populace-us@<commit>/releases/microcosm-us-2024-local/<build>/release_manifest.json" \
+  --accept-regional-model-mismatch "<who accepted it and why>" \
+  --regional-input-surface-evidence regional-input-surface.json \
+  --model-version "<policyengine-us-version>"
+```
+
+The audit, the comparison and the record are the UK's (see the accepted
+mismatch under [UK constituency and local-authority regions](#uk-constituency-and-local-authority-regions)),
+with three US differences:
+
+- **The file layout.** The audit reads either layout .py's US loader reads,
+  and only names, never values. The Microcosm ACS-local release stores entity
+  tables, one per US entity (`person`, `household`, `tax_unit`, `spm_unit`,
+  `family`, `marital_unit`), with `_time_period` naming its year; every entity
+  table is required, and a file storing them by year (`person/2024`) refuses,
+  since .py's US loader and policyengine-us's `Microsimulation(dataset=<path>)`
+  read single-year files only. A file in PolicyEngine core's variable/period layout (an
+  H5 group per variable, holding an array per period, and a `person_id`
+  group) is read by group and period names; a top-level dataset there is file
+  metadata, and a period that is not a year refuses. A file in neither layout
+  refuses.
+- **The input aliases.** policyengine-us renames inputs; .py maps a stored
+  legacy column onto its live input (`LEGACY_INPUT_RENAMES`), when the target
+  defines the live name and not the legacy one, and the file stores the
+  legacy column but not the live one (in any period, for the variable/period
+  layout). The audit records the target's pending renames as `input_aliases`.
+  For entity tables .py checks the live input's own entity table; the audit,
+  whose evidence keeps one column list per file, checks across all of them,
+  so a legacy column stored in another entity's table would count as mapped.
+  Where a rename applies, it counts the live input as supplied and the legacy
+  column as neither dropped nor never modelled; where it doesn't, the legacy
+  column is classified like any other. Certification re-reads the renames
+  from the installed target, without loading .py's US model.
+- **The models.** The build model is read with `--with policyengine-us==<build>`,
+  and certification runs with the target policyengine-us installed.
+
+.py writes execution receipts for UK results only, so a US run carries no
+per-run record of the exception. The record is `data_releases.us.regional_certification`
+in the bundle manifest, its evidence included, and the US TRO, which carries
+it separately from the national certification. The national certificate
+itself does not change.
+
 The script fetches and validates the manifest (every artifact must carry a
 revision pin; the certified dataset must be reachable), writes the canonical
 bundle manifest, exact-pins the country model package in that same manifest,

docs/households.md

@@ -122,7 +122,7 @@ set of keys is:
 | `as_of` | Optional information-date cutoff accepted by the pinned artifact. |
 
 Forecast paths, provider objects and fallback policies are not settings. The
-canonical artifact covers 2022–2035. Its receipt distinguishes published
+canonical artifact covers 2022–2036. Its receipt distinguishes published
 national inputs from forecast components and research geography; an unsupported
 year fails instead of being extrapolated by the wrapper. Scenario forecasts
 are conditional research estimates, not agency forecasts or uncertainty bounds.
@@ -158,7 +158,7 @@ separate benefit inputs; they do not override SPM measurement composition.
 
 These changes require the coordinated canonical country/calculator bundle, which
 the packaged production manifest now pins and certifies: `policyengine-us`
-2.2.1, `policyengine-core` 3.33.3, `spm-calculator` 1.0.0, and the US data
+2.2.2, `policyengine-core` 3.33.3, `spm-calculator` 1.1.0, and the US data
 release `populace-us-2024-spm-20260915`. Local-wheel development manifest
 fixtures remain explicitly uncertified. The measurement receipt identifies a
 calculation; it does not certify a population dataset or establish publication

docs/microsim.md

@@ -64,7 +64,7 @@ The same `spm=` selection is accepted by
 `pe.us.managed_microsimulation`; its returned country simulation exposes
 `spm_config` and `spm_provenance()` too. See the [household SPM
 contract](households.md#spm-geography-and-measurement-selection) for exact keys,
-the 2022–2035 artifact horizon and explicit geography errors. No provider or
+the 2022–2036 artifact horizon and explicit geography errors. No provider or
 forecast-file path can replace the bundle's pinned artifact.
 
 For a small local development dataset, explicitly opt out of managed data

pyproject.toml

@@ -41,18 +41,18 @@ graph = [
 ]
 models = [
     "policyengine-core==3.33.3",
-    "policyengine-us==2.2.1",
+    "policyengine-us==2.2.2",
     "policyengine-uk==2.174.1",
-    "spm-calculator==1.0.0",
+    "spm-calculator==1.1.0",
 ]
 uk = [
     "policyengine-core==3.33.3",
     "policyengine-uk==2.174.1",
 ]
 us = [
     "policyengine-core==3.33.3",
-    "policyengine-us==2.2.1",
-    "spm-calculator==1.0.0",
+    "policyengine-us==2.2.2",
+    "spm-calculator==1.1.0",
 ]
 dev = [
     "pytest",
@@ -71,9 +71,9 @@ dev = [
     "hypothesis>=6.100.0",
     "rfc8785>=0.1.4",
     "policyengine-core==3.33.3",
-    "policyengine-us==2.2.1",
+    "policyengine-us==2.2.2",
     "policyengine-uk==2.174.1",
-    "spm-calculator==1.0.0",
+    "spm-calculator==1.1.0",
 ]
 
 [tool.setuptools]

scripts/audit_regional_input_surface.py

@@ -1,15 +1,18 @@
-"""Write a byte-bound audit of regional H5 columns for an installed UK model.
+"""Write a byte-bound audit of regional H5 columns for an installed country model.
 
-Run in an environment with the exact requested policyengine-uk version and
-PyTables installed. Supply the H5 files from the immutable regional manifest;
-the audit verifies their hashes and downloads no microdata. The baseline is the
-build model and core the regional manifest records: the audit reads their
-variables in a throwaway ``uv run --with`` environment, resolved from PyPI as
-of the manifest's build time, so it needs ``uv`` and PyPI. Certification reads
-both models again. Columns the build model defines and the target does not
-refuse; columns neither model defines are recorded as never modelled; and
-target inputs the files do not store that were not build inputs are recorded
-as defaulted.
+Run in an environment with the exact requested target model installed: the
+policyengine-uk or policyengine-us the regional manifest's build model names,
+at ``--model-version``. Entity-table files (UK, and the US ACS-local
+release) are read with PyTables; a US file in PolicyEngine core's
+variable/period layout with h5py. Supply the H5
+files from the immutable regional manifest; the audit verifies their hashes
+and downloads no microdata. The baseline is the build model and core the
+regional manifest records: the audit reads their variables in a throwaway
+``uv run --with`` environment, resolved from PyPI as of the manifest's build
+time, so it needs ``uv`` and PyPI. Certification reads both models again.
+Columns the build model defines and the target does not refuse; columns
+neither model defines are recorded as never modelled; and target inputs the
+files do not store that were not build inputs are recorded as defaulted.
 """
 
 from __future__ import annotations
@@ -57,7 +60,7 @@ def main(argv: list[str] | None = None) -> int:
     args.output.write_text(evidence.model_dump_json(indent=2) + "\n")
     print(
         f"Saved {len(evidence.artifacts)} artifact input audits to {args.output}: "
-        f"policyengine-uk {evidence.model_version} against build model "
+        f"{evidence.model_package} {evidence.model_version} against build model "
         f"{evidence.build_model_version} (core {evidence.build_core_version}); "
         f"{len(evidence.dropped_columns)} dropped, "
         f"{len(evidence.never_modelled_columns)} never-modelled, "

scripts/bundle.py

@@ -238,7 +238,7 @@ def _parser() -> argparse.ArgumentParser:
         "--accept-regional-model-mismatch",
         metavar="REASON",
         help=(
-            "UK only: route regions to a regional release on another model "
+            "UK or US: route regions to a regional release on another model "
             "version, recording this reviewed reason; forwarded to "
             "scripts/certify_data_release.py."
         ),

scripts/certify_data_release.py

@@ -120,7 +120,7 @@ def main(argv=None) -> int:
         "--accept-regional-model-mismatch",
         metavar="REASON",
         help=(
-            "UK only: route regions to a regional release that does not cover "
+            "UK or US: route regions to a regional release that does not cover "
             "the pinned model, recording this reviewed reason in the bundle "
             "instead of refusing. Requires --regional-input-surface-evidence; "
             "dropped columns (defined by the regional build model, not by the "