[juaristi22]

Part of #553 (the infrastructure half; the certification PR will close it)

The infrastructure half of #553. The UK default stays the enhanced FRS (1.58.1 on main) for now, so manifest.json and both TRO sidecars are unchanged. This PR adds what a later certification PR needs to switch the default to a Microcosm national release. That works with a Microcosm local-area release, or, if the national release is ready first, with the enhanced FRS serving constituency and local-authority runs.

Based on main at 4773da7f, after #574, #605, #617 and #649. Commits 4–6 follow Anthony's review of 20 Oct, and commit 7 stops #574's automation from promoting a uk-data pair over a Microcosm certificate.

Commits

1. Package UK area lists and certify UK regional releases. Everything below except the next two items.
2. Name builds by cut tag in qualification and check a regional release's core. Adapts this PR to Qualify UK model and data candidates automatically before promotion #574; see "Fit with Qualify UK model and data candidates automatically before promotion #574's qualification" below.
3. Say which bundle entries lack regional_release_kind. A docs correction: the US entry routes to ACS-local again since Restore ACS-local US regions on the microcosm#1263 rebuild #649, but was certified before the field existed.
4. Share one model-basis decision and derive the regional release kinds. The primary and regional checks take the basis, its warning and the refusal reason from one helper. The kinds come from the RegionalReleaseKind literal.
5. Type the regional merge with its artifact and country types. A CertifiedCountry literal, DataReleaseArtifact parameters and typed regional artifacts.
6. Refuse dropping and re-routing regions in one run; test cut tags. --drop-regional-release with --regional-manifest-uri is refused, and the Zenodo and candidate-resolution paths have cut-tag tests.
7. Compare qualification candidates only within main's data line. See "Fit with Qualify UK model and data candidates automatically before promotion #574's qualification" below.

Packaged area lists

• Lists: src/policyengine/data/uk_geography/ ships constituencies_2024.csv and local_authorities_2023.csv. Their sources and hashes are in SOURCES.md.
  • Constituencies: 650 on the 2024 boundaries, including Northern Ireland, copied from policyengine-uk-data.
  • Local authorities: 361 on the April 2023 frame, with ONS names and sim-api's hex layout. This adds Ards and North Down (N09000011), and the four 2023 unitaries get their own hexes instead of a shared (0, 0).
• Regions: constituency and local-authority regions read these lists. On main the opt-in build_uk_region_registry(include_constituencies=True, include_local_authorities=True) returns no areas: the GCS helper it imports no longer exists in policyengine-core 3.33.0, and the loader swallows the ImportError.
• Outputs: compute_uk_constituency_impacts and compute_uk_local_authority_impacts label areas from the packaged lists. Add disclosure control and uncertainty for UK geography outputs #605 deprecated both in favour of compute_uk_geography_disclosure, but they still run. An explicit CSV path still wins. They no longer search local folders or download from GCS, because the local search included the old download cache and would prefer a stale 360-row file.
• Matrix lookups unchanged: the lookup CSVs paired with the legacy weight matrices (local_authorities_2021.csv) are untouched. A matrix indexes areas by that file's row order, and sim-api's fallback depends on it.

UK regional certification (certify-data --regional-manifest-uri)

#552's regional path was US-only. It now covers both UK cases:

• Ship together: a Microcosm non_default_local_area release routes region_datasets.constituency and .local_authority to its H5. This records regional_release_kind: local_area.

• National first: --regional-release-kind area_coded_national takes a certified policyengine-uk-data release, such as the enhanced FRS manifest. It carries:

  • that release's national dataset, with area codes;
  • its weights artifacts: the constituency and local-authority matrices and local_geography_weights.

  All of them keep their own repo and revision, so sim-api's same-release matrix binding still holds. The constituency and LA region types route to the enhanced FRS, and the run records regional_release_kind: area_coded_national.

• Either way:

  • Both area types are routed, which satisfies sim-api's uk_area_regions_routed.
  • The regional release must serve the pinned model, and the pinned core when --core-version is given, by its own build or claims.
  • Every carried artifact is HEAD-checked. Microcosm UK's vendored artifacts are HEAD-checked too, as US Populace's already were.

• Guard: re-certifying a bundle that routes regions, without --regional-manifest-uri, is refused instead of silently dropping the routing. --drop-regional-release removes it on purpose, and cannot be combined with --regional-manifest-uri. This applies to the US as well. Since Restore ACS-local US regions on the microcosm#1263 rebuild #649 the US entry routes states and districts to ACS-local again, so it applies to US re-certification today, as the runbook already asks.

Cut tag as build identity

Microcosm keeps build.build_id equal to the line name on every cut and records the cut in build.cut_tag. Certification and the import-time compatibility check both use the cut tag when there is one, through release_build_id. It feeds version, data_build_id and the changelog fragment name, so two cuts of one line no longer look identical. Releases without a cut tag, such as the enhanced FRS, are unchanged.

#682 moves Microcosm to one folder per build, whose manifests name the release and build and carry no cut_tag. It keeps manifests published before that layout behaving as they do now, so this PR reads cut_tag for them.

Fit with #574's qualification (commits 2 and 7)

• Build ids: Qualify UK model and data candidates automatically before promotion #574's runtime check for a qualified exact pair compared the bundle's data_build_id with the release manifest's build.build_id. With commit 1 the certificate names a Microcosm cut by its cut tag. Without this commit, a qualified cut would have been refused at import, and another cut of the same line, which shares the line name, would have matched. The runtime check, the UK candidate pair's data build id and the Zenodo description now use release_build_id. A test covers both cases.
• Core: with --core-version, the regional release must also cover the pinned core by its own build or claims. A --qualified-pair-id names the primary release only, so it does not vouch for the regional one.
• Merge note: git's automatic merge had moved Qualify UK model and data candidates automatically before promotion #574's core and qualified-pair logic into the regional helper, where its variables don't exist. validate_release_manifest keeps Qualify UK model and data candidates automatically before promotion #574's logic. Since commit 4 it shares the whole built-with and publisher-claim decision, warning and refusal with the regional check (_release_model_basis).
• Data line (commit 7): Qualify UK model and data candidates automatically before promotion #574 orders pairs by (model, data release, core) version, across data producers. With a Microcosm certificate on main (microcosm-data 0.1.0), every policyengine-uk release paired with a uk-data release (1.58.x) compared as newer. The workflow would qualify it, and promotion would ready a PR that switches the default back. Versions are now compared only within main's data line (data package and repository). A pair from another line gets the action other_line unless forced, and promotion neither readies nor closes it. Pairs on main's line behave as before.

UK region routing

• Constituency and local-authority regions take dataset_path=resolve_region_dataset_path("uk", …) and keep their row filters. That is None today, so behaviour is unchanged.
• build_uk_region_registry() adds the area regions by default only when regional_release_kind == "local_area". Explicit True/False still wins.

Universal Credit coverage floor (#491)

scripts/check_uk_uc_floor.py checks two Great Britain measures, each within 10% of its benchmark:

• benefit units with Universal Credit, against DWP's households on UC (6.7m in 2025);
• UC spend, against the OBR's March 2026 EFO Table 4.9 (£79.3bn in 2025-26).

These are the figures policyengine-uk-data calibrates to. Sources are in data/qualification/uk_uc_floor.json. The receipt holds weighted national totals only. On the enhanced FRS 1.58.0 at 2025 it passes: benefit units at 0.925 of DWP's figure and spend at 0.979 of the OBR's.

Checks

• Tests:
  • New: test_uk_area_lists.py, test_uk_uc_floor.py, UK merge and end-to-end certification tests (both modes, the model and core compatibility refusals, the re-certification guard, cut-tag identity, the sim-api predicate), UK routing tests, and runtime cut-tag tests (including a qualified pair on a cut).
  • Updated: three US certification tests now use a temporary bundle instead of the repo's manifest, and the label-lookup tests now expect the packaged lists.
  • Since the review: model-basis agreement tests, the flag-combination refusal, cut-tag tests for Zenodo and candidate resolution, and a cross-line qualification and promotion test.
  • Local, at commit 7: the touched and neighbouring modules pass (1173). CI runs the full suite.
• Bundle: bundle.py check --include-tros passes, and src/policyengine/data/bundle/ is unchanged.
• Live dry run (14 Oct, scratch copy, nothing committed): the published Microcosm national cut …5c6b3f68 was certified national-first against the enhanced FRS 1.58.0 manifest.
  • It is refused at pe-uk 2.100.0 (the cut's build version) on the enhanced FRS's own claim, as it should be.
  • Merging the live manifests routes both area types to enhanced_frs_2024_25.h5, with the matrices at policyengine-uk-data-private@1.58.0, and names the cut tag as the build.
  • All 12 dataset references and the carried artifacts answer HEAD requests.

Follow-up (certification PR, not this one)

• Certify a Microcosm national cut built on the pinned pe-uk, using one of the two modes above.
• Keep policyengine-uk upgrades automated with a Microcosm default. With commit 7, the automation skips uk-data pairs while main certifies Microcosm, so it stops proposing upgrades. To keep them, it must qualify each new policyengine-uk against the certified Microcosm builds and pass their regional release to certify-data.
• Certify Microcosm builds published side by side in their own release folders #682: certify Microcosm builds from their per-build folders.
• Add dataset_overlays.uk entries for the enhanced-FRS names no longer certified.
• Update the 14 tests pinned to the enhanced FRS, plus the docs and examples.
• Attach the Qualify UK model and data candidates automatically before promotion #574 qualification receipt and the UC floor receipt. Where data checks like the floor should live (here, in Qualify UK model and data candidates automatically before promotion #574, or in Microcosm's own gates) is for a separate issue that consolidates them with the data pipelines.
• Coordinate with sim-api#728.

🤖 Generated with Claude Code

[anth-volk]

model_compatibility_basis repeats the built-with and publisher-claim decision in validate_release_manifest, and copies its publisher-claim warning text word for word, so the two could drift apart. Could both use one helper that returns the basis (or None) and the warning, with validate_release_manifest adding its qualified-pair handling around it?

The two regional release kinds are also written out in four places: RegionalReleaseKind, REGIONAL_RELEASE_KINDS, and the argparse choices in scripts/bundle.py and scripts/certify_data_release.py. typing.get_args(RegionalReleaseKind) could supply the other three.

[juaristi22]

Done in 247b90d.

• One decision. validate_release_manifest and model_compatibility_basis both take the built-with or publisher-claim basis, its warning and the refusal reason from one helper, _release_model_basis. The primary path adds its core and qualified-pair handling around it. Each caller completes the refusal with its own remedy, because only the primary path can be covered by a qualification. A new test checks that both paths give the same warning and the same refusal reason.
• One list of kinds. REGIONAL_RELEASE_KINDS is now get_args(RegionalReleaseKind), and scripts/certify_data_release.py takes its choices from it. scripts/bundle.py now forwards the flag with no list of its own. Importing policyengine there would load the country models (about 26 s) on every bundle.py command, and certify_data_release.py checks the forwarded value.

[anth-volk]

A few of the new signatures could be typed more tightly:

• artifact here has no type; it is a DataReleaseArtifact.
• regional_artifacts in PopulaceDataProducerCertificationStrategy.certify is a bare dict; it could be dict[str, DataReleaseArtifact].
• The new merge functions take country: str, but only "us" and "uk" are valid. A country Literal would let mypy catch a wrong value before _local_area_region_types raises at run time.

[juaristi22]

Done in 9d10eb7.

• _add_regional_artifact takes a DataReleaseArtifact.
• The strategy keeps regional_artifacts: dict[str, DataReleaseArtifact].
• The new merge helpers and the three strategies take CertifiedCountry = Literal["us", "uk"]. certify_data_release narrows its country: str once, with the same "Unknown country" error as before.

mypy reports nothing new for certification.py.

[anth-volk]

Two gaps in the tests:

• Passing --drop-regional-release together with --regional-manifest-uri keeps the regional routing without any message, because the drop flag is only read when no regional URI is given. Should that combination be refused?
• The cut_tag handling in zenodo.deposition_metadata and uk_qualification.resolve_candidate has no test where cut_tag is set. The certification and runtime compatibility paths do have one.

[juaristi22]

Both done in 8eba4d3.

• Flag combination. --drop-regional-release with --regional-manifest-uri is now refused before anything is fetched ("cannot be combined"), and a test covers it. The help text and the certification skill say so too.
• cut_tag tests. New tests set build.cut_tag and check that zenodo.deposition_metadata and uk_qualification.resolve_candidate name the build by it.

This PR keeps reading cut_tag, because #682 asks for manifests published before the new layout to keep behaving as they do now.

[juaristi22]

@MaxGhenis, could you review 521ab47? It changes #574's resolve and promotion in uk_qualification.py.

• Problem. Pairs were ordered by (model, data release, core) across data producers. With a Microcosm certificate on main (microcosm-data 0.1.0), every policyengine-uk release paired with uk-data 1.58.x compared as newer. It would have been qualified, and promotion would have readied a PR that switches the default back.
• Change. A candidate's data line is its data package and repository. A pair from another line than main's certificate gets the new action other_line, unless forced. The workflow's later jobs run only on qualify, so these runs end cleanly. Both promotion paths (requalification_target and plan_promotion) leave such pairs out: they are neither readied nor closed. Pairs on main's line behave as before.
• Not in this change. While main certifies Microcosm, the automation therefore stops proposing policyengine-uk upgrades. Qualifying new releases against the certified Microcosm builds is listed as a follow-up in the PR body.