Repository navigation
Promotion for the constant-id national line: pointer and certified-loader extension #823
Description
#806 / #812 delivered the UK national line's inspect lane: assemble a certified cut, publish with --no-latest under an immutable per-cut tag (microcosm-uk-2024-25-national-<YYYYMMDDTHHMMSSZ>-<uuid8>), and adjudicate it on the dashboard via the release list / ?release=. Promotion — moving latest.json — is structurally refused there (publish_release rejects a pointer move for any tag that is not the release id itself), because the current pointer and loader contracts cannot express it. This issue is the design and implementation that lifts that refusal.
Why promotion cannot work today
The 2026-08-27 ruling makes the national release id constant across cuts; run identity lives in the per-cut tags and the Logbook chain. Two contracts still assume id == tag:
- The pointer cannot name a cut.
latest_pointer_payloadcarries only{schema_version, release_id, updated_at, paths}(release.py). A promoted pointer could say the current release ismicrocosm-uk-2024-25-nationalbut not which cut that means. - The certified loader follows the id as a tag.
loader.pyfetchesrelease_manifest.jsonatrevision=pointer.release_id— an immutable tag named exactly the release id, which the national line never mints (and must not: minted once at cut 1 it would pin cut 1 forever) — and separately hard-asserts the default artifact'srevision == release_id, which per-cut revisions fail by design.
The dashboard itself would survive a pointer move (it reads releases/<id>/ paths off main, which always hold the current cut); every certified consumer would break.
Design decisions this issue owns
- Pointer shape: add a
tag/revisionfield naming the current cut (bumpsLATEST_POINTER_SCHEMA_VERSION; the reader refuses unknown versions, so microcosm-data consumers, the dashboard, and the pe certification path upgrade in lockstep) — versus teaching the loader to read the manifest from main and trust its per-cut revision + sha pins (no schema bump, different immutability trust model). - Repo pointer semantics:
latest.jsonis repo-global, andpolicyengine/populace-uk-privatecurrently serves the June 2023 release to certified consumers. Promoting the national line re-points what "current" means for the whole repo: one shared pointer, per-line pointers, or separate repos is an adjudication, not a refactor. - Registry entry for the national line (the registry today has only
(uk, 2023) → populace_uk_2023.h5), so pe can load what the pointer names. - Lift the interim refusal: replace
publish_release's per-cut-tag pointer refusal with whatever the new contract permits, keeping the fail-closed default for tags the pointer cannot express.
Riding along
- The constant-named staging branch
release-staging/microcosm-uk-2024-25-nationalstrands when tag creation 409s mid-publish and blocks the retry until deleted by hand; a per-attempt staging branch name removes the recurrence.
Acceptance
- A promoted national cut resolves end-to-end: pointer → current cut → certified loader fetches the manifest and artifacts at an immutable ref and verifies their digests.
- Certified consumers of the 2023 line keep working through the transition, per the repo-semantics ruling.
- The inspect lane is unchanged:
--no-latestpublishes still work identically, and promotion of a tag the pointer cannot express still refuses.
Out of scope
- The
uk_target_fitadjudication line (UK WAS stage: split private pension wealth out of corporate_wealth (uk-data#452 M2) #750 et al.) — a shippable certification is a precondition for having anything to promote. - Dashboard feature work; it already reads current cuts from main.
Activity
juaristi22 commentedon Sep 18, 2026
Status check on 2026-09-18 (main at 8c44daa): still live, and it now gates more than the national line.
Unchanged since filing.
latest_pointer_payloadstill carries{schema_version, release_id, updated_at, paths}with nothing naming a cut;loader.pystill fetches the manifest atrevision=pointer.release_idand refuses a default artifact whoserevision != release_id;publish_releasestill refuses to movelatest.jsonfor any tag that is not the release id; the staging branch is stillrelease-staging/<release_id>, so the ride-along strand recurs. No change has touched these contracts inmicrocosm-datasince 30 August.Now also the dense line. #762's release path publishes
microcosm-uk-2024-25-densethe same way: constant release id, per-cut tag,--no-latestmandatory and enforced (docs/uk-dense-release-assembly-runbook-762.md§5). A dense cut is reachable by its tag and by the registry key("uk", 2025, "dense")only, so promotion of the full build's dense line waits on the same pointer and loader extension. The exact-count line (#898) has no permitted release identifier yet; this issue is a precondition it meets later, not what blocks it today.Not affected. The staged-dataset lane from #946 never reads or writes
latest.json, the pointer or the certified loader: a run's bundle lives understaged/<run_id>/and is fetched by run id and commit sha. Inspection of any run, releasable or not, works without this issue.The repo-pointer adjudication is more pressing than when written.
policyengine/populace-uk-privatenow holds the June 2023 release that certified consumers follow through the repo-globallatest.json, the dense line's per-cut tags, and (from #946) the staged bundles. Promoting any newer line re-points what "current" means for the whole repo, so the ruling this issue names (one shared pointer, per-line pointers, or separate repos) decides the shape of the pointer fix and should come first.MaxGhenis commentedon Sep 18, 2026
Ruling on the repo-pointer question this issue names, so the pointer fix has a shape to build to.
Pointer semantics: per-line pointers, in the one repo.
latest.jsoninpolicyengine/populace-uk-privatestays exactly where it is, naming the June 2023 release, and is not touched by the national or dense lines. Each promotable line gets its own pointer file.Pointer shape: the line pointer names the cut. It carries the cut's immutable tag, and the certified loader fetches the manifest at that tag. The loader does not read manifests from
main.Why, from the code on
main:loader.py,latest_release(spec.hf_repo)) and then requires the manifest's default artifact to equal the registry filename. One repo-global pointer can therefore serve one registry entry. The live pointer namespopulace-uk-2023-dd68c73-4aa4b14-20260619T023711Z; the moment it moves,load("uk", 2023)raises on every install, old and new. That fails this issue's own acceptance line about 2023 consumers.mainwould make certified consumers follow the newest unreviewed cut. The national release id is constant, and every--no-latestpublish that is not--tag-onlywritesreleases/<id>/release_manifest.jsontomain, so each inspect cut overwrites the same path. The sha pins do not help: they pin the artifact to whatever that manifest declares.LATEST_POINTER_SCHEMA_VERSIONis a fleet-wide event, not a UK one. The reader checks exact equality and the writer is shared, so the next US publish would emit a pointer every older install refuses. Five readers followlatest.jsontoday: microcosm's loader, bottom-50-tax-analysis, the calibration dashboard'slatest-artifact.ts, microcosm.institute, and calibration-diagnostics' Python evaluation harness, which also pinsschema_versionto 1. None of them ever sees a new pointer file, so nothing upgrades in lockstep.latest-evidence.json(Evidence-tier publishing: ship the best available artifact at all times, with its failures recorded in the manifest (dense first) #506) is a second pointer file with its own payload, its own reader, and a guard against cross-reading.What the implementation needs, so it is not rediscovered in review:
DatasetSpec.pointer_pathdefaulting tolatest.json. Forlatest.jsonpayloads the revision defaults to the release id, so that path stays byte-identical.pointer.revisionand asserts the default artifact's revision against it. Still a hard assertion, against the cut tag.prepare_releaseis a literal of the two existing paths, and without the third a release manifest could declare a root artifact at the line-pointer path and move the pointer as an ordinary upload.latest_releaserefuses arevisionkey outright, and the line reader requires both its discriminator and arevision. Otherwise a hand-editedlatest.jsoncarrying a cut tag would send every certified consumer to an unpromoted cut.("uk", 2024, "national"): the artifact ismicrocosm_uk_2024.h5and the release id ismicrocosm-uk-2024-25-national. Adding a variant does not disturbresolve("uk").publish_release's refusal from "no pointer may name a per-cut tag" to "latest.jsonmay not; a line pointer may", keeping the fail-closed default.The dense line is outside this ruling: its role forbids it from moving any pointer, and
("uk", 2025, "dense")does not resolve through the certified path today. Separate repos were the alternative I rejected: they multiply tokens and access control and buy nothing per-line pointers do not.What would change this: a certified consumer that cannot take a line key without a breaking change. policyengine.py is not one; its certification path pins an explicit repo, revision and path.
juaristi22 commentedon Sep 18, 2026
Thanks Max, building to that shape. Five things to settle before I write the implementation plan, each with the answer I would propose so you can just say yes or correct it.
1. Registry key and filenames for the 2024-25 lines. The raw data is published by fiscal year, so the dataset name is
microcosm-uk-2024-25and the national artifactmicrocosm_uk_2024_25.h5, matching the constant idmicrocosm-uk-2024-25-national.DatasetSpec.yearstill needs one integer: you keyed 2024, the existing dense entry is keyed 2025 (("uk", 2025, "dense")), and the H5's own time period is 2025, the calibration year. Proposal: key every 2024-25 line on 2025 (soload("uk", 2025)reads as the 2025 policy year, as the dense entry already does) and re-key and rename the dense entry to match (microcosm_uk_2024_25_dense.h5), so the two lines never disagree.2. The national line as the default variant, not a
nationalvariant. We are moving to 2025 as the modelled year by default, soresolve("uk")should land on the new line.latest_yearpicks the highest registered year for the default variant, so a("uk", 2025)default entry with its own pointer file does that on its own, while("uk", 2023)keepslatest.jsonuntouched exactly as you ruled. Proposal: amend item 5 to("uk", 2025)default withpointer_pathlatest-national.json(or whatever name you prefer), the 2023 entry unchanged and retired from "default" only by the year ordering.3. A line pointer for the local-area line, and narrowing the role's prohibition. You wrote the dense line out of the ruling because its role may move no pointer. The line we need people to consume is the exact-count one: the full-pool dense build is an intermediate we L0-penalise to exact-K files calibrated to all national and subnational targets, and those are the local-area dataset. Proposal: a
non_default_local_arealine may move its own line pointer and neverlatest.json; the certified loader, for that role, selects the singlemicrodataartifact the local-area contract already guarantees (itsdefault_datasetsis empty by contract); the dense full-pool cut stays tag-only or staged, without a pointer.4. The local-area line's identity. This is #898's naming decision and better taken now than rediscovered there. The contract already reserves
populace-uk-<year>-<tier>-k<N>and binds the count fields to the id. Proposal: one constant line id per approved count (microcosm-uk-2024-25-local-k55000, per-cut tags as on the national line), each with its own pointer, and the registry variantlocalnaming the approved default count; the dense line's constant id and assembler stay for inspection only. The alternative is a count-free line id with the count in the manifest; I prefer the count in the id because a pointer move can then never silently change the file size a consumer gets.5. The dashboard's UK default. With
latest.jsonfrozen on June 2023,?country=ukkeeps showing 2023 as current unless the dashboard reads the national line pointer. Proposal: the UK registration follows the line pointer (a small change inlatest-artifact.ts::loadPointerReleaseIdbeside theproduction_release_idpath), or gains a pinned production release id for UK once a cut is reviewed; your call which.Everything else in your ruling I am building to as written: per-line pointers in the one repo, the pointer naming the cut, no schema bump, the writer-side reservation,
latest_releaserefusing arevisionkey, and the narrowedpublish_releaserefusal. The build-side consolidation (the national dataset built by the rowwise driver without cloning, on national targets, and the standalone national command retired) is orthogonal and will come to you as a PR.Add per-line release pointers for the UK dataset lines (#823)
Register the UK 2024-25 national line as the 2025 default and follow …
Give the UK national assembler its promotion command and pin its data…
Score the UK national candidate on the surface both artifacts can mat…
juaristi22 commentedon Sep 22, 2026
Evaluation rule on the UK national line (microcosm#967, 2026-09-22): rule 1 (#578) is decided on the surface both artifacts can materialize. A target whose measure the incumbent cannot compute is pruned from BOTH arms only under a signed, in-force entry of the reviewed incumbent-unresolvable register (
uk/incumbent_unresolvable_measures.json); an unlisted measure refuses the evaluation. The receipt names every pruned row, its measure and the entry it stands on, and the release-cut certifier re-checks the pruned measures against the committed register. At v21 the six measures the enhanced FRS never carried remove 120 of 644 targets (UC administrative family 93, two-child limit 15, ONS household type 10, CGT asset type 2); the six entries are signed on the register as of 2026-09-22 (a8aa8684on #967), in force to 2026-12-22.Update (2026-09-22, after the rebase over #965): the signing commit on the branch is
2f97e087; the register ispackages/microcosm-build/src/microcosm/build/uk/incumbent_unresolvable_measures.json.Merge pull request #965 from PolicyEngine/uk-823-national-line-throug…
Score the UK national candidate on the surface both artifacts can mat…
Declare the UK rowwise driver's release role and move the national do…
Re-pin the childcare gate test to the family_equal doctrine rule (Pol…
Build the UK national line through the rowwise driver's national role (…
Add the seam-versus-national-role parity twin (PolicyEngine#823)
Retire the UK national calibration seam command (PolicyEngine#823)
Record the PolicyEngine#823 consolidation receipts: the national role…
Add per-line release pointers for the UK dataset lines (#823)
Register the UK 2024-25 national line as the 2025 default and follow …
Give the UK national assembler its promotion command and pin its data…
MaxGhenis commentedon Sep 24, 2026
Yes to all five, with one caveat on 4.
microcosm_uk_2024_25_dense.h5, soload("uk", 2025)means the same thing on every line.("uk", 2025)default with its ownlatest-national.jsonpointer. 2023 stays untouched, and the default moves only because 2025 is the higher year, solatest.jsonnever moves.latest.json. Its certified loader selects the single microdata artifact, and the full-pool dense cut stays tag-only or staged. People consume the exact-K file; the dense pool is an intermediate. That matches the principle that only the L0 budget should differ across sizes.microcosm-uk-2024-25-local-k55000), so a pointer move can never silently change the file size a consumer gets. This approves the naming scheme, not 55k as the default count. Your 9/11 report has most constituencies under the effective-sample-size floor at 55k, so the default count is still a separate call.latest-artifact.tschange with Anthony's diagnostics unification (Unify calibration diagnostics schema and publication #1007).Go ahead and land #966 on this basis.
Merge pull request #966 from PolicyEngine/uk-823-uk-publication-lines