[juaristi22]

Takes over #1348 and #1350 as one PR, as Max asked. It carries all 31 of #1348's commits and #1350's one commit, with their authorship kept, so it supersedes both.

What it carries

• Let anyone build, sign and certify a Microcosm UK cut #1348 (Max Ghenis, with the UK hub's rehearsal commits). Its description remains the reference for the design. It adds:
  • the signing-key registry microcosm.data.gate_signing_keys;
  • tools/run_uk_cut.py and tools/uk_cut_inputs.json;
  • tools/modal_uk_cut.py and tools/modal_rebuild_uk_chronicle_artifact.py;
  • tools/certify_uk_cut_in_policyengine_py.py;
  • docs/uk-cut-anyone.md.
• Register a team UK gate signing key (hold: needs María's agreement) #1350, cherry-picked as 252c168: the team key uk-gate-team-2026 (dd31e325…).

What changed since

María's review of #1348 (25 Oct)

1. The accepted-mismatch route certifies with policyengine.py's audited evidence (c550496). Following María's ruling on policyengine.py#699, its scripts/audit_regional_input_surface.py replaces this tool's own comparison.
   • The tool refuses a base without the audit, and downloads every regional microdata H5 at its manifest pin, checked by sha256.
   • In the worktree it installs the target model and core and runs the audit. Evidence that isn't build-relative (schema 2), or that drops a stored column, is refused.
   • It passes --regional-input-surface-evidence to certify-data, and the PR body lists what area runs default.
   • --audited-by and --regional-limitation are now required with a mismatch, and --allow-dropped-columns is gone.
   • policyengine.py commands no longer inherit the caller's UV_PROJECT_ENVIRONMENT.
2. A resumed cut is bound to its signer (810b2ec).
   • Preflight reads attestation.signing_key_sha256 from the reports of finished steps. It refuses any other key and names the signer.
   • The receipt takes its fingerprint from those reports.
   • Once the cut is certified, a signer that has since been retired can still finish it.
3. The one-writer check (c28cd37):
   • matches certify-uk-<id> anywhere in a remote branch's name or an open PR's head, so certify-uk-a133f37a-national is caught;
   • lists up to 1000 open PRs;
   • refuses when a listing can't be read.
4. all end to end: Vahid's run; see Verification.
5. UK_GATE_SIGNING_KEY_ENV is imported, not copied (fc288e1).
   • This covers the runner, the Modal wrapper and the local-release preflight.
   • The Modal wrapper can't import microcosm, so it loads the stdlib-only registry file by path: from the checkout locally, and from a copy the image carries in the container.
6. No machine-specific defaults (97bd378, ee91651).
   • The host-reservation check is opt-in (--reservation-tool with --reservation-caller), and a named tool that is missing fails.
   • The certifier has no load limit unless --max-load1 sets one. Its limits pass as keyword arguments through resource_guard, Shell and model_surface.
   • --pepy-repo is now required.
   • The doc no longer points at one machine's runbook.
7. Credits (61e20cb). Certification commits and PR bodies credit only the people the operator names, through --co-author and --pr-body-footer.
8. UC-floor receipts (ad73a94). A malformed receipt now fails operationally instead of crashing. A Hypothesis test over arbitrary JSON covers it.

Also

• Pins (96a9314). NTS 2002-2024 and LCFS 2024-25 are pinned at María's uploads (revisions 559b69e9 and 11051052), with their archive digests checked. Every hosted archive must now carry both pins. The doc notes that the LCFS zip is a repack of the extracted files.
• Team key (34daa6d). Its custodian is Max Ghenis. held_in names his agent store and says agent sessions on his build Mac may read it. María's key 66bf0e29… stays active.
• main merged (ef82ec8, at 5810fce, policyengine-uk 2.162.3), with no pin or fixture to regenerate. The Chronicle artifact's Hub path now follows chronicle_feed.json (b769292). It reads chronicle-uk/<source commit>/ at main and checks it against the feed's digests, so a feed re-pin no longer breaks the runner.
• Chronicle 272c987 hosted (7a97104). The artifact main pins is now on the Hub (commit 5d28b8f0), uploaded unchanged from the build UK: impute alcohol, tobacco and gambling consumption so policyengine-uk's excise duties stop scoring zero #1359 pinned. The runner's fetch --only chronicle_uk fetched and verified both files from the derived path.
• Orrery (5e1650a). The national step passes --no-publish-orrery, as the hand-run launchers do.
• Logbook (c212517). The release-cut certification's Logbook row now chains on the national build's own row, as the launchers chain it, instead of sharing the operator's predecessor.
• Doc fix (1e54fa8) repairs a doc table that one of these edits garbled.

Since Vahid's round 1 (at 6c6039941)

1. The accepted-mismatch route: already done in c550496, which landed after his read.
2. HMAC limits (9cddb43). The doc's § Signing keys says what the signature does not prove: only a key holder can verify, a retired key can still mint reports, and revocation is local. The Ed25519 follow-up is microcosm#1406.
3. Key recipes (833ea03). generate --keychain-service NAME stores the key through security -i on stdin, reads it back and prints only the fingerprint, so fingerprint.txt is no longer needed. Plain generate refuses to print a key to a terminal. The doc and this description now pass the key to one command at a time instead of exporting it.
4. Key scope (c3fe8f8). The runner gives the key only to spine, national, certify, assemble and publication. The certifier withholds it from every policyengine.py command, since policyengine.py reads no Microcosm key.
5. Modal (89f513c, 9cddb43). Planning is plain Python and sends nothing. modal run attaches credentials only with MICROCOSM_UK_CUT_MODAL_RUN=1 … --run. The doc says what a run puts on Modal: the ephemeral secret, the volume (readable across the workspace), and the key's custody.
6. Gated, not private (9cddb43). The doc now says that policyengine-uk-data-private is gated, with a public file list, and that approving access is what admits a builder.

Tests

On this head, 850 tests pass locally and none fail. They cover:

• the registry;
• the runner and the Modal wrapper (new stub-Modal tests);
• both certify suites;
• the build CLI and the national role;
• the calibration run and release certification;
• the local preflight;
• the gate battery and contract pins.

Lint and format are clean. The full suite runs in CI.

Verification

The last check before merging is the runner's first real run, which Vahid Ahmadi makes from his laptop with his own key.

What it needs:

• HF read access to the input repositories, and write access to policyengine/populace-uk-private and policyengine/populace-uk-staging;
• cover under the UKDS licences for the inputs;
• about 60 GB of free disk;
• AC power.

git fetch origin && git switch --detach origin/claude/uk-cut-anyone-takeover-974500
uv sync --all-packages --locked --extra uk
# Once: store a fresh key in your login keychain (it is never printed) and
# send only the fingerprint this prints, to be registered here.
uv run --no-sync python -m microcosm.data.gate_signing_keys generate --keychain-service microcosm-uk-gate-signing-key
# Then, with HF_TOKEN exported, give the key to one command at a time:
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  uv run --no-sync python -m microcosm.data.gate_signing_keys check
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  uv run --no-sync python tools/run_uk_cut.py all --label verify-<date> \
  --allow-off-main "María Juaristi: verification run before merging this PR"
uv run --no-sync python tools/run_uk_cut.py status --label verify-<date>

Leave out --publish. Report back which steps passed and how long each took, from status and the steps in cut_receipt.json, but no figures. Outputs stay on the laptop.

What's left

• policyengine.py#699. The accepted-mismatch route needs Run the UK test suite and a smoke-rung build in CI #699 merged; it now has María's ruling at f14690ee. Until then a base without the audit is refused before anything is downloaded.
• Vahid's key, which is registered here once he sends its fingerprint.
• Vahid's run. María then merges, with a merge commit.

The Modal wrappers have still never run on Modal; a separate plan replaces them.

Supersedes #1348 and #1350.

axiom: n/a: build infrastructure (no policy encoding)

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 round 2, at 74721a944.

Verdict: the must is fixed, and so are four of the five shoulds. The fifth, the gated repo, is now documented and left to the repo owners. Two new shoulds are small. One licence question is for Vahid and Max, not for this PR. The route still waits on policyengine.py#699 merging, and CI was still running when I looked.

Round 1, item by item

1. must, the accepted-mismatch route: fixed (c550496, 74721a9).
   • Reason, --audited-by and at least one --regional-limitation are required at parse time, and each needs a printable non-space character (tools/certify_uk_cut_in_policyengine_py.py:881, :980-995).
   • The certifier downloads the regional H5s at their manifest pins and checks each sha256. It deletes any earlier evidence file (:1182), runs Run the UK test suite and a smoke-rung build in CI #699's own audit (:1200), and refuses evidence that is not schema 2, is for other model versions, or drops a column (:712-733). Only then does it pass --regional-input-surface-evidence to certify-data (:1204-1207).
   • On our Run the UK test suite and a smoke-rung build in CI #699 finding. On Run the UK test suite and a smoke-rung build in CI #699 I found that certify-data checked the evidence only against itself, so a hand-edited file acted as an override (comment). This route doesn't inherit that, because it makes the evidence itself in the same run from hash-checked files instead of taking a file. Run the UK test suite and a smoke-rung build in CI #699's 1a93f7e70 also says certify-data now re-reads both models. I haven't re-verified that at f0870aa79. One part stays: the stored-column list rests on the auditor's word, bound to the H5 sha256, and a hand run of bundle.py certify-data still takes any evidence file. That belongs to Run the UK test suite and a smoke-rung build in CI #699, not here.
2. should, HMAC limits: done. § Signing keys says verifying needs the secret, a retired key can still mint, and revocation is local (docs/uk-cut-anyone.md:287-302). Ed25519 is Sign UK gate reports with Ed25519 so anyone can verify a cut #1406, which is open.
3. should, the key recipes: done. See the next section. The doc passes the key to one command only (docs/uk-cut-anyone.md:32-35).
4. should, key scope: done.
   • The runner gives the key only to spine, national, certify, assemble, publish-preflight and publish (tools/run_uk_cut.py:69-88, :1338). I checked these names against the real runner.run calls.
   • The certifier withholds the key, VIRTUAL_ENV and UV_PROJECT_ENVIRONMENT from every policyengine.py command (certify_uk_cut_in_policyengine_py.py:430-438, :489).
5. should, Modal: done for modal_uk_cut.py. With no opt-in, _secrets() returns nothing (tools/modal_uk_cut.py:95). Planning is plain Python, and main refuses unless both the opt-in and --run are set (:209). The doc now names the ephemeral secret and the volume's readers (docs/uk-cut-anyone.md:126-136). There's one gap, in the other Modal tool: see new finding B.
6. should, gated not private: the doc is now accurate, but the repo decision is open (docs/uk-cut-anyone.md:159-163). See the next section but one.
7. nit, team key readers. Left as Max's call. Fine.
8. nit, fingerprint.txt. Gone.

Key storage (833ea03)

I read this code and didn't run it. No key was generated, and no keychain was touched.

• argv. The key goes only to security -i on stdin (gate_signing_keys.py:286-287). The read-back command line holds only the account and service (:293), and the account and service names are checked as plain (:270, :281-283). Your test checks that no argv holds the key.
• Environment. generate puts the key into no environment variable.
• Logs and output.
  • Both security calls capture their output, and nothing prints it.
  • On failure, it prints the CalledProcessError text (:313), which holds the argv and exit code but not the stdin input.
  • A read-back mismatch prints only the service name (:298-301).
  • On success, only the fingerprint is printed (:316).
  • Plain generate refuses to print to a terminal (:318).
• Files. No temp file is written.

So the key never appears in argv, in an environment dump, in a log or in a file. One problem:

A. should: -U silently replaces an existing key (gate_signing_keys.py:287). The doc's own rotation steps trigger this. Step 1 uses the fixed service name microcosm-uk-gate-signing-key (docs/uk-cut-anyone.md:271-272), and step 4 says a retired key's cuts "still verify, but only for someone who holds that key" (:278-280). Anyone who rotates by following step 1 overwrites the old key with no backup, so after that nobody holds it. Its cuts then can't be verified, and step 4's promise no longer holds. Running it twice by mistake does the same to an active key. Please drop -U and refuse if the item already exists, or print a clear "already stored" refusal.

One fact for the doc, not a defect: an item that security adds without -T can be read back, with no prompt, by any process running as that user that calls security find-generic-password -w. That is how the doc's recipe reads it. Like Max's agent-secret, the login keychain shields the key from other users, not from the user's own agent sessions.

Gated, not private: for Vahid and Max

These are the facts. Read anonymously, policyengine/policyengine-uk-data-private reports private: false, gated: "manual". Its file names (the FRS, LCFS, NTS, WAS, SPI and ETB zips) are public. The files download only after a person approves an access request. On a gated repo, Hugging Face shows the owners the requester's details, but it doesn't check UKDS registration or licence cover. That check exists only if the approver does it by hand.

Whether that is enough for UKDS End User Licence data is a licence question. Vahid and Max should decide it with whoever holds the UKDS licences, not this PR. Two points to weigh: public file names that show which UKDS studies PolicyEngine holds, and access that depends on a manual approval step with no written check. The doc now describes the setup accurately, so nothing here blocks the PR.

New findings

B. should: modal_rebuild_uk_chronicle_artifact.py still sends HF_TOKEN to Modal in "plan (free)" mode (tools/modal_rebuild_uk_chronicle_artifact.py:17-19, :26-28, :74-80, :133). This is round 1 item 5 again, in the tool 89f513c didn't touch. _secrets() attaches HF_TOKEN whenever it is set. The doc's main recipe has the builder export HF_TOKEN (docs/uk-cut-anyone.md:31). So modal run tools/modal_rebuild_uk_chronicle_artifact.py sends a token with write access to the private repos as soon as it plans. The Chronicle data is public, but the token isn't. Its docstring and the doc (:217) still say "never stored as a Modal secret". Use the same opt-in as modal_uk_cut.py.

C. nit: the opt-in variable alone attaches the secrets, without --run. MICROCOSM_UK_CUT_MODAL_RUN=1 modal run … --label x (no --run) sends both secrets and only then refuses (modal_uk_cut.py:95 and :209). The doc says the variable is what attaches them, so this is just a sharp edge. It would help if the refusal said "credentials were already sent".

D. nit: secrets still reach third-party code in two places.

• The certifier passes HF_TOKEN and HUGGING_FACE_TOKEN, write tokens, to policyengine.py's uv pip install, uv sync and test suite (certify_uk_cut_in_policyengine_py.py:489). The H5 downloads run in the certifier's own process (:625), so the token may only be needed by certify-data. I haven't checked which policyengine.py steps read it.
• The Modal container passes its whole environment, signing key included, to uv sync before the runner scopes it (modal_uk_cut.py:141).

E. nit: the Modal refusal says export … before --run (modal_uk_cut.py:218), which goes against the doc's one-command-only advice.

CI

Run 40404384382 on 74721a944: lint, wheels, select-countries and integration-uk pass. engine-free, engine-us and engine-uk (3.13, 3.14) were still pending when I posted, so I can't say they are green. Your plan to re-run any engine-uk job that times out is fine.

What I ran

• Tests. In a fresh worktree and uv venv, with the key and HF variables unset, I ran the six touched test files: both registry suites, runner, Modal stub and both certify suites. 186 passed. I deselected 8 tests whose names mention generate, keychain or read_back, so that nothing created a key, even a throwaway one. The generate and keychain tests I read but didn't run.
• Reading. I read 833ea03, c3fe8f8 and 89f513c in full, plus the certifier's audit path, both Modal tools and the doc sections cited above.
• Not done. No key generation, no keychain access, no Verification run, no Modal runs and no builds. I didn't re-run Run the UK test suite and a smoke-rung build in CI #699's tests at f0870aa79.

Not approving here. Vahid will post his fingerprint separately.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi, round 2 is addressed; the head is now 7eb54e8cb.

• A, -U overwrote an existing key (c2c9d0a).
  • generate --keychain-service now checks for an existing item first. If one exists, it refuses and leaves it untouched; otherwise it adds without -U.
  • The doc's rotation step stores each new key under a new service name, so a retired key stays held and its cuts stay verifiable.
  • The doc also states your fact: the login keychain shields a key from other users, not from the user's own processes or agent sessions.
  • If you already have an item under microcosm-uk-gate-signing-key, the command refuses rather than replacing it. In that case, pass another --keychain-service name.
• B, the Chronicle rebuild sent HF_TOKEN when planning (cb39f2c).
  • Planning is now plain python tools/modal_rebuild_uk_chronicle_artifact.py, which contacts no Modal service.
  • The token is attached only with MICROCOSM_UK_CHRONICLE_MODAL_UPLOAD=1, a separate opt-in from the cut's, so consenting to one tool doesn't consent to the other. A rebuild without --upload sends no credential.
  • The docstring and the doc no longer say "never stored as a Modal secret".
• C, the opt-in without --run. The refusal now says the credentials were already sent and nothing was built (cb39f2c).
• D, tokens reaching third-party code.
  • The certifier's uv sync, uv lock and uv pip install steps now get no Hugging Face token (7eb54e8). policyengine.py's audit, certify-data, UC floor and tests keep it, since they read the private cut repositories, and policyengine.py's dataset tests need it.
  • In the Modal container, uv sync now runs without HF_TOKEN or the signing key (cb39f2c).
• E, export … before --run. The refusal now asks for the credentials to be given to that one command (cb39f2c).
• The gated repo. Noted as a licence question for you and Max, outside this PR. It stays as it is for now.

Locally, the touched suites pass: both registry suites, runner, the Modal stub for both tools, both certify suites, and the local preflight. CI is running on the new head, and I'll hold further pushes until your fingerprint, so it can finish.

For the verification run, these are in place:

• the inputs, each on the Hub at its pin, including Chronicle 272c987 (runner-fetched and verified today) and the NTS and LCFS zips;
• the branch, pushed;
• the recipe under Verification in the description.

Two things to check on your side first:

• that your HF account is approved for the gated policyengine-uk-data-private and can read microcosm-uk-build-inputs, which run_uk_cut.py fetch will show;
• your fingerprint, which I'll register as uk-gate-vahid-ahmadi-2026 as soon as you post it.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 my signing key is generated and stored in my login keychain, under service microcosm-uk-gate-signing-key, using generate --keychain-service at 7eb54e8cb. Only the fingerprint below has left the keychain:

key_sha256 e9ab40bddffa31e67c3f7c4a5fba2fe3d5f8a71d3200f92ecbc6592b877a7eb1

Please register it as uk-gate-vahid-ahmadi-2026, custodian Vahid Ahmadi, held in his laptop's keychain.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi, your key is registered as uk-gate-vahid-ahmadi-2026 (e9ab40bd…, active, held in your laptop's login keychain), in 928874f. You're clear to start the verification run from the head of this branch.

I checked the inputs on the Hub without downloading anything. Every pinned archive is at its revision, and the Hub's own sha256 for it equals the pin. That covers:

• FRS, SPI, WAS and ETB at a2039519;
• NTS at 559b69e9 and LCFS at 11051052;
• the enhanced FRS at a9e52499;
• Chronicle 272c987.

The HMRC workbook is still served by gov.uk at its pinned size. Access to the gated policyengine-uk-data-private has been granted to you. fetch is the first step, so it will tell you at once if anything is missing.

The recipe is under Verification in the description:

• run_uk_cut.py all --label verify-<date> --allow-off-main "María Juaristi: verification run before merging #1405";
• give the key to that one command;
• leave out --publish.

Please report only which steps passed and how long each took, from status and the receipt's steps, and no figures from the data.

CI restarted on the new head. If an engine-uk job hits the 30-minute limit, I'll re-run it.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi, here is how to run the verification build from your laptop. It is the runner's first real end-to-end run, and the last check before #1405 merges.

What to expect

• What it builds. One Microcosm UK national cut, built from scratch at this branch's head:
  1. it fetches every input by its pin;
  2. preflight;
  3. spine (about 15 min);
  4. national dry run, then the national calibration (about 1 h);
  5. release-cut certification (about 25 min);
  6. assembly;
  7. the publisher's preflight.
• Total: about 2.5–3 h.
• What leaves your laptop. Nothing is published or promoted. Like every cut, the national step stages its bundle to the private policyengine/populace-uk-private and its telemetry to the private policyengine/populace-uk-staging.
• Your laptop needs:
  • 60 GB free disk, which is preflight's floor (the cut itself uses about 30 GB);
  • AC power, with the lid open for the whole run.
• Credentials:
  • your Hugging Face login, with read access to the gated policyengine-uk-data-private (granted) and to microcosm-uk-build-inputs, and write access to populace-uk-private and populace-uk-staging;
  • your signing key in the login keychain (done).
• Outputs go to ~/microcosm-uk-work, outside any repository. They derive from UKDS-licensed data, so keep them there: don't upload, share or commit them.

Steps

1. Check out this branch, clean, and sync it.

git clone https://github.com/PolicyEngine/microcosm.git ~/microcosm-verify && cd ~/microcosm-verify
git fetch origin claude/uk-cut-anyone-takeover-974500
git switch --detach FETCH_HEAD
git log -1 --format=%h        # expect 928874f14 or later
uv sync --all-packages --locked --extra uk

2. Check the key with check. It should print "ok": true and uk-gate-vahid-ahmadi-2026.

MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  uv run --no-sync python -m microcosm.data.gate_signing_keys check

3. Fetch the inputs. This downloads about 9 GB and verifies every file against its pin. Every line should say ok. A 401 or 403 means a repository access is missing.

HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
  uv run --no-sync python tools/run_uk_cut.py fetch --work-dir ~/microcosm-uk-work

4. Run the cut.

• The credentials are given to this one command and not exported. They go in as shell prefixes rather than through env, so they appear in no process list.
• caffeinate stops the Mac sleeping while it runs.
• Leave out --publish.

HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  caffeinate -ims uv run --no-sync python tools/run_uk_cut.py all \
    --label verify-1405 --work-dir ~/microcosm-uk-work \
    --allow-off-main "María Juaristi: verification run before merging #1405"

all starts with preflight. If preflight refuses, it says why, and nothing has been built.

5. If a step fails, fix the cause and run the same command again. It resumes at the failed step without repeating finished ones, and never overwrites a finished output.

• An unfinished spine or national attempt is kept for inspection. Add --retry-spine or --retry-national to move it aside and start that step again.
• If staging hits Hugging Face's commit-rate limit, wait a few minutes and re-run with --retry-national.

6. Report back. When it finishes, it prints the assembled cut's tag and the receipt path. Post here only this:

uv run --no-sync python tools/run_uk_cut.py status --label verify-1405 --work-dir ~/microcosm-uk-work
python3 -c "import json,sys; r=json.load(open(sys.argv[1])); [print(k, s.get('status'), s.get('seconds')) for k, s in r['steps'].items()]" \
  ~/microcosm-uk-work/cuts/verify-1405-$(git rev-parse --short=9 HEAD)/cut_receipt.json

That is which steps passed and how long each took. Please post no figures from the data: no gate values, totals or file contents. If something fails, post the step name and the runner's REFUSED: line only. Don't paste from logs/ or the tools' output, since those can contain figures from the data.

🤖 Generated with Claude Code

[juaristi22]

@MaxGhenis, could you run the verification build for this PR on your build Mac? Vahid's laptop is too loaded to take a 3-hour run. This is the runner's first real end-to-end cut and the last check before #1405 merges. It will also be the first cut signed with the team key, uk-gate-team-2026, which agent sessions on your Mac can read, so it tests that key too.

What to expect

• Where it runs. On your build Mac, not through tools/modal_uk_cut.py. The Modal wrapper builds only from commits on main, and processing UKDS data on Modal still needs the data owners' OK.
• What it builds. One Microcosm UK national cut, from scratch, at this branch's head (17e8147cd, which includes main as of today, policyengine-uk 2.162.8):
  1. it fetches every input by its pin;
  2. preflight;
  3. spine;
  4. national dry run, then the national calibration;
  5. release-cut certification;
  6. assembly;
  7. the publisher's preflight.
• Total: about 2.5–3 h.
• What leaves your Mac. Nothing is published or promoted. Like every cut, the national step stages its bundle to the private populace-uk-private and its telemetry to the private populace-uk-staging.
• The inputs are ready. I checked each against the Hub's own sha256: FRS, SPI, WAS and ETB at a2039519, NTS at 559b69e9, LCFS at 11051052, the enhanced FRS at a9e52499, and Chronicle 272c987. The HMRC workbook is still served by gov.uk at its pinned size.
• Your Mac needs:
  • 60 GB free disk, preflight's floor;
  • AC power, with the lid open for the whole run;
  • a Hugging Face token that reads the gated policyengine-uk-data-private and microcosm-uk-build-inputs, and writes populace-uk-private and populace-uk-staging.
• Outputs go to ~/microcosm-uk-work, outside any repository. They derive from UKDS-licensed data, so keep them there.

Steps

1. Check out this branch, clean, and sync it.

git clone https://github.com/PolicyEngine/microcosm.git ~/microcosm-verify && cd ~/microcosm-verify
git fetch origin claude/uk-cut-anyone-takeover-974500
git switch --detach FETCH_HEAD
git log -1 --format=%h        # expect 17e8147cd or later
uv sync --all-packages --locked --extra uk

2. Check the team key with check. It should print "ok": true and uk-gate-team-2026.

MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(agent-secret get microcosm/uk-gate-signing-key-team maxghenis)" \
  uv run --no-sync python -m microcosm.data.gate_signing_keys check

3. Fetch the inputs. This downloads about 9 GB and verifies each file against its pin; every line should say ok.

HF_TOKEN="$(agent-secret get HUGGING_FACE_TOKEN_MAX maxghenis)" \
  uv run --no-sync python tools/run_uk_cut.py fetch --work-dir ~/microcosm-uk-work

4. Run the cut.

• The credentials go to this one command only, as shell prefixes and not through env, so they appear in no process list.
• Leave out --publish.

HF_TOKEN="$(agent-secret get HUGGING_FACE_TOKEN_MAX maxghenis)" \
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(agent-secret get microcosm/uk-gate-signing-key-team maxghenis)" \
  caffeinate -ims uv run --no-sync python tools/run_uk_cut.py all \
    --label verify-1405 --work-dir ~/microcosm-uk-work \
    --allow-off-main "María Juaristi: verification run before merging #1405"

5. If a step fails, fix the cause and run the same command again. It resumes at the failed step without repeating finished ones.

• --retry-spine or --retry-national moves an unfinished attempt aside and starts that step again.
• A resumed cut accepts only the key that signed its earlier steps, so keep using the team key throughout.

6. Report back here, with only which steps passed and how long each took:

uv run --no-sync python tools/run_uk_cut.py status --label verify-1405 --work-dir ~/microcosm-uk-work
python3 -c "import json,sys; r=json.load(open(sys.argv[1])); [print(k, s.get('status'), s.get('seconds')) for k, s in r['steps'].items()]" \
  ~/microcosm-uk-work/cuts/verify-1405-$(git rev-parse --short=9 HEAD)/cut_receipt.json

Please post no figures from the data. If something fails, post the step name and the runner's REFUSED: line only. Don't paste from logs/ or the tools' output, which can contain figures.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 verification run at 928874f14, label verify-1405, on my laptop with my key (uk-gate-vahid-ahmadi-2026), no --publish. Steps and times only, no figures from the data.

Step	Result	Time (from log start and end)
fetch	passed: 10 of 10 inputs verified against their pins	about 10 min (first fetch, about 9 GB)
preflight	passed	about 23 min (11:28 to 11:51)
spine	done	31 min
national dry run	done	11 min
national	done	49 min
score	passed	(part of the certify window)
certify	refused	14 min, then stopped
assemble, publish preflight	not reached

The refusal:

REFUSED: certify exited 1; see .../cuts/verify-1405-928874f14/logs/certify.log
GateBatteryBlockedError: Gate battery blocked at phase 'terminal'
  - [uk_degenerate_release_surface] household.beer_draught_share: persisted release column is constant ...; populate it with signal, drop it, or record a reviewed exclusion.
  - [uk_degenerate_release_surface] household.cider_draught_share: persisted release column is constant ...; populate it with signal, drop it, or record a reviewed exclusion.

I've left the constant values out. Both columns look like they come from the alcohol inputs in #1381, which is in this branch through the main merge. If so, a cut from main may hit the same gate, not just this PR. cut_receipt.json wasn't written, because the run stopped before assembly. The cut directory is kept, so after a fix I can re-run the same command and it will resume at certify.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 a summary of what the verification run did, to go with the step times above.

Built on my laptop (about 12.5 GB, in ~/microcosm-uk-work/cuts/verify-1405-928874f14/, staying there and not shared):

• the spine (10 GB);
• the national candidate, with its build record and evidence (2.5 GB).

What left the laptop: only what the national step stages for every cut.

• The bundle went to the private policyengine/populace-uk-private, under staged/uk-frs-calibration-attempt-20261027T123319Z-… (about 200 MB).
• Telemetry went to the private policyengine/populace-uk-staging.
• Nothing was published or promoted: no --publish and no release tag. The cut failed certification, so it can't be used as a release. Delete the staged attempt if you'd rather not keep it.

What it shows:

• The runner works end to end up to certification on a second builder's machine, with that builder's own registered key.
• The terminal gate then caught a real problem: household.beer_draught_share and household.cider_draught_share are constant across every household.
• The cut directory is kept, so once that's fixed I can re-run the same command and it resumes at certify.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 status at 17e8147cd, after my verification run at 928874f14.

The head only adds a merge of main (27 Oct) on top of 928874f14, and its diff against main is the same 32 files. So I have no new code findings. Two points on getting to a passing verification:

1. The certify blocker is fixed on main, but not on this branch. The refusal was the constant beer_draught_share and cider_draught_share. Sign the draught shares' degenerate exclusions and build on policyengine-uk #2695 #1411 signed their exclusions (d444eb5b9) and merged on 28 Oct, after this branch last merged main. Without it, a cut from this head would refuse at certify the same way. The branch merges cleanly with today's main (git merge-tree reports no conflicts, and GitHub shows it as clean).
2. A correction to what I said on the 27th: re-running the same command will not resume at certify. CutPaths.cut (tools/run_uk_cut.py:591) names the cut directory <label>-<commit[:9]>, and preflight needs a clean checkout. Once this branch merges main, its head changes, so the runner starts a fresh cut in a new directory. On my run that took about 2 hours 20 minutes and 12.5 GB to reach certify. Resuming the old directory at 928874f14 wouldn't help either, because that checkout doesn't have Sign the draught shares' degenerate exclusions and build on policyengine-uk #2695 #1411's exclusions.

Verdict: my earlier verdict stands. This needs a merge of main, then one fresh verification cut, on my laptop or on Max's build Mac.

What I ran: git log, merge-base and merge-tree against main, and read tools/run_uk_cut.py:579-591 at the head. Not run: a new cut.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi, the branch now has today's main, and it's ready for the fresh verification cut you asked for. The head is 7dad2eb5e.

What changed since your run at 928874f14

• main is merged (f47542076, main at f85c46970). That brings in Sign the draught shares' degenerate exclusions and build on policyengine-uk #2695 #1411's signed exclusions for the two constant draught shares that refused your certify step. The merge had no conflicts.

• Two new spine inputs (7dad2eb5e). Since the branch last merged main, the spine reads two more licensed tabs. Without them, a cut from the merged head would have refused at the spine:

  • the NTS vehicle tab (vehicle_eul_2002-2024.tab), for lcfs_consumption (UK #1095 A/B fixes: the SPI age band (uk-data#628), households' HRAD from Table 8a receipts, and the SPI pay floor (uk-data#629) #1426). It is already in the hosted NTS zip, so the runner now extracts one more member;
  • the APS April 2024-March 2025 person tab (apsp_a24m25_eul_pwta22.tab, UKDS SN 9408), for aps_occupation (Lock policyengine-uk 2.174.1 and follow what its releases change #1511, UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490). It is now hosted in policyengine-uk-data-private as aps_2024_25.zip @ ebebd1e0, a repack of the extracted files like the LCFS one.

  The runner fetches both and passes --nts-vehicle-tab and --aps-tab. A new test fails if a stage ever pins a licensed tab that the runner doesn't fetch and pass.

• The Chronicle feed moved from 272c987 to 64753803 (UK occupation facts for chronicle#407: APS employment by SOC 2020 group (Nomis APS218) and ASHE Table 2 pay by two-digit SOC chronicle#411). Its consumer artifact is now hosted under chronicle-uk/64753803…/ in microcosm-uk-build-inputs. Both files match the feed pin.

• One CI fix from the merge. main added a test that pins the modules the transport gates' binding identity reaches. This PR's signing-key check in the gate battery imports microcosm.data.gate_signing_keys, which brings in the microcosm.data package. Those modules are now listed in the test, as UK: carry the FRS occupation (SOC 2020 major group), and redraw it on the SPI channel #1490 did for microcosm.fit. The diff against main is now 33 files.

What I ran:

• The runner's, certifier's and signing-key tests, plus the tests of the files both sides changed: 855 passed on the merge.
• test_run_uk_cut.py and test_transport_evidence_kernels.py on the head: all passed.
• run_uk_cut.py fetch --only aps_2024_25 --only nts_2002_2024 from the Hub: both ok, with all six NTS tabs and the APS tab verified against their pins.

Not run: a cut. I also didn't run preflight's feed-gated surface check: on my Mac, the 64753803 feed needed more memory than the machine has, so I stopped it. Your preflight runs it before anything is built.

Before you start

• APS access. The APS is a new UKDS study for these builds, so your UKDS account needs SN 9408 under the End User Licence, as with the FRS, NTS and LCFS. Your Hub access to policyengine-uk-data-private already covers the download.
• Run it before 16 November. That's when the earliest reviewed register entry expires (local_target_fit_reviewed_exclusions.json), and after it preflight refuses.
• Disk. The new head starts a fresh cut directory (verify-1405-7dad2eb5e), so preflight's 60 GB floor applies again. Your old verify-1405-928874f14 directory (about 12.5 GB) can't be resumed. Delete it if you need the space.
• Same key (uk-gate-vahid-ahmadi-2026). The lock is now policyengine-uk 2.174.1.

Steps

The same as last time, at the new head:

cd ~/microcosm-verify
git fetch origin claude/uk-cut-anyone-takeover-974500
git switch --detach FETCH_HEAD
git log -1 --format=%h        # expect 7dad2eb5e
uv sync --all-packages --locked --extra uk

fetch keeps every input that still matches its pin. It downloads the new Chronicle artifact (about 8 GB) and the APS zip (about 54 MB), and extracts the vehicle tab from the NTS zip.

HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
  uv run --no-sync python tools/run_uk_cut.py fetch --work-dir ~/microcosm-uk-work

HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  caffeinate -ims uv run --no-sync python tools/run_uk_cut.py all \
    --label verify-1405 --work-dir ~/microcosm-uk-work \
    --allow-off-main "María Juaristi: verification run before merging #1405"

Leave out --publish. Your last run reached certify in about 2 h 20 min, so expect about 3 h in all.

Report back the same way: the step names, pass or fail and their times from status and cut_receipt.json. If a step refuses, post only the runner's REFUSED: line. As before, please post no figures from the data.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 verification run at 7dad2eb5e, label verify-1405, on my laptop with my key (uk-gate-vahid-ahmadi-2026), no --publish. It stopped at preflight, so nothing was built.

Step	Result	Time
fetch	ok: NTS (with the vehicle tab), APS 2024-25 and the 64753803 Chronicle artifact fetched, the rest present at their pins	a few minutes, before 10:39
all: preflight	refused: FAIL Hugging Face reachable: AttributeError, then "Preflight failed; nothing was built."	10:39 to 11:24

Every other preflight check passed, including the feed-gated target surface check that your Mac couldn't finish, the signing key (e9ab40bd…), policyengine-uk 2.174.1, disk and AC power.

Cause: the merge of main removed two methods the runner still calls. hub_checks (tools/run_uk_cut.py:1112-1113) calls storage.head_revision() and storage.credential_can_write() on HuggingFaceDatasetStorage. #1460 (334bf0e, 2 November) removed both when the build's emitter took over Hub writes and credential checks, and test_staging_v2.py:1118-1128 now asserts they are gone. I reproduced it outside the runner with the same token: AttributeError: 'HuggingFaceDatasetStorage' object has no attribute 'head_revision'. list_repo_refs on the publish repo works, so this is not access.

preflight calls hub_checks() unconditionally (:1797), so every cut from this head refuses at preflight, with or without --publish. No test runs hub_checks against the real storage class, so the suite cannot catch it.

Suggested fix: take the write-scope check from wherever #1460 moved it in the emitter (or call HfApi directly for repo_info and the token's scope), and add one test that runs hub_checks with a stub HfApi and the real HuggingFaceDatasetStorage, so a later rename fails in CI rather than at preflight. It would also help if the failure printed the message, not only the exception type.

The fetched inputs are kept in ~/microcosm-uk-work, so once this is fixed the rerun skips the downloads. The 16 November register expiry still applies, so there's about a week. Tell me when the head moves and I'll start it again.

🤖 Generated with Claude Code

[juaristi22]

@vahid-ahmadi, thanks. You're right, and it was the merge: my 6 November merge brought in #1460, and none of the tests I ran called the real storage class. The fix is in, and the branch is up to date with main again. The head is 4addf2ae9.

The fix (40be79bfe)

• Preflight checks write access the way the build now does. It starts a short-lived emitter with only its Hugging Face module, as tools/stage_uk_rowwise_candidate.py does, and asks the build's own write_access_problem about populace-uk-private and populace-uk-staging. So these now refuse at preflight, with the build's message and its remedy, instead of an hour into the national step:

  • a missing emitter;
  • an emitter older than minimum_emitter_version.json (0.13.0 today);
  • an emitter without its Hugging Face module;
  • a read-only token.

  An unread write scope still passes with a warning, as it does in the build.

• The emitter starts without the signing key. It inherits the runner's environment, and preflight signs nothing.

• Other refusals. HF_HUB_OFFLINE or TRANSFORMERS_OFFLINE refuse before any emitter starts. An unexpected error now prints its message, not only its type.

• Tests. Following your suggestion, the tests run hub_checks with the real write_access_problem. They patch start_bundle_emitter in telemetry_emitter, where the build defines it, so a rename there fails in CI. They cover the passing case, a read-only token, an outdated emitter, offline mode and an unexpected error. At 7dad2eb5e they fail.

• The doc lists the emitter among what a builder needs.

Bringing it up to date

• db210f492 merges 18 commits of main. The one conflict was the list in test_transport_evidence_kernels.py, where both sides had added entries; it keeps both.
• 4addf2ae9 merges the one commit main gained after that.
• The diff against main is still 33 files. No input pin and no feed pin changed since 7dad2eb5e, so everything you fetched still matches. fetch re-verifies it and downloads nothing.

What I ran

• test_run_uk_cut.py, the binding-identity and column-kernel tests, test_contract.py, test_gate_signing_keys.py, test_gate_battery_contract_pins.py and ci_test_plan.py verify: all passed.
• The fixed hub_checks against the real Hub from my machine, with emitter 0.13.0. With a write token, both repositories passed in about 3 s. With a read-only token, both were refused with the build's message.

Not run: a cut, or a whole preflight. The surface check needs more memory than my Mac has.

Before you rerun

• Install the emitter. Your 27 October run predates Hand staged dataset bundles to the emitter with stage_dataset_bundle #1460, so you may not have it. microcosm-emitter version should report 0.13.0 or later:

  uv tool install microcosm-emitter
  microcosm-emitter version

• HF_TOKEN must be able to write both repositories. The token from your token file staged your 27 October run, so it should pass.
• Cut directory. The new head starts a fresh one, verify-1405-4addf2ae9. Your 7dad2eb5e run stopped at preflight, so its directory holds only logs.
• Deadline. It still has to start before 16 November.

Steps

The same as before, at the new head:

cd ~/microcosm-verify
git fetch origin claude/uk-cut-anyone-takeover-974500
git switch --detach FETCH_HEAD
git log -1 --format=%h        # expect 4addf2ae9
uv sync --all-packages --locked --extra uk
HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
  uv run --no-sync python tools/run_uk_cut.py fetch --work-dir ~/microcosm-uk-work

HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
MICROCOSM_UK_TERMINAL_GATE_SIGNING_KEY="$(security find-generic-password -a "$USER" -s microcosm-uk-gate-signing-key -w)" \
  caffeinate -ims uv run --no-sync python tools/run_uk_cut.py all \
    --label verify-1405 --work-dir ~/microcosm-uk-work \
    --allow-off-main "María Juaristi: verification run before merging #1405"

Report back as before: the step names, pass or fail and their times. If a step refuses, post only its REFUSED: or FAIL line, and no figures from the data.

🤖 Generated with Claude Code

[vahid-ahmadi]

@juaristi22 rerun at 4addf2ae9, label verify-1405, on my laptop with my key (uk-gate-vahid-ahmadi-2026, e9ab40bd…), no --publish. Every step passed and the cut assembled. Nothing was published or promoted.

Step	Result	Time (UTC)
preflight	ok on every check, including both write-access checks through the emitter (your 40be79b fix), the target surface check, the register expiries (earliest 16 November) and AC power	12:44 to 13:25
spine	done	13:25 to 13:48 (23 min)
national dry run	done	13:48 to 14:07 (19 min)
national	done	14:07 to 14:46 (39 min)
score	passed	within national
certify	certified shippable	14:46 to 15:17 (30 min)
assemble	done, tag microcosm-uk-2024-25-national-20261109T140748Z-a1edef5a	15:17 (16 s)
publish preflight	{"valid": true, "published": false}	15:17

Stage 1 was skipped (no --previous-h5). Total about 2 h 33 min from start.

The receipt is at cut_receipt.json in my cut directory. The promote and policyengine.py certify steps were not run. Those are separate calls for whoever ships the cut.

The 16 November register expiry still applies, so a cut from this head needs to ship within the week or have that entry renewed.

🤖 Generated with Claude Code