Harden the GPT-6.1 Sol driver: pin kept sidecars, reference explanations, re-opened flags and the export receipt

[MaxGhenis]

Lands units 4 to 7 of the third review's fix brief for the GPT-6.1 Sol release driver (scripts/finish_gpt61sol.py, scripts/freeze_gpt61sol.py). Units 1 to 3 shipped with release dashboard-data-20260930 (#187). That release has shipped, so nothing here re-freezes or re-publishes it. These gates harden the driver for the follow-up re-judge release (judge-isolation-20260930b, whose driver imports this one) and any later single-model release.

The review's point was that the freeze rebuilt the payload from the staged bundle, but bound the bundle only by a list of hashes stored beside it. Each unit makes the freeze re-derive one more input, or pins it outside the stage.

Units

Each unit is one commit, and its new tests fail on the commit before it.

Unit	Commit	What it closes
4 [review-fix kept-sidecars]	d66e6e01	The seed digest (docs/gpt61sol/seed_digest.csv) now pins each of the 674 judged seed cases' verdict.meta.json sha256. validate_verdicts, which judge, triage, export and the freeze run, refuses a kept sidecar that is not the seed's. Before, a kept sidecar's judge_model_requested could be rewritten (with verdict_sha256 kept and the receipt re-hashed) and the snapshot manifest's judge tally would change.
5 [review-fix reference-explanations]	03dbdfac	The receipt must bind us_case_reference_explanations.csv. Export and the freeze require it to be release 20260929's (git at BASE_COMMIT) byte for byte. build_payload, shared by export and the freeze's rebuild, refuses a bundle that lacks any file export reads (EXPORT_INPUTS). So neither can fall back to the working directory's annotations/<RUN>/ any more.
6 [review-fix reference-flag]	ac44dfb3	verify_restatements checks every re-opened entry's top-level judge_reference_suspect and judge_reference_suspect_source against its current bound verdict and the 2026-09-22 wave (read at BASE_COMMIT). It uses the restate script's own reference_flag, the one definition: the flag is raised when the verdict or the wave raised it, and names a source exactly when the wave alone did.
7 [review-fix receipt-pin]	f6640978	After writing release-ready.json, export writes docs/gpt61sol/release_receipt.json (tag, payload sha256, receipt sha256). The freeze reads it with git show HEAD:, requires the working-tree copy to match, and refuses unless it names the stage's tag, payload and receipt. A stage edited after export, with its receipt re-hashed, now needs a visible commit.
pin	ceb84ee1	The pin export wrote for the real stage (below).
review	8a3108c6	From the first independent review: validate_verdicts refuses a verdict.json or sidecar in any case directory cases.jsonl does not list as judged. The snapshot manifest's judge tally counts every case directory holding a verdict, so such a file, added after export, changed the tally past every gate (reproduced on ceb84ee1). The pin's documented scope is narrowed to files the receipt binds.

Invariants these gates enforce, and their tests

• Kept seed cases: verdict, sidecar and prompt are byte-identical to the 20260929 seed, and the committed digest's bytes are pinned by SEED_DIGEST_SHA256 (e80ec95e…). Tests: test_a_kept_sidecar_must_be_the_seeds, test_bind_seed_*, test_the_freeze_refuses_a_verdict_edited_after_export[edited_kept_sidecar-*].
• Reference explanations: always equal to release 20260929's. Export, the freeze rebuild and the freeze never read a file the bundle lacks. Tests: test_the_freeze_refuses_edited_reference_explanations (entry removed, and entry re-hashed), test_the_rebuild_refuses_a_bundle_file_export_reads_and_the_receipt_omits, test_export_refuses_*.
• Re-opened entries: every one carries the reference flag the restate script writes from its current verdict. The test table covers each combination of verdict flag, wave flag, entry flag and source, including the brief's case (flag raised with a source while the verdict says false). There is also a Hypothesis differential test (test_the_freeze_accepts_exactly_the_reference_flag_restatements_write). Through any sequence of re-judges, the gate accepts exactly what the restate script writes, and refuses the same entry with its flag or source flipped.
• The freeze proceeds only on a stage whose tag, payload and receipt the committed pin names. Tests: test_the_freeze_refuses_a_stage_edited_after_its_pin_was_committed, and test_the_freeze_reads_the_receipt_pin_as_committed_at_head (real git: a pin never committed, then committed, then rewritten and not committed, then committed but naming another payload).

docs/gpt61sol/design.md's invariants table has a row or an amendment for each gate.

The real stage

The archived stage was not written. The freeze requires a stage inside the checkout, and the 20260930b driver checks the archive's receipt file by file. So the run used an APFS copy-on-write clone of it. Its stage.json already bound the three-field seed (an earlier partial attempt at unit 4 re-bound it), and all 540 kept sidecars match.

• --step export reproduced release-ready.json byte for byte, as the archive holds it (e8e0a51f…). It also reproduced the payload as published (d1cae7456cf91ab6fa04644a4d5d570e359522386a7c52f9645d5923bfc11258, the live pointer's). GPT-6.1 Sol: exact 90.595091%, rank 8 of 46, n = 1,928, as released.
• Freeze --dry-run (at ceb84ee1, and again at 8a3108c6): Validated local release inputs: dashboard-data-20260930, 46 models, d1cae745…; references unchanged; 0 adjudications added; 394 wording amendments. The working tree was unchanged.
• All 54 restated entries in the stage pass the new flag check. One of them, scenario_022's state income tax, carries the restate script's source wording in place of 20260929's, and the gate accepts either wording.

No published number or file changes.

Tests

The suite was run serially at 8a3108c6: test_finish_gpt61sol.py 244, test_freeze_gpt61sol.py 203, test_restate_gpt61sol_adjudications.py 34, test_run_audit_claude.py 98 and test_judge_provenance.py 10 passed, and ruff is clean. The local-only tests ran against the clone.

Independent review

The first review (an Opus 5.5 subfleet lane with no shell, verify/round3_review.md) requested changes. Responses:

1. Stray verdicts feed the judge tally: reproduced and fixed in 8a3108c6.
2. Two sentences overclaimed the pin's scope: narrowed.
3. A pin commit shows that the receipt changed, not what changed in the stage: accepted as a limit of the design, and stated in the design note.
4. The gate accepts FLAG_SOURCE_EARLIER_RUN where 20260929's entry had its own wording: kept, because the real stage's scenario_022 state income tax entry carries it. The flag and whether a source is named stay exact.
5. A run-record error (export does not write stage.json): corrected.

A second review, with a shell, is running.

Follow-ups (not in this PR)

• An extra us_*_annotations.csv in a stage bundle would be read by export and bound by the receipt, but re-derived by no gate. This needs confirming before any fix.
• A re-opened entry left exactly as 20260929 wrote it still names the old judge and day. The stage has none.
• The 20260930b re-judge driver (judge-isolation-20260930b) calls none of these gates yet. It needs to adopt them when it rebases onto main.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 4, 2026 11:09pm UTC

[MaxGhenis]

Heads-up: #197 edits us_case_reference_explanations.csv, which this PR's new committed-copy test pins.

#197 corrects one reference explanation, scenario_031 head_medicaid_eligible. The old text said countable income deducts a Medicare Part B premium; the engine applies California's $230 monthly disregard instead. It also corrects that output's case note and 43 row annotations. All changes are wording only, recorded in reference_audit/2026-10-05-medicaid-031-annotations/rewrites.json, and they change no score. The frozen run dir is untouched.

test_the_committed_reference_explanations_are_20260929s calls verify_reference_explanations(driver.ANNOTATIONS, "committed"). That compares the working tree's file with the BASE_COMMIT blob, so it fails as soon as both PRs are on main, whichever lands second.

The driver gates (export, freeze rebuild, freeze) check the stage's copy and are unaffected. Only the committed-copy test reads HEAD's working tree. Two ways to keep its intent ("the 20260930 freeze wrote 20260929's explanations") without blocking later corrections:

1. Check the release commit. Compare git show 8b4c0ca1:annotations/<RUN>/us_case_reference_explanations.csv (release 20260930 as merged) with the BASE_COMMIT blob, not the working tree.
2. Allow the ledger. Compare the working tree with the BASE_COMMIT blob after applying the listed rewrites (reference_audit/*/rewrites.json, file us_case_reference_explanations.csv).

If #190 merges first, I'll make the change in #197. If #197 merges first, this PR needs it on rebase.

Separately, the 20260930b re-judge driver builds from release 20260930's committed annotation bytes. Its freeze refuses a working tree that differs from them, and its gate says the payload differs from the base only in its five cases. Both need to accept #197's ledger once #197 is on main. I've told the session that owns 20260930b.

[MaxGhenis]

Release dashboard-data-20261006 (PR #202, 9ce4ade8) merged without this PR; it was out of that release's scope. This PR stays open on its own track, rebased on main as needed.