Version the diagnosis judge's template and drop its false review claim from v2

[MaxGhenis]

The diagnosis judge's prompt header now has two versions. v1 is the header every verdict so far was judged on. v2 drops v1's claim that the reference pipeline "has survived an adversarial review program" and that "the few real bugs found were fixed before this run". The claim is false: release dashboard-data-20261006's exclusion record stops scoring 28 outputs for engine defects (reason_code: reference_engine_defect at 9ce4ade), and the 2026-10-05 reference adversary confirmed four more (reference_audit/2026-10-05-reference-adversary/README.md).

This finishes the WIP commit 695dceb (2026-10-09), rebased onto main. It was held so that release dashboard-data-20261010 (#208, 5a8164a) judged every verdict on one template. That release is live, and every one of its verdicts is on v1.

What it does

• policybench/judge_template.py holds the headers, pinned by sha256. v2 is built from v1 by deleting V1_REVIEW_CLAIM.
• prepare_audit keeps a verdict only on the exact bytes its judge read. It renders the judged case on the version its sidecar records (judge_template_version; a sidecar without the field means v1). A verdict without a sidecar has no record of what its judge read, so it is re-opened, and audit-collect refuses it. That rendering must equal prompt.md byte for byte, and must match the sidecar's prompt_sha256 when there is one. Without prompt.md, only a matching prompt_sha256 keeps the verdict, and the prompt is restored. A sidecar that records verdict_sha256 must also describe the verdict. A kept case's files are not rewritten. Anything else is re-opened: its verdict is dropped, and its sidecar too, unless the sidecar describes another verdict when prepare reads it. Prompts are written as UTF-8 bytes.
• New and re-opened cases render on the version the caller names. render_case_prompt, prepare_audit and audit-prepare --template-version all require it. There is no default.
• Both runners have the judge read a private copy of prompt.md, record its prompt_sha256 and judge_template_version in verdict.meta.json, and publish nothing if prompt.md changed during the run. That is new for the Codex runner. Both remove an invalid verdict.json and any stale sidecar before judging a case, and publish the sidecar before the verdict. With one runner per case and one audit-prepare at a time, the two may share a tree: once both finish, no verdict passes collect, or survives the next prepare, on bytes its judge did not read.
• audit-collect refuses a tree whose verdicts disagree with their prompts on the version (template_version_problems), or whose sidecar prompt_sha256 is not prompt.md's bytes (prompt_hash_problems).
• The three past release drivers name JUDGE_TEMPLATE_VERSION = 1: finish_adds0928.py, finish_gpt61sol.py and finish_haiku55.py. Their releases were judged on v1, so re-running one reproduces its prompts. A new release's driver names v2.

Changes from the WIP

• No default version. The WIP defaulted every renderer to the current version, v2, so its callers switched silently. Now every caller names one.
• The drivers name v1. scripts/finish_haiku55.py reached main after the WIP's base. On the plain cherry-pick it rendered its in-memory audit on v2, re-opened every kept case, and 31 of its tests failed (19 failed, 12 errors; evidence/baseline_haiku.txt). Its tests now seed on v1, as release 20261006's seed was. That includes test_every_seed_prompt_rerenders_except_the_nine_reworded.
• Exact bytes (review round 1):
  • A verdict without prompt.md was kept unconditionally, both before this PR and in the WIP. Now it needs its sidecar's hash.
  • Prompts compared with read_text, which folds \r and CRLF. They now compare as bytes.
  • The Codex runner recorded the live prompt's version. It now records the copy it judged.
• Publication (review round 2):
  • A verdict published after a racing audit-prepare rewrote its case passed audit-collect. Collect now checks the sidecar's hash.
  • Runners moved the verdict before its sidecar, so an interruption could leave a new verdict that read as legacy v1. They now publish the sidecar first.
  • A terminated Codex worker now removes its prompt copy.
• Prepare's sidecar removal (review round 3): a prepare that dropped a truncated verdict also removed the sidecar a runner had just published, stranding the runner's verdict.
  • Prepare now removes a sidecar only when it describes the dropped verdict, or is legacy. This narrows the race but does not close it; round 4's fix below catches what slips through.
  • Runners remove an invalid verdict before judging.
• No sidecar, no verdict (review round 4): removal could still race publication, so any interleaving can leave a runner's verdict without its sidecar. Such a verdict is no longer read as legacy v1: audit-prepare re-opens it and audit-collect refuses it. Every verdict in releases 20260929 and 20260930 has a sidecar.
• Docs: docs/audit.md gains an invariants section and the --template-version 2 pipeline step. v1 is dated through release 20261010.

Invariants

Each holds for every input. They are tested in tests/test_judge_template.py.

1. Every committed prompt re-renders under its recorded template version. Each committed prompt record re-renders from the board of the release that judged it. Release 20261006 judged nothing: it carried release 20260930's audit while rewording nine explanations, which release 20261010 re-opened. The committed prompt records are:

   • release 20260929's 674 prompts (docs/gpt61sol/seed_digest.csv);
   • the 134 cases release 20260930 re-judged (docs/gpt61sol/judge_provenance.json);
   • release 20261006's 674, which is release 20260930's audit carried over (docs/haiku55/seed_digest.csv);
   • the 232 cases release 20261010 re-judged, for 677 in all (docs/haiku55/judge_provenance.json).

   Tests:

   • test_every_committed_prompt_rerenders_on_its_recorded_version is slow and local only, because the audit grounding is not committed. It renders each release's board, read from its release PR's squash commit, on the version each record names. All 674, 674 and 677 sha256s match exactly, and no other case renders.
   • test_the_committed_prompt_records_agree_and_name_a_version runs in CI. Every record resolves to v1, and release 20261006's seed digest equals release 20260929's digest overlaid with release 20260930's re-judged prompts and verdicts.
   • test_prepare_keeps_each_judged_case_on_its_own_template and test_a_verdict_is_kept_on_its_recorded_version_alone (Hypothesis, CI) check that a judged case keeps its files exactly when the case, rendered on its recorded version, is the bytes its judge read: prompt.md, and prompt_sha256 when recorded. The second draws \r text, CRLF copies, a missing prompt and four kinds of sidecar hash.
   • test_prompts_compare_and_keep_exact_bytes, test_a_verdict_without_its_prompt_stands_only_on_its_recorded_hash and test_a_changed_case_without_its_prompt_is_reopened are worked examples.
   • test_collect_refuses_a_verdict_published_on_rewritten_bytes, test_codex_publishes_no_verdict_for_a_prompt_rewritten_while_it_judged and test_an_interrupted_publish_never_leaves_a_verdict_without_its_sidecar (both runners, also from a truncated verdict) cover publication. So do test_prepare_racing_a_publish_never_strands_the_verdict, test_an_audit_prepare_between_a_publishs_two_moves_strands_nothing (end to end on the real Codex runner), test_a_dropped_verdict_takes_only_its_own_sidecar and test_a_verdict_its_sidecar_does_not_describe_is_reopened. Together they show that no verdict reaches audit-collect, or survives audit-prepare, on bytes its judge did not read.
   • The local frozen-tree test in test_finish_gpt61sol.py now prepares releases 20260929 and 20260930 on both versions, byte for byte.

2. v2 differs from v1 only by the dropped clause.

   • test_the_header_diff_is_one_deletion_of_the_clause: the common prefix and suffix bracket exactly V1_REVIEW_CLAIM, and difflib's only edit is that one deletion.
   • test_only_the_header_depends_on_the_version (Hypothesis): for any case, the v2 prompt is the v1 prompt without the clause, at the header's offset.
   • Each header's sha256 is pinned, and v1 equals origin/main's pre-version _PROMPT_HEADER, read from git.

3. Version selection is explicit, never inferred.

   • test_the_renderers_have_no_default_version and test_a_renderer_called_without_a_version_is_refused: the version is keyword-only with no default.
   • test_audit_prepare_requires_a_template_version: without the flag, the CLI exits 2 and writes nothing.
   • test_every_renderer_call_names_its_version: an AST scan of policybench/, scripts/ and reference_audit/ finds 8 calls. Every one names template_version, and none reads it off a prompt with template_version_of.
   • test_only_the_tree_check_reads_a_version_off_a_prompt: template_version_of is called only in template_version_problems, which renders nothing. A version read into a variable elsewhere fails it.
   • test_each_past_release_driver_renders_on_v1_by_name: each driver passes its JUDGE_TEMPLATE_VERSION constant, which is 1.
   • test_a_verdict_is_kept_on_its_recorded_version_alone (Hypothesis): a verdict is kept exactly when prompt.md is the case rendered on the recorded version. A prompt on another version is re-opened, even when the caller names that version.
   • The only fixed mapping is that a sidecar without the field means v1 (UNRECORDED_TEMPLATE_VERSION). That is a constant, never derived from the prompt.

Mutation check

I planted bugs one at a time in a detached worktree, and a test caught each one.

Round 1 (evidence/mutants.txt, evidence/mutants2.txt):

• reading the version off prompt.md;
• keeping a verdict whose prompt matches the caller's version;
• a default version on render_case_prompt, and on prepare_audit;
• v2 deleting one more word;
• a CLI default;
• the Haiku driver naming v2;
• a driver passing a literal 1;
• a one-space edit to the case body. The pinned headers miss this one; the committed-prompt test catches it.

Round 2 (evidence/mutants3.txt, each mutant run against the new tests alone):

• comparing prompt.md as text;
• keeping a promptless verdict that has no hash;
• ignoring the sidecar's hash;
• the Codex runner skipping its live-prompt check;
• a version read off a prompt into a variable.

Round 3 (evidence/mutants4.txt):

• each runner publishing the verdict before its sidecar;
• collect checking no hashes;
• removing the Codex worker's cleanup traps.

Removing only the TERM trap survives, because bash 3.2 still runs the EXIT trap on TERM.

Round 4 (evidence/mutants5.txt):

• prepare removing any sidecar;
• the keep rule ignoring verdict_sha256;
• each runner keeping an invalid verdict;
• both layers off together, caught end to end.

Either layer alone survives the end-to-end test, as intended, because the other layer covers it.

Round 5 (evidence/mutants6.txt): restoring the legacy reading of a verdict without a sidecar. Its new test and the property catch it.

Tests run

At fd55a1b (the PR's commit 80bad56 merged with main at cfd851d), with --basetemp outside the repo (evidence/round5.txt). The head, e23e33c, changes only docs/audit.md and a docstring after that; the template, provenance and audit tests pass there too (evidence/final_head.txt):

• test_judge_template.py: 37 passed, slow tests included.
• test_audit.py: 25 passed.
• test_judge_provenance.py: 18 passed.
• test_finish_haiku55.py: 471 passed, 7 skipped. The four seed tests, test_every_seed_prompt_rerenders_except_the_nine_reworded among them, ran against release 20260930's local audit tree, which equals docs/haiku55/seed_digest.csv.
• test_finish_gpt61sol.py: 213 passed, 3 skipped. Its frozen-tree tests re-prepare releases 20260929 and 20260930's real verdicts, sidecars and prompts on both versions and keep every byte.
• test_finish_adds0928.py: 82 passed.
• test_validate_verdict.py: 5 passed.
• test_stamp_gpt61sol_prompt_bindings.py: 7 passed.
• test_restate_gpt61sol_adjudications.py: 33 passed.
• test_model_response_window.py: 26 passed.
• test_adjudications.py: 23 passed.
• test_freeze_haiku55.py: 368 passed, 5 skipped.
• test_freeze_gpt61sol.py: 155 passed.
• test_build_references_upgrade.py: 83 passed.
• test_reference_adversary.py: 107 passed.
• test_run_audit_claude.py: 100 passed.
• test_held_references.py: 93 passed, and test_tax_table_audit.py: 54 passed. These are main's new files, run because the merge touched cli.py.

Free disk was 41 GB at the start but fell to 25–31 GB as other jobs ran, so I did not run the full suite locally; CI runs it. Evidence is in ~/reviews/policybench-judge-template-versions/evidence/.

Impact

No published number, prompt or verdict changes. Every committed prompt re-renders byte for byte. The next release's driver should name JUDGE_TEMPLATE_VERSION = 2.

axiom: n/a: benchmark audit tooling, no policy encoded

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 11, 2026 3:05pm UTC

[MaxGhenis]

Independent review, round 1 (GPT-6.1 Sol, at b11a959): REQUEST_CHANGES. It found three blocking issues and two non-blocking ones; all five are fixed in 575f257 and daf038b.

1. A verdict without prompt.md was kept unconditionally, as it was before this PR. Now it stands only when its sidecar's prompt_sha256 is the case rendered on its recorded version, and the prompt is restored. A sidecar hash that is not prompt.md's bytes also re-opens the case.
2. run_audit_codex.sh recorded the version of the live prompt.md. Codex now judges a private copy, the sidecar records that copy's prompt_sha256 and version, and nothing is published if prompt.md changed during the run.
3. prepare_audit compared prompts with read_text, which folds \r and CRLF. It now compares and writes bytes, and leaves a kept case's files untouched.
4. A new allowlist test covers a version read into a variable: template_version_of is called only in template_version_problems.
5. The docs now say each committed prompt re-renders from the board of the release that judged it.

Each fix has tests, including Hypothesis properties that draw \r text, CRLF copies, missing prompts and sidecar hashes. A mutant of each fix is caught by the new tests alone. Every affected test file passes at 575f257, and the head adds only a docstring commit. Round 2 of the review is running at daf038b.

🤖 Generated with Claude Code

[MaxGhenis]

Independent review, round 2 (GPT-6.1 Sol, at daf038b): REQUEST_CHANGES. It confirmed round 1's fixes and found one more blocking issue, in the publication window. That is fixed in 0770d99.

• A verdict published after a racing audit-prepare rewrote its case passed audit-collect. Collect now refuses any verdict whose sidecar's prompt_sha256 is not prompt.md's bytes (prompt_hash_problems).
• Both runners now publish verdict.meta.json before verdict.json. An interrupted publish can no longer leave a new verdict without its sidecar, which audit-prepare would read as legacy v1.
• Non-blocking: a terminated Codex worker now removes its prompt copy, via EXIT, TERM and INT traps.

The new tests are the race at collect time, an mv shim that kills each runner's worker between its two moves, and a Codex worker terminated mid-judge. A mutant of each fix is caught. Every affected test file passes at 0770d99. Round 3 of the review is running.

🤖 Generated with Claude Code

[MaxGhenis]

Independent review, round 3 (GPT-6.1 Sol, at 0770d99): REQUEST_CHANGES. It found one more blocking issue. A racing audit-prepare that dropped a truncated verdict also removed the sidecar a runner had just published, so the runner's verdict arrived without its record. This is fixed in 18595d7, from two sides:

• audit-prepare now removes a sidecar only together with the verdict it describes (verdict_sha256), or a legacy sidecar that records none. A verdict is kept only when its sidecar describes it.
• Both runners remove an invalid verdict.json before judging a case.

The review's schedule is now a test, both step by step and end to end on the real Codex runner, where an mv shim runs audit-prepare between the two moves. docs/audit.md now states what holds when a runner and audit-prepare share a tree. Mutants confirm each layer and the pair. Every affected test file passes at 18595d7. Round 4 of the review is running.

🤖 Generated with Claude Code

[MaxGhenis]

Independent review, round 4 (GPT-6.1 Sol, at 18595d7): REQUEST_CHANGES. _drop_verdict reads a sidecar and then unlinks its path, so a runner publishing in between could still lose its sidecar. Every such schedule ends in the same state: a runner's verdict without its sidecar. Fixed in 80bad56 by no longer accepting that state. A verdict with no sidecar has no record of what its judge read, so audit-prepare re-opens it and audit-collect refuses it. A sidecar that lacks only the version field still means v1. Every verdict in releases 20260929 and 20260930 has a sidecar.

The branch now also merges main at cfd851d (#213 and #214, a clean merge), giving head fd55a1b. All 18 affected test files pass there, main's two new ones included. Round 5 of the review is running.

🤖 Generated with Claude Code