Add the reference adversary and record its first run (evidence only; no score changes)

[MaxGhenis]

Adds the reference adversary, a pass whose only job is to attack PolicyBench's US references, and records its first run on the frozen 20260612 run (46 models, 1,928 scored cells). Evidence and tooling only: no reference, exclusion record or score changes. The full report is reference_audit/2026-10-05-reference-adversary/README.md.

What it found

• Flagged and run. The consensus trigger flagged 61 cells: a wrong cluster of at least 15 models, or 3 of the top 5. Another audit owns 9 of them (Audit Louisiana's 2026 standard deduction behind two scored references #192 Louisiana, Audit optional employer pass-through in the payroll tax references; propose excluding four outputs #194 payroll, Correct scenario_031's Medicaid annotations to the engine's California disregard #197 scenario_031), so 52 were run.
• Verdicts. The blind, law-first, two-stage adversary returned 45 reference_holds, 6 reference_wrong and 1 definition_mismatch.
• Search exposure and re-judge. The code review found that WebSearch results listing PolicyEngine and GitHub pages reached the judge. WebFetch had a deny rule, but WebSearch did not, and the run's audit read only the queries.
  • The audit now reads every search result. Run over the original transcripts, it flags 13 of 104, in 10 cases.
  • Those 10 were re-judged, with the prompt asking for blocked_domains on every search. None of the 20 new transcripts is flagged, and every verdict is unchanged.
  • One stage 1 moved. scenario_013 SNAP (Arizona) can no longer date the 185% → 200% eligibility change behind the $240 reference. Blind, it finds $288; stage 2 still holds $240. This is listed as an open item (verification/search_exposure.md).
• Independent verification of the 7 (verification/independent/):

  • CONFIRMED engine defects, all still on policyengine-us main 2.29.11:

    Cell	Defect	Reference	Law
    AZ 018	Standard deduction not indexed under A.R.S. 43-1041(H)	$1,146.05	$1,137.30
    OH 025	R.C. 5747.01(A)(10)(b) premiums dropped	$1,921.57	$1,916.60
    CO 043	Sales tax refund paid without the FY 2025-26 surplus condition	$19	$0
    NY 082	Pre-2026 child care credit instead of Tax Law 606(c-2)	$667	$1,187.61

  • AMBIGUOUS: PA 123. The definitions don't say whether a dependent's own required return counts. The same question reaches 4 cells: PA 123 and MO 093, state and federal.

  • REFUTED: NC 026 child1/child2. SPA NC-23-0009's 42 CFR 435.218 election covers insured children, so both references stand.

• Leaderboard impact (verification/leaderboard_impact.json, scored with policybench analyze). Excluding the 8 cells raises every model's exact rate by 0.40–0.65 pp and swaps claude-sonnet-5.5 and gpt-6-luna at ranks 4 and 5.
• Engine-side checks.
  • Definition conformance: the material mismatches are Audit optional employer pass-through in the payroll tax references; propose excluding four outputs #194's payroll components and the household scope above.
  • Publication sources: 1,194 2026 values, read by 1,542 scored cells, carry a may-not-have-been-published flag. Among them is Louisiana's CPI-computed deduction, the finding behind Audit Louisiana's 2026 standard deduction behind two scored references #192.
• Judge cost (reported by the CLI as API-price estimates; the calls billed subscription lanes):
  • the run: 104 calls, $91.22, 5.25 judge-hours;
  • the re-judge: 20 calls, $24.82, 4.70 judge-hours.

Rulings (recorded in proposed_changes.json status)

• d1022 (Max, 2026-10-06): exclude the 8 cells in the next release after dashboard-data-20261006, then regenerate the four defect cells once fixed policyengine-us versions land. The fixes are open policyengine-us PRs #9928 (AZ), #9946 (CO) and #9948 (NY). For Ohio, #9925 fixes a double count, and the premiums fix has no PR yet.
• d994 (Max, 2026-10-06): exclude Louisiana 051 and 077, and keep the published-amounts convention.

This PR installs neither ruling; the next release does.

Review fixes since the first review

The independent code review requested changes (blinding: search results). The report review found count and wording errors. Both are addressed:

• Blinding. claude_transcript_audit rejects a search whose result lists a blocked URL or names PolicyEngine or PolicyBench. The source rule asks for blocked_domains on every search. Docs and both runner headers now say the Codex runner cannot see search results.
• Collect. adversary-collect:
  • refuses a repeated judge label;
  • queues every case with an inconsistent verdict, whatever its class;
  • requires a verdict sidecar bound to the current stage 1;
  • exits non-zero on missing verdicts unless --allow-missing is given.
• Consensus. A model whose own answer is within the tolerance counts as exact and leaves the wrong cluster. consensus_flags.json and the prototype file reproduce byte-identically under the new rule.
• Credentials.
  • Codex runner: an allowlisted environment, with the Claude runner's locale set, and it refuses a Codex home holding an AGENTS.md.
  • check_login refuses a token lane that declares the desktop account.
• Scripts.
  • leaderboard_impact.py refuses a --scratch inside the repo and an unknown --only variant.
  • build_proposals.py records each record's ruling (decision: d1022) and keeps the pass's own expectations out of the rulings. It also states the Colorado basis as the verifier did; proposed_changes.json reproduces byte-identically.
• Report.
  • The derivations are 35 amount and 17 eligibility outputs, not 42 and 10.
  • The rank-shift text is corrected: no 5/10% ranks exist.
  • The d1022 bullets now keep to the ruling.
  • The AZ filing-status values are fixed.
• Evidence. runs/collected/ reproduces byte-identically under the new collect. The runner logs the README cites were git-ignored and are now committed.
• Diagnosis judge template. It is unchanged here (738d6a6). Dropping its stale "bugs were fixed before this run" sentence waits for a versioned judge template in a separate PR, so seed verdicts keep re-rendering byte-identically.

Invariants

The tests check each of these, with Hypothesis property tests where marked:

• Consensus trigger (tests/test_consensus.py):
  • its flags match a brute-force recount, which applies the exact-member rule (property);
  • no member of a wrong cluster is within the tolerance of an amount reference (property);
  • raising any threshold never adds a flag (property);
  • prediction order does not matter (property);
  • the report is deterministic and leaves the payload untouched (property).
• Adversary (tests/test_reference_adversary.py):
  • a stage-1 prompt never contains the derivation (property);
  • apply_adversary_flags changes nothing but the flags it sets (property);
  • the adjudication queue holds every non-holding verdict and every inconsistent one;
  • a verdict counts only with a sidecar bound to the current stage 1;
  • a changed stage-1 prompt invalidates everything downstream;
  • the audit rejects a search result that reaches a blocked source, and passes a clean one.
• Definition conformance: the component tree is deterministic and independent of graph order, and every node's signed path multiplies correctly (properties). On the real reference system, all 1,928 scored references reproduce within 1e-3, and 8,128 node-sum checks have 0 failures.
• Publication sources: citation dates are never after the text's stated years, and classification is total and deterministic (properties).
• Runner (tests/test_reference_adversary_runner.py, with fake CLIs):
  • blinding;
  • transcript rejection;
  • credential refusal;
  • the Codex runner's allowlisted environment and its AGENTS.md refusal;
  • a rolling pool that never exceeds its parallel cap.

Tests

reference_audit/2026-10-05-reference-adversary/verification/pytest_final.txt holds the final run:

• Files: the five adversary test files plus tests/test_audit.py.
• Result: 376 passed, at 10b3d16. The later commit a29cb3c changes only that file and the README.
• ruff check . and ruff format --check . pass with the locked ruff 0.15.2.

The previously failing test_codex_runner_passes_only_allowlisted_variables was a test bug. With no LANG, the fake codex, a Python script, sets LC_CTYPE itself under PEP 538, so the runner was right.

axiom: n/a: PolicyBench evidence and tooling; the engine defects are fixed in policyengine-us under d1022.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 9, 2026 3:29pm UTC

[MaxGhenis]

Removed the judge-template edit from this PR (commit 738d6a6). Dropping the stale sentence changes all 674 seed prompts and breaks byte-identical verdict carry-over at the next model addition (flagged by the release session building 20261006 and Haiku 5.5). The sentence's removal moves to a separate PR that versions the judge template. Review fixes left uncommitted by a stopped lane are parked on reference-adversary-review-wip: one runner test fails and lint is pending. A queued lane finishes them before this PR merges.

[MaxGhenis]

Merged at the reviewed head 395d81f (squash 4db91b5).

• Review. The round-5 independent review (GPT-6.1 Sol) approved this head: ~/reviews/policybench-reference-adversary-2026-10/pr200-r5/review_r5_out.md. Round 4 approved 8f968e8; its minor note and its nit were fixed in 4a62b1e.
• CI. gh pr checks exited 0 on this head: test, app, lint, paper, and the Vercel checks.
• Score impact. None. The PR adds tooling and evidence only, and changes no published reference, exclusion or score.
• Follow-up. The review's one remaining minor is open: reference_audit/2026-10-05-reference-adversary/scripts/leaderboard_impact.py regenerates its impact evidence from the working tree's run bundle, not from the pinned 8b4c0ca inputs. The committed results are accurate. It is tracked as a follow-up task.
• Rulings. d1022 governs the exclusions this evidence supports. The Haiku 5.5 release session will take this PR's proposed_changes.json when it rebases.