Pin the reference adversary's leaderboard impact to the pass's inputs

[MaxGhenis]

Summary

This fixes the one open finding from #200's round-5 review. reference_audit/2026-10-05-reference-adversary/scripts/leaderboard_impact.py read the working tree's run. The pass actually ran on release dashboard-data-20260930's run, as 8b4c0ca1 committed it. So regenerating the impact evidence after #202 silently scored #202's payload and exclusions. The README's Reproduce step restored only the payload and annotations, not the other scoring inputs.

The PR changes no evidence and no published number. No verification/ file, proposed_changes.json, reference, exclusion or score changes. The diff touches the impact script, the audit README and one new test file.

The bug, reproduced

I ran the unmodified script from a clean checkout of main. Main's working tree holds #202's payload b1da3eae… and exclusions 92741dfd….

• The script exited 0 and rewrote 19 of the 23 committed verification/leaderboard_impact* files.
• scored_outputs.published went from 1928 to 1920, the leader's exact rate from 95.0035 to 95.8279, and the top-5 order changed.
• Its own check that the unchanged copy reproduces the published payload passed, because Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's payload and exclusions agree with each other.

The fix

leaderboard_impact.py

• git_input() writes git show <commit>:<path> to scratch, and stops (SystemExit) unless the bytes match the pinned sha256.
• pass_inputs() stages:
  • all seven run files from 8b4c0ca1, including the payload 1e029aaa…;
  • proposed_changes.json from 4db91b5f (Add the reference adversary and record its first run (evidence only; no score changes) #200), sha256 3a6e5920….
• main() stages these before anything else. The variant copies, the regeneration's line check and the published-reproduction comparison all read only the staged copy. Any mismatch stops the script before it scores or writes anything.
• A new --out-dir writes the evidence elsewhere (default unchanged).
• The scoring code is still the checkout's, as the docstring says. The published-reproduction check and the slow test catch drift there.

README

• An Inputs table lists every pinned input with its sha256 and the scripts that read it, read from the code: the seven run files, the explanations CSV and the proposals.
• It also says which scripts check the pins.
• The Reproduce section now restores the whole run directory and the run's annotations directory from 8b4c0ca1 for the scripts that still read the working tree. I ran that round trip in a clean checkout of main: every file matched its pin after the restore, and the tree was clean after the put-back.

tests/test_reference_adversary_impact.py (new):

• The pins cover every file the script reads. The staged bytes match them, and the payload is the README's 46 models × 1,928 scored cells.
• A differential check: the payload pin equals definition_conformance.py's and publication_sources.py's.
• A property test (Hypothesis): flipping, truncating or extending any pinned run file is refused and nothing is written, while the unchanged bytes are accepted.
• Refusals: Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's payload, a wrong proposals hash and an unknown commit. main() refuses before analyze is ever called.
• A tampered checkout: a git clone --shared --no-checkout whose working-tree run and proposals are junk still scores the pinned bytes and writes all 23 file names. Each exclude variant changes only the exclusion record, and each regenerate variant changes only the reference CSV and its sidecar.
• @slow: a real full regeneration reproduces all 23 committed files byte for byte. It takes about four minutes on a loaded machine, so CI deselects it.

Invariants

• Pinned inputs. Every byte the script scores or compares against equals the pinned bytes, whatever the working tree holds. Covered by the tampered-checkout test and the property test.
• Refusal before use. A mismatched input stops the script before any scoring or output write. Covered by the refusal tests and the property test.
• Reproduction. A full regeneration equals the committed evidence byte for byte. Covered by the slow test.
• Variant isolation. The published copy is byte-identical to the pins, and each variant changes only the files its records touch. Covered by the tampered-checkout test.

Mutation check: I put each working-tree read back, disabled the sha256 check, copied the proposals from the working tree, and skipped the stale-directory cleanup. Each of the 7 mutants fails at least one fast test.

Tests

Run locally at head 95aaa8fb with the CI environment (Python 3.12, uv sync --locked --extra dev, policyengine-us 2.15.17), one file at a time on a loaded machine:

Run	Result
tests/test_reference_adversary_impact.py -m "not slow"	8 passed
tests/test_consensus.py	57 passed
tests/test_reference_adversary.py	107 passed
tests/test_reference_adversary_runner.py	27 passed
tests/test_definition_conformance.py	64 passed
tests/test_publication_sources.py	111 passed
tests/test_audit.py	25 passed
tests/test_reference_adversary_impact.py -m slow (full regeneration, byte for byte)	1 passed in 228 s
ruff check ., ruff format --check .	clean

git status was clean after the runs: nothing was written into the repository.

Out of scope (filed as follow-ups)

• definition_conformance.py, publication_sources.py, engine_probe.py and build_proposals.py in this audit still read the working tree's run. The README now documents exactly which inputs each one reads and checks.
• The Louisiana, Medicare Part B and payroll audits' leaderboard_impact.py scripts have the same working-tree pattern.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 9, 2026 6:12pm UTC