Pin the reference adversary's other scripts to the pass's inputs

[MaxGhenis]

Summary

This is stacked on #205, which pinned leaderboard_impact.py. It targets main so CI runs, which means #205's two commits (59c32d60, 95aaa8fb) appear in this diff until #205 merges. This PR's own change is 95aaa8fb..HEAD. Merge #205 first; this branch is then rebased onto main.

It gives the reference adversary's other scripts the same staged, sha256-checked inputs:

• definition_conformance.py
• publication_sources.py
• engine_probe.py
• build_proposals.py

The pins and the check now live in one module, scripts/pass_inputs.py, and leaderboard_impact.py uses it too.

No committed evidence file, reference, exclusion or score changes, and proposed_changes.json stays at 3a6e5920…. The diff touches only the audit's scripts and README, plus tests.

The bug, reproduced

These runs used the scripts as of #205's head (95aaa8fb), from a clean checkout of main. On main, release dashboard-data-20261006 (#202) has rewritten the run's payload and exclusion record, and #204 has rewritten one convention module. Record: bug_on_main.txt in the review folder named under Review.

• engine_probe.py exits 0 and silently writes different evidence. Run with the arguments the committed co_sales_tax_refund_scenario_043.json records, its probe has no payroll_tax reproduction row. Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202 later excluded that cell, and the script reads the working tree's exclusion record. By the same mechanism scenario_082's probe would lose its payroll_tax row too, since Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202 excluded that cell as well (deduced from the code, not run).
• definition_conformance.py, run after following Pin the reference adversary's leaderboard impact to the pass's inputs #205's README Reproduce step (restore the run and annotations from 8b4c0ca1), also exits 0. It records latest_alt_snap_mortgage_residence.py as 5e701f8f… (Stop passing the mortgage interest inputs policyengine-us #9605 deletes #204's rewrite) instead of the pass's 00cd1388…. The engine-side scripts copied the reference system's convention modules from the working tree.
• build_proposals.py writes the pinned proposed_changes.json in place. Today the bytes come out identical, but only because none of its eight cells is among Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's new exclusions.
• definition_conformance.py and publication_sources.py stop at their payload pin on an unrestored tree. They read every other input from the working tree unchecked.
• The independent review found one more working-tree read. policybench loads benchmark_specs.json itself as policybench.scenarios is imported, so the engine helpers looked up each output's engine variable and aggregation in the checkout's file. Its bytes equal the pass's today, so nothing has drifted yet.

The fix

scripts/pass_inputs.py (new) holds every pin, as commit, path and sha256:

• the seven run files at 8b4c0ca1;
• policybench/benchmark_specs.json at 8b4c0ca1;
• the 20 convention-module files latest_final is built from, at 8b4c0ca1;
• Audit optional employer pass-through in the payroll tax references; propose excluding four outputs #194's law table at 049f4f09;
• proposed_changes.json and verification/definition_conformance.json at 4db91b5f.

git_input writes git show <commit>:<path> to scratch and stops (SystemExit) unless the bytes match the pin. stage_run, stage_fixes and stage_specs stage sets of files. use_staged_specs points policybench at the staged output definitions for the current process.

Each script stages everything it reads before it computes or writes anything, and reads only the staged copy. The three engine-side scripts also stage before they import the engine or start a worker:

• definition_conformance.py:
  • stages the payload, references, sidecar, scenarios and the law table, then the conventions and output definitions;
  • points policybench at the staged definitions before importing anything else from it;
  • starts its workers with the spawn method and an initializer, so each worker stages its own conventions and definitions and none inherits the main process's system;
  • refuses a law-table mismatch instead of degrading to available: False;
  • still names each input by its repository path in the run record.
• publication_sources.py does the same for the payload, references, sidecar, exclusion record and scenarios. It loses --from-facts, which built both reports from a saved file with no pin check.
• engine_probe.py stages the scenarios, references and exclusion record, then the conventions and definitions through definition_conformance's builder.
• build_proposals.py:
  • stages the references, the exclusion record and the conformance scan;
  • writes only to a required --out, which it creates exclusively, so it never writes a file that already exists;
  • refuses the committed proposed_changes.json however the path is spelled.
• leaderboard_impact.py (Pin the reference adversary's leaderboard impact to the pass's inputs #205) imports the shared pins. pass_inputs() became stage_inputs(), and its scoring child now keeps the caller's PYTHONPATH after the checkout. It still scores with the checkout's policybench, output definitions included; its published-reproduction check covers that.
• Each script checks that the pass_inputs it imported is the file beside it.

README.

• The Inputs tables list every pin with its commit, sha256 and the scripts that stage it, including the 20 convention modules.
• "Who checks the pins" now says every script does, and how the output definitions are handled.
• Reproduce keeps the restore step only for the CLI commands, which record the payload path they are given. It lists the scripts' commands into scratch, and says what each regeneration test compares.

Tests.

• tests/test_reference_adversary_inputs.py (new) holds the new tests below.
• tests/working_tree_fence.py (new) is an audit hook that fails any open of the working tree's copy of a pinned input, benchmark_specs.json included. Armed through a generated sitecustomize, it reaches subprocesses and spawned workers.
• tests/test_reference_adversary_impact.py is adapted to the shared module, and its slow regeneration now runs fenced. Pin the reference adversary's leaderboard impact to the pass's inputs #205's byte-refusal property test moved to the new file, where it covers every pin.

Verification

Byte-for-byte regeneration. Each script, at head f6774fd1, ran against main's working tree (#202's run, #204's module) and wrote to scratch, with the fence armed in every process (pytest_slow_r2.txt, 11 passed in 27 minutes on a heavily loaded machine):

Output	Result
verification/definition_conformance.md	identical
verification/definition_conformance.json	identical except the values of run.seconds (wall time) and run.script_sha256 (the script's own hash; the committed file records 33ce4c3e…, the script as #200 merged it)
verification/publication_sources.md	identical
verification/publication_sources.json	identical except the value of meta.seconds
verification/probes/*.json (8)	all identical, each from the arguments it records
verification/leaderboard_impact* (23)	all identical
proposed_changes.json (via --out to scratch, fast suite)	identical, sha256 3a6e5920…

A baseline of the unmodified scripts, with the run and modules restored from 8b4c0ca1, differs from the committed files in the same seconds lines only (manual_regeneration.txt). So the environment (uv sync --locked --extra dev --python 3.12, policyengine-us 2.15.17) reproduces the evidence, and the scripts' changes alter none of it.

Tests.

• Fast suite: the two reference-adversary test files pass locally at f6774fd1 (50 passed). The full non-slow suite last ran locally before the review fixes (2,057 passed at 54a6a2ba, pytest_fast.txt); CI runs it at head.
• The slow regenerations are deselected in CI.

Mutation check. Partly run; the rest is pending.

• Run at f6774fd1 (mutants_r2.txt): 12 mutants that each reintroduce a working-tree read, drop the sha256 check, stage after engine work, drop the proposed_changes.json refusal, or put a pin in a script. Every one fails a fast test.
• Written but not yet run (mutants_r2.py): 17 mutants that reintroduce the review's findings one at a time. The machine is saturated and local test runs are on hold.
• Also pending: two mutants that only the fenced slow tests can catch (a main or always_zero reading the working tree's scenarios.csv, whose bytes equal the pin). Before the review fixes, the first of these passed the fast suite and failed the fenced slow regeneration with PermissionError (mutant_slow_only.txt).

Invariants

• Pinned inputs. Every input byte a script reads equals the pinned bytes, whatever the working tree holds. Covered by:
  • the fence tests, which run build_proposals.py and engine_probe.py whole and each script's staging and reading in-process;
  • the slow regenerations, fenced in every process;
  • the differential test that the committed evidence records the same hashes as the pins.
• Pinned output definitions. The engine helpers look up each output in the pinned benchmark_specs.json, not the checkout's. A copy of the package with a changed mapping, put first on the path, does not change the lookup. A process that has already loaded other definitions is refused.
• Refusal before use. Any byte that is not the pinned byte is refused, and nothing is written. A script pointed at Stop scoring eight tax outputs and date the response window from the last answer (release dashboard-data-20261006) #202's run, or given a wrong convention or definitions pin, stops before any engine import, system build, worker or output write. Covered by a Hypothesis property over all 31 pins and per-script refusal tests whose engine entry points raise if reached.
• Workers stage their own inputs. Both pools use the spawn context and an initializer that stages, whatever the platform default.
• Reproduction. Each script regenerates its committed outputs byte for byte, except the value on the volatile lines named above. _identical_but requires the same key, indentation and punctuation on those lines, and the regenerated script_sha256 to be the current script's hash.
• Existing files are never rewritten. build_proposals.py creates its output exclusively and refuses proposed_changes.json by path, so neither a hard link nor another checkout's copy is written.
• One source of pins. No script other than pass_inputs.py holds a sha256 or names the run's path. The README tables list each pin with exactly the scripts that stage it, checked by recording which pins each script stages.

Review

• Round 1 (GPT-6.1 Sol, subfleet run --task review --tier hard, at b4f2aac8): REQUEST_CHANGES, with two blockers and five other findings (review_sol.md).
• All seven are addressed in f6774fd1; response_r1.md maps each finding to its fix and tests.
• Round 2 has not been requested yet. This PR stays a draft until it has passed.

Everything cited above is in ~/reviews/policybench-reference-adversary-2026-10/other-scripts-pinned/.

Follow-ups

The Louisiana, Medicare Part B and payroll audits' impact scripts have the same working-tree pattern; #206 pins them. Their engine-side sweeps build latest_final from the working tree's convention modules and are a separate follow-up, which can reuse pass_inputs.py and the fence.

axiom: n/a: audit tooling only; no policy encoded or changed

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 10, 2026 12:56am UTC

[MaxGhenis]

Merge-order note, from a local merge reported by the session stacking on both PRs:

• Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 (head 1210c65e) adds tests/test_reference_audit_pins.py::test_the_pins_agree_across_the_scripts_and_with_the_adversary. That test AST-parses PAYLOAD_SHA256 from the adversary's definition_conformance.py and publication_sources.py.
• This PR moves that pin into scripts/pass_inputs.py (RUN_SHA256["data.json.gz"]). A merge of the two fails assert [] == ['1e029aaa…'], though each PR is green alone.
• I've suggested Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 make the check order-independent: read pass_inputs.py when it exists, else the old constants.
• If Pin the October 5 audits' leaderboard impact, proposals and answers to the passes' inputs #206 lands first without that, this PR's rebase onto main (needed anyway after Pin the reference adversary's leaderboard impact to the pass's inputs #205) will patch the test, and the full fast suite will be run on the rebased head before merge.