diff --git a/docs/audit.md b/docs/audit.md index 7f16906b..9dfe007d 100644 --- a/docs/audit.md +++ b/docs/audit.md @@ -65,3 +65,133 @@ genuine model errors (`llm_error`) should survive to a scored snapshot — Re-running the script is safe: a case is skipped once it has a verdict, so interrupted runs resume and failures can be re-attempted by re-invoking. + +# Reference adversary + +The diagnosis judge above is built to explain misses, not to doubt references: +its prompt says to treat the reference and its derivation as correct, labels +engine facts authoritative, and allows doubt only through `reference_suspect` +with a concrete contradiction. On 2026-10-05 that design missed reference issues +a reading of the data found: Louisiana's 2026 standard deduction was +PolicyEngine's own CPI computation rather than a published amount; the +`payroll_tax` reference counted paid-leave employee shares that the employer +may, but need not, pass on, though the output asks for "mandatory employee state +payroll taxes"; and the income tax references left out a dependent's own return. +The reference adversary is a separate pass whose only job is to attack the +reference. Its verdicts never change a score. + +## Pipeline + +```bash +# 1. Consensus trigger: scored cells where a cluster of models shares a wrong +# answer (>= 15 of the models, or >= 3 of the top 5). +uv run policybench consensus-flags \ + --payload paper/snapshot/20260501/runs//data.json.gz \ + --output /consensus_flags.json + +# 2. One two-stage case per flagged cell, minus cells another audit owns. +uv run policybench adversary-prepare \ + --payload paper/snapshot/20260501/runs//data.json.gz \ + --flags /consensus_flags.json \ + --annotations-dir annotations/ \ + --skip-cells /covered_elsewhere.json \ + --adversary-dir + +# 3. Judge inside a Subfleet lane (subscription billing, never an API key). +scripts/run_reference_adversary_claude.sh # Claude lane +scripts/run_reference_adversary_codex.sh # Codex lane + +# 4. Verdict table and adjudication queue (one or more judges). +uv run policybench adversary-collect \ + --adversary-dir claude= --adversary-dir codex= \ + --output-dir +``` + +`consensus-flags` records its parameters in the report. `--prototype` +reproduces the 2026-10-05 prototype (answers truncated to whole dollars, +eligibility outputs never flagged): 41 of the frozen run's 1,928 scored cells. +The defaults round answers to the nearest dollar, which merges answers such as +13,387.65 and 13,388, and compare eligibility outputs by mismatch: 61 cells. A +model whose own answer is within the tolerance of an amount reference counts as +exact and never toward a wrong cluster, even when its rounded key misses. + +## Two stages + +- **Stage 1, law first.** The judge sees the household prompt, the output's + definition from `benchmark_specs.json`, the consensus answer with each + member's model id, answer and explanation, and the reference value. It does + not see the engine derivation. It works the answer from primary law + (statutes, regulations, agency publications, forms and instructions), cites + each rule with its publication date and whether it predates the 2026-07-03 + reference freeze, and says which answer the law supports. + - The Claude runner gives it only web tools (no file access) and denies + WebFetch on the PolicyBench, PolicyEngine and GitHub domains + (`BLOCKED_DOMAINS`). WebSearch has no deny rule, so its results reach the + judge. The prompt asks the judge to pass those domains as + `blocked_domains` on every search. The transcript audit rejects an output + whose search results list a blocked URL or name PolicyEngine or + PolicyBench. + - The Codex runner works from an empty directory and rejects a stage 1 + whose log shows a read of derivation-bearing paths. Codex also adds text + to a session that produces no event, so the runner keeps the lane's + configuration from the judge: each call skips the lane's `config.toml` + (`--ignore-user-config`, which drops `developer_instructions`, + `model_instructions_file` and configured MCP servers), turns the + memories, hooks, plugins and apps features off, and gets a fresh, empty + `HOME`, so no user skills reach it. The runner refuses a Codex home + holding `AGENTS.override.md` or `AGENTS.md`, or any skills entry but the + bundled `.system` directory. It does not control what Codex bundles, an + administrator's `/etc/codex`, or a feature those flags leave on; calling + an app or MCP server is an MCP tool event, which the audit rejects. + Codex's event log records a search's query but not its results, so this + runner cannot check what a search returned; the prompt's request is its + only guard there. +- **Stage 2, reconcile.** A fresh call gets the frozen stage-1 JSON (bound by + its sha256) and only now the engine derivation. It returns a verdict: + `reference_holds`, `reference_wrong`, `definition_mismatch` or + `prompt_ambiguous`, with citations and a suggested adjudication. + +## Where verdicts go + +`adversary-collect` writes each judge's verdict table and +`adversary_adjudication_queue.csv`, in the case-notes schema with +`reference_suspect=true`. The queue holds every case with a verdict other than +`reference_holds`. It also holds every case whose verdict is inconsistent, such +as a holding verdict that drops stage 1's finding for the consensus without +naming a stage-1 error. Its rules: + +- A verdict counts only with a sidecar bound to the current stage 1. +- A case without one is listed as missing, and the command then exits + non-zero unless `--allow-missing` is given. +- Each `--adversary-dir` needs its own label (`LABEL=DIR`), which names its + files (`adversary_