Record two references held against a model consensus, and let the adversary list them (evidence only; no score changes)

docs/audit.md

@@ -89,12 +89,14 @@ uv run policybench consensus-flags \
   --payload paper/snapshot/20260501/runs/<run>/data.json.gz \
   --output <dir>/consensus_flags.json
 
-# 2. One two-stage case per flagged cell, minus cells another audit owns.
+# 2. One two-stage case per flagged cell, minus cells another audit owns and
+#    cells whose reference an earlier check already held.
 uv run policybench adversary-prepare \
   --payload paper/snapshot/20260501/runs/<run>/data.json.gz \
   --flags <dir>/consensus_flags.json \
   --annotations-dir annotations/<run> \
   --skip-cells <dir>/covered_elsewhere.json \
+  --held-references reference_audit/held_references.json \
   --adversary-dir <adv>
 
 # 3. Judge inside a Subfleet lane (subscription billing, never an API key).
@@ -182,6 +184,64 @@ each carries a developer `reference_verdict` (`policybench.adjudications`):
 the existing adjudication path. A change to a published reference or an
 exclusion still needs a release and the maintainer's ruling.
 
+## Held references
+
+The consensus trigger reads no history, so a cell whose reference a check has
+already worked from the law and held is flagged again for as long as the same
+models agree. `reference_audit/held_references.json` is the standing list of
+such checks. Each record names a cell, the scenario's prompt as the payload
+exports it (by its sha256), the reference value held, each consensus answer
+the check explained, and the write-up (`policybench/held_references.py` gives
+the schema).
+
+`adversary-prepare --held-references` gives a flagged cell on that list no
+case and writes `<adv>/held_references.json`, which lists it as checked and
+held beside the explanation of each triggering cluster. A record applies only
+while all of these are true:
+
+- the payload's exported prompt for the scenario (`prompt.tool`, the
+  canonical whole-scenario prompt under the tool answer contract) is the one
+  the check read, byte for byte. A scenario id is a position in a run
+  (`scenario_{i:03d}`), so the same id can name another household after a
+  regeneration. The hash binds the household and the output definitions; it
+  does not record each model's own request, which a model card may serve
+  under the JSON contract or in chunks;
+- the flag's reference is the held value; and
+- each triggering cluster is an answer the record explains: every member's
+  own answer matches the same explained answer.
+
+"Is" and "matches" mean within a dollar on an amount output, the benchmark's
+exact-match tolerance, and equal on an eligibility output. The bound belongs
+to the record: the pass's own `--tolerance` is not used.
+
+Otherwise the cell is judged like any other, and the listing gives the first
+reason that fails: `prompt_changed`, `reference_moved` when an engine or data
+change gave the cell another reference, or `unexplained_consensus` when models
+now agree on an answer the check never looked at. A record whose cell is not
+flagged is listed as `not_flagged`. The option decides only which cells get a
+case. It changes no score, reference or exclusion, and a hold is not a ruling:
+a record can carry an open question, as the first two do
+(`reference_audit/2026-10-10-held-references/`).
+
+With the option, the command also:
+
+- requires the flags to be the payload's. The flags report must record the
+  payload's sha256, and the command recomputes the flags from the payload at
+  the report's parameters and refuses a report that differs. Flags from an
+  older payload would lack a consensus that has formed since, and a hold
+  would hide it;
+- refuses to prepare a directory that holds a runner's files for a cell now
+  listed as held, because preparing removes the directory of a case no longer
+  listed. Preparation writes only `stage1_prompt.md` into a case directory,
+  so any other file there counts: a verdict, or what an interrupted run left
+  behind. Prepare each pass into a new directory;
+- marks a record that no longer applies with what happens to its cell:
+  `prepared`, or `covered_elsewhere` when `--skip-cells` lists it.
+
+To add a record, write the check up under `reference_audit/`, with the law,
+the engine run and the models' answers it rests on, and list only the
+consensus answers it explains.
+
 ## Definition and publication checks
 
 Two engine-side checks run with the reference venv (policyengine-us 2.15.17

policybench/cli.py

@@ -640,6 +640,16 @@ def main():
         help="JSON list of {scenario_id, variable, covered_by} cells another "
         "audit already covers; they get no adversary case",
     )
+    adversary_prepare_parser.add_argument(
+        "--held-references",
+        default=None,
+        help="Held-references JSON (reference_audit/held_references.json): a "
+        "flagged cell an earlier check held gets no adversary case while its "
+        "prompt and reference are the ones checked and every triggering "
+        "consensus answer is one the record explains; "
+        "<adversary-dir>/held_references.json lists those cells and every "
+        "record that no longer applies",
+    )
 
     adversary_collect_parser = subparsers.add_parser(
         "adversary-collect",
@@ -1464,25 +1474,132 @@ def main():
             prepare_adversary,
         )
 
-        flags = json.loads(Path(args.flags).read_text())["flags"]
+        flags_report = json.loads(Path(args.flags).read_text())
+        flags = flags_report["flags"]
+        payload_path = Path(args.payload)
+        payload = load_us_payload(payload_path)
+        adversary_dir = Path(args.adversary_dir)
         skipped: dict[tuple[str, str], str] = {}
         if args.skip_cells:
             for cell in json.loads(Path(args.skip_cells).read_text()):
                 key = (str(cell["scenario_id"]), str(cell["variable"]))
                 skipped[key] = str(cell.get("covered_by", ""))
-        kept = [f for f in flags if (f["scenario_id"], f["variable"]) not in skipped]
+        # Held references first, so "not flagged" means not flagged at all.
+        unheld = flags
+        held_report = None
+        if args.held_references:
+            from policybench.consensus import (
+                ConsensusParams,
+                consensus_flags,
+                file_sha256,
+            )
+            from policybench.held_references import (
+                apply_held_references,
+                load_held_references,
+            )
+            from policybench.reference_adversary import STAGE1_PROMPT
+
+            # A hold is checked against the payload it is applied with, so the
+            # flags must be that payload's: flags from an older payload would
+            # lack a consensus that has formed since, and a hold would hide it.
+            payload_sha256 = file_sha256(payload_path)
+            source_sha256 = flags_report.get("source_sha256")
+            if source_sha256 != payload_sha256:
+                raise SystemExit(
+                    f"adversary-prepare: --held-references needs flags computed "
+                    f"from {payload_path} (sha256 {payload_sha256}), and "
+                    f"{args.flags} records "
+                    + (f"sha256 {source_sha256}" if source_sha256 else "no payload")
+                    + "; recompute them with policybench consensus-flags"
+                )
+            # The recorded hash is a claim. The flags themselves must be what
+            # the trigger computes from this payload at their own parameters.
+            try:
+                flag_params = ConsensusParams(**flags_report["params"])
+            except (KeyError, TypeError, ValueError) as exc:
+                raise SystemExit(
+                    f"adversary-prepare: {args.flags} carries no usable "
+                    f"consensus parameters ({exc}); recompute the flags"
+                ) from exc
+            if consensus_flags(payload, flag_params) != flags:
+                raise SystemExit(
+                    f"adversary-prepare: {args.flags} is not what the consensus "
+                    f"trigger computes from {payload_path} at its recorded "
+                    "parameters; recompute it with policybench consensus-flags"
+                )
+            unheld, held_report = apply_held_references(
+                flags, load_held_references(args.held_references), payload
+            )
+            # The held flags' cases are built only for their ids.
+            unheld_ids = {id(flag) for flag in unheld}
+            held_cases = build_adversary_cases(
+                payload, [flag for flag in flags if id(flag) not in unheld_ids]
+            )
+            # Preparing removes the directory of a case no longer listed.
+            # Preparation itself writes only the stage-1 prompt there, so
+            # anything else is a runner's: a verdict, or what an interrupted
+            # run left behind.
+            runner_files = [
+                case.case_id
+                for case in held_cases
+                if (adversary_dir / "cases" / case.case_id).is_dir()
+                and any(
+                    child.name != STAGE1_PROMPT
+                    for child in (adversary_dir / "cases" / case.case_id).iterdir()
+                )
+            ]
+            if runner_files:
+                raise SystemExit(
+                    f"adversary-prepare: {adversary_dir} holds a runner's files "
+                    f"for cells now listed as held ({', '.join(runner_files)}), and "
+                    "preparing would delete them; prepare into a new directory "
+                    "or move those case directories aside"
+                )
+            for row in held_report["not_applied"]:
+                # A record that no longer applies leaves its cell to the pass,
+                # which judges it unless another audit covers it.
+                cell = (row["scenario_id"], row["variable"])
+                row["case"] = "covered_elsewhere" if cell in skipped else "prepared"
+            held_report = {
+                "held_references": str(args.held_references),
+                "held_references_sha256": file_sha256(args.held_references),
+                "flags": str(args.flags),
+                "flags_sha256": file_sha256(args.flags),
+                "payload_sha256": payload_sha256,
+                **held_report,
+            }
+        kept = [f for f in unheld if (f["scenario_id"], f["variable"]) not in skipped]
         derivations = (
             load_derivations(Path(args.annotations_dir))
             if args.annotations_dir
             else None
         )
-        cases = build_adversary_cases(
-            load_us_payload(Path(args.payload)), kept, derivations=derivations
-        )
-        prepare_adversary(Path(args.adversary_dir), cases)
+        cases = build_adversary_cases(payload, kept, derivations=derivations)
+        prepare_adversary(adversary_dir, cases)
+        # The directory describes this preparation, so a listing left by an
+        # earlier one with --held-references does not outlive it.
+        held_path = adversary_dir / "held_references.json"
+        held_note = ""
+        if held_report is None:
+            held_path.unlink(missing_ok=True)
+        else:
+            held_path.write_text(json.dumps(held_report, indent=2) + "\n")
+            held_note = (
+                f"; {len(held_report['held'])} listed as checked and held in "
+                f"{held_path}"
+            )
+            stale = [
+                f"{row['scenario_id']} {row['variable']} ({row['reason']}; "
+                + ("judged" if row["case"] == "prepared" else "covered elsewhere")
+                + ")"
+                for row in held_report["not_applied"]
+            ]
+            if stale:
+                held_note += f"; held records that no longer apply: {', '.join(stale)}"
         print(
             f"Prepared {len(cases)} adversary cases under {args.adversary_dir} "
-            f"({len(flags) - len(kept)} flagged cells skipped as covered elsewhere). "
+            f"({len(unheld) - len(kept)} flagged cells skipped as covered "
+            f"elsewhere{held_note}). "
             "Run scripts/run_reference_adversary_claude.sh or "
             "scripts/run_reference_adversary_codex.sh inside a Subfleet lane."
         )