Repository navigation
Record two references held against a model consensus, and let the adversary list them (evidence only; no score changes) - #213
Merged
Conversation
…ersary list them (evidence only; no score changes)
On 2026-10-10 a session checked two scored references of release
dashboard-data-20261010 that several strong models agree against, and found
both correct:
- scenario_039 (VA) federal income tax: $8,596.03. Six models answer $5,145.11
by filing the lone widowed head as a qualifying surviving spouse; 26 U.S.C.
2(a)(1)(B) requires a dependent child in the home.
- scenario_025 (OH) state income tax: $1,916.61. Eight models answer $1,589.56
by leaving out the $332.00 in R.C. 5747.02(A)(3)(c).
reference_audit/2026-10-10-held-references/ holds the session's report, the
engine traces on policyengine-us main 75cdd8019e (2.38.8) and on the release's
2.38.6, the hand derivations as an executable script, the law excerpts relied
on, the models' answers, and the two independent GPT-6.1 Sol reviews. Every
claim in the report was checked again for this record; the README lists the
three it corrects, including that the consensus trigger does not flag the
Virginia cell at its default parameters today.
Each reviewer would exclude its cell on a wording ground the consensus does
not raise ("estate income" states no character; "employer sponsored insurance
premiums" does not say who paid). That call is Max's, queued as decision
d1252, and is recorded as pending. Nothing here acts on it.
So that a later reference-adversary pass lists these cells as checked and
held instead of judging them again, reference_audit/held_references.json is a
standing list that `policybench adversary-prepare --held-references` reads. A
record applies to a flagged cell only while the reference is the held value
and every triggering consensus answer is one the record explains; otherwise
the cell is judged and the listing says why.
No reference, exclusion record, score, payload or frozen snapshot changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…s the review The independent review of 6d1f957 found that a record could cover a case the check never saw. The rule is reworked so that it cannot: - A record carries the sha256 of the prompt the models answered and applies only while the payload's prompt for that scenario is the same. A scenario id is a position in a run, so the same id can name another household later. - The bound is a dollar, PolicyBench's exact-match tolerance, and belongs to the record. The pass's own tolerance is no longer used, so flags computed at a loose tolerance cannot stretch a record over another answer. - A cluster is explained only when every member's own answer is within the bound of the same explained answer; the rounded key is not compared. - adversary-prepare checks each held flag against the payload as it checks a flag that gets a case, refuses flags that record another payload's hash, and refuses to prepare a directory that holds judge output for a cell now listed as held, since preparing would delete it. - A stale record whose cell another audit covers is reported as covered elsewhere, not as judged. A flag with no cluster is refused. The record: the opening verdict states its assumptions; the Ohio statute figure ($1,916.60) is told apart from the engine's ($1,916.61, from its 332.00204 base); the Virginia models' income is given from the prompt's whole-dollar facts; the short traces are described as pruning zero branches; what each review does and does not reconstruct is spelled out. Added: the engine and data-builder source lines the README relies on, the re-run on the report's engine commit, the excerpts the hand derivation still lacked, Ohio's 2026 exemption ceiling, and the mutant script itself (27 of 27 killed). The property test that asserted a false property is replaced by one over held cells only, with the review's counterexample pinned. The situations test now rebuilds both situations with PolicyBench's own builder. Still evidence and tooling only: no reference, exclusion, score, payload or snapshot changes, and decision d1252 stays pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second review found that flags with no recorded payload hash could still be applied, so flags from an older payload could list a cell as held while a consensus that formed since went unjudged. - With --held-references the flags report must record the payload's sha256, and the command recomputes the flags from the payload at the report's parameters and refuses a report that differs. The hash is a claim; the recomputation is the check. - A cluster is explained only when it gives exactly one finite answer for each of the n_models models it names; one that answers for another set of models is not. - The data-loss guard refuses when a cell that became held has any file but the prepared stage-1 prompt in its case directory, so what an interrupted runner left is protected too. - The prompt hash is described as what it is: the canonical whole-scenario prompt the payload exports under the tool answer contract, which binds the household and the definitions and is not each model's own request. - The short traces prune zero-valued branches below the output's immediate children, and are described so. - records.json keeps the two records as written, and the tests about them read it, so the standing list can change. A record that still cites this write-up must hold what it held. - The mutant harness runs the unmodified tests first and counts a kill only when pytest ran and a test failed: 32 of 32. Still evidence and tooling only; decision d1252 stays pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two maintenance notes from the third review, which approved 2cfff3a. Tests only; no source or evidence file changes. - The historical tests recomputed the release's flags at the trigger's current defaults, so a later change of a default would have broken them. They now use the parameters of the 2026-10-10 runs, stated in the test and checked against the committed reports. - A refused command is now checked against a directory an earlier preparation left: its manifest, schemas, case and held listing keep their bytes. The command is also run on each form of payload a release ships (plain, gzipped, wrapped, both), and on the same flags written with another key order and whole floats as integers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Review record Independent review by GPT-6.1 Sol on a read-only Subfleet lane (
Every round re-checked that the change is evidence and tooling only, that both reviewers' exclusion dissents are recorded in full, and that decision d1252 is pending and not acted on. Rounds 1 and 2 re-derived the arithmetic and checked the law excerpts, traces and hashes. The reviewer could not run the test suite or the mutant script (read-only lane). Those ran locally: 93 tests pass in Merging at |
This was referenced Oct 10, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records two verdicts from a 2026-10-10 check of release dashboard-data-20261010, with the evidence they rest on, and adds the smallest mechanism that lets a later reference-adversary pass list the two cells as checked and held. Evidence and tooling only: no reference, exclusion record, score, payload or frozen snapshot changes. It follows #200. The full record is
reference_audit/2026-10-10-held-references/README.md.The verdicts
Each reference is what the statute's rate schedule gives for the stated facts as the engine reads them. Whether the prompt's wording fixes that reading is the open question.
Still open: d1252
Each independent reviewer (GPT-6.1 Sol) agrees the consensus is wrong and would still exclude its cell, on a wording ground the consensus does not raise:
Whether to exclude is Max's call, queued as cos decision d1252 and not yet ruled. This PR records both dissents in the reviewers' own words and acts on neither.
What was checked again for this record
The session's report is committed as written (
report/REPORT.md) and was treated as claims:75cdd8019e(2.38.8): installed from that commit in a clean environment, the traces are byte-identical to the session's (traces/…/rerun/). They also reproduce on the release's own engine, policyengine-us 2.38.6: all 77 listed values and both short traces are equal.Scenario.to_pe_householdbuilds from the release'sscenarios.csv.scripts/hand_derivations.pyworks both cells in exact decimals with no engine. Virginia's reference, both consensus answers and each reviewer's alternative reproduce to the cent. Ohio's statute figure is $1,916.60; with the engine's fixed amount, $332.00204, it is the reference, $1,916.61.law/excerpts.md), with each document's address and sha256 (law/sources.json).engine_sources.mdquotes the engine and data-builder lines the README's statements about them rest on.Four corrections to the report, listed in the README:
The mechanism
covered_elsewhere.jsonbelongs to the October 5 pass and is keyed by cell alone, so it cannot say that a reference was checked at a value, under a prompt, against a particular answer. This PR adds:reference_audit/held_references.json: a standing list. Each record names the cell, the sha256 of the scenario's prompt as the payload exports it, the held reference, each consensus answer the check explained, and the write-up.policybench adversary-prepare --held-references <file>(policybench/held_references.py): a flagged cell with a record gets no case, and<adversary-dir>/held_references.jsonlists it as checked and held next to the explanation of each triggering cluster.A record applies only while all three are true; otherwise the cell is judged and the listing gives the reason:
prompt_changed). It is the canonical whole-scenario prompt, which binds the household and the output definitions, not each model's own request;reference_moved);unexplained_consensus).The dollar is the benchmark's exact-match tolerance and belongs to the record; the pass's own
--toleranceis not used.The flags must be the payload's own: the command requires the flags report to record the payload's sha256, then recomputes the flags from the payload at the report's parameters and refuses a report that differs. Flags from an older payload would lack a consensus that formed since, and a hold would hide it.
On the release's payload: at the default parameters OH 025 is listed as held (54 cases instead of 55) and VA 039 as not flagged; with
--min-top 2both are listed as held (verification/held_on_20261010*.json).Invariants
A hold can go stale, but it cannot cover a prompt, a reference or a consensus answer the check did not see. For every set of flags, records and prompts (
tests/test_held_references.py, Hypothesis where marked):held,not_appliedandnot_flagged(property).--held-referencesthe cases and the printed line are as before.Differential tests, each against the release's payload read from git at
5a8164a0:records.json, their copy beside the evidence, so the standing list can change);Scenario.to_pe_household's;manifest.jsonreproduces, so a changed evidence file fails.scripts/mutants_held_references.pyruns the unmodified tests, then breaks the rule and the command 32 ways; the tests catch all 32 (verification/mutants_held_references.txt).Review
Independent review (GPT-6.1 Sol, read-only) took three rounds. The first two requested changes, each on the hold rule's safety; the third approved:
6d1f9577: a record was bound only to the cell id, used the pass's tolerance, compared rounded cluster keys, and held flags skipped the payload check; one property test asserted a false property.e59f4369reworks the rule and addresses its nine other findings.e59f4369: flags with no recorded payload hash could still be applied, so flags from an older payload could hide a consensus that formed since.2cfff3acrequires the hash and recomputes the flags, and addresses its six other findings (cluster membership, the data-loss guard for interrupted runs, the prompt hash described as the exported canonical prompt, the short-trace description, records archived beside the evidence, the mutant harness's baseline).2cfff3ac: approved, with no blocking finding and two maintenance notes on the tests.96eb548faddresses both and changes tests only: the historical tests use the 2026-10-10 runs' own trigger parameters, not the current defaults; refusals are checked against an existing directory's bytes; and the command is run on each form of payload a release ships.Tests
uv run pytest tests/test_held_references.py: 93 passed. Withtests/test_consensus.py,test_reference_adversary_runner.py,test_cli_helpers.pyandtest_disclosures.py: 223 passed.ruff check .andruff format --check .pass. CI runs the full suite.tests/test_reference_adversary.py::test_apply_adversary_flags_changes_nothing_else, which this PR does not touch, has a wrong oracle for whitespace-padded text and can fail at random on main; it is split off as a separate task.Also
The two prompt-label notes this check produced ("is a surviving spouse" is the Code's name for a filing status while the data flag is only "widowed"; "estate income" states no character) are a comment on #165.
axiom: n/a: PolicyBench evidence and tooling; no policy is encoded or changed.
🤖 Generated with Claude Code