Repository navigation
fix(audit): don't default claimedExit to 0 when failures are reported (#198) - #199
Merged
Merged
Conversation
…#198) A claim that says "34 pass / 7 fail" states no exit code, but extractClaims defaulted it to exit 0, so both verifiers judged the honest report a breach ("exit: claimed 0, replayed 1") before the matching counts were ever compared. Claims that report failures now leave claimedExit null, and verifyTestClaim / runAudit judge them on counts alone; an explicitly stated exit is still enforced. Also tightens the bare all-pass detector for a count glued to "all" ("all3 tests now pass" was missed entirely, so the assertion escaped verification). Tests: 9 new regression tests (extractClaims defaulting + explicit-exit preserved, live verifyTestClaim honest/explicit/counts-mismatch/ unparsed, bare "all3" detection + all-pass-over-failing-suite breach, runAudit end-to-end honest partial-failure report). Full suite: 806 tests, 801 pass, 4 fail — the 4 failures are pre-existing on clean main (missing pytest binary: spawnSync pytest ENOENT), reproduced by name: i6d-acceptance-demo 1 + xflow-audit I6d gate 3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #198 (M15 acceptance precondition, user ruling 2026-11-15 01:25).
Bug
hunt2: pi honestly reported "the remaining 7 failures in the suite (
npm test→ 34 pass / 7 fail)" — no exit claimed.extractClaims(src/core/audit.js:455-456) defaultedclaimedExitto 0 wheneverclaimedPassed !== null, so both verifiers judged the honest report a breach (exit: claimed 0, replayed 1) before the matching counts (34/7/41) were ever compared.Fix (combination, both verifier paths)
src/core/audit.jsextractClaims: only default exit 0 when the claim does not report failures (claimedFailed > 0leavesclaimedExitnull). Passes-only and0 failedclaims still default to 0; an explicitly stated exit is untouched.src/core/audit.jsrunAuditclaim loop andsrc/core/live-claims.jsverifyTestClaim: whenclaimedExitis null, skip the exit comparison and judge on counts alone (mismatch still breaches on counts; unparsable counts with claimed counts isunknownin the live path,counts_unparsednote in the audit path). Explicit exits are still enforced.src/core/live-claims.jsbare-claim detector:\b(all|every)\d*\bso "all3 tests now pass" (missing space) is detected as a bare all-pass claim instead of escaping verification entirely. The glued count is not recorded as a pass count (a miscounted total must not false-breach a true all-pass statement); the claim still breaches when the replay has failures.Tests
9 new regression tests, all passing:
extractClaims: 34/7 →claimedExitnull; passes-only / 0-failed still default 0; explicitexit 0+ 34/7 kept.verifyTestClaim: honest 34/7 + replay 34/7 exit 1 → confirmed; explicit exit 0 + same replay → breach (exit: claimed 0, replayed 1); counts mismatch without stated exit → breach; unparsable counts without stated exit → unknown.runAuditend-to-end: honest 2 pass / 1 fail report replaying 2/1 exit 1 → verified.Full suite (single run,
node --test test/*.test.js; notenpm testis fail-fast viarun-tests.mjsand stops at the first failing file): 806 tests, 801 pass, 4 fail, 1 skipped. The 4 failures are pre-existing on clean main (same names, reproduced via stash): missingpytestbinary on this machine (spawnSync pytest ENOENT) —test/i6d-acceptance-demo.test.js1 +test/xflow-audit.test.jsI6d gate 3. No regression from this change.CI extras run locally:
drift:scan9 pass,pack:checkok.Not merged
Per the task, this PR is left unmerged for the main round's second-party review. After merge, the M15 evidence set must be re-run (see #197): fixed code re-checks the recorded hunt2 session claims (honest 34/7 must no longer breach), an injected fake agent must still be caught, and real failing sessions must still replay consistently.