Skip to content

fix(audit): don't default claimedExit to 0 when failures are reported (#198) - #199

Merged
surlymochan merged 1 commit into
mainfrom
fix/198-claimed-exit-default
Nov 14, 2026
Merged

surlymochan merged 1 commit into
mainfrom
fix/198-claimed-exit-default

Conversation

@surlymochan

Copy link
Copy Markdown
Owner

Fixes #198 (M15 acceptance precondition, user ruling 2026-11-15 01:25).

Bug

hunt2: pi honestly reported "the remaining 7 failures in the suite (npm test → 34 pass / 7 fail)" — no exit claimed. extractClaims (src/core/audit.js:455-456) defaulted claimedExit to 0 whenever claimedPassed !== null, so both verifiers judged the honest report a breach (exit: claimed 0, replayed 1) before the matching counts (34/7/41) were ever compared.

Fix (combination, both verifier paths)

  • src/core/audit.js extractClaims: only default exit 0 when the claim does not report failures (claimedFailed > 0 leaves claimedExit null). Passes-only and 0 failed claims still default to 0; an explicitly stated exit is untouched.
  • src/core/audit.js runAudit claim loop and src/core/live-claims.js verifyTestClaim: when claimedExit is null, skip the exit comparison and judge on counts alone (mismatch still breaches on counts; unparsable counts with claimed counts is unknown in the live path, counts_unparsed note in the audit path). Explicit exits are still enforced.
  • src/core/live-claims.js bare-claim detector: \b(all|every)\d*\b so "all3 tests now pass" (missing space) is detected as a bare all-pass claim instead of escaping verification entirely. The glued count is not recorded as a pass count (a miscounted total must not false-breach a true all-pass statement); the claim still breaches when the replay has failures.

Tests

9 new regression tests, all passing:

  • extractClaims: 34/7 → claimedExit null; passes-only / 0-failed still default 0; explicit exit 0 + 34/7 kept.
  • verifyTestClaim: honest 34/7 + replay 34/7 exit 1 → confirmed; explicit exit 0 + same replay → breach (exit: claimed 0, replayed 1); counts mismatch without stated exit → breach; unparsable counts without stated exit → unknown.
  • Bare claim: "all3 tests now pass" / "every3 tests pass" / "all3 tests are green" detected; all-pass claim over a failing replay → breach.
  • runAudit end-to-end: honest 2 pass / 1 fail report replaying 2/1 exit 1 → verified.

Full suite (single run, node --test test/*.test.js; note npm test is fail-fast via run-tests.mjs and stops at the first failing file): 806 tests, 801 pass, 4 fail, 1 skipped. The 4 failures are pre-existing on clean main (same names, reproduced via stash): missing pytest binary on this machine (spawnSync pytest ENOENT) — test/i6d-acceptance-demo.test.js 1 + test/xflow-audit.test.js I6d gate 3. No regression from this change.

CI extras run locally: drift:scan 9 pass, pack:check ok.

Not merged

Per the task, this PR is left unmerged for the main round's second-party review. After merge, the M15 evidence set must be re-run (see #197): fixed code re-checks the recorded hunt2 session claims (honest 34/7 must no longer breach), an injected fake agent must still be caught, and real failing sessions must still replay consistently.

…#198)

A claim that says "34 pass / 7 fail" states no exit code, but
extractClaims defaulted it to exit 0, so both verifiers judged the
honest report a breach ("exit: claimed 0, replayed 1") before the
matching counts were ever compared. Claims that report failures now
leave claimedExit null, and verifyTestClaim / runAudit judge them on
counts alone; an explicitly stated exit is still enforced.

Also tightens the bare all-pass detector for a count glued to "all"
("all3 tests now pass" was missed entirely, so the assertion escaped
verification).

Tests: 9 new regression tests (extractClaims defaulting + explicit-exit
preserved, live verifyTestClaim honest/explicit/counts-mismatch/
unparsed, bare "all3" detection + all-pass-over-failing-suite breach,
runAudit end-to-end honest partial-failure report). Full suite:
806 tests, 801 pass, 4 fail — the 4 failures are pre-existing on clean
main (missing pytest binary: spawnSync pytest ENOENT), reproduced by
name: i6d-acceptance-demo 1 + xflow-audit I6d gate 3.
@surlymochan
surlymochan merged commit e3332dc into main Nov 14, 2026
2 checks passed
@surlymochan
surlymochan deleted the fix/198-claimed-exit-default branch November 14, 2026 19:58
This was referenced Nov 14, 2026
surlymochan added a commit that referenced this pull request Nov 14, 2026
Milestone M15 live verify accepted (#197, #192): live session wrapper + streaming pairing (#194), real-time claim verification + --enforce kill on breach (#196), claimedExit default fix (#199, fixes #198).

Co-authored-by: Nick <209239467+surlymochan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

跟进:claimedExit 默认规则在 claimedFailed>0 时误判诚实报告为 breach

1 participant