diff --git a/README.md b/README.md index e03b280b..47674c64 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ The dashboard below is generated from an append-only event ledger and immutable, - Benchmark evidence dashboard. Equivalent detailed tables follow: 11 suites have accepted checkpoints, 1 is current, and 4 shipped agents have no benchmark evidence. + Benchmark evidence dashboard. Equivalent detailed tables follow: 11 suites have accepted checkpoints, 2 are current, and 4 shipped agents have no benchmark evidence. The tracker separates lifecycle, evidence provenance, freshness, change type, and assessment. A stale score remains visible but is never presented as current. Current history is too sparse and heterogeneous to support exponential-growth or diminishing-return claims; the honest classification is **insufficient comparable evidence**. @@ -141,7 +141,7 @@ The tracker separates lifecycle, evidence provenance, freshness, change type, an | Backend performance reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 7/7 (100.0%) · first-pass clean pass: 3/6 (50.0%) · block recall: 7/7 (100.0%) · clean pass: 6/6 (100.0%) | | Frontend security reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 11/11 (100.0%) · first-pass clean pass: 7/8 (87.5%) · block recall: 10/11 (90.9%) · clean pass: 8/8 (100.0%) | | Frontend performance reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 10/11 (90.9%) · first-pass clean pass: 7/8 (87.5%) · block recall: 10/11 (90.9%) · clean pass: 7/8 (87.5%) | -| Correctness reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 35/38 (92.1%) · first-pass clean pass: 24/28 (85.7%) | +| Correctness reviewer | shipped | accepted | current | coverage | ↕ mixed | first-pass FAIL recall: 44/50 (88.0%) · first-pass clean pass: 32/40 (80.0%) | | Decision governance | shipped | evidence-only | unknown | quality | ↑ improved | Detect accuracy: 45/49 (91.8%) · DECISION recall: 8/9 (88.9%) | | Sentry capture judge | shipped | evidence-only | unknown | quality | ↑ improved | Commit-message F1: 56/100 (56.0%) · Focused-diff F1: 87/100 (87.0%) | | Edge-case autonomy | no-ship | accepted | stale | no-ship | ? unknown | Judge-free ceiling: 51.2% · Pre-registered target: 35.0% | diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index b281a2b4..d2570b7c 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -79,7 +79,7 @@ Committed evidence rejects raw prompts, transcripts, absolute paths, email addre | Backend performance reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 7/7 (100.0%) · first-pass clean pass: 3/6 (50.0%) · block recall: 7/7 (100.0%) · clean pass: 6/6 (100.0%) | | Frontend security reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 11/11 (100.0%) · first-pass clean pass: 7/8 (87.5%) · block recall: 10/11 (90.9%) · clean pass: 8/8 (100.0%) | | Frontend performance reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 10/11 (90.9%) · first-pass clean pass: 7/8 (87.5%) · block recall: 10/11 (90.9%) · clean pass: 7/8 (87.5%) | -| Correctness reviewer | shipped | accepted | stale | coverage | ? unknown | first-pass FAIL recall: 35/38 (92.1%) · first-pass clean pass: 24/28 (85.7%) | +| Correctness reviewer | shipped | accepted | current | coverage | ↕ mixed | first-pass FAIL recall: 44/50 (88.0%) · first-pass clean pass: 32/40 (80.0%) | | Decision governance | shipped | evidence-only | unknown | quality | ↑ improved | Detect accuracy: 45/49 (91.8%) · DECISION recall: 8/9 (88.9%) | | Sentry capture judge | shipped | evidence-only | unknown | quality | ↑ improved | Commit-message F1: 56/100 (56.0%) · Focused-diff F1: 87/100 (87.0%) | | Edge-case autonomy | no-ship | accepted | stale | no-ship | ? unknown | Judge-free ceiling: 51.2% · Pre-registered target: 35.0% | diff --git a/docs/benchmarks/assets/dashboard-dark.svg b/docs/benchmarks/assets/dashboard-dark.svg index a54e41ca..e795f8d4 100644 --- a/docs/benchmarks/assets/dashboard-dark.svg +++ b/docs/benchmarks/assets/dashboard-dark.svg @@ -17,7 +17,7 @@ Backend performance reviewer! stalefirst-pass FAIL recall: 7/7 (100.0%) · first-… Frontend security reviewer! stalefirst-pass FAIL recall: 11/11 (100.0%) · firs… Frontend performance reviewer! stalefirst-pass FAIL recall: 10/11 (90.9%) · first… -Correctness reviewer! stalefirst-pass FAIL recall: 35/38 (92.1%) · first… +Correctness reviewer✓ acceptedfirst-pass FAIL recall: 44/50 (88.0%) · first… Decision governance· evidence-onlyDetect accuracy: 45/49 (91.8%) · DECISION rec… Sentry capture judge· evidence-onlyCommit-message F1: 56/100 (56.0%) · Focused-d… Edge-case autonomy!× stale no-shipJudge-free ceiling: 51.2% · Pre-registered ta… diff --git a/docs/benchmarks/assets/dashboard-light.svg b/docs/benchmarks/assets/dashboard-light.svg index 365e5b13..992a8a8a 100644 --- a/docs/benchmarks/assets/dashboard-light.svg +++ b/docs/benchmarks/assets/dashboard-light.svg @@ -17,7 +17,7 @@ Backend performance reviewer! stalefirst-pass FAIL recall: 7/7 (100.0%) · first-… Frontend security reviewer! stalefirst-pass FAIL recall: 11/11 (100.0%) · firs… Frontend performance reviewer! stalefirst-pass FAIL recall: 10/11 (90.9%) · first… -Correctness reviewer! stalefirst-pass FAIL recall: 35/38 (92.1%) · first… +Correctness reviewer✓ acceptedfirst-pass FAIL recall: 44/50 (88.0%) · first… Decision governance· evidence-onlyDetect accuracy: 45/49 (91.8%) · DECISION rec… Sentry capture judge· evidence-onlyCommit-message F1: 56/100 (56.0%) · Focused-d… Edge-case autonomy!× stale no-shipJudge-free ceiling: 51.2% · Pre-registered ta… diff --git a/docs/benchmarks/checkpoints/f69e993d15469053672364db5a68c1ab38ad12e3176a2d12f66bddc3d93bfb6d.json b/docs/benchmarks/checkpoints/f69e993d15469053672364db5a68c1ab38ad12e3176a2d12f66bddc3d93bfb6d.json new file mode 100644 index 00000000..7709175d --- /dev/null +++ b/docs/benchmarks/checkpoints/f69e993d15469053672364db5a68c1ab38ad12e3176a2d12f66bddc3d93bfb6d.json @@ -0,0 +1,781 @@ +{ + "acceptance": { + "accepted": true, + "reason": "Complete reviewer cohorts with row evidence, zero outages, and suite floors met" + }, + "adapter": "reviewer", + "capturedAt": "2026-08-01T17:35:02.000Z", + "comparisons": [ + { + "method": "shared-row outcome comparison", + "negativeDiscordant": 2, + "positiveDiscordant": 2, + "predecessorEventId": "evt-2026-08-01-reviewer-correctness-75d2d15bab20", + "sharedRows": 66, + "verdict": "mixed" + } + ], + "hashes": { + "corpus": "sha256:3ac68f7e77d58567eee38139b20bc211266904be19c54cd588fd2c8a54f8c0a4", + "implementation": "sha256:39f934cad948e1489a05e182884012a966cd19200cf3ff1f158aa7259c9bd57b", + "runner": "sha256:7fed284584ffc9abb04b2da5ae85345b20e85509e93e0c79097b2792f0367b12", + "scorer": "sha256:7fed284584ffc9abb04b2da5ae85345b20e85509e93e0c79097b2792f0367b12" + }, + "metrics": [ + { + "denominator": 50, + "direction": "higher", + "id": "correctness-reviewer@sonnet@cascade-off:first-fail-recall", + "inferenceUnit": "row", + "interval": { + "lower": 0.7619518261679701, + "method": "wilson-95", + "upper": 0.9438239984906773 + }, + "label": "correctness-reviewer@sonnet@cascade-off first-pass FAIL recall", + "numerator": 44, + "unit": "ratio", + "value": 0.88 + }, + { + "denominator": 40, + "direction": "higher", + "id": "correctness-reviewer@sonnet@cascade-off:first-clean-pass", + "inferenceUnit": "row", + "interval": { + "lower": 0.652426936536005, + "method": "wilson-95", + "upper": 0.895000102745623 + }, + "label": "correctness-reviewer@sonnet@cascade-off first-pass clean pass", + "numerator": 32, + "unit": "ratio", + "value": 0.8 + } + ], + "rows": { + "correctness-reviewer@sonnet@cascade-off:corr-asymmetric-flip-classifier": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "6de9cfbf2954" + }, + "correctness-reviewer@sonnet@cascade-off:corr-broadcast-fanout-no-dedup": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "47c008601f5a" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-broadcast-targeted": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "e938d8dfd98f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-cas-guarded-write": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "78f68c681a4b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-catch-rethrows": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "e8f30a7b534c" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-clears-stale-artifact": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "6b1d08fe1cda" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-conditional-update": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "2bdf6d299d0f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-early-exit-preserved": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "f0e7c45072f7" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-empty-recompute-clears": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "3bd72383bd3a" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-exit-code-fail-closed": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "21bd92d728ef" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-fs-recursive-mkdir": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "eb96bfed4c42" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-lock-finally": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "e11d0cbad3ba" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-loop-break-preserved": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "c2c122dff0af" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-only-validated": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "323fff96a42c" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-outage-exit-preserved": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "ab4b076ebb7b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-parses-json-string-result": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "44a17ae0e114" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-reset-failure-reasons": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "d1aa3924919d" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-retry-clears-error": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "c3a89e627c9d" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-retry-reset-counter": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "d1986643add2" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-return-checked-fails-closed": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "b0212aaba7aa" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-select-for-update": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "14a0f0b1e50a" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-signature-callsite-updated": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "8f24ea6cab36" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-targeted-broadcast": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "d98ec7291153" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-tight-anchor-classifier": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "32273403766b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-timer-debounce-clears": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "f6b1ee659889" + }, + "correctness-reviewer@sonnet@cascade-off:corr-decoy-worktree-finally": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "b7d85baf480b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-discarded-resume-return": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "ff1b6eaaf1ad" + }, + "correctness-reviewer@sonnet@cascade-off:corr-early-exit-loop-break-dropped": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "0d411082e3fc" + }, + "correctness-reviewer@sonnet@cascade-off:corr-early-exit-status-bypassed": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "bb1655183df9" + }, + "correctness-reviewer@sonnet@cascade-off:corr-empty-recompute-keeps-stale": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "97ecde7284e1" + }, + "correctness-reviewer@sonnet@cascade-off:corr-execfile-cmd-shim-swallowed": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "203e6fa03e0e" + }, + "correctness-reviewer@sonnet@cascade-off:corr-exit-code-only-one-fails": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "d145d1ad9a98" + }, + "correctness-reviewer@sonnet@cascade-off:corr-field-validated-wrong-state": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "ef5656e35718" + }, + "correctness-reviewer@sonnet@cascade-off:corr-fs-check-then-act-race": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "e14b7efdc4ba" + }, + "correctness-reviewer@sonnet@cascade-off:corr-group-map-last-write-wins": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "8884e07c6d1b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-guarded-result-ignored-mode-flip": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "5779051bc2d1" + }, + "correctness-reviewer@sonnet@cascade-off:corr-json-string-result-dropped": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "cb0d1314d164" + }, + "correctness-reviewer@sonnet@cascade-off:corr-lock-not-released-on-throw": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "7f9f3d490d13" + }, + "correctness-reviewer@sonnet@cascade-off:corr-lock-timeout-runs-unlocked": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "ba399eec3288" + }, + "correctness-reviewer@sonnet@cascade-off:corr-loose-cmd-classifier-anchor": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "6396f1f3b278" + }, + "correctness-reviewer@sonnet@cascade-off:corr-loose-json-error-classifier": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "9d6d1a732721" + }, + "correctness-reviewer@sonnet@cascade-off:corr-manifest-unknown-version-wiped": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "d49452801030" + }, + "correctness-reviewer@sonnet@cascade-off:corr-nearexpiry-permanent-reauth": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "5cbfcd319951" + }, + "correctness-reviewer@sonnet@cascade-off:corr-only-selector-silent-drop": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "15e6fb189b63" + }, + "correctness-reviewer@sonnet@cascade-off:corr-outage-exit-code-downgraded": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "8d4201a5cf5d" + }, + "correctness-reviewer@sonnet@cascade-off:corr-persist-unnormalized-signal": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "54e6d945cb0f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr138-ci-breaking-gate-engine-coverage": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "bbaa0d63b28f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr138-ci-breaking-gate-engine-coverage-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "03b109856fa8" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr172-duplicate-let-spawnoptions-declaration": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "2460b5333d56" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr172-duplicate-let-spawnoptions-declaration-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "58955fb31aea" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr200-important": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "5b4be62121e1" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr200-important-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "c0ae2bb9ba5b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr200-ref-mutated-during-render-move": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "7b4f10bd7c1f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr200-ref-mutated-during-render-move-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "87332a189081" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr21-fail-closed-when-the-remote": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "b5efdb08286f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr21-fail-closed-when-the-remote-pair": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "61e01ab552b4" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr21-validate-the-parsed-github-repo": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "25638f849b88" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr21-validate-the-parsed-github-repo-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "439eb60c508e" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr26-cover-the-restart-interrupted-flow": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "64bb8db9a785" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr26-cover-the-restart-interrupted-flow-pair": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "d5b2d88ce1a4" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr26-keep-the-mcp-inputschema-in": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "7e174420f714" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr26-keep-the-mcp-inputschema-in-pair": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "cd3e2928584d" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr38-check-the-full-r2-credential": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "3b141c8643d2" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr38-check-the-full-r2-credential-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "a327217fc231" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr38-require-the-full-azure-signing": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": true, + "okFinal": false, + "okFirst": true, + "rowHash": "4ea330953efb" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr38-require-the-full-azure-signing-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "50cf11f113c3" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr48-cn-ts-src": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "58f9cd6e3788" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr48-cn-ts-src-pair": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "920c44fa7a20" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr59-guard-config-json": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "d27f9cc76bb3" + }, + "correctness-reviewer@sonnet@cascade-off:corr-pr59-guard-config-json-pair": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "7487964245c4" + }, + "correctness-reviewer@sonnet@cascade-off:corr-read-then-write-clobber-terminal": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "85ac72a96350" + }, + "correctness-reviewer@sonnet@cascade-off:corr-reminder-ignores-plan-mode": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "71d93cd6e657" + }, + "correctness-reviewer@sonnet@cascade-off:corr-reopen-before-cas-strands": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "918e6e35a29f" + }, + "correctness-reviewer@sonnet@cascade-off:corr-retry-broadcast-double-fire": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "f55d220196ab" + }, + "correctness-reviewer@sonnet@cascade-off:corr-retry-leaves-stale-error": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "c05976dd01fd" + }, + "correctness-reviewer@sonnet@cascade-off:corr-retry-stuck-unclaimable": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "0640a8b18d1b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-revive-user-cancelled": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "b2a0d733679e" + }, + "correctness-reviewer@sonnet@cascade-off:corr-signature-change-callsite-missed": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "16d916a93384" + }, + "correctness-reviewer@sonnet@cascade-off:corr-stale-checklist-artifact": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "e90eb005ef97" + }, + "correctness-reviewer@sonnet@cascade-off:corr-stale-failure-reasons": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "09f050fde84d" + }, + "correctness-reviewer@sonnet@cascade-off:corr-submit-latch-stuck": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "8183fa0b5781" + }, + "correctness-reviewer@sonnet@cascade-off:corr-swallowed-config-parse-bypass": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "5929d8e219fe" + }, + "correctness-reviewer@sonnet@cascade-off:corr-sync-source-target-divergence": { + "expected": "FAIL", + "finalStatus": "pass", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "48bd49378904" + }, + "correctness-reviewer@sonnet@cascade-off:corr-timer-debounce-double-fire": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "ee84f73cfeb3" + }, + "correctness-reviewer@sonnet@cascade-off:corr-unguarded-unpark-clobber": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "868c6ced4079" + }, + "correctness-reviewer@sonnet@cascade-off:corr-worktree-leak-no-finally": { + "expected": "FAIL", + "finalStatus": "fail", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "844bffb4162b" + }, + "correctness-reviewer@sonnet@cascade-off:corr-xdomain-nplusone": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "a0bccc3e0faf" + }, + "correctness-reviewer@sonnet@cascade-off:corr-xdomain-render": { + "expected": "PASS", + "finalStatus": "fail", + "ok": false, + "okFinal": false, + "okFirst": false, + "rowHash": "a2babcecd2d6" + }, + "correctness-reviewer@sonnet@cascade-off:corr-xdomain-sqli": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "2ddabe1e3884" + }, + "correctness-reviewer@sonnet@cascade-off:corr-xdomain-xss": { + "expected": "PASS", + "finalStatus": "pass", + "ok": true, + "okFinal": true, + "okFirst": true, + "rowHash": "5562a8a8edb3" + } + }, + "schemaVersion": 1, + "sourceCommit": "16b73c102cda23ecb3917eec13980f090700066b", + "suiteId": "reviewer-correctness" +} diff --git a/docs/benchmarks/history.jsonl b/docs/benchmarks/history.jsonl index e5bcf80f..e1171bb1 100644 --- a/docs/benchmarks/history.jsonl +++ b/docs/benchmarks/history.jsonl @@ -21,3 +21,4 @@ {"assessment":"unknown","changeType":"coverage","checkpoint":{"path":"docs/benchmarks/checkpoints/1ab5991c759be40ceefdb7e804d6ab2464c12a32554448a0a6052447f496287e.json","sha256":"1ab5991c759be40ceefdb7e804d6ab2464c12a32554448a0a6052447f496287e"},"comparisons":[],"evidence":"accepted","freshness":"current","hashes":{"corpus":"sha256:a31b51b7343edce031dd43d79a0703118e8e5e96735a78cea2720c6cb5c38a14","implementation":"sha256:39f7131d35dc539e202d801bb6d2bd9e21064dad02c37b11d4348ced301df8d7","runner":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a","scorer":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a"},"id":"evt-2026-08-01-reviewer-frontend-security-8c1cbfd57121","lifecycle":"shipped","metrics":[{"denominator":11,"direction":"higher","id":"frontend-security-reviewer@haiku@cascade-on:first-fail-recall","inferenceUnit":"row","interval":{"lower":0.7411670330319684,"method":"wilson-95","upper":1},"label":"frontend-security-reviewer@haiku@cascade-on first-pass FAIL recall","numerator":11,"unit":"ratio","value":1},{"denominator":8,"direction":"higher","id":"frontend-security-reviewer@haiku@cascade-on:first-clean-pass","inferenceUnit":"row","interval":{"lower":0.5291118177871464,"method":"wilson-95","upper":0.9775825085499433},"label":"frontend-security-reviewer@haiku@cascade-on first-pass clean pass","numerator":7,"unit":"ratio","value":0.875},{"denominator":11,"direction":"higher","id":"frontend-security-reviewer@haiku@cascade-on:block-recall","inferenceUnit":"row","interval":{"lower":0.6226415635484043,"method":"wilson-95","upper":0.9837678271141151},"label":"frontend-security-reviewer@haiku@cascade-on block recall","numerator":10,"unit":"ratio","value":0.9090909090909091},{"denominator":8,"direction":"higher","id":"frontend-security-reviewer@haiku@cascade-on:clean-pass","inferenceUnit":"row","interval":{"lower":0.6755924351161198,"method":"wilson-95","upper":1},"label":"frontend-security-reviewer@haiku@cascade-on clean pass","numerator":8,"unit":"ratio","value":1}],"note":"First accepted frontend-security reviewer cohort at the shipped config, 131-row reviewer corpus, zero outages.","provenance":{"source":"committed sanitized checkpoint","sourceCommit":"d855b7b0de8524021905688b617ac836fb652b4f","tier":"accepted"},"recordedAt":"2026-08-01T11:19:02.000Z","schemaVersion":1,"subjectIds":["reviewer-frontend-security-reviewer","agent-frontend-security-reviewer","bin-guard-review"],"suiteId":"reviewer-frontend-security"} {"assessment":"unknown","changeType":"coverage","checkpoint":{"path":"docs/benchmarks/checkpoints/a7b6e01854cc49ad7085e4ee30392ab8be43972dca4ca139f9461a9a72360653.json","sha256":"a7b6e01854cc49ad7085e4ee30392ab8be43972dca4ca139f9461a9a72360653"},"comparisons":[],"evidence":"accepted","freshness":"current","hashes":{"corpus":"sha256:9818efd6009cb8f6080a56ecc00d2a7f47c78e59d2d268d63d4303d370e66f43","implementation":"sha256:c23535aa1da64b761dbf2b614e8f3f1dc7b7508f585ea83f870a2f1692fa48d4","runner":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a","scorer":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a"},"id":"evt-2026-08-01-reviewer-frontend-performance-50009f0eff89","lifecycle":"shipped","metrics":[{"denominator":11,"direction":"higher","id":"frontend-performance-reviewer@haiku@cascade-on:first-fail-recall","inferenceUnit":"row","interval":{"lower":0.6226415635484043,"method":"wilson-95","upper":0.9837678271141151},"label":"frontend-performance-reviewer@haiku@cascade-on first-pass FAIL recall","numerator":10,"unit":"ratio","value":0.9090909090909091},{"denominator":8,"direction":"higher","id":"frontend-performance-reviewer@haiku@cascade-on:first-clean-pass","inferenceUnit":"row","interval":{"lower":0.5291118177871464,"method":"wilson-95","upper":0.9775825085499433},"label":"frontend-performance-reviewer@haiku@cascade-on first-pass clean pass","numerator":7,"unit":"ratio","value":0.875},{"denominator":11,"direction":"higher","id":"frontend-performance-reviewer@haiku@cascade-on:block-recall","inferenceUnit":"row","interval":{"lower":0.6226415635484043,"method":"wilson-95","upper":0.9837678271141151},"label":"frontend-performance-reviewer@haiku@cascade-on block recall","numerator":10,"unit":"ratio","value":0.9090909090909091},{"denominator":8,"direction":"higher","id":"frontend-performance-reviewer@haiku@cascade-on:clean-pass","inferenceUnit":"row","interval":{"lower":0.5291118177871464,"method":"wilson-95","upper":0.9775825085499433},"label":"frontend-performance-reviewer@haiku@cascade-on clean pass","numerator":7,"unit":"ratio","value":0.875}],"note":"First accepted frontend-performance reviewer cohort at the shipped config, 131-row reviewer corpus, zero outages.","provenance":{"source":"committed sanitized checkpoint","sourceCommit":"1fb073f3612a7bbf9cb58e6c0372abc1f7fe7311","tier":"accepted"},"recordedAt":"2026-08-01T11:19:03.000Z","schemaVersion":1,"subjectIds":["reviewer-frontend-performance-reviewer","agent-frontend-performance-reviewer","bin-guard-review"],"suiteId":"reviewer-frontend-performance"} {"assessment":"unknown","changeType":"coverage","checkpoint":{"path":"docs/benchmarks/checkpoints/2520b632529db1c4341aa53d43f6f491e278aa86f4fd033f89be806a8d0f37db.json","sha256":"2520b632529db1c4341aa53d43f6f491e278aa86f4fd033f89be806a8d0f37db"},"comparisons":[],"evidence":"accepted","freshness":"current","hashes":{"corpus":"sha256:db96ac3d606bde78aa3555707727076d1375c79401ca32e8f2bf36197c6b6b2f","implementation":"sha256:c1483e0a07cb5f2cf5eddcd13edb8efb3d38a556f6120a2bae13787ea5167d71","runner":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a","scorer":"sha256:546a99e7239d201b9e017abd4b1662f133b387cb4bf553f1b40d6eaf7123045a"},"id":"evt-2026-08-01-reviewer-correctness-75d2d15bab20","lifecycle":"shipped","metrics":[{"denominator":38,"direction":"higher","id":"correctness-reviewer@sonnet@cascade-off:first-fail-recall","inferenceUnit":"row","interval":{"lower":0.7920062854222853,"method":"wilson-95","upper":0.9727854044930122},"label":"correctness-reviewer@sonnet@cascade-off first-pass FAIL recall","numerator":35,"unit":"ratio","value":0.9210526315789473},{"denominator":28,"direction":"higher","id":"correctness-reviewer@sonnet@cascade-off:first-clean-pass","inferenceUnit":"row","interval":{"lower":0.685101978364747,"method":"wilson-95","upper":0.9430099468025059},"label":"correctness-reviewer@sonnet@cascade-off first-pass clean pass","numerator":24,"unit":"ratio","value":0.8571428571428571}],"note":"First accepted correctness reviewer cohort at the shipped config, 131-row reviewer corpus, zero outages.","provenance":{"source":"committed sanitized checkpoint","sourceCommit":"4125ff02cfb655df76272edbb48b0fd7bfcafd33","tier":"accepted"},"recordedAt":"2026-08-01T11:19:03.000Z","schemaVersion":1,"subjectIds":["reviewer-correctness-reviewer","agent-correctness-reviewer","bin-guard-review"],"suiteId":"reviewer-correctness"} +{"assessment":"mixed","changeType":"coverage","checkpoint":{"path":"docs/benchmarks/checkpoints/f69e993d15469053672364db5a68c1ab38ad12e3176a2d12f66bddc3d93bfb6d.json","sha256":"f69e993d15469053672364db5a68c1ab38ad12e3176a2d12f66bddc3d93bfb6d"},"comparisons":[{"method":"shared-row outcome comparison","negativeDiscordant":2,"positiveDiscordant":2,"predecessorEventId":"evt-2026-08-01-reviewer-correctness-75d2d15bab20","sharedRows":66,"verdict":"mixed"}],"evidence":"accepted","freshness":"current","hashes":{"corpus":"sha256:3ac68f7e77d58567eee38139b20bc211266904be19c54cd588fd2c8a54f8c0a4","implementation":"sha256:39f934cad948e1489a05e182884012a966cd19200cf3ff1f158aa7259c9bd57b","runner":"sha256:7fed284584ffc9abb04b2da5ae85345b20e85509e93e0c79097b2792f0367b12","scorer":"sha256:7fed284584ffc9abb04b2da5ae85345b20e85509e93e0c79097b2792f0367b12"},"id":"evt-2026-08-01-reviewer-correctness-b0dc29a7d531","lifecycle":"shipped","metrics":[{"denominator":50,"direction":"higher","id":"correctness-reviewer@sonnet@cascade-off:first-fail-recall","inferenceUnit":"row","interval":{"lower":0.7619518261679701,"method":"wilson-95","upper":0.9438239984906773},"label":"correctness-reviewer@sonnet@cascade-off first-pass FAIL recall","numerator":44,"unit":"ratio","value":0.88},{"denominator":40,"direction":"higher","id":"correctness-reviewer@sonnet@cascade-off:first-clean-pass","inferenceUnit":"row","interval":{"lower":0.652426936536005,"method":"wilson-95","upper":0.895000102745623},"label":"correctness-reviewer@sonnet@cascade-off first-pass clean pass","numerator":32,"unit":"ratio","value":0.8}],"note":"Corpus growth 66->90 (batch 1 of the CodeRabbit mining effort: +12 mined golds, +12 minimal-pair decoys; charter docs/benchmarks/benchmark-methodology.md). Shipped pin config: sonnet single-pass, cascade-off — no clean-pass floor exists for this cohort (cascade-only floor), stated per the honesty rule. Metrics are LOCALIZED-FIXTURE recall/precision: mined gold measures relative recall only (methodology item 17); fixtures are 1-2 files <=25 lines vs production mean 7.9 files/26KB — calibration slice pending. first-fail-recall 44/50=0.88, first-clean-pass 32/40=0.80, zero outages. Minimal-pair decoys drew 5 of the 8 false FAILs — the hard precision probe the correctness-reviewer-precision record asked for.","provenance":{"source":"committed sanitized checkpoint","sourceCommit":"16b73c102cda23ecb3917eec13980f090700066b","tier":"accepted"},"recordedAt":"2026-08-01T17:35:02.000Z","schemaVersion":1,"subjectIds":["reviewer-correctness-reviewer","agent-correctness-reviewer","bin-guard-review"],"suiteId":"reviewer-correctness"}