From 06f5bbdf96b62cb48009bc40680b1f83007a576b Mon Sep 17 00:00:00 2001 From: norvalbv Date: Sun, 2 Aug 2026 13:00:50 +0100 Subject: [PATCH 1/2] =?UTF-8?q?docs(benchmarks):=20known-answer=20import?= =?UTF-8?q?=20scoping=20=E2=80=94=20c-CRAB/CR-Bench=20TS/JS=20premise=20fa?= =?UTF-8?q?lsified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scoped the queued absolute-recall import against the actual artifacts: c-CRAB builds on SWE-CARE and CR-Bench transforms SWE-Bench — both Python-only, so the planned "filtered to TS/JS" slice does not exist in either. c-CRAB's dataset repo additionally has no license; CR-Bench has released no artifact. Runbook item 2 now records the finding and proposes (unratified) replacements: GHSA/npm-advisory mining with fix commits for the security suites, and applying CR-Bench's transformation recipe to SWE-Bench Multimodal's JS/TS repos for correctness. Absolute recall stays blocked until one is ratified. Co-Authored-By: Claude Fable 5 --- docs/benchmarks/corpus-growth.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/benchmarks/corpus-growth.md b/docs/benchmarks/corpus-growth.md index 05d28204..1dbd72b0 100644 --- a/docs/benchmarks/corpus-growth.md +++ b/docs/benchmarks/corpus-growth.md @@ -82,8 +82,18 @@ runner); freshness goes stale exactly when one moves. The planned calibration slice (replay ~10 scope-confirmed misses at FULL context, locally, never committed) turns this caveat into a number. 2. **Mined gold = precision + relative recall only** (item 17). Absolute recall needs - known-answer imports — c-CRAB (arXiv 2603.23448) and CR-Bench (arXiv 2603.11078), filtered to - TS/JS. This is the only realistic absolute-recall path for the security suites. + known-answer imports. **Scoped 2026-08-02 — the original plan (c-CRAB arXiv 2603.23448 / + CR-Bench arXiv 2603.11078 "filtered to TS/JS") is not viable**: both are Python-derived + (c-CRAB builds on SWE-CARE, CR-Bench transforms SWE-Bench — neither contains any TS/JS + instances to filter), c-CRAB's artifact repo (github.com/c-CRAB-Benchmark/dataset) carries no + license, and CR-Bench has published no artifact at all. Proposed replacements, unratified: + (a) mine GHSA/npm advisories with fix commits directly — public known-answer facts, + re-expressed as anonymized fixtures like every other row, the natural api-security / + frontend-security source (SecBench.js catalogs ~600 such vulns but is itself unlicensed — + use it as an index, not a source); (b) apply CR-Bench's transformation recipe (blame → + PR lookup → detectability filter, Alg. 1 of the paper) to SWE-Bench Multimodal's JS/TS + repos for correctness-suite known-answer rows. Absolute-recall numbers stay blocked until + one of these is ratified and built. 3. **No delta is believed before the noise floor** (item 4): κ blind-relabel (40–60 rows, labeler must be non-Claude or human, sees ONLY repo.base/staged) + cleanlab confident-learning floor. Delta < (floor + paired CI) = unresolved, not a win. From 36d202d60bc9f16252e22d902116a7c8e38465a5 Mon Sep 17 00:00:00 2001 From: norvalbv Date: Sun, 2 Aug 2026 13:14:40 +0100 Subject: [PATCH 2/2] =?UTF-8?q?docs(benchmarks):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20scope=20the=20CR-Bench=20artifact=20claim,=20fix=20?= =?UTF-8?q?pending-list=20conflicts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round on #307: 'published no artifact at all' overclaimed (verified absence of a PUBLIC artifact only) — now 'no publicly released artifact'. Pending-work item 5 still requested the rejected c-CRAB/CR-Bench import, contradicting measurement-rule 2's viability finding — now points at ratifying one of the proposed replacements. Also marked pending item 2 (κ + noise floor) DONE per #304, same staleness class. Co-Authored-By: Claude Fable 5 --- docs/benchmarks/corpus-growth.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/benchmarks/corpus-growth.md b/docs/benchmarks/corpus-growth.md index 1dbd72b0..b354e1b4 100644 --- a/docs/benchmarks/corpus-growth.md +++ b/docs/benchmarks/corpus-growth.md @@ -86,7 +86,7 @@ runner); freshness goes stale exactly when one moves. CR-Bench arXiv 2603.11078 "filtered to TS/JS") is not viable**: both are Python-derived (c-CRAB builds on SWE-CARE, CR-Bench transforms SWE-Bench — neither contains any TS/JS instances to filter), c-CRAB's artifact repo (github.com/c-CRAB-Benchmark/dataset) carries no - license, and CR-Bench has published no artifact at all. Proposed replacements, unratified: + license, and CR-Bench has no publicly released artifact. Proposed replacements, unratified: (a) mine GHSA/npm advisories with fix commits directly — public known-answer facts, re-expressed as anonymized fixtures like every other row, the natural api-security / frontend-security source (SecBench.js catalogs ~600 such vulns but is itself unlicensed — @@ -136,8 +136,9 @@ runner); freshness goes stale exactly when one moves. above — that figure is the blind relabel's measurement and stays as measured. - Yield funnel from the first mining pass: 749 candidates → 479 fixed / 34 rebutted / 236 unresolved → hard drops (232 unresolved-outcome, 179 out-of-charter, 113 truncated-hunk, 27 - already-in-corpus) → 23 gold+pair sets landed across two batches. Rebutted threads mostly fail - extraction (no line anchors) — relaxing that path is the next miner improvement. + already-in-corpus) → 23 gold+pair sets landed across two batches. (The "rebutted threads mostly + fail extraction" premise was falsified 2026-08-02: 25 of 34 already survived; #306 waived the + two hunk drops for rebutted rows, freeing the remaining viable 3 — none die on line anchors.) ## Pending work, in order @@ -149,6 +150,9 @@ runner); freshness goes stale exactly when one moves. probabilities from a bench run, not another labeler. 3. Calibration slice (localized vs full-context delta). 4. Domain-suite cascade re-bench (needs opus) now that api-security has 30 rows. -5. c-CRAB / CR-Bench known-answer import for absolute recall (security suites especially). +5. Known-answer import for absolute recall (security suites especially): ratify and build one of + the replacements in measurement-rule 2 (GHSA/npm advisory mining, or the CR-Bench recipe over + SWE-Bench Multimodal) — the original c-CRAB / CR-Bench TS/JS import was scoped 2026-08-02 and + is not viable. 6. Mine the override-valve history + human review comments (decoy/gold sources already banked). 7. Optional: weekly scheduled routine (mine → propose → adapt → draft corpus PR).