bench(reviewer-eval): GHSA known-answer pipeline — absolute recall unblocked for the security suites (sc-1408)

docs/benchmarks/corpus-growth.md

@@ -86,7 +86,7 @@ runner); freshness goes stale exactly when one moves.
    CR-Bench arXiv 2603.11078 "filtered to TS/JS") is not viable**: both are Python-derived
    (c-CRAB builds on SWE-CARE, CR-Bench transforms SWE-Bench — neither contains any TS/JS
    instances to filter), c-CRAB's artifact repo (github.com/c-CRAB-Benchmark/dataset) carries no
-   license, and CR-Bench has no publicly released artifact. Proposed replacements, unratified:
+   license, and CR-Bench has no publicly released artifact. RATIFIED 2026-08-03 (owner): replacement (a) is BUILT — `mine-ghsa.mts` + `propose/propose-ghsa.mts` sweep npm advisories with fix-commit anchors (165 anchored candidates on the first sweep); adapted rows carry `provenance:'known-answer'`, the only provenance whose golds support absolute-recall claims. (b) stays deferred post-epic. Original options for the record:
    (a) mine GHSA/npm advisories with fix commits directly — public known-answer facts,
    re-expressed as anonymized fixtures like every other row, the natural api-security /
    frontend-security source (SecBench.js catalogs ~600 such vulns but is itself unlicensed —

docs/decisions/benchmarks-grow-from-telemetry.md

@@ -23,3 +23,4 @@ created: 2026-08-01
 - 2026-08-02 — the Target's known-answer path (c-CRAB/CR-Bench 'filtered to TS/JS') was falsified by scoping (#307): both are Python-derived (SWE-CARE / SWE-Bench), c-CRAB's artifact is unlicensed, CR-Bench has no publicly released artifact — no TS/JS slice exists to filter. Proposed replacements awaiting ratification: GHSA/npm advisory mining with fix commits (security suites), or CR-Bench's transformation recipe over SWE-Bench Multimodal's JS/TS repos (correctness). Absolute recall stays blocked until one is built
 - 2026-08-03 — sc-1416 triage of the 6 corpus rows minted from now-rebutted threads (post-#315 bot-login fix): ALL SIX stored labels stand — 3 threads were acknowledged-valid-but-deferred (sc-1055/1056/1010), 1 was fixed in a different file (line-touched missed it), 2 had false-positive/moot sources whose fixtures remain valid synthetic reproductions (one provenance-metadata correction: pr172's stale resolved+line-touched claim removed). Zero confirmed label errors — the 2/48 ≈ 4.2% noise floor is UNCHANGED. Taxonomy caveat now load-bearing for decoy mining: outcome='rebutted' bundles refuted-on-merits with deferred-valid and moot — decoy authors must read the thread; deferral-resolved threads are NOT decoys
 - 2026-08-03 — rowHash scope narrowed for comparisons (behaviorHash): pairing/salvage/staleness now key on the behavior-bearing slice (reviewer, expected, expectItems, reasonPattern, repo) when both sides carry it, falling back to strict full-row rowHash for old baselines — additive, no epoch break. Forced by the accepted→stale bouncing the owner flagged: honest metadata corrections (sc-1400, sc-1416 both produced them) were paying full staleness+exclusion cost for edits that cannot change a verdict. Full rowHash retained as the written record; documentation fields (note, provenance, source, outcomeEvidence, scopeConfirmed, caseId, difficulty, holdout) excluded from the comparison key
+- 2026-08-03 — sc-1408 ratified + built (owner chose now, option (a)): mine-ghsa.mts sweeps npm GitHub Security Advisories keeping only fix-commit-anchored entries (165/400 on first sweep); propose/propose-ghsa.mts enriches each with the fix commit's per-file patches — pre-image = confirmed-vulnerable, commit = confirmed fix. Adapted rows will carry provenance:'known-answer' (new enum value), the only provenance whose golds support ABSOLUTE recall. SecBench.js deliberately unread (unlicensed — index only). Option (b) (CR-Bench recipe over SWE-Bench Multimodal, correctness suite) deferred post-epic. First adapted batch = the remaining sc-1408 step

gate-engine/review/eval/reviewers/corpus.mts

@@ -40,7 +40,9 @@ export const casesFile = (reviewer) => path.join(here, `cases-${reviewer.skill}.
 const ROW_ENUMS = {
   expected: ['FAIL', 'PASS'],
   difficulty: ['clear', 'borderline', 'adversarial'],
-  provenance: ['authored', 'mined', 'adapted'],
+  // 'known-answer' = adapted from an external ground truth (GHSA fix commit — sc-1408):
+  // the only provenance whose golds support ABSOLUTE recall claims (methodology item 17).
+  provenance: ['authored', 'mined', 'adapted', 'known-answer'],
   // Optional provenance/labeling fields (mined-corpus tooling) — absent is fine, present-but-wrong
   // is a lint failure like every other enum here.
   outcomeEvidence: [

gate-engine/review/eval/reviewers/mine-ghsa.mts

@@ -0,0 +1,120 @@
+#!/usr/bin/env node
+// @ts-nocheck — BENCH-ONLY (excluded from tsc, see tsconfig.json exclude); loose types deliberate.
+
+/**
+ * mine-ghsa — sweep GitHub Security Advisories (npm ecosystem) into KNOWN-ANSWER candidates for
+ * the security suites. This is the ratified replacement (sc-1408) for the falsified
+ * c-CRAB/CR-Bench TS/JS import (#307): advisories with fix commits are public known-answer
+ * facts — the commit's parent tree is a confirmed-vulnerable state and the commit is the
+ * confirmed fix — giving the security corpora their first ABSOLUTE-recall anchor (methodology
+ * item 17: mined golds measure only precision + relative recall).
+ *
+ *   bun mine-ghsa.mts [--pages N] [--severity high,critical]   (defaults: 5 pages, high+critical)
+ *
+ * Keeps only advisories carrying at least one /commit/ reference (the known-answer anchor).
+ * Output: raw/candidates-ghsa.jsonl (gitignored), merged by ghsa url (new wins) — same contract
+ * as the other miners. Fixture authorship happens downstream (propose/propose-ghsa.mts →
+ * anonymized adapt session); nothing here writes corpus rows. SecBench.js is deliberately NOT
+ * read (unlicensed — usable as an index only, per #307); the advisory API is the source of truth.
+ *
+ * Read-only against GitHub (gh api). Facts (CVE ids, versions, commit shas) are not
+ * copyrightable; prose summaries are capped and never reach the public corpus verbatim — the
+ * adapt stage re-expresses everything as generic-identifier fixtures like every other row.
+ */
+
+import { execFileSync } from 'node:child_process';
+import { existsSync, mkdirSync, renameSync, writeFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { readJsonl } from './propose/propose-common.mts';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const RAW_DIR = path.join(here, 'raw');
+const OUT = path.join(RAW_DIR, 'candidates-ghsa.jsonl');
+const SUMMARY_CAP = 2000;
+const COMMIT_REF_RE = /\/commit\/([0-9a-f]{7,40})/;
+// GitHub Link header: `<...&after=CURSOR>; rel="next"` — cursor pagination for /advisories.
+const LINK_NEXT_RE = /<[^>]*[?&]after=([^&>]+)[^>]*>;\s*rel="next"/;
+
+function gh(args) {
+  return execFileSync('gh', args, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
+}
+
+function parseArgs(argv) {
+  const pagesIdx = argv.indexOf('--pages');
+  const pages = pagesIdx !== -1 ? Number.parseInt(argv[pagesIdx + 1], 10) : 5;
+  const sevIdx = argv.indexOf('--severity');
+  const severities = (sevIdx !== -1 ? argv[sevIdx + 1] : 'high,critical').split(',');
+  return { pages, severities };
+}
+
+function fetchAdvisories(pages, severity) {
+  const rows = [];
+  let cursor = null;
+  for (let page = 0; page < pages; page += 1) {
+    const after = cursor ? `&after=${encodeURIComponent(cursor)}` : '';
+    const raw = gh([
+      'api',
+      '-i',
+      `/advisories?ecosystem=npm&severity=${severity}&per_page=100${after}`,
+    ]);
+    const [head, body] = [
+      raw.slice(0, raw.indexOf('\r\n\r\n')),
+      raw.slice(raw.indexOf('\r\n\r\n')),
+    ];
+    rows.push(...JSON.parse(body));
+    const next = LINK_NEXT_RE.exec(head);
+    if (!next) break;
+    cursor = decodeURIComponent(next[1]);
+  }
+  return rows;
+}
+
+function main() {
+  const { pages, severities } = parseArgs(process.argv.slice(2));
+  const merged = new Map();
+  if (existsSync(OUT)) for (const row of readJsonl(OUT)) merged.set(row.url, row);
+
+  let seen = 0;
+  let anchored = 0;
+  for (const severity of severities) {
+    for (const a of fetchAdvisories(pages, severity)) {
+      seen += 1;
+      const fixCommits = (a.references ?? []).filter((r) => COMMIT_REF_RE.test(r));
+      if (fixCommits.length === 0) continue; // no known-answer anchor — skip
+      anchored += 1;
+      // Multi-ecosystem advisories (e.g. GHSA-85rg-p3fr-xc2f: maven+pip+npm) order entries
+      // arbitrarily — take the npm entry specifically, or the package/version fields describe
+      // the wrong ecosystem's artifact.
+      const vuln = (a.vulnerabilities ?? []).find((v) => v?.package?.ecosystem === 'npm') ?? {};
+      merged.set(a.html_url, {
+        kind: 'ghsa',
+        url: a.html_url,
+        ghsaId: a.ghsa_id,
+        cveId: a.cve_id ?? null,
+        severity: a.severity,
+        cwes: a.cwe_ids ?? [],
+        package: vuln.package?.name ?? null,
+        vulnerableRange: vuln.vulnerable_version_range ?? null,
+        firstPatched: vuln.first_patched_version ?? null,
+        publishedAt: a.published_at,
+        summary: String(a.summary ?? '').slice(0, SUMMARY_CAP),
+        description: String(a.description ?? '').slice(0, SUMMARY_CAP),
+        fixCommits,
+      });
+    }
+  }
+
+  mkdirSync(RAW_DIR, { recursive: true });
+  const rows = [...merged.values()];
+  const tmp = `${OUT}.tmp`;
+  writeFileSync(tmp, `${rows.map((r) => JSON.stringify(r)).join('\n')}\n`);
+  renameSync(tmp, OUT);
+  console.error(
+    `mine-ghsa: ${seen} advisories swept → ${anchored} with fix-commit anchors this run → ${rows.length} total in ${path.relative(here, OUT)}`,
+  );
+}
+
+const invokedDirectly =
+  process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url);
+if (invokedDirectly) main();

gate-engine/review/eval/reviewers/propose/propose-common.mts

@@ -7,7 +7,9 @@
  * clone gate flagged the duplicated drop-histogram block between them.
  */
 
-import { readFileSync } from 'node:fs';
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
 
 /** kebab slug from free text, for queue ids. */
 export function slugify(text, maxWords = 5) {
@@ -60,3 +62,53 @@ export function makeDropCounter() {
   };
   return { drops, bump };
 }
+
+/** Parse `--max N` (positive integer) or exit 2 under the script's name. */
+export function parseMaxArg(argv, name, fallback = 10) {
+  const maxIdx = argv.indexOf('--max');
+  const max = maxIdx !== -1 ? Number.parseInt(argv[maxIdx + 1], 10) : fallback;
+  if (!Number.isFinite(max) || max <= 0) {
+    console.error(`${name}: --max must be a positive integer`);
+    process.exit(2);
+  }
+  return max;
+}
+
+/** Exit 2 with a remedy hint when a required input file is absent. */
+export function requireFile(file, name, hint) {
+  if (!existsSync(file)) {
+    console.error(`${name}: missing ${path.basename(file)} — ${hint}`);
+    process.exit(2);
+  }
+}
+
+/** Run the hard-drop filter over candidates, bumping the histogram; returns survivors. */
+export function partitionDrops(candidates, hardDropReason, bump) {
+  const kept = [];
+  for (const c of candidates) {
+    const dropReason = hardDropReason(c);
+    if (dropReason) bump(dropReason);
+    else kept.push(c);
+  }
+  return kept;
+}
+
+/** Write a queue file (one JSON line per row), creating raw/ if needed. */
+export function writeQueue(outFile, rows) {
+  mkdirSync(path.dirname(outFile), { recursive: true });
+  writeFileSync(outFile, `${rows.map((r) => JSON.stringify(r)).join('\n')}\n`);
+}
+
+/** stderr triage summary: script-specific lines + the shared drops/output-path tail. */
+export function printSummary(lines, drops, baseDir, outFile) {
+  console.error(
+    [...lines, `  drops: ${formatDrops(drops)}`, `  → ${path.relative(baseDir, outFile)}`].join(
+      '\n',
+    ),
+  );
+}
+
+/** Invoke main() only when this module is the direct CLI entrypoint. */
+export function runIfMain(metaUrl, main) {
+  if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(metaUrl)) main();
+}

gate-engine/review/eval/reviewers/propose/propose-ghsa.mts

@@ -0,0 +1,121 @@
+#!/usr/bin/env node
+// @ts-nocheck — BENCH-ONLY (excluded from tsc, see tsconfig.json exclude); loose types deliberate.
+
+/**
+ * propose-ghsa — deterministic triage of raw/candidates-ghsa.jsonl (mine-ghsa's output) into an
+ * adaptation queue for the security suites. Known-answer path (sc-1408): every queued entry
+ * carries the FIX COMMIT's per-file patches fetched from the advisory's repo, so the adapt
+ * session sees the confirmed-vulnerable shape (patch pre-image) and the confirmed fix — gold =
+ * vulnerable state (expected FAIL, absolute-recall row), minimal-pair decoy = fix applied.
+ *
+ *   bun propose/propose-ghsa.mts [--max N]   (default 10)
+ *
+ * Pipeline: HARD DROPS (counted) → SORT (severity, then recency) → ENRICH (gh api commit —
+ * files + patches; a fetch failure drops the entry and the next-ranked is tried) → WRITE
+ * raw/queue-ghsa.jsonl. Suite routing happens at adaptation (api-security vs frontend-security
+ * is a fixture-placement judgment, not a path heuristic — advisories have no repo-relative
+ * frontend/backend split).
+ */
+
+import { execFileSync } from 'node:child_process';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import {
+  makeDropCounter,
+  parseMaxArg,
+  partitionDrops,
+  printSummary,
+  readJsonl,
+  requireFile,
+  runIfMain,
+  slugify,
+  uniqueId,
+  writeQueue,
+} from './propose-common.mts';
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+const reviewersDir = path.join(here, '..');
+const CANDIDATES_FILE = path.join(reviewersDir, 'raw', 'candidates-ghsa.jsonl');
+const OUT_FILE = path.join(reviewersDir, 'raw', 'queue-ghsa.jsonl');
+const COMMIT_URL_RE = /github\.com\/([^/]+)\/([^/]+)\/commit\/([0-9a-f]{7,40})/;
+const PATCH_CAP = 8000;
+const SEVERITY_RANK = { critical: 0, high: 1, medium: 2, low: 3 };
+
+function gh(args) {
+  return execFileSync('gh', args, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
+}
+
+/** Returns a drop reason string, or null if the candidate survives. */
+function hardDropReason(c) {
+  if (c.kind !== 'ghsa') return `unknown-kind:${c.kind}`;
+  if (!Array.isArray(c.fixCommits) || c.fixCommits.length === 0) return 'no-fix-commit';
+  if (!COMMIT_URL_RE.test(c.fixCommits[0])) return 'unparseable-commit-url';
+  return null;
+}
+
+function compareCandidates(a, b) {
+  const bySev = (SEVERITY_RANK[a.severity] ?? 4) - (SEVERITY_RANK[b.severity] ?? 4);
+  if (bySev !== 0) return bySev;
+  return Date.parse(b.publishedAt ?? 0) - Date.parse(a.publishedAt ?? 0);
+}
+
+/** Fix-commit files + patches — the known-answer evidence the adapt session works from. */
+function fetchFixCommit(commitUrl) {
+  const [, owner, repo, sha] = COMMIT_URL_RE.exec(commitUrl);
+  const raw = JSON.parse(gh(['api', `repos/${owner}/${repo}/commits/${sha}`]));
+  return {
+    repo: `${owner}/${repo}`,
+    sha,
+    files: (raw.files ?? []).map((f) => ({
+      filename: f.filename,
+      status: f.status,
+      patch: String(f.patch ?? '').slice(0, PATCH_CAP),
+    })),
+  };
+}
+
+function main() {
+  const max = parseMaxArg(process.argv.slice(2), 'propose-ghsa');
+  requireFile(CANDIDATES_FILE, 'propose-ghsa', 'run mine-ghsa.mts first');
+
+  const candidates = readJsonl(CANDIDATES_FILE);
+  const { drops, bump } = makeDropCounter();
+  const kept = partitionDrops(candidates, hardDropReason, bump);
+  kept.sort(compareCandidates);
+
+  const seenIds = new Set();
+  const rows = [];
+  let enrichFailures = 0;
+  for (const c of kept) {
+    if (rows.length >= max) break;
+    let fixCommit = null;
+    try {
+      fixCommit = fetchFixCommit(c.fixCommits[0]);
+    } catch (e) {
+      enrichFailures += 1;
+      console.error(
+        `propose-ghsa: enrich failed for ${c.ghsaId} — ${e.message?.split('\n')[0] ?? e}`,
+      );
+      continue;
+    }
+    rows.push({
+      queueId: uniqueId(`sec-ghsa-${slugify(c.summary)}`, seenIds),
+      knownAnswer: true,
+      candidate: c,
+      fixCommit,
+    });
+  }
+  if (enrichFailures) bump('enrich-failed', enrichFailures);
+
+  writeQueue(OUT_FILE, rows);
+  printSummary(
+    [
+      `propose-ghsa: ${candidates.length} candidates → ${kept.length} anchored → ${rows.length} queued (${enrichFailures} enrich failures)`,
+    ],
+    drops,
+    reviewersDir,
+    OUT_FILE,
+  );
+}
+
+runIfMain(import.meta.url, main);