diff --git a/docs/benchmarks/corpus-growth.md b/docs/benchmarks/corpus-growth.md index a0928184..28e99c0e 100644 --- a/docs/benchmarks/corpus-growth.md +++ b/docs/benchmarks/corpus-growth.md @@ -155,4 +155,7 @@ runner); freshness goes stale exactly when one moves. SWE-Bench Multimodal) — the original c-CRAB / CR-Bench TS/JS import was scoped 2026-08-02 and is not viable. 6. Mine the override-valve history + human review comments (decoy/gold sources already banked). -7. Optional: weekly scheduled routine (mine → propose → adapt → draft corpus PR). +7. ~~Weekly scheduled routine~~ — DONE 2026-08-03 (sc-1415): `propose/weekly-mining.sh` runs + both miners + both propose stages weekly via the owner's local crontab (Mon 09:00), + notify-only — mine-telemetry needs this machine's collector db, so local cron, not cloud; + adaptation stays a judgment session and is never automated. diff --git a/docs/decisions/benchmarks-grow-from-telemetry.md b/docs/decisions/benchmarks-grow-from-telemetry.md index 522f3c8b..4c561da2 100644 --- a/docs/decisions/benchmarks-grow-from-telemetry.md +++ b/docs/decisions/benchmarks-grow-from-telemetry.md @@ -24,3 +24,4 @@ created: 2026-08-01 - 2026-08-03 — sc-1416 triage of the 6 corpus rows minted from now-rebutted threads (post-#315 bot-login fix): ALL SIX stored labels stand — 3 threads were acknowledged-valid-but-deferred (sc-1055/1056/1010), 1 was fixed in a different file (line-touched missed it), 2 had false-positive/moot sources whose fixtures remain valid synthetic reproductions (one provenance-metadata correction: pr172's stale resolved+line-touched claim removed). Zero confirmed label errors — the 2/48 ≈ 4.2% noise floor is UNCHANGED. Taxonomy caveat now load-bearing for decoy mining: outcome='rebutted' bundles refuted-on-merits with deferred-valid and moot — decoy authors must read the thread; deferral-resolved threads are NOT decoys - 2026-08-03 — rowHash scope narrowed for comparisons (behaviorHash): pairing/salvage/staleness now key on the behavior-bearing slice (reviewer, expected, expectItems, reasonPattern, repo) when both sides carry it, falling back to strict full-row rowHash for old baselines — additive, no epoch break. Forced by the accepted→stale bouncing the owner flagged: honest metadata corrections (sc-1400, sc-1416 both produced them) were paying full staleness+exclusion cost for edits that cannot change a verdict. Full rowHash retained as the written record; documentation fields (note, provenance, source, outcomeEvidence, scopeConfirmed, caseId, difficulty, holdout) excluded from the comparison key - 2026-08-03 — sc-1408 ratified + built (owner chose now, option (a)): mine-ghsa.mts sweeps npm GitHub Security Advisories keeping only fix-commit-anchored entries (165/400 on first sweep); propose/propose-ghsa.mts enriches each with the fix commit's per-file patches — pre-image = confirmed-vulnerable, commit = confirmed fix. Adapted rows will carry provenance:'known-answer' (new enum value), the only provenance whose golds support ABSOLUTE recall. SecBench.js deliberately unread (unlicensed — index only). Option (b) (CR-Bench recipe over SWE-Bench Multimodal, correctness suite) deferred post-epic. First adapted batch = the remaining sc-1408 step +- 2026-08-03 — sc-1415 weekly routine ruled + built (owner chose automation over the wait-for-toil recommendation): local cron (Mon 09:00) runs propose/weekly-mining.sh — both miners + both propose stages, notify-only into ~/.claude-usage/weekly-mining.log. Local-not-cloud because mine-telemetry needs this machine's collector db/diff archive; adaptation (fixture authoring) deliberately stays un-automated — it is the judgment step. The cron entry points at the main checkout so it always runs the merged code diff --git a/gate-engine/review/eval/reviewers/propose/weekly-mining.sh b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh new file mode 100755 index 00000000..5f7f2865 --- /dev/null +++ b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh @@ -0,0 +1,59 @@ +#!/bin/sh +# Weekly corpus-growth sweep (sc-1415, epic 1399): run both miners + both propose stages, +# append the funnel summaries to a log, and pop a notification. NOTIFY-ONLY by design — the +# adapt stage (fixture authoring) is the judgment step and stays a human/agent session; this +# script never writes to cases-*.jsonl and never commits. +# +# Installed into the owner's crontab (Mondays 09:00 local): +# 0 9 * * 1 "/gate-engine/review/eval/reviewers/propose/weekly-mining.sh" +# cron ships a minimal environment, so PATH is set explicitly (bun + homebrew + gh). +# mine-telemetry needs THIS machine's ~/.claude-usage db + diff archive — that is why this is a +# local cron job and not a cloud routine (recorded on the sc-1415 ticket). +# +# Each stage's EXIT STATUS is captured before its output is tailed into the log (a plain +# `cmd | tail` pipeline would report tail's status and mask miner failures — review finding on +# #321). A failed stage marks the sweep failed: propose-telemetry is skipped when its miner +# failed, the notification says FAILED naming the stages, and the script exits non-zero so cron +# records the failure instead of a false "refreshed". +set -u +PATH="$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" +export PATH + +cd "$(dirname "$0")/.." || exit 1 +LOG="${HOME}/.claude-usage/weekly-mining.log" +TMP="$(mktemp)" || exit 1 +trap 'rm -f "$TMP"' EXIT +FAILED="" + +# run_stage