From 3720c013f277e554a4b0918b0a3155b0795337cc Mon Sep 17 00:00:00 2001 From: norvalbv Date: Mon, 3 Aug 2026 13:41:10 +0100 Subject: [PATCH 1/2] =?UTF-8?q?bench(reviewer-eval):=20weekly=20mining=20r?= =?UTF-8?q?outine=20=E2=80=94=20local=20cron=20sweep,=20notify-only=20(sc-?= =?UTF-8?q?1415)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit propose/weekly-mining.sh runs both miners (mine-bots, mine-telemetry) and both propose stages weekly, appending funnel summaries to ~/.claude-usage/weekly-mining.log and popping a notification. Registered in the owner's crontab (Mon 09:00), pointed at the main checkout so it always runs merged code. Deliberately notify-only and local: mine-telemetry needs this machine's collector db + diff archive (a cloud routine could only cover the bot half), and adaptation stays a human/agent judgment session — the script never touches cases-*.jsonl and never commits. Runbook pending item 7 closed; decision note recorded on the epic axis. Co-Authored-By: Claude Fable 5 --- docs/benchmarks/corpus-growth.md | 5 ++- .../benchmarks-grow-from-telemetry.md | 1 + .../eval/reviewers/propose/weekly-mining.sh | 32 +++++++++++++++++++ 3 files changed, 37 insertions(+), 1 deletion(-) create mode 100755 gate-engine/review/eval/reviewers/propose/weekly-mining.sh diff --git a/docs/benchmarks/corpus-growth.md b/docs/benchmarks/corpus-growth.md index a0928184..28e99c0e 100644 --- a/docs/benchmarks/corpus-growth.md +++ b/docs/benchmarks/corpus-growth.md @@ -155,4 +155,7 @@ runner); freshness goes stale exactly when one moves. SWE-Bench Multimodal) — the original c-CRAB / CR-Bench TS/JS import was scoped 2026-08-02 and is not viable. 6. Mine the override-valve history + human review comments (decoy/gold sources already banked). -7. Optional: weekly scheduled routine (mine → propose → adapt → draft corpus PR). +7. ~~Weekly scheduled routine~~ — DONE 2026-08-03 (sc-1415): `propose/weekly-mining.sh` runs + both miners + both propose stages weekly via the owner's local crontab (Mon 09:00), + notify-only — mine-telemetry needs this machine's collector db, so local cron, not cloud; + adaptation stays a judgment session and is never automated. diff --git a/docs/decisions/benchmarks-grow-from-telemetry.md b/docs/decisions/benchmarks-grow-from-telemetry.md index 522f3c8b..4c561da2 100644 --- a/docs/decisions/benchmarks-grow-from-telemetry.md +++ b/docs/decisions/benchmarks-grow-from-telemetry.md @@ -24,3 +24,4 @@ created: 2026-08-01 - 2026-08-03 — sc-1416 triage of the 6 corpus rows minted from now-rebutted threads (post-#315 bot-login fix): ALL SIX stored labels stand — 3 threads were acknowledged-valid-but-deferred (sc-1055/1056/1010), 1 was fixed in a different file (line-touched missed it), 2 had false-positive/moot sources whose fixtures remain valid synthetic reproductions (one provenance-metadata correction: pr172's stale resolved+line-touched claim removed). Zero confirmed label errors — the 2/48 ≈ 4.2% noise floor is UNCHANGED. Taxonomy caveat now load-bearing for decoy mining: outcome='rebutted' bundles refuted-on-merits with deferred-valid and moot — decoy authors must read the thread; deferral-resolved threads are NOT decoys - 2026-08-03 — rowHash scope narrowed for comparisons (behaviorHash): pairing/salvage/staleness now key on the behavior-bearing slice (reviewer, expected, expectItems, reasonPattern, repo) when both sides carry it, falling back to strict full-row rowHash for old baselines — additive, no epoch break. Forced by the accepted→stale bouncing the owner flagged: honest metadata corrections (sc-1400, sc-1416 both produced them) were paying full staleness+exclusion cost for edits that cannot change a verdict. Full rowHash retained as the written record; documentation fields (note, provenance, source, outcomeEvidence, scopeConfirmed, caseId, difficulty, holdout) excluded from the comparison key - 2026-08-03 — sc-1408 ratified + built (owner chose now, option (a)): mine-ghsa.mts sweeps npm GitHub Security Advisories keeping only fix-commit-anchored entries (165/400 on first sweep); propose/propose-ghsa.mts enriches each with the fix commit's per-file patches — pre-image = confirmed-vulnerable, commit = confirmed fix. Adapted rows will carry provenance:'known-answer' (new enum value), the only provenance whose golds support ABSOLUTE recall. SecBench.js deliberately unread (unlicensed — index only). Option (b) (CR-Bench recipe over SWE-Bench Multimodal, correctness suite) deferred post-epic. First adapted batch = the remaining sc-1408 step +- 2026-08-03 — sc-1415 weekly routine ruled + built (owner chose automation over the wait-for-toil recommendation): local cron (Mon 09:00) runs propose/weekly-mining.sh — both miners + both propose stages, notify-only into ~/.claude-usage/weekly-mining.log. Local-not-cloud because mine-telemetry needs this machine's collector db/diff archive; adaptation (fixture authoring) deliberately stays un-automated — it is the judgment step. The cron entry points at the main checkout so it always runs the merged code diff --git a/gate-engine/review/eval/reviewers/propose/weekly-mining.sh b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh new file mode 100755 index 00000000..b3a3ed9d --- /dev/null +++ b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh @@ -0,0 +1,32 @@ +#!/bin/sh +# Weekly corpus-growth sweep (sc-1415, epic 1399): run both miners + both propose stages, +# append the funnel summaries to a log, and pop a notification. NOTIFY-ONLY by design — the +# adapt stage (fixture authoring) is the judgment step and stays a human/agent session; this +# script never writes to cases-*.jsonl and never commits. +# +# Installed into the owner's crontab (Mondays 09:00 local): +# 0 9 * * 1 "/gate-engine/review/eval/reviewers/propose/weekly-mining.sh" +# cron ships a minimal environment, so PATH is set explicitly (bun + homebrew + gh). +# mine-telemetry needs THIS machine's ~/.claude-usage db + diff archive — that is why this is a +# local cron job and not a cloud routine (recorded on the sc-1415 ticket). +set -u +PATH="$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" +export PATH + +cd "$(dirname "$0")/.." || exit 1 +LOG="${HOME}/.claude-usage/weekly-mining.log" + +{ + echo "=== weekly mining sweep $(date -u '+%Y-%m-%dT%H:%MZ') ===" + echo "--- mine-bots ---" + bun mine-bots.mts 2>&1 | tail -6 + echo "--- mine-telemetry ---" + bun mine-telemetry.mts 2>&1 | tail -8 + echo "--- propose (bot-mined, correctness) ---" + bun propose/propose.mts --suite correctness --max 10 2>&1 | tail -4 + echo "--- propose (telemetry) ---" + bun propose/propose-telemetry.mts --max 10 2>&1 | tail -4 + echo "" +} >>"$LOG" 2>&1 + +osascript -e 'display notification "Corpus sweep done — queues refreshed. See ~/.claude-usage/weekly-mining.log" with title "devkit weekly mining"' 2>/dev/null || true From 1008610b1cd0eee913743ce891c21cea10629b12 Mon Sep 17 00:00:00 2001 From: norvalbv Date: Mon, 3 Aug 2026 15:10:49 +0100 Subject: [PATCH 2/2] =?UTF-8?q?bench(reviewer-eval):=20weekly=20sweep=20tr?= =?UTF-8?q?acks=20per-stage=20exit=20status=20=E2=80=94=20review=20finding?= =?UTF-8?q?=20on=20#321?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stages ran as `bun ... | tail` pipelines whose status came from tail, so a miner failure was masked, the notification still said "queues refreshed", and the trailing `|| true` let cron record success. Each stage's own exit status is now captured before its output is tailed into the log; a failure marks the sweep failed, propose-telemetry is skipped when its miner failed, the notification names the failed stages, and the script exits non-zero so cron records it. (The same pipe-masks-status trap this session already hit with backgrounded pushes — now fixed at both sites.) Co-Authored-By: Claude Fable 5 --- .../eval/reviewers/propose/weekly-mining.sh | 51 ++++++++++++++----- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/gate-engine/review/eval/reviewers/propose/weekly-mining.sh b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh index b3a3ed9d..5f7f2865 100755 --- a/gate-engine/review/eval/reviewers/propose/weekly-mining.sh +++ b/gate-engine/review/eval/reviewers/propose/weekly-mining.sh @@ -9,24 +9,51 @@ # cron ships a minimal environment, so PATH is set explicitly (bun + homebrew + gh). # mine-telemetry needs THIS machine's ~/.claude-usage db + diff archive — that is why this is a # local cron job and not a cloud routine (recorded on the sc-1415 ticket). +# +# Each stage's EXIT STATUS is captured before its output is tailed into the log (a plain +# `cmd | tail` pipeline would report tail's status and mask miner failures — review finding on +# #321). A failed stage marks the sweep failed: propose-telemetry is skipped when its miner +# failed, the notification says FAILED naming the stages, and the script exits non-zero so cron +# records the failure instead of a false "refreshed". set -u PATH="$HOME/.bun/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" export PATH cd "$(dirname "$0")/.." || exit 1 LOG="${HOME}/.claude-usage/weekly-mining.log" +TMP="$(mktemp)" || exit 1 +trap 'rm -f "$TMP"' EXIT +FAILED="" + +# run_stage