eval: monthly review-quality probe of omni-reviewbot as a harness arm - #109
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cdef1de798
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| out = output or {} | ||
| if not out: | ||
| return False # ran, no usable final — keep its route, re-ask |
There was a problem hiding this comment.
When DeepSeekHarnessTransport.run_session() catches an authentication, quota, timeout, or SDK exception, it returns an AgentOutcome with empty text (providers/deepseek.py:410-417,453-461); run_agent_step() consequently returns (StepResult(False), {}) (runner.py:208-211). This branch classifies that empty output as a live/truncated seat, so the routed retry remains on the same dead member and the tier fallback added by this change never runs, ultimately dropping the lens from the ensemble. Empty output therefore cannot unconditionally return False here without also consulting the transport failure/refusal signal.
Useful? React with 👍 / 👎.
| matched = MONTH_TAG.match(args.tag) | ||
| name = matched.group(1) if matched else args.tag | ||
| out = DS / "results" / "reviewbot" / f"REVIEWBOT_{name}.md" |
There was a problem hiding this comment.
When the documented smoke command passes smoke, the monthly script constructs the tag reviewbot_smoke; this fallback keeps that entire tag and therefore writes REVIEWBOT_reviewbot_smoke.md, not the documented results/reviewbot/REVIEWBOT_smoke.md. Any operator or automation following the runbook will look for the wrong output after every non-month run.
Useful? React with 👍 / 👎.
One command — `bash eval/dataset/run_reviewbot_monthly.sh 2026-09` —
generates fresh omni-reviewbot Direct reviews over the dataset's
train+val PR items (15; test stays frozen) and judges them blind against
the pinned claudecode_opus5 baseline with the pinned claude-sonnet-5
judge, 3 generation x 3 judge replicates, reported with the campaign's
paired-within-verdict, item-clustered statistics. The bot is a
loop+model combination: results live in results/reviewbot/ and are
never merged into model_comparison.md.
run_reviewbot_arm.py enforces the measurement contract per invocation,
fail-closed: POST_MODE=shadow asserted from CLI output (nothing can
reach GitHub); REVIEW_CONTEXT_MODE=no_discussion asserted via the bot's
printed provenance line (on these PRs the historical review threads ARE
the ground truth); reviewed head pinned to expected_pr_heads.json and
changed_files matched to the frozen GT diff; sanitized artifacts
(markers stripped, neutral heading — labels must not reach the judge)
must fit the judge's 24k cap. Replication has one owner (the runner:
ARM_TAG -> arms/<tag>_r1..rN), manifests record resolved config and
refuse to mix a changed bot into a half-generated arm, and the cursor
provider is refused without the contamination-vault ack.
build_reviewbot_report.py --verify audits exact denominators before any
statistics: judge_val.py skips existing verdicts and succeeds on empty
job sets, so the verifier independently checks {manifest stems} x
{judge reps} per replicate dir, pinned judge identity, and each
verdict's arm/baseline sha256 against the CURRENT capped candidate
texts — stale judgments fail here, not in the numbers. Reports are
written atomically; a <YYYY-MM> tag upserts one row in
results/reviewbot/INDEX.md (rerun replaces, never duplicates); smoke
tags never touch the series.
Companion: omni-reviewbot#20 (REVIEW_CONTEXT_MODE + printed provenance).
Offline coverage in test/test_reviewbot_eval.py (10 cases).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PchwdRpCaWWiRbJDVHKUiD
cdef1de to
4a7d8fe
Compare
1 participant
A monthly, one-command review-quality probe of the live omni-reviewbot configuration, reusing this repo's eval arms: dataset train+val PR items (15; test stays frozen), blind pairwise judging against the pinned
baselines/claudecode_opus5with the pinnedclaude-sonnet-5judge (3 generation × 3 judge replicates), reported with the campaign's paired-within-verdict, item-clustered statistics.Companion PR: zuiho-kai/omni-reviewbot#20 (
REVIEW_CONTEXT_MODE=no_discussion+ printed provenance) — the bot normally injects the PR's historical review threads into the prompt, and on dataset PRs those threads are the GT; every existing runner here enforces the same no-discussion rule.What the harness enforces (fail-closed, per item)
status: shadowandreview_context: no_discussion (0 threads)asserted from the bot's CLI output; reviewed head pinned togoal-eval/expected_pr_heads.json;changed_filesmatched to the frozen GT diff; sanitized artifacts (markers stripped, neutral heading — labels must not reach the blind judge) must fit the judge's 24k cap.build_reviewbot_report.py --verify):judge_val.pyskips existing verdicts and succeeds on empty job sets, so the verifier independently checks {manifest stems}×{judge reps} per replicate dir, pinned judge identity, strict verdict schema (blinding map, all rubric values in [0,1], winner, gap_hit — malformed verdicts abort rather than shrink denominators), and each verdict's arm/baseline sha256 against the current capped candidate texts (stale judgments fail here, not in the numbers).<YYYY-MM>tags upsert one row inresults/reviewbot/INDEX.md(rerun replaces, never duplicates); smoke tags never touch the series; GOLD latent-gap items are reported even on a 0/n double miss. The fixed header states: monitoring probe, not a gate; loop+model numbers, never merged intomodel_comparison.md(2026-08-14 decision).Verification
python -m pytest test/test_reviewbot_eval.py— 18 offline cases: sanitizer, train+val pinning, dry-run isolation, paired-delta math vs hand-computed values, missing/extra/stale/malformed-verdict aborts, judge pinning, INDEX idempotency, provenance fingerprint (dirty refusal, secret exclusion, resume mismatch).🤖 Generated with Claude Code
https://claude.ai/code/session_01PchwdRpCaWWiRbJDVHKUiD