You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
enhancement: recovered judge verdicts are indistinguishable from clean ones in report.json #321
When the judge agent errors but leaves a salvageable FinalMessage, agentCallError intentionally recovers it (CHANGELOG-documented and unit-tested behavior — this request does not question that design; the recovered output passes the same strict validation as a clean one).
The gap is observability: in report.json, a recovered verdict is indistinguishable from a clean one. The recovery is WARN-logged to the console, but nothing machine-readable records it — so a CI gate consuming reports cannot tell "judge was healthy" from "judge crashed/timed out and its output was salvaged". Across a benchmark run, a degrading judge engine is invisible exactly where judgments are aggregated.
Notably, the precedent already exists in the same schema: gate-short-circuited criteria carry "skipped": true, and judge.Result has skip_reason/error_reason slots — skill-up already treats abnormal verdicts as explainable; the recovery path just isn't wired into it.
Proposed solution
Surface the recovery in report.json (not grading.json, which is an Anthropic-compatible contract). The plumbing mostly exists: SessionResult.Warnings is already a field and buildResult already holds the session — carrying the recovery warning along that channel would make the change small. Any machine-readable marker (e.g. a recovered: true / warning entry on the judge result) solves it.
Alternatives considered
Do nothing (console WARN is enough for humans) — leaves automated consumers blind; infrastructure noise then hides inside aggregate scores.
Put it in grading.json — rejected: that file is an Anthropic-compatible contract.
Additional context
Found via deterministic fault injection of the judge channel (custom engine emitting exit_code=124 with a valid final_message): recovered verdicts produce reports structurally identical to healthy controls except timing fields. Related: #319 / #320 (found in the same fault-injection pass). The gate-skip path already demonstrates the desired disclosure shape ("skipped": true on judge criteria when the expect gate fails).
Implemented this as PR #332: it preserves a machine-readable judge_recovery marker in report.json for recovered agent_judge results, distinguishes the initial and correction-call recovery paths without serializing provider error text, and keeps grading.json unchanged.
Fixed on main by #332. Recovered judge verdicts now carry a machine-readable judge_recovery marker in report.json, while grading.json remains unchanged. Closing as completed. Thanks for reporting!
Activity
huong8373tt-beep commentedon Oct 21, 2026
Implemented this as PR #332: it preserves a machine-readable
judge_recoverymarker inreport.jsonfor recoveredagent_judgeresults, distinguishes the initial and correction-call recovery paths without serializing provider error text, and keepsgrading.jsonunchanged.zpzjzj commentedon Oct 23, 2026
Fixed on main by #332. Recovered judge verdicts now carry a machine-readable judge_recovery marker in report.json, while grading.json remains unchanged. Closing as completed. Thanks for reporting!