Skip to content

enhancement: recovered judge verdicts are indistinguishable from clean ones in report.json #321

Closed

Description

@BiBoyang

Problem / motivation

When the judge agent errors but leaves a salvageable FinalMessage, agentCallError intentionally recovers it (CHANGELOG-documented and unit-tested behavior — this request does not question that design; the recovered output passes the same strict validation as a clean one).

The gap is observability: in report.json, a recovered verdict is indistinguishable from a clean one. The recovery is WARN-logged to the console, but nothing machine-readable records it — so a CI gate consuming reports cannot tell "judge was healthy" from "judge crashed/timed out and its output was salvaged". Across a benchmark run, a degrading judge engine is invisible exactly where judgments are aggregated.

Notably, the precedent already exists in the same schema: gate-short-circuited criteria carry "skipped": true, and judge.Result has skip_reason/error_reason slots — skill-up already treats abnormal verdicts as explainable; the recovery path just isn't wired into it.

Proposed solution

Surface the recovery in report.json (not grading.json, which is an Anthropic-compatible contract). The plumbing mostly exists: SessionResult.Warnings is already a field and buildResult already holds the session — carrying the recovery warning along that channel would make the change small. Any machine-readable marker (e.g. a recovered: true / warning entry on the judge result) solves it.

Alternatives considered

  • Do nothing (console WARN is enough for humans) — leaves automated consumers blind; infrastructure noise then hides inside aggregate scores.
  • Put it in grading.json — rejected: that file is an Anthropic-compatible contract.

Additional context

Found via deterministic fault injection of the judge channel (custom engine emitting exit_code=124 with a valid final_message): recovered verdicts produce reports structurally identical to healthy controls except timing fields. Related: #319 / #320 (found in the same fault-injection pass). The gate-skip path already demonstrates the desired disclosure shape ("skipped": true on judge criteria when the expect gate fails).

Activity

  1. huong8373tt-beep commented on Oct 21, 2026

    @huong8373tt-beep
    Contributor

    Implemented this as PR #332: it preserves a machine-readable judge_recovery marker in report.json for recovered agent_judge results, distinguishes the initial and correction-call recovery paths without serializing provider error text, and keeps grading.json unchanged.

  2. zpzjzj commented on Oct 23, 2026

    @zpzjzj
    Collaborator

    Fixed on main by #332. Recovered judge verdicts now carry a machine-readable judge_recovery marker in report.json, while grading.json remains unchanged. Closing as completed. Thanks for reporting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions