fix: opencode non-cached input semantics + LLM grader retry

[tardigrde]

Summary

Two fixes found while live-verifying v0.6.0:

1. OpenCode reports input tokens excluding cache reads

Live runs show cached_tokens consistently larger than input_tokens (e.g. 12,481 input vs 93,184 cached across all 4 evals of a run) — impossible if input contained the cache reads. The harness assumed codex-style semantics (input includes cache reads), so non_cached_input_tokens clamped to a bogus 0 and cached_pct capped at 100% for opencode. OpenCodeHarness now overrides the split like claude-code; README/comments updated.

Pass/fail, costs, and all other numbers were unaffected — only the billable-input interpretation for opencode.

2. LLM grader retries dropped assertions once

The judge intermittently omits an assertion from its JSON answer; the harness correctly skipped it (grader failure ≠ agent failure), but the strict live e2e test asserts zero skips and flaked on different assertions across runs. The grader now retries once with only the dropped assertions before skipping. While in there: matched results are filtered to the requested assertions and deduplicated, so a retry (or a chatty judge) can never double-count rows in grading.json.

Test plan

• pytest -m "not live": 313 passed (4 new grader-retry tests, 1 new opencode-semantics test)
• ruff check + ruff format --check clean
• Live verify with opencode + free flash model (codex excluded: no credits currently)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a retry mechanism in the LLM grader to recover assertions dropped by the judge during grading, and updates the OpenCode harness to correctly handle input tokens that exclude cache reads. Additionally, corresponding unit tests were added to verify these changes. Feedback on the pull request suggests adding defensive type and key checks when parsing the LLM grader's JSON response to prevent potential runtime exceptions from malformed outputs.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The LLM grader's response parsing is susceptible to runtime exceptions (such as AttributeError, KeyError, or TypeError) if the LLM returns an unexpected JSON structure (e.g., a top-level list instead of a dictionary, or results containing objects missing expected keys like 'text', 'passed', or 'evidence'). Since _grade_attempt is called within a try-except block, any such exception will cause the entire grading attempt to fail, skipping all assertions.

By adding defensive type and key checks, we can gracefully ignore malformed individual results and retry or skip only the affected assertions, while preserving the validly graded ones.

        if not isinstance(data, dict):
            return [], assertions
        # Only keep results that answer a requested assertion, once each:
        # judges sometimes return extra, duplicated, or unrelated rows, and
        # on a retry that would double-count assertions already graded.
        requested = {_normalize_assertion_text(a) for a in assertions}
        results = []
        answered: set[str] = set()
        for r in data.get("results", []):
            if not isinstance(r, dict) or "text" not in r or "passed" not in r or "evidence" not in r:
                continue
            key = _normalize_assertion_text(r["text"])
            if key not in requested or key in answered:
                continue
            results.append(
                AssertionResult(
                    text=r["text"],
                    passed=r["passed"],
                    evidence=r["evidence"],
                    method="llm",
                )
            )
            answered.add(key)

[tardigrde]

Done in 45b2677 — also rejects non-string text and a non-list results, and the dropped row's assertion now flows into the retry instead of being lost.