fix(core): pass only MCP arguments as Codex tool-call input

[barryroodt]

The Codex parser put the whole MCP item, tool result included, into tool_call.input. Judges that use serializeTranscript(ctx.transcript, { includeToolCallInputs: true }) therefore got full MCP outputs (e.g. search_docs results up to ~150k chars) as input, for Codex runs only. Input is now just the call arguments, matching the Claude Code, Grok and OpenCode parsers. The result stays on the tool result.

Changes

• packages/core/src/agents/codex/parser.ts: input is item.arguments when it is an object, else {}. Result is item.result ?? item.output ?? item.error.message. The error fallback matters because when Codex fails a call itself it sends result: null with error: {message}, and that message used to reach judges only through the whole-item input. Normalized args (path/command/url) stay {} for MCP calls, so this only changes the input/result split.
• packages/core/src/docs-results.ts: drop the body.arguments.graphql_query fallback, which only existed for the old shape.
• 4 new parser tests. All 4 fail against the old parser.

Effects

• Codex judge input shrinks, so Codex judge scores may shift.
• resolve-performance-001-slow-query-cpu-spike executedSql now sees Codex execute_sql queries.
• Saved files under results/ keep the old shape. Nothing re-derives from them.

The before/after judge-vs-agent spend measurement shows an 85–88% cut in Codex judge spend per full pass; see the measurement comment.

Testing

• pnpm --filter @supabase-evals/core test: 170/170 pass
• pnpm typecheck: pass
• pnpm format:check: pass
• pnpm check not run as a gate: its framework smoke test is already red on main.

Linear: AI-1269

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored		Oct 5, 2026 10:02am UTC

[barryroodt]

Measured: on main's own configuration, this fix cuts Codex judge spend per full benchmark pass by 85–88%, with 0 fix-related verdict regressions.

Experiment	Judge $/pass before → after	Judge share of pass spend
codex-gpt-6-luna	$0.475 → $0.059 (−87.6%)	57.0% → 14.0%
codex-gpt-6-sol	$0.406 → $0.063 (−84.6%)	5.65% → 0.96%

The drop comes from judges that read tool-call inputs: median input per call went from 48,919 → 2,453 tokens (luna) and 31,822 → 2,419 (sol). Judges that don't read tool inputs stayed flat. After the fix, 0 of 381 Codex MCP calls carry the old whole-item input; before, all 358 did. An offline replay of the before transcripts through the fixed parser agrees with the live run to within 0.2 points.

Verdicts: 19 checks flipped, none attributable to the fix. 16 were deterministic checks, 2 were on a judge that never sees tool inputs, and 1 was a FAIL → PASS on luna investigate-auth-001, where a deterministic check flipped the same way.

Caveats:

• n=1, so agent cost and deterministic verdicts vary run to run. The replay backs the judge-side result. All 56 runs completed with 0 errors.
• Luna's judge share stays at ~14% because a luna agent run costs about $0.01, while each judge call is still ~2k tokens on gpt-6-sol. That comes from the configuration, not leftover tool output: dependent judge inputs are now the same size as independent ones.
• Full-pass figures sum all 23 evals' own agent and judge cost. 14 were measured (every judged eval). The 9 unjudged local evals have their agent cost filled from main's published 3-run means (luna $0.195, sol $3.040 per pass) identically on both sides, so the filled values change the denominator but not the before/after difference.
• Codex reports usage per turn, so short-context pricing is assumed.

Base cc479f2 (origin/main) + local usage logging; before = base, after = base + this PR's two commits; judge gpt-6-sol; Codex CLI 0.154.0; experiments codex-gpt-6-luna / codex-gpt-6-sol; 14 evals (every judged eval included); n=1 per eval, side and experiment; prices from the OpenAI pricing page, 2026-10-05.

[mattrossman]

Thanks for finding and fixing! Judge input looks better in my local testing.