Skip to content

feat(agents): type a model key refusal as model_key_budget / model_key_auth, not agent_error - #695

Draft
andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-agent-typed-errors
Draft

andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-agent-typed-errors

Conversation

@andrewpark-scaleai

Copy link
Copy Markdown
Contributor

Why

AgentEnv should say why every run failed, and heal itself where code can. Today a run whose model key is out of budget or rejected is blamed on the agent.

When the model endpoint refuses the key, the Claude Code CLI exits 1 and its terminal result event names the cause as fields: api_error_status and api_error_type (for example 400 / budget_exceeded from LiteLLM, 401 / authentication_error from Anthropic). The claude-code agent's A2A server ignored those fields and called builder.failed("cli_error", ...) with no error_type, so the protocol's default agent_error reached the framework's PromptResponse.error_type. Graders and dashboards then count a platform key failure as the agent's mistake, and nothing downstream can tell the two apart without matching error text.

What

  • agentenv-framework-protocol: TaskError.error_type is now the closed type TaskErrorType, which adds model_key_budget and model_key_auth to agent_error and infra_error. The validator and every signature that took the old two-value Literal use it.
  • claude-code agent: the bridge parses the CLI's terminal result event into that type when the CLI fails. api_error_type == "budget_exceeded" gives model_key_budget. A 401 or 403, or a type of auth_error, authentication_error or permission_error, gives model_key_auth. Anything else stays agent_error. The A2A server passes it to builder.failed.
  • The agent's protocol floor rises to the first release with the new values. The protocol README and agents/AGENTS.md document them.

The framework side needs no change: TerminalResponse.from_message already copies error_type from the data part into PromptResponse.error_type.

Design decisions and tradeoffs

  • New error_type values, not a new field. Every reader of PromptResponse.error_type already treats any value as "the turn errored", so verifiers keep skipping these turns. A later breaker or classifier can match the two key values exactly instead of matching text. The cost is that a reader comparing error_type == "infra_error" will not match them. Nothing in this repo does.
  • Parse once, at the CLI boundary. The bridge already owns the CLI's event parsing (_terminal_result, _status), so the mapping lives there and the server only forwards a typed value.
  • Unknown stays agent_error. Rate limits (429) and bad requests (prompt too long) keep the old type. This PR only reclassifies the two causes the endpoint states outright.
  • Only claude-code for now. The gemini and codex servers have the same failed("cli_error", ...) default. I did not have their CLIs to capture what they print for a budget or auth refusal, so they are left for a follow-up rather than guessed.
  • Protocol floor. >=0.1.551 assumes this lands in the next protocol release. If another release goes out first, bump the floor to the version that carries this change. The image installs the newest protocol at or above the floor, so the floor documents the need rather than gating it.

How to verify

The test events are trimmed from captures of Claude Code CLI 2.1.325 run against a local fake endpoint that answers with each error body.

.venv/bin/python -m pytest tst/unit/agents/claude_code tst/unit/a2a_agent/protocol_test.py packages/agentenv-protocol/tests tst/unit/test_release_protocol_pins.py -q

Result: 1955 passed, 1 skipped in 576s. ruff check on the touched files passes.

Without the fix, test_model_key_errors.py fails for the four key cases (litellm-budget, litellm-auth, anthropic-auth, anthropic-permission) with error_type == "agent_error", and the protocol test fails with ValueError: error_type must be 'agent_error' or 'infra_error'.

End to end, I ran the agent's A2A server in-process with the real claude binary against a local endpoint answering 400 {"error": {"type": "budget_exceeded", ...}}. The terminal message carried error_type: model_key_budget, error_code: cli_error. A 404 endpoint still gave agent_error.

Ubuntu and others added 2 commits November 15, 2026 07:55
…odel_key_budget or model_key_auth

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ey_auth, not agent_error

TaskError.error_type gains model_key_budget and model_key_auth. The claude-code bridge
reads the CLI's api_error_type / api_error_status from its terminal result event and the
A2A server passes the type to builder.failed, so the framework's PromptResponse no longer
blames the agent for an out-of-budget or rejected key.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant