Skip to content

feat(agents): shipped agents report cache reads and writes as Usage fields, not provider_details - #809

Open
earakely-scale wants to merge 1 commit into
mainfrom
edgararakelyan/fd5979-agent-usage-cache
Open

earakely-scale wants to merge 1 commit into
mainfrom
edgararakelyan/fd5979-agent-usage-cache

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The five shipped agents now report the prompt cache in Usage's own fields, added in #791, and their input_tokens counts the cache wherever the agent can tell:

Agent Before After
pi input_tokens left the cache out; reads and writes in provider_details the cache is added back; cache_read_tokens, cache_write_tokens and input_includes_cache: true are filled in
Hermes same as pi same as pi
Pydantic AI the run's totals; Claude's cache, sent in Anthropic's keys, was dropped summed call by call: Anthropic's keys are added to a prompt smaller than the cache, and the flag is left unknown when a call can't say
OpenHands the run's totals; Claude's prompt left the cache out summed call by call, like Pydantic AI; a Responses API call's input already counts the cache
browser-use its counts already included the cache; the cache and the cost sat in provider_details the same counts, now in the Usage fields, with cost_usd

When a model call returns no usage, the agent reports the counts as unknown (None) and records the call in provider_details.unreported_calls; before, it reported zero tokens. This covers Claude's streamed answers through Scale's LiteLLM proxy, which come back without usage.

Moved keys (for the release note)

  • provider_details.cache_read_tokens and provider_details.cache_write_tokens move to Usage.cache_read_tokens and Usage.cache_write_tokens for all five agents.
  • browser-use's provider_details.cost_usd moves to Usage.cost_usd.
  • input_tokens now counts the cache for pi, Hermes, and for Claude on Pydantic AI and OpenHands. A reader that used to add the cache back itself must stop doing so.
  • provider_details.unreported_calls is new.

The agents' and the proxy's protocol floor rises to 0.1.597, the first release whose Usage has the cache fields.

Why

Measured against the real proxy, the five agents disagreed on what input_tokens meant. The proxy's Chat Completions answers for Claude also leave the cache out of the prompt and carry no usage when streamed. Each runtime counts differently, so each agent normalizes its own runtime's counts, as agreed on #791.

Tests

  • Unit tests for each agent's usage, built from the per-call usage the proxy actually returned: GPT's cached reads, Claude's Anthropic keys, an all-zero (unreported) call, and the ambiguous case where the prompt is at least as large as the cache.
  • Integration: the Pydantic AI and OpenHands image tests (run_bundle_*_test.py) each gain a test of three turns on one deployed agent: Claude's recorded shape with a tool call, GPT's, and an answer with no usage. tst/util/fake_openai.py gains a per-request usage, None for an answer without one.

E2E against the real proxy

Each agent's smoke task ran on claude-haiku-4-5 and gpt-4.1-mini through Scale's LiteLLM proxy, behind a logging pass-through. The agent images were built from this branch, and all 10 runs passed. The table compares each turn's reported usage with the gateway's own answers for that turn, read by Usage.from_model_response:

Agent Model main: input / read / write This branch: input / read / write / flag Gateway
Pydantic AI Claude 0 / – / – unknown, unreported_calls: 2 2 streamed answers, no usage
Pydantic AI GPT 4,472 / 2,048* / 0* 4,474 / 2,048 / 0 / true 4,474 / 2,048 / 0
OpenHands Claude 23 / 10,363* / 10,562* 20,939 / 10,363 / 10,553 / true 20,939 / 10,363 / 10,553
OpenHands GPT 15,681 / 7,680* / 0* 15,681 / 7,680 / 0 / true 15,681 / 7,680 / 0
Hermes Claude 0 / 0* / 0* unknown, unreported_calls: 2 2 streamed answers, no usage
Hermes GPT 10,750 / 10,496* / 0* 21,249 / 10,496 / 0 / true 21,249 / 10,496 / 0
pi Claude 0 / 0* / 0* unknown, unreported_calls: 2 2 streamed answers, no usage
pi GPT 1,224 / 1,024* / 0* 2,248 / 1,024 / 0 / true 2,248 / 1,024 / 0
browser-use Claude 5,752 / 0* / 0* 5,775 / 0 / 0 / true 5,775 / 0 / 0
browser-use GPT 5,581 / 3,456* / 0* 9,784 / 5,632 / 0 / true 9,784 / 5,632 / 0

* in provider_details. The main numbers come from an earlier run, so prompts differ slightly between the two runs. The Hermes gateway figures leave out its auxiliary non-streamed call (280 and 421 tokens), which Hermes keeps out of the turn's api_calls.

Chaos

Ran on the VM against the shipped Pydantic AI and OpenHands images and the fake endpoint, with two seeds of 10 turns each on one conversation. Each call carried Claude's proxy shape, GPT's shape with or without a cached read, or no usage, at random. Single 500s were retried by the agent's client. Each seed ended with a turn stalled past its 5 s timeout, then one more turn. All 40 turns matched the expected usage. Each timed-out turn failed with unknown counts, and the turn after it carried only its own call.

Notes

  • _counted and _summed are copied in the Pydantic AI and OpenHands folders rather than shared. Each image builds only from its own folder, and the protocol has no public helper for this.
  • OpenHands on a GPT model it calls through Chat Completions reports input_includes_cache: null when the cache read is smaller than the prompt. LiteLLM gives OpenHands the same counts for both shapes, so the agent can't tell which one it got.
  • Hermes still reports cost_usd: 0.0 when its cost_status is unknown. I left that to the pricing work.

Docs

The agents page should say that each shipped agent's input_tokens counts the prompt cache, which fields hold reads and writes, and what unreported_calls means.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Moves prompt-cache counts into shared Usage fields and updates how the shipped agents count cached input.

  • Shipped agents report cache-aware token usage in shared Usage fields.

earakely-scale intentionally keeps the copied usage helpers inside each image’s build folder and defers Hermes reporting zero cost when its price is unknown.

Reviews (1) · Last reviewed commit: "feat(agents): shipped agents report cach..." · Reviewed by Greptile

…ields, not provider_details

Each shipped agent (Pydantic AI, OpenHands, pi, Hermes, browser-use) now fills Usage's
cache_read_tokens, cache_write_tokens and input_includes_cache, and its input_tokens counts the
cache where the agent can tell: pi and Hermes add back the cache their runtimes take out of the
prompt; Pydantic AI and OpenHands decide call by call, adding Anthropic's cache keys to a prompt
smaller than them and leaving the flag unknown when the counts can't say. A model call that
returned no usage, as Claude's streamed answers through a LiteLLM proxy, leaves the counts unknown
and is counted in provider_details.unreported_calls instead of reading as zero tokens.

provider_details.cache_read_tokens and cache_write_tokens move to the Usage fields for all five
agents, and browser-use's cost to Usage.cost_usd. The agents' and the proxy's protocol floor is
0.1.597, the first whose Usage has the cache fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 19, 2026 08:17
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 08:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant