Repository navigation
feat(agents): shipped agents report cache reads and writes as Usage fields, not provider_details - #809
Open
earakely-scale wants to merge 1 commit into
Open
feat(agents): shipped agents report cache reads and writes as Usage fields, not provider_details#809earakely-scale wants to merge 1 commit into
earakely-scale wants to merge 1 commit into
Conversation
…ields, not provider_details Each shipped agent (Pydantic AI, OpenHands, pi, Hermes, browser-use) now fills Usage's cache_read_tokens, cache_write_tokens and input_includes_cache, and its input_tokens counts the cache where the agent can tell: pi and Hermes add back the cache their runtimes take out of the prompt; Pydantic AI and OpenHands decide call by call, adding Anthropic's cache keys to a prompt smaller than them and leaving the flag unknown when the counts can't say. A model call that returned no usage, as Claude's streamed answers through a LiteLLM proxy, leaves the counts unknown and is counted in provider_details.unreported_calls instead of reading as zero tokens. provider_details.cache_read_tokens and cache_write_tokens move to the Usage fields for all five agents, and browser-use's cost to Usage.cost_usd. The agents' and the proxy's protocol floor is 0.1.597, the first whose Usage has the cache fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
marked this pull request as ready for review
November 19, 2026 08:17
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The five shipped agents now report the prompt cache in
Usage's own fields, added in #791, and theirinput_tokenscounts the cache wherever the agent can tell:input_tokensleft the cache out; reads and writes inprovider_detailscache_read_tokens,cache_write_tokensandinput_includes_cache: trueare filled inprovider_detailsUsagefields, withcost_usdWhen a model call returns no usage, the agent reports the counts as unknown (
None) and records the call inprovider_details.unreported_calls; before, it reported zero tokens. This covers Claude's streamed answers through Scale's LiteLLM proxy, which come back without usage.Moved keys (for the release note)
provider_details.cache_read_tokensandprovider_details.cache_write_tokensmove toUsage.cache_read_tokensandUsage.cache_write_tokensfor all five agents.provider_details.cost_usdmoves toUsage.cost_usd.input_tokensnow counts the cache for pi, Hermes, and for Claude on Pydantic AI and OpenHands. A reader that used to add the cache back itself must stop doing so.provider_details.unreported_callsis new.The agents' and the proxy's protocol floor rises to 0.1.597, the first release whose
Usagehas the cache fields.Why
Measured against the real proxy, the five agents disagreed on what
input_tokensmeant. The proxy's Chat Completions answers for Claude also leave the cache out of the prompt and carry no usage when streamed. Each runtime counts differently, so each agent normalizes its own runtime's counts, as agreed on #791.Tests
run_bundle_*_test.py) each gain a test of three turns on one deployed agent: Claude's recorded shape with a tool call, GPT's, and an answer with no usage.tst/util/fake_openai.pygains a per-requestusage,Nonefor an answer without one.E2E against the real proxy
Each agent's smoke task ran on
claude-haiku-4-5andgpt-4.1-minithrough Scale's LiteLLM proxy, behind a logging pass-through. The agent images were built from this branch, and all 10 runs passed. The table compares each turn's reported usage with the gateway's own answers for that turn, read byUsage.from_model_response:main: input / read / writeunreported_calls: 2unreported_calls: 2unreported_calls: 2* in
provider_details. Themainnumbers come from an earlier run, so prompts differ slightly between the two runs. The Hermes gateway figures leave out its auxiliary non-streamed call (280 and 421 tokens), which Hermes keeps out of the turn'sapi_calls.Chaos
Ran on the VM against the shipped Pydantic AI and OpenHands images and the fake endpoint, with two seeds of 10 turns each on one conversation. Each call carried Claude's proxy shape, GPT's shape with or without a cached read, or no usage, at random. Single 500s were retried by the agent's client. Each seed ended with a turn stalled past its 5 s timeout, then one more turn. All 40 turns matched the expected usage. Each timed-out turn failed with unknown counts, and the turn after it carried only its own call.
Notes
_countedand_summedare copied in the Pydantic AI and OpenHands folders rather than shared. Each image builds only from its own folder, and the protocol has no public helper for this.input_includes_cache: nullwhen the cache read is smaller than the prompt. LiteLLM gives OpenHands the same counts for both shapes, so the agent can't tell which one it got.cost_usd: 0.0when itscost_statusisunknown. I left that to the pricing work.Docs
The agents page should say that each shipped agent's
input_tokenscounts the prompt cache, which fields hold reads and writes, and whatunreported_callsmeans.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Moves prompt-cache counts into shared
Usagefields and updates how the shipped agents count cached input.earakely-scale intentionally keeps the copied usage helpers inside each image’s build folder and defers Hermes reporting zero cost when its price is unknown.
Reviews (1) · Last reviewed commit: "feat(agents): shipped agents report cach..." · Reviewed by Greptile