Skip to content

feat(protocol): Usage gains prompt-cache counts and an includes-cache flag, and the proxy reports its turns' model usage - #791

Merged
earakely-scale merged 8 commits into
mainfrom
edgararakelyan/fd-5979-p7-usage-cache-fields
Nov 19, 2026
Merged

earakely-scale merged 8 commits into
mainfrom
edgararakelyan/fd-5979-p7-usage-cache-fields

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

The protocol's Usage can now say how many prompt tokens came from the provider's prompt cache, and whether input_tokens already counts them. Without that, a price can't tell cached input from fresh input.

Whoever makes a model call knows what its counts hold, so they report usage themselves; the framework only guesses when nobody tells it. This is the protocol and host half of FD-5979 (P7 of FD-5974). The five agents that count cache tokens move to the new fields in a second PR, once a protocol release has them.

What changes

  • Usage gains three fields (all None by default):
    • cache_read_tokens and cache_write_tokens, validated like the other counters;
    • input_includes_cache: bool | None, strictly a bool. True means input_tokens counts both cache counts, False means it counts neither, and None means unknown.
  • Usage.from_dict reads what to_dict wrote under any protocol version. A field it leaves out is None, a key this version doesn't know is ignored, and a value a field can't hold raises.
  • TaskResult.failure(..., usage=), so a failure carries usage as success does.
  • A trainer can report a call's usage itself. record_model_call(message, usage=Usage(...)) sends the Usage to the proxy, which takes it as given.
    • It goes under REPORTED_USAGE_KEY ("agentenv_usage") inside the record's existing usage field, not as a new field. Extension requests refuse unknown fields, so a proxy on an older protocol still takes the record.
    • A reported value that isn't a valid Usage gets 400 and leaves the turn as it was.
  • Usage.from_model_response(usage) is the default reader for a model response's own usage. It handles OpenAI's Chat Completions (prompt_tokens) and Responses (input_tokens) shapes and Anthropic's, always returning input_tokens counting the cache:
    • If prompt_tokens_details or input_tokens_details lists cached_tokens, the input already counts the cache. That's OpenAI and a LiteLLM client, and cache_write_tokens there is counted too.
    • Otherwise Anthropic's cache_read_input_tokens and cache_creation_input_tokens are added to the input and the total. That covers Anthropic itself, and a LiteLLM proxy answering for Claude, which sends them beside a prompt_tokens that leaves them out. They're also added whenever the input is smaller than the cache it would hold.
    • It keeps each provider's split of cache writes by how long they're kept, since those are priced differently: Anthropic's cache_creation (5 minutes, 1 hour) and OpenAI's cache_write_tokens_details. They go in provider_details.
    • It marks a usage in none of these shapes as unrecognized, and leaves out a value that isn't a count.
  • The proxy reports its turns' usage. Before, a turn's Usage was only tool_call_count.
    • Each recorded call is read once and made to count the cache. A reported call with the flag False has its cache added to its input and total. One with no flag but with cache counts stays as given and leaves the turn's flag unknown.
    • The call's chat span shows the same counts, so the trajectory and the usage agree.
    • The turn sums each count over the calls that gave it, cost_usd included.
    • provider_details gives model_calls; unreported (how many calls left out a count only some gave, so a partial sum reads as the lower bound it is); cache_unsettled; unrecognized; and the cache-write splits summed, counting only real counts.
  • Usage's field list is written once. from_dict, is_empty, to_dict and the proxy's sums read it from the dataclass.
  • Docs. The protocol README describes Usage, the default reader and the proxy's usage. agents/AGENTS.md tells agent authors to fill the cache fields and set the flag per call.

What a real gateway showed

Calls through the Scale LiteLLM proxy, which are now the unit fixtures:

Route Reported input Counts the cache?
Claude, /chat/completions prompt_tokens: 9, cache beside it, no details no
GPT, /chat/completions prompt_tokens: 7823, prompt_tokens_details.cached_tokens: 7680 yes
Claude and GPT, /v1/responses input_tokens with input_tokens_details yes
Claude, /v1/messages input_tokens: 9, cache beside it no; a 1-hour write shows in cache_creation

So whether the input counts the cache depends on the route and the model behind it, not on the agent. An adapter sets the flag per call. FD-5979's table gives OpenHands True, which holds for GPT on this gateway but not for Claude.

Mixed versions

  • An old agent with a new host: the new fields read as None.
  • A new agent with an old host: the old host reads only tool_call_count from usage and ignores the rest.
  • A reported usage sent to an older proxy: the proxy takes the record and simply doesn't count it. Sandboxed proxies install the protocol from PyPI, so they count usage only after the second PR raises their floor.
  • New adapter code on an old protocol would raise TypeError. That's why the adapters wait for the second PR.

Tests

  • test_usage.py (new):
    • the round trip, with and without the new fields;
    • older and newer agents' usage;
    • bad counts and non-bool flags refused;
    • a provider_details that isn't a mapping, even an empty one, refused;
    • failure(usage=);
    • every recorded gateway response, plus a Bedrock-shaped one marked unrecognized.
  • test_a2a_proxy.py:
    • summed turns, and spans agreeing with them;
    • reported usages: taken as given, a False flag normalized, an unknown flag leaving the turn's flag unknown;
    • an invalid reported usage refused without touching the turn;
    • cache-write splits summed, malformed ones in a reported usage ignored;
    • unrecognized shapes counted;
    • a turn with no recorded calls.
  • tst/unit/a2a_agent/protocol_test.py: the host parses a terminal message whose usage has the new fields, and a failed turn sends its usage.
  • tst/integration/agent/proxy_usage_local_test.py (new, about 1 s, no Docker): an episode's trainer records an OpenAI-shaped and an Anthropic-shaped response and one reported Usage through record_model_call. The host stores their sum with the agent's reply.

E2E and chaos (Linux VM)

  • Real model, about $0.10 over all runs. Eleven calls went through the gateway: Chat Completions for Claude and GPT, Messages for Claude (5-minute and 1-hour writes), and Responses for Claude and GPT, each prompt sent so the cache is written and then read.
    • All Claude calls read as one input whichever API they went through, and so do GPT's.
    • The 1-hour write is kept as such.
    • Recorded in one episode turn with a reported Usage whose input left the cache out, the host stored the 11 calls summed, the reported one normalized.
    • The key was handed over through an object deleted once read, and no process held it afterwards.
  • Chaos, against an oracle written apart from the code:
    • 15 minutes on 71e23c1: 5.5 million readings and 137,550 proxy turns, with 124,071 reported calls and 40,809 invalid reported usages refused.
    • 5 minutes on the head: 1.77 million readings and 44,285 turns.
    • 0 problems in both.
    • Four planted bugs, one in each new rule, were each caught within 5 seconds.
  • The usage suites and the new integration test passed on the VM.

Next

  • The adapters PR, after the protocol release:
    • browser-use, Hermes, OpenHands, pi and pydantic-ai write the new fields instead of their provider_details cache keys;
    • those reading gateway responses use from_model_response;
    • each raises its protocol floor, the proxy agent's included;
    • the release note names the moved keys.

Docs (for P13, FD-6263)

  • Usage's cache fields and the flag, set per call.
  • Reporting usage from a trainer with record_model_call(usage=Usage(...)).
  • from_model_response: the shapes it reads, the cache-write splits, and unrecognized.
  • The proxy turn's provider_details.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The changes appear safe to merge; no outstanding issues were found.

Summary

Adds prompt-cache counts to Usage and reports summed model usage for proxy turns, including failed replies.

  • Usage records prompt-cache counts and whether input includes them.
  • Proxy turns now report the usage of their recorded model calls.

Reviews (11) · Last reviewed commit: "fix(protocol): a reported Usage's cache-..." · Reviewed by Greptile

… flag, and the proxy reports its turns' model usage

Usage takes cache_read_tokens, cache_write_tokens and input_includes_cache, validated like its other fields, and
Usage.from_dict reads what any protocol version wrote. TaskResult.failure carries usage. The proxy sums the usage of
the model calls its driver records into each turn's Usage, OpenAI- and Anthropic-shaped alike, with input_tokens
counting the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 23:02
earakely-scale and others added 3 commits November 18, 2026 15:12
…e cache Anthropic's keys beside it name

A real LiteLLM proxy answers Claude through /chat/completions with prompt_tokens counting neither cache count it sends
beside it, while OpenAI and a LiteLLM client say what prompt_tokens counts through prompt_tokens_details.cached_tokens.
The normalizer, now Usage.from_model_response so agents can read their own gateway calls the same way, adds the cache
to prompt_tokens and total_tokens unless prompt_tokens_details says it's already there. The fixtures are the recorded
responses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… mapping, empty ones too; the usage episode test abandons its episode

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…9-p7-usage-cache-fields

# Conflicts:
#	tst/unit/a2a_agent/protocol_test.py
earakely-scale and others added 3 commits November 18, 2026 16:14
… and a sum short of some calls says so

A chat span took its token counts from the raw usage while the turn's Usage read it with from_model_response, so a
LiteLLM proxy's Claude call showed 9 input tokens in the trajectory and 7,818 in the usage. Both now read the call
once. The turn's provider_details give model_calls and, for a count only some calls gave, how many left it out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l_response decides once whether a response's input counts the cache

from_dict, is_empty, to_dict and the proxy's sums read Usage's fields from the dataclass, so a count added to Usage
reaches them all. from_model_response makes the one decision its docstring describes and applies it to the input and
the total together. Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fault reader knows the Responses shape

Whoever makes a model call knows what its counts hold. record_model_call takes a Usage, sent to the proxy under
REPORTED_USAGE_KEY inside the record's usage, so a proxy on an earlier protocol still takes the record, and taken as
given; one that isn't a Usage gets 400 and leaves the turn as it was. Each call's input is made to count the cache
before its span and its turn read it, a call whose flag is unknown while it has cache counts leaving the turn's flag
unknown, and the turn sums cost_usd too.

Usage.from_model_response, the default for a response's own usage, reads OpenAI's Responses shape and the
cache_write_tokens OpenAI's details now give, keeps each provider's split of its cache writes by how long they're kept
(priced apart) in provider_details, and marks a usage in no shape it knows as unrecognized. The fixtures are responses
recorded through a LiteLLM proxy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ere it holds counts

A driver's Usage may carry any provider_details, so a split with a None count no longer fails the turn when it ends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 5865e43 into main Nov 19, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5979-p7-usage-cache-fields branch November 19, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants