Conversation
Usage.CacheReadInputTokens and CacheWriteInputTokens are *int64: nil when the backend reported no such figure, a pointer to the count, zero included, when it did, which is how CostUSDMicro already tells unknown from zero. A plain integer folded "not measured" into a measured zero, and the Anthropic frontend then told a Messages client that a cache served nothing where an engine had reported nothing at all. This is the type change and the mechanical adaptation of every consumer: the bridge and the frontends dereference with zero for nil, and each backend still sets the count it decoded, so no wire byte changes here. Each backend's report is narrowed to what its wire carried in its own change. The changelog lists the API change under Changed with the migration.
… carried it prompt_tokens_details and its cached_tokens are pointers on the wire struct, so a Chat Completions server without per-request cache accounting, which writes neither, decodes to a nil cache read count rather than a measured zero, and a present cached_tokens, zero included, decodes to its value. This dialect has no cache write count and never sets one. The frontend keeps writing cached_tokens as 0 for nil, since the wire always carries the member and its readers take 0 as a call that read no cache. A table test pins the buffered and streamed decodes and a fuzz test holds the count to its wire member.
… carried it input_tokens_details and its cached_tokens are pointers on the wire struct, so a Responses server without per-request cache accounting, which writes neither, decodes to a nil cache read count rather than a measured zero, and a present cached_tokens, zero included, decodes to its value. This dialect has no cache write count and never sets one. The frontend keeps writing cached_tokens as 0 for nil, since the wire always carries the member and its readers take 0 as a call that read no cache. A table test pins the buffered body and response.completed and a fuzz test holds the count to its wire member.
…rame carried it backendUsage's two cache members are pointers, so a usage that omits one, or writes it null as older responses did, decodes to an unreported count rather than a measured zero, and a present member, zero included, decodes to its value. On the stream, message_delta merges each count through mergeCount: a count it carries fills one message_start did not, a zero never erases a count already reported, and a nonzero one replaces it; the write count is merged there too, where before only the read count was read. A table test pins the buffered and streamed decodes and a fuzz test holds each count to its own wire member.
lux.Usage's cache_read_input_tokens and cache_write_input_tokens are *int64 with omitempty, as cost_usd_micro already is: nil is no key, and a count the backend reported as zero travels as 0 and stays distinguishable from a backend that reported none, where the integer members with omitempty folded a reported zero into absence. The conversions copy rather than alias, so a mutation on one side of the wire/IR boundary cannot reach the other. luxsdk.Usage is the same type and changes with it. A test pins absent, null, zero, and a count on both legs against literal wire bytes, and a fuzz test holds each optional member to its own key and round-trips a decoded usage.
…d not report encodeUsage writes cache_read_input_tokens and cache_creation_input_tokens only when the count is non-nil, zero included, on the response body, message_start, and message_delta alike. A Messages-API client reads cache_read_input_tokens: 0 as a measurement that found nothing cached, which is not what an engine without cache accounting said, so the key must not appear for it. The Anthropic-fronted bridge goldens change by removals only, the keys that were zero for a backend that reported none; the .usage goldens and every other golden are byte-identical.
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ir.Usage.CacheReadInputTokensandCacheWriteInputTokenswere plainintegers. A backend that reports no cache figure (an OpenAI-compatible
server without per-request cache accounting writes no
prompt_tokens_details) therefore decoded to zero, and the Anthropicfrontend wrote
cache_read_input_tokens: 0andcache_creation_input_tokens: 0. A Messages-API client reads those as ameasurement that found nothing cached, which is the opposite of "not
measured".
This change makes the two counts
*int64, following the patternUsage.CostUSDMicroalready uses for the same nil-versus-zero distinction:openaichatandopenairespbackends set the read count only whenprompt_tokens_details.cached_tokens(Chat) orinput_tokens_details.cached_tokens(Responses) is present, zeroincluded, and never set a write count, since neither wire has one.
anthropicbackend sets each count from its own member when present(a
nullor absent member is nil), onmessage_deltaas well asmessage_start; a later zero never erases a count already reported, and acount only
message_deltacarried is still a report.anthropicfrontend omits each key whose count is nil and writes it,zero included, when it is not, on the response body,
message_start, andmessage_deltaalike.cached_tokensas0for nil:those wires always carry the member and their readers take
0as "nocache read".
lux.Usage(and soluxsdk.Usage) carries the two as*int64withomitempty, as it carriescost_usd_micro: nil is no key, an explicitzero travels as
0. Conversions copy, never alias.bridge.Usageis unchanged: it is the floored total a meter wants.This is a breaking change to a
v0.xmodule, listed under Changed withthe migration.
Reproduction
Upstream Chat Completions response from a server with no cache accounting:
{"id":"chatcmpl-1","model":"m","choices":[{"index":0,"finish_reason":"stop", "message":{"role":"assistant","content":"hi"}}], "usage":{"prompt_tokens":12,"completion_tokens":3}}Before, translated to an Anthropic Messages caller:
After:
The same upstream with
"prompt_tokens_details":{"cached_tokens":0}(aserver that measured and found no hit) still yields
"cache_read_input_tokens":0, and one with{"cached_tokens":2}yields"cache_read_input_tokens":2withinput_tokensreduced by 2 as before;neither writes
cache_creation_input_tokens, which that wire cannot report.On a stream,
message_startis synthesized before any usage is known.Before it carried all four members as
0; after it carriesinput_tokensandoutput_tokensonly, and the cache keys appear onmessage_deltawhen the backend reported them.The bridge goldens under
llmdialect/bridge/testdata/translate/for theAnthropic-fronted pairs change accordingly; the diff is removals only
(
cache_creation_input_tokens:0where a read count was reported, bothzero keys where nothing was), and the
.usagegoldens are unchanged.Migration
ir.Usageorlux.Usageliteral: take the address of thecount.
ir.Usagevalues with==now compares pointers; comparemembers or use
reflect.DeepEqual.Tests
openaichat:TestDecodeUsageCacheReporting(absent, null, detailswithout the member, reported zero, reported count; buffered and stream
agree),
TestFrontendEncodeUsageUnreportedCache,FuzzUsageDecode.openairesp:TestBackendDecodeUsageCacheReporting(buffered andresponse.completed),FuzzBackendUsageDecode.anthropic:TestEncodeUsageOmitsUnreportedCache(body,message_start,message_delta, and one-of-two reported),TestBackendDecodeUsageCacheReporting(neither, null, zeros, read only; and the four stream merge cases),
FuzzBackendUsageDecode.lux:TestUsageCacheCountsNilVsZero(literal wire bytes both ways, andno aliasing across the wire/IR boundary),
FuzzDecodeResponseUsage(member set only when its key appeared; round trip stable).
bridge: goldens regenerated for the Anthropic-fronted pairs; everyother golden and every
.usagegolden is byte-identical.go test -race ./llmdialect/... ./luxsdk/...passes. Coverage measured asthe repository's gate does (
-coverpkgover the subtree):ir92.7 %,anthropic97.5 %,openaichat97.6 %,openairesp94.8 %,lux99.6 %,bridge99.5 %,luxsdk98.3 %.Changelog
Under
## Unreleased/### Changed, with the migration.