Skip to content

core: count Google cache reads inside the reported input - #51

Closed
sam-bretz wants to merge 1 commit into
mainfrom
fix/google-input-includes-cache-reads
Closed

sam-bretz wants to merge 1 commit into
mainfrom
fix/google-input-includes-cache-reads

Conversation

@sam-bretz

Copy link
Copy Markdown
Collaborator

Found while fixing MET-61 (cached tokens billed twice for OTLP spans). This is the same bug class for Google traffic, independent of OTLP.

@VasiliyRad this contradicts a reading you encoded deliberately, in 70565c2 and 453634c ("google-api bills cache reads on top of full input"), so please weigh the evidence below before merging. It changes what Google traffic costs.

Problem

No Google-channel price records input_includes_cache_read, yet 16 of them bill a cache-read rate. So a cached Google token is charged twice: at the input rate, because it is inside the reported prompt, and again at the cache rate. Caching makes a prompt more expensive than not caching it.

Worked example on the shipped catalog, gemini-2.5-pro, 100,000 reported input of which 80,000 were cache reads:

today with this change
cost $0.14980000 $0.04980000

Evidence

  1. Google's API reference, UsageMetadata: promptTokenCount is "Number of tokens in the prompt. When cachedContent is set, this is still the total effective prompt size meaning this includes the number of tokens in the cached content." cachedContentTokenCount is a subset of it, not an addition to it.
  2. The catalog already treats OpenAI this way. prompt_tokens has the same meaning, and those entries carry the rule. Core is currently inconsistent between two providers that report identically.
  3. The hosted app catalog already prices Google this way. metergraph-internal migration 0040 sets {"input_includes_cache_read":true} on the Google entries, and has priced production Google traffic with it. The two catalogs disagree today, which is the drift MET-33 describes.
  4. The same model on the Vercel gateway already carries the rule in this catalog, so one Gemini model is priced two different ways depending on channel.

What I could not find: an explicit statement on Google's pricing page that the caching rate replaces the input rate. The pricing page lists them as separate line items without saying how they combine. Points 1 to 4 are the basis for this change, plus the fact that the alternative reading makes caching a cost increase, which would defeat the feature.

Change

input_includes_cache_read: true on the 15 Google-published prices on google-api and google-vertex-ai that bill a cache-read rate. Untouched: Google models with no published cache rate (nothing is billed per cached token), and anthropic/claude-opus-4.6 on google-vertex-ai, which reports Anthropic's usage shape where input already excludes the cache.

The YAML edit was applied by a script that re-parses the file and refuses to write if anything other than those 15 rules changed, so the diff is 15 added lines.

Verification

  • pytest core/tests -q: 231 passed.
  • Two existing assertions encoded the older reading and are updated, with the provider's wording in the docstring:
    • gemini-2.5-flash on google-api, 1M input of which 1M cached: $0.375 becomes $0.075;
    • gemini-3.5-flash-lite, same shape: $0.33 becomes $0.03.
  • New test_google_published_models_count_cache_reads_inside_reported_input fails if any Google price bills cache reads without the rule.
  • Negative control: with the data change reverted, those 4 tests fail and the rest pass.

Release note

No version bump here, to avoid colliding with #50 which takes 0.2.24. Whoever merges both should ship this as the next version. Once released and pinned, already-stored calls rows keep their old costs; repricing history is a separate decision.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DkYVigksoSN9N7coWLgmvS

Google's UsageMetadata defines promptTokenCount as "the total effective
prompt size meaning this includes the number of tokens in the cached
content". The Google-channel prices did not record that, so every cached
token was billed twice: once at the input rate as part of the reported
prompt, and again at the cache-read rate. Caching a prompt made it more
expensive than not caching it.

Add input_includes_cache_read to the 15 Google-published prices on
google-api and google-vertex-ai that bill a cache-read rate. Models with no
published cache rate are untouched, and so is Anthropic's model served on
Vertex, which reports Anthropic's usage shape where input already excludes
the cache.

This matches how the catalog already treats OpenAI, whose prompt_tokens has
the same meaning, and how the hosted app catalog has priced Google traffic
since migration 0040.

Two existing assertions encoded the older reading and are updated with the
provider's wording. A catalog-wide test now fails if a Google price bills
cache reads without recording the convention.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DkYVigksoSN9N7coWLgmvS
@sam-bretz

Copy link
Copy Markdown
Collaborator Author

Superseded by #52, which fixes the same defect at the channel level rather than row by row: a new price row cannot forget the rule there, which is the better shape.

Carried over to #52 as a review comment: its channel set includes google-vertex-ai, which also carries anthropic/claude-opus-4.6. Claude on Vertex reports Anthropic's usage shape, where input already excludes cache reads, so that row would have them deducted twice over ($0.55 instead of $6.05 at 1M input and 1M cache reads). #51 avoided it by only touching Google-published rows.

The evidence from this PR still stands and is recorded in MET-136: Google's UsageMetadata defines promptTokenCount as including cached content, the app's own migration 0040 has priced Google that way in production, and core already applied the rule to OpenAI.

@sam-bretz sam-bretz closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant