Fix Gemini/Vertex estimated cost excluding thinking tokens#920
Open
colinbennettbrain wants to merge 1 commit into
Open
Fix Gemini/Vertex estimated cost excluding thinking tokens#920colinbennettbrain wants to merge 1 commit into
colinbennettbrain wants to merge 1 commit into
Conversation
geminiUsageToOpenAIUsage mapped completion_tokens from candidatesTokenCount only, which excludes thoughtsTokenCount. Every cost path prices completion_tokens, so thinking tokens were never charged, undercounting cost for Gemini thinking models on Vertex. Follow the OpenAI/Anthropic/Lingua convention where completion_tokens includes reasoning and reasoning_tokens is the subset breakdown, so all downstream cost paths (proxy metrics, ingestion, query, UI) price thinking tokens correctly. Add streaming + non-streaming regression tests. Pylon #18680
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vertex/Gemini thinking (reasoning) tokens were excluded from estimated LLM cost.
geminiUsageToOpenAIUsagemappedcompletion_tokensfromcandidatesTokenCountonly, which per Google's usage metadata excludesthoughtsTokenCount(totalTokenCount = prompt + candidates + toolUse + thoughts). Since every cost path pricescompletion_tokens, thinking tokens were never charged — undercounting cost for Gemini thinking models on Vertex (e.g.gemini-3.1-pro-preview).Root cause / fix
The canonical convention across the codebase is that
completion_tokensis the total billable output including reasoning, withreasoning_tokensas a subset breakdown:completion_tokensalready includes reasoning (pass-through).completion_tokens = output_tokens, which includes thinking.completion_tokens = candidates + thoughts.The legacy Gemini converter was the lone violator. This aligns it with the convention:
reasoning_tokensstays the subset breakdown. This corrects every downstream cost path at once (proxy metric logging, ingestion, query-time, and UI) without touching the cost logic — and deliberately avoids adding reasoning inside the cost paths, which would double-count for OpenAI/Anthropic/Lingua where reasoning is already part ofcompletion_tokens.Tests
Added streaming + non-streaming regression tests covering the reported payload (
candidatesTokenCount=50,thoughtsTokenCount=25→completion_tokens=75,reasoning_tokens=25).Follow-up
Fixes new logs going forward via the proxy. Historical spans keep the undercounted
completion_tokensand would need a separate backfill if corrected cost is required. A submodule bump inbraintrustdata/braintrustis the deploy step after this merges.Pylon #18680