feat: honest spend attribution and context-size lens in usage display#3772
feat: honest spend attribution and context-size lens in usage display#3772pauldambra wants to merge 1 commit into
Conversation
The usage banner's top-tool headline ("<tool> drives X% of your spend")
was computed from share_of_scoped, a co-occurrence artifact that
overstates whichever tool is most ubiquitous (Bash), since a single
turn's cost gets counted against every tool it touched. This adopts
the new optional API fields (cost_attributed_usd, share_attributed,
by_input_size) that split each turn's cost fractionally across its
tools and break spend down by input-context size, and reworks the
suggestions/tables to use them when present while falling back to the
prior behavior when a backend hasn't rolled them out yet.
Generated-By: PostHog Code
Task-Id: 68fa35a8-56d2-4be3-a0c9-d870e966acd2
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
React Doctor found no issues in the changed files. 🎉 Reviewed by React Doctor for commit |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 92730143ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| : topByCost.share_of_scoped > LARGE_CONTEXT_SHARE_THRESHOLD | ||
| ? topByCost |
There was a problem hiding this comment.
When a low-context bucket is the largest cost bucket and exceeds 40%—for example <50k with min_input_tokens: 0—this fallback selects it and tells the user that calls with 0+ tokens prove trimming context is their biggest lever. Since one ordinary bucket can naturally contain most spend due to call volume, filter the fallback to large-context buckets before making this recommendation.
Useful? React with 👍 / 👎.
| function topToolSuggestion( | ||
| toolItems: SpendAnalysisResponse["by_tool"]["items"], | ||
| ): string | null { | ||
| const top = toolItems[0]; |
There was a problem hiding this comment.
When the returned rows remain ordered by legacy co-occurrence cost, the first row need not have the greatest share_attributed. For example, a co-occurrence-heavy first tool at 20% followed by a tool at 50% causes this function to return no tool suggestion, even though the latter exceeds the threshold; choose the maximum attributed-share row when attribution is available.
Useful? React with 👍 / 👎.
pauldambra
added
the
Stamphog
|
Note 🤖 stamphog reviewed Gates denied this PR (deny-list match on billing, tier classified T2-never), and it carries two unresolved Codex P2 comments pointing at real logic bugs (fallback selecting low-context buckets, top-tool selection not using max attributed share) with no human or team sign-off addressing them.
Gate mechanics and policy version
|
stamphog
Bot
removed
the
Stamphog
1 participant
Problem
The usage banner's top-tool headline literally says "
<tool>drives X% of your spend," computed fromshare_of_scopedinpackages/core/src/billing/spendSuggestions.ts. That field is a co-occurrence artifact: a single turn can call several tools, and the backend attributes the full turn cost to every tool it touched, so whichever tool is most ubiquitous (Bash) looks artificially dominant, and the numbers don't reconcile to total spend.This is the client half of a companion change to the posthog monorepo's
/api/llm_analytics/@me/spend/endpoint, which adds honest attribution fields plus a genuinely new lever: spend broken down by input-context size.Changes
packages/api-client/src/spend-analysis.tsandpackages/core/src/billing/spendAnalysisTypes.ts, kept in sync): added optionalcost_attributed_usd/share_attributedtoSpendAnalysisToolRow(fractional cost split across a turn's tools, reconciling to scoped spend), a newSpendAnalysisInputSizeRowtype, and an optionalby_input_size?breakdown onSpendAnalysisResponse.packages/core/src/billing/spendSuggestions.ts): the top-tool suggestion now usesshare_attributedand is worded as co-occurrence-aware ("X was involved in N% of your spend when each turn's cost is split across the tools it used") instead of claiming a tool "drives" the spend. A new primary suggestion readsby_input_sizeand recommends trimming context (clearing between unrelated tasks, avoiding re-reads of large files/output) when the largest-context bucket or the top bucket by cost exceeds ~40% of scoped spend. The no-tool suggestion wording was tightened. All of this falls back to the previous behavior when the new fields are absent.packages/ui/src/features/usage/components/SpendBreakdownTables.tsx,SpendAnalysisSection.tsx): the tool table gains an "Attributed" cost column when the field is present, and a new "By context size" table rendersby_input_sizewhen the backend serves it.All new fields are optional on the client, exactly like the existing
by_day?field, so this PR is backwards-compatible and can merge before or after the API rollout.How did you test this?
pnpm --filter @posthog/core typecheck,pnpm --filter @posthog/api-client typecheck,pnpm --filter @posthog/ui typecheck, and a full workspacepnpm typecheck(24/24 tasks) all pass.pnpm --filter @posthog/core test: 2502/2502 pass, including a newspendSuggestions.test.ts(13 cases) covering the attributed-cost wording, the share_of_scoped/share_attributedfallback, theby_input_sizethreshold logic (including the>300k-under-threshold-but-another-bucket-over fallback), and the absent-field paths.pnpm --filter @posthog/api-client test: 130/130 pass.pnpm --filter @posthog/ui test: 2012/2012 pass.biome checkon all changed files: clean.pnpm test(turbo, whole workspace): the only failures are in@posthog/workspace-server(git-network-integration and HogQL-fetch tests unrelated to this change, timing out in this sandbox with no diff in that package) — pre-existing and not touched by this PR.Automatic notifications
Created with PostHog Code