Include inner interactive-toolkit cost in session cost

[josephschorr]

Motivation

A session's status.estimatedCost under-reports total spend — potentially by
more than an order of magnitude — because it only prices the orchestrator
model's tokens and ignores the cost of inner interactive toolkits the agent
drives.

A passthrough coding agent that drives Claude Code, for example, can spend the
large majority of its cost inside those sub-runs. That spend is already recorded
per invocation in the tool_session audit Kind's content.costUsd, but it
never reaches the cost reporter — so the stamped total reflects only the
orchestrator model and omits the inner toolkit spend entirely.

Summary of changes

Rolls each interactive toolkit's provider-reported cost into the session total,
itemized so provenance stays visible. Generic — keyed by outer tool name, no
per-toolkit branching.

• Types (pkg/platform/pipeline, pkg/apis/v1alpha1): new ToolUsage and
  ToolCostBucket; SessionEndInfo.ByTool and EstimatedSessionCost.ByTool.
  Regenerated deepcopy, CRDs, install.yaml, and the CRD reference doc.
• Accumulation (pkg/agent/runner): Loop.AddToolCost mirrors the existing
  addModelUsage (same usageMu, first-seen ordering, negative-cost guard,
  single round at the USD→micro-USD boundary). Fed in-process from the runner's
  interactive-tool result callback, regardless of the ToolSessionLog persist
  gate. Surfaced through SessionEndInfo.ByTool at fireSessionEnd.
• Reporting (pkg/agent/postsession/cost): grand total is now
  Σ ByModel + Σ ByTool; tool cost is always provider-reported (never
  table-priced) so top-level PricingKnown is unchanged for existing sessions.
  The AmountMicroUSD/PricingKnown doc contract is updated: when a served
  model is unpriced, the total is now a documented lower bound (the priced
  components) rather than forced to 0. The closing cost notice gains a
  Sub-agent tools line.
• admind (pkg/web/admind): the session-detail breakdown carries byTool;
  the budget breakdown and the Overview headline spend panel add each session's
  tool cost on top of the token-repriced estimate, over the same session scope
  that feeds the per-model rollup (no double-count).

Report-only: there is no cost-budget enforcement, and no payer/passthrough
marker (the inner spend is billed to the user's own subscription under
passthrough — a follow-up could annotate this).

Alternatives considered

• Fold inner cost silently into amountMicroUSD — rejected: it breaks the
  "byModel buckets sum to the total" invariant and loses the provenance of a
  non-model cost. Itemizing under byTool keeps one headline number while
  showing where it came from.
• Two separate top-level totals (operator-metered vs passthrough) — rejected
  for v1 in favor of a single itemized total; the split can be reintroduced as a
  billedTo marker later without a schema break.
• Re-read tool_session records at SessionEnd instead of in-process
  accumulation — rejected: the result callback fires regardless of the
  ToolSessionLog persist gate, so in-process accumulation is strictly more
  reliable and matches the existing addModelUsage idiom.

Provenance

• Author (person, or model and version): Joseph Schorr
• Harness or tooling, with version: local development; mage targets for
  codegen and the test gate
• Person who read the diff: Joseph Schorr (on review)

Ship gate

• mage test:unit
• mage test:integration
• mage test:e2e

Results:

test:unit          exit 0, 0 failures (re-run at HEAD after the Overview fix)
test:integration   exit 0, 0 failures
test:e2e           exit 0, 0 failures — bronzethread + steelthread both ok

Regeneration

• Changed a CRD-shaping field in pkg/apis/v1alpha1/*_types.go → ran
  mage gen:api and mage manifests
• Edited anything under config/** directly → n/a (only via gen:api)
• Changed a CRD schema → ran mage docs:crd
• mage fmt:check clean

Coverage

• User-visible behavior is exercised by tests — unit coverage for the
  accumulator, the SessionEnd wiring, the cost reporter (grand total,
  itemization, unknown-model lower bound, no-tool regression), and the three
  admind surfaces (session-detail, budget, Overview). Not a new agent-facing
  tool call, so no bronzethread bundle is required; the whole e2e + steel
  suites remain green.
• Touches authorization / SpiceDB schema / approver model → n/a (none
  touched; estimatedCost is controller-owned status)

Before requesting review

• Single change: session cost now includes inner interactive-toolkit spend.
• A person reviews the diff on this PR. The generated files
  (zz_generated.deepcopy.go, install.yaml, config/**,
  site/content/docs/crd-agentsession.mdx) are mechanical — re-run
  mage gen:api && mage manifests && mage docs:crd and confirm the output
  is identical rather than reading them line by line.

Known follow-ups (deferred, non-blocking)

• The byModel budget axis total excludes tool cost (tool spend has no served
  model to attribute under) while the other axes include it — a deliberate
  asymmetry worth a doc note or a synthetic "sub-agent tools" row.
• An interactive-tool result that reports no billing still records a
  {tool, $0, pricingKnown: true} bucket in status.byTool (the parser
  collapses "reported 0" and "reported nothing"); no displayed total is affected
  since all surfaces gate on > 0. A costUSD > 0 gate or a threaded "reported"
  bool would tidy the raw status.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
openagentprimitives	Ready	Preview	Oct 5, 2026 5:28pm UTC

[samkim]

Review of the interactive toolkit cost changes (Claude Code /code-review, medium effort). Inline comments cover lines in the diff. Three issues sit on lines outside the diff:

1. The cost stamp's "keep the higher total" check now compares totals that mix model and tool cost (pkg/agent/runner/status.go:197). The per-tool costs live only in the pod's memory, so this can fail in two ways:

• Tool spend wiped: pod A stamps $1 of model cost plus $5 of tool cost ($6). After idle, reap and wake, pod B spends $7 on the model alone and stamps $7. That replaces the record and clears ByTool, so the $5 of sub-agent spend disappears from status and admind.
• Priced breakdown blocked: an earlier unpriced stamp of $5 in tool cost alone (PricingKnown=false) blocks a later priced $3 stamp, so the priced breakdown is never recorded.

2. The cost notice hides priced tool cost when model pricing is unknown (pkg/agent/postsession/cost/cost.go:193). costNotice returns "Session cost unavailable" and ignores c.ByTool and c.AmountMicroUSD. A session on an unpriced model that drives a $5 claude sub-run stamps $5 into status, but the notice never mentions it, so the notice and status/admind disagree.

3. Subscription (OAuth) sub-run costs are counted as billed spend. This is a product question rather than a bug: on a Max subscription, the claude CLI's total_cost_usd is an API-equivalent estimate, not money actually charged. Adding it to the notice and the admind budget overstates spend and could trip cost alerts or ceilings. Should these be excluded, or labeled as estimated?

[samkim]

With tool cost folded into Est, the byModel axis now disagrees with the other axes:

• Sessions with ByModel buckets: byModel emits only the bucket costs, so ByTool spend is dropped. With ByModel=[sonnet $1] and ByTool=[claude-oauth $5], byAgentClass, bySession and byUser each show $6, but byModel totals $1.
• Sessions without buckets: byModel emits s.Est, so the full $6 is charged to the outer model and no row shows where the $5 went.

Giving tool spend its own row in byModel (or keeping it out of Est for that axis) would keep the axes consistent.

[samkim]

Tool cost comes from a live a.agg.Snapshot() on every request, but ov.ByModel comes from the Overview engine's cache, which can be up to one TTL stale. Within that window the headline mixes two points in time: a session that just finished contributes its tool cost but not its model cost, and sessions that left the aggregator lose their tool cost but keep their model cost. The "no double-count" claim in the comment only holds when the two line up. Carrying tool cost on overview.LiveSession and summing it inside the engine's compute would avoid this.

[samkim]

ok is passed from main.go through buildToolSessionEventPublisher and then discarded with _ = ok. Since this is a new exported method, could it be dropped to AddToolCost(outerTool string, costUSD float64)? Readers will otherwise assume success affects the accounting.

[samkim]

The bucket is appended before the costUSD < 0 guard returns, so a single negative report leaves {Tool: x, Amount: 0, PricingKnown: false} in status. That contradicts the "tool buckets are always priced" contract toolCostUSD relies on. Moving the negative check above the bucket creation fixes it.

[samkim]

costReported is set for any non-negative cost, including 0. If the parser's ev.cost() returns 0 when total_cost_usd is absent (worth confirming), a toolkit that never reported a cost still produces a bucket with PricingKnown=true, asserting a known $0 instead of "not reported". That is the absent-versus-zero confusion Outcome.Unbilled's doc warns about. The comment in TestLoopAddToolCost_ZeroCostUnbilled says zero should not mark the bucket reported, but the test never asserts it.