Skip to content

feat: report subagent credit usage alongside elapsed time - #8003

Open
welikoiwanenko wants to merge 1 commit into
kirodotdev:mainfrom
welikoiwanenko:codex/subagent-credit-usage
Open

feat: report subagent credit usage alongside elapsed time#8003
welikoiwanenko wants to merge 1 commit into
kirodotdev:mainfrom
welikoiwanenko:codex/subagent-credit-usage

Conversation

@welikoiwanenko

@welikoiwanenko welikoiwanenko commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Managed subagent completion currently reports elapsed time but not the credits consumed by that run. Parents and users cannot attribute the cost of a successful, failed, or stopped subagent without comparing noisy account-level totals.

For maintainer approval: cumulative positive reported credits appear in expanded Subagents activity cards, with elapsed time retained in compact headers and usage included in completion messages and spawn_status.

Why it matters

A failed or stopped attempt can consume credits even when it produces no final answer. Per-run attribution makes those costs visible, including billed transient retries, without relying on unrelated activity in the same account.

What changed (motivation → approach → change)

  • Accumulate credits per attempted provider turn. Completion events supply usage on success; fresh prompt stats supply usage on interrupted or failed attempts. Each attempt settles once, including cancellation during event handling and force-reaping.
  • Persist terminal credits and elapsed time, expose them through spawn status responses, and replay credits on dashboard reconnect. The list response remains unchanged. Legacy retained records remain readable without invented usage fields.
  • Add usage to single completion, batch digest, delivery-failure, and retained orphan notifications, plus full and paged MCP status responses.
  • Display positive credits in expanded completed, failed, and stopped activity cards using the existing localized elapsed-and-credit summary. Keep elapsed time alone in compact headers; omit zero/unreported credit summaries. Backend summaries use matching credit precision.
  • Bound best-effort terminal persistence to five seconds before parent delivery proceeds. Keep pending write tasks owned and retain the conversation hold until every overlapping writer settles, protecting retention state from stale writes.

No dependency, configuration, schema migration, model-default, or changelog changes. Native harness subagents remain unattributed because they share the parent's billed turn. Providers that do not report credit billing return zero; displays omit that credit label rather than claim the run was free. This feature does not estimate charges or add dollar billing. Run totals are completion summaries, not additional billing-ledger entries; failed-attempt credits can make them larger than successful-turn records alone.

Tests

Added regression coverage for successful and failed attempts, billed retries, cancellation during consumer work, force-reaping during a state-write drain, stalled and late-failing terminal persistence, overlapping writer holds, restart/status compatibility, MCP error and paging responses, completion digests, Redux terminal events, and activity-card rendering.

On the rebased head, 601 focused backend tests passed across every changed backend test file plus the ACP frame-replay regression suite. The two changed frontend suites passed 73 tests; TypeScript, ESLint, documentation lint, and diff checks also passed. The full repository suite was not rerun locally; CI remains the authoritative full-matrix validation.

Manual verification

Rendered the real production SPA with the repository's scripted Playwright harness and fixture-only HTTP/WebSocket responses. Verified expanded elapsed-and-credit summaries and elapsed-only headers for completed, failed, and stopped subagents in dark and light themes, including a constrained 320px activity region. No live gateway or paid subagent calls were used. Live provider billing verification remains for maintainers.

Screenshots / video

Fixture data, real built dashboard; each card shows its own terminal usage.

Completed, failed, and stopped credit usage — dark theme

Completed, failed, and stopped credit usage — light theme

Full dashboard context

Dashboard with subagent credit usage

Related Issues

Implements the credit-attribution request in #4721. The issue has an earlier automated claim; this PR is for coordination and maintainer approval, not an assertion that design approval has already been given.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Fixes #4721

@iamwhatever

Copy link
Copy Markdown
Collaborator

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@welikoiwanenko
welikoiwanenko marked this pull request as ready for review September 2, 2026 22:05
@welikoiwanenko
welikoiwanenko requested a review from a team September 2, 2026 22:05
@welikoiwanenko
welikoiwanenko requested a review from a team as a code owner September 2, 2026 22:05
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 2, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 1dcffc7 to 6bb4677 Compare September 2, 2026 23:10
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

UX Review (Fable 5, fork) — 🟡 CONCERNS

UX-level review of 1bfa6119da6bc0351d1a176618387c785c60f800 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

The diff, PR body, base-tree string usage, and screenshot availability are all established. The only user-visible dashboard change is one line in the expanded subagent card, reusing the existing localized turn_took_credits string; the PR's screenshots are fork-added binaries not on disk here, and no blind read ran in this lane.

UX-Verdict: CONCERNS

"Turn took…" now labels a run-cumulative total; the new credit line's screenshots exist only on the fork, unseen by any first-time reader.

Watch

  • Vocabulary collision: ActivityViewer.tsx reuses pages.chat.assistantMessage.turn_took_credits ("Turn took {{elapsed}} and used {{credits}} credits") for a value the spec defines as cumulative across all attempted turns including billed retries, while the same string in AssistantMessage.tsx means one turn. A user comparing a multi-turn subagent's "Turn took 1m 5s and used 12.5 credits" against per-turn chat stats reads one phrase meaning two scopes and undercounts what the number covers. Every expanded terminal card, mild-to-moderate comprehension impact, persistent. Smallest fix: a run-scoped key ("Run took … and used … credits") instead of reusing the per-turn one.

Evidence gaps

  • The expanded-card credit summary (completed, failed, and stopped states; dark + light) — credits-dark.png/credits-light.png are added by the fork PR and not materialized here, and no blind read ran; push the branch to this repository so the blind-read lane can see them.

Suggestions

  • In the chat-visible delivery-failure/orphan messages, format_subagent_usage renders Usage: 12.5s when credits are zero — "Usage" labeling a bare duration; use "Elapsed: 12.5s" (or drop the label) for the zero-credit branch.

[UX-REVIEWED] 1bfa611

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 1bfa6119da6bc0351d1a176618387c785c60f800 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Verification complete. The design rests on existing contracts (resolve_billing_stats, TurnUsage.credits, per-prompt carry_over() holder replacement — all present in the base tree), the writer-hold generalization strictly widens the existing #6298 protection with the spec updated in the same commit, temp-screenshots/ is the repo's documented PR-screenshot convention, and legacy-record/zero-credit compatibility is handled explicitly rather than by invented zeros.

Design-Verdict: PASS

Attribution rides existing billing seams end-to-end; the settle-once accounting and widened writer holds map to real, tested hazards, not invented ones.

Suggestions

[DESIGN-REVIEWED] 1bfa611

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 1bfa6119da6bc0351d1a176618387c785c60f800 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All verifications are done: the frontend reuses the existing fmtCredits/turn_took_credits mechanism, the per-prompt holder-replacement premise is real (session_handle.py:1013), temp-screenshots/ is an established convention (1,288 files), the doc's timeout rewrite matches base code (run.py:379), and I counted zero readers of a credits field in tombstone.json (readers use only cause, outcome, died, cleanup identities).

First-Principles-Verdict: CONCERNS

tombstone.json gains a credits field with zero readers — state.json already persists the same value on every terminal path, including reaps.

Not justified as shipped

  1. tombstone credits field — zero consumers: grepped every read_tombstone/tombstone.json reader (backfill.py:358-360, messaging.py:632, continuation.py:145); none reads credits, and the reap test shows the state.json write carries it anyway.
  2. spec timeout-message rewrite — undeclared, rides along (verified correct against run.py:379).

What this change ships

Intent: let parents and users see what a subagent run cost, per run, at completion — an ADDITION (linked feature request #4721).

  1. Expanded terminal cards show "Turn took … and used … credits"; headers keep elapsed only — justified
  2. Completion, digest, delivery-failure, orphan and force-reap messages gain a Usage: line — justified
  3. spawn_status MCP responses gain a [usage: …] prefix (full, paged, error) — justified
  4. GET /api/spawn/{id} returns elapsed and credits for finished runs — justified
  5. subagent_done WS frames carry credits, live and on reconnect replay — justified
  6. Credits count every attempted turn once, including billed retries and cancels — justified
  7. Terminal usage persisted to state.json; legacy records stay usage-free — justified
  8. Every pending state write now holds the conversation until settled; delivery waits ≤5s for the terminal write — justified
  9. tombstone.json gains a credits field — undeclared, zero consumers
  10. Spec's timeout message corrected from fixed "180 minutes" to configured value — undeclared, rides along

Watch

The failed-attempt accounting rests on "providers replace this holder for each prompt" — verified for ACP (session_handle.py:1013 carry_over()); the Claude Code path merely degrades to zero via resolve_billing_stats, which is honest. No action needed beyond the tombstone field.
Clears when: credits=info.credits is removed from the _write_tombstone call, or a real tombstone-credits reader lands.

Subtractions

  • Drop credits=info.credits from the tombstone write in src/kiro_crew/subagent.py (_write_tombstone kwargs) — 0 readers counted across all read_tombstone/tombstone.json consumers; state.json persistence already covers restart survival.

[FIRST-PRINCIPLES-REVIEWED] 1bfa611

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 1bfa6119da6bc0351d1a176618387c785c60f800 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1bfa611

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 1bfa6119da6bc0351d1a176618387c785c60f800 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1bfa611

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Sep 3, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 6bb4677 to e3d8211 Compare September 3, 2026 08:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 3, 2026
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from dd99628 to 1630c44 Compare September 7, 2026 08:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch 2 times, most recently from 77c8d40 to e092b20 Compare September 7, 2026 09:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from e092b20 to a1aeb5b Compare September 7, 2026 10:05
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from a1aeb5b to 63a6148 Compare September 7, 2026 10:14
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 7, 2026
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 63a6148 to 845224a Compare September 7, 2026 10:33
Attribute reported credits to managed subagent runs, including failed and
cancelled attempts, so users and parent agents can compare their costs.

Persist terminal usage and show it in completion events, status queries,
and dashboard cards. Bound best-effort persistence before parent delivery
and settle active accounting before reap reports capture their totals.
@welikoiwanenko
welikoiwanenko force-pushed the codex/subagent-credit-usage branch from 845224a to 1bfa611 Compare September 8, 2026 10:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@bolichen97

Copy link
Copy Markdown
Collaborator

@welikoiwanenko Thanks for this. Keeping it: it is the only implementation of #4721, and nothing on main covers any part of it. Two open PRs touch the same code, so we should agree a landing order first.

#9008 (@javenciu) inserts at the exact line this PR does: TerminalCoordinator._report_terminal_impl in src/kiro_crew/subagent_manager/terminal.py, immediately after info.done = True. #9008 moves the whole report body into a new _report_terminal_guarded_impl; this PR adds info._credit_accounting.settle(), the credits key on the subagent_done payload, and a bounded terminal-usage state write. Both also add lines at the child-escalation and turn-limit bails in src/kiro_crew/subagent_manager/run.py, and both edit the inspect.getsource pin in test/test_subagent_scale.py. No behaviour is duplicated. #9008 is further along in that region because it already relocates that pin, so please let #9008 land first, then re-place the settle() call and the payload key inside _report_terminal_guarded_impl.

#9101 (@Premshay) is much smaller (5 files against 27 here) and collides only textually: both add a field to the SubagentInfo dataclass in src/kiro_crew/subagent.py, both add statements around _run_inner_impl in run.py (this PR changes the signature at line 620, #9101 sets _first_stream_started at 629 and 1167), and both edit neighbouring bullets in docs/system-specs/modules/subagent.md. Letting #9101 go first costs you a rebase and nothing else.

Please rebase either way: at audit time the branch was 46 commits behind main, and #7195 rewrites the same subagent.md timeout bullet you reword here.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface per-session credit usage in subagent completion events

3 participants