fix(chat): suppress leaked token budget banners

[rubencu]

Problem / Motivation

Long-running model turns can emit a provider-internal context-budget sentence such as You have 8154 weighted tokens left as ordinary assistant prose. In observed dashboard sessions, that sentence became the final assistant row and inherited the turn's file-change card instead of the user receiving a real completion summary.

Why it matters

The artifact is confusing on its own, but it can also make incomplete work look finished. Normal chats may end without a useful answer, and orchestrated stages can be captured or skipped even though the model emitted only provider metadata.

What changed (motivation → approach → change)

The symptom comes from provider text entering the same streamed-text path as a real answer. This change handles it narrowly at the authoritative turn and orchestration boundaries:

• Recognize only the leading You have <N> weighted tokens left shape for provenance-bearing tool, orchestrator-stage, and synthetic provider-recovery turns. Preserve quoted/natural token discussion, mid-turn text, any real answer that follows, and all text when the user explicitly requested this literal phrase.
• Route normal and exceptional partial output through one sanitizer so cancellation, authentication, process death, prompt-busy, retryable, post-token, and terminal-error paths cannot persist the provider artifact.
• For a banner-only opted-in turn, continue at most once on the same conversation without replaying the original prompt or completed tools. Provider-recovery queue entries use enqueue-time Stop generations and restore only their represented budget when superseded.
• For orchestrated stages, retry synchronously under one absolute deadline and nested prompt depth. Only a clean end_turn or structural provider artifact is terminal-eligible; capture requires a real non-compaction assistant result after the current stage separator.
• Track the active uncaptured stage immediately after record_round and remove only that round on every pre-capture exit, including Stop, cancellation, timeout, internal error, missing/timed-out subagent manager, empty/compaction-only result, capture failure, and teardown. Prior completed results, escalation state, and task-failure history remain authoritative.
• Preserve regeneration variants and bind file-change metadata to the current turn rather than the preceding assistant row.

Broader no-provenance and no-tool/status-only compaction handling is intentionally out of scope and tracked in #7961.

Tests

• Added focused matcher and lifecycle regressions for standalone/newline/chunked/quoted/natural variants, provenance and explicit-request gates, mid-turn preservation, shared exceptional sanitization, same-conversation bounded continuation, Stop supersession, regeneration, and current-turn file-change attribution.
• Added orchestrator regressions for nested prompt depth, one shared deadline, clean terminal gating, repeated artifacts, compaction-only output, Stop/cancel/timeout/error paths, subagent-manager failures, result-capture failure, centralized round rollback, and stage-2 resume state.
• 928 tests passed on fresh origin/main (18aef5166): the complete test_dashboard_chat.py plus provider-banner, post-compaction, regeneration, file-change, orchestrator-cancellation, and config-load suites.
• Black passes the five diff-heavy source/test files; the installed Black version reports three unrelated baseline formatting hunks in context_management.py, reproduced on pristine main and outside the changed abort_round docstring. isort, flake8, mypy, and git diff --check pass across all changed files.
• Aggregate mutation proof passes: reverting the production hunks while retaining the tests produces assertion failures.
• Independent GPT 5.6 and Opus 4.8/AUTOSDE mirrors pass on the reviewed fresh-base tree with no blocking or advisory findings.

Manual verification

Verified two independent saved dashboard screenshots/transcripts showing the same exact provider artifact attached to different file-change cards. Both raw rows were assistant content, not a context-meter UI label; the second screenshot was treated as corroborating evidence for the same defect, not as evidence to widen the matcher beyond the proven provenance boundary.

Related Issues

no linked issue: the dashboard bug was reported directly; #7961 tracks the broader cross-surface/no-provenance follow-up.

Pattern harvest

Rule candidate: review-prompt
Pattern: provider-only status text emitted through the assistant channel must be sanitized at authoritative persistence boundaries using explicit provenance, with bounded continuation and cancellation-safe orchestration.

Checklist

• At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
• Existing tests pass and new tests added for new functionality
• Self-review completed; code follows project style guidelines
• Documentation updated (N/A — no public interface or operator workflow changed)
• No secrets, credentials, or internal references in the diff

[github-actions]

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of e00588a63bac81069ef88789884a49a4e12209c2 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound, bounded mitigation at the right shared layer, but it fingerprints one provider's exact English artifact and smuggles a broader Stop/rollback rework along.

Watch

• The whole defense is one exact regex (You have <N> weighted tokens left) in chat_runner. If the backend rewords, localizes, or grows a second artifact shape, the filter silently misses and the original harm returns with no signal — nothing logs a near-miss. This is inherently symptom-side (the leak originates in the provider); make sure the upstream leak is reported/tracked, or this becomes a growing regex museum.
• The stage-loop Stop-generation snapshot + abort_round rollback changes cancellation/capture semantics for every orchestrated stage — including pre-existing await points (context prep, subagent wait) unrelated to the banner retry. It's documented and tested, but it's a second behavior change with its own regression surface riding a banner fix; treat regressions in "Stop during a stage" as suspects of this PR, not the banner logic.
• _provider_artifact_context requires tool calls, stage execution, or a recovery turn — so a banner leaked on a plain no-tool chat turn (a harm the description names: "Normal chats may end without a useful answer") still lands verbatim. Deliberate precision tradeoff, but the stated problem is only partially covered.

[DESIGN-REVIEWED] e00588a

[github-actions]

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of e00588a63bac81069ef88789884a49a4e12209c2 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All lenses run; verifying counts are complete. Final review:

First-Principles-Verdict: CONCERNS

The banner fix earns its place, but it patches one of six delivery surfaces, and a pre-existing Stop-race fix rides along inside it.

What this change ships

Intent: stop a provider-internal "You have N weighted tokens left" line from becoming the final assistant message (and stealing the file-change card) — a FIX, backed by two screenshots.

1. Leading budget-banner stripped before a dashboard message persists — justified (reported defect).
2. Banner-only turn auto-continues once via the existing _POSTTOKEN_RECOVER_MSG/_posttoken_retry_used machinery — justified, reuses the existing mechanism rather than adding one.
3. "Press Continue" notice when the one continuation is spent — justified (bounded, no loop).
4. Already-streamed banner replaced client-side by an empty assistant frame — justified.
5. Banner-only turn's file-change card pinned to an empty current-turn row — justified (the observed misattribution).
6. New provider_budget_artifact stop reason — justified; 2 real consumers in chat_orchestrator.py.
7. Orchestrated stage retries once under the original stage deadline — justified.
8. Second banner in a stage halts auto-run before result capture — justified.
9. Stop-generation guard across context prep, turn, continuation, subagent wait, capture + abort_round rollback — rides along; fixes a pre-existing Stop race, not the banner.

Watch

• Point patch, 5+ unfixed siblings. Grep EVENT_TEXT_CHUNK → 24 files; the same provider stream is persisted/delivered by slack/handler.py, messaging/driver.py, cli_chat.py, channel.py, task_executor.py, subagent.py — none strips the banner, and the helper is private to dashboard/chat_runner.py so they can't reuse it. The description's own harvest rule ("must be sanitized at both live-stream and persistence boundaries") is delivered for the dashboard only. Accepted-and-deferred, but say so.
• Item 9 is a second fix inside this one. The base loop already had zero stopped-check between _build_stage_context and dispatch (chat_orchestrator.py:479-510); guarding that window fixes a real pre-existing race (its zero option costs a user whose Stop lands there), so it deserves to exist — but it is a separate defect shipping under the banner title. Declared in the description, so noted, not blocked.

[FIRST-PRINCIPLES-REVIEWED] e00588a

[github-actions]

GPT 5.6 Review (fork) — ⚠️ review incomplete

Reviewed e00588a63bac81069ef88789884a49a4e12209c2 via the fork AI-review pipeline; updated in place on each push.

No completed GPT verdict for this commit; see the Fork GPT 5.6 Review job logs.

[github-actions]

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed e00588a63bac81069ef88789884a49a4e12209c2 via the fork AI-review pipeline; updated in place on each push.

Review details

The candidate list contains one item. Let me evaluate it against the falsification bar.

CANDIDATE 1 — hardcoded literal "provider_budget_artifact" in chat_orchestrator.py duplicating _STOP_REASON_PROVIDER_BUDGET_ARTIFACT.

Verifying the evidence: the diff line 162 defines _STOP_REASON_PROVIDER_BUDGET_ARTIFACT = "provider_budget_artifact", line 280 assigns it to slot._last_stop_reason, and the orchestrator (diff line ~541) compares slot._last_stop_reason == "provider_budget_artifact". The values are byte-identical today.

Applying the three-part bar:

• (a) concrete input occurring in practice: the candidate's own input is "a future change to the value of the constant" — that is a "might/if" hypothesis, not a condition that occurs in practice.
• (c) observable wrong outcome: none today; the candidate explicitly concedes "no present-tense misbehavior."

This is a latent-coupling/duplication concern, and Step 1 mandates rejecting anything in the duplication category. It also fails (a) and (c). Dropped.

No candidate survives. During falsification I read the recovery path, the stage-loop stop-generation gating, abort_round, _flush_segment, and the banner-strip guard; the logic is internally consistent with its test matrix and I found no grounded 80+ defect to originate.

No findings.

[OPUS-REVIEWED] e00588a

[rubencu]

• File-change attribution — fixed in e00588a63. Banner-only opted-in suppression now creates a non-broadcast empty assistant placeholder whenever the current turn has file changes. _attach_turn_stats and _flush_file_changes therefore bind to that current-turn row, never the preceding assistant message. test_banner_only_file_changes_use_current_turn_placeholder reproduces the screenshot shape and asserts the preceding row receives no file_changes metadata.

"Banner-only tool turns misattribute file changes"

• Text-only normal-turn suppression — rebutted as an intentional safety boundary. The previous review correctly established that content-only matching can delete a legitimate requested answer. The matcher is therefore restricted to observed structural provenance: terminal tool activity, orchestrator execution, or the synthetic recovery turn; it also defers whenever the user's prompt contains weighted tokens left. test_mid_turn_exact_text_is_preserved and test_explicit_requested_text_is_never_suppressed lock both preservation paths. The reported screenshots are tool turns with file-change cards, so the observed defect remains covered without reintroducing content-only answer deletion.

"text-only normal turns ... banner still persists"

[rubencu]

Maintainer rerun requested for the current-head GPT 5.6 Review (e00588a63bac81069ef88789884a49a4e12209c2). The check ran from 17:38:03Z to 18:43:18Z, then concluded failure with title review incomplete, zero annotations, and no current-head finding comment. All CI/coverage checks and the Opus, Design, First-Principles, and UX reviews are green; local GPT also passed this exact revision. Please rerun the fork GPT stage-2 review. This is a rerun request only—not an /ai-review override.

[rubencu]

• Exact provider sentence and future variants — accepted-and-deferred to Generalize provider-artifact sanitization across chat surfaces #7961. This PR intentionally matches only the independently observed You have <N> weighted tokens left artifact. Earlier content-only suppression deleted legitimate requested answers, so widening the matcher without provenance would regress correctness. Generalize provider-artifact sanitization across chat surfaces #7961 tracks a shared, provenance-aware boundary across stream consumers plus privacy-safe diagnostics for unknown variants; the current fix does not log message content.

"The whole defense is one exact regex ... make sure the upstream leak is reported/tracked"

• Stage Stop/rollback semantics — rebutted as required in-scope correctness. The new same-conversation stage continuation adds await and capture boundaries where a Stop can land. A monotonic Stop snapshot and rollback of only the uncaptured current round are required to prevent ignored cancellation, false stage completion, or a skipped stage on the next Go. Focused regressions cover Stop during context preparation, initial turn, continuation, subagent wait, and capture, plus interrupted stage-2 resume and repeated-artifact rollback. Prior stages, results, escalation history, and failure history remain preserved.

"The stage-loop Stop-generation snapshot + abort_round rollback changes cancellation/capture semantics"

• Plain no-tool turns — accepted-and-deferred to Generalize provider-artifact sanitization across chat surfaces #7961. The observed incidents are terminal tool turns with file-change cards, and this PR covers that evidence. Suppressing the sentence solely by content on normal turns is unsafe because the user can explicitly request that exact text; test_mid_turn_exact_text_is_preserved and test_explicit_requested_text_is_never_suppressed lock that boundary. Generalize provider-artifact sanitization across chat surfaces #7961 owns investigation of a reusable provenance signal that can safely cover additional delivery contexts.

"a banner leaked on a plain no-tool chat turn ... still lands verbatim"

[rubencu]

• Additional delivery surfaces — accepted-and-deferred to Generalize provider-artifact sanitization across chat surfaces #7961. This PR is scoped to the dashboard evidence: a provider banner persisted as the final assistant row after tool activity and inherited the turn file-change card. Generalize provider-artifact sanitization across chat surfaces #7961 is an actionable follow-up to trace all EVENT_TEXT_CHUNK consumers and add shared, provenance-aware live-stream and persistence handling for Slack, messaging, CLI chat, channel, task executor, and subagent paths. It explicitly preserves the no-content-only-suppression invariant proven necessary here.

"Point patch, 5+ unfixed siblings"

• Stop-generation guards and abort_round — rebutted as inseparable from safe stage recovery. A banner-only stage now continues synchronously on the same conversation under one absolute deadline. Without Stop checks around every await/capture boundary, or without removing the uncaptured current round, the new recovery path can ignore Stop, capture a false result, or skip the interrupted stage on resume. The implementation narrows rollback to that stage key and timeout clock while preserving prior completed stages/results, escalations, and task-failure history; focused tests cover each cancellation boundary and stage-2 resume.

"Item 9 is a second fix inside this one"

[rubencu]

• Owned provider-artifact stop reason — fixed in 28d6bbb1b (reviewed tree 120aef957). The orchestrator now imports _STOP_REASON_PROVIDER_BUDGET_ARTIFACT from chat_runner and uses it at all three terminal/continuation gates. A future stop-reason rename therefore cannot silently desynchronize stage capture.
• abort_round contract documentation — fixed in 28d6bbb1b (reviewed tree 120aef957). The docstring now names every pre-capture abandonment class: stop, timeout, error, empty result, capture failure, and teardown. This matches all eight orchestrator call sites and preserves the retry-not-skip invariant.

[bolichen97]

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

• PR #3771 is OVERLAPPING relative to this PR. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #3771: REBASE. Both branches edit the same _stage_loop capture block for unrelated reasons. Sequence them (land one, rebase the other) and keep both invariants: the off-loop write and the uncaptured-round rollback. Files: src/kiro_crew/dashboard/chat_orchestrator.py, test/test_dashboard_chat.py.
• This PR is OVERLAPPING with PR #6850. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7626: REBASE. Genuinely complementary detectors for two different non-answer text shapes; no code collision, only a consistency concern. Files: src/kiro_crew/dashboard/chat_runner.py.
• This PR is OVERLAPPING with PR #8288. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7626: REBASE. Two independent fixes for two different provider artifacts converging on one shared recovery mechanism and duplicating its purge/guard edits. They should agree on one owner for the _POSTTOKEN_RECOVER_MSG purge set and one policy for sharing (or splitting) the one-shot before either lands. Files: src/kiro_crew/dashboard/chat_runner.py.
• This PR is OVERLAPPING with PR #8318. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7626: REBASE. Different goals (banner suppression vs redaction-notice parity) but a hard mechanical collision on the same seven persist sites, plus an accidental overlap in the user-visible outcome. Sequence them and let the loser rebase onto the winner's shape. Files: src/kiro_crew/dashboard/chat_runner.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

[NicholasRBowers]

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author.

Assessment: Three mechanical blockers: (1) rebase onto current main and reconcile _stage_loop so both the merged off-loop completion-read behavior and this PR's uncaptured-round rollback hold — the 10 failures in test_completion_result_read_off_loop.py are the spec for that reconcile; (2) route the STOP_REASON_END_TURN import through kiro_crew.agent_sdk to clear the agent-sdk-boundary lint gate; (3) the fresh head re-fires the incomplete fork GPT lane.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

[bolichen97]

@rubencu Thanks for this. The audit found two open PRs that overlap this one, so sharing the detail here to let the three of you sequence the work.

#8288 (@Pearcekieser) is the hard collision. Both PRs rewrite the same _purgeable tuple and the same log string in _start_next_queued_turn in src/kiro_crew/dashboard/chat_runner.py, and both spend the same slot._posttoken_retry_used one-shot. The detectors differ (this PR strips "You have N weighted tokens left", #8288 matches the fixed retry sentence), so neither is a duplicate. Two things need a single owner. First, the purge set and its reset: this PR's conditional form (if _POSTTOKEN_RECOVER_MSG in purged_contents) is stricter than #8288's unconditional three-budget reset, so #8288 should adopt it. Second, the one-shot itself, which is pre-existing on main and already owned by the post-token transient-error retry at chat_runner.py:11891, so a third consumer needs an arbitration rule. You also both fix file-change misattribution independently, this PR by appending an empty current-turn row in _flush_segment, #8288 by adding a turn_boundary parameter to _flush_file_changes. Please keep one implementation.

#9156 (@buluoray) is lighter. It moves the slot._file_changes clear into _dedupe_file_changes, so the "is anything pending" probe your _flush_segment hunk relies on runs against a different accumulator lifetime. Both can land; whoever rebases second should recheck that probe.

Suggested order: agree on one file-change-attribution implementation and one one-shot policy, ideally as a small shared PR, then rebase onto it. Both this PR and #8288 are currently dirty.

Posted from the 2026-09-08 open-PR relationship audit (read-only, one auditor per PR); reply here if any of this is wrong.

[bolichen97]

Rebase onto main 1192049a3 attempted as part of the 2026-09-08 open-PR audit, then aborted. Your branch is untouched (still 28d6bbb1b), because the resolution needs decisions only you can make.

• chat_orchestrator.py: merged fix(autopilot): enforce the round cap, bound the plan, refuse dead plans #8618 replaced record_round(stage_num) with start_stage() and split capture into _collect_stage_result_parts + to_thread(_write_stage_result). Auto-merge silently kept only 2 of your 7 abort_round calls and dropped _active_uncaptured_stage, with no conflict marker. abort_round is still genuinely needed; the rollback points must be re-anchored to the new capture boundary.
• chat_runner.py: main now gates the answer branch on _answer_text (_answer_text_only), overlapping your strip; merged fix(chat): fire redaction notice on the 7 exception-path assistant persists #8318 added _append_redaction_notice to all 8 exception persists your _persist_partial_assistant() replaces, so a naive rebase drops that notice; and merged 422cf3e added a productive-empty arm beside your new elif _provider_budget_banner.

Also worth settling: _flush_segment(strip_provider_banner=True) has no production caller, and _posttoken_retry_used is main's pre-existing transient-error one-shot that open #8288 also spends. No gates were run. Reply if anything here looks wrong.