test(dashboard): give the slot-close race double the body surface read_bounded_json reads - #8536
Conversation
…d_bounded_json reads api_chat_slots_cleanup moved onto read_bounded_json, which touches request.can_read_body before anything else. The _Req double in test_slot_close_recreation_race.py only stubbed json(), so every test_cleanup_* case raised AttributeError inside the handler. Ten of the eleven never reported that error: they park on an asyncio.Event the monkeypatched save sets, and the handler now raises before reaching the save, so the event never fires and the test blocks to its timeout -- 120s each in CI, or an xdist worker crash. One file cost 181s locally and 15 failures across two CI lanes. Give _Req the surface the helper actually reads: can_read_body, content_length, charset and a chunked content stream, all derived from the body it was constructed with, so the capped streaming path sees the same bytes json() would have returned.
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only double widened to the exact surface [DESIGN-REVIEWED] f710b13 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
## Problem / Motivation A dashboard session produced three consecutive `Empty model response` warnings. Two of them were **not empty**: the transcript shows an assistant preamble and successfully completed tool calls, and the usage records show real billed inference ending on a clean `end_turn`. `assistant_text` is reset at every tool boundary, so a turn that answers and then calls a tool reaches the terminal chain with an empty *final segment*. Nothing recorded that visible output had already been flushed — `_produced_visible_output` is deliberately narrow (it is only set on paths that reset the buffer mid-turn, and the promise-only guard depends on that meaning). So the empty-response ladder classified a productive turn as "the model returned nothing" and took rung 1, which **re-queues the ORIGINAL user message**. That re-runs tool calls that already completed — a second `send_message`, a second write, a second PR — and re-derives an answer the user has already read. The third attempt was genuinely empty, and could not be diagnosed at all. Five physically different realities collapsed onto the same branch and the same log line: - the provider generated nothing (`end_turn`, no content), - the terminal carried **no stop reason**, - **no terminal event arrived**, - the terminal was **synthesized** by the provider layer, - the turn produced only **tool calls or thinking**. The warning printed none of the state the runner already held, so the three attempts were indistinguishable in the log even though they had three different owners. ## What changed **A productive turn can never be replayed verbatim.** `EmptyTurnActivity.productive` is the load-bearing predicate — a flushed visible segment, a dispatched tool call, or thinking. Such a turn skips rung 1 entirely and gets at most **one** continuation, whose body does not tell the model its completed work produced no output (that wording is itself an invitation to redo the side effect). That continuation consumes the remaining recovery budget, so a second continuation can never follow. **A genuinely activity-free empty turn is unchanged** — it keeps the bounded replay → continue → give-up ladder, which is the self-heal a real provider-side empty depends on. **The verdict now names its cause.** One warning per verdict, emitted after the rung is chosen, carrying a closed cause (`no_terminal_event`, `synthetic_completion`, `visible_partial`, `tool_only`, `thinking_only`, `provider_empty`, `other`), the rung, a normalised stop reason, and booleans. An **omitted** stop reason is deliberately distinct from a clean `end_turn`: "the provider said the turn ended and produced nothing" is a model-side event, "the provider never said why it stopped" is a transport-side one, and that is the distinction the incident's third attempt needed and did not have. **Two ACP-side silences are closed.** The pre-turn drain destroys frames from an abandoned turn — possibly including that turn's terminal — and said nothing; it now reports **how many**, once per turn. And a prompt stream that exhausts *cleanly* without ever yielding a terminal completion now warns, which is the one state the dashboard structurally cannot tell apart from a model that answered with nothing. Only clean exhaustion is reported, so an ordinary consumer close or cancellation stays silent. ## Privacy Every diagnostic value is a bool or a member of a closed set. No prompt, no response text, no thinking, no tool arguments or results, no paths, no identities, **no token counts and no costs** — billing is a bool (`billed`), which is exactly the fact needed to separate "the provider ran and charged for this turn" from "it never ran". Frame counts stay in the drain's own log line and carry no sizes, because a size leaks response length. ## Testing - `TestProductiveTurnNeverReplaysVerbatim` — the incident's own shape (`TEXT → TOOL_CALL → TOOL_RESULT → COMPLETE(end_turn)`) must not re-queue the prompt, must queue exactly one continuation, and the completed tool must be dispatched once. Plus tool-only, thinking-only, and the scoping test that a genuinely activity-free turn **keeps** rung 1. - `TestEmptyTurnDiagnostics` — cause and rung vocabularies asserted from the classifier's own inputs (the ranking between overlapping causes is what regresses silently); provider-empty vs no-terminal reported distinctly; an omitted stop reason not laundered into `provider_empty`; and a privacy test that drives a turn carrying secret-marked text, a secret-marked tool title and real billing amounts, then asserts none of them appear in the rendered line while `billed=True` does. - ACP: the drain reports a count and leaks no frame content; a clean exhaustion without a terminal warns, while a consumer close does not. - **Mutation-checked.** Replacing the productive-turn guard with `True` turns exactly the three productive cases red (`the ORIGINAL user message was re-queued after a turn that already ran a tool`) while the provider-empty case stays green. `test_subagent_delivery_ttl_anchor.py`'s consumption-signal guard is re-anchored on the rung marker rather than the branch condition, because that condition now carries the guard and is reformatted whenever it grows a term. ## Validation Rebased onto current `main`; the three affected files are **1,097 passed**, and `mypy --platform linux` is clean across **1,288** files. Black ratchet, subprocess-encoding ratchet, isort, flake8, docs-lint and the brand gate all pass. `session_handle.py` stays grandfathered in the black baseline — the added hunks are black-clean, and formatting the file wholesale would rewrite 33 pre-existing regions. **Correcting an earlier claim in this PR's first revision:** the first push attributed a set of local shard failures to the environment. That was wrong for the ones that mattered. `test/test_slot_close_recreation_race.py` was failing because the branch was based on a `main` that predated #8536, #8549 and #8583 — the three commits that widened the slot-race request double's `can_read_body` surface — and #5697, which serializes slot model switches under the slot lock. On the stale base that file timed out; on current `main` it is **42 passed in 4.8s**, with no change to this PR's code. The rebase is the fix, and no source change was needed to obtain it. `test/test_transcribe.py` still cannot be collected in the local CI-parity venv (`imageio_ffmpeg` absent). That one is genuinely unrelated — this branch touches no transcribe, voice or STT code. ## Pattern harvest Rule candidate: semgrep Pattern: a boolean "did this turn produce output" flag consulted on a recovery or retry path that can re-send a side-effecting request **Defect class:** a predicate whose narrow, documented meaning is correct for its original caller, reused by a later branch that needs a *broader* question answered — here "did this turn produce visible output" answered by a flag that only means "was the buffer reset mid-turn". The second caller acts on a false negative, and because its action is a *replay*, the cost is duplicated side effects rather than a merely wrong message. The two callers are 4,000 lines apart, so neither reads like a misuse locally. **Retired by:** the activity snapshot is assembled from state the runner already held (terminal seen, synthetic, text, flushed, tools, thinking, billing) and the recovery decision reads `productive`, so a future branch asking the same question gets an answer scoped to it rather than borrowing one. The mutation check pins that the guard is load-bearing, and the cause enum makes a misclassification visible in the log instead of silent.
Problem / Motivation
test/test_slot_close_recreation_race.pyfails on every PR whose merge commit carries it, on both Backend Tests shard-4 lanes, and it is blocking unrelated PRs from reaching green (#8518).Every one of the eleven
test_cleanup_*cases is broken, and only one of them says why:api_chat_slots_cleanupreads its body throughread_bounded_json, whose very first statement isif allow_absent and not request.can_read_body. The_Reqtest double in this file only stubsjson(), so the handler raisesAttributeErrorbefore it does anything else.Ten of the eleven never report that error. They arm a live turn, launch the handler as a task, and then park on an
asyncio.Eventthat the monkeypatchedsave_slot_off_loopsets — but the handler now raises before it reaches that save, so the event never fires and the test blocks until its own deadline. On Linux that isTimeout >120.0s; on Windows the xdist worker is torn down and the failure readsworker 'gwN' crashed.Why it matters
The one-line cause was invisible and the cost was not: the file alone burned 181s locally and ~20 minutes of CI wall-clock per lane, and produced 15 failures across the two lanes in #8518's evidence — none of which named the defect. Because it fails on the merge ref rather than on a branch, it reds PRs that never touched the dashboard, and the loudest symptom (a crashed worker) points at the harness instead of at a missing attribute on a fake.
What changed (motivation → approach → change)
Symptom: eleven
test_cleanup_*failures with three different shapes. Root cause: one —api_chat_slots_cleanupwas migrated ontoread_bounded_json(the consolidation in #5587's line of work), and the test double was left stubbing onlyjson(). The three shapes are oneAttributeErrorplus ten tests whose parking event is downstream of where it now raises.The fix is to give
_Reqthe request surface the helper actually reads, rather than the one surface the handler used to read:can_read_body— the first thingread_bounded_jsontouches, and the whole failure.content_length/content/charset— the capped path (max_bytesdefaults to 64 KiB) streams the body offrequest.content.iter_chunkedinstead of callingrequest.json(), so a double that stubs onlyjson()would read an empty body there even once the attribute exists. A small_Streamchunks at the helper's own 8 KiB step.json()is kept: themax_bytes=Nonepath still calls it.Every field is derived from the
bodythe double was constructed with, so the streaming path sees exactly the bytesjson()would have returned.body is None(what all 42 call sites in this file pass) is no body and reads ascan_read_body == False; an explicitly-passed{}is a body that is present and empty, which is what aiohttp would report as readable — keying off truthiness instead would have made the double lie about that case.Scope is deliberately the double, not the handler:
read_bounded_jsonis correct, andapi_chat_slot_delete— the other handler these tests drive, and the source of the 31 cases that were already passing — never reads the body at all.Tests
No new tests. This repairs the 42 existing assertions in
test/test_slot_close_recreation_race.py, which pin the close-vs-recreate teardown guards from #7212 and were all unreachable on the cleanup half.Before and after, same file, same machine:
AttributeError, 9Timeout >120.0s)That before/after is the proof the change is load-bearing; there is no production hunk to revert, so
prove.pyhas nothing to prove here.Manual verification
N/A — unit coverage sufficient; the repaired assertions are the verification.
Local gates run on this diff, all green:
run_scoped_tests --test,leaf_test_scope --test,check_black_formatting,isort,flake8,mypy src/kiro_crew/, subprocess-encoding, agent-SDK-boundary, sync-IO-in-async, lockdown-before-publish, brand-name, harness-parity, feature-map, loop-bound-locks, builtin-skill-scope, testpaths-coverage, changelog-history, focus-cue, docs-lint, per-file-coverage, scrub-lint. Frontend/deploy gates were not run: the diff touches no frontend, template or deploy path.The full backend suite was also run on this branch and on pristine
origin/main.test_slot_close_recreation_race.pycontributes zero failures on this branch. The residual reds are host-environment artifacts that reproduce identically on the base — the runner's real$HOME(test_host_isolation_floor,test_file_explorer_app),uid 65534on/local/home(test_service),AF_UNIX path too long(test_dashboard_peer_auth), host CPU count (test_xdist_host_budget), and an absentghbinary (test_issue_radar_gh_bin). The failure sets on branch and base are identical.Related Issues
Fixes #8518
Pattern harvest
Rule candidate:
review-promptPattern: migrating a handler onto a shared request-reading helper without updating the hand-rolled request doubles that drive it — the helper reads attributes (
can_read_body,content,content_length,charset) the oldawait request.json()never touched, so a double that stubs onlyjson()breaks, and it breaks at the top of the handler, upstream of any event a test parks on. That ordering is what turns a one-lineAttributeErrorinto a fleet of 120s timeouts and crashed workers.read_bounded_json's own docstring already warns that the capped path moves a caller offrequest.json()and that "that handler's unit tests must feedcontent/content_lengthrather than mockingjson" — the note exists and was not applied at this call site, so the generalizable rule is to treat the docstring's warning as a checklist item on every conversion, and to look for hand-rolled_Req-style doubles in the converted handler's test files.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)