Skip to content

test(dashboard): give the slot-close race double the body surface read_bounded_json reads - #8536

Merged
bolichen97 merged 1 commit into
mainfrom
test/slot-close-race-req-double
Sep 4, 2026
Merged

test(dashboard): give the slot-close race double the body surface read_bounded_json reads#8536
bolichen97 merged 1 commit into
mainfrom
test/slot-close-race-req-double

Conversation

@iamwhatever

Copy link
Copy Markdown
Collaborator

Problem / Motivation

test/test_slot_close_recreation_race.py fails on every PR whose merge commit carries it, on both Backend Tests shard-4 lanes, and it is blocking unrelated PRs from reaching green (#8518).

Every one of the eleven test_cleanup_* cases is broken, and only one of them says why:

FAILED test_cleanup_ordinary_archive_still_saves_and_removes - AttributeError: '_Req' object has no attribute 'can_read_body'
FAILED test_cleanup_recreate_during_save_preserves_replacement - Failed: Timeout >120.0s
FAILED test_cleanup_first_guard_hands_the_marker_to_the_replacement - worker 'gw2' crashed

api_chat_slots_cleanup reads its body through read_bounded_json, whose very first statement is if allow_absent and not request.can_read_body. The _Req test double in this file only stubs json(), so the handler raises AttributeError before it does anything else.

Ten of the eleven never report that error. They arm a live turn, launch the handler as a task, and then park on an asyncio.Event that the monkeypatched save_slot_off_loop sets — but the handler now raises before it reaches that save, so the event never fires and the test blocks until its own deadline. On Linux that is Timeout >120.0s; on Windows the xdist worker is torn down and the failure reads worker 'gwN' crashed.

Why it matters

The one-line cause was invisible and the cost was not: the file alone burned 181s locally and ~20 minutes of CI wall-clock per lane, and produced 15 failures across the two lanes in #8518's evidence — none of which named the defect. Because it fails on the merge ref rather than on a branch, it reds PRs that never touched the dashboard, and the loudest symptom (a crashed worker) points at the harness instead of at a missing attribute on a fake.

What changed (motivation → approach → change)

Symptom: eleven test_cleanup_* failures with three different shapes. Root cause: oneapi_chat_slots_cleanup was migrated onto read_bounded_json (the consolidation in #5587's line of work), and the test double was left stubbing only json(). The three shapes are one AttributeError plus ten tests whose parking event is downstream of where it now raises.

The fix is to give _Req the request surface the helper actually reads, rather than the one surface the handler used to read:

  • can_read_body — the first thing read_bounded_json touches, and the whole failure.
  • content_length / content / charset — the capped path (max_bytes defaults to 64 KiB) streams the body off request.content.iter_chunked instead of calling request.json(), so a double that stubs only json() would read an empty body there even once the attribute exists. A small _Stream chunks at the helper's own 8 KiB step.
  • json() is kept: the max_bytes=None path still calls it.

Every field is derived from the body the double was constructed with, so the streaming path sees exactly the bytes json() would have returned. body is None (what all 42 call sites in this file pass) is no body and reads as can_read_body == False; an explicitly-passed {} is a body that is present and empty, which is what aiohttp would report as readable — keying off truthiness instead would have made the double lie about that case.

Scope is deliberately the double, not the handler: read_bounded_json is correct, and api_chat_slot_delete — the other handler these tests drive, and the source of the 31 cases that were already passing — never reads the body at all.

Tests

No new tests. This repairs the 42 existing assertions in test/test_slot_close_recreation_race.py, which pin the close-vs-recreate teardown guards from #7212 and were all unreachable on the cleanup half.

Before and after, same file, same machine:

result wall-clock
before 10 failed (1 AttributeError, 9 Timeout >120.0s) 181.43s
after 42 passed 1.52s

That before/after is the proof the change is load-bearing; there is no production hunk to revert, so prove.py has nothing to prove here.

Manual verification

N/A — unit coverage sufficient; the repaired assertions are the verification.

Local gates run on this diff, all green: run_scoped_tests --test, leaf_test_scope --test, check_black_formatting, isort, flake8, mypy src/kiro_crew/, subprocess-encoding, agent-SDK-boundary, sync-IO-in-async, lockdown-before-publish, brand-name, harness-parity, feature-map, loop-bound-locks, builtin-skill-scope, testpaths-coverage, changelog-history, focus-cue, docs-lint, per-file-coverage, scrub-lint. Frontend/deploy gates were not run: the diff touches no frontend, template or deploy path.

The full backend suite was also run on this branch and on pristine origin/main. test_slot_close_recreation_race.py contributes zero failures on this branch. The residual reds are host-environment artifacts that reproduce identically on the base — the runner's real $HOME (test_host_isolation_floor, test_file_explorer_app), uid 65534 on /local/home (test_service), AF_UNIX path too long (test_dashboard_peer_auth), host CPU count (test_xdist_host_budget), and an absent gh binary (test_issue_radar_gh_bin). The failure sets on branch and base are identical.

Related Issues

Fixes #8518

Pattern harvest

Rule candidate: review-prompt
Pattern: migrating a handler onto a shared request-reading helper without updating the hand-rolled request doubles that drive it — the helper reads attributes (can_read_body, content, content_length, charset) the old await request.json() never touched, so a double that stubs only json() breaks, and it breaks at the top of the handler, upstream of any event a test parks on. That ordering is what turns a one-line AttributeError into a fleet of 120s timeouts and crashed workers. read_bounded_json's own docstring already warns that the capped path moves a caller off request.json() and that "that handler's unit tests must feed content/content_length rather than mocking json" — the note exists and was not applied at this call site, so the generalizable rule is to treat the docstring's warning as a checklist item on every conversion, and to look for hand-rolled _Req-style doubles in the converted handler's test files.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…d_bounded_json reads

api_chat_slots_cleanup moved onto read_bounded_json, which touches
request.can_read_body before anything else. The _Req double in
test_slot_close_recreation_race.py only stubbed json(), so every
test_cleanup_* case raised AttributeError inside the handler.

Ten of the eleven never reported that error: they park on an asyncio.Event
the monkeypatched save sets, and the handler now raises before reaching
the save, so the event never fires and the test blocks to its timeout --
120s each in CI, or an xdist worker crash. One file cost 181s locally and
15 failures across two CI lanes.

Give _Req the surface the helper actually reads: can_read_body,
content_length, charset and a chunked content stream, all derived from
the body it was constructed with, so the capped streaming path sees the
same bytes json() would have returned.
@iamwhatever
iamwhatever requested a review from a team as a code owner September 4, 2026 19:26
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of f710b131caaa34ae6c042d8c1ac4f507d8630c14 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Test-only double widened to the exact surface read_bounded_json reads, following that helper's own documented testing contract — proportionate and sound.

[DESIGN-REVIEWED] f710b13

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f710b131caaa34ae6c042d8c1ac4f507d8630c14 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f710b13

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f710b131caaa34ae6c042d8c1ac4f507d8630c14: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f710b131caaa34ae6c042d8c1ac4f507d8630c14 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f710b13

Verdict parsed from the review's SHA-scoped output markers for commit f710b131caaa34ae6c042d8c1ac4f507d8630c14.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f710b131caaa34ae6c042d8c1ac4f507d8630c14: <one-sentence reason>

@bolichen97
bolichen97 merged commit 5bc2fc7 into main Sep 4, 2026
60 of 63 checks passed
@bolichen97
bolichen97 deleted the test/slot-close-race-req-double branch September 4, 2026 20:47
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 4, 2026
pepmach added a commit that referenced this pull request Sep 5, 2026
## Problem / Motivation

A dashboard session produced three consecutive `Empty model response` warnings. Two of them were **not empty**: the transcript shows an assistant preamble and successfully completed tool calls, and the usage records show real billed inference ending on a clean `end_turn`.

`assistant_text` is reset at every tool boundary, so a turn that answers and then calls a tool reaches the terminal chain with an empty *final segment*. Nothing recorded that visible output had already been flushed — `_produced_visible_output` is deliberately narrow (it is only set on paths that reset the buffer mid-turn, and the promise-only guard depends on that meaning). So the empty-response ladder classified a productive turn as "the model returned nothing" and took rung 1, which **re-queues the ORIGINAL user message**. That re-runs tool calls that already completed — a second `send_message`, a second write, a second PR — and re-derives an answer the user has already read.

The third attempt was genuinely empty, and could not be diagnosed at all. Five physically different realities collapsed onto the same branch and the same log line:

- the provider generated nothing (`end_turn`, no content),
- the terminal carried **no stop reason**,
- **no terminal event arrived**,
- the terminal was **synthesized** by the provider layer,
- the turn produced only **tool calls or thinking**.

The warning printed none of the state the runner already held, so the three attempts were indistinguishable in the log even though they had three different owners.

## What changed

**A productive turn can never be replayed verbatim.** `EmptyTurnActivity.productive` is the load-bearing predicate — a flushed visible segment, a dispatched tool call, or thinking. Such a turn skips rung 1 entirely and gets at most **one** continuation, whose body does not tell the model its completed work produced no output (that wording is itself an invitation to redo the side effect). That continuation consumes the remaining recovery budget, so a second continuation can never follow.

**A genuinely activity-free empty turn is unchanged** — it keeps the bounded replay → continue → give-up ladder, which is the self-heal a real provider-side empty depends on.

**The verdict now names its cause.** One warning per verdict, emitted after the rung is chosen, carrying a closed cause (`no_terminal_event`, `synthetic_completion`, `visible_partial`, `tool_only`, `thinking_only`, `provider_empty`, `other`), the rung, a normalised stop reason, and booleans. An **omitted** stop reason is deliberately distinct from a clean `end_turn`: "the provider said the turn ended and produced nothing" is a model-side event, "the provider never said why it stopped" is a transport-side one, and that is the distinction the incident's third attempt needed and did not have.

**Two ACP-side silences are closed.** The pre-turn drain destroys frames from an abandoned turn — possibly including that turn's terminal — and said nothing; it now reports **how many**, once per turn. And a prompt stream that exhausts *cleanly* without ever yielding a terminal completion now warns, which is the one state the dashboard structurally cannot tell apart from a model that answered with nothing. Only clean exhaustion is reported, so an ordinary consumer close or cancellation stays silent.

## Privacy

Every diagnostic value is a bool or a member of a closed set. No prompt, no response text, no thinking, no tool arguments or results, no paths, no identities, **no token counts and no costs** — billing is a bool (`billed`), which is exactly the fact needed to separate "the provider ran and charged for this turn" from "it never ran". Frame counts stay in the drain's own log line and carry no sizes, because a size leaks response length.

## Testing

- `TestProductiveTurnNeverReplaysVerbatim` — the incident's own shape (`TEXT → TOOL_CALL → TOOL_RESULT → COMPLETE(end_turn)`) must not re-queue the prompt, must queue exactly one continuation, and the completed tool must be dispatched once. Plus tool-only, thinking-only, and the scoping test that a genuinely activity-free turn **keeps** rung 1.
- `TestEmptyTurnDiagnostics` — cause and rung vocabularies asserted from the classifier's own inputs (the ranking between overlapping causes is what regresses silently); provider-empty vs no-terminal reported distinctly; an omitted stop reason not laundered into `provider_empty`; and a privacy test that drives a turn carrying secret-marked text, a secret-marked tool title and real billing amounts, then asserts none of them appear in the rendered line while `billed=True` does.
- ACP: the drain reports a count and leaks no frame content; a clean exhaustion without a terminal warns, while a consumer close does not.
- **Mutation-checked.** Replacing the productive-turn guard with `True` turns exactly the three productive cases red (`the ORIGINAL user message was re-queued after a turn that already ran a tool`) while the provider-empty case stays green.

`test_subagent_delivery_ttl_anchor.py`'s consumption-signal guard is re-anchored on the rung marker rather than the branch condition, because that condition now carries the guard and is reformatted whenever it grows a term.

## Validation

Rebased onto current `main`; the three affected files are **1,097 passed**, and `mypy --platform linux` is clean across **1,288** files. Black ratchet, subprocess-encoding ratchet, isort, flake8, docs-lint and the brand gate all pass. `session_handle.py` stays grandfathered in the black baseline — the added hunks are black-clean, and formatting the file wholesale would rewrite 33 pre-existing regions.

**Correcting an earlier claim in this PR's first revision:** the first push attributed a set of local shard failures to the environment. That was wrong for the ones that mattered. `test/test_slot_close_recreation_race.py` was failing because the branch was based on a `main` that predated #8536, #8549 and #8583 — the three commits that widened the slot-race request double's `can_read_body` surface — and #5697, which serializes slot model switches under the slot lock. On the stale base that file timed out; on current `main` it is **42 passed in 4.8s**, with no change to this PR's code. The rebase is the fix, and no source change was needed to obtain it.

`test/test_transcribe.py` still cannot be collected in the local CI-parity venv (`imageio_ffmpeg` absent). That one is genuinely unrelated — this branch touches no transcribe, voice or STT code.

## Pattern harvest

Rule candidate: semgrep
Pattern: a boolean "did this turn produce output" flag consulted on a recovery or retry path that can re-send a side-effecting request

**Defect class:** a predicate whose narrow, documented meaning is correct for its original caller, reused by a later branch that needs a *broader* question answered — here "did this turn produce visible output" answered by a flag that only means "was the buffer reset mid-turn". The second caller acts on a false negative, and because its action is a *replay*, the cost is duplicated side effects rather than a merely wrong message. The two callers are 4,000 lines apart, so neither reads like a misuse locally.

**Retired by:** the activity snapshot is assembled from state the runner already held (terminal seen, synthetic, text, flushed, tools, thinking, billing) and the recovery decision reads `productive`, so a future branch asking the same question gets an answer scoped to it rather than borrowing one. The mutation check pins that the guard is load-bearing, and the cause enum makes a misclassification visible in the log instead of silent.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_slot_close_recreation_race fails on every PR merge commit (worker crash on Windows, timeouts + stub AttributeError on Linux)

2 participants