Skip to content

Reopen Claude's /tasks view until it reports completion; ask CUJ 4 for one subagent - #1247

Open
rohita5l wants to merge 5 commits into
mainfrom
fix-cuj4-background-task-wait
Open

rohita5l wants to merge 5 commits into
mainfrom
fix-cuj4-background-task-wait

Conversation

@rohita5l

Copy link
Copy Markdown
Collaborator

Stacked on #1245 (base ci-flake-hardening); this diff contains only the follow-up commit. Retarget to main once #1245 merges.

What

Fixes the two remaining CUJ 4 flake causes from the CI analysis in #1245. The diagnosis replays the raw PTY transcripts from CI's CUJ evidence artifacts (runs 38801294447 and 39696214480, with passing siblings) through pyte, plus the final screens from 19 failing jobs.

  1. wait_for_background_tasks (tests/integration/utils/terminal.py). "TUI did not show Claude's task view reporting no running tasks within 180s" was 17 of 43 flaky CUJ 4 jobs, and the same wait ends the Claude skill-toggle tests. The work had finished every time. The check was never flickering: in the failing transcript it is False continuously until the final repaint. There are two real causes:

    • No task view on screen (14 of 19). Claude was back at its idle prompt with every agent finished, either because a /tasks typed mid-turn was swallowed or because the view closed itself. The old wait never reopened it.
    • Frozen running snapshot (5 of 19). /tasks opened mid-turn and froze on a running state ("Background task update waiting while this panel is open"). The running header names the subagent type (ug-smart-router:… › Read file contents, footer … · x to stop · …), so the old agent › regex never even recognized it.

    The wait is now a bounded loop:

    • It resubmits /tasks when no task view is shown at an idle prompt, at most every 3 s.
    • It sends Esc, then /tasks, when the view hasn't reported completion for 10 s.
    • The detail regex accepts any <name> › <title> header and stays within one line.
    • Success still requires Claude's own task view to report that nothing is running, held for 0.3 s within the same 180 s deadline. Completion is never inferred from scrolled history.
  2. CUJ 4 "exactly one completed child" (about 9 failures with assert 2 == 1). These were genuine second subagents, not one subagent counted twice: every failing job has two distinct agent-<id>.jsonl transcripts. The parent sometimes started a "relay" subagent just to message the first one for its result, e.g. Agent(subagent_type="fork", description="Get subagent result", …) in job 119554254666.

    • The delegation prompt now adds "Use only that one subagent: do not start any other subagent."
    • assert len(completed_children) == 1 and all routing assertions are unchanged.

Both changes stay within tests/AGENTS.md. Reopening /tasks is real keystrokes through the visible UI, and the prompt change is scenario input. There are no mocks, skips, retries of the task, or weaker assertions.

Not covered / risks

  • The prompt change is policy-sensitive and unverified live. If relay spawns persist, the next option would be counting only children whose first prompt names the task file. That changes the assertion, so it isn't done here.
  • Esc race. Esc is sent when the view was open on the latest screen. If Claude closes the view in that instant, Esc lands on the main prompt. The old final close step had the same race.
  • First matching header wins. The detail regex matches the first <name> › <title> header on screen. A scrollback line containing " › " could make it misread the panel. That can only cause a missed completion, meaning another reopen, never a false success.
  • The 14/19 shape (swallowed or self-closed view) is confirmed from final screens only; the transcripts for those runs were unavailable.

Docs

  • tests/e2e_cuj/README.md: describes the reopen behaviour and the one-subagent prompt.

Tests

  • tests/test_e2e_cuj_helpers.py:
    • Predicate checks on the real CI screens: a view that closed itself, a stale running panel, and a completed panel.
    • Idle-prompt checks.
    • Fake-ug PTY cases: a swallowed /tasks followed by completion; a stale running view that gets reopened; both together; and a never-completing case that must still raise at the deadline without sending /exit.
    • The existing background-wait cases still pass.
  • tests/test_integration_contract.py, tests/test_e2e_cuj_helpers.py, tests/test_cuj_evidence.py and tests/test_tui_request_recorder.py: 339 passed.
  • tests/e2e_cuj collects 111 tests; tests/integration collects 151 of 163. ruff check and format are clean.
  • Not run live. This PR's CI run is the first real exercise of the reopen path.

This pull request and its description were written by Isaac.

rohita5l and others added 5 commits October 22, 2026 14:51
…outs once

The per-model launch tests looped over the live workspace catalog and failed
as a whole when any one model misbehaved, so a single bad model turned every
PR red. Parametrize them at collection time from a cached catalog fetch
(shared with the e2e_state fixture) so each model is its own case. Without
UCODE_TEST_WORKSPACE nothing touches the network; a catalog fetch error is a
single failing case, not a silent skip.

Add pytest-rerunfailures and rerun an e2e shard case once only for live
timeouts and transient upstream errors (TimeoutExpired, "timed out after",
"TIMEOUT after", TEMPORARILY_UNAVAILABLE, REQUEST_LIMIT_EXCEEDED); reruns are
listed in the summary. Integration and CUJ suites keep failing on the first
attempt per tests/AGENTS.md.

Co-authored-by: Isaac <no-reply@databricks.com>
The tracing tests slept 30s and queried once, and gave each SQL statement
180s, so a cold warehouse or ingestion lag failed them ("SQL statement did
not finish", "assert 0 > 0"). Give a statement 300s and poll the trace-row
count every 15s for up to 300s. The tests still assert the row exists.

Co-authored-by: Isaac <no-reply@databricks.com>
The recorder's handler threads are daemons, and teardown closed the httpx
client while a forward could still be streaming. That frees the socket
descriptor while OpenSSL keeps reading the old number, so a later connection
in the same process (the next CUJ class's SDK calls and inference requests)
had bytes stolen: "Response ended prematurely", SSL RECORD_LAYER_FAILURE,
"incomplete chunked read" and "Bad file descriptor". Every one of the 22
flaky CUJ 2 Claude failures ran right after a passing Codex class.

Teardown now waits for in-flight forwards before closing the client and
leaves a forward that outlives the wait to close its own connection.

Also:
- Record an upstream transport failure as the 502 the agent saw, so waiters
  fail immediately with the cause instead of timing out after 240s.
- Do not reuse pooled upstream connections.
- CUJ 2 asserts the first marker-bearing inference request that was served
  with HTTP 200 (an agent-retried transport failure no longer fails it), and
  uses one budget for the request and response waits.

Co-authored-by: Isaac <no-reply@databricks.com>
…tests

Both ran one long live session per config version, so one slow subagent or
model call failed every assertion chained after it.

- CUJ 4: test_smart_router_config_version becomes a per-agent preset routing
  test and a Claude status-band/hint/toggle test that needs no delegated work.
- Skill toggles: each agent's test becomes at-launch, off and back-on tests,
  each in a fresh session; back-on keeps the off->on sequence (and Claude's
  launch-recipe comparison) that it proves.

All existing assertions and config versions are kept. Coverage READMEs are
updated for the new test names.

Co-authored-by: Isaac <no-reply@databricks.com>
…r one subagent

wait_for_background_tasks submitted /tasks once and waited 180s for the view
to report completion. In CI (17 of 43 flaky CUJ 4 jobs) that never happened
although the work had finished:
- Claude swallowed a /tasks typed mid-turn, or closed the view itself, and
  the wait sat at the idle prompt.
- The view opened as a running snapshot and stayed frozen ("Background task
  update waiting while this panel is open"); its header names the subagent
  type, not "agent", so the detail regex never matched it.

The wait now reopens the view through the visible UI: /tasks again when no
view is shown at an idle prompt, and Esc then /tasks when the view has not
reported completion for 10s. Success still requires Claude's own task view
to report that nothing is running, within the same bounded deadline.

CUJ 4's "exactly one completed child" failures were real second subagents:
the parent sometimes started a "relay" subagent to message the first one for
its result. The delegation prompt now asks for that one subagent only; the
assertion is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
Base automatically changed from ci-flake-hardening to main October 28, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant