Repository navigation
Conversation
…outs once The per-model launch tests looped over the live workspace catalog and failed as a whole when any one model misbehaved, so a single bad model turned every PR red. Parametrize them at collection time from a cached catalog fetch (shared with the e2e_state fixture) so each model is its own case. Without UCODE_TEST_WORKSPACE nothing touches the network; a catalog fetch error is a single failing case, not a silent skip. Add pytest-rerunfailures and rerun an e2e shard case once only for live timeouts and transient upstream errors (TimeoutExpired, "timed out after", "TIMEOUT after", TEMPORARILY_UNAVAILABLE, REQUEST_LIMIT_EXCEEDED); reruns are listed in the summary. Integration and CUJ suites keep failing on the first attempt per tests/AGENTS.md. Co-authored-by: Isaac <no-reply@databricks.com>
The tracing tests slept 30s and queried once, and gave each SQL statement
180s, so a cold warehouse or ingestion lag failed them ("SQL statement did
not finish", "assert 0 > 0"). Give a statement 300s and poll the trace-row
count every 15s for up to 300s. The tests still assert the row exists.
Co-authored-by: Isaac <no-reply@databricks.com>
The recorder's handler threads are daemons, and teardown closed the httpx client while a forward could still be streaming. That frees the socket descriptor while OpenSSL keeps reading the old number, so a later connection in the same process (the next CUJ class's SDK calls and inference requests) had bytes stolen: "Response ended prematurely", SSL RECORD_LAYER_FAILURE, "incomplete chunked read" and "Bad file descriptor". Every one of the 22 flaky CUJ 2 Claude failures ran right after a passing Codex class. Teardown now waits for in-flight forwards before closing the client and leaves a forward that outlives the wait to close its own connection. Also: - Record an upstream transport failure as the 502 the agent saw, so waiters fail immediately with the cause instead of timing out after 240s. - Do not reuse pooled upstream connections. - CUJ 2 asserts the first marker-bearing inference request that was served with HTTP 200 (an agent-retried transport failure no longer fails it), and uses one budget for the request and response waits. Co-authored-by: Isaac <no-reply@databricks.com>
…tests Both ran one long live session per config version, so one slow subagent or model call failed every assertion chained after it. - CUJ 4: test_smart_router_config_version becomes a per-agent preset routing test and a Claude status-band/hint/toggle test that needs no delegated work. - Skill toggles: each agent's test becomes at-launch, off and back-on tests, each in a fresh session; back-on keeps the off->on sequence (and Claude's launch-recipe comparison) that it proves. All existing assertions and config versions are kept. Coverage READMEs are updated for the new test names. Co-authored-by: Isaac <no-reply@databricks.com>
…r one subagent
wait_for_background_tasks submitted /tasks once and waited 180s for the view
to report completion. In CI (17 of 43 flaky CUJ 4 jobs) that never happened
although the work had finished:
- Claude swallowed a /tasks typed mid-turn, or closed the view itself, and
the wait sat at the idle prompt.
- The view opened as a running snapshot and stayed frozen ("Background task
update waiting while this panel is open"); its header names the subagent
type, not "agent", so the detail regex never matched it.
The wait now reopens the view through the visible UI: /tasks again when no
view is shown at an idle prompt, and Esc then /tasks when the view has not
reported completion for 10s. Success still requires Claude's own task view
to report that nothing is running, within the same bounded deadline.
CUJ 4's "exactly one completed child" failures were real second subagents:
the parent sometimes started a "relay" subagent to message the first one for
its result. The delegation prompt now asks for that one subagent only; the
assertion is unchanged.
Co-authored-by: Isaac <no-reply@databricks.com>
lilly-luo
force-pushed
the
ci-flake-hardening
branch
from
October 28, 2026 14:44
ae94b3f to
dd012c7
Compare
rohita5l
force-pushed
the
ci-flake-hardening
branch
from
October 28, 2026 15:33
dd012c7 to
4e50fd6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1245 (base
ci-flake-hardening); this diff contains only the follow-up commit. Retarget tomainonce #1245 merges.What
Fixes the two remaining CUJ 4 flake causes from the CI analysis in #1245. The diagnosis replays the raw PTY transcripts from CI's CUJ evidence artifacts (runs 38801294447 and 39696214480, with passing siblings) through pyte, plus the final screens from 19 failing jobs.
wait_for_background_tasks(tests/integration/utils/terminal.py). "TUI did not show Claude's task view reporting no running tasks within 180s" was 17 of 43 flaky CUJ 4 jobs, and the same wait ends the Claude skill-toggle tests. The work had finished every time. The check was never flickering: in the failing transcript it is False continuously until the final repaint. There are two real causes:/taskstyped mid-turn was swallowed or because the view closed itself. The old wait never reopened it./tasksopened mid-turn and froze on a running state ("Background task update waiting while this panel is open"). The running header names the subagent type (ug-smart-router:… › Read file contents, footer… · x to stop · …), so the oldagent ›regex never even recognized it.The wait is now a bounded loop:
/taskswhen no task view is shown at an idle prompt, at most every 3 s./tasks, when the view hasn't reported completion for 10 s.<name> › <title>header and stays within one line.CUJ 4 "exactly one completed child" (about 9 failures with
assert 2 == 1). These were genuine second subagents, not one subagent counted twice: every failing job has two distinctagent-<id>.jsonltranscripts. The parent sometimes started a "relay" subagent just to message the first one for its result, e.g.Agent(subagent_type="fork", description="Get subagent result", …)in job 119554254666.assert len(completed_children) == 1and all routing assertions are unchanged.Both changes stay within
tests/AGENTS.md. Reopening/tasksis real keystrokes through the visible UI, and the prompt change is scenario input. There are no mocks, skips, retries of the task, or weaker assertions.Not covered / risks
<name> › <title>header on screen. A scrollback line containing " › " could make it misread the panel. That can only cause a missed completion, meaning another reopen, never a false success.Docs
tests/e2e_cuj/README.md: describes the reopen behaviour and the one-subagent prompt.Tests
tests/test_e2e_cuj_helpers.py:/tasksfollowed by completion; a stale running view that gets reopened; both together; and a never-completing case that must still raise at the deadline without sending/exit.tests/test_integration_contract.py,tests/test_e2e_cuj_helpers.py,tests/test_cuj_evidence.pyandtests/test_tui_request_recorder.py: 339 passed.tests/e2e_cujcollects 111 tests;tests/integrationcollects 151 of 163. ruff check and format are clean.This pull request and its description were written by Isaac.