[rohita5l]

What

Hardens the flakiest CI tests found in a 60-day analysis of ci.yml (5,320 runs; job-level detail for the last 14 days). Over the last 14 days, 44% of completed runs had a failing blocking job, and 263 of 385 manual reruns over the 60 days ended green. This PR targets causes that are proven flakes (the job failed, then passed on a rerun of the same commit). It changes tests and CI only; no product code.

1. Recorder teardown corrupted other connections (tests/integration/utils/request_recorder.py). Handler threads are daemons, and __exit__ closed the httpx client while a forward could still be streaming. That frees the socket descriptor while OpenSSL keeps using the old number, so a later connection in the same process had its bytes stolen. This produced Response ended prematurely, SSL: RECORD_LAYER_FAILURE, incomplete chunked read and Bad file descriptor. All 22 flaky CUJ 2 Claude failures ran right after a passing CUJ 2 Codex class. Teardown now waits for in-flight forwards before closing the client.
   • Also: a transport failure is recorded as the 502 the agent saw, so a waiter fails immediately with the cause instead of timing out after 240 s. Pooled upstream connections are no longer reused.
   • CUJ 2 now asserts the first marker-bearing inference request that the gateway served with HTTP 200, and uses one budget for the request and response waits.
2. Per-model e2e launch tests (tests/test_e2e.py, tests/conftest.py). These looped over the live catalog and failed as a whole when any one model misbehaved; the same test failed across 19–29 unrelated PRs. They are now parametrized, one case per model, from a cached catalog fetch shared with e2e_state.
   • Without UCODE_TEST_WORKSPACE nothing touches the network.
   • A catalog fetch error is one failing case, not a skip.
3. Automatic rerun for live timeouts in the e2e shards only (ci.yml). This adds pytest-rerunfailures and reruns once, only for TimeoutExpired, timed out after Ns, TIMEOUT after, TEMPORARILY_UNAVAILABLE and REQUEST_LIMIT_EXCEEDED. -rfER lists reruns.
   • Integration and CUJ suites keep failing on the first attempt, per tests/AGENTS.md rule 6.
4. Tracing tests poll longer (tests/integration/utils/sql.py). Each statement now gets 300 s instead of 180 s. The trace row is polled every 15 s for up to 300 s, instead of one query after a 30 s sleep. The tests still assert the row exists.
5. Smaller smart-routing journeys. One slow live call no longer fails everything chained after it, and each failure names the step.
   • CUJ 4 test_smart_router_config_version becomes a per-agent preset routing test and a Claude status-band/toggle test.
   • Each agent's skill-toggle test becomes at-launch, off and back-on tests in fresh sessions. back-on keeps the off→on sequence it proves.
   • All assertions and config versions are kept.

Not covered

• The most common CUJ 4 failure (task view reporting no running tasks within 180s, 17 of 43 flaky jobs) is a bug in wait_for_background_tasks in tests/integration/utils/terminal.py. In 14 of 19 replayed screens the /tasks panel had already closed with all work finished. In the other 5, the panel showed ✔ Completed, but the check never stayed true for the required 0.3 s. The split isolates this failure but does not fix it; that will be a follow-up.
• The CUJ 4 "exactly one completed child" assertion (the model sometimes spawns two) is unchanged.
• The splits improve diagnosis but do not reduce the red-run rate. Expected overall effect is roughly 2–3 percentage points, plus fewer manual reruns.
• The launch preflight's misleading "Unity Gateway is not enabled" on transient HTTP 499 is fixed separately in fix-model-services-probe-retry.

Docs

• tests/README.md, tests/integration/README.md and tests/e2e_cuj/README.md: renamed the CUJ 4 and skill-toggle tests and updated their coverage descriptions.

Tests

• New unit tests in tests/test_tui_request_recorder.py cover recorder teardown. test_teardown_leaves_client_open_for_forward_that_outlives_drain fails on the old code and passes now.
• Contract plus recorder/CUJ helper tests pass (333). Integration collects 151 of 163 (12 deselected); e2e_cuj collects 111.
• The full unit suite has the same 25 failures as clean origin/main on the dev machine (environment-related PTY, smart-router and installer tests; compared test by test).
• ruff check and format are clean. ty is clean on src/.
• Not run live. The integration, CUJ and e2e changes need this PR's CI to confirm them.

This pull request and its description were written by Isaac.

[lilly-luo]

The recorder teardown fix is well diagnosed, and test_teardown_leaves_client_open_for_forward_that_outlives_drain covers it properly. The per-model parametrization and the scoped e2e reruns look right.

Blocking: rebase onto main. The branch is 42 commits behind and conflicts. Since it was cut, #1222, #1212 and #1270 changed test_ug_smart_routing_hooks.py:

• They added MANAGED_DEFAULT cases.
• They added a smart_routing= kwarg to use_smart_routing_fixture. Legacy env-flag cases pass smart_routing=False because "managed smart routing would override the legacy env vars".
• #1270 quarantined the Claude toggle journeys that this PR splits.

The new _select_smart_router_version has neither the MANAGED_DEFAULT cases nor the kwarg, so a plain conflict resolution would silently drop coverage. See the inline comment for a version that keeps both.

CI is also stale. The last run (2026-10-22) is red on e2e, all integration tests, and Codex · Windows full journeys (10 failed). The Windows unit failure (test_catalog_browsing small-terminal render) looks unrelated. These live changes still need a green run after the rebase.

Non-blocking nits are inline.

[lilly-luo]

Re-review after rebasing onto main (dd012c7e, force-pushed). The branch now merges cleanly. Here is how I resolved the conflicts, so you can check them:

Main's changes carried into the split tests

• test_ug_smart_routing_hooks.py:
  • (True, MANAGED_DEFAULT) is in SMART_ROUTER_VERSIONS.
  • _select_smart_router_version skips the selector env var for MANAGED_DEFAULT and passes smart_routing=config_version is not None, as main does (#1212).
  • The Agent/Task tool-description assertions from #1290 now live in _assert_claude_calculation_requests, through a new orchestration_enabled kwarg (condition orchestration_enabled and routed).
  • The #1270 quarantine (@pytest.mark.soft_fail) is on all three Claude split tests: 18 cases. The Codex splits stay blocking.
• test_cuj4_smart_routing.py: main's route-options checks are in the first-prompt-routed block. Child routes are matched by decision task_name (#1258) instead of child_task.filename.

Fixes for code that only broke after the rebase

• scripts/run_integration.py: WINDOWS_UNSUPPORTED_NODES deselected the old test_smart_router_skill_toggles_codex_subagent_routing by node id. It now lists the three Codex splits; without that, they would have started running on Windows.
• tests/test_integration_contract.py: the #1270 soft-fail contract checks now use a CLAUDE_TOGGLE_TESTS tuple of the split names. The Codex retained test is now test_smart_router_skill_turns_codex_subagent_routing_back_on.
• READMEs: merged both sides. The integration README soft-fail section now names the three split tests.

Local checks

• ruff check and format are clean. Contract and runner tests: 102 passed. Recorder tests pass.
• Collection: 46 smart-routing integration cases (18 soft_fail), 109 e2e_cuj, and 36 e2e with no workspace set.
• The full unit suite has 7 failures (test_e2e_user_agent and test_smart_router). The same 7 also fail on origin/main on my machine.
• No live runs. CI is running now.

Still open from my first review (the code is unchanged):

• test_cuj2_mps_mcp.py:227: the waits use separate 240 s timeouts, not one shared budget, and an abandoned later candidate masks the real 502. This is the one I'd fix before merging.
• Nits: the recorder's mid-stream 502 comment, the shallow copy in e2e_state, and the collection-time catalog fetch.

[lilly-luo]

Second re-review at dd012c7e, this time with an independent read that doesn't rely on my conflict resolution.

CI: 52 checks passed. The only blocking failure is CUJ 4 test_smart_router_preset_routes_first_prompt_then_explicit_native_subagent[subagent_orch_v1-claude] (assert 2 == 1 at test_cuj4_smart_routing.py:445). Claude spawned two subagents for the same delegated prompt, and both hit routes:select with identical prompts (seq 104 and 109). That's the known extra-subagent flake (see #1230 and this PR's "Not covered" section), not the rebase. All integration tests is red only because of it. Codex · Windows full journeys pass, which confirms the updated WINDOWS_UNSUPPORTED_NODES deselect.

Should fix: the split dropped the routed → off → unrouted scenario. On main, each toggle journey routed a child, turned routing off, and then checked that the next child was not routed. In the split, every off toggle happens either before anything was routed (*_off, and Codex _back_on) or right before an on toggle (Claude _back_on). No test now catches a hook or mod that caches routing state after the first routed spawn, so /smart-router off stops working mid-session. See the inline suggestion.

Minor

• The tracing tests now poll count and then model_count, each for up to about 300 s plus 60 s more for a slow statement. Both queries use time > current_timestamp() - INTERVAL 10 MINUTES (test_ug_claude_tracing.py:56 and test_ug_codex_tracing.py:56). A slow agent run plus a late export can push the span outside that window, which reports a false "no rows". Consider widening the window or anchoring it to the test's start time.
• The earlier comments on CUJ 2 _served_marker_request, the shallow copy, and the recorder's mid-stream 502 still apply. Two more are inline.

Checked with no findings: the CUJ 4 split keeps every assertion and preset, the band and hint-row checks, the env and machine-wide restore, and main's #1258 and #1290 changes. The hooks file keeps #1270 soft_fail on all three Claude tests, and its ids and markers match main. No stale references to the renamed tests remain in scripts, contract tests, workflows or READMEs. The --only-rerun patterns match both the pytest.fail messages and a raw TimeoutExpired.