Repository navigation
test: tracedecay lib tests that fail only under full-suite load #2650
Description
What happens
These tracedecay lib tests fail only under load. Each one passes when run alone. I found them while looping the lib suite for #2513 and did not investigate them.
How it was reproduced
- Binary: master d842ec3,
cargo test -p tracedecay --features test-transport --lib --no-run. - Load: the whole lib suite, 772 tests, run with
--test-threads=64. Two to four copies ran at once on a 96-core host that was shared with other agents' builds. - 50 runs in total.
| Test | Failed runs (of 50) |
|---|---|
mcp::server::routing::tests::many_slow_initialize_roots_share_one_discovery_budget |
14 |
daemon::production_harness::advisory_cycle_language_journey_test::first_advisory_cycle_after_a_reopen_answers_without_a_retry |
13 |
daemon::tests::replay::client_identity_startup_replays_retained_profile_receipts |
10 |
mcp::server::host_admission_tests::owned_project_replay_worker_continues_past_one_bounded_batch |
7 |
daemon::production_harness::lcm_preserved_profile_journey_test::preserved_profile_lcm_discovery_converges_without_blocking_retrieval |
7 |
daemon::tests::lifecycle::one_shot_tool_call_allows_long_response_while_daemon_stays_live |
3 |
daemon::production_harness::generation_retention_test::mounted_code_generation_retention_continues_capped_segment_reclamation |
3 |
session_temporal_benchmark::tests::fresh_benchmark_db_provisions_key_for_rank_and_hydration |
2 |
session_temporal_benchmark::tests::fixture_refresh_persists_progress_before_measurement |
2 |
daemon::tests::route_discovery::a_deferred_discovery_converges_on_the_next_request |
1 |
corrupt_graph_restart_rebuilds_from_the_sealed_segments_without_decoding also failed in 10 of the 50 runs. #2633 already tracks it, so it is not listed here.
Known cause for one test
many_slow_initialize_roots_share_one_discovery_budget compares wall-clock time: it asserts elapsed < 3s across three 1.5s injected probes (crates/tracedecay/src/mcp/server/routing.rs). Under CPU contention the budget check fails before the deadline logic it is meant to test gets a chance to run.
I did not capture the panic text for the other tests. The run above reproduces them.
Repro
B=$(cargo test -p tracedecay --features test-transport --lib --no-run 2>&1 | sed -n 's/.*(\(target[^)]*\)).*/\1/p')
cd crates/tracedecay
for i in 1 2 3; do "../../$B" --test-threads=64 & done; wait
Activity
ScriptedAlchemy commentedon Sep 29, 2026
Claimed by fleet slug load-flake-sweep (looping lib, mcp_suite, session_suite, daemon_suite, application_suite under fleet load; fixes grouped by area).
ScriptedAlchemy commentedon Sep 29, 2026
Also seen under the same load: two full lib-suite runs hung for more than an hour (one on d842ec3 plus test-only changes, one on 8d4fd5e plus test-only changes). In both, the single remaining libtest thread was named
daemon::store_r…, so the hang is in one of thedaemon::store_runtime_tests::*tests. The thread name is truncated, so I can't tell which one. Hauler lanes should wrap each loop iteration intimeout.ScriptedAlchemy commentedon Sep 29, 2026
Findings from
load-flake-sweep-harness(the three lib-suite flakes delegated from this sweep). Wholetracedecay --libsuite, default parallelism, debug build, 96 cores.advisory_cycle_language_journey_test::first_advisory_cycle_after_a_reopen_answers_without_a_retrylatest_feedback_generation_for_scopeisNone, so the placeholder advisory owner'smount()answeredAnswersand the request gotfeedback.advisory-cycle.unavailableimmediately. Holding that proof with the existing retained-recovery gate reproduced it 10/10 alone (refusal in 1.6 ms while the harness already reported ready).Mountingwhile the scope holds a retained text owner, so the request waits for the deferred mount within its own deadline.03ba683fbccode_index_runtime_graph_activation_tests::corrupt_graph_restart_rebuilds_from_the_sealed_segments_without_decodingdatabase file is locked by another process: …/tracedecay.grafeo, and the scheduler took its documented fallback decode (sealed_decode_count1).shutdown_terminal_tasks()before the reconciliation shutdown.5aa5fd06b8(closes #2633)lcm_preserved_profile_journey_test::preserved_profile_lcm_discovery_converges_without_blocking_retrievalstale/partial). Instrumented alone (~25-40 s total): one project refresh worker runs everything serially. Its first history pass takes ~12 s (Codex ~1.5 s, Claude observation admission ~7 s over ~120 submits, Cursor ~2 s), returnsRetryable{ingest_pass_backpressured}, then about ten projection-only passes run, then a second history pass installs the 40-span Git evidence (~4.7 s), and only then does summary convergence getAdmitted(Full). In a failing loaded run (load ~200), the Codex-to-Git-evidence phase took 77 s against 9-19 s alone. #843 names no convergence deadline, so the 2-minuteCONVERGENCE_WAITis the test's own.SessionTemporalRefreshWakeState::is_idle()is not one: it also reports idle while a pass waits on the historical-work permit (worker.rs ~333-339). The other option is to make the first history pass cheaper (Claude admission is one submit per transcript write group), measured with Hotpath. Both are larger than this lane.Other failures seen in my loops (the parent's, not mine):
concurrent_same_identity_worktrees_keep_exact_server_and_scheduler_bindings(every run),daemon_http_shutdown_releases_loopback_listener,client_identity_startup_replays_retained_profile_receipts,session_temporal_benchmark::tests::*,remote_tls_listener_bounds_connections_and_expires_incomplete_headers,rmcp_peer_disconnect_mid_delivery_settles_dropped_rather_than_unknown,owned_project_replay_worker_continues_past_one_bounded_batch,many_slow_initialize_roots_share_one_discovery_budget,maintenance_reclaims_a_removed_linked_worktree_and_the_text_artifact_only_it_named.ScriptedAlchemy commentedon Sep 29, 2026
Status from fleet slug load-flake-sweep (whole lib suite at default parallelism on the shared host, load 57-233; baseline master 35fe79e, 20 runs; post-fix 20 runs, load 72-161).
Merged:
concurrent_same_identity_worktrees_.... All 0/20 post-fix.Still open here:
store_runtime_tests::schema_convergence_is_reported_and_completes_after_admission(seed insert hitsdatabase is lockedwhile the fixture's dropped runtime closes, 1/20) andruntime_identity::maintenance_reclaims_a_removed_linked_worktree_...(1/20), both assigned to sibling lanes. The product race behind the liveness fake-daemon failures is #2654.ScriptedAlchemy commentedon Oct 1, 2026
claimed by fleet slug close-all-issues/daemon-flakes