Repository navigation
test(daemon): drive tracedecay journey tests by signals, not wall bounds - #4174
Conversation
source_body reads with the Engine requirement and never waits on the held catalog, so its 1 s caller deadline only measured CPU share. The admission test awaited admission under a 10 s timeout; it now awaits admission and then the convergence gate, so an admission that waited on convergence hangs instead of racing a wall clock.
settled_session_ingest retried only application.runtime.mounting. Under load the reopened composition answered temporal_refresh_unavailable, which is retryable after 250 ms, and the helper returned it as settled. It now retries any problem marked retryable and drops its 30 s wall, so a server that never settles fails at the harness timeout.
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if answer["structuredContent"]["problem"]["retryable"] != true { | ||
| return answer; | ||
| } | ||
| tokio::time::sleep(std::time::Duration::from_millis(250)).await; |
There was a problem hiding this comment.
🟡 Permanent ingest refusals stall journey tests
If the refresh owner stays unavailable, settled_server_session_ingest retries its retryable refusal without a bound. The session journeys run until Bazel kills the whole test target instead of reporting the failed ingest.
Learn more
The session journeys use this helper to wait for a mounted project to finish ingesting. A missing or permanently failed refresh owner can return temporal_refresh_unavailable with retryable: true on every call through join_hook_ingest_refresh. Removing the previous 30-second bound leaves no way for the helper to identify that failure locally; the Bazel test target is size = "large" and times out as a whole. The repository rules require tests to stay falsifiable and forbid weakening assertions or raising timeouts.
Example: If the project's refresh owner remains unavailable, each ingest answers temporal_refresh_unavailable; the helper sleeps 250 ms and repeats until Bazel terminates the suite, rather than failing this journey with its last refusal.
Recommended fix: Keep a bounded, diagnostic failure path around the retry loop while allowing transient retryable refusals. Report the last typed problem when the bound is exhausted.
Was this helpful? React with 👍 or 👎 to provide feedback.
| let database = registry | ||
| .project_sessions(project_id, [project_root]) | ||
| .await | ||
| .expect("registered project sessions"); | ||
| convergence_gate.wait_until_blocked().await; |
There was a problem hiding this comment.
🟡 Admission regressions hang the test target
If project_sessions waits for convergence, its await cannot finish while convergence_gate remains held. Removing the deadline makes Bazel time out the entire test target instead of failing this admission check.
Learn more
This journey checks that foreground admission does not wait for background schema convergence. The convergence worker waits inside block until this test calls release() after its assertions. If a regression makes admission await that worker, the unbounded project_sessions call never returns and cannot reach release(). Removing the old timeout weakens this failure check despite the repository's instruction to keep tests falsifiable and not weaken assertions.
Example: If admission starts awaiting convergence, convergence holds at the gate, and the test holds at project_sessions; neither reaches release(). Bazel eventually aborts every test in the target instead of naming this failed admission.
Recommended fix: Retain a sufficiently measured finite admission guard that fails with a targeted diagnostic, while using the gate to assert that admission does not require convergence to finish.
Was this helpful? React with 👍 or 👎 to provide feedback.
Fixes #4149
This is the
crates/tracedecayhalf of #4149. #4167 is the daemon-service half. Three//crates/tracedecay:unit_testjourneys raced wall clocks under the #4149 load recipe. Each now waits on the signal production gives it. The residual tests are recorded on #4149.Fixes
source_body_serves_while_the_released_catalog_cannot_rewarmCodeGraphReadinessRequirement::Engineand never waits on the held catalogreason_code=tool_dispatch_deadline_exceeded ... tool 'tracedecay_source_body' exceeded its absolute deadline before commit(3 of 20)CodeGraphReadinessRequirement::Catalog: fails in 11.4 s,tool faileddaemon_admission_remains_ready_while_lcm_indexes_converge_in_backgroundstore_runtime_testsstalls (issue item 3)Complete: TIMEOUTshut_down_composition_releases_its_session_stores_for_an_immediate_reopensettled_session_ingestretried onlyapplication.runtime.mounting. The reopened composition answeredtemporal_refresh_unavailable, which isretryable: true, retry_after_millis: 250, and the helper returned it as settledthe reopened composition must remount the session stores: ... Problem: temporal_refresh_unavailable ... Retryable: true(7 of 20)project_server_capacity_journey_testaccepted_for_replayand on WAL bytesThe helper now retries any problem with
structuredContent.problem.retryable == true. Its 30 s wall bound is removed, so a server that never settles fails at the harness timeout.Admission time under load
The earlier lane saw
project_sessionsadmission take 135 ms alone and 1.2 to 7 s under load. Phase probes inmount_project_session_storeshow where the time goes. Opening the store takes about 10 ms.attach_registeredtakes the rest: about 110 ms of wall time, which matches 11 to 13 ticks of process CPU time. Admission is CPU-bound, with no sleep or lock wait, so it scales with CPU starvation. No production change was made.Verification
//crates/tracedecay:unit_testfiltered toproject_server_capacity_journey_test: 30/30 under the recipe.//crates/tracedecay:unit_testfiltered tosource_body_readiness_journey_testandstore_runtime_tests: 30/30 under the recipe.bazel build --config=clippy //crates/tracedecay:unit_test: clean.