Skip to content

test(daemon): drive tracedecay journey tests by signals, not wall bounds #4174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ScriptedAlchemy merged 4 commits into from
Oct 17, 2026
Merged

test(daemon): drive tracedecay journey tests by signals, not wall bounds #4174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dbaaa05
test(daemon): drop wall bounds from source-body and admission tests
ScriptedAlchemy Oct 17, 2026
d0ca91d
Merge remote-tracking branch 'origin/master' into fix/load-flakes-2b
ScriptedAlchemy Oct 17, 2026
e8d63f9
Merge remote-tracking branch 'origin/master' into fix/load-flakes-2b
ScriptedAlchemy Oct 17, 2026
c49608d
test(daemon): retry every retryable hook refusal, not only mounting
ScriptedAlchemy Oct 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 13 additions & 28 deletions crates/tracedecay/src/daemon/production_harness/project_server_capacity_journey_test.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,18 +498,17 @@ async fn twelve_project_journey_retires_idle_owners_without_empty_graphs() {
harness.shutdown().await;
}

/// Ingest through the session stores until the full server has mounted them,
/// returning the first settled answer, or `None` once the bound expires.
/// Ingest through the session stores, retrying each retryable refusal, and
/// return the first settled answer. A server that never settles fails at the
/// harness timeout.
async fn settled_session_ingest(
harness: &ProductionProjectCompositionHarnessV1,
project: &Path,
) -> Option<serde_json::Value> {
) -> serde_json::Value {
settled_server_session_ingest(&harness.server(project).expect("mounted project")).await
}

async fn settled_server_session_ingest(
server: &crate::mcp::McpServer,
) -> Option<serde_json::Value> {
async fn settled_server_session_ingest(server: &crate::mcp::McpServer) -> serde_json::Value {
let request = serde_json::from_value::<JsonRpcRequest>(serde_json::json!({
"jsonrpc": "2.0",
"id": 1,
Expand All @@ -524,7 +523,6 @@ async fn settled_server_session_ingest(
},
}))
.expect("hook runtime request");
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
loop {
let response = server
.handle_request(&request)
Expand All @@ -535,11 +533,10 @@ async fn settled_server_session_ingest(
"hook runtime failed: {response:?}"
);
let answer = response.result.expect("hook runtime result");
if !answer.to_string().contains("application.runtime.mounting") {
return Some(answer);
}
if std::time::Instant::now() >= deadline {
return None;
// Mounting is one retryable refusal; a refresh that missed its
// publish deadline under load is another.
if answer["structuredContent"]["problem"]["retryable"] != true {
return answer;
}
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
Comment on lines +538 to 541

@devin-ai-integration devin-ai-integration Bot Oct 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Permanent ingest refusals stall journey tests

If the refresh owner stays unavailable, settled_server_session_ingest retries its retryable refusal without a bound. The session journeys run until Bazel kills the whole test target instead of reporting the failed ingest.

Learn more

The session journeys use this helper to wait for a mounted project to finish ingesting. A missing or permanently failed refresh owner can return temporal_refresh_unavailable with retryable: true on every call through join_hook_ingest_refresh. Removing the previous 30-second bound leaves no way for the helper to identify that failure locally; the Bazel test target is size = "large" and times out as a whole. The repository rules require tests to stay falsifiable and forbid weakening assertions or raising timeouts.

Example: If the project's refresh owner remains unavailable, each ingest answers temporal_refresh_unavailable; the helper sleeps 250 ms and repeats until Bazel terminates the suite, rather than failing this journey with its last refusal.

Recommended fix: Keep a bounded, diagnostic failure path around the retry loop while allowing transient retryable refusals. Report the last typed problem when the bound is exhausted.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

}
Expand All @@ -559,11 +556,7 @@ async fn shut_down_composition_releases_its_session_stores_for_an_immediate_reop
let harness = ProductionProjectCompositionHarnessV1::open(isolation.path(), [project.clone()])
.await
.expect("first production composition");
let first = settled_session_ingest(&harness, &project).await;
assert!(
first.is_some(),
"the first composition mounts its session stores"
);
settled_session_ingest(&harness, &project).await;
let project_data_root = harness
.project_data_root(&project)
.await
Expand Down Expand Up @@ -600,11 +593,8 @@ async fn shut_down_composition_releases_its_session_stores_for_an_immediate_reop
.await
.expect("reopened production composition");
let reopened = settled_session_ingest(&harness, &project).await;
assert_eq!(
reopened
.as_ref()
.map(|answer| answer.to_string().contains("accepted_for_replay")),
Some(true),
assert!(
reopened.to_string().contains("accepted_for_replay"),
"the reopened composition must remount the session stores: {reopened:?}"
);
harness.shutdown().await;
Expand Down Expand Up @@ -651,12 +641,7 @@ async fn capacity_retirement_releases_session_stores_its_owner_used_on_the_first
.unwrap_or_else(|error| {
panic!("project {ordinal} must open over released capacity: {error}")
});
assert!(
settled_server_session_ingest(&opened.server)
.await
.is_some(),
"project {ordinal} mounts its session stores"
);
settled_server_session_ingest(&opened.server).await;
drop(opened);
opened_projects += 1;
}
Expand Down
8 changes: 4 additions & 4 deletions crates/tracedecay/src/daemon/production_harness/source_body_readiness_journey_test.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use tempfile::TempDir;
use tracedecay_code_index::graph_projection::{
CodeGraphCatalogReleaseV1, CodeGraphReadinessRequirement,
};
use tracedecay_domain::{ProjectId, SymbolOccurrenceId, UtcMicros};
use tracedecay_domain::{ProjectId, SymbolOccurrenceId};
use tracedecay_graph_db::NeverCancelled;

use super::ProductionProjectCompositionHarnessV1;
Expand Down Expand Up @@ -82,13 +82,13 @@ async fn source_body_serves_while_the_released_catalog_cannot_rewarm() {
// These point reads prove that the retained engine and occurrence are ready.
let point = reader.symbol_summary(&occurrence, Arc::new(NeverCancelled));
let catalog_pending = store.await_rewarm(Duration::ZERO);
let deadline = UtcMicros(tracedecay_contracts::now_micros().0 + 1_000_000);
// A read that waited on the held catalog would spend its re-warm budget
// and answer unavailable instead of evidence.
let response = harness
.call_tool_with_deadline(
.call_tool(
&project,
"tracedecay_source_body",
serde_json::json!({"node_id": node_id, "format": "json"}),
deadline,
)
.await;
let still_pending = store.await_rewarm(Duration::ZERO);
Expand Down
25 changes: 7 additions & 18 deletions crates/tracedecay/src/daemon/store_runtime_tests.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1176,24 +1176,13 @@ async fn daemon_admission_remains_ready_while_lcm_indexes_converge_in_background
.expect("session runtime registry");
let convergence_gate = registry.block_registered_schema_convergence_for_test();

let admission = registry.project_sessions(project_id, [project_root]);
tokio::pin!(admission);
let convergence_blocked = convergence_gate.wait_until_blocked();
tokio::pin!(convergence_blocked);
let database = tokio::time::timeout(std::time::Duration::from_secs(10), async {
tokio::select! {
result = &mut admission => {
let database = result.expect("registered project sessions");
convergence_blocked.await;
database
}
() = &mut convergence_blocked => {
admission.await.expect("registered project sessions")
}
}
})
.await
.expect("daemon admission and convergence scheduling must not stall");
// The gate holds convergence until the assertions below, so an admission
// that waited on convergence never returns.
let database = registry
.project_sessions(project_id, [project_root])
.await
.expect("registered project sessions");
convergence_gate.wait_until_blocked().await;
Comment on lines +1181 to +1185

@devin-ai-integration devin-ai-integration Bot Oct 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Admission regressions hang the test target

If project_sessions waits for convergence, its await cannot finish while convergence_gate remains held. Removing the deadline makes Bazel time out the entire test target instead of failing this admission check.

Learn more

This journey checks that foreground admission does not wait for background schema convergence. The convergence worker waits inside block until this test calls release() after its assertions. If a regression makes admission await that worker, the unbounded project_sessions call never returns and cannot reach release(). Removing the old timeout weakens this failure check despite the repository's instruction to keep tests falsifiable and not weaken assertions.

Example: If admission starts awaiting convergence, convergence holds at the gate, and the test holds at project_sessions; neither reaches release(). Bazel eventually aborts every test in the target instead of naming this failed admission.

Recommended fix: Retain a sufficiently measured finite admission guard that fails with a targeted diagnostic, while using the gate to assert that admission does not require convergence to finish.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


assert_eq!(
registry.registered_schema_convergence_status(&shard_id),
Expand Down
Loading