flaky: remaining load-sensitive daemon tests #4149

Activity

1. ScriptedAlchemy commented on Oct 17, 2026

   

   ScriptedAlchemy

   on Oct 17, 2026

   OwnerAuthor

   More actions

   Status of #4149 after #4167 (daemon-service) and #4174 (crates/tracedecay). The two PRs fix the listed tests. The tests below were left unchanged, with the reason and evidence for each.

   Not reproduced, left unchanged

   • capacity_retirement_blocked_by_a_foreign_store_lease_is_typed_and_not_replayed: 1 failure in about 150 runs in the earlier lane. It passed in the project_server_capacity_journey_test round of test(daemon): drive tracedecay journey tests by signals, not wall bounds #4174 (30/30).
   • linked_roots_alias_one_store_producer_until_the_last_alias_shuts_down and store_shutdown_checkpoint_test::capacity_retirement_completes_while_a_maintenance_tick_runs: both passed 30/30 in a full //crates/tracedecay:daemon_suite round under the recipe (host load average 300 to 900). Earlier evidence shows three linked-roots failure modes, all at extreme host load:
     • left: 4 (or 1), right: 5 persisted events. The producer drops a batch whose write exceeds the fixed 2 s ObservabilityProducerDeadlinesV1::persistence.
     • restarted producer: ... delivery_settlement_recorder_already_running. expire_all returned while a timed-out retirement drain still held the store's delivery spool lock.
     • SQLite transaction lease expired during schema initialization.
   • The maintenance-tick failure in the 2026-10-16 gates was store runtime still held at shutdown (sessions.db, ClientLeases { count: 2 }). It followed work owner-observation recovery shutdown deadline and observability_persistence_deadline lost work. Store close spent exactly its fixed 3 s hold drain (SHUTDOWN_HOLD_DRAIN_LIMIT, from flaky: graceful stop leaves WAL when store leases are still held at close #4138) with about 9 s of the store-close phase budget left. A candidate production fix is to pass the store-close phase deadline into close_idle_for_shutdown in place of the fixed 3 s. It is not made here, because no current run reproduces the failure.

   Real-time deadline tests, left unchanged

   shutdown::orchestration::tests::a_stuck_background_owner_cannot_starve_project_servers_or_store_close and a_project_server_drain_that_ignores_its_deadline_hits_the_backstop failed once each in one 30-run round. Both reported left: ["shutdown_coordinator"]: the coordinator deadline (2 s or 4 s) fired before the 50 to 300 ms phase timers. native_runtime has no clock seam. Injecting one would mean threading a timer through coordinate_daemon_shutdown, run_daemon_shutdown and the owner joins, which is not a small local change.

   Outside this issue: daemon_suite under the recipe

   The same full daemon_suite round failed 22 of 30 runs, all in tests not listed here:

   • sealed_generation_crash_test::sigkill_during_the_seal_keeps_the_prior_generation_published: 19/30, index never became ready.
   • git_watch_test::one_reconciliation_covers_a_fifty_commit_frontier: 17/30, timed out waiting for "f49".
   • store_shutdown_checkpoint_test::stop_after_independent_owners_releases_every_project_store: 10/30, project open never reached independent_owners_registered.
   • advanced_workflow_journey_test::mounted_fan_out_recovers_then_synthesizes_and_hands_off: 8/30, timed out waiting for project application mount.
   • graceful_stop_truncates_profile_and_project_store_wals: 4/30, global.db WAL 3048832 bytes.
   • stop_during_project_open_truncates_project_store_wals: 3/30.

   Most of these are readiness waits bounded by wall clocks while indexing is CPU-starved. They need their own issue.

   Admission time

   project_sessions admission is CPU-bound. attach_registered is about 110 ms of wall time and 11 to 13 CPU ticks unloaded. There is no hidden wait, so 1.2 to 7 s under load is starvation.
2. ScriptedAlchemy

   closed this as completedin #4174on Oct 17, 2026