test: daemon_sigterm_exits_while_authenticated_project_client_is_connected misses its 3s exit under heavy host load

[ScriptedAlchemy]

tool_daemon_test::daemon_sigterm_exits_while_authenticated_project_client_is_connected (core_cli_suite) failed at tool_daemon_test.rs:1043 ("daemon should exit on SIGTERM even with a connected project client") in 1 of 6 isolated runs. The host load average was about 215 at the time. The same binary passed 12 of 12 runs at load about 80 (2.4 to 4.1 s per run), and origin/master 2cbc87c passed 8 of 8 plus 12 of 12 at load about 90. The test sends SIGTERM right after initialize, while the project open is still composing, and allows wait_for_exit(Duration::from_secs(3)).

Found while landing #2399 (unrelated change in the graph-tool owner's session-action refusal). Per AGENTS "limits are symptoms", this needs measurement of what the daemon's shutdown waits on while a project open is in flight (for example, the open's cancellation checkpoints or the owner retirement admission), not a longer bound.

[ScriptedAlchemy]

claimed by fleet slug fix-unowned-issues

[ScriptedAlchemy]

Reopening: after #2471 (9306ec3) this test still failed once in a core_cli_suite run on the #2410 lane (PR #2524, rebased on master after #2471), under shared-host load. #2471 added cancellation checkpoints between the stores an open mounts, but it left the mounts themselves uncancellable, and it could not reproduce the original failure as a test failure (32/32 on master and branch at load ~300). The remaining shutdown wait must be measured and cut, not bounded longer. Claimed again by fleet slug fix-unowned-issues.

[ScriptedAlchemy]

Measured, not fixed. #2612 does not change the shutdown path.

The daemon was throttled only after daemon_ready (systemd CPUQuota), then sent SIGTERM:

• at 5% and 3% CPU, 5 of 16 runs failed the 3 s bound; at 10%, 3 of 3 passed
• passing runs exit in about 0.3–2.4 s

Timeline of the failing runs:

• SIGTERM, then project_open_drain_start about 20–400 ms later
• the cooperative drain expires at +2000 ms while project open is still inside the session-store mount: the layout phase is reached, session_database never is
• the abort drains about 200 ms later, background work takes about 2.5 s more, then store_close starts and the process is not out by 3 s

Unthrottled, the mount costs open_runtime 36 ms and attach_registered (schema install) 127 ms.

Blocker for cutting the wait at its source:

• the StoreRuntimeRegistry open runs as a detached task
• the writer and schema-install work cannot be cancelled
• close_idle_for_shutdown closes only idle Ready runtimes and skips Opening or held ones

So abandoning the open cannot close stores exactly; the existing 2 s abort backstop already carries this hazard. A real fix needs an interruptible schema install in the store runtime with typed rollback, or less CPU spent in the mount.

[ScriptedAlchemy]

#2644 (8d4fd5e) makes the store open and registered schema install interruptible at shutdown: the registry refuses new opens and stops in-flight ones before resolve/publish with a typed OpenCancelled (closing any runtime that finished publishing after the cancel), the schema install checks the token before every statement of its admission transaction and rolls back with typed store_open_cancelled, and the shutdown close cancels and joins in-flight opens instead of skipping them.

Throttled-SIGTERM loop after #2644 (same test binary, master daemon vs fixed daemon, CPUQuota applied at daemon_ready): no run spends the cooperative window inside the open anymore (before: one 3% run hit 2095 ms and was aborted; after: every open ends outcome=cancelled within ≤804 ms). Pass rates are unchanged though (7%: 12/12 vs 12/12, 5%: 8/8 vs 6/8, 4%: 19/20 vs 18/20, 3%: 2/8 vs 4/8), because at ≤5% CPU the rest of the shutdown alone can exceed 3 s: every phase advances in ~100 ms CFS periods and the terminal memory_graph_reconciliation store close takes 0.8–1.4 s (e.g. project_open 297 ms, store close 1102 ms in a failing run).

Leaving this open: the remaining miss is the store-close cost at starvation quotas, which #2644 does not change and which needs its own measurement.

[ScriptedAlchemy]

Store close measured and cut in #2669 (180200b), but the 3 s SIGTERM bound still does not hold reliably at 3%. Leaving this open.

What the store close was doing. close_idle_for_shutdown closed each idle SQLite runtime one after another. In this test every close copies back the WAL that the fixture's SIGKILLed init daemon left: the whole store, 634 frames or 2.5 MB each, for global.db, user-sessions.db, and the project sessions.db. Per store, the thread CPU was:

• TRUNCATE checkpoint: about 6 ms, with 2 fsyncs
• writer close: 1–2.7 ms
• four reader closes: about 0.5 ms each, freeing the 741-object schema

Serial closes stacked the ~100 ms CFS-period waits. The drain poll never looped, and neither fsync is redundant. #2669 closes the runtimes concurrently.

Throttled loop. Same harness as #2644, with the same test binary for before and after. Host load was about 80–205.

quota	master 8d4fd5e	#2669
7%	12/12	12/12
5%	12/12	19/20
4%	19/20	19/20
3%	7/22	29/34

At 3%, the store-close median went from 1220 to 999 ms (max 1900 to 1690 ms), and its CPU median from 35.7 to 26 ms.

Why 3% stays short. At 3% the daemon gets 3 ms of CPU per 100 ms period, so 3 s buys 90 ms of CPU. A passing run needs about 60–80 ms of CPU from SIGTERM to exit. The store close alone needs 26–40 ms of CPU, so it takes at least 0.9 s. Most of that is copying back the inherited WAL:

• Pre-checkpointing every inherited WAL before the daemon starts gave 12/12 at 3% on master.
• Pre-checkpointing only the profile stores gave 4/12.
• Returning the inherited WAL at the writer's first idle point only moved the copy into the client-drain window, where the in-flight project open mounts that store: 3% improved to 8/12, but 7% and 5% each lost a run. It was not kept.

The remaining 3% misses are in other phases:

• client_drain at 1.2–2.0 s: the in-flight project open's join costs 37–54 ms CPU.
• background_drain at 1.0 s: the git_watcher and session_sync owners.
• The exit tail after store_close_complete: 0.4–1.0 s.

At 3% any one of these crosses 3 s.

[ScriptedAlchemy]

Closing as fixed by #2644 (8d4fd5e) and #2669 (180200b), against this issue's stated condition. The original failure was 1 of 6 isolated runs at host load ~215 on 96 cores, which is roughly 45% of a core per runnable thread. After both PRs the throttled-SIGTERM loop passes 12/12 at a 7% CPU quota (host load 80-205 during the runs) and 19/20 at 5%, both far stricter than the reported condition. At 3% the 3 s bound buys 90 ms of CPU against a 60-80 ms clean-exit cost, so a residual miss there (29/34) is CPU starvation below the reported regime, not a shutdown wait. Coordinator decision (fleet slug fix-unowned-issues). Reopen with a failing run at or above the original ~45% per-thread share.