Skip to content

fix: wait for daemon exit during server shutdown - #266

Merged
yingdi-shan merged 1 commit into
kvcache-ai:mainfrom
zlzGithub-0801:fix/wait-for-ublk-daemon-exit
Sep 9, 2026
Merged

fix: wait for daemon exit during server shutdown#266
yingdi-shan merged 1 commit into
kvcache-ai:mainfrom
zlzGithub-0801:fix/wait-for-ublk-daemon-exit

Conversation

@zlzGithub-0801

Copy link
Copy Markdown
Collaborator

What

Wait for the ublk daemon process to fully exit before completing server shutdown.

Why

The daemon may acknowledge shutdown before finishing cleanup. If the server restarts during this window, the old daemon can remove the new daemon's Unix socket, causing subsequent requests to fail.

Related issue

Closes #158

Scope and non-goals

This change only coordinates shutdown between the server and its managed ublk daemon. It does not add daemon auto-restart or change systemd configuration.

Design and behavior changes

The daemon watchdog publishes a retained exit notification after child.wait() completes. shutdown() sends the shutdown request and then waits for that notification.

Compatibility and operations

  • Public API or generated protocol: N/A.
  • Configuration or defaults: N/A.
  • Snapshot manifest, artifact layout, or storage format: N/A.
  • Upgrade and rollback: N/A.
  • Host requirements, permissions, ports, or dependencies: N/A.

Validation

  • make fmt
  • make clippy
  • make test-unit
  • Relevant Rust integration tests
  • make -C services test (required when services/ changes)
  • Generated clients/server regenerated with the documented make target
  • Documentation updated
  • Benchmarks or performance comparison completed

Skipped checks and reasons:

No generated code, services, documentation, or performance-sensitive behavior changed.

Risks and reviewer notes

The server shutdown now waits until the daemon process has actually exited.

Checklist

  • The PR contains one coherent change and no unrelated formatting or refactoring.
  • New behavior is covered by tests, or I explained why testing is impractical.
  • Logs and examples contain no credentials, tokens, or private registry information.
  • I did not manually edit generated code without updating its source and regenerating it.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

🔍 OpenCodeReview found 2 issue(s) in this PR.

  • ✅ Successfully posted inline: 2 comment(s)

Comment on lines +499 to +504
self.inner
.daemon_exited
.subscribe()
.wait_for(|exited| *exited)
.await
.context("wait for ublk daemon exit")?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · high
This wait has no deadline or fallback, unlike the 5-second RPC call above. If the daemon acknowledges (or ignores) Shutdown but remains alive, or the watchdog never publishes its notification, every caller of shutdown() will hang indefinitely and can block application teardown. Bound this wait (and ideally terminate/escalate the child on timeout) so best-effort shutdown cannot become an unbounded cleanup operation.

Comment on lines 650 to +651
daemon_dead: AtomicBool::new(daemon_dead),
daemon_exited: tokio::sync::watch::channel(true).0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · low
The fixture marks daemon_exited true regardless of daemon_dead. Thus new_for_test(..., false) models a live daemon as already exited, and any shutdown test that does not manually reset the channel can pass without exercising the watchdog wait (the existing shutdown_succeeds_even_without_response test does exactly that). Initialize this channel from daemon_dead, or make the fixture explicitly configure the exit state.

@yingdi-shan yingdi-shan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@yingdi-shan
yingdi-shan merged commit 66ccef8 into kvcache-ai:main Sep 9, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

First sandbox start after 'systemctl restart aenv' fails with ublk-daemon.sock: No such file or directory

2 participants