Skip to content

fix(app): stop signals only the process that holds the PID file lock - #777

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/710-stop-checks-pid-file-lock
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/710-stop-checks-pid-file-lock

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

extenddb stop read the PID file, checked that the process existed with kill(pid, 0), and sent it SIGTERM. When a server exited without removing its PID file, for example after a crash, and the system later gave its PID to another process, stop signalled that process.

Reproduction on main

A dev-mode build under a fresh HOME: start sleep 300, write its PID to ~/.extenddb/run/extenddb-19891.pid, run extenddb stop --port 19891. It printed extenddb stopped (pid ...) and the sleep process ended.

Change

A running server holds an exclusive flock on its PID file, and stop and status trust the PID only while the file is held.

  • Daemon mode already held the lock: daemonize 0.5 opens the PID file, takes LOCK_EX, writes the PID and keeps that descriptor open until the process exits.
  • Foreground mode with --write-pid-file now takes it too, through the new pid_file::hold. It writes the PID to a new file, locks it and renames it into place, so a reader sees either an old file no process holds or the server's file, complete and locked.
  • flock locks belong to open file descriptions, so hold leaves a file that already names this process and is locked (daemon mode). It also leaves a file another running process holds, for example a server on the same port number bound to another address; this server then runs without a PID file and logs a warning. serve's error path removes the PID file only when it names this process.
  • In a run directory the server cannot write, hold rewrites an existing PID file in place, as earlier releases did, so a file an init system created for the server still works.
  • stop reads the PID and probes the lock through one open of the file (pid_file::read). It checks the lock before it checks that the process exists, because kill(pid, 0) fails with EPERM for a server another user runs, and after SIGTERM it removes only the file it read. For a file nothing holds whose process is alive:
    • When nothing accepts a connection on the port at the bind address the config file and the EXTENDDB__ variables give (an IPv6 address may be in brackets, as serve accepts it), at both loopback addresses for a wildcard one, or at 127.0.0.1, the file is stale. stop says the process is not an extenddb server, removes the file it read (pid_file::remove_stale checks that the path still names it), signals nothing, and exits 0.
    • When something accepts a connection there, stop exits 1 with a kill hint and keeps the file, because a server that an earlier release started with --foreground --write-pid-file holds no lock.
  • status shows the PID only when the file is held, and says pid unknown otherwise.
  • docs/troubleshooting.md lists the two new stop messages, and the devtools/run-tests restart hint names the kill fallback.
  • pid_file::try_lock repeats the flock call of main's backup_store::try_lock_exclusive, with the operation as an argument, because pid_file::read probes with LOCK_SH and releases with LOCK_UN. The standard library's File::try_lock_shared and File::unlock would replace the probe and the release, but they need Rust 1.89, and the workspace's rust-version is 1.88.

An older lock-less server bound to an address the config stop reads does not name is still taken for a stale file: stop removes its PID file without signalling it, and the troubleshooting entry says how to find it.

Tests

  • Unit tests in pid_file.rs: a held file is seen as held until it is dropped; a stale file is replaced by a new one; a writable file in a read-only directory is rewritten in place and held; a free file that names this process is taken over; a file locked on another descriptor for this process is left to it; a file another process holds is left alone; remove_stale removes only the file that was read; a missing file is NotFound. Unit tests in cmd_stop.rs cover the probed addresses, with and without brackets.
  • TestStopPidFileLock in tests/test_cli_container_readiness.py, which needs no backend: with nothing listening, stop leaves a sleep whose PID is in the file running and removes the file; with the port in use, stop exits 1, keeps the file and leaves the sleep running, and status shows no PID. Both tests pass on this branch and fail on a build of main.

Testing

  • cargo +1.97.0 fmt --all -- --check and cargo +1.97.0 clippy -p extenddb-server -p extenddb-app --all-targets -- -D warnings: clean.
  • cargo +1.88.0 test --locked -p extenddb-server -p extenddb-app: 305 passed.
  • Dev-mode builds on loopback under an isolated HOME:
Case stop status
PID file names a live sleep, nothing listening exit 0, not an extenddb server, sleep left running, file removed
foreground --write-pid-file server from this branch stopped it showed its PID
daemon from this branch stopped it showed its PID
foreground --write-pid-file server from a build of main, on 127.0.0.1 exit 1 with the kill hint, server left running, file kept
the same on ::1, stop given the same EXTENDDB__SERVER__BIND_ADDR exit 1 with the kill hint, server left running, file kept

Not in this change

  • catalog-check still decides whether a server is running from the PID alone (catalog-check: the server-running check misses a foreground server and a server started with --port #678 covers its other gaps).
  • Daemon start still removes an existing PID file before daemonize writes its own, without checking whether a running server holds it, as on main. A server on the same port number bound to another address loses its file, and that server's shutdown later removes the daemon's.
  • A foreground server started while another server on the same port drains after SIGTERM finds the file held and runs without a PID file, the state main also ended in.
  • hold recognizes daemon mode by the held file naming the server's own PID. A server in another PID namespace that shares the run directory and has the same PID number would be taken for the daemon. Passing the mode from serve needs a ServeParams change in cmd_serve.rs, which several open PRs edit.

Fixes #710.

extenddb stop read the PID file, checked that the process existed with
kill(pid, 0), and sent it SIGTERM. When a server exited without removing
its PID file and the system later gave its PID to another process, stop
signalled that process. With a build of main, a PID file naming a sleep
process under a fresh HOME made stop print "extenddb stopped" and end the
sleep.

A running server now holds an exclusive flock on its PID file. In daemon
mode daemonize already took that lock on its own descriptor and keeps it
until the process exits. In foreground mode with --write-pid-file the
server takes it through the new pid_file::hold, which writes the PID to a
new file, locks it and renames it into place, so a reader sees either an
old file no process holds or the server's file, complete and locked.
Because flock locks belong to open file descriptions, hold leaves a file
that already names this process and is locked to daemonize, and leaves a
file another running process holds alone; that server then serves without
a PID file and logs a warning. In a run directory the server cannot
write, hold rewrites an existing PID file in place, as earlier releases
did, so a file an init system created for the server still works. serve's
error path removes the PID file only when it names this process.

stop reads the PID and probes the lock through one open of the file, with
pid_file::read. It checks the lock before it checks that the process
exists, because kill(pid, 0) fails with EPERM for a server another user
runs, and after SIGTERM it removes only the file it read. For a file
nothing holds:

- A file nothing holds whose process is alive is stale when nothing
  accepts a connection on the port at the bind address the config file
  and the EXTENDDB__ variables give (an IPv6 address may be in
  brackets, as serve accepts it), at the loopback addresses for a
  wildcard one, or at 127.0.0.1. stop says the process is not an extenddb
  server, removes the file it read (pid_file::remove_stale checks it is
  still the same file), signals nothing, and exits 0.
- When something accepts a connection there, stop exits 1 with a kill
  hint and keeps the file, because a server that an earlier release
  started with --foreground --write-pid-file holds no lock.

status shows the PID only when the file is held. The troubleshooting
guide lists the two new stop messages, including the case an older
server bound to an address the config does not name still falls into,
and the run-tests restart hint names the kill fallback.

Unit tests cover a held file, a stale file replaced by a new one, a
writable file in a read-only directory rewritten in place, a free file
that names this process, a file daemonize holds for this process, a file
another process holds, removal of only the file that was read, a missing
file, and the probed addresses. CLI tests write a sleep process's
PID to the PID file and check that stop leaves it running and removes the
file, and that with the port in use stop exits 1, keeps the file, and
status shows no PID; both fail against a build of main. On loopback
under an isolated HOME, stop and status worked against a foreground
server and a daemon from this change, and stop refused, without
signalling it, a foreground server from a build of main on 127.0.0.1 and
on ::1.

Fixes #710.
crates/app/src/cmd_stop.rs keeps the shutdown wait test from main and the address probe tests from this branch in one test module.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extenddb stop sends SIGTERM to whatever process the pid file names

1 participant