Repository navigation
fix(app): stop signals only the process that holds the PID file lock - #777
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
extenddb stop read the PID file, checked that the process existed with kill(pid, 0), and sent it SIGTERM. When a server exited without removing its PID file and the system later gave its PID to another process, stop signalled that process. With a build of main, a PID file naming a sleep process under a fresh HOME made stop print "extenddb stopped" and end the sleep. A running server now holds an exclusive flock on its PID file. In daemon mode daemonize already took that lock on its own descriptor and keeps it until the process exits. In foreground mode with --write-pid-file the server takes it through the new pid_file::hold, which writes the PID to a new file, locks it and renames it into place, so a reader sees either an old file no process holds or the server's file, complete and locked. Because flock locks belong to open file descriptions, hold leaves a file that already names this process and is locked to daemonize, and leaves a file another running process holds alone; that server then serves without a PID file and logs a warning. In a run directory the server cannot write, hold rewrites an existing PID file in place, as earlier releases did, so a file an init system created for the server still works. serve's error path removes the PID file only when it names this process. stop reads the PID and probes the lock through one open of the file, with pid_file::read. It checks the lock before it checks that the process exists, because kill(pid, 0) fails with EPERM for a server another user runs, and after SIGTERM it removes only the file it read. For a file nothing holds: - A file nothing holds whose process is alive is stale when nothing accepts a connection on the port at the bind address the config file and the EXTENDDB__ variables give (an IPv6 address may be in brackets, as serve accepts it), at the loopback addresses for a wildcard one, or at 127.0.0.1. stop says the process is not an extenddb server, removes the file it read (pid_file::remove_stale checks it is still the same file), signals nothing, and exits 0. - When something accepts a connection there, stop exits 1 with a kill hint and keeps the file, because a server that an earlier release started with --foreground --write-pid-file holds no lock. status shows the PID only when the file is held. The troubleshooting guide lists the two new stop messages, including the case an older server bound to an address the config does not name still falls into, and the run-tests restart hint names the kill fallback. Unit tests cover a held file, a stale file replaced by a new one, a writable file in a read-only directory rewritten in place, a free file that names this process, a file daemonize holds for this process, a file another process holds, removal of only the file that was read, a missing file, and the probed addresses. CLI tests write a sleep process's PID to the PID file and check that stop leaves it running and removes the file, and that with the port in use stop exits 1, keeps the file, and status shows no PID; both fail against a build of main. On loopback under an isolated HOME, stop and status worked against a foreground server and a daemon from this change, and stop refused, without signalling it, a foreground server from a build of main on 127.0.0.1 and on ::1. Fixes #710.
LeeroyHannigan
requested review from
amrith,
c33howard,
eliseong,
hitesh96db,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
November 3, 2026 02:25
crates/app/src/cmd_stop.rs keeps the shutdown wait test from main and the address probe tests from this branch in one test module.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
extenddb stopread the PID file, checked that the process existed withkill(pid, 0), and sent it SIGTERM. When a server exited without removing its PID file, for example after a crash, and the system later gave its PID to another process,stopsignalled that process.Reproduction on main
A dev-mode build under a fresh
HOME: startsleep 300, write its PID to~/.extenddb/run/extenddb-19891.pid, runextenddb stop --port 19891. It printedextenddb stopped (pid ...)and thesleepprocess ended.Change
A running server holds an exclusive
flockon its PID file, andstopandstatustrust the PID only while the file is held.daemonize0.5 opens the PID file, takesLOCK_EX, writes the PID and keeps that descriptor open until the process exits.--write-pid-filenow takes it too, through the newpid_file::hold. It writes the PID to a new file, locks it and renames it into place, so a reader sees either an old file no process holds or the server's file, complete and locked.flocklocks belong to open file descriptions, soholdleaves a file that already names this process and is locked (daemon mode). It also leaves a file another running process holds, for example a server on the same port number bound to another address; this server then runs without a PID file and logs a warning.serve's error path removes the PID file only when it names this process.holdrewrites an existing PID file in place, as earlier releases did, so a file an init system created for the server still works.stopreads the PID and probes the lock through one open of the file (pid_file::read). It checks the lock before it checks that the process exists, becausekill(pid, 0)fails withEPERMfor a server another user runs, and after SIGTERM it removes only the file it read. For a file nothing holds whose process is alive:EXTENDDB__variables give (an IPv6 address may be in brackets, asserveaccepts it), at both loopback addresses for a wildcard one, or at 127.0.0.1, the file is stale.stopsays the process is not an extenddb server, removes the file it read (pid_file::remove_stalechecks that the path still names it), signals nothing, and exits 0.stopexits 1 with akillhint and keeps the file, because a server that an earlier release started with--foreground --write-pid-fileholds no lock.statusshows the PID only when the file is held, and sayspid unknownotherwise.docs/troubleshooting.mdlists the two newstopmessages, and thedevtools/run-testsrestart hint names thekillfallback.pid_file::try_lockrepeats theflockcall of main'sbackup_store::try_lock_exclusive, with the operation as an argument, becausepid_file::readprobes withLOCK_SHand releases withLOCK_UN. The standard library'sFile::try_lock_sharedandFile::unlockwould replace the probe and the release, but they need Rust 1.89, and the workspace'srust-versionis 1.88.An older lock-less server bound to an address the config
stopreads does not name is still taken for a stale file:stopremoves its PID file without signalling it, and the troubleshooting entry says how to find it.Tests
pid_file.rs: a held file is seen as held until it is dropped; a stale file is replaced by a new one; a writable file in a read-only directory is rewritten in place and held; a free file that names this process is taken over; a file locked on another descriptor for this process is left to it; a file another process holds is left alone;remove_staleremoves only the file that was read; a missing file isNotFound. Unit tests incmd_stop.rscover the probed addresses, with and without brackets.TestStopPidFileLockintests/test_cli_container_readiness.py, which needs no backend: with nothing listening,stopleaves asleepwhose PID is in the file running and removes the file; with the port in use,stopexits 1, keeps the file and leaves thesleeprunning, andstatusshows no PID. Both tests pass on this branch and fail on a build of main.Testing
cargo +1.97.0 fmt --all -- --checkandcargo +1.97.0 clippy -p extenddb-server -p extenddb-app --all-targets -- -D warnings: clean.cargo +1.88.0 test --locked -p extenddb-server -p extenddb-app: 305 passed.HOME:stopstatussleep, nothing listeningsleepleft running, file removed--write-pid-fileserver from this branch--write-pid-fileserver from a build of main, on 127.0.0.1killhint, server left running, file kept::1,stopgiven the sameEXTENDDB__SERVER__BIND_ADDRkillhint, server left running, file keptNot in this change
catalog-checkstill decides whether a server is running from the PID alone (catalog-check: the server-running check misses a foreground server and a server started with --port #678 covers its other gaps).daemonizewrites its own, without checking whether a running server holds it, as on main. A server on the same port number bound to another address loses its file, and that server's shutdown later removes the daemon's.holdrecognizes daemon mode by the held file naming the server's own PID. A server in another PID namespace that shares the run directory and has the same PID number would be taken for the daemon. Passing the mode fromserveneeds aServeParamschange incmd_serve.rs, which several open PRs edit.Fixes #710.