Repository navigation
fix(store): lock the telemetry data directory against a second process - #490
Merged
Merged
Conversation
A telemetry data directory has no flock or lockfile, so nothing stops a second process from opening the same directory as a live one. When it does, the newcomer deletes the live process's staging/ and compaction/ directories mid-write, removes _schema.batch, and runs CollectRetired against an empty pin table; pitot repair quarantine likewise renames a .batch directory out from under a running store with no liveness check. DuckDB's own file lock only stops a second server incidentally, depending on open order, so it is not a guarantee. Take an exclusive, non-blocking flock (LOCK_EX|LOCK_NB) on a dedicated root/LOCK file in store.Open, held for the lifetime of the Repository and released in Close. A second Open on the same directory now fails fast with an error naming the directory and saying another process holds it, instead of racing into corruption. The same lock guards the repair CLI entrypoints (VerifyBatches and QuarantineBatch), so repair refuses to run against a live data directory and releases the lock when it returns. The lock is a per-process guard taken once at Open, off every hot path. A crashed process releases its flock on exit, so the normal Open -> Close -> reopen path and crash-recovery startup are unaffected; the interrupted-compaction-swap test now closes the crashed repository before the recovering reopen, modelling that process exit. Closes #485
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
There was no flock or lockfile on the telemetry data directory. A second process started on the same directory would delete the live process's
staging/andcompaction/work mid-write, remove_schema.batch, and runCollectRetiredagainst an empty pin table.pitot repair quarantinewould rename a.batchdirectory with no pin and no liveness check. DuckDB's own file lock might stop a second server incidentally, depending on open order, but that is not guaranteed.Closes #485.
Fix
Take an exclusive, non-blocking advisory flock on the data directory for the lifetime of the process:
internal/telemetry/store/lock.go:acquireDataDirLock(root)opens a dedicatedroot/LOCKfile and takesunix.Flock(fd, LOCK_EX|LOCK_NB)(using the existinggolang.org/x/sys/unixdependency — no new modules). When another process already holds it, the call fails fast with an error naming the directory and telling the operator to stop the running Pitot or repair first, rather than blocking.Close()doesLOCK_UNand closes the descriptor.store.Openacquires the lock right afterMkdirAll, before touching any batch file, and releases it inClose()(after the Parquet handles close, so the directory is quiescent before another process can acquire it). All existing failure paths run throughClose(), so the lock is always released.VerifyBatchesandQuarantineBatchtake the same lock on entry and release it on return, so repair refuses to run against a live data directory.The
LOCKfile is created at runtime inside the data directory, never in the repository or the Docker build context; its presence on disk is meaningless without a live descriptor holding the flock, so it survives restarts harmlessly.No same-process self-conflict
A single invocation runs either repair or the server, never both:
runRepairreturns before the server'sstore.Open, and within repair exactly one ofVerifyBatches/QuarantineBatchruns.store.Openis called once in the server path. So a process-level flock cannot reject itself.Tests
TestOpenRejectsSecondProcessOnSameDataDir(written first, confirmed red before the fix): a secondOpenon a held directory fails with an error naming the directory and another process; afterClose, a clean reopen succeeds.TestRepositoryRecoversInterruptedCompactionSwapnow closes the crashed repository before the recovering reopen. This faithfully models a crashed process exiting — the OS releases the flock on exit, andParquetStore.Close()is a no-op, so the on-disk marker, staged output, and retired inputs are left intact for recovery. Without this, the test would self-deadlock against the new lock.No hot-path cost: the lock is taken once at
Openand released once atClose, nothing per operation.Verification
just vet,just check-static(gofmt, deadcode, behavioral test names), andjust test "./internal/telemetry/..."all green, plus every other package that callsstore.Open.