Skip to content

fix(store): lock the telemetry data directory against a second process - #490

Merged
vishr merged 1 commit into
mainfrom
fix/data-dir-flock
Nov 16, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/data-dir-flock

Conversation

@vishr

@vishr vishr commented Nov 15, 2026

Copy link
Copy Markdown
Member

Problem

There was no flock or lockfile on the telemetry data directory. A second process started on the same directory would delete the live process's staging/ and compaction/ work mid-write, remove _schema.batch, and run CollectRetired against an empty pin table. pitot repair quarantine would rename a .batch directory with no pin and no liveness check. DuckDB's own file lock might stop a second server incidentally, depending on open order, but that is not guaranteed.

Closes #485.

Fix

Take an exclusive, non-blocking advisory flock on the data directory for the lifetime of the process:

  • New internal/telemetry/store/lock.go: acquireDataDirLock(root) opens a dedicated root/LOCK file and takes unix.Flock(fd, LOCK_EX|LOCK_NB) (using the existing golang.org/x/sys/unix dependency — no new modules). When another process already holds it, the call fails fast with an error naming the directory and telling the operator to stop the running Pitot or repair first, rather than blocking. Close() does LOCK_UN and closes the descriptor.
  • store.Open acquires the lock right after MkdirAll, before touching any batch file, and releases it in Close() (after the Parquet handles close, so the directory is quiescent before another process can acquire it). All existing failure paths run through Close(), so the lock is always released.
  • The repair CLI entrypoints VerifyBatches and QuarantineBatch take the same lock on entry and release it on return, so repair refuses to run against a live data directory.

The LOCK file is created at runtime inside the data directory, never in the repository or the Docker build context; its presence on disk is meaningless without a live descriptor holding the flock, so it survives restarts harmlessly.

No same-process self-conflict

A single invocation runs either repair or the server, never both: runRepair returns before the server's store.Open, and within repair exactly one of VerifyBatches/QuarantineBatch runs. store.Open is called once in the server path. So a process-level flock cannot reject itself.

Tests

  • New TestOpenRejectsSecondProcessOnSameDataDir (written first, confirmed red before the fix): a second Open on a held directory fails with an error naming the directory and another process; after Close, a clean reopen succeeds.
  • TestRepositoryRecoversInterruptedCompactionSwap now closes the crashed repository before the recovering reopen. This faithfully models a crashed process exiting — the OS releases the flock on exit, and ParquetStore.Close() is a no-op, so the on-disk marker, staged output, and retired inputs are left intact for recovery. Without this, the test would self-deadlock against the new lock.

No hot-path cost: the lock is taken once at Open and released once at Close, nothing per operation.

Verification

just vet, just check-static (gofmt, deadcode, behavioral test names), and just test "./internal/telemetry/..." all green, plus every other package that calls store.Open.

A telemetry data directory has no flock or lockfile, so nothing stops a
second process from opening the same directory as a live one. When it
does, the newcomer deletes the live process's staging/ and compaction/
directories mid-write, removes _schema.batch, and runs CollectRetired
against an empty pin table; pitot repair quarantine likewise renames a
.batch directory out from under a running store with no liveness check.
DuckDB's own file lock only stops a second server incidentally,
depending on open order, so it is not a guarantee.

Take an exclusive, non-blocking flock (LOCK_EX|LOCK_NB) on a dedicated
root/LOCK file in store.Open, held for the lifetime of the Repository
and released in Close. A second Open on the same directory now fails
fast with an error naming the directory and saying another process holds
it, instead of racing into corruption. The same lock guards the repair
CLI entrypoints (VerifyBatches and QuarantineBatch), so repair refuses
to run against a live data directory and releases the lock when it
returns.

The lock is a per-process guard taken once at Open, off every hot path.
A crashed process releases its flock on exit, so the normal
Open -> Close -> reopen path and crash-recovery startup are unaffected;
the interrupted-compaction-swap test now closes the crashed repository
before the recovering reopen, modelling that process exit.

Closes #485
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 16, 2026
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit c4c5c66 Nov 16, 2026
31 checks passed
@vishr
vishr deleted the fix/data-dir-flock branch November 16, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: no data-directory lock

1 participant