Skip to content

fix(telemetry): publish the compaction swap catalog atomically - #500

Merged
vishr merged 1 commit into
mainfrom
fix/compaction-swap-atomic-snapshot
Nov 16, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/compaction-swap-atomic-snapshot

Conversation

@vishr

@vishr vishr commented Nov 16, 2026

Copy link
Copy Markdown
Member

Summary

Fixes #499. A reader querying concurrently with a compaction swap could under-count rows that the swap conserves, losing a whole compaction group.

Root cause

PublishReplacement swaps N inputs plus 1 output under a single p.mu.Lock(), but it did so by calling setBatchLocked once per batch. setBatchLocked republishes the canonical reader snapshot (metadataSnapshot atomic.Pointer) on every call, so a swap of 8 inputs republished the catalog 9 times, stepping through intermediate catalogs where the inputs had already been removed but the output was not yet installed.

Readers call BatchMetadata() lock-free and build their read_parquet([...explicit files]) list from it (internal/engine/file_snapshot.go). A reader that captured the snapshot in that gap built a file list that had lost a whole compaction group but not yet gained its replacement, under-counting rows. The snapshot pin only defers deletion of retired files so captured files stay on disk; it does not make snapshot capture atomic, which is correct and unchanged.

This reliably reproduced under -race:

--- FAIL: TestCompactionAndRetentionUnderConcurrentReads
    retirement_test.go:103: windowed read saw spans=72 logs=72 metrics=72, want 80 each
--- FAIL: TestCompactionAndRetentionUnderConcurrentReads
    retirement_test.go:103: unwindowed read saw 78 spans, want 80 or 81

Fix

Make the whole swap publish the catalog snapshot exactly once, so lock-free readers only ever observe the pre-swap or the post-swap catalog, never an intermediate.

  • Split setBatchLocked into setBatchLockedNoPublish (the map and generation-0 inventory accounting, no snapshot store, no debt-gate close) and publishBatchesLocked (the single publishMetadataLocked plus the debt-gate close). setBatchLocked is now the composition of the two, so every other caller — commit, restoration, rollback, startup, retention — keeps identical per-call behaviour.
  • PublishReplacement now retires the inputs and installs the output with setBatchLockedNoPublish, then calls publishBatchesLocked once after recordReplacesLocked. The single publish is the only metadataSnapshot store for the swap. The on-disk rename, fsync, replaces.json write, retire, and deferred-deletion logic are all unchanged.

The retention path (PruneBefore) also publishes per batch and is intentionally left as is: retention only drops out-of-window expired batches, so it does not conserve rows and every intermediate catalog is an independently valid state. The regression test encodes this — windowed reads must equal the conserved count while unwindowed reads tolerate the expired batch being visible or already pruned.

Verification

  • just test-race "-run TestCompactionAndRetentionUnderConcurrentReads -count=20 ./internal/engine/": pass, no data race (reproduced 6/10 failures before the change).
  • just test "./internal/telemetry/... ./internal/engine/...": pass.
  • just vet, gofmt: clean.
  • No test assertions changed; the regression relies on -count under -race with no production test seams.

PublishReplacement swaps a compacted output for its inputs under p.mu, but
it did so by calling setBatchLocked once per input and once for the output.
Every setBatchLocked call republishes the lock-free metadataSnapshot that
readers use to build their read_parquet file lists, so a swap of N inputs
republished the catalog N+1 times and stepped through intermediate states
in which the inputs were already gone but the replacement was not yet
installed.

A reader that captured BatchMetadata in that gap built a file list missing
a whole compaction group and under-counted rows that the swap conserves.
TestCompactionAndRetentionUnderConcurrentReads reproduced this under -race:
windowed reads saw fewer than the committed rows, and unwindowed reads
dropped well below the retention-tolerant range.

Split setBatchLocked into a mutate-only setBatchLockedNoPublish plus a
single publishBatchesLocked step. setBatchLocked keeps its exact behavior
for every other caller (commit, restoration, rollback, startup, retention).
PublishReplacement now mutates all inputs and the output with the no-publish
variant and publishes once after recording the replaces list, so readers
only ever observe the pre-swap or post-swap catalog. The pin, retire,
deletion, fsync, marker and on-disk rename sequences are unchanged; only the
in-memory snapshot publication becomes atomic.

Fixes #499
@vishr
vishr added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit ff28b4f Nov 16, 2026
31 checks passed
@vishr
vishr deleted the fix/compaction-swap-atomic-snapshot branch November 16, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flaky: TestCompactionAndRetentionUnderConcurrentReads loses rows under -race on repeated runs

1 participant