Repository navigation
fix(telemetry): publish the compaction swap catalog atomically - #500
Merged
Merged
Conversation
PublishReplacement swaps a compacted output for its inputs under p.mu, but it did so by calling setBatchLocked once per input and once for the output. Every setBatchLocked call republishes the lock-free metadataSnapshot that readers use to build their read_parquet file lists, so a swap of N inputs republished the catalog N+1 times and stepped through intermediate states in which the inputs were already gone but the replacement was not yet installed. A reader that captured BatchMetadata in that gap built a file list missing a whole compaction group and under-counted rows that the swap conserves. TestCompactionAndRetentionUnderConcurrentReads reproduced this under -race: windowed reads saw fewer than the committed rows, and unwindowed reads dropped well below the retention-tolerant range. Split setBatchLocked into a mutate-only setBatchLockedNoPublish plus a single publishBatchesLocked step. setBatchLocked keeps its exact behavior for every other caller (commit, restoration, rollback, startup, retention). PublishReplacement now mutates all inputs and the output with the no-publish variant and publishes once after recording the replaces list, so readers only ever observe the pre-swap or post-swap catalog. The pin, retire, deletion, fsync, marker and on-disk rename sequences are unchanged; only the in-memory snapshot publication becomes atomic. Fixes #499
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #499. A reader querying concurrently with a compaction swap could under-count rows that the swap conserves, losing a whole compaction group.
Root cause
PublishReplacementswaps N inputs plus 1 output under a singlep.mu.Lock(), but it did so by callingsetBatchLockedonce per batch.setBatchLockedrepublishes the canonical reader snapshot (metadataSnapshot atomic.Pointer) on every call, so a swap of 8 inputs republished the catalog 9 times, stepping through intermediate catalogs where the inputs had already been removed but the output was not yet installed.Readers call
BatchMetadata()lock-free and build theirread_parquet([...explicit files])list from it (internal/engine/file_snapshot.go). A reader that captured the snapshot in that gap built a file list that had lost a whole compaction group but not yet gained its replacement, under-counting rows. The snapshot pin only defers deletion of retired files so captured files stay on disk; it does not make snapshot capture atomic, which is correct and unchanged.This reliably reproduced under
-race:Fix
Make the whole swap publish the catalog snapshot exactly once, so lock-free readers only ever observe the pre-swap or the post-swap catalog, never an intermediate.
setBatchLockedintosetBatchLockedNoPublish(the map and generation-0 inventory accounting, no snapshot store, no debt-gate close) andpublishBatchesLocked(the singlepublishMetadataLockedplus the debt-gate close).setBatchLockedis now the composition of the two, so every other caller — commit, restoration, rollback, startup, retention — keeps identical per-call behaviour.PublishReplacementnow retires the inputs and installs the output withsetBatchLockedNoPublish, then callspublishBatchesLockedonce afterrecordReplacesLocked. The single publish is the onlymetadataSnapshotstore for the swap. The on-disk rename, fsync,replaces.jsonwrite, retire, and deferred-deletion logic are all unchanged.The retention path (
PruneBefore) also publishes per batch and is intentionally left as is: retention only drops out-of-window expired batches, so it does not conserve rows and every intermediate catalog is an independently valid state. The regression test encodes this — windowed reads must equal the conserved count while unwindowed reads tolerate the expired batch being visible or already pruned.Verification
just test-race "-run TestCompactionAndRetentionUnderConcurrentReads -count=20 ./internal/engine/": pass, no data race (reproduced 6/10 failures before the change).just test "./internal/telemetry/... ./internal/engine/...": pass.just vet,gofmt: clean.-countunder-racewith no production test seams.