Repository navigation
refactor(telemetry): remove dead compaction crash-recovery marker - #495
Merged
Merged
Conversation
vishr
enabled auto-merge
November 16, 2026 00:45
vishr
force-pushed
the
chore/remove-dead-compaction-rollback
branch
from
November 16, 2026 02:22
f39dc83 to
432f582
Compare
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Nov 16, 2026
vishr
force-pushed
the
chore/remove-dead-compaction-rollback
branch
from
November 16, 2026 13:18
432f582 to
c740ff3
Compare
Compaction never produced the state its recovery machinery was built to undo. Nothing creates `<id>.retired-<output>` input directories: inputs stay `.batch` and a published output's `replaces.json` retires them. So the `COMPACTION.json` marker, `RestoreRetiredInputs` and its move branch, the `.retired-<output>` handling, the marker-protected cleanup set, and the `Open` path that read the marker and refused to start were all dead or redundant with recovery that already runs on every boot. A compaction is now purely in-process: it merges into `compaction/<output>` staging and commits by renaming that into the batch namespace, where the output's `replaces.json` retires its inputs. The whole recovery protocol is to discard the `compaction/` staging on boot and let `replaces.json` decide. That is equivalent and crash-safe on both sides of the single rename: a crash before it leaves the inputs live as `.batch` and the staging is dropped; a crash after it leaves the output present with its `replaces.json`, which `loadBatches` applies to retire the inputs. Neither window loses rows or orphans inputs. Deleted: `compactionMarker`, `recoverCompaction`, `completeCompaction`, `RecoverParquet`, `validateCompactionMarker`, `RestoreRetiredInputs`, `protectedRetiredSuffix`, `logUnresolvedCompaction`, `compactionRunbookURL`, the `COMPACTION.json` write, and the unused `pathExists`/`parquetPublishFunc` helpers. `compactParquet` now calls `PublishReplacement` directly after `PrepareReplacement`. The interrupted-merge attempt-file backoff (`COMPACTION-ATTEMPT.json`) and `replaces.json` itself are untouched. The repair quarantine guard that consulted the marker to refuse a batch mid-compaction is removed; it is superseded by the data-directory flock (#485), so repair cannot run against a live data dir and has no in-process compaction to collide with. A crash-matrix test (`TestInterruptedCompaction*`, `TestFailedCompactionPublishKeepsInputsLiveAndRetryPreservesRows`) drives each window on a real filesystem and asserts exact row counts and catalog composition, so it fails if `replaces.json` masking regresses, if boot completes a merge it should discard, or if any row is lost. The removed marker tests' crash-safety coverage is preserved by these. Closes #486
vishr
force-pushed
the
chore/remove-dead-compaction-rollback
branch
from
November 16, 2026 14:09
c740ff3 to
52c4824
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was dead
Nothing in the current code creates
<id>.retired-<output>directories — inputs stay as.batchand each compacted output'sreplaces.jsoncommits the swap. So the move/rollback branch ofRestoreRetiredInputs,protectedRetiredSuffix, and the.retired-handling informat.gowere dead. AndCOMPACTION.jsononly ever rolled a stage→final rename forward, which boot'sloadBatchesalready redoes from the output'sreplaces.json; yet a marker whose stage and final were both missing madeOpenrefuse to start.Closes #486.
The surviving recovery is equivalent
PublishReplacementwrites and syncs the output'sreplaces.jsoninto the staging directory before the singleos.Rename(stage, final), so it travels atomically with the rename. On boot,loadBatches/retiredAtLoadreconstruct retirement from each present output'sreplaces.json. That makes both sides of the rename consistent with no durable marker:defer), the selected inputs are still live.batch, and a later pass redoes the merge. No rows lost.replaces.json, whichloadBatchesapplies to retire the inputs exactly once. No rows lost, no double counting.So the entire compaction boot-recovery protocol is now: discard
compaction/staging and letreplaces.jsondecide.Openno longer refuses to start on a marker.Removed
compactionMarker,recoverCompaction,completeCompaction,RecoverParquet,validateCompactionMarker, the pre-pass marker check, theCOMPACTION.jsonwrite,RestoreRetiredInputs,protectedRetiredSuffix,logUnresolvedCompaction+compactionRunbookURL+ the refuse-to-start path, and the.retired-<output>handling (bare.retiredretention handling stays).compactParquetnow completes by callingPublishReplacementdirectly.The interrupted-merge attempt-file backoff (
COMPACTION-ATTEMPT.json,writeCompactionAttempt/readCompactionAttempt/clearCompactionAttempt,mergeCompactionhalving,errCompactionPreempted) is a separate mechanism and is untouched, as arereplaces.json/retiredAtLoadand retention's bare.retireddelete.Repair-quarantine guard
The marker-based "refuse to quarantine a live compaction member" guard (
ensureBatchOutsideLiveCompaction) is removed. It is superseded by the data-directory flock (#485 / #490): repair cannot run against a live data directory, so there is no in-process compaction to collide with. This PR should therefore merge after #490. The merge queue is serial and #490 is already ahead, so ordering holds.Tests — crash-matrix (the whole justification)
New
internal/telemetry/store/compaction_recovery_test.godrives a realParquetStore/Repositoryon a real filesystem, abandons it in each crash window, reopens the same directory, and asserts exact row counts and catalog composition:TestInterruptedCompactionBeforeRenameKeepsInputsLive— W1: all inputs live, output absent, staging discarded.TestInterruptedCompactionAfterRenameRetiresInputsFromReplaces— W2: output live, inputs masked once (not double-counted).TestInterruptedCompactionMidInputDeletionRetiresRemainder— W3: re-expresses the removed interrupted-swap recovery test.TestFailedCompactionPublishKeepsInputsLiveAndRetryPreservesRows— a failed publish leaves inputs live; a retry preserves rows.These were confirmed to fail if
loadBatchesstops masking inputs byreplaces.json(W2 and W3 double-counted), proving they actually exercise the recovery. The removed marker tests' crash-safety coverage is preserved by these.Verification
just test "./internal/telemetry/..."andjust test "./internal/engine/...",just vet, andjust check-static(gofmt, deadcode — nothing newly orphaned — behavioral test names) all green. Net diff −888/+382 across 15 files; troubleshooting/data-layout docs updated to drop the removed marker.