Repository navigation
fix(engine): self-heal a missed rollup-cache checkpoint instead of degrading readiness for an hour - #494
Merged
Conversation
…grading readiness for an hour The rollup-cache checkpoint folds the write-ahead log back into its file. Its non-FORCE form refuses while a read that began before the last write commit is still open, and the old 250ms retry budget was shorter than almost any such read, so a busy instance reliably missed the fold. The miss folded into the whole maintenance error, which flipped /readyz to degraded until the next maintenance pass -- an hour away by default. A missed fold harms nothing a restart or a page would fix: the WAL stays correct and queryable, just larger until the next successful fold. Separate checkpoint health from maintenance health. A checkpoint-only failure is recorded in new CheckpointHealth state (lastCheckpointOK/At/Err, checkpointFailures, a pending flag) and never touches maintenance health. Retention DELETEs, prune, compaction, and recovery still fail maintenance as before -- those let storage grow unchecked, so they must degrade readiness immediately. The retention DELETEs move into their own gated helper (pruneRollupCache) so the fold can be driven separately. Drive the checkpoint off a pending flag that both the hourly pass and the 10s compaction cycle honor, rather than checkpointing every 10s. When nothing is pending the cycle takes no write gate, so an idle WAL adds no contention with the 60s rollup tick; when a fold is pending it holds both write gates and the background slot exactly as the hourly pass did. The hourly pass marks the fold pending (an interval of rollup commits plus the retention DELETEs always leave work) and attempts it; a miss stays pending and the next ~10s cycle retries, so a transient conflict self-heals in seconds instead of an hour. Only a context cancellation propagates, so the maintenance loop still stops promptly. Raise the retry budget from 250ms to 3s. Both write gates are idle-cheap relative to the 60s rollup tick, so giving an older reader a few seconds to drain lets the common transient conflict clear in place. The exact-message retry gate (checkpointRetryable/checkpointBusyMessage) is unchanged. Report checkpoint separately in readiness via a new checkpoint check. Because the fold retries every 10s, a single recent miss is not degraded; only a fold that keeps failing past several cycles (checkpointDegradeAfterFailures) is. It never reports unhealthy -- the instance serves while the WAL grows -- and a genuinely wedged engine is still caught by the independent telemetry SELECT-1 probe. Closes #482
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The DuckDB
CHECKPOINTthat folds the rollup cache's write-ahead log back into its file ran only inside the hourlyrunRepositoryMaintenance, with a 250ms retry budget shorter than any read that began before the last write commit and is still running. So a busy instance reliably missed the fold. The miss folded into the whole maintenance error, which made/readyzreportdegradeduntil the next maintenance pass — one hour later by default — even though ingest and queries keep working and a restart would not help. Starting maintenance at boot also put the first checkpoint in the window with the most startup reads.Closes #482.
Fix
Separate checkpoint health from maintenance health. A checkpoint-only failure no longer touches
lastMaintenanceErr/maintenanceFailures. Retention DELETEs (extracted topruneRollupCache), prune, compaction, and recovery failures still degrade maintenance readiness immediately, because those let storage grow unchecked. The checkpoint gets its own health fields and aCheckpointHealth()accessor mirroringMaintenanceHealth().Retry a missed fold on the 10s compaction cycle. A
checkpointPendingflag is honored by both the hourly pass andrunParquetCompactionCycle. When nothing is pending the cycle takes no write gate, so an idle WAL adds no contention with the 60s rollup tick; when a fold is pending it holds both write gates and the background slot exactly as the old maintenance checkpoint did. A transient refusal therefore self-heals in ~10s instead of ~1h, and the boot checkpoint no longer has to succeed in the busy-read startup window.Budget raised to a few seconds.
checkpointRetryBudgetis now 3s (from 250ms); both gates are idle-cheap relative to the 60s rollup tick, so giving an older reader a few seconds to drain lets the common transient conflict clear in place. The exact-message retry gate (checkpointRetryable/checkpointBusyMessage) is unchanged, socheckpoint_retry_test.go's real-conflict reproduction still pins the diagnostic.Readiness surfaces the checkpoint separately and self-healingly. A new
checkpointreadiness check reportsdegradedonly aftercheckpointDegradeAfterFailures(6, ~1 minute at the 10s cadence) consecutive failures, and neverunhealthy. A single transient miss staysok, so one conflict does not flap the instance out of rotation; a fold that is genuinely stuck still surfaces. A wedged engine is still caught independently by the telemetrySELECT 1probe.Tests
Reconciled the previously-passing contracts that asserted a checkpoint failure increments maintenance failures, moving those assertions to checkpoint health, and kept the exact-message and caller-budget-retention tests intact. New behavioral tests (red before the fix):
TestCheckpointMissDoesNotDegradeMaintenanceHealth— a miss leaves maintenance health clean and is recorded only in checkpoint health.TestMissedCheckpointRetriesOnCompactionCycle— drivingrunParquetCompactionCycledirectly, the first cycle records the miss and the second retries and clears it (self-heal proven without a real clock).TestRetentionFailureDegradesMaintenance— a real retention/prune failure still degrades maintenance (no readiness regression for genuine maintenance failures).TestReadinessToleratesTransientCheckpointMiss—checkpointResult: never-attempted / clean / a few recent misses areok; past the threshold isdegraded; neverunhealthy.TestMaintenanceCheckpointRetryExhaustionIsExplicitAndBoundednow asserts maintenance stays clean, the miss is recorded in checkpoint health with the exact budget-exhausted and transaction-active text, and the pass is bounded by the 3s budget.Verification
just test "./internal/engine/..."andjust test "./internal/api/..."(full suites),just vet, andjust check-static(gofmt, deadcode, behavioral test names) all green. No config, schema, or migration changes.