You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
flaky: TestCompactionAndRetentionUnderConcurrentReads loses rows under -race on repeated runs #499
TestCompactionAndRetentionUnderConcurrentReads (internal/engine/retirement_test.go:103) fails intermittently under the race detector: a concurrent read sees fewer rows than were seeded.
Symptoms
Under -race with more than one iteration it fails most runs, with messages like:
retirement_test.go:103: windowed read saw spans=73 logs=73 metrics=73, want 80 each
retirement_test.go:103: unwindowed read saw 76 spans, want 80 or 81
Observed counts range ~73–77 against an expected 80 (the unwindowed assertion already tolerates 80 or 81, so the shortfall is well outside the intended tolerance, not an off-by-one).
Reproduction
just test "./internal/engine/ -run TestCompactionAndRetentionUnderConcurrentReads -race -count=5"
Fails on multiple runs. CI runs this check at -count=1, where it usually passes, which is why it only occasionally reddens a PR's "Go race (engine)" check rather than failing every time.
Not a recent regression
It reproduces on 77fa266 (the #488 merge) as well as current main, so it predates the recent engine-audit fixes (#491 OOM backoff, #494 checkpoint readiness) and the in-flight read-path PRs. It is a property of the test under race load, not of those changes.
Hypotheses to check
The concurrent reader may race the publication of the seeded batches: if the read fires before all 80 rows are durably visible, it legitimately sees fewer, and the assertion's fixed expectation (80, or 80 or 81) is too tight for the concurrency the test itself sets up. The race detector widens the window, so it shows far more often under -race -count>1.
Alternatively, a read could be losing committed rows while compaction retires inputs — which would be a real pin/epoch visibility bug. The crash-matrix and batch-cache tests pass, so this is less likely, but the test should be made to distinguish the two: synchronize "all seeds published" before asserting, or assert against the actual published count rather than a constant.
Suggested fix direction
Make the test deterministic about how many rows are guaranteed visible at the moment it reads (wait for the publication horizon to cover all seeds, or read the published count and assert equality to that), so a genuine row-loss regression would still fail while concurrency timing does not. If investigation shows real row loss during retire, treat it as a correctness bug instead.
Found while running the full race suite against an engine-audit read-path PR; the failure is pre-existing and unrelated to that PR.
Fixed in #500 (commit ff28b4f). Root cause was PublishReplacement publishing the catalog snapshot once per batch during a compaction swap, so a lock-free reader could capture an intermediate catalog that had removed a compaction group's inputs but not yet installed the output, under-counting conserved rows. The swap now mutates all inputs and the output under mu and publishes the reader snapshot exactly once, so readers only ever observe the pre-swap or post-swap catalog.
Verified with TestCompactionAndRetentionUnderConcurrentReads under -race: reproduced 6/10 failures before the change, 20/20 pass after, no data race.
Activity
vishr commentedon Nov 16, 2026
Fixed in #500 (commit ff28b4f). Root cause was
PublishReplacementpublishing the catalog snapshot once per batch during a compaction swap, so a lock-free reader could capture an intermediate catalog that had removed a compaction group's inputs but not yet installed the output, under-counting conserved rows. The swap now mutates all inputs and the output undermuand publishes the reader snapshot exactly once, so readers only ever observe the pre-swap or post-swap catalog.Verified with
TestCompactionAndRetentionUnderConcurrentReadsunder-race: reproduced 6/10 failures before the change, 20/20 pass after, no data race.