Skip to content

test(flaky): silence the FSWatcher echo and stop the sizing waits from giving up quietly - #43

Merged
rducom merged 1 commit into
mainfrom
fix/flaky-fsevents-echo-and-silent-waits
Aug 16, 2026
Merged

rducom merged 1 commit into
mainfrom
fix/flaky-fsevents-echo-and-silent-waits

Conversation

@rducom

@rducom rducom commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Two unrelated flakes surfaced in the same CI run (31976426203).

1. denseBurstCoalescesToItsClusterAncestor — reproduced

✘ IncrementalRefreshTests.swift:317: Expectation failed: (c.autoRestatCount → 2) == 1

The suite's premise — "forge FSEvents batches directly into handleDiskChanges — no FSEvents latency" — was never true. Self.scanned(root) calls scan(url:), which arms a real FSWatcher (1 s latency) on the fixture. The tests write real files, so the watcher echoes their own writes back a second later. That second batch re-marks paths the cycle already reconciled and lands as an extra re-stat or subtree pass.

Idempotent for the tree — which is why waitUntil still exits and the aggregate assertions pass — but fatal for anything that counts the passes. On a loaded runner a reconcile cycle (20 directories, parallel) outlives the watcher's latency, so the echo arrives while the test is still waiting.

The suite half-knew this already: preciseEventLeadsToRestatOnRefresh carried an NB: dropping a dirtyPaths.isEmpty assert for exactly this reason. The workaround was never generalised.

Fix. ScanController gains a liveWatchingSuspended flag — a flag rather than a bare stopWatching(), because a rescan re-arms the stream — and the scanned() helper pins the watcher off so the forged batches are the only input. The dropped assert comes back.

2. toggleAllCyclesTriState — diagnosed, not reproduced

✘ CleanupTests.swift:343: Expectation failed: (c.selectedRows.count → 1) == 2
✘ CleanupTests.swift:346: Expectation failed: (c.selectAllState → .all) == .some
… 3 more, the whole tri-state cycle shifted by one

Never fell locally in 40 loaded runs, so this one is a diagnostic rather than a reproduction — worth flagging when reviewing.

All five failures are consistent with a single cause: the test-cache-2 row was still .pending, so not bulk-selectable, so every step of the cycle shifts. Alternatives ruled out — blockedReason and detect are deterministic for that id, and a sized(0) would need the scanner to miss a 4 KiB file.

The culprit is waitForReady. It polls RunLoop.main.run(until:), and all 26 parallel @MainActor suites pump that same shared run loop, so each round costs far more than its 20 ms slice: the 5 s budget went to mutual blocking, not to work. Then it gave up in silence — which is what let a timeout masquerade as five incoherent assertions further down.

Fix. The deadline exists only to stop a hang, so it is now generous (60 s); more importantly both waits now #expect their condition, reporting timed out waiting for .ready, still sizing at the call site instead of failing obscurely downstream.

Verification

Full suite, 20 iterations, under 28 spinners on 14 cores:

before after
IncrementalRefreshTests:317 10 / 20 failed 0 / 20

SwiftLint --strict clean.

Out of scope

StructuralIntegrityTests.swift:33 also fell once in 20 loaded runs — not in the CI run above, and untouched here. The single await Task.yield() at line 29 assumes the child task reaches its first suspension in one scheduler turn; under load it sometimes runs to completion, ok is true and the file is really deleted. It needs an observable rendezvous point, so it belongs in its own change. Its 0/20 here proves nothing at a 1-in-20 rate.

…m giving up quietly

Two unrelated flakes in the same CI run (run 31976426203).

denseBurstCoalescesToItsClusterAncestor asserts exactly which reconcile
decision one forged FSEvents batch produces. But the suite's premise —
"no FSEvents latency" — was never true: scan(url:) arms a real FSWatcher
on the fixture, and it echoes the test's own writes back a second later.
The extra batch re-marks paths the cycle already reconciled, landing as
an extra re-stat. Idempotent for the tree, fatal for a counter. On a
loaded runner a reconcile cycle outlives the watcher's latency, so the
echo arrives while the test is still waiting.

The suite half-knew this: preciseEventLeadsToRestatOnRefresh carried a
NB dropping a dirtyPaths assert for exactly this reason. Generalise it
instead — ScanController gains a suspend flag (not a bare stopWatching,
which a rescan would undo) and the scanned() helper pins the watcher off,
so the forged batches are the only input. The dropped assert comes back.

Reproduced under CPU load: 10 failures in 20 runs before, 0 in 20 after.

toggleAllCyclesTriState is the second, and the fix is diagnostic rather
than reproduced — it never fell locally in 40 loaded runs. Its five
failures are consistent with one cause: a row still .pending reads
exactly like a row that was never there, and the whole tri-state cycle
shifts by one. waitForReady polls RunLoop.main, which every parallel
@mainactor suite pumps too, so a round costs far more than its 20 ms
slice and 5 s went to mutual blocking rather than to work. The deadline
only exists to stop a hang, so it is now generous — but the real defect
was expiring in silence, which let a timeout masquerade as five bogus
assertions downstream. Both waits now report it.
@rducom
rducom merged commit 23f0fef into main Aug 16, 2026
6 checks passed
@rducom
rducom deleted the fix/flaky-fsevents-echo-and-silent-waits branch August 16, 2026 22:54
@rducom rducom added the bug Something isn't working label Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant