test(flaky): silence the FSWatcher echo and stop the sizing waits from giving up quietly - #43
Merged
Merged
Conversation
…m giving up quietly Two unrelated flakes in the same CI run (run 31976426203). denseBurstCoalescesToItsClusterAncestor asserts exactly which reconcile decision one forged FSEvents batch produces. But the suite's premise — "no FSEvents latency" — was never true: scan(url:) arms a real FSWatcher on the fixture, and it echoes the test's own writes back a second later. The extra batch re-marks paths the cycle already reconciled, landing as an extra re-stat. Idempotent for the tree, fatal for a counter. On a loaded runner a reconcile cycle outlives the watcher's latency, so the echo arrives while the test is still waiting. The suite half-knew this: preciseEventLeadsToRestatOnRefresh carried a NB dropping a dirtyPaths assert for exactly this reason. Generalise it instead — ScanController gains a suspend flag (not a bare stopWatching, which a rescan would undo) and the scanned() helper pins the watcher off, so the forged batches are the only input. The dropped assert comes back. Reproduced under CPU load: 10 failures in 20 runs before, 0 in 20 after. toggleAllCyclesTriState is the second, and the fix is diagnostic rather than reproduced — it never fell locally in 40 loaded runs. Its five failures are consistent with one cause: a row still .pending reads exactly like a row that was never there, and the whole tri-state cycle shifts by one. waitForReady polls RunLoop.main, which every parallel @mainactor suite pumps too, so a round costs far more than its 20 ms slice and 5 s went to mutual blocking rather than to work. The deadline only exists to stop a hang, so it is now generous — but the real defect was expiring in silence, which let a timeout masquerade as five bogus assertions downstream. Both waits now report it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two unrelated flakes surfaced in the same CI run (31976426203).
1.
denseBurstCoalescesToItsClusterAncestor— reproducedThe suite's premise — "forge FSEvents batches directly into
handleDiskChanges— no FSEvents latency" — was never true.Self.scanned(root)callsscan(url:), which arms a realFSWatcher(1 s latency) on the fixture. The tests write real files, so the watcher echoes their own writes back a second later. That second batch re-marks paths the cycle already reconciled and lands as an extra re-stat or subtree pass.Idempotent for the tree — which is why
waitUntilstill exits and the aggregate assertions pass — but fatal for anything that counts the passes. On a loaded runner a reconcile cycle (20 directories, parallel) outlives the watcher's latency, so the echo arrives while the test is still waiting.The suite half-knew this already:
preciseEventLeadsToRestatOnRefreshcarried anNB:dropping adirtyPaths.isEmptyassert for exactly this reason. The workaround was never generalised.Fix.
ScanControllergains aliveWatchingSuspendedflag — a flag rather than a barestopWatching(), because a rescan re-arms the stream — and thescanned()helper pins the watcher off so the forged batches are the only input. The dropped assert comes back.2.
toggleAllCyclesTriState— diagnosed, not reproducedNever fell locally in 40 loaded runs, so this one is a diagnostic rather than a reproduction — worth flagging when reviewing.
All five failures are consistent with a single cause: the
test-cache-2row was still.pending, so not bulk-selectable, so every step of the cycle shifts. Alternatives ruled out —blockedReasonanddetectare deterministic for that id, and asized(0)would need the scanner to miss a 4 KiB file.The culprit is
waitForReady. It pollsRunLoop.main.run(until:), and all 26 parallel@MainActorsuites pump that same shared run loop, so each round costs far more than its 20 ms slice: the 5 s budget went to mutual blocking, not to work. Then it gave up in silence — which is what let a timeout masquerade as five incoherent assertions further down.Fix. The deadline exists only to stop a hang, so it is now generous (60 s); more importantly both waits now
#expecttheir condition, reportingtimed out waiting for .ready, still sizingat the call site instead of failing obscurely downstream.Verification
Full suite, 20 iterations, under 28 spinners on 14 cores:
IncrementalRefreshTests:317SwiftLint
--strictclean.Out of scope
StructuralIntegrityTests.swift:33also fell once in 20 loaded runs — not in the CI run above, and untouched here. The singleawait Task.yield()at line 29 assumes the child task reaches its first suspension in one scheduler turn; under load it sometimes runs to completion,okistrueand the file is really deleted. It needs an observable rendezvous point, so it belongs in its own change. Its 0/20 here proves nothing at a 1-in-20 rate.