Repository navigation
Flaky under load: run note tests, session temp-dir races, workspace and provider timing #1441
Description
These tests fail intermittently under load on santos/dev2. They also fail on a clean trunk, so they are not caused by the changes they were seen beside. Each was rerun alone to confirm.
| Test | Package | Seen | Rate on clean trunk |
|---|---|---|---|
TestNotesBeyondTheBoundWaitForTheNextBoundary, TestANoteIsHandedToAWorkerOnceAndTheScreenStillReadsIt, TestANoteDoesNotChangeWhatATaskWasAskedFor |
internal/run | "stopped at its step cap after 9 steps" | about half the runs at load 27 to 35 on 20 cores (3 of 5 alone) |
TestTwoWorkersLiveAtOnceEachGetOnlyItsOwnNote |
internal/run | note routing | 4 of 20 |
TestARunsRowNamesTheStoreTaskThePlanReadAnswersUnder |
internal/session | temp directory cleanup race | 3 of 20 |
TestANewDirectionArrivesWithTheClaimUnspent |
internal/session | 1 of 10 | |
TestEightyConcurrentAddsFromSeparateHandlesAllLand |
internal/workspace | only under load; passes alone | |
TestAWireBelowItsPaceWithNothingToSpendSaysSoAndIsSpent |
internal/provider | failed in 2 full runs; passed alone 3 of 3 |
The internal/run note tests are the costly ones: they turn a full make check red about half the time, so a green check on trunk is partly luck.
Suggested order: the run note tests first (a step cap reached under a slow scheduler reads as a real failure), then the two temp-directory races.
Drafted with CodeAF · reviewed and owned by the author
Activity
AbirAbbas commentedon Sep 24, 2026
Rechecked #1410 at
008363c98on 2026-09-24. The run worker's step handshake removes thestopped at its step cap after 9 stepscause from the note tests: 3,000/3,000 passes under-cpu 1,2,8 -count=1000at low load. Under load 10–19, 3 of 600 runs instead failed at step 1 withthe run's wall stopped the worker. Their 10-second test-harness wall elapsed while the test's own store write was in SQLitefsync, before the first tool-end handshake. A separate capped-step finish incorrectly reported as the cap remains and is being fixed on #1410.santoshkumarradha commentedon Sep 25, 2026
More tests that fail only under load, seen on santos/dev2 during full
make checkruns on 2026-09-24. Each one's package code is identical to trunk.TestPlandbCliConcurrentWritersLandEveryTaskTestTheLosingArmsRowSaysItIsExhaustAndNotAFailureTestS3ItComesBackTestAJudgedWideTaskWhoseFirstPartsAreRefusedStillGainsTheDivideVerbTestAgreementDoesNotBuyASecondCompletionsantoshkumarradha commentedon Sep 25, 2026
One more, seen 2026-09-25 in a full
make checkon #1485's branch:TestTheVisionGateReadsTheDiskCacheWhileTheCatalogIsWarming(cmd/codeaf) failed once. It passed 10 of 10 alone on that branch and 10 of 10 on clean dev.