Skip to content

fix(tests): wait on real signals in three flaky channel, headless, and SQLite recovery tests - #2790

Merged
Aaronontheweb merged 2 commits into
devfrom
fix/flaky-channel-headless-sqlite-tests
Nov 6, 2026
Merged

Aaronontheweb merged 2 commits into
devfrom
fix/flaky-channel-headless-sqlite-tests

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes three flaky tests. Each test waited on a signal that did not prove the data it then read.

Earlier PRs fixed parts of tests 2 and 3. #2612 and #2625 fixed an earlier part of each. This PR fixes the parts that remained.

1. ChannelBotFilterOrderTests (macOS)

  • CI symptom: Slack_person_denied_by_ACL_is_dropped failed twice in a row with Expected ["user_not_allowed"], Actual [].
  • Cause: ChannelMetrics.RecordEventDropped first increments the snapshot total. It then calls _eventsDropped.Add, which runs the meter listener that records the reason. The test waited on the total and read the reasons once. It could read between the two lines. RecordEventFiltered has the same order.
  • Fix: Both assert helpers now wait on the reason and the total together with AwaitAssertAsync. The negative checks run after that wait. ChannelMetrics is unchanged.
  • Verification:
    • With a temporary Thread.Sleep(300) between the two lines in ChannelMetrics, the old test failed 6 of 6 with the CI error. The new test passed 6 of 6. The temporary change is removed.
    • 20 of 20 runs passed.

2. HeadlessStopRuleMutationTests (Windows)

  • CI symptom: The run did not stop within the 10 s StopCeiling.
  • Cause: Connect the headless client before the stop-rule timer starts #2625 moved only ConnectAsync out of the ceiling. Other cold work stayed inside it. This work includes the session create RPC, the first log file open, the first send, and the output pump. On a loaded Windows runner ConnectAsync alone took about 11 s.
  • Fix: The wait has two phases. A TaskCompletionSource completes when the fake transport sees SendMessage. The test waits on it for up to 30 s. Then the existing 10 s stop ceiling starts. Each timeout names its phase. The stop ceiling stays short, because Stryker pays it in full for each mutant that never stops.
  • Verification:
    • ./scripts/run-headless-stop-rule-mutations.sh passed. It tested 3 mutants and killed 3, as expected. The expected count is unchanged.
    • 20 of 20 runs of the test class passed. The full Netclaw.Cli.MutationTests project passed.

3. LegacyCoordinationSqliteRecoveryTests (Windows)

  • Run A symptom: A 5 s Ask for SendUserMessage timed out.
  • Run A cause: The ack waits behind the recovery persist and the admission persists. A durable SQLite write cost about 0.5 s on that runner.
  • Run A fix: The three SQLite SendUserMessage asks now use the existing ColdRecoveryCeiling. No production command is added.
  • Run B symptom: StartOneAtATimeAsync waited 30 s for the journal RecoverySuccess. Then DeleteOwnedDirectories failed with IOException: journal.db in use.
  • Run B cause: The journal stayed blocked in its first CREATE TABLE. Its connection stayed checked out. The delete then failed. The cause of the journal stall itself is not proven. The new diagnostic is meant to show it.
  • Run B fix, in RecordedJournalTestBase and ChildParkSqliteRecoveryTests:
    • Teardown waits for the actor system to terminate, with a 15 s ceiling. This covers the plugin actors and the seeders, which run in the same system. This wait runs before the pool clear and the delete.
    • Each delete attempt clears the pools first. Three attempts run, and DisposableTempDir retries between them.
    • If the wait times out, the teardown still tries the delete. It then reports the stall.
    • A start-up timeout now names the plugin (journal or snapshot store), its actor path, and the database path. StartOneAtATimeAsync takes the database path as a required parameter.
    • The 30 s ceiling is unchanged.
  • Verification:
    • A temporary 1 ms ceiling showed the new message. The temporary change is removed.
    • 20 of 20 runs of the two SQLite recovery classes passed.

Checks

  • dotnet test src/Netclaw.Actors.Tests: 7281 passed, 41 skipped, 0 failed.
  • TestKitTeardownGuard tests: 5 passed.
  • dotnet slopwatch analyze: 0 issues.
  • Add-FileHeaders.ps1 -Verify did not run, because the sandbox blocked pwsh. This PR adds no .cs file, so no header changed.

var id = new SessionId("park/sqlite");
await SqlPersistencePlugins.StartOneAtATimeAsync(this, ColdRecoveryCeiling);
await SqlPersistencePlugins.StartOneAtATimeAsync(this, ColdRecoveryCeiling,
Path.Combine(_sqliteDirectory!.Path, "journal.db"));
Comment on lines +104 to +106
=> Path.Combine(
_sqliteDirectory?.Path ?? throw new InvalidOperationException("The test SQLite directory was not configured."),
"journal.db");
…d SQLite recovery tests

ChannelBotFilterOrderTests: wait on the reason tag, not on the snapshot total.
The total changes before the meter listener records the reason.

HeadlessStopRuleMutationTests: wait in two phases. A 30 s cold start phase
ends when the fake transport sees SendMessage. The 10 s stop ceiling then
starts. A timeout names its phase.

LegacyCoordinationSqliteRecoveryTests: give the three SendUserMessage asks
the cold recovery ceiling. The ack waits behind durable SQLite writes.

RecordedJournalTestBase and ChildParkSqliteRecoveryTests: wait for the actor
system to stop before the pool clear and the delete. Clear the pools on each
delete attempt. A plugin start timeout now names the plugin and the database.
@Aaronontheweb
Aaronontheweb force-pushed the fix/flaky-channel-headless-sqlite-tests branch from bca64da to cd52875 Compare November 5, 2026 23:51
…t Slopwatch reads

The inline slopwatch-ignore comment did not suppress the rule, so the
Slopwatch check failed.
Comment on lines +343 to +345
catch (IOException) when (attempt < DeleteAttempts)
{
}
@Aaronontheweb
Aaronontheweb merged commit 78e4e38 into dev Nov 6, 2026
41 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/flaky-channel-headless-sqlite-tests branch November 6, 2026 02:01
@Aaronontheweb Aaronontheweb mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant