Skip to content

Close the races behind the suite's intermittent failures - #941

Merged
Inok merged 5 commits into
mainfrom
pavel/gh-916-flaky-ci-tests
Sep 14, 2026
Merged

Inok merged 5 commits into
mainfrom
pavel/gh-916-flaky-ci-tests

Conversation

@Inok

@Inok Inok commented Sep 14, 2026

Copy link
Copy Markdown
Member

Closes #918 — AI-2734
Closes #707 — AI-2370

What & why

Two intermittent CI failures and one harness race, each a test losing a race rather than a product fault. The Avalonia one needed more than claiming Dispatcher.UIThread once at assembly start: per-test isolation releases that claim and retakes it on every dispatch, so a single claim closes only the first window. One application for the assembly leaves nothing to reclaim. Git's own auto maintenance ran inside fixture repositories, writing a lock file into a tree a test was comparing and detaching a gc that outlived the TempDir it ran in.

The permission bridge's port seam goes with them: a settable delegate no production caller filled, so the behaviour worth pinning — a taken port is retried, not fatal — was reachable only by reaching into the instance under test. It is a constructor argument now, with the real probe behind it.

Where to look

AvaloniaSession keeps the assembly-level claim and changes the isolation level under it. HeadlessUnitTestSession.EnsureIsolatedApplication calls Dispatcher.ResetBeforeUnitTests, which nulls the process-global the claim lives in, and it runs per dispatch while isolation is per-test. The cost is a shared Application: nothing in this suite uses AvaloniaLocator, and each dispatch still ends draining its jobs.

GitConfigGlobalSetup pinned an empty global config — hermetic, but git's own defaults (gc.auto 6700, maintenance.auto true) stay in force through an empty file.

The ACP fake recorded a server-request reply only after resuming through a completion source that publishes continuations asynchronously, leaving the test two thread-pool hops behind the wire with a wall clock as its only synchronisation.

Verification

Three mutations, each failing the test that claims the behaviour: isolation returned to per-test; gc.auto restored to git's default; the bridge's port source left to the OS where a collision is arranged.

Suite Result
Capacitor.App.Tests.Unit 1939/1939
Capacitor.Cli.Daemon.Tests.Unit 3192/3192
Capacitor.Cli.Tests.Unit 4084/4084
Capacitor.Cli.Core.Tests.Unit 3284/3284

Zero warnings across the solution; dotnet publish -c Release clean of IL2026/IL3050 for the daemon.

#916 and #917 were fixed by #924 while this branch was open, and its versions are better than what this branch had: a scripted HttpMessageHandler needs no port at all, and spawning ping.exe directly makes the tracked process the real working-directory owner. Both commits were dropped rather than merged.

#543 is closed separately: its deterministic failures no longer reproduce on macOS under the default TMPDIR, with no test skipped to achieve it. Its production question — the Codex config guard rejecting any symlinked ancestor, so a ~/.codex symlinked into a dotfiles repo is refused — is untouched here and wants its own issue.

🤖 Generated with Claude Code

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Eliminate intermittent CI races across test suites

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Eliminates races in dispatcher ownership, process cleanup, asynchronous replies, and port reuse.
• Pins Git maintenance settings to keep fixture repositories hermetic and disposable.
• Injects loopback port selection for deterministic permission-bridge retry coverage.
Diagram

graph TD
  CI["CI test suites"] --> AV["Assembly session"] --> UI["UI dispatcher"]
  CI --> PS["Port sources"] --> PB["Permission bridge"]
  CI --> GC["Pinned Git config"] --> FR["Fixture repositories"]
  CI --> SYNC["Lifecycle synchronization"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Script transport failures in an HTTP handler
  • ➕ Eliminates real-port allocation and reuse races entirely.
  • ➕ Runs faster without stopping and restarting a WireMock server.
  • ➕ Can deterministically control the exact exception and retry sequence.
  • ➖ Requires an injectable HTTP transport seam in the tested path.
  • ➖ Exercises less of the real loopback networking stack.
  • ➖ May not validate behavior against an actual refused TCP connection.
2. Serialize affected tests
  • ➕ Provides a simpler mitigation for shared dispatcher and port resources.
  • ➕ Requires fewer production-facing abstractions.
  • ➖ Masks ownership and lifecycle defects rather than closing their race windows.
  • ➖ Slows the suites and remains vulnerable to unrelated processes.
  • ➖ Does not address detached Git maintenance or asynchronous publication ordering.

Recommendation: Keep the PR’s explicit synchronization and injected port-source design: they model the actual ownership boundaries and preserve production-path coverage. A scripted HTTP handler is worth considering specifically for the hook retry test if the transport is already injectable, but broad test serialization would only conceal the underlying races.

Files changed (31) +203 / -68

Enhancement (2) +34 / -0
EphemeralLoopbackPortSource.csProvide OS-selected loopback port candidates +24/-0

Provide OS-selected loopback port candidates

• Introduces the production port source, which briefly binds port zero through 'TcpListener' and returns the selected port for a later 'HttpListener' bind attempt.

src/Capacitor.Cli.Daemon/Services/EphemeralLoopbackPortSource.cs

ILoopbackPortSource.csDefine the permission bridge port-selection seam +10/-0

Define the permission bridge port-selection seam

• Adds an injectable interface for supplying each loopback bind candidate, allowing retries to request a fresh port and tests to arrange collisions deterministically.

src/Capacitor.Cli.Daemon/Services/ILoopbackPortSource.cs

Bug fix (6) +50 / -24
DaemonRunner.csRegister the production loopback port source +1/-0

Register the production loopback port source

• Adds the singleton 'ILoopbackPortSource' registration required by 'LocalPermissionBridge' construction and hosted-service startup.

src/Capacitor.Cli.Daemon/DaemonRunner.cs

LocalPermissionBridge.csInject port allocation into permission bridge binding +2/-17

Inject port allocation into permission bridge binding

• Replaces the mutable test-only delegate and inline TCP probe with a constructor-injected port source. Existing retry and cancellation behavior now operates through the explicit dependency.

src/Capacitor.Cli.Daemon/Services/LocalPermissionBridge.cs

AvaloniaSession.csShare one Avalonia application per assembly +5/-1

Share one Avalonia application per assembly

• Changes the headless session from per-test to per-assembly isolation so dispatches do not repeatedly release and reclaim the global UI dispatcher.

test/Capacitor.App.Tests.Unit/AvaloniaSession.cs

FakeAcpAgent.csPublish ACP response state on the reader loop +12/-4

Publish ACP response state on the reader loop

• Records server-request results and errors before completing the pending request. Tests can now observe the state immediately after request completion without waiting for an additional asynchronous continuation.

test/Capacitor.Cli.Daemon.Tests.Unit/Acp/FakeAcpAgent.cs

WorktreeManagerTests.csAwait snapshot-holder termination before cleanup +7/-1

Await snapshot-holder termination before cleanup

• Waits for the killed child process to exit before removing its working directory, preventing Windows cleanup failures while the process still holds the path.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs

HookForwardingTests.csKeep retry-test ports below ephemeral ranges +23/-1

Keep retry-test ports below ephemeral ranges

• Allocates the stopped WireMock server from a bounded non-ephemeral range so neighboring servers cannot receive the freed port during the retry window. The helper retries occupied candidates while retaining a real refused-connection scenario.

test/Capacitor.Cli.Tests.Unit/HookForwardingTests.cs

Tests (22) +102 / -39
AvaloniaSessionTests.csVerify Avalonia application and dispatcher persistence +16/-0

Verify Avalonia application and dispatcher persistence

• Adds a regression test proving consecutive dispatches use the same 'Application' and 'Dispatcher.UIThread' instances.

test/Capacitor.App.Tests.Unit/AvaloniaSessionTests.cs

AvaloniaUiThreadClaim.csClaim the Avalonia UI thread before tests run +13/-0

Claim the Avalonia UI thread before tests run

• Adds an assembly setup hook that performs the first dispatch before tests can race to create and claim the global dispatcher.

test/Capacitor.App.Tests.Unit/AvaloniaUiThreadClaim.cs

AcpHostedAgentRuntimePermissionTests.csAssert nullable ACP response presence explicitly +3/-1

Assert nullable ACP response presence explicitly

• Checks 'HasValue' before reading the nullable JSON response, producing a meaningful assertion failure when no response was recorded.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/AcpHostedAgentRuntimePermissionTests.cs

AgentActivityClockTests.csSupply the production port source in activity tests +1/-1

Supply the production port source in activity tests

• Updates direct 'LocalPermissionBridge' construction for the new injected port-source dependency.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/AgentActivityClockTests.cs

AgentOrchestratorHarness.csWire port allocation into the orchestrator harness +1/-1

Wire port allocation into the orchestrator harness

• Constructs the harness permission bridge with the production ephemeral loopback port source.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/AgentOrchestratorHarness.cs

AgentResolvedTitleTests.csUpdate resolved-title permission bridge setup +1/-1

Update resolved-title permission bridge setup

• Passes the production loopback port source when building the test fixture’s permission bridge.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/AgentResolvedTitleTests.cs

AgentStatusSnapshotTests.csUpdate status snapshot permission bridge setup +1/-1

Update status snapshot permission bridge setup

• Supplies the new loopback port dependency in the status snapshot fixture.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/AgentStatusSnapshotTests.cs

ConsentRulesPutV2Tests.csUpdate consent-rules bridge construction +1/-1

Update consent-rules bridge construction

• Adds the production port source to the consent-rules test harness.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/ConsentRulesPutV2Tests.cs

DaemonStatusIpcTests.csUpdate daemon-status IPC bridge fixtures +2/-2

Update daemon-status IPC bridge fixtures

• Passes the production port source in both daemon-status IPC harness construction paths.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/DaemonStatusIpcTests.cs

DaemonStatusWiringTests.csMirror port-source registration in daemon wiring tests +1/-0

Mirror port-source registration in daemon wiring tests

• Registers 'ILoopbackPortSource' so the test service graph matches the daemon’s production dependency injection setup.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/DaemonStatusWiringTests.cs

FakeLoopbackPortSource.csAdd deterministic loopback collision provider +16/-0

Add deterministic loopback collision provider

• Introduces a fake source that returns a chosen first port, tracks reservation attempts, and delegates later candidates to the production source.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/FakeLoopbackPortSource.cs

LaunchConsentIpcTests.csUpdate launch-consent bridge construction +1/-1

Update launch-consent bridge construction

• Supplies the production loopback port source to the launch-consent IPC harness.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LaunchConsentIpcTests.cs

LocalControlHelloTests.csUpdate local-control hello bridge construction +1/-1

Update local-control hello bridge construction

• Adds the required port source to the local-control hello test fixture.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalControlHelloTests.cs

LocalControlOpsV2PutTests.csUpdate local-control operations bridge construction +1/-1

Update local-control operations bridge construction

• Passes the production ephemeral port source into the operations test harness.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalControlOpsV2PutTests.cs

LocalControlProbeTests.csUpdate local-control probe bridge construction +1/-1

Update local-control probe bridge construction

• Provides the new loopback port dependency in probe test setup.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalControlProbeTests.cs

LocalPermissionBridgeInputWaitTests.csUpdate input-wait bridge fixture +1/-1

Update input-wait bridge fixture

• Constructs the permission bridge with the production port source while retaining its attribution and wait-state handlers.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalPermissionBridgeInputWaitTests.cs

LocalPermissionBridgeInteractiveTests.csUpdate interactive permission bridge fixture +1/-1

Update interactive permission bridge fixture

• Inserts the production port source before the broker and decision-log dependencies.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalPermissionBridgeInteractiveTests.cs

LocalPermissionBridgePolicyTests.csUpdate policy permission bridge fixture +1/-1

Update policy permission bridge fixture

• Adds the production port source to policy-oriented bridge construction.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalPermissionBridgePolicyTests.cs

LocalPermissionBridgeShutdownTests.csUpdate shutdown permission bridge fixture +1/-1

Update shutdown permission bridge fixture

• Supplies the production port source while preserving the shutdown and cancellation scenario.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalPermissionBridgeShutdownTests.cs

LocalPermissionBridgeTests.csTest bind retries through injected port sources +24/-22

Test bind retries through injected port sources

• Reworks bridge factories and DI setup for the new dependency. Collision and cancellation tests now use 'FakeLoopbackPortSource' instead of mutating an internal delegate, making retry behavior deterministic.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/LocalPermissionBridgeTests.cs

PermissionWiringTests.csRegister port allocation in permission wiring tests +1/-0

Register port allocation in permission wiring tests

• Adds the port-source singleton so the permission service graph resolves with the same dependencies as production.

test/Capacitor.Cli.Daemon.Tests.Unit/Services/PermissionWiringTests.cs

GitRepoHermeticityTests.csVerify fixture repositories disable Git maintenance +13/-1

Verify fixture repositories disable Git maintenance

• Adds assertions for disabled automatic GC, detached GC, and maintenance. It also updates the exported global-config assertion to reference the pinned configuration file.

test/Capacitor.Cli.Tests.Unit/GitRepoHermeticityTests.cs

Other (1) +17 / -5
GitConfigGlobalSetup.csDisable automatic Git work in fixture repositories +17/-5

Disable automatic Git work in fixture repositories

• Replaces the empty global Git configuration with a pinned file disabling automatic GC, detached GC, and maintenance. The exported environment continues to isolate tests from user and system configuration.

test/Capacitor.Tests.Helpers/Guards/GitConfigGlobalSetup.cs

@qodo-code-review

qodo-code-review Bot commented Sep 14, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Snapshot cleanup kills trees unsafely ✓ Resolved 📘 Rule violation ≡ Correctness
Description
BorrowedSnapshot_RefreshPreservesRunningExecutionDirectory in WorktreeManagerTests calls
holder.Kill(entireProcessTree: true) and waits only for the parent shell instead of routing
termination through ProcessTree.Kill and ensuring its spawned descendants have exited. When the
snapshot holder launches ping on Windows or sleep on POSIX, a descendant can remain alive and
retain snapshot.Path while WorktreeManager.RemoveAsync recursively deletes the snapshot.
Code

test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs[318]

+                holder.Kill(entireProcessTree: true);
Relevance

●●● Strong

Recent accepted precedent favors descendant-aware process cleanup; this directly matches the PR’s
stated snapshot deletion race.

PR-#841

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Compliance rule 3209096 requires process-tree termination to use ProcessTree.Kill, but the changed
cleanup directly invokes Process.Kill with entireProcessTree: true. The holder commands
explicitly spawn ping on Windows and sleep on POSIX, while the added WaitForExitAsync observes
only the parent shell before RemoveAsync immediately performs recursive deletion; existing
ProcessTreeTests instead wait for the root and independently verify that captured descendants
disappear.

Rule 3209096: Process tree termination must use ProcessTree.Kill with SIGKILL-only, child-first, and single-signal semantics
test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs[317-321]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs[285-300]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs[317-325]
test/Capacitor.Cli.Daemon.Tests.Unit/Services/ProcessTreeTests.cs[20-43]
src/Capacitor.Cli.Daemon/Services/WorktreeManager.cs[577-590]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Snapshot-holder cleanup directly invokes `Process.Kill(bool)` and waits only for the parent shell, bypassing the repository's required process-tree termination semantics. A spawned `ping` or `sleep` descendant may therefore still hold the snapshot working directory when recursive deletion begins.

## Fix Focus Areas
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/WorktreeManagerTests.cs[285-325]
- test/Capacitor.Cli.Daemon.Tests.Unit/Services/ProcessTreeTests.cs[20-43]

## Recommended Fix
Replace `holder.Kill(entireProcessTree: true)` with `ProcessTree.Kill(holder)` and retain a bounded `WaitForExitAsync()` for the holder. Also ensure no descendant can retain `snapshot.Path` before calling `WorktreeManager.RemoveAsync`: either capture each descendant's process identity before termination and wait for every captured identity to disappear, following the existing process-tree test approach, or make the shell block with an input-reading builtin so that it creates no descendant.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. The retry test flakes on some hosts ✗ Dismissed 🐞 Bug ☼ Reliability
Description
StartBelowTheEphemeralRange hard-codes ports 20000–29999 as outside the ephemeral range, but a
host can configure its dynamic-port range to include those ports. When another parallel test is
assigned the freed port during the delayed restart window, PostWithRetryAsync accepts that
server's HTTP response rather than retrying to the intended server.
Code

test/Capacitor.Cli.Tests.Unit/HookForwardingTests.cs[73]

+                return WireMockServer.Start(Random.Shared.Next(20000, 30000));
Relevance

●●● Strong

Recent history accepts deterministic retry-test fixes for host and scheduler races, matching this
PR’s explicit reliability intent.

PR-#360
PR-#132
PR-#375

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new helper selects only a random port from the fixed 20000–29999 interval. The test deliberately
releases that port for 600ms before restarting its intended server, while the suite has many tests
that start WireMock with an OS-assigned port; if the host's configured ephemeral range overlaps this
interval, a neighboring server can take the released port. The retry implementation returns an HTTP
response when its status is not retryable, so the neighboring server's response is treated as the
test's result rather than a transport failure.

test/Capacitor.Cli.Tests.Unit/HookForwardingTests.cs[36-58]
test/Capacitor.Cli.Tests.Unit/HookForwardingTests.cs[70-77]
src/Capacitor.Cli.Core/HttpClientExtensions.cs[270-274]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`StartBelowTheEphemeralRange` assumes 20000–29999 is always outside the OS ephemeral range. Hosts can configure a dynamic-port range that includes these ports, allowing a parallel dynamically-bound WireMock server to claim the port after `tempServer.Stop()` and before the delayed restart.

## Fix Focus Areas
- test/Capacitor.Cli.Tests.Unit/HookForwardingTests[70-77]

## Recommended Fix
Avoid relying on a hard-coded platform-default ephemeral range. Obtain a port from a test-controlled allocator that verifies it is outside the active dynamic-port range for the current host, or serialize this test with dynamically-bound WireMock tests and make the delayed restart task observable so a bind failure fails immediately.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 64 rules
✅ Cross-repo context — repo relationships
Review mode: 🧠 Deep: This PR changes production dependency wiring and retry behavior while modifying several independent concurrency, process-lifecycle, test-isolation, Git-environment, and networking paths, creating a high density of easy-to-miss defects.

Grey Divider

Tip of the day
💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread test/Capacitor.Cli.Tests.Unit/HookForwardingTests.cs Outdated
Inok and others added 4 commits September 14, 2026 18:12
Per-test isolation releases Dispatcher.UIThread and reclaims it on every
dispatch, so claiming it once before the suite starts closes only the
first window and every later dispatch reopens one. One application for
the assembly leaves nothing to reclaim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fake assigned it after resuming through a completion source that
publishes continuations asynchronously, leaving a test two thread-pool
hops behind the wire with only a wall clock to say when to give up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Auto maintenance and auto gc are on by git's defaults, which an empty
global config leaves in force: one writes .git/objects/maintenance.lock
under a tree a test is comparing, the other detaches and outlives the
TempDir holding the repository it ran in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bridge held a settable delegate no production caller filled, so the
one behaviour worth pinning — that a taken port is retried rather than
fatal — was only reachable by reaching into the instance under test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Inok
Inok force-pushed the pavel/gh-916-flaky-ci-tests branch from e0a2190 to 1f322d3 Compare September 14, 2026 16:21
@Inok
Inok requested a review from realtonyyoung September 14, 2026 16:40
Comment thread test/Capacitor.Tests.Helpers/Guards/GitConfigGlobalSetup.cs Outdated

@realtonyyoung realtonyyoung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review complete. I left one actionable inline finding. Per request, I did not build or run tests.

A raw string literal keeps whatever indentation precedes its lines, so a
tab inside a space-indented one lands as space-before-tab and
`git diff --check` reports it. Git config reads spaces the same way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Inok
Inok requested a review from realtonyyoung September 14, 2026 18:41
@realtonyyoung

Copy link
Copy Markdown
Collaborator

NO FINDINGS

@realtonyyoung realtonyyoung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static re-review complete. The prior finding is addressed. Per request, I did not build or run tests.

@Inok
Inok merged commit 5ded952 into main Sep 14, 2026
8 checks passed
@Inok
Inok deleted the pavel/gh-916-flaky-ci-tests branch September 14, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants