Skip to content

test(tunnel): wait for listener shutdown after context cancellation - #1256

Merged
skevetter merged 1 commit into
mainfrom
fix/tunnel-context-cancellation-flake
Sep 21, 2026
Merged

skevetter merged 1 commit into
mainfrom
fix/tunnel-context-cancellation-flake

Conversation

@skevetter

Copy link
Copy Markdown
Contributor

Root cause

TestLocalTunnel_ContextCancellation flaked on the macOS Build CLI Binary run on #1248 (29bd1b4) with expected connection to be refused after context cancellation, then passed on the same code ~2h earlier.

The test asserted an asynchronous effect with a fixed 100ms sleep:

  1. cancel() cancels the parent context. Propagation to the tunnel's child context is synchronous, so by the time cancel() returns, tunnelCtx.Done() is closed.
  2. But the listener is closed by a separate goroutine (<-tunnelCtx.Done(); listener.Close() in NewLocalTunnel). That goroutine must be scheduled before the port stops accepting.
  3. The test slept 100ms, then dialed. If the watcher goroutine had not run yet, the listen socket was still open, the kernel completed the handshake, DialTimeout succeeded, and the test failed.

The macOS "Build CLI Binary" job runs go test ./... -race -short (goreleaser pre-build hook): race-instrumented test binaries from many packages run in parallel on a 3-vCPU runner, where a >100ms scheduling delay for one goroutine is entirely plausible. Same code passing 2h earlier is consistent with load-dependent scheduling, not a code regression (#1248 does not touch pkg/tunnel).

Evidence

  • The failure message matches the only assertion after the fixed sleep (local_listener_test.go:189).
  • Reproduced deterministically: with the watcher goroutine artificially delayed 150ms (go build -overlay, no source change), the unmodified test fails 100% with the exact CI message. At 0ms artificial delay it passes, including 400 iterations under CPU saturation on 2 cores with -race.
  • Code inspection shows no other path that keeps the port open: a single listener, closed exactly once, and Close() is not involved in this test.

Fix

Test-side only: replace the fixed sleep with polling until the port refuses connections, bounded at 2s, matching the pattern the same file already uses in TestLocalTunnel_HealthCheckShutdown. The deadline is a failure bound, not a wait - the test passes as soon as shutdown is observed (normally immediate). No blanket retry, no inflated timeout: the scheduling assumption is removed rather than widened.

The product contract is unchanged and correct: context cancellation shuts the tunnel down promptly but asynchronously, so the test must observe the effect rather than assume a delay.

Verification

  • Fixed test passes 20/20 runs against the 150ms-delayed-watcher overlay (the condition that failed the old test 100%).
  • Fixed test passes 400/400 iterations under CPU saturation with -race.
  • Full pkg/tunnel suite passes with -race -short.
  • gofmt clean.

The test asserted an asynchronous effect with a fixed 100ms sleep: after cancel(), the listener is closed by a separate goroutine watching the context, so under -race on a loaded macOS runner the close can land after the dial and the connection succeeds. Poll until the port refuses connections (bounded at 2s) instead, matching the existing health-check shutdown test.

Signed-off-by: Samuel K <skevetter@pm.me>
@coderabbitai

coderabbitai Bot commented Sep 21, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 31 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: a50b0cff-03be-4cd3-8ae3-89e25c170f43

📥 Commits

Reviewing files that changed from the base of the PR and between 9d47372 and 72c0802.

📒 Files selected for processing (1)
  • pkg/tunnel/local_listener_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@netlify

netlify Bot commented Sep 21, 2026

Copy link
Copy Markdown

Deploy Preview for devsydev canceled.

Name Link
🔨 Latest commit 72c0802
🔍 Latest deploy log https://app.netlify.com/projects/devsydev/deploys/6ab1715609743900076a11be

@netlify

netlify Bot commented Sep 21, 2026

Copy link
Copy Markdown

Deploy Preview for images-devsy-sh canceled.

Name Link
🔨 Latest commit 72c0802
🔍 Latest deploy log https://app.netlify.com/projects/images-devsy-sh/deploys/6ab1715658cc2f000898b9d3

@skevetter

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Sep 21, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

The test stabilization is generally safe to merge, with a non-blocking timing-bound weakness in the new polling loop.

Findings

  1. P2 Dial Failure Bypasses Deadline

Summary

The PR replaces a fixed post-cancellation sleep with bounded polling for asynchronous listener shutdown.

  • Polls the listener every 10 milliseconds rather than assuming shutdown completes within 100 milliseconds.
  • Stops polling when a dial fails or reports failure after a nominal 2-second deadline.
  • Leaves production tunnel behavior unchanged.

Reviews (1) · Last reviewed commit: "test(tunnel): wait for listener shutdown..."

Comment on lines +193 to +195
if err != nil {
return
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Dial failure bypasses deadline

The loop returns on a dial error before checking the two-second deadline. If the deadline fires while DialTimeout is still running and the listener closes afterward, the test passes even though shutdown exceeded the intended bound. Check the deadline or elapsed time before treating the dial failure as success.

@skevetter

Copy link
Copy Markdown
Contributor Author

On the Greptile P2 (dial failure bypasses deadline) - non-blocking per its own summary, so explaining rather than applying, per the review workflow:

The deadline is a failure bound, not a latency contract. Normal shutdown completes in microseconds after cancel(); the 2s exists only to turn "shutdown never happens" into a test failure instead of a hang. The overshoot the finding describes is bounded by a single dial cycle: the last in-flight DialTimeout can land at most ~50ms (its own timeout, loopback RTT is sub-millisecond) past the deadline. Asserting elapsed < deadline on the success path would make the test strictly enforce 2.000s vs 2.05s, which is not a property the tunnel contract has. The loop also deliberately mirrors the existing structure of TestLocalTunnel_HealthCheckShutdown in the same file, which has the same shape.

If a strict bound is ever wanted, the fix is a three-line select/default on the deadline channel before the success return - happy to add it on request.

@skevetter
skevetter marked this pull request as ready for review September 21, 2026 19:23
@mergify

mergify Bot commented Sep 21, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@skevetter
skevetter merged commit bfbaba7 into main Sep 21, 2026
150 of 153 checks passed
@skevetter
skevetter deleted the fix/tunnel-context-cancellation-flake branch September 21, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant