handshake: close accepted streams — port 444 leaked a goroutine per connection - #34
Merged
Conversation
…onnection
Close appeared NOWHERE in handshake.go. Every stream accepted on port 444
was leaked, on both the timeout and the success path.
The timeout branch is the worse half:
ch := make(chan readResult, 1)
go func() {
buf := make([]byte, maxMsgSize) // 64 KiB
n, err := stream.Read(buf)
...
}()
select {
case res := <-ch: ...
case <-time.After(handshakeRecvTimeout): // 10s
slog.Warn("handshake timeout waiting for message", ...)
}
On timeout handleConnection returned without closing. The child goroutine
stayed parked in stream.Read forever — nothing could ever unblock it,
because the only thing that would is a Close and there wasn't one. Its stack
retained the 64 KiB buffer, the stream, and the underlying daemon Connection
with its RecvBuf/SendBuf channels.
Port 444 accepts from any peer. A peer that connects and sends nothing leaked
roughly 64 KiB plus a goroutine stack plus a full Connection every 10s, with
no cap and no recovery — the same shape as the dead idle-timeout that
OOM-killed the service fleet, reached through a different door.
Closing is what frees the parked reader: Close reaches CloseConnection, which
calls CloseRecvBuf, so the blocked receive returns !ok and the goroutine
exits with io.EOF.
The handshakeCloseDelay pause is why that constant exists. It was declared at
line 80 for exactly this purpose ("delay before closing after send to let
data flush") and had NO remaining reference anywhere in the package —
stranded when the close path was removed at some point. processMessage writes
its reply synchronously, so closing without the pause would truncate an
in-flight response. Each handleConnection already runs in its own goroutine
(see the accept loop), so the pause delays nothing but its own teardown.
Pre-existing unrelated failure TestSendRequestDirectFailsRelaySucceeds fails
identically with and without this change; verified against a clean stash.
Found during a codebase-wide sweep for safety mechanisms that silently fail
to engage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closing accepted streams (previous commit) made TestHandleConnection_
TimeoutBranch panic: the test double implemented
func (s *slowStream) Close() error { close(s.done); return nil }
so the test's own `defer stream.Close()` became a SECOND close once
handleConnection started closing the stream itself, and close of a closed
channel panics.
The real transport is idempotent — *Connection guards teardown with a
sync.Once (CloseRecvBuf → c.CloseOnce.Do), so double Close is safe in
production. The double simply did not model that, so it was less faithful
than the thing it stood in for. Fixed with sync.Once to match.
The alternative — dropping the test's own defer — would have been wrong: it
would leave the test unable to fail if handleConnection ever stopped closing
the stream again, which is precisely the regression this guards.
Pre-existing unrelated failure TestSendRequestDirectFailsRelaySucceeds fails
identically on origin/main; verified by running the suite on a clean checkout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closeappeared nowhere inhandshake.go. Every stream accepted on port 444 was leaked — on both the timeout and the success path.The timeout branch is the worse half
On timeout,
handleConnectionreturned without closing. The child goroutine stayed parked instream.Readforever — nothing could ever unblock it, because the only thing that would is aClose, and there wasn't one. Its stack retained the 64 KiB buffer, the stream, and the underlying daemonConnectionwith itsRecvBuf/SendBufchannels.Port 444 accepts from any peer. A peer that connects and sends nothing leaked ~64 KiB + a goroutine stack + a full
Connectionevery 10 seconds, uncapped, with no recovery.That is the same shape as the dead idle-timeout that OOM-killed the 431-agent service fleet — reached through a different door.
Why closing fixes the parked goroutine
Close→CloseConnection→CloseRecvBuf, so the blocked<-RecvBufreturns!okand the reader exits withio.EOF. Closing is the only thing that can reclaim it.The dead constant was the clue
handshakeCloseDelaywas declared at line 80 for precisely this — "delay before closing after send to let data flush" — and had no remaining reference anywhere in the package. It was stranded when the close path was removed at some point.processMessagewrites its reply synchronously, so closing without the pause would truncate an in-flight response. That's why the delay existed, and why simply adding a baredefer Close()would have been wrong.Each
handleConnectionalready runs in its own goroutine (see the accept loop), so the pause delays nothing but its own teardown.Tests
go build,go vetclean.TestSendRequestDirectFailsRelaySucceedsfails identically with and without this change — verified against a clean stash, so it's pre-existing and unrelated.Found during a codebase-wide sweep for safety mechanisms that silently fail to engage.
🤖 Generated with Claude Code