Repository navigation
fix: flush data buffered by send_data before the next write and before finish - #360
Merged
Ruben2424 merged 3 commits intoSep 20, 2026
Merged
Conversation
…e finish h3's `stream::write` is `send_data` followed by `poll_ready`. When that future is dropped between the two, for example by `tokio::time::timeout` or `select!`, the transport keeps the rest of the frame in its buffer. The next `send_data` on that stream then finds the stream not ready; h3-quinn treats that as a misuse of the trait and answers with `ConnectionErrorIncoming::InternalError`, which closes the whole QUIC connection with H3_INTERNAL_ERROR. On the first request stream of a connection a plain `finish()` is enough to get there, because it writes a grease frame first. Independently, h3-quinn's `poll_finish` sent FIN without flushing that buffer, so the peer saw a truncated frame. `stream::write` now polls `poll_ready` before `send_data` (a no-op when nothing is buffered) and keeps the trailing `poll_ready`, so a write still completes within the call. h3-quinn's `poll_finish` flushes before `finish`, matching s2n-quic-h3's implementation of the same trait. Both paths get a regression test that fails without the corresponding change.
This was referenced Oct 24, 2026
xenorune
added a commit
to runewarp/runewarp
that referenced
this pull request
Oct 24, 2026
## What - **Receive slots:** adds the receive-slot Visitor carrier to the `tunnel_h3` module. It is still not wired into the Client or Server runtimes. - *Client:* after the event-stream gate, it keeps up to 32 plain-`CONNECT` receive slots open per Tunnel connection. Each slot holds a permit from the Client-instance stream budget. A slot is replenished whenever one is assigned or fails, and assigned slots arrive through `accept_tunnel_stream`. - *Server:* queues slots up to the same cap and rejects extras with `H3_REQUEST_REJECTED`. `open_tunnel_stream(deadline)` assigns the oldest slot by completing its `200` response at once; the caller then writes the Tunnel framing and ClientHello, so assignment needs no Client round trip. GOAWAY rejects idle slots, and graceful shutdown waits for open Tunnel streams. - **Byte streams:** `TunnelStream` implements `AsyncRead` + `AsyncWrite` over an assigned slot, with `into_split`. - FIN maps to half-close in each direction. - Dropping a write half before `shutdown` resets the stream, so an aborted Visitor never looks like a clean end. Quinn would otherwise finish a dropped send stream. - **Glossary:** adds the "Receive slot" term to `CONTEXT.md`, and the roadmap is updated. ## Why This is PR2a of the HTTP/3 Tunnel migration. Ordinary HTTP/3 forbids Server-opened request streams (#224), so Visitors arrive on Client-preopened receive slots, as chosen in [#246](#246) (plain `CONNECT`). Doing the carrier inside the module first keeps the next PR, which switches the runtimes and deletes `runewarp/1`, mostly about wiring. ## How - **Budgets:** - 32 idle slots per connection, chosen to bound the Server's idle-stream cost (up to 4,096 connections) while allowing about 32 placements per round trip per connection; - the Server admits 1,024 slot streams plus 8 signaling streams per connection, matching the Client-instance handler budget that no single connection can exceed; - `open_tunnel_stream` takes the caller's deadline; the runtime will pass today's 5-second stream-open deadline. - **Stale slots:** a peer STOP_SENDING is only seen on write (hyperium/h3#350), so a slot the Client abandoned fails its `200` and the next slot is tried within the same deadline. - **Writes:** `h3` exposes only async sends, so each write moves the send half into one boxed future. It is started immediately and always run to completion, because cancelling an `h3` write corrupts the stream (hyperium/h3#360). As with a buffered writer, callers flush before waiting on the peer. - **Connection lifetime:** `h3` closes a client connection when its last `SendRequest` drops. `ClientTunnelH3` therefore keeps one for its lifetime, so the slot pool and event follower winding down after GOAWAY can't end Visitors already placed. The shutdown test found this. - **Refusal handling:** a Client slot that fails, or a Server that keeps rejecting, pauses opening for 1 second instead of busy-looping. - **Tests:** loopback tests over real Quinn/rustls mTLS cover: - assignment with no Client round trip, and half-close both ways; - the budget bounding slots, and released streams replenishing them; - the Client idle cap, and Server rejection beyond it; - an abandoned slot being skipped; - dropping a half resetting only that stream; - an 8 MiB transfer each way with a slow reader; - GOAWAY rejecting idle slots while an open Tunnel stream finishes; - a closed connection opening nothing. The module's tests pass in 12 out of 12 repeated runs. - **Deferred:** runtime wiring, deleting `runewarp/1`, protocol and architecture docs, ADR 0005 and the CHANGELOG all come in PR2b. Withdrawal is PR3. 🤖 Written by @dtcristo's agent (Claude Opus 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
stream::writein h3 issend_datafollowed bypoll_ready. That future is not cancel-safe: when it is dropped between the two (atokio::time::timeoutaroundRequestStream::send_data, aselect!branch losing, an application deadline), the transport keeps the rest of the frame in itswritingbuffer.From there two things go wrong:
send_dataon that stream findswriting.is_some(). h3-quinn treats that as a misuse of the trait (// This can only happen if the traits are misused by h3 itself) and returnsConnectionErrorIncoming::InternalError, whichhandle_quic_stream_errorturns into closing the whole QUIC connection withH3_INTERNAL_ERROR. Every other request multiplexed on that connection is gone. On the first request stream of a connection a plainfinish()is enough to hit this, becausefinishwrites a grease frame first; on later streams any furthersend_dataorsend_trailersdoes it.poll_finishcallsquinn::SendStream::finish()without flushingwriting, so FIN goes out while the tail of the last DATA frame is still buffered. This one does not even need a cancellation:send_data+poll_finishwith nopoll_readyin between already loses the data.Reproduction for (1), as in the new test: send a response body larger than the peer's stream receive window, poll the
send_datafuture once, drop it, then callfinish(). Before this change the server getsConnectionError(Remote(InternalError("internal error in the http stack")))and the client seesApplicationClose(H3_INTERNAL_ERROR).The change
h3/src/stream.rs:writepollspoll_readybeforesend_dataas well as after. The leading poll is a no-op when nothing is buffered and otherwise flushes the leftover of a cancelled write, sosend_datais never called on a stream that is not ready.h3-quinn/src/lib.rs:poll_finishflushes viapoll_readybeforefinish(). This matches what s2n-quic-h3's implementation of the same trait already does (ready!(self.poll_ready(cx))?; self.stream.finish()); the twosend_dataguards are word-for-word identical, h3-quinn just lacked the flushing half.Two regression tests, one per change; each fails without its change and passes with it (
finish_after_cancelled_send_data_keeps_connection_openintests/request.rs,h3_quinn_finish_flushes_buffered_dataintests/connection.rs).cargo testpasses for the three feature sets CI runs.Relation to #78 / #81
#81 swapped the order to
poll_readythensend_dataand dropped the trailingpoll_ready; that stalled transfers because h3-quinn only writes onpoll_ready, so data sat in the buffer until the next call. This PR keeps the trailingpoll_ready, so a write still completes within the call, and only adds the leading one. It does not change the trait contract or any public API.Found while running a MASQUE (CONNECT-UDP) proxy on h3, where a client that stops reading one tunnel used to take down every other tunnel on the connection.