Skip to content

fix: flush data buffered by send_data before the next write and before finish - #360

Merged
Ruben2424 merged 3 commits into
hyperium:masterfrom
vcarus:fix/flush-buffered-data-before-finish
Sep 20, 2026
Merged

Ruben2424 merged 3 commits into
hyperium:masterfrom
vcarus:fix/flush-buffered-data-before-finish

Conversation

@vcarus

@vcarus vcarus commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What happens

stream::write in h3 is send_data followed by poll_ready. That future is not cancel-safe: when it is dropped between the two (a tokio::time::timeout around RequestStream::send_data, a select! branch losing, an application deadline), the transport keeps the rest of the frame in its writing buffer.

From there two things go wrong:

  1. The connection dies. The next send_data on that stream finds writing.is_some(). h3-quinn treats that as a misuse of the trait (// This can only happen if the traits are misused by h3 itself) and returns ConnectionErrorIncoming::InternalError, which handle_quic_stream_error turns into closing the whole QUIC connection with H3_INTERNAL_ERROR. Every other request multiplexed on that connection is gone. On the first request stream of a connection a plain finish() is enough to hit this, because finish writes a grease frame first; on later streams any further send_data or send_trailers does it.
  2. The peer sees a truncated frame. h3-quinn's poll_finish calls quinn::SendStream::finish() without flushing writing, so FIN goes out while the tail of the last DATA frame is still buffered. This one does not even need a cancellation: send_data + poll_finish with no poll_ready in between already loses the data.

Reproduction for (1), as in the new test: send a response body larger than the peer's stream receive window, poll the send_data future once, drop it, then call finish(). Before this change the server gets ConnectionError(Remote(InternalError("internal error in the http stack"))) and the client sees ApplicationClose(H3_INTERNAL_ERROR).

The change

  • h3/src/stream.rs: write polls poll_ready before send_data as well as after. The leading poll is a no-op when nothing is buffered and otherwise flushes the leftover of a cancelled write, so send_data is never called on a stream that is not ready.
  • h3-quinn/src/lib.rs: poll_finish flushes via poll_ready before finish(). This matches what s2n-quic-h3's implementation of the same trait already does (ready!(self.poll_ready(cx))?; self.stream.finish()); the two send_data guards are word-for-word identical, h3-quinn just lacked the flushing half.

Two regression tests, one per change; each fails without its change and passes with it (finish_after_cancelled_send_data_keeps_connection_open in tests/request.rs, h3_quinn_finish_flushes_buffered_data in tests/connection.rs). cargo test passes for the three feature sets CI runs.

Relation to #78 / #81

#81 swapped the order to poll_ready then send_data and dropped the trailing poll_ready; that stalled transfers because h3-quinn only writes on poll_ready, so data sat in the buffer until the next call. This PR keeps the trailing poll_ready, so a write still completes within the call, and only adds the leading one. It does not change the trait contract or any public API.

Found while running a MASQUE (CONNECT-UDP) proxy on h3, where a client that stops reading one tunnel used to take down every other tunnel on the connection.

vcarus and others added 2 commits August 20, 2026 20:33
…e finish

h3's `stream::write` is `send_data` followed by `poll_ready`. When that future is dropped between the two, for example by `tokio::time::timeout` or `select!`, the transport keeps the rest of the frame in its buffer. The next `send_data` on that stream then finds the stream not ready; h3-quinn treats that as a misuse of the trait and answers with `ConnectionErrorIncoming::InternalError`, which closes the whole QUIC connection with H3_INTERNAL_ERROR. On the first request stream of a connection a plain `finish()` is enough to get there, because it writes a grease frame first.

Independently, h3-quinn's `poll_finish` sent FIN without flushing that buffer, so the peer saw a truncated frame.

`stream::write` now polls `poll_ready` before `send_data` (a no-op when nothing is buffered) and keeps the trailing `poll_ready`, so a write still completes within the call. h3-quinn's `poll_finish` flushes before `finish`, matching s2n-quic-h3's implementation of the same trait.

Both paths get a regression test that fails without the corresponding change.

@Ruben2424 Ruben2424 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch, thanks.

@Ruben2424
Ruben2424 merged commit 14a1422 into hyperium:master Sep 20, 2026
16 checks passed
@vcarus
vcarus deleted the fix/flush-buffered-data-before-finish branch September 20, 2026 21:27
xenorune added a commit to runewarp/runewarp that referenced this pull request Oct 24, 2026
## What

- **Receive slots:** adds the receive-slot Visitor carrier to the
`tunnel_h3` module. It is still not wired into the Client or Server
runtimes.
- *Client:* after the event-stream gate, it keeps up to 32
plain-`CONNECT` receive slots open per Tunnel connection. Each slot
holds a permit from the Client-instance stream budget. A slot is
replenished whenever one is assigned or fails, and assigned slots arrive
through `accept_tunnel_stream`.
- *Server:* queues slots up to the same cap and rejects extras with
`H3_REQUEST_REJECTED`. `open_tunnel_stream(deadline)` assigns the oldest
slot by completing its `200` response at once; the caller then writes
the Tunnel framing and ClientHello, so assignment needs no Client round
trip. GOAWAY rejects idle slots, and graceful shutdown waits for open
Tunnel streams.
- **Byte streams:** `TunnelStream` implements `AsyncRead` + `AsyncWrite`
over an assigned slot, with `into_split`.
  - FIN maps to half-close in each direction.
- Dropping a write half before `shutdown` resets the stream, so an
aborted Visitor never looks like a clean end. Quinn would otherwise
finish a dropped send stream.
- **Glossary:** adds the "Receive slot" term to `CONTEXT.md`, and the
roadmap is updated.

## Why

This is PR2a of the HTTP/3 Tunnel migration. Ordinary HTTP/3 forbids
Server-opened request streams (#224), so Visitors arrive on
Client-preopened receive slots, as chosen in
[#246](#246) (plain
`CONNECT`). Doing the carrier inside the module first keeps the next PR,
which switches the runtimes and deletes `runewarp/1`, mostly about
wiring.

## How

- **Budgets:**
- 32 idle slots per connection, chosen to bound the Server's idle-stream
cost (up to 4,096 connections) while allowing about 32 placements per
round trip per connection;
- the Server admits 1,024 slot streams plus 8 signaling streams per
connection, matching the Client-instance handler budget that no single
connection can exceed;
- `open_tunnel_stream` takes the caller's deadline; the runtime will
pass today's 5-second stream-open deadline.
- **Stale slots:** a peer STOP_SENDING is only seen on write
(hyperium/h3#350), so a slot the Client abandoned fails its `200` and
the next slot is tried within the same deadline.
- **Writes:** `h3` exposes only async sends, so each write moves the
send half into one boxed future. It is started immediately and always
run to completion, because cancelling an `h3` write corrupts the stream
(hyperium/h3#360). As with a buffered writer, callers flush before
waiting on the peer.
- **Connection lifetime:** `h3` closes a client connection when its last
`SendRequest` drops. `ClientTunnelH3` therefore keeps one for its
lifetime, so the slot pool and event follower winding down after GOAWAY
can't end Visitors already placed. The shutdown test found this.
- **Refusal handling:** a Client slot that fails, or a Server that keeps
rejecting, pauses opening for 1 second instead of busy-looping.
- **Tests:** loopback tests over real Quinn/rustls mTLS cover:
  - assignment with no Client round trip, and half-close both ways;
  - the budget bounding slots, and released streams replenishing them;
  - the Client idle cap, and Server rejection beyond it;
  - an abandoned slot being skipped;
  - dropping a half resetting only that stream;
  - an 8 MiB transfer each way with a slow reader;
  - GOAWAY rejecting idle slots while an open Tunnel stream finishes;
  - a closed connection opening nothing.

  The module's tests pass in 12 out of 12 repeated runs.
- **Deferred:** runtime wiring, deleting `runewarp/1`, protocol and
architecture docs, ADR 0005 and the CHANGELOG all come in PR2b.
Withdrawal is PR3.

🤖 Written by @dtcristo's agent (Claude Opus 5.5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants