H1 upstream: chunked terminator 0\r\n\r\n written twice when an H2 downstream ends the request body with an empty DATA frame (END_STREAM), poisoning upstream keep-alive connections

[songhieu]

Summary

When proxying HTTP/2 downstream → HTTP/1.1 upstream, if the client finishes the request body with an empty DATA frame carrying END_STREAM (very common: Cloudflare's HTTP/2 to Origin does this, so does curl -T - and most streaming clients that don't know body length up front), pingora writes the chunked terminating sequence 0\r\n\r\n to the upstream twice.

Strict HTTP/1.1 parsers on the upstream (e.g. uvicorn/h11, which rejects with 400 Invalid HTTP request received.) parse the duplicate terminator as the start of a bogus pipelined request. The upstream connection is then desynchronized: the stray error response is delivered to whichever request reuses the pooled keep-alive connection next, so unrelated requests intermittently receive 400s, and the connection pool churns (Upstream body is already finished. Nothing to write / Data received on idle client connection, close it warnings flood the logs).

Verified on pingora-proxy 0.6.0; by code inspection the same gap is still present on current main.

Mechanism

1. proxy_h1.rs — the downstream read path deliberately forwards a final empty chunk when end_of_body is true (the guard only skips empty chunks mid-stream):

/* It is normal to get 0 bytes because of multi-chunk ...
 * Don't write 0 bytes to the network since it will be
 * treated as the terminating chunk */
if !upstream_end_of_body && data.as_ref().is_some_and(|d| d.is_empty()) {
    return Ok(false);
}

2. proxy_h1.rs::send_body_to1 — the empty body slice is passed to write_body, then finish_body runs because end == true:

HttpTask::Body(data, end) => {
    body_done = end;
    if let Some(d) = data {
        let m = client_session.write_body(&d).await;   // <-- d is empty
        ...
    }
}
...
if body_done {
    match client_session.finish_body().await {         // <-- terminator again

3. pingora-core/src/protocols/http/v1/body.rs::do_write_chunked_body has no empty-buffer guard, so a 0-byte write is the terminator:

let chunk_size = buf.len();                            // 0
let chuck_size_buf = format!("{:X}\r\n", chunk_size);  // "0\r\n"
let mut output_buf = Bytes::from(chuck_size_buf).chain(buf).chain(&b"\r\n"[..]);
// wire: 0\r\n\r\n  == chunked terminator
self.body_mode = BM::ChunkedEncoding(written + chunk_size); // stays ChunkedEncoding

Because body_mode stays ChunkedEncoding, the subsequent finish_body() → do_finish_chunked_body() writes LAST_CHUNK (0\r\n\r\n) a second time.

Reproduction

Any ProxyHttp service with an h2-enabled TLS listener and a plain-HTTP/1.1 upstream. Point the upstream at a byte-capture sink:

nc -l 9999 > captured.txt        # upstream: capture raw bytes

# client: HTTP/2, body streamed from stdin => no content-length,
# curl ends the stream with an empty DATA frame + END_STREAM
printf '%s' '{"hello":"world"}' | \
  curl -sk --http2 -X POST -T - https://localhost:8443/test \
       -H 'Content-Type: application/json'

Captured upstream bytes (pingora 0.6.0):

POST /test HTTP/1.1
user-agent: curl/8.7.1
accept: */*
Content-Type: application/json
Transfer-Encoding: chunked
Host: localhost:8443

11
{"hello":"world"}
0

0

Note the two 0\r\n\r\n sequences. The same request sent as HTTP/1.1 chunked (-H 'Transfer-Encoding: chunked', no -T) produces a single terminator — the difference is the trailing empty DATA frame that only the h2 path delivers as Body(Some(empty), end=true).

Real-world trigger: Cloudflare HTTP/2 to Origin streams POST bodies this way, so every such request through a pingora-based origin proxy corrupts the upstream connection.

Suggested fix

Either (or both):

• send_body_to1: skip the write for empty data — finish_body() alone terminates correctly:

if let Some(d) = data {
    if !d.is_empty() {
        let m = client_session.write_body(&d).await;
        ...
    }
}

• BodyWriter::do_write_chunked_body: return early on an empty buffer so a 0-byte application write can never emit the protocol terminator.

Workaround for users

Drop the final empty chunk in ProxyHttp::request_body_filter:

async fn request_body_filter(
    &self,
    _session: &mut Session,
    body: &mut Option<Bytes>,
    end_of_stream: bool,
    _ctx: &mut Self::CTX,
) -> Result<()> {
    // Only when end_of_stream: an empty mid-stream chunk turned into None
    // would falsely signal end-of-body in proxy_h1.
    if end_of_stream && body.as_ref().is_some_and(|b| b.is_empty()) {
        *body = None;
    }
    Ok(())
}

Environment

• pingora / pingora-proxy / pingora-core 0.6.0 (boringssl feature), Linux x86_64
• Downstream: HTTP/2 over TLS (ALPN h2), e.g. Cloudflare HTTP/2-to-Origin or curl --http2
• Upstream: HTTP/1.1 cleartext (gunicorn + uvicorn worker; reproduced with raw nc byte capture)

[drcaramelsyrup]

@songhieu Looks like you're prepping an MR we can review for this?

[songhieu]

Yes — PR is up: #936. Zero-length writes are now a no-op in both chunked write paths (do_write_chunked_body and the cancel-safe poll task path), with unit tests asserting no bytes hit the wire for an empty chunk and that finish() emits a single terminator. cargo test -p pingora-core --lib protocols::http::v1::body passes (75/75).