[Bug]: reattached stdout stream stalls after auto-resume (regression in python-sdk 2.29.1)

[nseniak]

Summary

After a sandbox auto-pauses and resumes, reattaching to an existing
long-lived background process via commands.connect(pid) produces a
stdout stream that intermittently delivers its first response and then
goes silent — the on_stdout callback stops firing for subsequent
responses even though the process is still running and still writing.

This is a regression introduced in python-sdk 2.29.1 and present in
every release since (through 2.34.0). The likely change is cb061d2
("Fix several command and PTY streaming issues"), which reworked
incremental stdout/stderr decoding and stream-connection teardown.

Bisect (strong evidence)

Identical workload, same account/templates; only the SDK version changed.
The workload is a real stdio server (@modelcontextprotocol/server-everything
over npx) driven through repeated pause → resume → commands.connect(pid)
reattach cycles. "Stall rate" = fraction of reattach cycles where the
reconnected stdout stream failed to deliver a complete response, over 6
cycles:

Version	Stall rate
2.20.2	1/6
2.21.0	1/6
2.27.0	1/6
2.29.0	1/6 ← last good
2.29.1	6/6 ← first bad
2.30.0	6/6
2.34.0	6/6

The transition is sharp and clean: every version up to and including
2.29.0 stalls ~1/6; every version from 2.29.1 on stalls ~6/6. The only
command/stream change in 2.29.1 is cb061d2.

Reproduction conditions

The stall reproduces reliably only through a realistic streaming
workload — a long-lived process reattached after resume that answers
several requests with sizeable, multi-chunk stdout responses (in
our case JSON-RPC tools/list / resources/list / prompts/list from an
stdio MCP server). The pattern observed: the first response after reattach
arrives, later ones never do.

A trivial synthetic process (one small line per second, or a single large
one-shot write) does not isolate the regression — those either stream
fine on both versions, or hit an unrelated single-write truncation that is
version-independent. So the reliable repro is the real stdio-server flow
above; we can share the exact harness on request.

Expected vs actual

• Expected (≤ 2.29.0): after reattach, on_stdout keeps firing for
  every subsequent response; a full multi-request refresh completes.
• Actual (≥ 2.29.1): after reattach, the stream delivers the first
  response (sometimes partially) then goes silent; subsequent responses
  never arrive, while a fresh reconnect shows the process still alive.

Impact

Any workload relying on a long-lived background process surviving
auto-pause and being reattached after resume (e.g. an stdio server behind
a gateway) loses output on nearly every resume from 2.29.1 on, forcing a
full fresh-sandbox rebuild as a workaround. On 2.29.0 this was rare
(~1/6); from 2.29.1 it is effectively every resume.

Environment

• python-sdk 2.29.1 first bad, confirmed through 2.34.0; 2.29.0 last good
• AsyncSandbox.create(lifecycle={"on_timeout": "pause", "auto_resume": True})
• Reattach via AsyncSandbox.connect(sandbox_id=...) + commands.connect(pid=...)
• Suspected change: cb061d2 (incremental stdout/stderr decode + stream teardown)

[mishushakov]

Thanks for reporting, I think we have found the root cause of the issue with Fable:

httpcore itself never cancels a stream on early response close — it only
stops reading it. On the shared envd connection an abandoned stream then
stays attached server-side and blocks envd's process output fan-out for
every other consumer once its flow-control window fills

We are in process migrating to the official version of Python connect client that should land soon:
#1558

[mishushakov]

Thanks for the excellent bisect — it made this much easier to chase down. I reproduced the stall with a harness modeled on your workload (long-lived stdio responder, multi-KB responses, repeated pause → resume → commands.connect(pid) cycles) and traced it end to end. Root cause below.

TL;DR

The stall is server-side in envd, not in the SDK's stream handling: one dead stream consumer wedges envd's process-output fan-out for every consumer of that process. The 2.29.1 SDK change didn't introduce the bug — it removed an accidental client-side cleanup that had been masking it. Fixes:

• fix(envd): kick stalled process-output subscribers instead of wedging the fan-out infra#3362 — the real fix (envd stops letting one stuck consumer block everyone). Takes effect as it rolls out; newly created sandboxes get the fixed envd.
• fix(python-sdk): cancel HTTP/2 server streams closed before the server ends them #1590 — python-sdk fix for the related client-side defect (commands.connect to a resumed sandbox stops delivering stdout after a large response #1352): abandoned HTTP/2 streams are now actually cancelled (RST_STREAM) instead of lingering half-open on the shared envd connection.

What actually happens

envd fans process output out to all attached streams (run, every connect(pid)) via a loop that does a blocking send to each consumer in sequence. Any stream still attached when the sandbox auto-pauses is captured in the pause snapshot; after resume, its restored TCP connection leads nowhere — writes to it block forever without erroring. The handler goroutine for that ghost stream gets stuck mid-send, the fan-out loop blocks on it, and from that point no consumer of that process receives anything — including your fresh post-resume connect(pid) stream. That's why you see the first response (sometimes partially — it flows until the ghost's remaining buffer space runs out) and then silence, and why a fresh reconnect shows the process alive but silent.

I confirmed the wedge is server-side by reattaching from a completely separate process with a fresh TCP connection after a stall: it also receives 0 bytes.

Why your bisect flagged 2.29.1

Up to 2.29.0, the SDK set a per-chunk read timeout on command streams equal to the command timeout (60s default for connect()). All sandboxes share one pooled HTTP/2 connection, and in httpcore a single read timeout poisons that whole connection: the pool discards it, and the TCP close cascades through the proxy to envd, which detaches all stream consumers while the sandbox is still running — so the auto-pause snapshot was usually clean, and resume worked (your ~1/6 residual failures were the cycles where the timing didn't line up). 2.29.1 shipped #1448, which removed that read timeout (it raced the server-side deadline and caused spurious TimeoutExceptions — the fix itself was correct). With nothing tearing the old streams down, every cycle's abandoned stream survives into the snapshot, becomes a ghost on resume, and wedges the fan-out: 6/6. (cb061d2's decoding changes turn out to be innocent.)

One more wrinkle: current envd sends keepalive events every 50s on open streams, which resets the old 60s read timeout — so against current envd my harness stalls deterministically even on 2.29.0. The old behavior you measured depends on the envd build in your templates. That's the strongest reason the durable fix is server-side.

Until the infra fix is deployed

• If a reattach stalls, kill and restart the background process (commands.kill(pid) + start it again) — the wedge is per-process, so a restarted process streams fine. Much cheaper than rebuilding the sandbox.
• Avoiding auto-pause (longer timeout + explicit lifecycle) avoids the ghost-consumer scenario entirely.

Keeping this open until infra#3362 is deployed and I've re-verified the pause/resume harness end to end.

[nseniak]

Thanks for the incredibly thorough root-cause. This is much deeper than I expected, and the envd fan-out explanation matches everything we saw: first response partial, then total silence, process still alive on a fresh reconnect.

Your note about current envd stalling even on 2.29.0 lines up with our setup. Our integration suite runs against our own custom templates built a while back, so they're on an older envd. That's almost certainly why we measured ~1/6 on 2.29.0 rather than deterministic stalls. Good to know a template rebuild would change that picture until the infra fix lands.

We've pinned to 2.29.0 as an interim measure and will track infra#3362 / #1590. Happy to re-test against our workload once the envd fix rolls out. Thanks again for chasing this down so fast.