Summary
On a self-hosted E2B-compatible deployment we occasionally observe sandboxes where the envd process is clearly alive — Firecracker's fc_api log shows a steady Get /mmds → 200 every 5–15 s — but a plain curl from inside the sandbox netns is refused instantly with no listener:
GET http://<vm-ip>:49983/ -> Failed to connect after 0 ms: Couldn't connect to server
GET http://<vm-ip>:49983/health -> Failed to connect after 0 ms: Couldn't connect to server
A healthy sandbox on the same host, same template, same veth topology answers HTTP 404 in ~0.4 ms / HTTP 204 in ~0.4 ms immediately.
The stuck sandbox stays in this state for the rest of its life. Because envd is running under a runsv-like supervisor and never exits, the supervisor does not restart it. Any orchestrator that only waits for the Firecracker API socket (not for envd:49983) marks the sandbox Running, so every subsequent proxy request fails with hyper/Go connect: connection refused.
Rate: sparse — well under 1 % of fresh cold boots. Not tied to any single template — the same template runs healthy on sibling hosts / other sandboxes.
What this is NOT
Suspected mechanism
packages/envd/main.go run() orders startup like this:
go host.PollForMMDSOpts(ctx, mmdsChan, defaults.EnvVars) — MMDS goroutine is the first thing spawned.- A chain of synchronous initializers:
chi.NewRouter, filesystemRpc.Handle, createCgroupManager(), cgroups.NewWorkloadFreezer(...), processRpc.Handle, optional resumeHandover block, api.New, publicport.NewScanner, publicport.NewForwarder, portForwarder.StartForwarding(ctx) (go), portScanner.ScanAndBroadcast() (go).
- Finally
s.ListenAndServe().
If any of the synchronous steps in (2) blocks or panics into a swallowed / recovered path, the MMDS goroutine keeps ticking while ListenAndServe is never reached. Externally the process looks fully alive: MMDS side effects visible, /proc entry present, RSS non-zero, no exit — but no one is listening on 49983.
We could not grab an in-VM goroutine dump on a live stuck instance because getting a shell into the VM depends on envd's exec path, which is exactly what's broken. pprof served over HTTP needs the very listener that's missing. A vsock- or MMDS-mediated diagnostic channel would be the only way to surface it — mentioning this partly to ask whether the project would accept such a probe.
Suggested mitigations (either would independently avoid the observable state)
1. Listener-first startup. Move net.Listen("tcp", ":<port>") before MMDS poll / port scanner / cgroup manager, and only start those background goroutines once the socket is bound (or fail fast). Then wrap the post-listener init in bounded-timeout goroutines so a hang there surfaces as a listener that accepts + returns 5xx, rather than a listener that never binds.
2. Phased startup logging. At minimum, log each phase of run() before entering it (envd startup: opening cgroup manager etc.), and turn every recovered-panic path in run() into an os.Exit(1). That converts the silent variant into a visible crash, which the supervisor can restart and the operator can see in /var/log.
3. Readiness contract for consumers. Document that anything driving envd (orchestrator, snapshot resume, self-hosted forks) MUST TCP-dial the envd port before marking a sandbox usable, not just wait on the Firecracker socket. Every self-host consumer currently has to reinvent this.
Repro
Not reduced to a minimal repro yet — it's a rare startup-race pattern. Happens across multiple kernel versions, multiple templates, multiple hosts. Happy to gather additional data (in-VM strace of envd, MMDS packet counts, guest ss -tlnp at boot) against any specific hypothesis you'd like tested.
Summary
On a self-hosted E2B-compatible deployment we occasionally observe sandboxes where the envd process is clearly alive — Firecracker's
fc_apilog shows a steadyGet /mmds→ 200 every 5–15 s — but a plaincurlfrom inside the sandbox netns is refused instantly with no listener:A healthy sandbox on the same host, same template, same veth topology answers
HTTP 404 in ~0.4 ms/HTTP 204 in ~0.4 msimmediately.The stuck sandbox stays in this state for the rest of its life. Because envd is running under a runsv-like supervisor and never exits, the supervisor does not restart it. Any orchestrator that only waits for the Firecracker API socket (not for envd:49983) marks the sandbox
Running, so every subsequent proxy request fails with hyper/Goconnect: connection refused.Rate: sparse — well under 1 % of fresh cold boots. Not tied to any single template — the same template runs healthy on sibling hosts / other sandboxes.
What this is NOT
/health→ 204 in <1 ms) and only wedgesProcess.*/Filesystem.*. Here/healthitself is refused instantly.ip routeinside the netns is identical between healthy and stuck. The guest TCP stack is up. envd's HTTP listener is the specific thing missing.Suspected mechanism
packages/envd/main.go run()orders startup like this:go host.PollForMMDSOpts(ctx, mmdsChan, defaults.EnvVars)— MMDS goroutine is the first thing spawned.chi.NewRouter,filesystemRpc.Handle,createCgroupManager(),cgroups.NewWorkloadFreezer(...),processRpc.Handle, optionalresumeHandoverblock,api.New,publicport.NewScanner,publicport.NewForwarder,portForwarder.StartForwarding(ctx)(go),portScanner.ScanAndBroadcast()(go).s.ListenAndServe().If any of the synchronous steps in (2) blocks or panics into a swallowed / recovered path, the MMDS goroutine keeps ticking while
ListenAndServeis never reached. Externally the process looks fully alive: MMDS side effects visible, /proc entry present, RSS non-zero, no exit — but no one is listening on 49983.We could not grab an in-VM goroutine dump on a live stuck instance because getting a shell into the VM depends on envd's exec path, which is exactly what's broken.
pprofserved over HTTP needs the very listener that's missing. A vsock- or MMDS-mediated diagnostic channel would be the only way to surface it — mentioning this partly to ask whether the project would accept such a probe.Suggested mitigations (either would independently avoid the observable state)
1. Listener-first startup. Move
net.Listen("tcp", ":<port>")before MMDS poll / port scanner / cgroup manager, and only start those background goroutines once the socket is bound (or fail fast). Then wrap the post-listener init in bounded-timeout goroutines so a hang there surfaces as a listener that accepts + returns 5xx, rather than a listener that never binds.2. Phased startup logging. At minimum, log each phase of
run()before entering it (envd startup: opening cgroup manageretc.), and turn every recovered-panic path inrun()into anos.Exit(1). That converts the silent variant into a visible crash, which the supervisor can restart and the operator can see in/var/log.3. Readiness contract for consumers. Document that anything driving envd (orchestrator, snapshot resume, self-hosted forks) MUST TCP-dial the envd port before marking a sandbox usable, not just wait on the Firecracker socket. Every self-host consumer currently has to reinvent this.
Repro
Not reduced to a minimal repro yet — it's a rare startup-race pattern. Happens across multiple kernel versions, multiple templates, multiple hosts. Happy to gather additional data (in-VM strace of envd, MMDS packet counts, guest
ss -tlnpat boot) against any specific hypothesis you'd like tested.