bug(envd): HTTP listener bound after blocking init chain — connect-refused during startup race

[AdaAibaby]

Summary

On fresh cold boot, run() in packages/envd/main.go calls s.ListenAndServe() as the last statement after a long synchronous init chain. Any blocking call in that chain leaves the process alive but port :49983 returning ECONNREFUSED — indistinguishable from a crash to any TCP-based health check.

Root Cause

run() startup order (main.go):
  1. go PollForMMDSOpts(...)          ← goroutine, async
  2. logs.NewLogger(...)              ← sync
  3. filesystemRpc.Handle(...)        ← sync
  4. createCgroupManager()            ← sync + kernel I/O  ← most likely hang point
  5. NewWorkloadFreezer(...)          ← sync
  6. processRpc.Handle(...)           ← sync
  7. api.New(...)                     ← sync
  8. publicport.NewScanner/Forwarder  ← sync
  9. s.ListenAndServe()               ← TCP port bound HERE, too late

Most likely blocking point: createCgroupManager() → createCgroup() → writeCgroupProp() (cgroup2.go:117) writes to cgroupv2 files (memory.high, memory.max) via os.OpenFile + WriteString with no timeout. Under host memory pressure the kernel blocks the write() syscall during page reclaim. The process stays alive (context not cancelled, goroutines running) but :49983 returns ECONNREFUSED.

Verified on dev (main branch, Linux 6.8 cgroupv2 host)

Instrumented reproduction with a 200 ms synthetic init delay:

Current behavior (listener-last):
  [10ms]  port CLOSED — process alive, ECONNREFUSED  ← RACE WINDOW
  [100ms] port CLOSED — process alive, ECONNREFUSED  ← RACE WINDOW
  [190ms] port CLOSED — process alive, ECONNREFUSED  ← RACE WINDOW
  [201ms] port OPEN   — ListenAndServe finally reached

Fixed behavior (listener-first):
  [0ms]  port OPEN — kernel queues SYNs immediately
  [10ms] port OPEN — still open during entire init
  [201ms] port OPEN — Serve() starts accepting

Relation to #3609 and #3559

This is the mechanism behind #3609 (listener-last race). The two issues are independent:

• This bug fires before the HTTP listener binds (PostInit is never called)
• bug(envd): PostInit spawns unbounded MMDS-poll goroutines — one per /init retry #3559 fires after the listener is up (PostInit is called repeatedly)

The "5-15 s MMDS traffic" described in #3609 does not match the current PollForMMDSOpts implementation (one-shot, exits after first success), suggesting the reporter observed an older version or an external MMDS source.

Fix

Call net.Listen() as the first operation in run(), before any blocking initializer. Replace s.ListenAndServe() with s.Serve(ln). Add stderr phase logs around createCgroupManager so a hung init is visible in logs.

PR with the fix: coming separately.

Impact

• Rare (< 1 % of cold boots per bug(envd): rare startup race — MMDS poller keeps running but HTTP listener never binds (sandbox refuses :49983 instantly, orchestrator can't tell) #3609) but permanent — the stuck sandbox never recovers since envd never exits
• Affects any orchestrator that TCP-dials :49983 as a readiness signal
• Self-hosted deployments are most exposed (no automatic node recycling)