Skip to content

fix(orchestrator): early-bind gRPC port + serve /health during startup reclaim - #3616

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/orchestrator-early-bind-health
Open

fix(orchestrator): early-bind gRPC port + serve /health during startup reclaim#3616
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/orchestrator-early-bind-health

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What

Bind the orchestrator's service port (GRPC_PORT) and start the HTTP /health server before the potentially long sandbox-runtime initialization (startup reclaim + network pool populate), instead of after it.

Refs #3615 (Solution A).

Why

On a host that previously ran many sandboxes and was drained to 0, a large number of ns-* network namespaces can remain in /run/netns (leaked slots). On the next orchestrator upgrade/restart, startupreclaim.Runnetwork.ReclaimLeakedSlots tears these down one namespace at a time, and the whole reclaim chain runs before cmux binds the port. With thousands of residual namespaces this takes 20–30 minutes, during which:

  • the port is closed → Nomad health checks fail with connection-refused,
  • the node reads as failed (not starting) → restart churn.

How

  • pkg/service/info.go: ServiceInfo now starts Unhealthy instead of Healthy.
  • pkg/factories/run.go:
    • Create the cmux server, match the HTTP and gRPC listeners, and start cmux.Serve() + the HTTP /health server early, right after the closers slice is set up.
    • The long init chain (acquireOrchestratorLock stays first and blocking → startup reclaim → NewStorageLocal → pool populate → server.NewRegisterService) runs afterwards, unchanged in order.
    • After all RegisterService calls complete, grpcServer.Serve(grpcListener) is started and the status flips to Healthy.
    • The late /upload handler (local build storage, only created during template-manager setup) is wired through an atomic.Pointer[localupload.Handler] so the HTTP mux stays immutable once serving; it returns 503 until the real handler is stored.

Effect

The port is open the whole time and /health answers 503 until the runtime is ready, so Nomad treats the node as starting rather than failed. No new work is routed to the node until it flips to Healthy (edge/service discovery gates on CanAcceptNewRequests, which is Healthy-only).

Correctness notes

  • gRPC ordering: gRPC forbids RegisterService after Serve(). The gRPC listener is matched at early-bind but only Served after all registrations complete; connections arriving in between buffer in the cmux matcher. During that window /health is 503, so the edge should not be sending gRPC traffic here.
  • Lock stays first: acquireOrchestratorLock (flock) remains first and blocking — it is the single-instance guard, and startup reclaim mutates host-level netns/iptables so it must hold the exclusive lock.
  • Drain path: the shutdown drain only acts when status is Healthy/Standby, so a shutdown received mid-init (still Unhealthy) correctly skips draining a node that never became ready.

Testing

Built and tested on Linux (linux/amd64, go1.26.3):

  • go build ./... for packages/orchestrator — full binary compiles.
  • go vet ./pkg/factories/ ./pkg/service/ ./pkg/healthcheck/ — clean.
  • go test ./pkg/factories/ ./pkg/service/ — pass.
  • go test ./pkg/sandbox/network/... ./pkg/startupreclaim/... — pass (covers the reclaim path this PR reorders around).
  • Full orchestrator test tree compiles (go test -run=^$ ./...).

Follow-ups (not in this PR)

@cla-bot cla-bot Bot added the cla-signed label Sep 3, 2026
…p reclaim

On a host with many leaked ns-* in /run/netns (e.g. after draining a node
that had run many sandboxes), startup reclaim tears down namespaces serially
and runs before the listener binds, so GRPC_PORT stays unbound for 20-30 min.
Nomad health checks get connection-refused and the node reads as failed.

Bind the cmux listener and start the HTTP /health server before the long
sandbox-runtime init. serviceInfo now starts Unhealthy, so /health returns
503 (Nomad sees "starting", not "failed") until the gRPC server is wired up,
at which point the status flips to Healthy. The gRPC listener is matched at
early-bind but only Served after all RegisterService calls complete; early
gRPC connections buffer in the cmux matcher. The late /upload handler is wired
through an atomic pointer so the mux stays immutable once serving.

Refs e2b-dev#3615
@AdaAibaby
AdaAibaby force-pushed the fix/orchestrator-early-bind-health branch from 880e5e7 to 0a76c19 Compare September 3, 2026 09:49
@AdaAibaby
AdaAibaby marked this pull request as ready for review September 3, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants