fix(orchestrator): early-bind gRPC port + serve /health during startup reclaim - #3616
Open
AdaAibaby wants to merge 1 commit into
Open
fix(orchestrator): early-bind gRPC port + serve /health during startup reclaim#3616AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
…p reclaim On a host with many leaked ns-* in /run/netns (e.g. after draining a node that had run many sandboxes), startup reclaim tears down namespaces serially and runs before the listener binds, so GRPC_PORT stays unbound for 20-30 min. Nomad health checks get connection-refused and the node reads as failed. Bind the cmux listener and start the HTTP /health server before the long sandbox-runtime init. serviceInfo now starts Unhealthy, so /health returns 503 (Nomad sees "starting", not "failed") until the gRPC server is wired up, at which point the status flips to Healthy. The gRPC listener is matched at early-bind but only Served after all RegisterService calls complete; early gRPC connections buffer in the cmux matcher. The late /upload handler is wired through an atomic pointer so the mux stays immutable once serving. Refs e2b-dev#3615
AdaAibaby
force-pushed
the
fix/orchestrator-early-bind-health
branch
from
September 3, 2026 09:49
880e5e7 to
0a76c19
Compare
AdaAibaby
marked this pull request as ready for review
September 3, 2026 10:07
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
September 3, 2026 10:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bind the orchestrator's service port (
GRPC_PORT) and start the HTTP/healthserver before the potentially long sandbox-runtime initialization (startup reclaim + network pool populate), instead of after it.Refs #3615 (Solution A).
Why
On a host that previously ran many sandboxes and was drained to 0, a large number of
ns-*network namespaces can remain in/run/netns(leaked slots). On the next orchestrator upgrade/restart,startupreclaim.Run→network.ReclaimLeakedSlotstears these down one namespace at a time, and the whole reclaim chain runs beforecmuxbinds the port. With thousands of residual namespaces this takes 20–30 minutes, during which:How
pkg/service/info.go:ServiceInfonow startsUnhealthyinstead ofHealthy.pkg/factories/run.go:cmuxserver, match the HTTP and gRPC listeners, and startcmux.Serve()+ the HTTP/healthserver early, right after the closers slice is set up.acquireOrchestratorLockstays first and blocking → startup reclaim →NewStorageLocal→ pool populate →server.New→RegisterService) runs afterwards, unchanged in order.RegisterServicecalls complete,grpcServer.Serve(grpcListener)is started and the status flips toHealthy./uploadhandler (local build storage, only created during template-manager setup) is wired through anatomic.Pointer[localupload.Handler]so the HTTP mux stays immutable once serving; it returns 503 until the real handler is stored.Effect
The port is open the whole time and
/healthanswers 503 until the runtime is ready, so Nomad treats the node as starting rather than failed. No new work is routed to the node until it flips toHealthy(edge/service discovery gates onCanAcceptNewRequests, which isHealthy-only).Correctness notes
RegisterServiceafterServe(). The gRPC listener is matched at early-bind but only Served after all registrations complete; connections arriving in between buffer in the cmux matcher. During that window/healthis 503, so the edge should not be sending gRPC traffic here.acquireOrchestratorLock(flock) remains first and blocking — it is the single-instance guard, and startup reclaim mutates host-level netns/iptables so it must hold the exclusive lock.Healthy/Standby, so a shutdown received mid-init (stillUnhealthy) correctly skips draining a node that never became ready.Testing
Built and tested on Linux (linux/amd64, go1.26.3):
go build ./...forpackages/orchestrator— full binary compiles.go vet ./pkg/factories/ ./pkg/service/ ./pkg/healthcheck/— clean.go test ./pkg/factories/ ./pkg/service/— pass.go test ./pkg/sandbox/network/... ./pkg/startupreclaim/...— pass (covers the reclaim path this PR reorders around).go test -run=^$ ./...).Follow-ups (not in this PR)
ReclaimLeakedSlotsto cut reclaim wall-time. Complementary to this change.ns-*leak on drain is expected, or points to a normal-operation teardown path that should have cleaned these up.