Repository navigation
fix(gateway): drain connections and route to ready engines (FB-4385) - #113
simonzelenskifbio wants to merge 6 commits into
Conversation
Separate peer discovery from serving readiness, keep health probes independent of the query listener, and drain active connections within the pod termination budget. Avoid DNS search expansion for fully qualified engine authorities.
Mintlify docs previewDocs were aggregated into the Firebolt documentation site.
Mintlify may take a few minutes to finish building. This comment updates on each sync while the source PR is open. |
There was a problem hiding this comment.
Stale comment
Risk: medium. Left a non-blocking comment because Cursor Bugbot stayed pending and did not reach a terminal state within the 8-minute wait, so this run is not approving. Cursor Security Agent passed with no findings; reviewers fstr and gm42 were assigned for human review.
Sent by Cursor Approval Agent: Infra Reviewer
There was a problem hiding this comment.
Stale comment
Risk: medium. Approved because Cursor Bugbot completed with no findings that need human review, Cursor Security Agent was not present after the first poll, and risk stays within the approval threshold. No prior approval from this automation needed to be dismissed.
Sent by Cursor Approval Agent: Infra Reviewer
There was a problem hiding this comment.
Risk: medium. Left a non-blocking comment because Cursor Bugbot stayed pending and did not reach a terminal state within the 8-minute wait, so this run is not approving. Cursor Security Agent was not present after the first poll; human review is needed and no reviewers were assigned.
Sent by Cursor Approval Agent: Infra Reviewer
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b890e95. Configure here.
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: http |
There was a problem hiding this comment.
Liveness can abort extended drains
Medium Severity
/healthcheck/fail makes the metrics /healthz health-check filter return 503 for both readiness and liveness. Raising the new gateway.terminationGracePeriodSeconds budget above the liveness window (period 15s, default three failures) lets kubelet kill the container mid-preStop and drop the in-flight queries that longer budget was meant to protect.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit b890e95. Configure here.
There was a problem hiding this comment.
Checked this against the Kubernetes v1.35.0 kubelet source. For normal pod deletion/rolling replacement, the probe worker detects DeletionTimestamp, sets the liveness/startup result to success, and stops probing:
https://github.com/kubernetes/kubernetes/blob/v1.35.0/pkg/kubelet/prober/worker.go#L275-L286
Consequently, the metrics /healthz returning 503 during the deletion-triggered preStop does not cause three liveness failures to cut an extended drain short. The pod termination grace period still bounds the shutdown, and active requests can be interrupted when that budget expires.
This conclusion applies to Kubernetes-managed pod deletion. Manually invoking /healthcheck/fail on a pod that is not being deleted is a different scenario and can fail its liveness probe. No code change is needed for the reported deletion scenario.




Background
FB-4385: the Gateway used bootstrap discovery endpoints and a fixed shutdown sleep, which could route requests to unready nodes or terminate active connections without an explicit drain.
Summary
Add a ready-only headless Service for query traffic while retaining not-ready peer discovery. Include the serving name in Engine certificate SANs. Port bounded Envoy connection draining, use an independent health listener, expose the total termination budget, and disable c-ares search-domain expansion for fully qualified authorities.
Stack 3/6 for FB-4382, based on #112. Existing BYO Engine TLS certificates must cover the new serving hostname before the routing transition; the documentation calls out pre-provisioning or a maintenance window. Engine StatefulSet updates are still not coordinated blue-green rollouts.
The two headless Services deliberately separate Engine discovery from query routing. Both select the same pods and return pod IPs directly:
…-hlkeepspublishNotReadyAddresses: true, exposing ready and unready nodes for Engine peer discovery during startup. It is readiness-independent, not a Service exclusively for unready pods.…-readysetspublishNotReadyAddresses: false. The Gateway resolves this Service, so Kubernetes readiness filters the pod IPs offered as query destinations. Envoy still performs per-pod load balancing and active health checks.For example, when one of three nodes is starting, peer discovery can still find all three through
…-hl, while the Gateway discovers only the ready nodes through…-ready. Making the existing discovery Service ready-only could introduce a bootstrap dependency if nodes need to discover peers before becoming ready.This adds a readiness safeguard alongside Envoy health checks; it does not guarantee uninterrupted queries. Readiness detection, DNS propagation, and Envoy refreshes are asynchronous, and DNS changes do not immediately terminate existing connections. If no nodes are ready, there is no alternative serving capacity. The additional Service is declarative and requires no controller or operator changes.
Test Plan
Stack navigation: #111 → #112 → #113 → #114 → #117 → #120. Parent: FB-4382. Merge in this order. Signing-key rotation documentation is in #120 (FB-4481).
Audit follow-up: existing Engine TLS certificates must contain each new ready-only serving hostname before the Gateway switches routing. The TLS guide now documents certificate preparation, Engine restart, verification, and chart upgrade as separate steps. Missing serving SANs can break queries while Engine pods remain ready. This requires a planned migration; Helm does not coordinate certificate issuance and routing automatically.
Note
Medium Risk
Changes live query routing, gateway shutdown, and TLS SAN requirements; mis-timed TLS upgrades can break Gateway→Engine verification while pods still look ready.
Overview
Gateway query routing now targets a new per-engine headless Service (
…-ready) that only publishes ready pod IPs, while the existing…-hlService stays on the StatefulSets for peer discovery (publishNotReadyAddresses: true). Envoy’s Lua filter and dynamic forward proxy authority switch from…-hlto…-ready; c-ares is configured withno_default_search_domainfor those FQDNs.Graceful gateway shutdown replaces the fixed preStop sleep with
files/gateway-prestop.sh: fail admin readiness, wait 5s for endpoint propagation, gracefully drain INBOUND query listeners, then wait until active downstream connections hit zero. Probes move to the separate metrics listener (with health-check filters there too), Envoy gets--drain-time-s 0/ immediate drain strategy, andgateway.terminationGracePeriodSeconds(default 15) bounds the whole shutdown.TLS adds each engine’s
…-readyhostname to cert-manager /fbinstance.engineTlsDnsNamesSANs. Docs describe upgrade/drain behavior and a maintenance-window sequence for TLS installs moving off…-hlrouting (reissue certs, restart engines, then upgrade).Render and Docker runtime tests cover the dual Services, drain/probe wiring, serving SANs, and preStop behavior with an in-flight request.
Reviewed by Cursor Bugbot for commit b890e95. Bugbot is set up for automated code reviews on this repo. Configure here.