Skip to content

fix(gateway): drain connections and route to ready engines (FB-4385) - #113

Open
simonzelenskifbio wants to merge 6 commits into
FB-4384-configuration-correctnessfrom
FB-4385-gateway-reliability
Open

simonzelenskifbio wants to merge 6 commits into
FB-4384-configuration-correctnessfrom
FB-4385-gateway-reliability

Conversation

@simonzelenskifbio

@simonzelenskifbio simonzelenskifbio commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Background

FB-4385: the Gateway used bootstrap discovery endpoints and a fixed shutdown sleep, which could route requests to unready nodes or terminate active connections without an explicit drain.

Summary

Add a ready-only headless Service for query traffic while retaining not-ready peer discovery. Include the serving name in Engine certificate SANs. Port bounded Envoy connection draining, use an independent health listener, expose the total termination budget, and disable c-ares search-domain expansion for fully qualified authorities.

Stack 3/6 for FB-4382, based on #112. Existing BYO Engine TLS certificates must cover the new serving hostname before the routing transition; the documentation calls out pre-provisioning or a maintenance window. Engine StatefulSet updates are still not coordinated blue-green rollouts.

The two headless Services deliberately separate Engine discovery from query routing. Both select the same pods and return pod IPs directly:

  • …-hl keeps publishNotReadyAddresses: true, exposing ready and unready nodes for Engine peer discovery during startup. It is readiness-independent, not a Service exclusively for unready pods.
  • …-ready sets publishNotReadyAddresses: false. The Gateway resolves this Service, so Kubernetes readiness filters the pod IPs offered as query destinations. Envoy still performs per-pod load balancing and active health checks.

For example, when one of three nodes is starting, peer discovery can still find all three through …-hl, while the Gateway discovers only the ready nodes through …-ready. Making the existing discovery Service ready-only could introduce a bootstrap dependency if nodes need to discover peers before becoming ready.

This adds a readiness safeguard alongside Envoy health checks; it does not guarantee uninterrupted queries. Readiness detection, DNS propagation, and Envoy refreshes are asynchronous, and DNS changes do not immediately terminate existing connections. If no nodes are ready, there is no alternative serving capacity. The additional Service is declarative and requires no controller or operator changes.

Test Plan

  • Full Helm 3.16.3 validation and eight render regressions: passed.
  • Docker runtime: retry safety passed; actual preStop retained an active request, waited for it to finish, closed query admission, and left the health listener reachable.
  • Docs checks and generated values docs: passed.
  • Local kind upgrade: Gateway and Engines ready; full SQL and Helm-hook validation recorded in stack update.

Stack navigation: #111 → #112 → #113 → #114 → #117 → #120. Parent: FB-4382. Merge in this order. Signing-key rotation documentation is in #120 (FB-4481).

Audit follow-up: existing Engine TLS certificates must contain each new ready-only serving hostname before the Gateway switches routing. The TLS guide now documents certificate preparation, Engine restart, verification, and chart upgrade as separate steps. Missing serving SANs can break queries while Engine pods remain ready. This requires a planned migration; Helm does not coordinate certificate issuance and routing automatically.


Note

Medium Risk
Changes live query routing, gateway shutdown, and TLS SAN requirements; mis-timed TLS upgrades can break Gateway→Engine verification while pods still look ready.

Overview
Gateway query routing now targets a new per-engine headless Service (…-ready) that only publishes ready pod IPs, while the existing …-hl Service stays on the StatefulSets for peer discovery (publishNotReadyAddresses: true). Envoy’s Lua filter and dynamic forward proxy authority switch from …-hl to …-ready; c-ares is configured with no_default_search_domain for those FQDNs.

Graceful gateway shutdown replaces the fixed preStop sleep with files/gateway-prestop.sh: fail admin readiness, wait 5s for endpoint propagation, gracefully drain INBOUND query listeners, then wait until active downstream connections hit zero. Probes move to the separate metrics listener (with health-check filters there too), Envoy gets --drain-time-s 0 / immediate drain strategy, and gateway.terminationGracePeriodSeconds (default 15) bounds the whole shutdown.

TLS adds each engine’s …-ready hostname to cert-manager / fbinstance.engineTlsDnsNames SANs. Docs describe upgrade/drain behavior and a maintenance-window sequence for TLS installs moving off …-hl routing (reissue certs, restart engines, then upgrade).

Render and Docker runtime tests cover the dual Services, drain/probe wiring, serving SANs, and preStop behavior with an in-flight request.

Reviewed by Cursor Bugbot for commit b890e95. Bugbot is set up for automated code reviews on this repo. Configure here.

Separate peer discovery from serving readiness, keep health probes independent of the query listener, and drain active connections within the pod termination budget. Avoid DNS search expansion for fully qualified engine authorities.
@fireboltdb-docs-integration

fireboltdb-docs-integration Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Mintlify docs preview

Docs were aggregated into the Firebolt documentation site.

Status Preview Updated (UTC)
🟢 Ready View Preview Oct 02, 2026, 9:34 AM

Mintlify may take a few minutes to finish building. This comment updates on each sync while the source PR is open.

@simonzelenskifbio
simonzelenskifbio marked this pull request as ready for review October 1, 2026 15:33
@cursor
cursor Bot requested review from fstr and gm42 October 1, 2026 15:42

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: medium. Left a non-blocking comment because Cursor Bugbot stayed pending and did not reach a terminal state within the 8-minute wait, so this run is not approving. Cursor Security Agent passed with no findings; reviewers fstr and gm42 were assigned for human review.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

cursor[bot]
cursor Bot previously approved these changes Oct 2, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: medium. Approved because Cursor Bugbot completed with no findings that need human review, Cursor Security Agent was not present after the first poll, and risk stays within the approval threshold. No prior approval from this automation needed to be dismissed.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium. Left a non-blocking comment because Cursor Bugbot stayed pending and did not reach a terminal state within the 8-minute wait, so this run is not approving. Cursor Security Agent was not present after the first poll; human review is needed and no reviewers were assigned.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b890e95. Configure here.

livenessProbe:
httpGet:
path: /healthz
port: http

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Liveness can abort extended drains

Medium Severity

/healthcheck/fail makes the metrics /healthz health-check filter return 503 for both readiness and liveness. Raising the new gateway.terminationGracePeriodSeconds budget above the liveness window (period 15s, default three failures) lets kubelet kill the container mid-preStop and drop the in-flight queries that longer budget was meant to protect.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit b890e95. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this against the Kubernetes v1.35.0 kubelet source. For normal pod deletion/rolling replacement, the probe worker detects DeletionTimestamp, sets the liveness/startup result to success, and stops probing:

https://github.com/kubernetes/kubernetes/blob/v1.35.0/pkg/kubelet/prober/worker.go#L275-L286

Consequently, the metrics /healthz returning 503 during the deletion-triggered preStop does not cause three liveness failures to cut an extended drain short. The pod termination grace period still bounds the shutdown, and active requests can be interrupted when that budget expires.

This conclusion applies to Kubernetes-managed pod deletion. Manually invoking /healthcheck/fail on a pod that is not being deleted is a different scenario and can fail its liveness probe. No code change is needed for the reported deletion scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant