Repository navigation
fix(tls): make certificate restarts explicit and verify trust (FB-4386) - #114
Open
simonzelenskifbio wants to merge 9 commits into
Open
simonzelenskifbio wants to merge 9 commits into
simonzelenskifbio wants to merge 9 commits into
Conversation
Provide offline-compatible restart tokens and a CA-verifying Helm hook. Document staged renewal and maintenance requirements for protocol and trust changes.
Mintlify docs previewDocs were aggregated into the Firebolt documentation site.
Mintlify may take a few minutes to finish building. This comment updates on each sync while the source PR is open. |
This was referenced Oct 1, 2026
simonzelenskifbio
marked this pull request as ready for review
October 1, 2026 15:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Background
FB-4386: Secret checksums are evaluated only during online Helm rendering, so certificate issuance or renewal alone does not ensure workloads serve the replacement certificate.
Summary
Expose generic Engine StatefulSet and Gateway Deployment annotations for optional externally installed reload controllers. Document explicit workload restarts after replacement Secrets exist and opt-in Stakater Reloader configuration; no custom restart tokens are needed. Document certificate issuance ordering, same-CA renewal, independent protocol changes, plaintext-listener persistence during TLS enablement, and the lack of automatic CA cutover. Add a Helm TLS hook that verifies every Engine node and optionally a configured Gateway hostname using trusted CA material only, without insecure verification flags.
Stack 4/6 for FB-4382, based on #113. The chart does not install a reload controller. Reloader annotations belong on workload resources, not pods. A shared Engine Secret can trigger concurrent restarts of independent single-replica StatefulSets; manual sequencing is required when that concurrency is undesirable. Neither path guarantees uninterrupted queries or coordinates CA migration.
Test Plan
Stack navigation: #111 → #112 → #113 → #114 → #117 → #120. Parent: FB-4382. Merge in this order. Signing-key rotation documentation is in #120 (FB-4481).
Audit follow-up: the TLS hook now verifies every Engine node against both its node hostname and, when Gateway is enabled, its ready-only serving hostname. On the dedicated kind release, the old certificate passed node verification but failed serving-hostname verification (curl exit 60). After certificate replacement and Engine restart, both Engine identities and the full SQL/Helm suite passed. Render regressions cover two nodes and Gateway-disabled behavior.
Revision validation: full chart validation and 20 render regressions passed on the final stack. Verified annotation placement and absent restart-token annotations in live kind workloads. Sequential Engine restarts plus Gateway and Metadata Service restarts passed, followed by all Helm hooks, SQL writes, and Engine TLS verification. Reloader controller execution was not tested; its optional configuration follows the upstream documented annotation interface.
Note
Medium Risk
Changes operational TLS/certificate lifecycle and rollout behavior; mis-timed restarts or CA cutover can still break queries, though runtime TLS logic is mostly documentation plus opt-in annotations and test hooks.
Overview
Addresses FB-4386: renewing or replacing TLS Secrets does not by itself reload serving certs—workloads must be restarted (or use external Secret reload automation) after the new Secret exists.
Chart changes: Adds
engineSpec.annotationsandgateway.annotationson Engine StatefulSets and the Gateway Deployment (not pods), e.g. for Stakater Reloader. Introducestls.gateway.verification(serverName, optionalcaSecret) and a newhelm testhook (test-tls.yaml) that verifies every Engine node (node hostname plus ready-only serving identity when Gateway is enabled) and optionally the Gateway—with full cert verification and CA-only Secret projection, no insecure curl flags. Schema, values, helm-docs, and render regressions cover annotation scoping and hook behavior.Docs:
docs/usage/tls.mdxexpands renewal/restart runbooks, limits of online Secret checksum rollouts, optional Reloader setup, CA migration and plaintext-during-TLS-enablement caveats, and how to run verified TLS checks viahelm test.Reviewed by Cursor Bugbot for commit a469878. Bugbot is set up for automated code reviews on this repo. Configure here.