Skip to content

fix(tls): make certificate restarts explicit and verify trust (FB-4386) - #114

Open
simonzelenskifbio wants to merge 9 commits into
FB-4385-gateway-reliabilityfrom
FB-4386-certificate-lifecycle
Open

simonzelenskifbio wants to merge 9 commits into
FB-4385-gateway-reliabilityfrom
FB-4386-certificate-lifecycle

Conversation

@simonzelenskifbio

@simonzelenskifbio simonzelenskifbio commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Background

FB-4386: Secret checksums are evaluated only during online Helm rendering, so certificate issuance or renewal alone does not ensure workloads serve the replacement certificate.

Summary

Expose generic Engine StatefulSet and Gateway Deployment annotations for optional externally installed reload controllers. Document explicit workload restarts after replacement Secrets exist and opt-in Stakater Reloader configuration; no custom restart tokens are needed. Document certificate issuance ordering, same-CA renewal, independent protocol changes, plaintext-listener persistence during TLS enablement, and the lack of automatic CA cutover. Add a Helm TLS hook that verifies every Engine node and optionally a configured Gateway hostname using trusted CA material only, without insecure verification flags.

Stack 4/6 for FB-4382, based on #113. The chart does not install a reload controller. Reloader annotations belong on workload resources, not pods. A shared Engine Secret can trigger concurrent restarts of independent single-replica StatefulSets; manual sequencing is required when that concurrency is undesirable. Neither path guarantees uninterrupted queries or coordinates CA migration.

Test Plan

  • Helm 3.16.3 full validation, ten render regressions, strict lint, generated docs, and docs checks: passed.
  • Enabled Engine and Gateway TLS on the dedicated kind release with a temporary private CA and SAN-correct certificate: rollout passed.
  • Full helm test suite including managed-table SQL and both new verified TLS paths: passed.
  • Negative Gateway hostname verification with the trusted CA failed as expected (curl exit 60).

Stack navigation: #111 → #112 → #113 → #114 → #117 → #120. Parent: FB-4382. Merge in this order. Signing-key rotation documentation is in #120 (FB-4481).

Audit follow-up: the TLS hook now verifies every Engine node against both its node hostname and, when Gateway is enabled, its ready-only serving hostname. On the dedicated kind release, the old certificate passed node verification but failed serving-hostname verification (curl exit 60). After certificate replacement and Engine restart, both Engine identities and the full SQL/Helm suite passed. Render regressions cover two nodes and Gateway-disabled behavior.

Revision validation: full chart validation and 20 render regressions passed on the final stack. Verified annotation placement and absent restart-token annotations in live kind workloads. Sequential Engine restarts plus Gateway and Metadata Service restarts passed, followed by all Helm hooks, SQL writes, and Engine TLS verification. Reloader controller execution was not tested; its optional configuration follows the upstream documented annotation interface.


Note

Medium Risk
Changes operational TLS/certificate lifecycle and rollout behavior; mis-timed restarts or CA cutover can still break queries, though runtime TLS logic is mostly documentation plus opt-in annotations and test hooks.

Overview
Addresses FB-4386: renewing or replacing TLS Secrets does not by itself reload serving certs—workloads must be restarted (or use external Secret reload automation) after the new Secret exists.

Chart changes: Adds engineSpec.annotations and gateway.annotations on Engine StatefulSets and the Gateway Deployment (not pods), e.g. for Stakater Reloader. Introduces tls.gateway.verification (serverName, optional caSecret) and a new helm test hook (test-tls.yaml) that verifies every Engine node (node hostname plus ready-only serving identity when Gateway is enabled) and optionally the Gateway—with full cert verification and CA-only Secret projection, no insecure curl flags. Schema, values, helm-docs, and render regressions cover annotation scoping and hook behavior.

Docs: docs/usage/tls.mdx expands renewal/restart runbooks, limits of online Secret checksum rollouts, optional Reloader setup, CA migration and plaintext-during-TLS-enablement caveats, and how to run verified TLS checks via helm test.

Reviewed by Cursor Bugbot for commit a469878. Bugbot is set up for automated code reviews on this repo. Configure here.

Provide offline-compatible restart tokens and a CA-verifying Helm hook. Document staged renewal and maintenance requirements for protocol and trust changes.
@fireboltdb-docs-integration

fireboltdb-docs-integration Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Mintlify docs preview

Docs were aggregated into the Firebolt documentation site.

Status Preview Updated (UTC)
🟢 Ready View Preview Oct 02, 2026, 9:34 AM

Mintlify may take a few minutes to finish building. This comment updates on each sync while the source PR is open.

@simonzelenskifbio
simonzelenskifbio marked this pull request as ready for review October 1, 2026 15:33
cursor[bot]
cursor Bot previously approved these changes Oct 1, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: medium. Approved. Cursor Bugbot and Cursor Security Agent both completed successfully with no findings that need human review.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

cursor[bot]
cursor Bot previously approved these changes Oct 2, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium. Approved. Cursor Bugbot completed successfully with no findings that need human review; Cursor Security Agent was not present, so that signal was skipped.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

cursor[bot]
cursor Bot previously approved these changes Oct 2, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium. Approved. Cursor Bugbot completed successfully with no findings that need human review; Cursor Security Agent was not present, so that signal was skipped.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium. Approved. Cursor Bugbot completed successfully with no findings that need human review; Cursor Security Agent was not present, so that signal was skipped.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant