From afafe6357305da7342c087c78bd4fe7067d7d5ce Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Tue, 29 Sep 2026 17:40:16 +0200 Subject: [PATCH 1/5] fix(tls): make certificate restarts explicit and verify trust (FB-4386) Provide offline-compatible restart tokens and a CA-verifying Helm hook. Document staged renewal and maintenance requirements for protocol and trust changes. --- docs/usage/tls.mdx | 43 +++++++++++++++ helm/AGENTS.md | 2 + helm/README.md | 5 ++ helm/scripts/test_render.py | 33 +++++++++++ helm/templates/engine-statefulset.yaml | 1 + helm/templates/gateway-deployment.yaml | 1 + helm/templates/tests/test-tls.yaml | 76 ++++++++++++++++++++++++++ helm/values.schema.json | 18 ++++++ helm/values.yaml | 11 ++++ 9 files changed, 190 insertions(+) create mode 100644 helm/templates/tests/test-tls.yaml diff --git a/docs/usage/tls.mdx b/docs/usage/tls.mdx index 7e7c67c..167ecd9 100644 --- a/docs/usage/tls.mdx +++ b/docs/usage/tls.mdx @@ -81,6 +81,49 @@ The engine's certificate's DNS names are derived automatically from every engine The issuer behind `tls.engine.certManager` must populate `ca.crt` in the Certificate's Secret, not just `tls.crt`/`tls.key`. CA and Vault issuers do this; a bare self-signed or ACME issuer may not. The chart's `tls-chain-setup` init container needs `ca.crt` to build the engine's own health-check trust bundle and fails the pod at startup if it's missing. +### Renewal and restart procedure + +Secret updates do not independently trigger Helm upgrades. Engines load serving certificates at startup, and their full certificate chain is assembled by an init container. Treat certificate replacement and workload restart as separate steps: + +1. Renew or replace the certificate Secret, preserving its required keys and SANs. For cert-manager, wait for the replacement Certificate revision to be issued and its Secret to contain the new certificate. Merely observing an old `Ready=True` condition does not establish that renewal finished. +2. For a same-CA Engine renewal, change `engineSpec.restartToken`. For a Gateway certificate or upstream trust change, change `gateway.restartToken`. Upgrade with the full deployment values and wait for every affected workload to finish rolling out. +3. Run the verified TLS hooks described below and confirm the served certificate is the renewed certificate before its predecessor expires. + +Use a new arbitrary, non-secret token for each restart: + +```yaml +engineSpec: + restartToken: certificate-renewal-2 +gateway: + restartToken: certificate-renewal-2 +``` + +Online Helm rendering hashes the current auth/TLS Secrets as an additional restart trigger. It cannot hash a future Secret that cert-manager has not issued yet. Offline and GitOps rendering cannot read live Secrets at all; use the explicit restart tokens after issuance in either deployment mode. The chart does not install a reload controller. + +A CA change requires overlapping trust and coordinated restarts. Do not replace the only trusted CA and both leaf certificates in a single upgrade. The chart does not automatically stage trust bundles or gate traffic cutover. Plan a maintenance window, or manage a separately validated staged migration outside the chart. + +Enabling TLS on an existing plaintext release does not immediately close old plaintext listeners. During a rolling update, old Gateway pods can continue serving while replacement pods wait for certificates. Restrict external access during a security-posture change and verify that all old pods have terminated before restoring it. Engine TLS protocol changes also require a maintenance window because Engine and Gateway updates are independent. + +### Verify certificate trust + +When Engine TLS is enabled, `helm test` checks the CA chain and hostname of every Engine node without disabling certificate verification. To verify the Gateway as well, supply a certificate SAN as `serverName`. The test connects to the internal Service while verifying that hostname. For a private CA, supply a Secret containing `ca.crt`: + +```yaml +tls: + gateway: + verification: + serverName: firebolt.example.com + caSecret: gateway-test-ca +``` + +Run all chart hooks after the rollout: + +```bash +helm test firebolt --namespace firebolt --logs +``` + +An empty Gateway `serverName` skips its verified handshake test. Existing readiness and SQL hooks test functionality and do not establish Gateway certificate trust. The verified TLS hook projects only CA material and never mounts signing keys or TLS private keys. + ### Caveats - The `core-ui` sidecar (`engineSpec.uiSidecar`) follows `tls.engine.enabled` automatically, proxying to the engine over HTTPS without verifying its certificate — the connection is over loopback within the pod, so this doesn't need a trusted CA. diff --git a/helm/AGENTS.md b/helm/AGENTS.md index ba156ed..e4cf4dc 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -86,3 +86,5 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g ## Known issues - Symptom: `make test` (i.e. `helm test --logs`) prints `Phase: Succeeded` for every suite and then exits 1 with `unable to get pod logs ... pods "-test-" not found`. Cause: a `helm.sh/hook-delete-policy` of `before-hook-creation,hook-succeeded` on the test pod tells helm to delete the pod the moment the hook succeeds, so by the time `helm test --logs` iterates the pods to fetch their logs they are already gone (helm/helm#8949). Resolution: test pods MUST set `helm.sh/hook-delete-policy: before-hook-creation` only — never include `hook-succeeded`. Stale pods are still cleaned up on the next `helm test` run by `before-hook-creation`, and `make test-cleanup` deletes them on demand. + +Auth/TLS Secret hashes are evaluated only during online Helm rendering. For offline/GitOps rendering, change the component restart token after new Secret material exists. `tests/test-tls.yaml` verifies every Engine node with the CA and optionally verifies the Gateway using `tls.gateway.verification`; it never disables certificate verification. diff --git a/helm/README.md b/helm/README.md index 3ace23c..55f4556 100644 --- a/helm/README.md +++ b/helm/README.md @@ -48,6 +48,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | engineSpec.podSecurityContext.fsGroupChangePolicy | string | `"OnRootMismatch"` | When to re-apply `fsGroup` ownership. `OnRootMismatch` skips the chown when already correct — much faster on large PVCs. | | engineSpec.podSecurityContext.runAsNonRoot | bool | `true` | Reject the pod if any container runs as UID 0. Also gates the chart's container-level `runAs*` defaults, memlock init, and the entrypoint UID check. | | engineSpec.readiness | bool | `true` | When true, a readiness probe is added to the core container. | +| engineSpec.restartToken | string | `""` | Change after Secret rotation to restart Engines, including offline/GitOps rendering. Wait for replacement Secret material before upgrading. | | engineSpec.serviceAccount | string | `""` | ServiceAccount used by engine pods. Empty (the default): the chart creates `-engine` with `automountServiceAccountToken: false`, so a code-execution exploit in the engine container has no SA token to talk to the apiserver with. Engines do not call the Kubernetes API; the dedicated SA replaces the namespace `default` SA, which automounts a token and inherits any RoleBindings accumulated on `default` from unrelated installs. Non-empty: the chart references the named SA verbatim and does NOT create one — bring your own (the documented IRSA / Pod Identity flow at docs/usage/object-storage/amazon-s3.mdx works this way). The chart cannot influence `automountServiceAccountToken` on a SA it does not own; set it explicitly in your SA manifest if you want the same hardening. | | engineSpec.storageHostPath | object | {} | Host path configuration used when `hostPathStorageEnabled` is true. | | engineSpec.storageHostPath.path | string | `"/var/lib/firebolt-core"` | Host path for engine data. | @@ -104,6 +105,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | gateway.podTemplate.volumes | list | `[]` | Extra volumes added to the gateway pod, merged with the chart-managed volumes. Mount them on the Envoy container via `volumeMounts` below. | | gateway.replicas | int | `2` | Number of gateway replicas. | | gateway.resources | object | `{"limits":{"memory":"512Mi"},"requests":{"cpu":"100m","memory":"256Mi"}}` | Resource requests and limits for the Envoy container. | +| gateway.restartToken | string | `""` | Change after certificate or trust-bundle rotation to restart the Gateway, including offline/GitOps rendering. | | gateway.service | object | {} | Gateway Service configuration. | | gateway.service.port | int | `80` | External service port proxied to `containerPort`. | | gateway.service.type | string | `"ClusterIP"` | Service type. One of `ClusterIP`, `LoadBalancer`, or `NodePort`. | @@ -187,6 +189,9 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | tls.gateway.enabled | bool | `false` | Enable TLS on the gateway's client-facing listener. | | tls.gateway.existingSecret | object | {} | Existing `kubernetes.io/tls` Secret (`tls.crt` / `tls.key`). | | tls.gateway.existingSecret.secretRef | string | `""` | Secret name. | +| tls.gateway.verification | object | {} | Optional verified Gateway TLS Helm test. Set serverName to a certificate SAN; caSecret is optional for publicly trusted certificates. | +| tls.gateway.verification.caSecret | string | `""` | Optional Secret containing ca.crt for the Gateway verification hook. Only the CA key is projected. | +| tls.gateway.verification.serverName | string | `""` | Hostname to verify while connecting to the in-cluster Gateway Service. Empty skips the Gateway verification hook; Engine TLS verification remains automatic. | | utilitiesImage | string | `"debian:stable-slim@sha256:5012d0517aa0075a7150a45aae67586641e898913b7af3b08228108565b5f90c"` | Image used for utility init/sidecar containers (e.g. the memlock-setup sidecar). Pinned to an immutable digest so a registry-side tag override cannot silently change what runs in production. Bump the digest together with the tag when upgrading. | ---------------------------------------------- diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py index a3f0cb5..44d7268 100644 --- a/helm/scripts/test_render.py +++ b/helm/scripts/test_render.py @@ -125,5 +125,38 @@ def test_reject_invalid_combinations(self): self.assertIn(message, render(values, expect_error=True)) +class CertificateLifecycleTests(unittest.TestCase): + def test_restart_tokens_are_component_scoped(self): + baseline = render() + def pod(docs, kind, name): + return resource(docs, kind, name)["spec"]["template"] + cases = [("engineSpec", "StatefulSet", "audit-engine-default-node-0"), + ("gateway", "Deployment", "audit-gateway"), + ("metadata", "Deployment", "audit-metadata-service")] + for component, kind, name in cases: + docs = render({component: {"restartToken": "rotation-2"}}) + self.assertNotEqual(pod(baseline, kind, name), pod(docs, kind, name)) + for other, other_kind, other_name in cases: + if other != component: + self.assertEqual(pod(baseline, other_kind, other_name), pod(docs, other_kind, other_name)) + + def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self): + values = {"tls": { + "engine": {"enabled": True, "existingSecret": {"secretRef": "engine-tls"}}, + "gateway": {"enabled": True, "existingSecret": {"secretRef": "gateway-tls"}, + "verification": {"serverName": "firebolt.example.com", "caSecret": "gateway-ca"}}, + }} + hook = resource(render(values), "Pod", "audit-test-tls")["spec"] + script = hook["containers"][0]["command"][-1] + self.assertNotIn(" -k", script) + self.assertNotIn("--insecure", script) + self.assertIn("--cacert /trust/engine/ca.crt", script) + self.assertIn("--connect-to", script) + self.assertIn("firebolt.example.com", script) + for volume in hook["volumes"]: + self.assertEqual(volume["secret"]["items"], [{"key": "ca.crt", "path": "ca.crt"}]) + self.assertFalse(any(d["metadata"]["name"] == "audit-test-tls" for d in render())) + + if __name__ == "__main__": unittest.main() diff --git a/helm/templates/engine-statefulset.yaml b/helm/templates/engine-statefulset.yaml index 2217904..5e09ca6 100644 --- a/helm/templates/engine-statefulset.yaml +++ b/helm/templates/engine-statefulset.yaml @@ -58,6 +58,7 @@ spec: template: metadata: annotations: + firebolt.io/restart-token: {{ $.Values.engineSpec.restartToken | quote }} # Roll this engine's pods when its own config.yaml changes. The # config hash is scoped to this engine so a change to one engine # doesn't restart the others. diff --git a/helm/templates/gateway-deployment.yaml b/helm/templates/gateway-deployment.yaml index 37159f9..8169116 100644 --- a/helm/templates/gateway-deployment.yaml +++ b/helm/templates/gateway-deployment.yaml @@ -33,6 +33,7 @@ spec: template: metadata: annotations: + firebolt.io/restart-token: {{ .Values.gateway.restartToken | quote }} checksum/config: {{ include (print $.Template.BasePath "/gateway-configmap.yaml") . | sha256sum }} {{- /* See the equivalent comment in engine-statefulset.yaml: these Secrets diff --git a/helm/templates/tests/test-tls.yaml b/helm/templates/tests/test-tls.yaml new file mode 100644 index 0000000..b8030aa --- /dev/null +++ b/helm/templates/tests/test-tls.yaml @@ -0,0 +1,76 @@ +{{- $gatewayTest := and .Values.gateway.enabled .Values.tls.gateway.enabled .Values.tls.gateway.verification.serverName -}} +{{- if or .Values.tls.engine.enabled $gatewayTest }} +apiVersion: v1 +kind: Pod +metadata: + name: {{ include "fbinstance.fullname" . }}-test-tls + namespace: {{ .Release.Namespace }} + annotations: + helm.sh/hook: test + helm.sh/hook-delete-policy: before-hook-creation +spec: + restartPolicy: Never + automountServiceAccountToken: false + {{- if or .Values.tls.engine.enabled (and $gatewayTest .Values.tls.gateway.verification.caSecret) }} + volumes: + {{- if .Values.tls.engine.enabled }} + - name: engine-ca + secret: + secretName: {{ include "fbinstance.secretSourceName" (dict "source" .Values.tls.engine "certManagerSecretName" (printf "%s-tls-engine" (include "fbinstance.fullname" .)) "context" "tls.engine") }} + items: + - key: ca.crt + path: ca.crt + {{- end }} + {{- if and $gatewayTest .Values.tls.gateway.verification.caSecret }} + - name: gateway-ca + secret: + secretName: {{ .Values.tls.gateway.verification.caSecret }} + items: + - key: ca.crt + path: ca.crt + {{- end }} + {{- end }} + containers: + - name: check + image: curlimages/curl:latest + {{- if or .Values.tls.engine.enabled (and $gatewayTest .Values.tls.gateway.verification.caSecret) }} + volumeMounts: + {{- if .Values.tls.engine.enabled }} + - name: engine-ca + mountPath: /trust/engine + readOnly: true + {{- end }} + {{- if and $gatewayTest .Values.tls.gateway.verification.caSecret }} + - name: gateway-ca + mountPath: /trust/gateway + readOnly: true + {{- end }} + {{- end }} + command: + - sh + - -c + - | + set -eu + {{- include "fbinstance.testShellHelpers" . | nindent 10 }} + {{- if .Values.tls.engine.enabled }} + {{- range $engine := .Values.engines }} + {{- range $i := until (int $engine.replicas) }} + HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $.Values.engineSpec.nodeHostSuffix }}" + log "Verify Engine TLS chain and hostname: $HOST" + curl --fail --silent --show-error --max-time 30 \ + --cacert /trust/engine/ca.crt "https://$HOST:3473/health/ready" + pass "Engine TLS verified: $HOST" + {{- end }} + {{- end }} + {{- end }} + {{- if $gatewayTest }} + log "Verify Gateway TLS chain and hostname" + curl --fail --silent --show-error --max-time 30 \ + {{- if .Values.tls.gateway.verification.caSecret }} + --cacert /trust/gateway/ca.crt \ + {{- end }} + --connect-to "{{ .Values.tls.gateway.verification.serverName }}:{{ .Values.gateway.service.port }}:{{ include "fbinstance.fullname" . }}-gateway.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.gateway.service.port }}" \ + "https://{{ .Values.tls.gateway.verification.serverName }}:{{ .Values.gateway.service.port }}/healthz" + pass "Gateway TLS verified" + {{- end }} +{{- end }} diff --git a/helm/values.schema.json b/helm/values.schema.json index c3dd01e..74d6bd7 100644 --- a/helm/values.schema.json +++ b/helm/values.schema.json @@ -62,6 +62,9 @@ ] } } + }, + "restartToken": { + "type": "string" } } }, @@ -180,6 +183,9 @@ "terminationGracePeriodSeconds": { "type": "integer", "minimum": 10 + }, + "restartToken": { + "type": "string" } } }, @@ -344,6 +350,18 @@ }, "certManager": { "$ref": "#/definitions/certManagerSource" + }, + "verification": { + "type": "object", + "properties": { + "serverName": { + "type": "string", + "pattern": "^([A-Za-z0-9]([A-Za-z0-9.-]*[A-Za-z0-9])?)?$" + }, + "caSecret": { + "type": "string" + } + } } } }, diff --git a/helm/values.yaml b/helm/values.yaml index d4bb09f..bc2bc44 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -18,6 +18,8 @@ createNamespace: false # -- Shared engine pod defaults applied to all engines unless overridden per-engine. # @default -- {} engineSpec: + # -- Change after Secret rotation to restart Engines, including offline/GitOps rendering. Wait for replacement Secret material before upgrading. + restartToken: "" image: # -- Container repository for the Firebolt engine image. repository: oci.firebolt.io/firebolt-db/engine @@ -330,6 +332,8 @@ postgresql: # the upstream to `{engine}-service:3473` via dynamic forward proxy. # @default -- {} gateway: + # -- Change after certificate or trust-bundle rotation to restart the Gateway, including offline/GitOps rendering. + restartToken: "" # -- Total pod shutdown budget in seconds, including a 5s endpoint-propagation wait and explicit connection draining. Requests still active at the deadline can be interrupted. terminationGracePeriodSeconds: 15 # -- Set to true to deploy the Envoy gateway proxy. @@ -530,6 +534,13 @@ tls: # -- TLS termination at the Envoy gateway (client → gateway). # @default -- {} gateway: + # -- Optional verified Gateway TLS Helm test. Set serverName to a certificate SAN; caSecret is optional for publicly trusted certificates. + # @default -- {} + verification: + # -- Hostname to verify while connecting to the in-cluster Gateway Service. Empty skips the Gateway verification hook; Engine TLS verification remains automatic. + serverName: "" + # -- Optional Secret containing ca.crt for the Gateway verification hook. Only the CA key is projected. + caSecret: "" # -- Enable TLS on the gateway's client-facing listener. enabled: false # -- Existing `kubernetes.io/tls` Secret (`tls.crt` / `tls.key`). From 22f059cdec0bdc1c6a899ea567f07ef0e182bb51 Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Thu, 1 Oct 2026 16:22:02 +0200 Subject: [PATCH 2/5] docs(tls): exempt non-secret restart examples from scanning (FB-4386) --- .legitignore | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.legitignore b/.legitignore index 14ebf3e..159e040 100644 --- a/.legitignore +++ b/.legitignore @@ -2,3 +2,7 @@ value_67A867BC754C44390E7B54969C75EEE8 # Example Kubernetes Secret name "firebol value_8FAFF5AF6AA62BB96EAF8C4A1C3ED99F # Example Kubernetes Secret name "firebolt-signing-key-2026-01" in docs/usage/authentication.mdx — a resource reference, not a credential value. value_0BAC68E67DA5B3BED13A4A58E76B535A # Dummy postgresql.password used by helm/scripts/validate-chart.sh to satisfy the values schema during template rendering — not a credential. value_01kp98j0000000000000000000 # Dummy customEngineConfig.instance.id (lowercase Crockford ULID placeholder in values.yaml / helm-docs / schema tests) — not a credential. + +# Historical documentation examples: public rollout markers, never credentials. +instance_D0360DAADFA20D225333000C4486F464 # docs/usage/tls.mdx:96 in afafe63 +instance_84F2094DD13F984A9CD7224F6E08C27E # docs/usage/tls.mdx:98 in afafe63 From 000c1dcda5157e69142eab286382bafefb1ebe03 Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Thu, 1 Oct 2026 17:07:29 +0200 Subject: [PATCH 3/5] test(tls): verify each engine serving identity (FB-4386) --- docs/usage/tls.mdx | 2 +- helm/scripts/test_render.py | 7 ++++++- helm/templates/tests/test-tls.yaml | 9 +++++++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/usage/tls.mdx b/docs/usage/tls.mdx index d7632b5..7103e8e 100644 --- a/docs/usage/tls.mdx +++ b/docs/usage/tls.mdx @@ -119,7 +119,7 @@ Enabling TLS on an existing plaintext release does not immediately close old pla ### Verify certificate trust -When Engine TLS is enabled, `helm test` checks the CA chain and hostname of every Engine node without disabling certificate verification. To verify the Gateway as well, supply a certificate SAN as `serverName`. The test connects to the internal Service while verifying that hostname. For a private CA, supply a Secret containing `ca.crt`: +When Engine TLS is enabled, `helm test` checks each Engine node against its node hostname and, when the Gateway is enabled, its ready-only serving hostname. These checks never disable certificate verification. To verify the Gateway as well, supply a certificate SAN as `serverName`. The test connects to the internal Service while verifying that hostname. For a private CA, supply a Secret containing `ca.crt`: ```yaml tls: diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py index 54ac7e9..2aab981 100644 --- a/helm/scripts/test_render.py +++ b/helm/scripts/test_render.py @@ -155,7 +155,7 @@ def pod(docs, kind, name): self.assertEqual(pod(baseline, other_kind, other_name), pod(docs, other_kind, other_name)) def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self): - values = {"tls": { + values = {"engines": [{"name": "default", "replicas": 2}], "tls": { "engine": {"enabled": True, "existingSecret": {"secretRef": "engine-tls"}}, "gateway": {"enabled": True, "existingSecret": {"secretRef": "gateway-tls"}, "verification": {"serverName": "firebolt.example.com", "caSecret": "gateway-ca"}}, @@ -165,11 +165,16 @@ def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self): self.assertNotIn(" -k", script) self.assertNotIn("--insecure", script) self.assertIn("--cacert /trust/engine/ca.crt", script) + self.assertEqual(script.count('--connect-to "$SERVING:3473:$HOST:3473"'), 2) + self.assertIn("audit-engine-default-ready.audit.svc.cluster.local", script) self.assertIn("--connect-to", script) self.assertIn("firebolt.example.com", script) for volume in hook["volumes"]: self.assertEqual(volume["secret"]["items"], [{"key": "ca.crt", "path": "ca.crt"}]) self.assertFalse(any(d["metadata"]["name"] == "audit-test-tls" for d in render())) + values["gateway"] = {"enabled": False} + direct = resource(render(values), "Pod", "audit-test-tls")["spec"]["containers"][0]["command"][-1] + self.assertNotIn("SERVING=", direct) if __name__ == "__main__": diff --git a/helm/templates/tests/test-tls.yaml b/helm/templates/tests/test-tls.yaml index b8030aa..dc2f83d 100644 --- a/helm/templates/tests/test-tls.yaml +++ b/helm/templates/tests/test-tls.yaml @@ -60,6 +60,15 @@ spec: curl --fail --silent --show-error --max-time 30 \ --cacert /trust/engine/ca.crt "https://$HOST:3473/health/ready" pass "Engine TLS verified: $HOST" + {{- if $.Values.gateway.enabled }} + # Check the Gateway's serving identity against every node, not just + # the node identity. Old certificates may cover only the latter. + SERVING="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-ready.{{ $.Release.Namespace }}.svc.cluster.local" + curl --fail --silent --show-error --max-time 30 \ + --cacert /trust/engine/ca.crt \ + --connect-to "$SERVING:3473:$HOST:3473" "https://$SERVING:3473/health/ready" + pass "Engine serving identity verified: $SERVING on $HOST" + {{- end }} {{- end }} {{- end }} {{- end }} From 3fa2fa637d9d4d8b4abc8cb84e61422c08dad97c Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Fri, 2 Oct 2026 10:38:07 +0200 Subject: [PATCH 4/5] fix(tls): use workload annotations and explicit restarts (FB-4386) --- docs/usage/tls.mdx | 43 +++++++++++++++++++++----- helm/AGENTS.md | 2 +- helm/README.md | 4 +-- helm/scripts/test_render.py | 16 ++++++---- helm/templates/engine-statefulset.yaml | 5 ++- helm/templates/gateway-deployment.yaml | 5 ++- helm/values.schema.json | 14 ++++++--- helm/values.yaml | 10 +++--- 8 files changed, 73 insertions(+), 26 deletions(-) diff --git a/docs/usage/tls.mdx b/docs/usage/tls.mdx index 7103e8e..8544010 100644 --- a/docs/usage/tls.mdx +++ b/docs/usage/tls.mdx @@ -99,19 +99,48 @@ The issuer behind `tls.engine.certManager` must populate `ca.crt` in the Certifi Secret updates do not independently trigger Helm upgrades. Engines load serving certificates at startup, and their full certificate chain is assembled by an init container. Treat certificate replacement and workload restart as separate steps: 1. Renew or replace the certificate Secret, preserving its required keys and SANs. For cert-manager, wait for the replacement Certificate revision to be issued and its Secret to contain the new certificate. Merely observing an old `Ready=True` condition does not establish that renewal finished. -2. For a same-CA Engine renewal, change `engineSpec.restartToken`. For a Gateway certificate or upstream trust change, change `gateway.restartToken`. Upgrade with the full deployment values and wait for every affected workload to finish rolling out. -3. Run the verified TLS hooks described below and confirm the served certificate is the renewed certificate before its predecessor expires. +2. Restart affected workloads after the replacement Secret exists. For a same-CA Engine renewal, restart every Engine StatefulSet. For a Gateway serving-certificate change, restart the Gateway Deployment. If Engine CA trust changes, coordinate restarts of both Engines and the Gateway using a staged trust migration. +3. Wait for the rollouts, run the verified TLS hooks described below, and confirm the served certificate is the renewed certificate before its predecessor expires. -Use a new arbitrary, non-secret token for each restart: +For the `firebolt` release in the `firebolt` namespace, restart Engine nodes one StatefulSet at a time during a maintenance window: + +```bash +# Stop if discovery or any rollout fails. +set -e +workloads=$(kubectl get statefulsets --namespace firebolt \ + --selector app.kubernetes.io/instance=firebolt,firebolt/component=engine -o name) +for workload in $workloads; do + kubectl rollout restart "$workload" --namespace firebolt + kubectl rollout status "$workload" --namespace firebolt --timeout=10m +done +``` + +Run the loop in Bash. Waiting between nodes avoids initiating all restarts together, but does not guarantee uninterrupted distributed queries. Change the release and namespace in these examples to match your deployment. + +After replacing a Gateway certificate, restart its Deployment: + +```bash +kubectl rollout restart deployment/firebolt-gateway --namespace firebolt +kubectl rollout status deployment/firebolt-gateway --namespace firebolt --timeout=10m +``` + +Online Helm rendering hashes current auth/TLS Secrets as an additional rollout trigger. It cannot hash a future Secret that cert-manager has not issued yet. Offline rendering cannot read live Secrets. Neither mechanism watches for changes between Helm operations. + +### Optional automatic restarts + +If your platform already runs [Stakater Reloader](https://github.com/stakater/Reloader), you can opt workloads into Secret-triggered rollouts. The chart does not install Reloader. Its annotations belong on Deployment or StatefulSet metadata, not on pod metadata. + +For example, enable Secret-triggered Gateway restarts through workload annotations: ```yaml -engineSpec: - restartToken: certificate-renewal-2 gateway: - restartToken: certificate-renewal-2 + annotations: + secret.reloader.stakater.com/auto: "true" ``` -Online Helm rendering hashes the current auth/TLS Secrets as an additional restart trigger. It cannot hash a future Secret that cert-manager has not issued yet. Offline and GitOps rendering cannot read live Secrets at all; use the explicit restart tokens after issuance in either deployment mode. The chart does not install a reload controller. +The same annotation can be set under `engineSpec.annotations` for Engine StatefulSets or `metadata.annotations` for the Metadata Service. Automatic reload watches referenced Secrets, including those updated by cert-manager. It does not stage CA trust changes, rotate database passwords, or coordinate signing-key promotion. + +Each Engine node has its own single-replica StatefulSet. A shared Secret update can trigger concurrent restarts across those StatefulSets. Use manual restarts when you need to control that sequence. Reloader's rolling updates do not provide cross-node coordination or a query-availability guarantee. Configure annotation ownership and rollout strategy according to your Reloader and GitOps controller configuration. A CA change requires overlapping trust and coordinated restarts. Do not replace the only trusted CA and both leaf certificates in a single upgrade. The chart does not automatically stage trust bundles or gate traffic cutover. Plan a maintenance window, or manage a separately validated staged migration outside the chart. diff --git a/helm/AGENTS.md b/helm/AGENTS.md index 628139a..be51f8b 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -87,4 +87,4 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - Symptom: `make test` (i.e. `helm test --logs`) prints `Phase: Succeeded` for every suite and then exits 1 with `unable to get pod logs ... pods "-test-" not found`. Cause: a `helm.sh/hook-delete-policy` of `before-hook-creation,hook-succeeded` on the test pod tells helm to delete the pod the moment the hook succeeds, so by the time `helm test --logs` iterates the pods to fetch their logs they are already gone (helm/helm#8949). Resolution: test pods MUST set `helm.sh/hook-delete-policy: before-hook-creation` only — never include `hook-succeeded`. Stale pods are still cleaned up on the next `helm test` run by `before-hook-creation`, and `make test-cleanup` deletes them on demand. -Auth/TLS Secret hashes are evaluated only during online Helm rendering. For offline/GitOps rendering, change the component restart token after new Secret material exists. `tests/test-tls.yaml` verifies every Engine node with the CA and optionally verifies the Gateway using `tls.gateway.verification`; it never disables certificate verification. +Auth/TLS Secret hashes are evaluated only during online Helm rendering. After replacing Secret material, explicitly restart affected workloads or configure an external reload controller. `engineSpec.annotations`, `gateway.annotations`, and `metadata.annotations` annotate workload resources, not pods. Each Engine node is an independent StatefulSet; automatic Secret-triggered rollouts do not coordinate across nodes. `tests/test-tls.yaml` verifies every Engine node with the CA and optionally verifies the Gateway using `tls.gateway.verification`; it never disables certificate verification. diff --git a/helm/README.md b/helm/README.md index 5a14b77..597606d 100644 --- a/helm/README.md +++ b/helm/README.md @@ -28,6 +28,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | customEngineConfig.instance.id | string | `"01kp98j0000000000000000000"` | Lowercase Crockford ULID for the Firebolt instance. Must match the account reconciled by the metadata service at startup. `values.schema.json` rejects uppercase and any character outside the Crockford alphabet. | | engineSpec | object | {} | Shared engine pod defaults applied to all engines unless overridden per-engine. | | engineSpec.affinity | object | `{}` | Affinity rules for engine pod scheduling. | +| engineSpec.annotations | object | {} | Annotations on Engine StatefulSets, for example to opt into an externally installed Secret reload controller. These do not annotate pods. | | engineSpec.customInitContainersTemplate | list | `[]` | Custom init containers injected into engine pods (supports templating). | | engineSpec.customVolumeMounts | list | `[]` | Custom volume mounts injected into the engine `core` container, paired with `customVolumes` above — a volume declared there is inert until also mounted here. | | engineSpec.customVolumes | list | `[]` | Custom volumes injected into engine pods. | @@ -48,7 +49,6 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | engineSpec.podSecurityContext.fsGroupChangePolicy | string | `"OnRootMismatch"` | When to re-apply `fsGroup` ownership. `OnRootMismatch` skips the chown when already correct — much faster on large PVCs. | | engineSpec.podSecurityContext.runAsNonRoot | bool | `true` | Reject the pod if any container runs as UID 0. Also gates the chart's container-level `runAs*` defaults, memlock init, and the entrypoint UID check. | | engineSpec.readiness | bool | `true` | When true, a readiness probe is added to the core container. | -| engineSpec.restartToken | string | `""` | Change after Secret rotation to restart Engines, including offline/GitOps rendering. Wait for replacement Secret material before upgrading. | | engineSpec.serviceAccount | string | `""` | ServiceAccount used by engine pods. Empty (the default): the chart creates `-engine` with `automountServiceAccountToken: false`, so a code-execution exploit in the engine container has no SA token to talk to the apiserver with. Engines do not call the Kubernetes API; the dedicated SA replaces the namespace `default` SA, which automounts a token and inherits any RoleBindings accumulated on `default` from unrelated installs. Non-empty: the chart references the named SA verbatim and does NOT create one — bring your own (the documented IRSA / Pod Identity flow at docs/usage/object-storage/amazon-s3.mdx works this way). The chart cannot influence `automountServiceAccountToken` on a SA it does not own; set it explicitly in your SA manifest if you want the same hardening. | | engineSpec.storageHostPath | object | {} | Host path configuration used when `hostPathStorageEnabled` is true. | | engineSpec.storageHostPath.path | string | `"/var/lib/firebolt-core"` | Host path for engine data. | @@ -75,6 +75,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | extraLabels | object | `{"firebolt/product":"core"}` | Extra labels applied to all resources and pods. | | gateway | object | {} | Envoy gateway proxy configuration. Routes queries to engine Services based on the `X-Firebolt-Engine` HTTP header. A Lua filter extracts the engine name and rewrites the upstream to `{engine}-service:3473` via dynamic forward proxy. | | gateway.adminPort | int | `9901` | Envoy admin interface port (used for health checks). | +| gateway.annotations | object | {} | Annotations on the Gateway Deployment, for example to opt into an externally installed Secret reload controller. These do not annotate pods. | | gateway.containerPort | int | `8080` | Envoy listener port for client traffic. | | gateway.enabled | bool | `true` | Set to true to deploy the Envoy gateway proxy. | | gateway.image.pullPolicy | string | `"IfNotPresent"` | Image pull policy. | @@ -105,7 +106,6 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | gateway.podTemplate.volumes | list | `[]` | Extra volumes added to the gateway pod, merged with the chart-managed volumes. Mount them on the Envoy container via `volumeMounts` below. | | gateway.replicas | int | `2` | Number of gateway replicas. | | gateway.resources | object | `{"limits":{"memory":"512Mi"},"requests":{"cpu":"100m","memory":"256Mi"}}` | Resource requests and limits for the Envoy container. | -| gateway.restartToken | string | `""` | Change after certificate or trust-bundle rotation to restart the Gateway, including offline/GitOps rendering. | | gateway.service | object | {} | Gateway Service configuration. | | gateway.service.port | int | `80` | External service port proxied to `containerPort`. | | gateway.service.type | string | `"ClusterIP"` | Service type. One of `ClusterIP`, `LoadBalancer`, or `NodePort`. | diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py index 9d424a3..ba56c01 100644 --- a/helm/scripts/test_render.py +++ b/helm/scripts/test_render.py @@ -146,19 +146,23 @@ def test_reject_invalid_combinations(self): class CertificateLifecycleTests(unittest.TestCase): - def test_restart_tokens_are_component_scoped(self): + def test_workload_annotations_are_component_scoped(self): baseline = render() - def pod(docs, kind, name): - return resource(docs, kind, name)["spec"]["template"] + custom = {"secret.reloader.stakater.com/auto": "true"} cases = [("engineSpec", "StatefulSet", "audit-engine-default-node-0"), ("gateway", "Deployment", "audit-gateway"), ("metadata", "Deployment", "audit-metadata-service")] for component, kind, name in cases: - docs = render({component: {"restartToken": "rotation-2"}}) - self.assertNotEqual(pod(baseline, kind, name), pod(docs, kind, name)) + docs = render({component: {"annotations": custom}}) + updated = resource(docs, kind, name) + original = resource(baseline, kind, name) + self.assertEqual(updated["metadata"]["annotations"], custom) + self.assertEqual(updated["spec"]["template"], original["spec"]["template"]) + self.assertNotIn("firebolt.io/restart-token", original["spec"]["template"]["metadata"]["annotations"]) for other, other_kind, other_name in cases: if other != component: - self.assertEqual(pod(baseline, other_kind, other_name), pod(docs, other_kind, other_name)) + self.assertEqual(resource(baseline, other_kind, other_name), resource(docs, other_kind, other_name)) + render({component: {"annotations": {"invalid": True}}}, expect_error=True) def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self): values = {"engines": [{"name": "default", "replicas": 2}], "tls": { diff --git a/helm/templates/engine-statefulset.yaml b/helm/templates/engine-statefulset.yaml index 5e09ca6..45503c1 100644 --- a/helm/templates/engine-statefulset.yaml +++ b/helm/templates/engine-statefulset.yaml @@ -37,6 +37,10 @@ kind: StatefulSet metadata: namespace: {{ $.Release.Namespace }} name: {{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }} + {{- with $.Values.engineSpec.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} labels: {{- include "fbinstance.labels" $ | nindent 4 }} firebolt/component: engine @@ -58,7 +62,6 @@ spec: template: metadata: annotations: - firebolt.io/restart-token: {{ $.Values.engineSpec.restartToken | quote }} # Roll this engine's pods when its own config.yaml changes. The # config hash is scoped to this engine so a change to one engine # doesn't restart the others. diff --git a/helm/templates/gateway-deployment.yaml b/helm/templates/gateway-deployment.yaml index 8169116..b248373 100644 --- a/helm/templates/gateway-deployment.yaml +++ b/helm/templates/gateway-deployment.yaml @@ -16,6 +16,10 @@ kind: Deployment metadata: name: {{ include "fbinstance.fullname" . }}-gateway namespace: {{ .Release.Namespace }} + {{- with .Values.gateway.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} labels: {{- include "fbinstance.labels" . | nindent 4 }} firebolt/component: gateway @@ -33,7 +37,6 @@ spec: template: metadata: annotations: - firebolt.io/restart-token: {{ .Values.gateway.restartToken | quote }} checksum/config: {{ include (print $.Template.BasePath "/gateway-configmap.yaml") . | sha256sum }} {{- /* See the equivalent comment in engine-statefulset.yaml: these Secrets diff --git a/helm/values.schema.json b/helm/values.schema.json index 6803ba8..9b9cf07 100644 --- a/helm/values.schema.json +++ b/helm/values.schema.json @@ -63,8 +63,11 @@ } } }, - "restartToken": { - "type": "string" + "annotations": { + "type": "object", + "additionalProperties": { + "type": "string" + } } } }, @@ -187,8 +190,11 @@ "type": "integer", "minimum": 10 }, - "restartToken": { - "type": "string" + "annotations": { + "type": "object", + "additionalProperties": { + "type": "string" + } } } }, diff --git a/helm/values.yaml b/helm/values.yaml index 66a9b45..de52f64 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -18,8 +18,9 @@ createNamespace: false # -- Shared engine pod defaults applied to all engines unless overridden per-engine. # @default -- {} engineSpec: - # -- Change after Secret rotation to restart Engines, including offline/GitOps rendering. Wait for replacement Secret material before upgrading. - restartToken: "" + # -- Annotations on Engine StatefulSets, for example to opt into an externally installed Secret reload controller. These do not annotate pods. + # @default -- {} + annotations: {} image: # -- Container repository for the Firebolt engine image. repository: oci.firebolt.io/firebolt-db/engine @@ -333,8 +334,9 @@ postgresql: # the upstream to `{engine}-service:3473` via dynamic forward proxy. # @default -- {} gateway: - # -- Change after certificate or trust-bundle rotation to restart the Gateway, including offline/GitOps rendering. - restartToken: "" + # -- Annotations on the Gateway Deployment, for example to opt into an externally installed Secret reload controller. These do not annotate pods. + # @default -- {} + annotations: {} # -- Total pod shutdown budget in seconds, including a 5s endpoint-propagation wait and explicit connection draining. Requests still active at the deadline can be interrupted. terminationGracePeriodSeconds: 15 # -- Set to true to deploy the Envoy gateway proxy. From b49fb4ca7bcc22da870968c6ab9f991fb36d0a12 Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Fri, 2 Oct 2026 11:06:08 +0200 Subject: [PATCH 5/5] docs(tls): present reloader as an optional example (FB-4386) --- docs/usage/tls.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/usage/tls.mdx b/docs/usage/tls.mdx index 8544010..f4da730 100644 --- a/docs/usage/tls.mdx +++ b/docs/usage/tls.mdx @@ -128,9 +128,9 @@ Online Helm rendering hashes current auth/TLS Secrets as an additional rollout t ### Optional automatic restarts -If your platform already runs [Stakater Reloader](https://github.com/stakater/Reloader), you can opt workloads into Secret-triggered rollouts. The chart does not install Reloader. Its annotations belong on Deployment or StatefulSet metadata, not on pod metadata. +After replacing a Secret, restart affected workloads explicitly or use your platform's Secret-change automation. The chart does not install or require a reload controller. -For example, enable Secret-triggered Gateway restarts through workload annotations: +The following example configures Secret-triggered Gateway restarts with [Stakater Reloader](https://github.com/stakater/Reloader). Its annotations belong on Deployment or StatefulSet metadata, not on pod metadata: ```yaml gateway: