Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .legitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,7 @@ value_67A867BC754C44390E7B54969C75EEE8 # Example Kubernetes Secret name "firebol
value_8FAFF5AF6AA62BB96EAF8C4A1C3ED99F # Example Kubernetes Secret name "firebolt-signing-key-2026-01" in docs/usage/authentication.mdx — a resource reference, not a credential value.
value_0BAC68E67DA5B3BED13A4A58E76B535A # Dummy postgresql.password used by helm/scripts/validate-chart.sh to satisfy the values schema during template rendering — not a credential.
value_01kp98j0000000000000000000 # Dummy customEngineConfig.instance.id (lowercase Crockford ULID placeholder in values.yaml / helm-docs / schema tests) — not a credential.

# Historical documentation examples: public rollout markers, never credentials.
instance_D0360DAADFA20D225333000C4486F464 # docs/usage/tls.mdx:96 in afafe63
instance_84F2094DD13F984A9CD7224F6E08C27E # docs/usage/tls.mdx:98 in afafe63
72 changes: 72 additions & 0 deletions docs/usage/tls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,78 @@ The engine's certificate's DNS names are derived automatically from every engine
The issuer behind `tls.engine.certManager` must populate `ca.crt` in the Certificate's Secret, not just `tls.crt`/`tls.key`. CA and Vault issuers do this; a bare self-signed or ACME issuer may not. The chart's `tls-chain-setup` init container needs `ca.crt` to build the engine's own health-check trust bundle and fails the pod at startup if it's missing.
</Note>

### Renewal and restart procedure

Secret updates do not independently trigger Helm upgrades. Engines load serving certificates at startup, and their full certificate chain is assembled by an init container. Treat certificate replacement and workload restart as separate steps:

1. Renew or replace the certificate Secret, preserving its required keys and SANs. For cert-manager, wait for the replacement Certificate revision to be issued and its Secret to contain the new certificate. Merely observing an old `Ready=True` condition does not establish that renewal finished.
2. Restart affected workloads after the replacement Secret exists. For a same-CA Engine renewal, restart every Engine StatefulSet. For a Gateway serving-certificate change, restart the Gateway Deployment. If Engine CA trust changes, coordinate restarts of both Engines and the Gateway using a staged trust migration.
3. Wait for the rollouts, run the verified TLS hooks described below, and confirm the served certificate is the renewed certificate before its predecessor expires.

For the `firebolt` release in the `firebolt` namespace, restart Engine nodes one StatefulSet at a time during a maintenance window:

```bash
# Stop if discovery or any rollout fails.
set -e
workloads=$(kubectl get statefulsets --namespace firebolt \
--selector app.kubernetes.io/instance=firebolt,firebolt/component=engine -o name)
for workload in $workloads; do
kubectl rollout restart "$workload" --namespace firebolt
kubectl rollout status "$workload" --namespace firebolt --timeout=10m
done
```

Run the loop in Bash. Waiting between nodes avoids initiating all restarts together, but does not guarantee uninterrupted distributed queries. Change the release and namespace in these examples to match your deployment.

After replacing a Gateway certificate, restart its Deployment:

```bash
kubectl rollout restart deployment/firebolt-gateway --namespace firebolt
kubectl rollout status deployment/firebolt-gateway --namespace firebolt --timeout=10m
```

Online Helm rendering hashes current auth/TLS Secrets as an additional rollout trigger. It cannot hash a future Secret that cert-manager has not issued yet. Offline rendering cannot read live Secrets. Neither mechanism watches for changes between Helm operations.

### Optional automatic restarts

After replacing a Secret, restart affected workloads explicitly or use your platform's Secret-change automation. The chart does not install or require a reload controller.

The following example configures Secret-triggered Gateway restarts with [Stakater Reloader](https://github.com/stakater/Reloader). Its annotations belong on Deployment or StatefulSet metadata, not on pod metadata:

```yaml
gateway:
annotations:
secret.reloader.stakater.com/auto: "true"
```

The same annotation can be set under `engineSpec.annotations` for Engine StatefulSets or `metadata.annotations` for the Metadata Service. Automatic reload watches referenced Secrets, including those updated by cert-manager. It does not stage CA trust changes, rotate database passwords, or coordinate signing-key promotion.

Each Engine node has its own single-replica StatefulSet. A shared Secret update can trigger concurrent restarts across those StatefulSets. Use manual restarts when you need to control that sequence. Reloader's rolling updates do not provide cross-node coordination or a query-availability guarantee. Configure annotation ownership and rollout strategy according to your Reloader and GitOps controller configuration.

A CA change requires overlapping trust and coordinated restarts. Do not replace the only trusted CA and both leaf certificates in a single upgrade. The chart does not automatically stage trust bundles or gate traffic cutover. Plan a maintenance window, or manage a separately validated staged migration outside the chart.

Enabling TLS on an existing plaintext release does not immediately close old plaintext listeners. During a rolling update, old Gateway pods can continue serving while replacement pods wait for certificates. Restrict external access during a security-posture change and verify that all old pods have terminated before restoring it. Engine TLS protocol changes also require a maintenance window because Engine and Gateway updates are independent.

### Verify certificate trust

When Engine TLS is enabled, `helm test` checks each Engine node against its node hostname and, when the Gateway is enabled, its ready-only serving hostname. These checks never disable certificate verification. To verify the Gateway as well, supply a certificate SAN as `serverName`. The test connects to the internal Service while verifying that hostname. For a private CA, supply a Secret containing `ca.crt`:

```yaml
tls:
gateway:
verification:
serverName: firebolt.example.com
caSecret: gateway-test-ca
```

Run all chart hooks after the rollout:

```bash
helm test firebolt --namespace firebolt --logs
```

An empty Gateway `serverName` skips its verified handshake test. Existing readiness and SQL hooks test functionality and do not establish Gateway certificate trust. The verified TLS hook projects only CA material and never mounts signing keys or TLS private keys.

### Caveats

- The `core-ui` sidecar (`engineSpec.uiSidecar`) follows `tls.engine.enabled` automatically, proxying to the engine over HTTPS without verifying its certificate — the connection is over loopback within the pod, so this doesn't need a trusted CA.
Expand Down
2 changes: 2 additions & 0 deletions helm/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,5 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g
## Known issues

- Symptom: `make test` (i.e. `helm test --logs`) prints `Phase: Succeeded` for every suite and then exits 1 with `unable to get pod logs ... pods "<release>-test-<name>" not found`. Cause: a `helm.sh/hook-delete-policy` of `before-hook-creation,hook-succeeded` on the test pod tells helm to delete the pod the moment the hook succeeds, so by the time `helm test --logs` iterates the pods to fetch their logs they are already gone (helm/helm#8949). Resolution: test pods MUST set `helm.sh/hook-delete-policy: before-hook-creation` only — never include `hook-succeeded`. Stale pods are still cleaned up on the next `helm test` run by `before-hook-creation`, and `make test-cleanup` deletes them on demand.

Auth/TLS Secret hashes are evaluated only during online Helm rendering. After replacing Secret material, explicitly restart affected workloads or configure an external reload controller. `engineSpec.annotations`, `gateway.annotations`, and `metadata.annotations` annotate workload resources, not pods. Each Engine node is an independent StatefulSet; automatic Secret-triggered rollouts do not coordinate across nodes. `tests/test-tls.yaml` verifies every Engine node with the CA and optionally verifies the Gateway using `tls.gateway.verification`; it never disables certificate verification.
5 changes: 5 additions & 0 deletions helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines
| customEngineConfig.instance.id | string | `"01kp98j0000000000000000000"` | Lowercase Crockford ULID for the Firebolt instance. Must match the account reconciled by the metadata service at startup. `values.schema.json` rejects uppercase and any character outside the Crockford alphabet. |
| engineSpec | object | {} | Shared engine pod defaults applied to all engines unless overridden per-engine. |
| engineSpec.affinity | object | `{}` | Affinity rules for engine pod scheduling. |
| engineSpec.annotations | object | {} | Annotations on Engine StatefulSets, for example to opt into an externally installed Secret reload controller. These do not annotate pods. |
| engineSpec.customInitContainersTemplate | list | `[]` | Custom init containers injected into engine pods (supports templating). |
| engineSpec.customVolumeMounts | list | `[]` | Custom volume mounts injected into the engine `core` container, paired with `customVolumes` above — a volume declared there is inert until also mounted here. |
| engineSpec.customVolumes | list | `[]` | Custom volumes injected into engine pods. |
Expand Down Expand Up @@ -74,6 +75,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines
| extraLabels | object | `{"firebolt/product":"core"}` | Extra labels applied to all resources and pods. |
| gateway | object | {} | Envoy gateway proxy configuration. Routes queries to engine Services based on the `X-Firebolt-Engine` HTTP header. A Lua filter extracts the engine name and rewrites the upstream to `{engine}-service:3473` via dynamic forward proxy. |
| gateway.adminPort | int | `9901` | Envoy admin interface port (used for health checks). |
| gateway.annotations | object | {} | Annotations on the Gateway Deployment, for example to opt into an externally installed Secret reload controller. These do not annotate pods. |
| gateway.containerPort | int | `8080` | Envoy listener port for client traffic. |
| gateway.enabled | bool | `true` | Set to true to deploy the Envoy gateway proxy. |
| gateway.image.pullPolicy | string | `"IfNotPresent"` | Image pull policy. |
Expand Down Expand Up @@ -187,6 +189,9 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines
| tls.gateway.enabled | bool | `false` | Enable TLS on the gateway's client-facing listener. |
| tls.gateway.existingSecret | object | {} | Existing `kubernetes.io/tls` Secret (`tls.crt` / `tls.key`). |
| tls.gateway.existingSecret.secretRef | string | `""` | Secret name. |
| tls.gateway.verification | object | {} | Optional verified Gateway TLS Helm test. Set serverName to a certificate SAN; caSecret is optional for publicly trusted certificates. |
| tls.gateway.verification.caSecret | string | `""` | Optional Secret containing ca.crt for the Gateway verification hook. Only the CA key is projected. |
| tls.gateway.verification.serverName | string | `""` | Hostname to verify while connecting to the in-cluster Gateway Service. Empty skips the Gateway verification hook; Engine TLS verification remains automatic. |
| utilitiesImage | string | `"debian:stable-slim@sha256:5012d0517aa0075a7150a45aae67586641e898913b7af3b08228108565b5f90c"` | Image used for utility init/sidecar containers (e.g. the memlock-setup sidecar). Pinned to an immutable digest so a registry-side tag override cannot silently change what runs in production. Bump the digest together with the tag when upgrading. |

----------------------------------------------
Expand Down
42 changes: 42 additions & 0 deletions helm/scripts/test_render.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,5 +145,47 @@ def test_reject_invalid_combinations(self):
self.assertIn(message, render(values, expect_error=True))


class CertificateLifecycleTests(unittest.TestCase):
def test_workload_annotations_are_component_scoped(self):
baseline = render()
custom = {"secret.reloader.stakater.com/auto": "true"}
cases = [("engineSpec", "StatefulSet", "audit-engine-default-node-0"),
("gateway", "Deployment", "audit-gateway"),
("metadata", "Deployment", "audit-metadata-service")]
for component, kind, name in cases:
docs = render({component: {"annotations": custom}})
updated = resource(docs, kind, name)
original = resource(baseline, kind, name)
self.assertEqual(updated["metadata"]["annotations"], custom)
self.assertEqual(updated["spec"]["template"], original["spec"]["template"])
self.assertNotIn("firebolt.io/restart-token", original["spec"]["template"]["metadata"]["annotations"])
for other, other_kind, other_name in cases:
if other != component:
self.assertEqual(resource(baseline, other_kind, other_name), resource(docs, other_kind, other_name))
render({component: {"annotations": {"invalid": True}}}, expect_error=True)

def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self):
values = {"engines": [{"name": "default", "replicas": 2}], "tls": {
"engine": {"enabled": True, "existingSecret": {"secretRef": "engine-tls"}},
"gateway": {"enabled": True, "existingSecret": {"secretRef": "gateway-tls"},
"verification": {"serverName": "firebolt.example.com", "caSecret": "gateway-ca"}},
}}
hook = resource(render(values), "Pod", "audit-test-tls")["spec"]
script = hook["containers"][0]["command"][-1]
self.assertNotIn(" -k", script)
self.assertNotIn("--insecure", script)
self.assertIn("--cacert /trust/engine/ca.crt", script)
self.assertEqual(script.count('--connect-to "$SERVING:3473:$HOST:3473"'), 2)
self.assertIn("audit-engine-default-ready.audit.svc.cluster.local", script)
self.assertIn("--connect-to", script)
self.assertIn("firebolt.example.com", script)
for volume in hook["volumes"]:
self.assertEqual(volume["secret"]["items"], [{"key": "ca.crt", "path": "ca.crt"}])
self.assertFalse(any(d["metadata"]["name"] == "audit-test-tls" for d in render()))
values["gateway"] = {"enabled": False}
direct = resource(render(values), "Pod", "audit-test-tls")["spec"]["containers"][0]["command"][-1]
self.assertNotIn("SERVING=", direct)


if __name__ == "__main__":
unittest.main()
4 changes: 4 additions & 0 deletions helm/templates/engine-statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ kind: StatefulSet
metadata:
namespace: {{ $.Release.Namespace }}
name: {{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}
{{- with $.Values.engineSpec.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "fbinstance.labels" $ | nindent 4 }}
firebolt/component: engine
Expand Down
4 changes: 4 additions & 0 deletions helm/templates/gateway-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ kind: Deployment
metadata:
name: {{ include "fbinstance.fullname" . }}-gateway
namespace: {{ .Release.Namespace }}
{{- with .Values.gateway.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "fbinstance.labels" . | nindent 4 }}
firebolt/component: gateway
Expand Down
85 changes: 85 additions & 0 deletions helm/templates/tests/test-tls.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
{{- $gatewayTest := and .Values.gateway.enabled .Values.tls.gateway.enabled .Values.tls.gateway.verification.serverName -}}
{{- if or .Values.tls.engine.enabled $gatewayTest }}
apiVersion: v1
kind: Pod
metadata:
name: {{ include "fbinstance.fullname" . }}-test-tls
namespace: {{ .Release.Namespace }}
annotations:
helm.sh/hook: test
helm.sh/hook-delete-policy: before-hook-creation
spec:
restartPolicy: Never
automountServiceAccountToken: false
{{- if or .Values.tls.engine.enabled (and $gatewayTest .Values.tls.gateway.verification.caSecret) }}
volumes:
{{- if .Values.tls.engine.enabled }}
- name: engine-ca
secret:
secretName: {{ include "fbinstance.secretSourceName" (dict "source" .Values.tls.engine "certManagerSecretName" (printf "%s-tls-engine" (include "fbinstance.fullname" .)) "context" "tls.engine") }}
items:
- key: ca.crt
path: ca.crt
{{- end }}
{{- if and $gatewayTest .Values.tls.gateway.verification.caSecret }}
- name: gateway-ca
secret:
secretName: {{ .Values.tls.gateway.verification.caSecret }}
items:
- key: ca.crt
path: ca.crt
{{- end }}
{{- end }}
containers:
- name: check
image: curlimages/curl:latest
{{- if or .Values.tls.engine.enabled (and $gatewayTest .Values.tls.gateway.verification.caSecret) }}
volumeMounts:
{{- if .Values.tls.engine.enabled }}
- name: engine-ca
mountPath: /trust/engine
readOnly: true
{{- end }}
{{- if and $gatewayTest .Values.tls.gateway.verification.caSecret }}
- name: gateway-ca
mountPath: /trust/gateway
readOnly: true
{{- end }}
{{- end }}
command:
- sh
- -c
- |
set -eu
{{- include "fbinstance.testShellHelpers" . | nindent 10 }}
{{- if .Values.tls.engine.enabled }}
{{- range $engine := .Values.engines }}
{{- range $i := until (int $engine.replicas) }}
HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $.Values.engineSpec.nodeHostSuffix }}"
log "Verify Engine TLS chain and hostname: $HOST"
curl --fail --silent --show-error --max-time 30 \
--cacert /trust/engine/ca.crt "https://$HOST:3473/health/ready"
pass "Engine TLS verified: $HOST"
{{- if $.Values.gateway.enabled }}
# Check the Gateway's serving identity against every node, not just
# the node identity. Old certificates may cover only the latter.
SERVING="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-ready.{{ $.Release.Namespace }}.svc.cluster.local"
curl --fail --silent --show-error --max-time 30 \
--cacert /trust/engine/ca.crt \
--connect-to "$SERVING:3473:$HOST:3473" "https://$SERVING:3473/health/ready"
pass "Engine serving identity verified: $SERVING on $HOST"
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{- if $gatewayTest }}
log "Verify Gateway TLS chain and hostname"
curl --fail --silent --show-error --max-time 30 \
{{- if .Values.tls.gateway.verification.caSecret }}
--cacert /trust/gateway/ca.crt \
{{- end }}
--connect-to "{{ .Values.tls.gateway.verification.serverName }}:{{ .Values.gateway.service.port }}:{{ include "fbinstance.fullname" . }}-gateway.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.gateway.service.port }}" \
"https://{{ .Values.tls.gateway.verification.serverName }}:{{ .Values.gateway.service.port }}/healthz"
pass "Gateway TLS verified"
{{- end }}
{{- end }}
24 changes: 24 additions & 0 deletions helm/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@
]
}
}
},
"annotations": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
},
Expand Down Expand Up @@ -183,6 +189,12 @@
"terminationGracePeriodSeconds": {
"type": "integer",
"minimum": 10
},
"annotations": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
},
Expand Down Expand Up @@ -347,6 +359,18 @@
},
"certManager": {
"$ref": "#/definitions/certManagerSource"
},
"verification": {
"type": "object",
"properties": {
"serverName": {
"type": "string",
"pattern": "^([A-Za-z0-9]([A-Za-z0-9.-]*[A-Za-z0-9])?)?$"
},
"caSecret": {
"type": "string"
}
}
}
}
},
Expand Down
Loading