From 81adc9e978a55d938ad87536cee2bfb25c61da2a Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Tue, 29 Sep 2026 17:26:28 +0200 Subject: [PATCH 1/3] fix(gateway): prevent replay of delivered queries (FB-4383) Restrict retries to pre-delivery failures and explicit drained responses. Document staged signing-key distribution and promotion to keep token verification compatible across rolling pods. --- Makefile | 8 ++ docs/usage/authentication.mdx | 31 ++++++-- helm/AGENTS.md | 4 +- helm/README.md | 2 +- helm/scripts/test_gateway_runtime.py | 107 ++++++++++++++++++++++++++ helm/scripts/test_render.py | 48 ++++++++++++ helm/scripts/validate-chart.sh | 3 + helm/templates/gateway-configmap.yaml | 6 +- helm/values.yaml | 9 ++- 9 files changed, 204 insertions(+), 14 deletions(-) create mode 100644 helm/scripts/test_gateway_runtime.py create mode 100644 helm/scripts/test_render.py diff --git a/Makefile b/Makefile index f688c1e..fd04d6b 100644 --- a/Makefile +++ b/Makefile @@ -40,6 +40,8 @@ THOROUGH ?= false docs \ docs-check \ lint \ + test-render \ + test-gateway-runtime \ create \ delete \ floci \ @@ -110,6 +112,12 @@ lint: ## Lint and template-render the helm chart helm template $(RELEASE) $(CHART) > /dev/null @echo "All helm checks passed." +test-render: ## Run deterministic chart regression tests (Python + PyYAML) + $(or $(PYTHON),python3) helm/scripts/test_render.py + +test-gateway-runtime: ## Exercise Envoy retries against a fault-injecting backend (Docker + PyYAML) + $(or $(PYTHON),python3) helm/scripts/test_gateway_runtime.py + ##@ Local kind cluster (single-node dev) create: ## Create a local kind cluster diff --git a/docs/usage/authentication.mdx b/docs/usage/authentication.mdx index 0f3c80e..68ba428 100644 --- a/docs/usage/authentication.mdx +++ b/docs/usage/authentication.mdx @@ -55,19 +55,38 @@ Kubernetes health probes and the gateway's active engine health check keep worki ### Signing key rotation -`signingKeys` is an ordered list. The **first** entry is the active signer used for new tokens; every entry in the list remains valid for verifying tokens already issued under it. To rotate, prepend a new entry rather than replacing the list, and remove the old entry only once every token it signed has expired: +`signingKeys` is ordered. The first entry signs new tokens, and every entry verifies tokens. Rotate in separate upgrades so every node learns the new key before any node signs with it: + +1. Create a new Secret or cert-manager entry with a fresh key ID. **Append** it after the current signer. Upgrade and wait for every Engine StatefulSet to finish rolling out. With cert-manager, wait for the new Secret to be issued as well. +2. Move the new entry to the first position. Upgrade and wait for every Engine StatefulSet to finish rolling out again. +3. From the completion of the second rollout, retain the old key for the maximum lifetime of tokens it could have issued, including clock-skew tolerance. If that lifetime is unknown or unbounded, do not remove the key until you have established a safe retention policy. +4. Remove the old entry, upgrade, and verify authentication across every Engine. + +Do not replace key material in an existing Secret or prepend a new signer in a single upgrade. An updated node could issue tokens that an older node cannot verify. These steps preserve token verification compatibility; Engine restarts can still interrupt queries. + +During the first upgrade, keep the old signer first: ```yaml auth: signingKeys: - - id: key-2026-02 # now active - existingSecret: - secretRef: firebolt-signing-key-2026-02 - - id: key-2026-01 # still verifies old tokens until they expire + - id: key-2026-01 existingSecret: secretRef: firebolt-signing-key-2026-01 + - id: key-2026-02 + existingSecret: + secretRef: firebolt-signing-key-2026-02 +``` + +After each upgrade, wait for all Engine StatefulSets in the release: + +```bash +kubectl rollout status statefulset --namespace firebolt \ + --selector app.kubernetes.io/instance=firebolt,firebolt/component=engine \ + --timeout=10m ``` +Check tokens issued by different nodes against every other node before promotion and after removal. A successful login against one node does not establish fleet-wide compatibility. + ### cert-manager as a signing-key source As an alternative to an existing Secret, a signing key entry can request a [cert-manager](https://cert-manager.io) `Certificate` instead: @@ -84,7 +103,7 @@ auth: kind: ClusterIssuer ``` -The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Rotate by prepending a new `signingKeys` entry, the same as with an existing Secret. +The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Use the same staged distribution, promotion, and retirement procedure as with an existing Secret. ### OIDC and other auth settings diff --git a/helm/AGENTS.md b/helm/AGENTS.md index 33632bf..1bef500 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -32,7 +32,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - **`customEngineConfig` is merged into the rendered engine `config.yaml`.** `fbinstance.engineConfig` builds a canonical document on the new Firebolt Core schema (`schema_version: "1.0"`, `engine.{id,nodes,termination_grace_period}`, `instance.{type,multi_engine.metadata_endpoint}`, plus `instance.auth`/`endpoints` when `auth.enabled`/`tls.engine.enabled`) and deep-merges `.Values.customEngineConfig` on top at the root. Use it to set `instance.id` (a lowercase Crockford ULID; `values.schema.json` rejects uppercase. The engine fans it out to `account_id`/`account_name`/`organization_id`/`organization_name`), add a `logging:` block, or add auth settings the chart doesn't expose as first-class values (`instance.auth.oidc`, `password_login`, `jwt`, `preferred_authorization_server`, `signing_algorithm`) — these deep-merge alongside the admin account and signing keys the chart renders. Chart-authoritative paths are silently stripped before the merge: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. Without that auth stripping, a `customEngineConfig.instance.auth.admin`/`local.signing_keys` set by the user would win the merge and could point the engine at paths the chart never mounts, or flip `enabled` back off while the admin block remains — which the engine itself then refuses to start on; `instance.auth.oidc`/`password_login`/`jwt`/`preferred_authorization_server`/`signing_algorithm` are untouched and merge normally. `engine.termination_grace_period` is derived from `engineSpec.terminationGracePeriodSeconds` (minus a 5s margin, floored at 1s). Add new fields here rather than carving out new top-level value keys. The rendered file is YAML; the engine reads it from `/var/lib/firebolt/config.yaml` (the data dir, `--data-dir /var/lib/firebolt`). - **Pensieve `default_account_id` reads from `customEngineConfig.instance.id`.** `templates/metadata-service-configmap.yaml` populates `` from the same value the engine sees as `instance.id`, so Pensieve and the engine agree on the account ULID without two separate value keys. - **Auth and TLS secrets are provisioned by `existingSecret` or `certManager`, never both.** `fbinstance.secretSourceName` (`_helpers.tpl`) resolves each of `auth.signingKeys[]`, `tls.gateway`, and `tls.engine` to the Secret name to mount, and `fail`s the render if a source sets both or neither (detected via a non-empty `existingSecret.secretRef` vs. a non-empty `certManager.issuerRef.name` — the `certManager` map is always present with structural defaults, so `issuerRef.name` is the actual signal). `fbinstance.usesCertManager` mirrors the same check as a non-failing template guard, used by `certificates.yaml` (which also gates each artifact's `Certificate` on its owning feature — `tls.gateway.enabled` / `tls.engine.enabled` / `auth.enabled` — so a stale `certManager` block on a disabled feature never renders one) to decide whether to render a `Certificate`. `auth.admin.password` mirrors the same `existingSecret.secretRef` shape for consistency but only ever supports `existingSecret` — a password isn't a certificate, so there's no `certManager` alternative — and is validated by its own `fail` check rather than the shared helper. Nothing under `instance.auth` is rendered at all unless `auth.enabled` — the engine refuses to start if `admin`/`oidc`/`preferred_authorization_server` are present while auth is disabled. -- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation is prepend-a-new-entry, not replace-the-list — removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. +- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation uses separate distribution, promotion, and removal upgrades: append the new verifier and roll every node before moving it to the first position; retain the old verifier until tokens issued by the last old signer expire. Removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. - **The engine's own TLS cert must be a full chain, not leaf-only.** When `tls.engine.enabled`, the engine's internal self-health-check dials its own query listener over HTTPS and validates using `certificate_file` as its curl trust bundle — a leaf-only cert (cert-manager's default `tls.crt`) fails self-verification. The `tls-chain-setup` init container in `engine-statefulset.yaml` concatenates `tls.crt` + `ca.crt` from the resolved Secret into `certificate_file`'s path; `ca.crt` is therefore a required key in `tls.engine`'s Secret (BYO or cert-manager), not optional. The Envoy gateway's `trusted_ca` reads `ca.crt` directly (standard chain validation needs only the CA, no concatenation). - **`endpoints.http.listeners` only ever governs the query port (3473).** Confirmed against the engine source: health (8122), metrics (9090), and metadata (6500) listeners are configured by independent keys untouched by `endpoints`, so the chart's minimal TLS overlay never needs to re-declare them. kubelet probes stay on plain HTTP against 8122 regardless of `tls.engine.enabled`. - **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines via the chart-owned `engine-serviceaccount.yaml` (skipped only when `engineSpec.serviceAccount` names a BYO SA the chart does not own). The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived. @@ -62,7 +62,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - `templates/namespace.yaml` — optional Namespace. - `templates/tests/test-*.yaml` — `helm test` Pod manifests. Each is independent; `helm test` runs them in declaration order. - `files/memlock-setup.sh` — the sidecar script loaded into the engine pod when `engineSpec.memlockSetup: true`. -- `scripts/validate-chart.sh` — full validation pipeline used by CI and runnable locally. +- `scripts/validate-chart.sh` — full validation pipeline used by CI and runnable locally, including `scripts/test_render.py` (Helm + Python/PyYAML). `make test-gateway-runtime` uses Docker to exercise the rendered retry policy against a backend that disconnects after accepting a mutation. - `charts/` — populated by `helm dependency build`; contents are not committed. ## Adding a new value diff --git a/helm/README.md b/helm/README.md index 5993673..b16ab9d 100644 --- a/helm/README.md +++ b/helm/README.md @@ -21,7 +21,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | auth.admin.password.existingSecret | object | {} | Existing Secret containing the admin password under key `password`. Mounted read-only and passed to the engine as `password_file`. | | auth.admin.password.existingSecret.secretRef | string | `""` | Secret name. | | auth.enabled | bool | `false` | Enable authentication on the engine (`instance.auth.enabled`). | -| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, prepend a new entry** rather than replacing the list, so tokens signed by the outgoing key keep validating until it's removed. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by prepending a new list entry instead. | +| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, first append the new entry** and finish every Engine rollout before moving it to the first position. Finish every rollout again, then retain the old key for the maximum token lifetime plus clock skew before removing it. Prepending immediately lets updated nodes mint tokens that older nodes cannot verify. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by distributing, promoting, and retiring list entries in stages. | | createNamespace | bool | `false` | When true, a Namespace resource is included in the chart output. Pair with `helm install --create-namespace --set createNamespace=false`. | | customEngineConfig | object | {} | Custom engine configuration deep-merged into the rendered engine config.yaml at the root. The rendered document follows the Firebolt Core configuration schema (`schema_version: "1.0"`); user-supplied keys at the top of `customEngineConfig` become siblings of the chart-managed `engine:` and `instance:` blocks (e.g. `logging:`), and keys nested under `instance:` merge into the instance block (e.g. `instance.id`, or `instance.auth.*` alongside the admin account and signing keys the `auth` value block below renders when `auth.enabled` is true — see `auth` for what belongs there instead of here). Chart-authoritative paths are silently stripped from this input and cannot be overridden: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. | | customEngineConfig.instance | object | {} | Instance identity. `id` propagates internally to `account_id`, `account_name`, `organization_id`, and `organization_name`, so the chart only needs to set the ULID once. | diff --git a/helm/scripts/test_gateway_runtime.py b/helm/scripts/test_gateway_runtime.py new file mode 100644 index 0000000..6f5e472 --- /dev/null +++ b/helm/scripts/test_gateway_runtime.py @@ -0,0 +1,107 @@ +"""Run the chart's Envoy retry policy against a deliberately disconnecting backend. + +Requires Docker, Helm, and PyYAML. Only the created test container is removed. +The fixture substitutes the upstream address and removes routing/discovery filters; +the chart's route and retry policy are exercised unmodified. +""" +import http.client +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +import subprocess +import sys +import tempfile +import threading +import time +import unittest + +import yaml +from test_render import envoy_config, render, resource + + +class RetryRuntimeTests(unittest.TestCase): + def test_delivered_reset_is_not_replayed_but_drained_response_is(self): + calls = {"/reset": 0, "/drained": 0} + + class Backend(BaseHTTPRequestHandler): + def log_message(self, *_args): + pass + + def do_POST(self): + self.rfile.read(int(self.headers.get("Content-Length", 0))) + calls[self.path] += 1 + if self.path == "/reset": + # The mutation has happened. Close before response headers. + self.close_connection = True + return + drained = calls[self.path] == 1 + self.send_response(503 if drained else 200) + if drained: + self.send_header("X-Firebolt-Drained", "true") + self.send_header("Content-Length", "0") + self.end_headers() + + server = ThreadingHTTPServer(("0.0.0.0", 0), Backend) + threading.Thread(target=server.serve_forever, daemon=True).start() + config = envoy_config() + listener = config["static_resources"]["listeners"][0] + hcm = listener["filter_chains"][0]["filters"][0]["typed_config"] + hcm["http_filters"] = [f for f in hcm["http_filters"] if f["name"] in { + "envoy.filters.http.health_check", "envoy.filters.http.router", + }] + config["static_resources"]["clusters"] = [{ + "name": "dynamic_forward_proxy", "type": "LOGICAL_DNS", "connect_timeout": "1s", + "dns_lookup_family": "V4_ONLY", + "load_assignment": {"cluster_name": "dynamic_forward_proxy", "endpoints": [{ + "lb_endpoints": [{"endpoint": {"address": {"socket_address": { + "address": "host.docker.internal", "port_value": server.server_port, + }}}}], + }]}, + }] + config["static_resources"]["listeners"] = [listener] + image = resource(render(), "Deployment", "audit-gateway")["spec"]["template"]["spec"]["containers"][0]["image"] + container = None + try: + with tempfile.TemporaryDirectory(prefix="firebolt-gateway-test-") as temp: + path = Path(temp) / "envoy.yaml" + path.write_text(yaml.safe_dump(config)) + container = subprocess.check_output([ + "docker", "run", "-d", + *(["--add-host=host.docker.internal:host-gateway"] if sys.platform == "linux" else []), + "-p", "127.0.0.1::8080", "-v", f"{path}:/etc/envoy/envoy.yaml:ro", + image, "envoy", "-c", "/etc/envoy/envoy.yaml", "--concurrency", "1", + ], text=True).strip() + port = int(subprocess.check_output(["docker", "port", container, "8080/tcp"], text=True).strip().rsplit(":", 1)[1]) + + def request(method, path, body=None): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20) + try: + conn.request(method, path, body=body) + response = conn.getresponse() + response.read() + return response.status + finally: + conn.close() + + deadline = time.monotonic() + 30 + while True: + try: + if request("GET", "/healthz") == 200: + break + except (OSError, http.client.HTTPException): + pass + if time.monotonic() >= deadline: + self.fail(subprocess.check_output(["docker", "logs", container], text=True, stderr=subprocess.STDOUT)) + time.sleep(0.1) + self.assertEqual(request("POST", "/reset", "mutation"), 503) + self.assertEqual(calls["/reset"], 1, subprocess.check_output(["docker", "logs", container], text=True, stderr=subprocess.STDOUT)) + self.assertEqual(request("POST", "/drained", "mutation"), 200) + self.assertEqual(calls["/drained"], 2, "the pre-work fence should be retried") + finally: + if container: + subprocess.run(["docker", "rm", "-f", container], check=True, stdout=subprocess.DEVNULL) + server.shutdown() + server.server_close() + + +if __name__ == "__main__": + unittest.main() diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py new file mode 100644 index 0000000..a2e5ccb --- /dev/null +++ b/helm/scripts/test_render.py @@ -0,0 +1,48 @@ +"""Chart behavior regressions. Requires Helm and PyYAML (also used by yamllint).""" +import json +import os +from pathlib import Path +import subprocess +import unittest + +import yaml + +CHART = Path(__file__).resolve().parents[1] + + +def render(values=None, release="audit", namespace="audit", expect_error=False): + result = subprocess.run( + [os.environ.get("HELM", "helm"), "template", release, str(CHART), + "--namespace", namespace, "-f", "-"], + input=json.dumps(values or {}), text=True, capture_output=True, + ) + if expect_error: + if result.returncode == 0: + raise AssertionError("invalid values rendered successfully") + return result.stderr + if result.returncode: + raise AssertionError(result.stderr) + return [doc for doc in yaml.safe_load_all(result.stdout) if doc] + + +def resource(docs, kind, name): + return next(d for d in docs if d["kind"] == kind and d["metadata"]["name"] == name) + + +def envoy_config(values=None): + return yaml.safe_load(resource(render(values), "ConfigMap", "audit-gateway")["data"]["envoy.yaml"]) + + +class GatewaySafetyTests(unittest.TestCase): + def test_retries_do_not_replay_delivered_requests(self): + config = envoy_config() + hcm = config["static_resources"]["listeners"][0]["filter_chains"][0]["filters"][0]["typed_config"] + policy = hcm["route_config"]["virtual_hosts"][0]["routes"][0]["route"]["retry_policy"] + self.assertEqual(set(policy["retry_on"].split(",")), { + "connect-failure", "refused-stream", "reset-before-request", "retriable-headers", + }) + self.assertEqual(policy["retriable_headers"], [{"name": "X-Firebolt-Drained", "present_match": True}]) + + +if __name__ == "__main__": + unittest.main() diff --git a/helm/scripts/validate-chart.sh b/helm/scripts/validate-chart.sh index 69da655..e1167ee 100755 --- a/helm/scripts/validate-chart.sh +++ b/helm/scripts/validate-chart.sh @@ -153,5 +153,8 @@ done echo "" echo "===================================" + +# PyYAML is installed with yamllint in CI. Override PYTHON for a local venv. +"${PYTHON:-python3}" "$SCRIPT_DIR/test_render.py" echo "✓ All validation checks passed!" echo "===================================" diff --git a/helm/templates/gateway-configmap.yaml b/helm/templates/gateway-configmap.yaml index c8124dc..95307e4 100644 --- a/helm/templates/gateway-configmap.yaml +++ b/helm/templates/gateway-configmap.yaml @@ -133,7 +133,7 @@ data: cluster: dynamic_forward_proxy timeout: 0s # Retry contract: - # - connect-failure / refused-stream / reset: + # - connect-failure / refused-stream / reset-before-request: # transport-level failures where the engine # could not have observed the request, so a # retry cannot duplicate side effects. @@ -148,6 +148,8 @@ data: # between SIGTERM and the next health-check # ejection would propagate a 503 to the # client. + # Never retry a generic reset: the engine may have + # accepted and mutated state before disconnecting. # We deliberately do NOT match bare 5xx: a # 5xx returned by the engine after work has # started may have applied side effects. @@ -165,7 +167,7 @@ data: # short exponential back-off lets a DNS refresh or # health-check ejection take effect between tries. retry_policy: - retry_on: connect-failure,refused-stream,reset,retriable-headers + retry_on: connect-failure,refused-stream,reset-before-request,retriable-headers retriable_headers: - name: X-Firebolt-Drained present_match: true diff --git a/helm/values.yaml b/helm/values.yaml index 7b94290..4bee74e 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -475,8 +475,11 @@ auth: # (`instance.auth.local.signing_keys`). An ordered list: the **first** # entry is the active signer used for new tokens; every entry remains # valid for verifying tokens already issued under it. **To rotate the - # signing key, prepend a new entry** rather than replacing the list, so - # tokens signed by the outgoing key keep validating until it's removed. + # signing key, first append the new entry** and finish every Engine + # rollout before moving it to the first position. Finish every rollout + # again, then retain the old key for the maximum token lifetime plus + # clock skew before removing it. Prepending immediately lets updated + # nodes mint tokens that older nodes cannot verify. # At least one entry is required when `auth.enabled` is true — an engine # with no explicit signing key falls back to a per-pod dev key that # differs across nodes and breaks token validation in a multi-node @@ -492,7 +495,7 @@ auth: # and its CRDs installed in-cluster. The chart pins # `privateKey.rotationPolicy: Never` so cert-manager never silently # rotates a key out from under already-issued tokens — rotate by - # prepending a new list entry instead. + # distributing, promoting, and retiring list entries in stages. # @default -- [] signingKeys: [] # Example: From 65f9adbfd3f161d6c4d715cf7930c26e8b07b9b6 Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Thu, 1 Oct 2026 17:04:39 +0200 Subject: [PATCH 2/3] docs(auth): align signing key comment with staged rotation (FB-4383) --- helm/templates/certificates.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/helm/templates/certificates.yaml b/helm/templates/certificates.yaml index eb68d36..19aca61 100644 --- a/helm/templates/certificates.yaml +++ b/helm/templates/certificates.yaml @@ -84,8 +84,9 @@ spec: size: {{ $key.certManager.size | default 2048 }} # Signing keys must never be silently rotated by cert-manager's renewal timer — a # rotated key that vanishes from the active `signing_keys[0]` slot invalidates every - # token signed under it. Rotation is a deliberate chart-value change (prepend a new - # signingKeys entry), not a cert-manager renewal. + # token signed under it. Rotate through separate chart upgrades: append a verifier, + # complete every Engine rollout, promote it to signer, then retire the old key + # only after its tokens expire. Certificate renewal must preserve the key. rotationPolicy: Never issuerRef: name: {{ $key.certManager.issuerRef.name }} From 81177ec6450f74a2f2036c3bccfb3f2db4407a14 Mon Sep 17 00:00:00 2001 From: Simon Zelenski Date: Fri, 2 Oct 2026 10:33:00 +0200 Subject: [PATCH 3/3] docs(auth): move rotation guidance to a separate PR (FB-4383) --- docs/usage/authentication.mdx | 31 ++++++------------------------- helm/AGENTS.md | 2 +- helm/README.md | 2 +- helm/templates/certificates.yaml | 5 ++--- helm/values.yaml | 9 +++------ 5 files changed, 13 insertions(+), 36 deletions(-) diff --git a/docs/usage/authentication.mdx b/docs/usage/authentication.mdx index 68ba428..0f3c80e 100644 --- a/docs/usage/authentication.mdx +++ b/docs/usage/authentication.mdx @@ -55,38 +55,19 @@ Kubernetes health probes and the gateway's active engine health check keep worki ### Signing key rotation -`signingKeys` is ordered. The first entry signs new tokens, and every entry verifies tokens. Rotate in separate upgrades so every node learns the new key before any node signs with it: - -1. Create a new Secret or cert-manager entry with a fresh key ID. **Append** it after the current signer. Upgrade and wait for every Engine StatefulSet to finish rolling out. With cert-manager, wait for the new Secret to be issued as well. -2. Move the new entry to the first position. Upgrade and wait for every Engine StatefulSet to finish rolling out again. -3. From the completion of the second rollout, retain the old key for the maximum lifetime of tokens it could have issued, including clock-skew tolerance. If that lifetime is unknown or unbounded, do not remove the key until you have established a safe retention policy. -4. Remove the old entry, upgrade, and verify authentication across every Engine. - -Do not replace key material in an existing Secret or prepend a new signer in a single upgrade. An updated node could issue tokens that an older node cannot verify. These steps preserve token verification compatibility; Engine restarts can still interrupt queries. - -During the first upgrade, keep the old signer first: +`signingKeys` is an ordered list. The **first** entry is the active signer used for new tokens; every entry in the list remains valid for verifying tokens already issued under it. To rotate, prepend a new entry rather than replacing the list, and remove the old entry only once every token it signed has expired: ```yaml auth: signingKeys: - - id: key-2026-01 - existingSecret: - secretRef: firebolt-signing-key-2026-01 - - id: key-2026-02 + - id: key-2026-02 # now active existingSecret: secretRef: firebolt-signing-key-2026-02 + - id: key-2026-01 # still verifies old tokens until they expire + existingSecret: + secretRef: firebolt-signing-key-2026-01 ``` -After each upgrade, wait for all Engine StatefulSets in the release: - -```bash -kubectl rollout status statefulset --namespace firebolt \ - --selector app.kubernetes.io/instance=firebolt,firebolt/component=engine \ - --timeout=10m -``` - -Check tokens issued by different nodes against every other node before promotion and after removal. A successful login against one node does not establish fleet-wide compatibility. - ### cert-manager as a signing-key source As an alternative to an existing Secret, a signing key entry can request a [cert-manager](https://cert-manager.io) `Certificate` instead: @@ -103,7 +84,7 @@ auth: kind: ClusterIssuer ``` -The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Use the same staged distribution, promotion, and retirement procedure as with an existing Secret. +The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Rotate by prepending a new `signingKeys` entry, the same as with an existing Secret. ### OIDC and other auth settings diff --git a/helm/AGENTS.md b/helm/AGENTS.md index 1bef500..9c41855 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -32,7 +32,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - **`customEngineConfig` is merged into the rendered engine `config.yaml`.** `fbinstance.engineConfig` builds a canonical document on the new Firebolt Core schema (`schema_version: "1.0"`, `engine.{id,nodes,termination_grace_period}`, `instance.{type,multi_engine.metadata_endpoint}`, plus `instance.auth`/`endpoints` when `auth.enabled`/`tls.engine.enabled`) and deep-merges `.Values.customEngineConfig` on top at the root. Use it to set `instance.id` (a lowercase Crockford ULID; `values.schema.json` rejects uppercase. The engine fans it out to `account_id`/`account_name`/`organization_id`/`organization_name`), add a `logging:` block, or add auth settings the chart doesn't expose as first-class values (`instance.auth.oidc`, `password_login`, `jwt`, `preferred_authorization_server`, `signing_algorithm`) — these deep-merge alongside the admin account and signing keys the chart renders. Chart-authoritative paths are silently stripped before the merge: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. Without that auth stripping, a `customEngineConfig.instance.auth.admin`/`local.signing_keys` set by the user would win the merge and could point the engine at paths the chart never mounts, or flip `enabled` back off while the admin block remains — which the engine itself then refuses to start on; `instance.auth.oidc`/`password_login`/`jwt`/`preferred_authorization_server`/`signing_algorithm` are untouched and merge normally. `engine.termination_grace_period` is derived from `engineSpec.terminationGracePeriodSeconds` (minus a 5s margin, floored at 1s). Add new fields here rather than carving out new top-level value keys. The rendered file is YAML; the engine reads it from `/var/lib/firebolt/config.yaml` (the data dir, `--data-dir /var/lib/firebolt`). - **Pensieve `default_account_id` reads from `customEngineConfig.instance.id`.** `templates/metadata-service-configmap.yaml` populates `` from the same value the engine sees as `instance.id`, so Pensieve and the engine agree on the account ULID without two separate value keys. - **Auth and TLS secrets are provisioned by `existingSecret` or `certManager`, never both.** `fbinstance.secretSourceName` (`_helpers.tpl`) resolves each of `auth.signingKeys[]`, `tls.gateway`, and `tls.engine` to the Secret name to mount, and `fail`s the render if a source sets both or neither (detected via a non-empty `existingSecret.secretRef` vs. a non-empty `certManager.issuerRef.name` — the `certManager` map is always present with structural defaults, so `issuerRef.name` is the actual signal). `fbinstance.usesCertManager` mirrors the same check as a non-failing template guard, used by `certificates.yaml` (which also gates each artifact's `Certificate` on its owning feature — `tls.gateway.enabled` / `tls.engine.enabled` / `auth.enabled` — so a stale `certManager` block on a disabled feature never renders one) to decide whether to render a `Certificate`. `auth.admin.password` mirrors the same `existingSecret.secretRef` shape for consistency but only ever supports `existingSecret` — a password isn't a certificate, so there's no `certManager` alternative — and is validated by its own `fail` check rather than the shared helper. Nothing under `instance.auth` is rendered at all unless `auth.enabled` — the engine refuses to start if `admin`/`oidc`/`preferred_authorization_server` are present while auth is disabled. -- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation uses separate distribution, promotion, and removal upgrades: append the new verifier and roll every node before moving it to the first position; retain the old verifier until tokens issued by the last old signer expire. Removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. +- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation is prepend-a-new-entry, not replace-the-list — removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. - **The engine's own TLS cert must be a full chain, not leaf-only.** When `tls.engine.enabled`, the engine's internal self-health-check dials its own query listener over HTTPS and validates using `certificate_file` as its curl trust bundle — a leaf-only cert (cert-manager's default `tls.crt`) fails self-verification. The `tls-chain-setup` init container in `engine-statefulset.yaml` concatenates `tls.crt` + `ca.crt` from the resolved Secret into `certificate_file`'s path; `ca.crt` is therefore a required key in `tls.engine`'s Secret (BYO or cert-manager), not optional. The Envoy gateway's `trusted_ca` reads `ca.crt` directly (standard chain validation needs only the CA, no concatenation). - **`endpoints.http.listeners` only ever governs the query port (3473).** Confirmed against the engine source: health (8122), metrics (9090), and metadata (6500) listeners are configured by independent keys untouched by `endpoints`, so the chart's minimal TLS overlay never needs to re-declare them. kubelet probes stay on plain HTTP against 8122 regardless of `tls.engine.enabled`. - **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines via the chart-owned `engine-serviceaccount.yaml` (skipped only when `engineSpec.serviceAccount` names a BYO SA the chart does not own). The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived. diff --git a/helm/README.md b/helm/README.md index b16ab9d..5993673 100644 --- a/helm/README.md +++ b/helm/README.md @@ -21,7 +21,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | auth.admin.password.existingSecret | object | {} | Existing Secret containing the admin password under key `password`. Mounted read-only and passed to the engine as `password_file`. | | auth.admin.password.existingSecret.secretRef | string | `""` | Secret name. | | auth.enabled | bool | `false` | Enable authentication on the engine (`instance.auth.enabled`). | -| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, first append the new entry** and finish every Engine rollout before moving it to the first position. Finish every rollout again, then retain the old key for the maximum token lifetime plus clock skew before removing it. Prepending immediately lets updated nodes mint tokens that older nodes cannot verify. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by distributing, promoting, and retiring list entries in stages. | +| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, prepend a new entry** rather than replacing the list, so tokens signed by the outgoing key keep validating until it's removed. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by prepending a new list entry instead. | | createNamespace | bool | `false` | When true, a Namespace resource is included in the chart output. Pair with `helm install --create-namespace --set createNamespace=false`. | | customEngineConfig | object | {} | Custom engine configuration deep-merged into the rendered engine config.yaml at the root. The rendered document follows the Firebolt Core configuration schema (`schema_version: "1.0"`); user-supplied keys at the top of `customEngineConfig` become siblings of the chart-managed `engine:` and `instance:` blocks (e.g. `logging:`), and keys nested under `instance:` merge into the instance block (e.g. `instance.id`, or `instance.auth.*` alongside the admin account and signing keys the `auth` value block below renders when `auth.enabled` is true — see `auth` for what belongs there instead of here). Chart-authoritative paths are silently stripped from this input and cannot be overridden: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. | | customEngineConfig.instance | object | {} | Instance identity. `id` propagates internally to `account_id`, `account_name`, `organization_id`, and `organization_name`, so the chart only needs to set the ULID once. | diff --git a/helm/templates/certificates.yaml b/helm/templates/certificates.yaml index 19aca61..eb68d36 100644 --- a/helm/templates/certificates.yaml +++ b/helm/templates/certificates.yaml @@ -84,9 +84,8 @@ spec: size: {{ $key.certManager.size | default 2048 }} # Signing keys must never be silently rotated by cert-manager's renewal timer — a # rotated key that vanishes from the active `signing_keys[0]` slot invalidates every - # token signed under it. Rotate through separate chart upgrades: append a verifier, - # complete every Engine rollout, promote it to signer, then retire the old key - # only after its tokens expire. Certificate renewal must preserve the key. + # token signed under it. Rotation is a deliberate chart-value change (prepend a new + # signingKeys entry), not a cert-manager renewal. rotationPolicy: Never issuerRef: name: {{ $key.certManager.issuerRef.name }} diff --git a/helm/values.yaml b/helm/values.yaml index 4bee74e..7b94290 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -475,11 +475,8 @@ auth: # (`instance.auth.local.signing_keys`). An ordered list: the **first** # entry is the active signer used for new tokens; every entry remains # valid for verifying tokens already issued under it. **To rotate the - # signing key, first append the new entry** and finish every Engine - # rollout before moving it to the first position. Finish every rollout - # again, then retain the old key for the maximum token lifetime plus - # clock skew before removing it. Prepending immediately lets updated - # nodes mint tokens that older nodes cannot verify. + # signing key, prepend a new entry** rather than replacing the list, so + # tokens signed by the outgoing key keep validating until it's removed. # At least one entry is required when `auth.enabled` is true — an engine # with no explicit signing key falls back to a per-pod dev key that # differs across nodes and breaks token validation in a multi-node @@ -495,7 +492,7 @@ auth: # and its CRDs installed in-cluster. The chart pins # `privateKey.rotationPolicy: Never` so cert-manager never silently # rotates a key out from under already-issued tokens — rotate by - # distributing, promoting, and retiring list entries in stages. + # prepending a new list entry instead. # @default -- [] signingKeys: [] # Example: