diff --git a/docs/usage/authentication.mdx b/docs/usage/authentication.mdx index 0f3c80e..3066e4f 100644 --- a/docs/usage/authentication.mdx +++ b/docs/usage/authentication.mdx @@ -55,19 +55,31 @@ Kubernetes health probes and the gateway's active engine health check keep worki ### Signing key rotation -`signingKeys` is an ordered list. The **first** entry is the active signer used for new tokens; every entry in the list remains valid for verifying tokens already issued under it. To rotate, prepend a new entry rather than replacing the list, and remove the old entry only once every token it signed has expired: +`signingKeys` is ordered: the first entry signs new tokens, and all entries verify tokens. Engines load keys at startup, so Secret changes require Engine restarts. + +For routine rotation while preserving existing tokens, use separate chart upgrades: + +1. Append the new key with a fresh ID and Secret name. Wait for the Secret to exist and every Engine StatefulSet to complete its rollout before proceeding. +2. Move the new key to the first position and complete every Engine rollout again. +3. Retain the old key until its tokens have expired, including clock-skew tolerance, measured from when the last node stopped signing with it. Then remove its entry and upgrade. + +This gives the sequence `[old]` → `[old, new]` → `[new, old]` → `[new]`. Promoting the new key before all nodes have loaded it can break token validation. Engine restarts can still interrupt queries. + +After distributing the new key to every node, configure it as the active signer while retaining the old key for verification: ```yaml auth: signingKeys: - - id: key-2026-02 # now active + - id: key-2026-02 # Active: signs new tokens existingSecret: secretRef: firebolt-signing-key-2026-02 - - id: key-2026-01 # still verifies old tokens until they expire + - id: key-2026-01 # Retiring: verifies old tokens until they expire existingSecret: secretRef: firebolt-signing-key-2026-01 ``` +During a maintenance window with authentication and query traffic stopped, you can skip the initial distribution upgrade and switch directly to `[new, old]`. Complete all Engine rollouts before restoring traffic. + ### cert-manager as a signing-key source As an alternative to an existing Secret, a signing key entry can request a [cert-manager](https://cert-manager.io) `Certificate` instead: @@ -84,7 +96,7 @@ auth: kind: ClusterIssuer ``` -The chart pins `privateKey.rotationPolicy: Never` on signing-key Certificates, so cert-manager's renewal timer never silently replaces an active signing key out from under already-issued tokens. Rotate by prepending a new `signingKeys` entry, the same as with an existing Secret. +The chart pins `privateKey.rotationPolicy: Never`, so certificate renewal preserves the signing key. Rotate signing keys through new `signingKeys` entries using the procedure above. cert-manager provisions the key material; it does not coordinate signer promotion or retirement. ### OIDC and other auth settings diff --git a/helm/AGENTS.md b/helm/AGENTS.md index f515f67..0cbe5d9 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -32,7 +32,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - **`customEngineConfig` is merged into the rendered engine `config.yaml`.** `fbinstance.engineConfig` builds a canonical document on the new Firebolt Core schema (`schema_version: "1.0"`, `engine.{id,nodes,termination_grace_period}`, `instance.{type,multi_engine.metadata_endpoint}`, plus `instance.auth`/`endpoints` when `auth.enabled`/`tls.engine.enabled`) and deep-merges `.Values.customEngineConfig` on top at the root. Use it to set `instance.id` (a lowercase Crockford ULID; `values.schema.json` rejects uppercase. The engine fans it out to `account_id`/`account_name`/`organization_id`/`organization_name`), add a `logging:` block, or add auth settings the chart doesn't expose as first-class values (`instance.auth.oidc`, `password_login`, `jwt`, `preferred_authorization_server`, `signing_algorithm`) — these deep-merge alongside the admin account and signing keys the chart renders. Chart-authoritative paths are silently stripped before the merge: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. Without that auth stripping, a `customEngineConfig.instance.auth.admin`/`local.signing_keys` set by the user would win the merge and could point the engine at paths the chart never mounts, or flip `enabled` back off while the admin block remains — which the engine itself then refuses to start on; `instance.auth.oidc`/`password_login`/`jwt`/`preferred_authorization_server`/`signing_algorithm` are untouched and merge normally. `engine.termination_grace_period` is derived from `engineSpec.terminationGracePeriodSeconds` (minus a 5s margin, floored at 1s). Add new fields here rather than carving out new top-level value keys. The rendered file is YAML; the engine reads it from `/var/lib/firebolt/config.yaml` (the data dir, `--data-dir /var/lib/firebolt`). - **Pensieve `default_account_id` reads from `customEngineConfig.instance.id`.** `templates/metadata-service-configmap.yaml` populates `` from the same value the engine sees as `instance.id`, so Pensieve and the engine agree on the account ULID without two separate value keys. - **Auth and TLS secrets are provisioned by `existingSecret` or `certManager`, never both.** `fbinstance.secretSourceName` (`_helpers.tpl`) resolves each of `auth.signingKeys[]`, `tls.gateway`, and `tls.engine` to the Secret name to mount, and `fail`s the render if a source sets both or neither (detected via a non-empty `existingSecret.secretRef` vs. a non-empty `certManager.issuerRef.name` — the `certManager` map is always present with structural defaults, so `issuerRef.name` is the actual signal). `fbinstance.usesCertManager` mirrors the same check as a non-failing template guard, used by `certificates.yaml` (which also gates each artifact's `Certificate` on its owning feature — `tls.gateway.enabled` / `tls.engine.enabled` / `auth.enabled` — so a stale `certManager` block on a disabled feature never renders one) to decide whether to render a `Certificate`. `auth.admin.password` mirrors the same `existingSecret.secretRef` shape for consistency but only ever supports `existingSecret` — a password isn't a certificate, so there's no `certManager` alternative — and is validated by its own `fail` check rather than the shared helper. Nothing under `instance.auth` is rendered at all unless `auth.enabled` — the engine refuses to start if `admin`/`oidc`/`preferred_authorization_server` are present while auth is disabled. -- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation is prepend-a-new-entry, not replace-the-list — removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. +- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation uses separate distribution, promotion, and removal upgrades: append the new verifier and roll every node before moving it to the first position; retain the old verifier until tokens issued by the last old signer expire. Removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. - **The engine's own TLS cert must be a full chain, not leaf-only.** When `tls.engine.enabled`, the engine's internal self-health-check dials its own query listener over HTTPS and validates using `certificate_file` as its curl trust bundle — a leaf-only cert (cert-manager's default `tls.crt`) fails self-verification. The `tls-chain-setup` init container in `engine-statefulset.yaml` concatenates `tls.crt` + `ca.crt` from the resolved Secret into `certificate_file`'s path; `ca.crt` is therefore a required key in `tls.engine`'s Secret (BYO or cert-manager), not optional. The Envoy gateway's `trusted_ca` reads `ca.crt` directly (standard chain validation needs only the CA, no concatenation). - **`endpoints.http.listeners` only ever governs the query port (3473).** Confirmed against the engine source: health (8122), metrics (9090), and metadata (6500) listeners are configured by independent keys untouched by `endpoints`, so the chart's minimal TLS overlay never needs to re-declare them. kubelet probes stay on plain HTTP against 8122 regardless of `tls.engine.enabled`. - **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines at pod level, including when a BYO ServiceAccount is selected. The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived. diff --git a/helm/README.md b/helm/README.md index a993d51..0a0076a 100644 --- a/helm/README.md +++ b/helm/README.md @@ -21,7 +21,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | auth.admin.password.existingSecret | object | {} | Existing Secret containing the admin password under key `password`. Mounted read-only and passed to the engine as `password_file`. | | auth.admin.password.existingSecret.secretRef | string | `""` | Secret name. | | auth.enabled | bool | `false` | Enable authentication on the engine (`instance.auth.enabled`). | -| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, prepend a new entry** rather than replacing the list, so tokens signed by the outgoing key keep validating until it's removed. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by prepending a new list entry instead. | +| auth.signingKeys | list | [] | JWT signing keys for the engine's embedded Authorization Server (`instance.auth.local.signing_keys`). An ordered list: the **first** entry is the active signer used for new tokens; every entry remains valid for verifying tokens already issued under it. **To rotate the signing key, first append the new entry** and finish every Engine rollout before moving it to the first position. Finish every rollout again, then retain the old key for the maximum token lifetime plus clock skew before removing it. Prepending immediately lets updated nodes mint tokens that older nodes cannot verify. At least one entry is required when `auth.enabled` is true — an engine with no explicit signing key falls back to a per-pod dev key that differs across nodes and breaks token validation in a multi-node engine. Each entry sets exactly one of `existingSecret` / `certManager`: - `id` — key identifier, published as the JWT `kid` header (`instance.auth.local.signing_keys[].id`). - `existingSecret.secretRef` — Secret containing the PEM private key under key `tls.key`. - `certManager` — a chart-rendered cert-manager `Certificate` request (`algorithm`, `size`, `issuerRef.{name,kind}`). Requires cert-manager and its CRDs installed in-cluster. The chart pins `privateKey.rotationPolicy: Never` so cert-manager never silently rotates a key out from under already-issued tokens — rotate by distributing, promoting, and retiring list entries in stages. | | createNamespace | bool | `false` | When true, a Namespace resource is included in the chart output. Pair with `helm install --create-namespace --set createNamespace=false`. | | customEngineConfig | object | {} | Custom engine configuration deep-merged into the rendered engine config.yaml at the root. The rendered document follows the Firebolt Core configuration schema (`schema_version: "1.0"`); user-supplied keys at the top of `customEngineConfig` become siblings of the chart-managed `engine:` and `instance:` blocks (e.g. `logging:`), and keys nested under `instance:` merge into the instance block (e.g. `instance.id`, or `instance.auth.*` alongside the admin account and signing keys the `auth` value block below renders when `auth.enabled` is true — see `auth` for what belongs there instead of here). Chart-authoritative paths are silently stripped from this input and cannot be overridden: `schema_version`, `engine.id`, `engine.nodes`, `engine.termination_grace_period`, `instance.type`, `instance.multi_engine`, and — only while `auth.enabled` / `tls.engine.enabled` are actually rendering them — `instance.auth.{enabled,admin,local.signing_keys}` and `endpoints.http.listeners`. | | customEngineConfig.instance | object | {} | Instance identity. `id` propagates internally to `account_id`, `account_name`, `organization_id`, and `organization_name`, so the chart only needs to set the ULID once. | diff --git a/helm/templates/certificates.yaml b/helm/templates/certificates.yaml index eb68d36..19aca61 100644 --- a/helm/templates/certificates.yaml +++ b/helm/templates/certificates.yaml @@ -84,8 +84,9 @@ spec: size: {{ $key.certManager.size | default 2048 }} # Signing keys must never be silently rotated by cert-manager's renewal timer — a # rotated key that vanishes from the active `signing_keys[0]` slot invalidates every - # token signed under it. Rotation is a deliberate chart-value change (prepend a new - # signingKeys entry), not a cert-manager renewal. + # token signed under it. Rotate through separate chart upgrades: append a verifier, + # complete every Engine rollout, promote it to signer, then retire the old key + # only after its tokens expire. Certificate renewal must preserve the key. rotationPolicy: Never issuerRef: name: {{ $key.certManager.issuerRef.name }} diff --git a/helm/values.yaml b/helm/values.yaml index 8a0850a..9421e42 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -518,8 +518,11 @@ auth: # (`instance.auth.local.signing_keys`). An ordered list: the **first** # entry is the active signer used for new tokens; every entry remains # valid for verifying tokens already issued under it. **To rotate the - # signing key, prepend a new entry** rather than replacing the list, so - # tokens signed by the outgoing key keep validating until it's removed. + # signing key, first append the new entry** and finish every Engine + # rollout before moving it to the first position. Finish every rollout + # again, then retain the old key for the maximum token lifetime plus + # clock skew before removing it. Prepending immediately lets updated + # nodes mint tokens that older nodes cannot verify. # At least one entry is required when `auth.enabled` is true — an engine # with no explicit signing key falls back to a per-pod dev key that # differs across nodes and breaks token validation in a multi-node @@ -535,7 +538,7 @@ auth: # and its CRDs installed in-cluster. The chart pins # `privateKey.rotationPolicy: Never` so cert-manager never silently # rotates a key out from under already-issued tokens — rotate by - # prepending a new list entry instead. + # distributing, promoting, and retiring list entries in stages. # @default -- [] signingKeys: [] # Example: