Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/security.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,9 @@ Object-storage credentials for engines are not represented as Helm values. The c

The chart never sees those cloud provider credentials. For an S3-compatible endpoint (`managed_table_storage: s3` with `aws.endpoint`), the engine authenticates with the `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` you inject via `engineSpec.extraEnv` (from a Secret in production); see [S3-compatible storage](./usage/object-storage/s3-compatible).

### TLS to PostgreSQL is not exposed
### TLS to external PostgreSQL

The chart does not currently expose `sslmode` or a CA-bundle mount for the Metadata Service's PostgreSQL connection. Encrypt the PostgreSQL connection at the network layer (VPC peering, sidecar TLS proxy, or service-mesh mTLS) until the chart grows native support.
Set `postgresql.tls.enabled: true` and `postgresql.tls.caSecret` to require certificate-chain and hostname verification (`verify-full`). The CA bundle key defaults to `ca.crt` and can be changed with `postgresql.tls.caKey`. The certificate must cover `postgresql.host`. The bundled database does not provide a TLS listener. See [External PostgreSQL](./usage/external-postgres) for setup and CA rotation.

### Authentication and TLS

Expand Down
4 changes: 3 additions & 1 deletion docs/usage/external-postgres.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ kubectl -n firebolt logs deployment/firebolt-metadata-service | grep -i postgres

## Caveats

- TLS to PostgreSQL is not currently exposed. There is no `sslmode` value or CA-bundle mount.
- Enable `postgresql.tls.enabled` for verified TLS. Set `postgresql.tls.caSecret` to a Secret in the release namespace and `postgresql.tls.caKey` to its CA bundle key (default `ca.crt`). The chart sets `PGSSLMODE=verify-full` and mounts only the CA bundle. The database certificate must cover `postgresql.host`. This option requires `local_enabled: false`.
- `host`, `database`, and `schema` are pattern-restricted by `values.schema.json` and quoted as YAML scalars at render time. User input cannot break the rendered Metadata Service config.
- The Metadata Service creates its schema on first connect and migrates the schema in place when pointed at a database from an older Metadata Service version. No manual migration step is required.

Expand All @@ -79,3 +79,5 @@ Online Helm upgrades also track the existing credential Secret's resource versio


The bundled PostgreSQL path accepts `credentials.existingSecret` too. Supply `username`, `password`, and `database` keys, with username and database matching the chart values. Updating that Secret does not change a password inside an already-initialized database.

After changing the database CA Secret, restart the Metadata Service using the commands above, or use your configured external reload controller. Online Helm rendering also tracks the CA Secret resource version. Coordinate CA rotation with the database server.
22 changes: 22 additions & 0 deletions docs/usage/multi-engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,25 @@ Engine names must match RFC 1123 DNS labels (lowercase alphanumerics and hyphens
## Change the engine set

Edit `engines:` and run `helm upgrade`. Removing an engine deletes its StatefulSets but leaves the PVCs (Kubernetes default `Retain`).

## Override shared settings

Each Engine inherits `engineSpec`. Explicit per-Engine fields replace inherited values, including `false`, empty lists, and empty maps. Image objects inherit individual fields. `storage` inherits individual fields from `engineSpec.defaultStorage`. Per-Engine `customEngineConfig` merges over the global configuration, but `instance.id` remains global because all Engines share the Metadata Service. Chart-owned configuration fields remain protected.

For example, set `engines[].extraEnv: []` to clear inherited environment additions or `engines[].uiSidecar: false` to disable the inherited UI sidecar. Each Engine can select its own `serviceAccount`; the chart disables automatic API token mounts even with a supplied ServiceAccount.

## Select an Engine in the URL

Send the routing selector as a query parameter:

```bash
curl -s 'http://localhost:8080/?engine=analytics' --data 'SELECT 1'
```

The Gateway accepts either `X-Firebolt-Engine` or `engine=`. If both are present, they must match. Duplicate query selectors, invalid names, and conflicting selectors return HTTP 400. The Gateway removes the routing parameter before forwarding the request and preserves other query parameters.

## Use ephemeral local storage

Set `engineSpec.storageType: emptyDir`, or set `storageType: emptyDir` on one Engine, to use ephemeral local storage. Optional `emptyDir.sizeLimit` and `emptyDir.medium` configure the volume. Memory-backed volumes count against pod memory limits. Pod removal discards local data and caches, so use durable object storage and PostgreSQL for the durable data and metadata. Expect cold-cache recovery after replacement.

The default `storageType: pvc` retains PVCs when an Engine is removed. `storageType: hostPath` and the legacy `hostPathStorageEnabled` setting remain available. An explicit per-Engine `storageType` overrides an inherited `engineSpec.hostPathStorageEnabled: true`, allowing PVC or ephemeral Engines alongside legacy hostPath Engines. Changing storage type on an existing StatefulSet requires a planned migration because its volume claim templates are immutable. Prefer a new Engine name and retire the old Engine after validation.
2 changes: 1 addition & 1 deletion helm/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g
- **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation is prepend-a-new-entry, not replace-the-list — removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with.
- **The engine's own TLS cert must be a full chain, not leaf-only.** When `tls.engine.enabled`, the engine's internal self-health-check dials its own query listener over HTTPS and validates using `certificate_file` as its curl trust bundle — a leaf-only cert (cert-manager's default `tls.crt`) fails self-verification. The `tls-chain-setup` init container in `engine-statefulset.yaml` concatenates `tls.crt` + `ca.crt` from the resolved Secret into `certificate_file`'s path; `ca.crt` is therefore a required key in `tls.engine`'s Secret (BYO or cert-manager), not optional. The Envoy gateway's `trusted_ca` reads `ca.crt` directly (standard chain validation needs only the CA, no concatenation).
- **`endpoints.http.listeners` only ever governs the query port (3473).** Confirmed against the engine source: health (8122), metrics (9090), and metadata (6500) listeners are configured by independent keys untouched by `endpoints`, so the chart's minimal TLS overlay never needs to re-declare them. kubelet probes stay on plain HTTP against 8122 regardless of `tls.engine.enabled`.
- **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines via the chart-owned `engine-serviceaccount.yaml` (skipped only when `engineSpec.serviceAccount` names a BYO SA the chart does not own). The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived.
- **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines at pod level, including when a BYO ServiceAccount is selected. The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived.
- **No secrets in `values.yaml`.** `postgresql.password` is required when `local_enabled: true` and is left empty in `values.yaml`. External-DB users should set `postgresql.credentials.existingSecret` instead.
- **`shareProcessNamespace` + memlock sidecar.** When `engineSpec.memlockSetup: true`, the engine pod runs the `files/memlock-setup.sh` script as a sidecar that calls `prlimit` against the Core PID inside the shared PID namespace. The script is loaded via `.Files.Get` through the `fbinstance.memlockSetupScript` helper — keep it inside `helm/files/` so the loader path stays valid.
- **Test pod helper.** Helm test scripts use `fbinstance.testShellHelpers` for the shared `log` / `pass` / `fail` shell functions. Reuse it instead of redefining the helpers.
Expand Down
Loading