diff --git a/docs/security.mdx b/docs/security.mdx index e16361e..805ced8 100644 --- a/docs/security.mdx +++ b/docs/security.mdx @@ -105,9 +105,9 @@ Object-storage credentials for engines are not represented as Helm values. The c The chart never sees those cloud provider credentials. For an S3-compatible endpoint (`managed_table_storage: s3` with `aws.endpoint`), the engine authenticates with the `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` you inject via `engineSpec.extraEnv` (from a Secret in production); see [S3-compatible storage](./usage/object-storage/s3-compatible). -### TLS to PostgreSQL is not exposed +### TLS to external PostgreSQL -The chart does not currently expose `sslmode` or a CA-bundle mount for the Metadata Service's PostgreSQL connection. Encrypt the PostgreSQL connection at the network layer (VPC peering, sidecar TLS proxy, or service-mesh mTLS) until the chart grows native support. +Set `postgresql.tls.enabled: true` and `postgresql.tls.caSecret` to require certificate-chain and hostname verification (`verify-full`). The CA bundle key defaults to `ca.crt` and can be changed with `postgresql.tls.caKey`. The certificate must cover `postgresql.host`. The bundled database does not provide a TLS listener. See [External PostgreSQL](./usage/external-postgres) for setup and CA rotation. ### Authentication and TLS diff --git a/docs/usage/external-postgres.mdx b/docs/usage/external-postgres.mdx index d610346..fafbabb 100644 --- a/docs/usage/external-postgres.mdx +++ b/docs/usage/external-postgres.mdx @@ -60,7 +60,7 @@ kubectl -n firebolt logs deployment/firebolt-metadata-service | grep -i postgres ## Caveats -- TLS to PostgreSQL is not currently exposed. There is no `sslmode` value or CA-bundle mount. +- Enable `postgresql.tls.enabled` for verified TLS. Set `postgresql.tls.caSecret` to a Secret in the release namespace and `postgresql.tls.caKey` to its CA bundle key (default `ca.crt`). The chart sets `PGSSLMODE=verify-full` and mounts only the CA bundle. The database certificate must cover `postgresql.host`. This option requires `local_enabled: false`. - `host`, `database`, and `schema` are pattern-restricted by `values.schema.json` and quoted as YAML scalars at render time. User input cannot break the rendered Metadata Service config. - The Metadata Service creates its schema on first connect and migrates the schema in place when pointed at a database from an older Metadata Service version. No manual migration step is required. @@ -79,3 +79,5 @@ Online Helm upgrades also track the existing credential Secret's resource versio The bundled PostgreSQL path accepts `credentials.existingSecret` too. Supply `username`, `password`, and `database` keys, with username and database matching the chart values. Updating that Secret does not change a password inside an already-initialized database. + +After changing the database CA Secret, restart the Metadata Service using the commands above, or use your configured external reload controller. Online Helm rendering also tracks the CA Secret resource version. Coordinate CA rotation with the database server. diff --git a/docs/usage/multi-engine.mdx b/docs/usage/multi-engine.mdx index caa24ad..6d88d02 100644 --- a/docs/usage/multi-engine.mdx +++ b/docs/usage/multi-engine.mdx @@ -83,3 +83,25 @@ Engine names must match RFC 1123 DNS labels (lowercase alphanumerics and hyphens ## Change the engine set Edit `engines:` and run `helm upgrade`. Removing an engine deletes its StatefulSets but leaves the PVCs (Kubernetes default `Retain`). + +## Override shared settings + +Each Engine inherits `engineSpec`. Explicit per-Engine fields replace inherited values, including `false`, empty lists, and empty maps. Image objects inherit individual fields. `storage` inherits individual fields from `engineSpec.defaultStorage`. Per-Engine `customEngineConfig` merges over the global configuration, but `instance.id` remains global because all Engines share the Metadata Service. Chart-owned configuration fields remain protected. + +For example, set `engines[].extraEnv: []` to clear inherited environment additions or `engines[].uiSidecar: false` to disable the inherited UI sidecar. Each Engine can select its own `serviceAccount`; the chart disables automatic API token mounts even with a supplied ServiceAccount. + +## Select an Engine in the URL + +Send the routing selector as a query parameter: + +```bash +curl -s 'http://localhost:8080/?engine=analytics' --data 'SELECT 1' +``` + +The Gateway accepts either `X-Firebolt-Engine` or `engine=`. If both are present, they must match. Duplicate query selectors, invalid names, and conflicting selectors return HTTP 400. The Gateway removes the routing parameter before forwarding the request and preserves other query parameters. + +## Use ephemeral local storage + +Set `engineSpec.storageType: emptyDir`, or set `storageType: emptyDir` on one Engine, to use ephemeral local storage. Optional `emptyDir.sizeLimit` and `emptyDir.medium` configure the volume. Memory-backed volumes count against pod memory limits. Pod removal discards local data and caches, so use durable object storage and PostgreSQL for the durable data and metadata. Expect cold-cache recovery after replacement. + +The default `storageType: pvc` retains PVCs when an Engine is removed. `storageType: hostPath` and the legacy `hostPathStorageEnabled` setting remain available. An explicit per-Engine `storageType` overrides an inherited `engineSpec.hostPathStorageEnabled: true`, allowing PVC or ephemeral Engines alongside legacy hostPath Engines. Changing storage type on an existing StatefulSet requires a planned migration because its volume claim templates are immutable. Prefer a new Engine name and retire the old Engine after validation. diff --git a/helm/AGENTS.md b/helm/AGENTS.md index be51f8b..f515f67 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -35,7 +35,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - **`auth.signingKeys` is ordered; the first entry signs.** The engine always uses `signing_keys[0]` to sign new tokens; every other entry only verifies. Rotation is prepend-a-new-entry, not replace-the-list — removing an entry immediately invalidates tokens signed under it. `auth.enabled` with an empty `signingKeys` fails the render (a per-pod dev key would differ across nodes and break cross-node token validation), and so does any entry with an empty `id` (schema-required, plus a template `fail` for defense in depth) — it's published as the JWT `kid` header and used as the cert-manager Certificate's `commonName`. `fbinstance.authSigningKeySecretName` derives each cert-manager signing key's Secret/Certificate name from a hash of its `id`, not its list position — prepending shifts every existing entry to a higher index, and an index-derived name would make cert-manager treat the shifted entry as a new request and reissue fresh material, orphaning the key that already-issued tokens were signed with. - **The engine's own TLS cert must be a full chain, not leaf-only.** When `tls.engine.enabled`, the engine's internal self-health-check dials its own query listener over HTTPS and validates using `certificate_file` as its curl trust bundle — a leaf-only cert (cert-manager's default `tls.crt`) fails self-verification. The `tls-chain-setup` init container in `engine-statefulset.yaml` concatenates `tls.crt` + `ca.crt` from the resolved Secret into `certificate_file`'s path; `ca.crt` is therefore a required key in `tls.engine`'s Secret (BYO or cert-manager), not optional. The Envoy gateway's `trusted_ca` reads `ca.crt` directly (standard chain validation needs only the CA, no concatenation). - **`endpoints.http.listeners` only ever governs the query port (3473).** Confirmed against the engine source: health (8122), metrics (9090), and metadata (6500) listeners are configured by independent keys untouched by `endpoints`, so the chart's minimal TLS overlay never needs to re-declare them. kubelet probes stay on plain HTTP against 8122 regardless of `tls.engine.enabled`. -- **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines via the chart-owned `engine-serviceaccount.yaml` (skipped only when `engineSpec.serviceAccount` names a BYO SA the chart does not own). The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived. +- **No workload pod gets a ServiceAccount token.** None of the chart's components call the Kubernetes API, so every long-running pod spec the chart owns sets `automountServiceAccountToken: false` unconditionally — `gateway-deployment.yaml`, `metadata-service-deployment.yaml`, and `postgresql-statefulset.yaml` at pod level, engines at pod level, including when a BYO ServiceAccount is selected. The setting only suppresses the default SA token mount; webhook-injected projected tokens (IRSA, EKS Pod Identity) are separate volumes and keep working, which is why a user-supplied `serviceAccountName` does not need an escape hatch. A new workload template MUST set it too. The `templates/tests/` hook pods are deliberately excluded — they are ephemeral and short-lived. - **No secrets in `values.yaml`.** `postgresql.password` is required when `local_enabled: true` and is left empty in `values.yaml`. External-DB users should set `postgresql.credentials.existingSecret` instead. - **`shareProcessNamespace` + memlock sidecar.** When `engineSpec.memlockSetup: true`, the engine pod runs the `files/memlock-setup.sh` script as a sidecar that calls `prlimit` against the Core PID inside the shared PID namespace. The script is loaded via `.Files.Get` through the `fbinstance.memlockSetupScript` helper — keep it inside `helm/files/` so the loader path stays valid. - **Test pod helper.** Helm test scripts use `fbinstance.testShellHelpers` for the shared `log` / `pass` / `fail` shell functions. Reuse it instead of redefining the helpers. diff --git a/helm/README.md b/helm/README.md index 613faf0..a993d51 100644 --- a/helm/README.md +++ b/helm/README.md @@ -28,31 +28,37 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | customEngineConfig.instance.id | string | `"01kp98j0000000000000000000"` | Lowercase Crockford ULID for the Firebolt instance. Must match the account reconciled by the metadata service at startup. `values.schema.json` rejects uppercase and any character outside the Crockford alphabet. | | engineSpec | object | {} | Shared engine pod defaults applied to all engines unless overridden per-engine. | | engineSpec.affinity | object | `{}` | Affinity rules for engine pod scheduling. | -| engineSpec.annotations | object | {} | Annotations on Engine StatefulSets, for example to opt into an externally installed Secret reload controller. These do not annotate pods. | +| engineSpec.annotations | object | {} | Default annotations on Engine StatefulSets, replaced as a whole by engines[].annotations (an empty map clears them). These do not annotate pods. | | engineSpec.customInitContainersTemplate | list | `[]` | Custom init containers injected into engine pods (supports templating). | | engineSpec.customVolumeMounts | list | `[]` | Custom volume mounts injected into the engine `core` container, paired with `customVolumes` above — a volume declared there is inert until also mounted here. | | engineSpec.customVolumes | list | `[]` | Custom volumes injected into engine pods. | | engineSpec.defaultStorage | object | {} | Default PVC storage spec for engines. `storageClassName` is intentionally absent — the cluster default storage class is used. Override here or per-engine to specify a class (e.g. `storageClassName: gp3`). | | engineSpec.defaultStorage.accessModes | list | `["ReadWriteOnce"]` | Access modes for the default PVC. | | engineSpec.defaultStorage.resources.requests.storage | string | `"100Gi"` | Default storage size for engine PVCs. | +| engineSpec.emptyDir | object | {} | Options for storageType=emptyDir, such as sizeLimit or medium: Memory. Local cache is lost when the pod is removed; metadata and object storage remain authoritative. | | engineSpec.extraEnv | list | [] | Extra environment variables for the engine container. Use this to inject AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (for example via `valueFrom.secretKeyRef`) for a custom S3-compatible store (`managed_table_storage: s3` + `aws.endpoint`). | | engineSpec.extraEnvFrom | list | [] | Extra `envFrom` sources for the engine container. Use a `secretRef` to load a Secret holding AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for a custom S3-compatible store. | -| engineSpec.hostPathStorageEnabled | bool | `false` | When true, uses hostPath instead of PVC for engine data. | +| engineSpec.hostPathStorageEnabled | bool | `false` | Legacy alias for storageType=hostPath. An explicit engines[].storageType overrides the inherited alias. Setting both hostPathStorageEnabled=true and storageType=emptyDir on the same Engine is invalid. | | engineSpec.image.pullPolicy | string | `"IfNotPresent"` | Image pull policy. | | engineSpec.image.repository | string | `"oci.firebolt.io/firebolt-db/engine"` | Container repository for the Firebolt engine image. | | engineSpec.image.tag | string | `""` | Image tag. Defaults to `Chart.appVersion` when empty. | | engineSpec.memlockSetup | bool | `false` | When true, a memlock-setup init container is added to configure memory locking limits. | | engineSpec.nodeHostSuffix | string | `".cluster.local"` | Suffix appended after `.svc` in node FQDNs in `config.yaml`. | | engineSpec.nodeSelector | object | `{}` | Node selector for engine pod scheduling. | +| engineSpec.podAnnotations | object | {} | Default custom pod annotations. checksum/ and firebolt.io/ are reserved. | +| engineSpec.podLabels | object | {} | Default extra pod labels. Chart selector labels remain authoritative. | | engineSpec.podSecurityContext | object | {} | Pod-level security context for engine pods. | | engineSpec.podSecurityContext.fsGroup | int | `3473` | Group applied to mounted volumes. Matches the engine UID/GID so the data PVC is chowned on mount. | | engineSpec.podSecurityContext.fsGroupChangePolicy | string | `"OnRootMismatch"` | When to re-apply `fsGroup` ownership. `OnRootMismatch` skips the chown when already correct — much faster on large PVCs. | | engineSpec.podSecurityContext.runAsNonRoot | bool | `true` | Reject the pod if any container runs as UID 0. Also gates the chart's container-level `runAs*` defaults, memlock init, and the entrypoint UID check. | +| engineSpec.priorityClassName | string | `""` | Default priority class, overridden per Engine (an empty override clears it). | | engineSpec.readiness | bool | `true` | When true, a readiness probe is added to the core container. | -| engineSpec.serviceAccount | string | `""` | ServiceAccount used by engine pods. Empty (the default): the chart creates `-engine` with `automountServiceAccountToken: false`, so a code-execution exploit in the engine container has no SA token to talk to the apiserver with. Engines do not call the Kubernetes API; the dedicated SA replaces the namespace `default` SA, which automounts a token and inherits any RoleBindings accumulated on `default` from unrelated installs. Non-empty: the chart references the named SA verbatim and does NOT create one — bring your own (the documented IRSA / Pod Identity flow at docs/usage/object-storage/amazon-s3.mdx works this way). The chart cannot influence `automountServiceAccountToken` on a SA it does not own; set it explicitly in your SA manifest if you want the same hardening. | -| engineSpec.storageHostPath | object | {} | Host path configuration used when `hostPathStorageEnabled` is true. | +| engineSpec.resources | object | {} | Shared container resource requests and limits, overridden as a whole by engines[].resources. | +| engineSpec.serviceAccount | string | `""` | ServiceAccount used by engine pods. Empty (the default): the chart creates `-engine` with `automountServiceAccountToken: false`, so a code-execution exploit in the engine container has no SA token to talk to the apiserver with. Engines do not call the Kubernetes API; the dedicated SA replaces the namespace `default` SA, which automounts a token and inherits any RoleBindings accumulated on `default` from unrelated installs. Non-empty: this Engine references the named SA verbatim (bring your own). A managed SA is still created if another Engine needs it. Every Engine pod sets automountServiceAccountToken: false, including with a supplied SA. Webhook-injected workload-identity tokens remain supported; see docs/usage/object-storage/amazon-s3.mdx. | +| engineSpec.storageHostPath | object | {} | Host path configuration used with storageType=hostPath or the legacy hostPathStorageEnabled alias. | | engineSpec.storageHostPath.path | string | `"/var/lib/firebolt-core"` | Host path for engine data. | | engineSpec.storageHostPath.type | string | `"DirectoryOrCreate"` | Host path type. | +| engineSpec.storageType | string | `"pvc"` | Engine-local cache backend: pvc (retained by default), emptyDir (ephemeral), or hostPath. Changing backend on an existing StatefulSet requires a migration. | | engineSpec.terminationGracePeriodSeconds | int | `60` | Termination grace period in seconds for engine pods. Sized to give in-flight queries time to drain before SIGKILL during rolling updates and node drains. Also rendered into the engine `config.yaml` as `engine.termination_grace_period` with a 5s safety margin (floored at 1s), so the engine's own in-flight-query wait stays below this value. | | engineSpec.tolerations | list | `[]` | Tolerations for engine pod scheduling. | | engineSpec.topologySpreadConstraints | list | `[]` | Topology spread constraints for engine pod scheduling. Set this to force zone or node spread across an engine's nodes so a single zone or node failure cannot take down the whole engine. Overridable per-engine via `engines[].topologySpreadConstraints`. | @@ -60,18 +66,10 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | engineSpec.uiSidecarImage.pullPolicy | string | `"Always"` | Image pull policy for the Core UI sidecar. Defaults to `Always` (the Kubernetes default for `latest`-tagged images) so published UI fixes reach existing installs; nodes would otherwise pin the first cached copy. | | engineSpec.uiSidecarImage.repository | string | `"ghcr.io/firebolt-db/firebolt-core-ui"` | Container repository for the Core UI sidecar image. | | engineSpec.uiSidecarImage.tag | string | `"latest"` | Core UI sidecar image tag. A mutable alias tracking the latest published UI. | -| engines | list | [] | Engine definitions. Each entry produces one StatefulSet per node (`replicas` controls node count), plus a shared headless Service, ClusterIP Service, and ConfigMap. Per-engine values override the shared `engineSpec` defaults. | -| engines[0].affinity | object | `{}` | Affinity rules for engine pod scheduling. | -| engines[0].name | string | `"default"` | Engine name. Used to derive resource names across the chart. | -| engines[0].nodeSelector | object | `{}` | Node selector for engine pod scheduling. | -| engines[0].podAnnotations | object | `{}` | Annotations applied to engine pods. | -| engines[0].podLabels | object | `{}` | Extra labels applied to this engine's pod template. Chart-reserved keys (`app.kubernetes.io/{name,instance,managed-by}` and `firebolt/{component,engine,node}`) are silently dropped from user input so the StatefulSet selector cannot be detached by a typo. | -| engines[0].priorityClassName | string | `""` | Priority class name for engine pods. | -| engines[0].replicas | int | `1` | Number of nodes in this engine group (one StatefulSet replica per node). | -| engines[0].resources | object | `{"limits":{"memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}` | Resource requests and limits for engine containers. Firebolt Core is memory-bound: more RAM directly improves cache hit rates and query throughput. CPU governs parallel query execution threads. Typical sizing guidance: Development / functional testing: 2 vCPU / 8 Gi (request) Small production workload: 4 vCPU / 32 Gi Medium production workload: 8 vCPU / 64 Gi Large production workload: 16 vCPU / 128 Gi Storage I/O is also significant — use an SSD-backed StorageClass and size the PVC to hold your working dataset plus ~30 % headroom. | -| engines[0].storage | object | `{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100Gi"}}}` | PVC storage configuration for this engine. Falls back to `engineSpec.defaultStorage` if omitted. Set `storageClassName` to select a class, or an empty string to disable dynamic provisioning. Existing claim templates cannot be resized or switched by a normal Helm upgrade. | -| engines[0].tolerations | list | `[]` | Tolerations for engine pod scheduling. | -| engines[0].topologySpreadConstraints | list | `[]` | Topology spread constraints for engine pod scheduling. Overrides `engineSpec.topologySpreadConstraints` for this engine when set. | +| engines | list | [] | Engine definitions. Each entry produces one StatefulSet per node (`replicas` controls node count), plus a shared headless Service, ClusterIP Service, and ConfigMap. Per-engine values replace shared `engineSpec` defaults, including explicit false and empty maps/lists. Image fields and storage fields inherit individually; use per-engine storage for engineSpec.defaultStorage. | +| engines[0].customEngineConfig | object | {} | Optional per-Engine config overlay, merged over customEngineConfig. Installation identity and chart-managed topology/auth/TLS fields cannot be overridden. | +| engines[0].name | string | `"default"` | Engine name. Generated pod and Service names must fit in 63 characters together with the release prefix and node suffix. | +| engines[0].replicas | int | `1` | Number of nodes (one single-replica StatefulSet per node). Upgrades and topology changes can interrupt queries. | | extraLabels | object | `{"firebolt/product":"core"}` | Extra labels applied to all resources and pods. | | gateway | object | {} | Envoy gateway proxy configuration. Routes queries to engine Services based on the `X-Firebolt-Engine` HTTP header. A Lua filter extracts the engine name and rewrites the upstream to `{engine}-service:3473` via dynamic forward proxy. | | gateway.adminPort | int | `9901` | Envoy admin interface port (used for health checks). | @@ -166,6 +164,10 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | postgresql.port | int | `5432` | PostgreSQL port. | | postgresql.resources | object | `{"limits":{"cpu":"250m","memory":"256Mi"},"requests":{"cpu":"25m","memory":"64Mi"}}` | Resource requests and limits for the bundled PostgreSQL container. | | postgresql.schema | string | `"public"` | PostgreSQL schema. | +| postgresql.tls | object | {} | TLS to external PostgreSQL. The bundled database does not provide a TLS listener. | +| postgresql.tls.caKey | string | `"ca.crt"` | Key containing the PEM CA bundle in caSecret. | +| postgresql.tls.caSecret | string | `""` | Secret containing the PostgreSQL server CA bundle, mounted read-only into the Metadata Service. | +| postgresql.tls.enabled | bool | `false` | Require certificate-chain and hostname verification (verify-full). | | postgresql.username | string | `"firebolt"` | Database username. | | tls | object | {} | TLS configuration for the Envoy gateway's client-facing listener and the engine's query listener. Each of `gateway` / `engine` is provisioned by exactly one of `existingSecret` (a `kubernetes.io/tls` Secret) or `certManager` (a chart-rendered cert-manager `Certificate`); setting both or neither on an enabled block fails the render. | | tls.engine | object | {} | TLS on the engine's query listener (port 3473) and, correspondingly, the gateway's upstream connection to engines (including the active engine health check, which runs over the same connection). | diff --git a/helm/files/gateway-routing.lua b/helm/files/gateway-routing.lua new file mode 100644 index 0000000..9fb784f --- /dev/null +++ b/helm/files/gateway-routing.lua @@ -0,0 +1,117 @@ +local function is_valid_engine(s) + if s == nil or #s == 0 or #s > 63 then return false end + if not string.match(s, "^[%l%d][-%l%d]*$") then return false end + if string.sub(s, -1) == "-" then return false end + return true +end + +local function url_decode(value) + local decoded = {} + local i = 1 + while i <= #value do + local c = string.sub(value, i, i) + if c == "%" then + local hex = string.sub(value, i + 1, i + 2) + if #hex ~= 2 or not string.match(hex, "^%x%x$") then + return nil + end + table.insert(decoded, string.char(tonumber(hex, 16))) + i = i + 3 + elseif c == "+" then + table.insert(decoded, " ") + i = i + 1 + else + table.insert(decoded, c) + i = i + 1 + end + end + return table.concat(decoded) +end + +-- Extracts exactly one engine query parameter and +-- removes it without decoding or re-encoding any +-- other parameter. The downstream SQL endpoint +-- interprets query parameters as settings and does +-- not support the routing-only engine setting. +local function extract_engine_from_path(path) + if path == nil then return nil, path, nil end + + local query_start = string.find(path, "?", 1, true) + if query_start == nil then return nil, path, nil end + + local base = string.sub(path, 1, query_start - 1) + local query = string.sub(path, query_start + 1) + local kept = {} + local engine = nil + local engine_count = 0 + + for part in string.gmatch(query .. "&", "(.-)&") do + local raw_key, raw_value = string.match(part, "^([^=]*)=(.*)$") + if raw_key == nil then + raw_key = part + raw_value = "" + end + + local key = url_decode(raw_key) + if key == "engine" then + engine_count = engine_count + 1 + if engine_count > 1 then + return nil, path, "multiple engine query parameters" + end + engine = url_decode(raw_value) + if engine == nil then + return nil, path, "invalid engine query parameter encoding" + end + else + table.insert(kept, part) + end + end + + if engine_count == 0 then return nil, path, nil end + + local stripped_query = table.concat(kept, "&") + if stripped_query == "" then + return engine, base, nil + end + return engine, base .. "?" .. stripped_query, nil +end + +function envoy_on_request(handle) + local headers = handle:headers() + local header_engine = headers:get("x-firebolt-engine") + local query_engine, stripped_path, query_error = + extract_engine_from_path(headers:get(":path")) + + if query_error ~= nil then + handle:respond({[":status"] = "400"}, query_error) + return + end + + local engine = query_engine + if header_engine ~= nil then + if not is_valid_engine(header_engine) then + handle:respond({[":status"] = "400"}, "invalid X-Firebolt-Engine header") + return + end + if query_engine ~= nil and query_engine ~= header_engine then + handle:respond({[":status"] = "400"}, "conflicting engine selectors") + return + end + engine = header_engine + end + + if not is_valid_engine(engine) then + handle:respond({[":status"] = "400"}, "invalid or missing engine selector") + return + end + + if query_engine ~= nil then + headers:replace(":path", stripped_path) + end + headers:replace("x-firebolt-engine", engine) + -- Keep the chart's supported-engine compatibility setting. + local path = headers:get(":path") + headers:replace(":path", path .. (path:find("?", 1, true) and "&" or "?") .. "advanced_mode=true") + headers:replace(":authority", "{{ include "fbinstance.fullname" . }}-engine-" .. engine .. "-ready.{{ .Release.Namespace }}.svc.cluster.local:3473") + +end diff --git a/helm/scripts/test_gateway_runtime.py b/helm/scripts/test_gateway_runtime.py index 654c91a..c66c69b 100644 --- a/helm/scripts/test_gateway_runtime.py +++ b/helm/scripts/test_gateway_runtime.py @@ -1,10 +1,11 @@ """Run the chart's Envoy retry policy against a deliberately disconnecting backend. Requires Docker, Helm, and PyYAML. Only the created test container is removed. -The fixture substitutes the upstream address and removes routing/discovery filters; +The fixture substitutes the upstream address and removes the discovery filter; the chart's route and retry policy are exercised unmodified. """ import http.client +from urllib.parse import urlsplit from concurrent.futures import ThreadPoolExecutor from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path @@ -21,7 +22,8 @@ class RetryRuntimeTests(unittest.TestCase): def test_delivered_reset_is_not_replayed_but_drained_response_is(self): - calls = {"/reset": 0, "/drained": 0, "/hold": 0} + calls = {"/reset": 0, "/drained": 0, "/hold": 0, "/echo": 0} + observed = [] accepted = threading.Event() release = threading.Event() @@ -31,15 +33,17 @@ def log_message(self, *_args): def do_POST(self): self.rfile.read(int(self.headers.get("Content-Length", 0))) - calls[self.path] += 1 - if self.path == "/reset": + observed.append((self.path, self.headers.get("Host"))) + path = urlsplit(self.path).path + calls[path] += 1 + if path == "/reset": # The mutation has happened. Close before response headers. self.close_connection = True return - if self.path == "/hold": + if path == "/hold": accepted.set() release.wait(15) - drained = self.path == "/drained" and calls[self.path] == 1 + drained = path == "/drained" and calls[path] == 1 self.send_response(503 if drained else 200) if drained: self.send_header("X-Firebolt-Drained", "true") @@ -52,7 +56,7 @@ def do_POST(self): listener = config["static_resources"]["listeners"][0] hcm = listener["filter_chains"][0]["filters"][0]["typed_config"] hcm["http_filters"] = [f for f in hcm["http_filters"] if f["name"] in { - "envoy.filters.http.health_check", "envoy.filters.http.router", + "envoy.filters.http.health_check", "envoy.filters.http.router", "envoy.filters.http.lua", }] admin_cluster = config["static_resources"]["clusters"][-1] config["static_resources"]["clusters"] = [{ @@ -82,10 +86,10 @@ def do_POST(self): metrics_port = int(subprocess.check_output(["docker", "port", container, "9090/tcp"], text=True).strip().rsplit(":", 1)[1]) - def request(method, path, body=None, target_port=None): + def request(method, path, body=None, target_port=None, headers=None): conn = http.client.HTTPConnection("127.0.0.1", target_port or port, timeout=20) try: - conn.request(method, path, body=body) + conn.request(method, path, body=body, headers={"X-Firebolt-Engine": "default"} if headers is None else headers) response = conn.getresponse() response.read() return response.status @@ -106,6 +110,18 @@ def request(method, path, body=None, target_port=None): self.assertEqual(calls["/reset"], 1, subprocess.check_output(["docker", "logs", container], text=True, stderr=subprocess.STDOUT)) self.assertEqual(request("POST", "/drained", "mutation"), 200) self.assertEqual(calls["/drained"], 2, "the pre-work fence should be retried") + for path, headers, expected in [ + ("/echo?engine=default&x=hello%20there", {}, 200), + ("/echo?%65ngine=def%61ult", {}, 200), + ("/echo?engine=other", {"X-Firebolt-Engine": "default"}, 400), + ("/echo?engine=default&engine=default", {}, 400), + ("/echo?engine=bad%2Fname", {}, 400), + ("/echo?engine=%ZZ", {}, 400), + ("/echo", {}, 400), + ]: + self.assertEqual(request("POST", path, "query", headers=headers), expected, path) + self.assertIn(("/echo?x=hello%20there&advanced_mode=true", "audit-engine-default-ready.audit.svc.cluster.local:3473"), observed) + self.assertEqual(calls["/echo"], 2, "invalid selectors must not reach upstream") # Run the actual chart preStop while a request is active. It must # retain the connection, close query admission, and leave probes up. with ThreadPoolExecutor(max_workers=1) as pool: diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py index ba56c01..244e2f3 100644 --- a/helm/scripts/test_render.py +++ b/helm/scripts/test_render.py @@ -187,5 +187,122 @@ def test_tls_hook_verifies_hosts_and_projects_only_ca_keys(self): self.assertNotIn("SERVING=", direct) +class EngineOptionsTests(unittest.TestCase): + def test_engine_workload_annotations_inherit_replace_and_clear(self): + shared = {"secret.reloader.stakater.com/auto": "true", "example.com/owner": "shared"} + custom = {"secret.reloader.stakater.com/reload": "engine-tls"} + values = {"engineSpec": {"annotations": shared}, "engines": [ + {"name": "inherited", "replicas": 2}, + {"name": "overridden", "replicas": 1, "annotations": custom}, + {"name": "cleared", "replicas": 1, "annotations": {}}, + ]} + docs = render(values) + for name, count, expected in [("inherited", 2, shared), ("overridden", 1, custom), ("cleared", 1, {})]: + for node in range(count): + workload = resource(docs, "StatefulSet", f"audit-engine-{name}-node-{node}") + self.assertEqual(workload["metadata"].get("annotations", {}), expected) + pod_annotations = workload["spec"]["template"]["metadata"]["annotations"] + self.assertFalse(set(shared) & set(pod_annotations)) + self.assertFalse(set(custom) & set(pod_annotations)) + render({"engines": [{"name": "invalid", "replicas": 1, "annotations": {"bad": 1}}]}, expect_error=True) + + def test_per_engine_overrides_and_empty_values_do_not_leak(self): + docs = render({"engineSpec": {"image": {"tag": "shared"}, "nodeSelector": {"pool": "shared"}, + "extraEnv": [{"name": "CUSTOM", "value": "shared"}], "uiSidecar": True}, + "engines": [{"name": "first", "replicas": 1, "image": {"tag": "specific"}, + "nodeSelector": {}, "extraEnv": [], "uiSidecar": False, + "terminationGracePeriodSeconds": 90, + "customEngineConfig": {"logging": {"level": "debug"}}}, + {"name": "second", "replicas": 1}]}) + first = resource(docs, "StatefulSet", "audit-engine-first-node-0")["spec"]["template"]["spec"] + second = resource(docs, "StatefulSet", "audit-engine-second-node-0")["spec"]["template"]["spec"] + self.assertTrue(first["containers"][0]["image"].endswith(":specific")) + self.assertTrue(second["containers"][0]["image"].endswith(":shared")) + self.assertNotIn("nodeSelector", first) + self.assertEqual(second["nodeSelector"], {"pool": "shared"}) + self.assertNotIn("CUSTOM", [e["name"] for e in first["containers"][0]["env"]]) + self.assertIn("CUSTOM", [e["name"] for e in second["containers"][0]["env"]]) + self.assertEqual(len(first["containers"]), 1) + self.assertEqual(len(second["containers"]), 2) + self.assertEqual(first["terminationGracePeriodSeconds"], 90) + config = yaml.safe_load(resource(docs, "ConfigMap", "audit-engine-first-config")["data"]["config.yaml"]) + self.assertEqual(config["engine"]["termination_grace_period"], "85s") + self.assertEqual(config["logging"]["level"], "debug") + other = yaml.safe_load(resource(docs, "ConfigMap", "audit-engine-second-config")["data"]["config.yaml"]) + self.assertNotIn("level", other["logging"]) + self.assertNotIn("web-ui", [p["name"] for p in resource(docs, "Service", "audit-engine-first-ready")["spec"]["ports"]]) + + def test_node_hooks_use_effective_host_suffix_for_each_engine(self): + docs = render({"engineSpec": {"nodeHostSuffix": ".shared.example"}, "engines": [ + {"name": "first", "replicas": 1, "nodeHostSuffix": ".specific.example"}, + {"name": "second", "replicas": 1}]}) + for hook in ("dns", "pods", "ready"): + script = resource(docs, "Pod", "audit-test-engine-" + hook)["spec"]["containers"][0]["command"][-1] + self.assertIn("audit-engine-first-hl.audit.svc.specific.example", script) + self.assertIn("audit-engine-second-hl.audit.svc.shared.example", script) + self.assertNotIn(".svc.cluster.local", script) + + def test_default_engine_inherits_shared_scheduling_and_resources(self): + docs = render({"engineSpec": {"nodeSelector": {"pool": "shared"}, "resources": {"limits": {"memory": "8Gi"}}}}) + spec = resource(docs, "StatefulSet", "audit-engine-default-node-0")["spec"]["template"]["spec"] + self.assertEqual(spec["nodeSelector"], {"pool": "shared"}) + self.assertEqual(spec["containers"][0]["resources"]["limits"]["memory"], "8Gi") + + def test_emptydir_is_explicit_and_pvc_retention_is_unchanged(self): + docs = render({"engines": [{"name": "cache", "replicas": 1, "storageType": "emptyDir", "emptyDir": {"sizeLimit": "2Gi"}}, + {"name": "persist", "replicas": 1}]}) + cache = resource(docs, "StatefulSet", "audit-engine-cache-node-0")["spec"] + self.assertNotIn("volumeClaimTemplates", cache) + self.assertIn({"name": "data", "emptyDir": {"sizeLimit": "2Gi"}}, cache["template"]["spec"]["volumes"]) + persistent = resource(docs, "StatefulSet", "audit-engine-persist-node-0")["spec"] + self.assertIn("volumeClaimTemplates", persistent) + self.assertNotIn("persistentVolumeClaimRetentionPolicy", persistent) + self.assertIn("mutually exclusive", render({"engineSpec": {"storageType": "emptyDir", "hostPathStorageEnabled": True}}, expect_error=True)) + + def test_explicit_storage_type_overrides_inherited_legacy_hostpath(self): + docs = render({"engineSpec": {"hostPathStorageEnabled": True}, "engines": [ + {"name": "legacy", "replicas": 1}, + {"name": "persistent", "replicas": 1, "storageType": "pvc"}, + {"name": "ephemeral", "replicas": 1, "storageType": "emptyDir"}, + {"name": "disabled", "replicas": 1, "hostPathStorageEnabled": False}, + ]}) + for name, backend in [("legacy", "hostPath"), ("persistent", "pvc"), + ("ephemeral", "emptyDir"), ("disabled", "pvc")]: + spec = resource(docs, "StatefulSet", f"audit-engine-{name}-node-0")["spec"] + self.assertEqual("volumeClaimTemplates" in spec, backend == "pvc") + volumes = spec["template"]["spec"]["volumes"] + data = next((v for v in volumes if v["name"] == "data"), {}) + self.assertEqual("hostPath" in data, backend == "hostPath") + self.assertEqual("emptyDir" in data, backend == "emptyDir") + self.assertIn("mutually exclusive", render({"engines": [{ + "name": "conflict", "replicas": 1, "storageType": "emptyDir", + "hostPathStorageEnabled": True}]}, expect_error=True)) + + def test_mixed_service_accounts(self): + docs = render({"engineSpec": {"serviceAccount": "external"}, "engines": [ + {"name": "first", "replicas": 1, "serviceAccount": ""}, {"name": "second", "replicas": 1}]}) + self.assertEqual(resource(docs, "ServiceAccount", "audit-engine")["automountServiceAccountToken"], False) + for name, account in [("first", "audit-engine"), ("second", "external")]: + spec = resource(docs, "StatefulSet", f"audit-engine-{name}-node-0")["spec"]["template"]["spec"] + self.assertEqual(spec["serviceAccountName"], account) + self.assertFalse(spec["automountServiceAccountToken"]) + + def test_external_postgres_tls_requires_verified_configuration(self): + docs = render({"postgresql": {"local_enabled": False, "host": "db.example.com", "tls": { + "enabled": True, "caSecret": "database-ca", "caKey": "root.pem"}}}) + spec = resource(docs, "Deployment", "audit-metadata-service")["spec"]["template"]["spec"] + env = {e["name"]: e.get("value") for e in spec["containers"][0]["env"]} + self.assertEqual(env["PGSSLMODE"], "verify-full") + self.assertEqual(env["PGSSLROOTCERT"], "/secrets/postgres-ca/ca.crt") + volume = next(v for v in spec["volumes"] if v["name"] == "postgres-ca") + self.assertEqual(volume["secret"]["items"], [{"key": "root.pem", "path": "ca.crt"}]) + self.assertIn("external database", render({"postgresql": {"tls": {"enabled": True, "caSecret": "ca"}}}, expect_error=True)) + self.assertIn("caSecret", render({"postgresql": {"local_enabled": False, "host": "db", "tls": {"enabled": True}}}, expect_error=True)) + + def test_per_engine_identity_override_is_rejected(self): + self.assertIn("installation identity", render({"engines": [{"name": "default", "replicas": 1, + "customEngineConfig": {"instance": {"id": "01kp98j0000000000000000001"}}}]}, expect_error=True)) + + if __name__ == "__main__": unittest.main() diff --git a/helm/templates/_helpers.tpl b/helm/templates/_helpers.tpl index 70824de..1d80b63 100644 --- a/helm/templates/_helpers.tpl +++ b/helm/templates/_helpers.tpl @@ -59,18 +59,6 @@ not mutate the pod template and trigger an unwanted rollout. app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} -{{/* -Engine ServiceAccount name. Empty `engineSpec.serviceAccount` -> -`-engine` (chart-managed; the SA template renders it). -Non-empty `engineSpec.serviceAccount` -> verbatim, and the chart does -not render a SA manifest (bring your own — IRSA / Pod Identity flow). -Both the SA template and the engine StatefulSet podSpec MUST resolve -the name through this helper so a single value drives both sides. -*/}} -{{- define "fbinstance.engineServiceAccountName" -}} -{{- default (printf "%s-engine" (include "fbinstance.fullname" .)) .Values.engineSpec.serviceAccount -}} -{{- end -}} - {{/* Engine Service ports. Only externally-meaningful endpoints are declared: http-query (SQL), health (probe / Envoy active health check), metrics @@ -159,13 +147,14 @@ Usage: {{ include "fbinstance.engineConfig" (dict "root" $ "engine" $engine) }} {{- define "fbinstance.engineConfig" -}} {{- $root := .root -}} {{- $engine := .engine -}} +{{- $effective := include "fbinstance.effectiveEngine" . | fromYaml -}} {{- $baseName := printf "%s-engine-%s" (include "fbinstance.fullname" $root) $engine.name -}} {{- $svcName := printf "%s-hl" $baseName -}} {{- $ns := $root.Release.Namespace -}} {{- $pensieveSvc := printf "%s-metadata-service" (include "fbinstance.fullname" $root) -}} {{- $nodes := list -}} {{- range $i := until (int $engine.replicas) -}} -{{- $fqdn := printf "%s-node-%d-0.%s.%s.svc%s" $baseName $i $svcName $ns $root.Values.engineSpec.nodeHostSuffix -}} +{{- $fqdn := printf "%s-node-%d-0.%s.%s.svc%s" $baseName $i $svcName $ns $effective.nodeHostSuffix -}} {{- $nodes = append $nodes (dict "host" $fqdn) -}} {{- end -}} {{- $metadataEndpoint := printf "%s.%s.svc.cluster.local:%d" $pensieveSvc $ns (int $root.Values.metadata.server.port) -}} @@ -175,11 +164,16 @@ Usage: {{ include "fbinstance.engineConfig" (dict "root" $ "engine" $engine) }} the kubelet escalates to SIGKILL, floored at 1s. A single clamp keeps the budget monotonic non-decreasing in the grace period. */}} -{{- $gracePeriod := int $root.Values.engineSpec.terminationGracePeriodSeconds -}} +{{- $gracePeriod := int $effective.terminationGracePeriodSeconds -}} {{- $shutdownWait := sub $gracePeriod 5 -}} {{- if lt $shutdownWait 1 -}}{{- $shutdownWait = 1 -}}{{- end -}} {{- $user := deepCopy (default (dict) $root.Values.customEngineConfig) -}} +{{- $overlay := deepCopy ($engine.customEngineConfig | default dict) -}} +{{- if and (hasKey $overlay "instance") (kindIs "map" $overlay.instance) -}} + {{- $_ := unset $overlay.instance "id" -}} +{{- end -}} +{{- $user = mergeOverwrite $user $overlay -}} {{- $_ := unset $user "schema_version" -}} {{/* When user.engine / user.instance is not a map (string, number, list…) drop @@ -364,12 +358,13 @@ Usage: {{ include "fbinstance.engineTlsDnsNames" . }} {{- $ns := $root.Release.Namespace -}} {{- $names := list "localhost" -}} {{- range $engine := $root.Values.engines -}} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $root "engine" $engine) | fromYaml -}} {{- $baseName := printf "%s-engine-%s" (include "fbinstance.fullname" $root) $engine.name -}} {{- $svcName := printf "%s-hl" $baseName -}} {{- $names = append $names (printf "%s-ready.%s.svc.cluster.local" $baseName $ns) -}} -{{- $names = append $names (printf "%s.%s.svc%s" $svcName $ns $root.Values.engineSpec.nodeHostSuffix) -}} +{{- $names = append $names (printf "%s.%s.svc%s" $svcName $ns $effective.nodeHostSuffix) -}} {{- range $i := until (int $engine.replicas) -}} -{{- $names = append $names (printf "%s-node-%d-0.%s.%s.svc%s" $baseName $i $svcName $ns $root.Values.engineSpec.nodeHostSuffix) -}} +{{- $names = append $names (printf "%s-node-%d-0.%s.%s.svc%s" $baseName $i $svcName $ns $effective.nodeHostSuffix) -}} {{- end -}} {{- end -}} {{- toYaml $names -}} @@ -401,3 +396,22 @@ fail() { printf '\033[1;31mFAIL:\033[0m %s\n' "$*" >&2; exit 1; } {{- define "fbinstance.postgresSecretName" -}} {{- .Values.postgresql.credentials.existingSecret | default (printf "%s-metadata-postgres-creds" (include "fbinstance.fullname" .)) -}} {{- end -}} + +{{/* Images and PVC specs inherit individual fields; other explicit engine keys replace defaults, including empty maps/lists and false. */}} +{{- define "fbinstance.effectiveEngine" -}} +{{- $out := deepCopy .root.Values.engineSpec -}} +{{- range $key, $value := .engine -}} + {{- if and (has $key (list "image" "uiSidecarImage")) (kindIs "map" $value) -}} + {{- $_ := set $out $key (mergeOverwrite (deepCopy (index $out $key | default dict)) $value) -}} + {{- else -}} + {{- $_ := set $out $key (deepCopy $value) -}} + {{- end -}} +{{- end -}} +{{- $_ := set $out "storage" (mergeOverwrite (deepCopy .root.Values.engineSpec.defaultStorage) (.engine.storage | default dict)) -}} +{{/* An explicit per-engine storage type supersedes an inherited legacy alias. */}} +{{- if and $out.hostPathStorageEnabled (or (not (hasKey .engine "storageType")) (hasKey .engine "hostPathStorageEnabled")) -}} + {{- if eq $out.storageType "emptyDir" -}}{{- fail "hostPathStorageEnabled and storageType=emptyDir are mutually exclusive" -}}{{- end -}} + {{- $_ := set $out "storageType" "hostPath" -}} +{{- end -}} +{{- toYaml $out -}} +{{- end -}} diff --git a/helm/templates/_validation.tpl b/helm/templates/_validation.tpl index a1dbb24..8241a4c 100644 --- a/helm/templates/_validation.tpl +++ b/helm/templates/_validation.tpl @@ -52,6 +52,9 @@ {{- range .Values.engines -}} {{- if hasKey $seen .name -}}{{- fail (printf "duplicate engine name: %s" .name) -}}{{- end -}} {{- $_ := set $seen .name true -}} + {{- if and .customEngineConfig (kindIs "map" .customEngineConfig.instance) (hasKey .customEngineConfig.instance "id") -}} + {{- fail "engines[].customEngineConfig.instance.id cannot override installation identity" -}} + {{- end -}} {{- if or (gt (len (printf "%s-engine-%s-node-%d-0" $name .name (int (sub .replicas 1)))) 63) (gt (len (printf "%s-engine-%s-ready" $name .name)) 63) -}} {{- fail (printf "engine %s: generated pod or Service name exceeds 63 characters; shorten the release or engine name" .name) -}} {{- end -}} @@ -60,7 +63,7 @@ {{- include "fbinstance.validatePodExtras" (dict "context" "engineSpec" "env" .Values.engineSpec.extraEnv "volumes" .Values.engineSpec.customVolumes "mounts" .Values.engineSpec.customVolumeMounts "annotations" .Values.engineSpec.podAnnotations "reservedEnv" (list "FIREBOLT_CORE_NODE" "POD_UID" "NODES_COUNT" "FB_AWS_EC2_METADATA_CLIENT_ENABLED" "FIREBOLT_CORE_MODE") "reservedVolumes" (list "data" "runtime" "engine-config" "auth-admin" "engine-tls-raw" "engine-tls-chain" "nginx-writable-dir") "reservedPaths" (list "/" "/var/lib/firebolt" "/var/lib/firebolt/config.yaml" "/run/firebolt" "/secrets" "/etc/firebolt/tls/engine")) -}} {{- range $component := list "metadata" "gateway" -}} {{- $p := index $root.Values $component "podTemplate" -}} - {{- include "fbinstance.validatePodExtras" (dict "context" $component "env" $p.extraEnv "volumes" $p.volumes "mounts" $p.volumeMounts "annotations" $p.podAnnotations "reservedEnv" (list "POSTGRES_USERNAME_FILE" "POSTGRES_PASSWORD_FILE" "POSTGRES_SSLMODE" "POSTGRES_SSLROOTCERT") "reservedVolumes" (list "config" "postgres-creds" "postgres-ca" "tmp" "gateway-config" "gateway-tls" "engine-ca") "reservedPaths" (list "/" "/configs" $root.Values.postgresql.credentials.mountPath "/etc/envoy" "/tmp")) -}} + {{- include "fbinstance.validatePodExtras" (dict "context" $component "env" $p.extraEnv "volumes" $p.volumes "mounts" $p.volumeMounts "annotations" $p.podAnnotations "reservedEnv" (list "POSTGRES_USERNAME_FILE" "POSTGRES_PASSWORD_FILE" "PGSSLMODE" "PGSSLROOTCERT") "reservedVolumes" (list "config" "postgres-creds" "postgres-ca" "tmp" "gateway-config" "gateway-tls" "engine-ca") "reservedPaths" (list "/" "/configs" "/secrets/postgres-ca" $root.Values.postgresql.credentials.mountPath "/etc/envoy" "/tmp")) -}} {{- end -}} {{- $seen = dict -}} {{- range .Values.auth.signingKeys -}} @@ -70,6 +73,10 @@ {{- if and (not .Values.postgresql.local_enabled) (not .Values.postgresql.host) -}} {{- fail "postgresql.host is required when postgresql.local_enabled is false" -}} {{- end -}} +{{- if .Values.postgresql.tls.enabled -}} + {{- if .Values.postgresql.local_enabled -}}{{- fail "postgresql.tls requires an external database (local_enabled=false)" -}}{{- end -}} + {{- if or (not .Values.postgresql.tls.caSecret) (not .Values.postgresql.tls.caKey) -}}{{- fail "postgresql.tls requires caSecret and caKey" -}}{{- end -}} +{{- end -}} {{/* Helm client-side rendering has no live state; deployment tooling must enforce identity there. */}} {{- if .Release.IsUpgrade -}} {{- $existing := lookup "v1" "ConfigMap" .Release.Namespace (printf "%s-metadata-service" $name) -}} diff --git a/helm/templates/engine-clusterip-service.yaml b/helm/templates/engine-clusterip-service.yaml index 509823d..d88819a 100644 --- a/helm/templates/engine-clusterip-service.yaml +++ b/helm/templates/engine-clusterip-service.yaml @@ -1,5 +1,6 @@ {{- range .Values.engines }} {{- $engine := . }} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} --- apiVersion: v1 kind: Service @@ -16,5 +17,5 @@ spec: {{- include "fbinstance.selectorLabels" $ | nindent 4 }} firebolt/engine: {{ $engine.name }} ports: - {{- include "fbinstance.enginePorts" $ | nindent 4 }} + {{- include "fbinstance.enginePorts" (dict "Values" (dict "engineSpec" $effective)) | nindent 4 }} {{- end }} diff --git a/helm/templates/engine-headless-service.yaml b/helm/templates/engine-headless-service.yaml index f0fa5b8..54eb3a8 100644 --- a/helm/templates/engine-headless-service.yaml +++ b/helm/templates/engine-headless-service.yaml @@ -1,5 +1,6 @@ {{- range .Values.engines }} {{- $engine := . }} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} --- apiVersion: v1 kind: Service @@ -17,5 +18,5 @@ spec: {{- include "fbinstance.selectorLabels" $ | nindent 4 }} firebolt/engine: {{ $engine.name }} ports: - {{- include "fbinstance.enginePorts" $ | nindent 4 }} + {{- include "fbinstance.enginePorts" (dict "Values" (dict "engineSpec" $effective)) | nindent 4 }} {{- end }} diff --git a/helm/templates/engine-ready-service.yaml b/helm/templates/engine-ready-service.yaml index 917c2b7..dc11a1b 100644 --- a/helm/templates/engine-ready-service.yaml +++ b/helm/templates/engine-ready-service.yaml @@ -1,5 +1,6 @@ {{- range .Values.engines }} {{- $engine := . }} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} --- apiVersion: v1 kind: Service @@ -17,5 +18,5 @@ spec: {{- include "fbinstance.selectorLabels" $ | nindent 4 }} firebolt/engine: {{ $engine.name }} ports: - {{- include "fbinstance.enginePorts" $ | nindent 4 }} + {{- include "fbinstance.enginePorts" (dict "Values" (dict "engineSpec" $effective)) | nindent 4 }} {{- end }} diff --git a/helm/templates/engine-serviceaccount.yaml b/helm/templates/engine-serviceaccount.yaml index 23653e8..4b27b66 100644 --- a/helm/templates/engine-serviceaccount.yaml +++ b/helm/templates/engine-serviceaccount.yaml @@ -1,8 +1,13 @@ -{{- if empty .Values.engineSpec.serviceAccount }} +{{- $needsManaged := false -}} +{{- range $engine := .Values.engines -}} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml -}} +{{- if empty $effective.serviceAccount -}}{{- $needsManaged = true -}}{{- end -}} +{{- end -}} +{{- if $needsManaged }} apiVersion: v1 kind: ServiceAccount metadata: - name: {{ include "fbinstance.engineServiceAccountName" . }} + name: {{ include "fbinstance.fullname" . }}-engine namespace: {{ .Release.Namespace }} labels: {{- include "fbinstance.labels" . | nindent 4 }} @@ -10,13 +15,9 @@ metadata: # Engine pods do not call the Kubernetes API. Suppressing the token # mount means a code-execution exploit inside the engine cannot use a # projected SA token to talk to the apiserver at all, even if the SA -# accumulates accidental RBAC later. Setting this on the SA (rather -# than at the pod level) is the more reliable knob: kubelet honours -# it for any pod referencing this SA, regardless of who renders the -# pod template. +# accumulates accidental RBAC later. Engine pods also disable automounting +# explicitly, including when they use a supplied ServiceAccount. # -# Rendered only when `engineSpec.serviceAccount` is empty. A non-empty -# value means the user brings their own SA (e.g. IRSA), and the chart -# does not own its `automountServiceAccountToken` setting. +# Rendered when any Engine uses the managed ServiceAccount. automountServiceAccountToken: false {{- end }} diff --git a/helm/templates/engine-statefulset.yaml b/helm/templates/engine-statefulset.yaml index 45503c1..98124dd 100644 --- a/helm/templates/engine-statefulset.yaml +++ b/helm/templates/engine-statefulset.yaml @@ -30,6 +30,7 @@ whole instance), not per-node, so there's no reason to recompute them inside the {{- end -}} {{- range .Values.engines }} {{- $engine := . }} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} {{- range $i := until (int $engine.replicas) }} --- apiVersion: apps/v1 @@ -37,7 +38,7 @@ kind: StatefulSet metadata: namespace: {{ $.Release.Namespace }} name: {{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }} - {{- with $.Values.engineSpec.annotations }} + {{- with $effective.annotations }} annotations: {{- toYaml . | nindent 4 }} {{- end }} @@ -83,7 +84,7 @@ spec: {{- if $.Values.tls.engine.enabled }} checksum/tls-engine: {{ (lookup "v1" "Secret" $.Release.Namespace $engineTlsSecret).data | toYaml | sha256sum }} {{- end }} - {{- with $engine.podAnnotations }} + {{- with $effective.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} labels: @@ -92,7 +93,7 @@ spec: firebolt/engine: {{ $engine.name }} firebolt/node: {{ $i | quote }} {{- $reserved := list "firebolt/component" "firebolt/engine" "firebolt/node" "app.kubernetes.io/name" "app.kubernetes.io/instance" "app.kubernetes.io/managed-by" }} - {{- range $k, $v := $engine.podLabels }} + {{- range $k, $v := $effective.podLabels }} {{- if not (has $k $reserved) }} {{ $k }}: {{ $v | quote }} {{- end }} @@ -105,31 +106,32 @@ spec: # that also risks colliding with firebolt-core's own config keys # (the `FIREBOLT_*` prefix is owned by the engine binary). enableServiceLinks: false - {{- if $engine.priorityClassName }} - priorityClassName: {{ $engine.priorityClassName }} + {{- if $effective.priorityClassName }} + priorityClassName: {{ $effective.priorityClassName }} {{- end }} - serviceAccountName: {{ include "fbinstance.engineServiceAccountName" $ }} - terminationGracePeriodSeconds: {{ $.Values.engineSpec.terminationGracePeriodSeconds }} - {{- with $.Values.engineSpec.podSecurityContext }} + serviceAccountName: {{ $effective.serviceAccount | default (printf "%s-engine" (include "fbinstance.fullname" $)) }} + automountServiceAccountToken: false + terminationGracePeriodSeconds: {{ $effective.terminationGracePeriodSeconds }} + {{- with $effective.podSecurityContext }} securityContext: {{- toYaml . | nindent 8 }} {{- end }} - {{- if and $.Values.engineSpec.memlockSetup $.Values.engineSpec.podSecurityContext.runAsNonRoot }} + {{- if and $effective.memlockSetup $effective.podSecurityContext.runAsNonRoot }} shareProcessNamespace: true {{- end }} - {{- with ($engine.tolerations | default $.Values.engineSpec.tolerations) }} + {{- with $effective.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} - {{- with ($engine.affinity | default $.Values.engineSpec.affinity) }} + {{- with $effective.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} - {{- with ($engine.nodeSelector | default $.Values.engineSpec.nodeSelector) }} + {{- with $effective.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} - {{- with ($engine.topologySpreadConstraints | default $.Values.engineSpec.topologySpreadConstraints) }} + {{- with $effective.topologySpreadConstraints }} topologySpreadConstraints: {{- toYaml . | nindent 8 }} {{- end }} @@ -140,8 +142,8 @@ spec: {{- end }} containers: - name: core - image: "{{ $.Values.engineSpec.image.repository }}:{{ default $.Chart.AppVersion $.Values.engineSpec.image.tag }}" - imagePullPolicy: {{ $.Values.engineSpec.image.pullPolicy }} + image: "{{ $effective.image.repository }}:{{ default $.Chart.AppVersion $effective.image.tag }}" + imagePullPolicy: {{ $effective.image.pullPolicy }} env: - name: FIREBOLT_CORE_NODE value: {{ $i | quote }} @@ -161,10 +163,10 @@ spec: # startup. - name: FIREBOLT_CORE_MODE value: "1" - {{- with $.Values.engineSpec.extraEnv }} + {{- with $effective.extraEnv }} {{- toYaml . | nindent 12 }} {{- end }} - {{- with $.Values.engineSpec.extraEnvFrom }} + {{- with $effective.extraEnvFrom }} envFrom: {{- toYaml . | nindent 12 }} {{- end }} @@ -196,13 +198,13 @@ spec: echo -n "[Setup] Firebolt Core version " firebolt --version - {{- if $.Values.engineSpec.podSecurityContext.runAsNonRoot }} + {{- if $effective.podSecurityContext.runAsNonRoot }} if [[ $(id -u) -eq 0 ]]; then echo "[Setup] Core container is running as root, however deployment used podSecurityContext.runAsNonRoot=true" 1>&2 exit 4 fi - {{- if $.Values.engineSpec.memlockSetup }} + {{- if $effective.memlockSetup }} echo "[Setup] Core entrypoint PID: $BASHPID" rm -f /var/lib/firebolt/entrypoint-*.pid PID_FILE="/var/lib/firebolt/entrypoint-$POD_UID.pid" @@ -222,7 +224,7 @@ spec: exit 4 fi - {{- if $.Values.engineSpec.memlockSetup }} + {{- if $effective.memlockSetup }} ## NOTE: no 'ulimit' call happens here because root images support is soon to be deprecated echo "WARNING: memlockSetup=true is unnecessary when podSecurityContext.runAsNonRoot is false (or unset)" 1>&2 {{- end }} @@ -260,7 +262,7 @@ spec: httpGet: path: /health/live port: 8122 - {{- if $.Values.engineSpec.readiness }} + {{- if $effective.readiness }} readinessProbe: initialDelaySeconds: 1 periodSeconds: 3 @@ -278,7 +280,7 @@ spec: path: /health/live port: 8122 resources: - {{- toYaml $engine.resources | nindent 12 }} + {{- toYaml $effective.resources | nindent 12 }} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: @@ -310,7 +312,7 @@ spec: mountPath: /etc/firebolt/tls/engine readOnly: true {{- end }} - {{- with $.Values.engineSpec.customVolumeMounts }} + {{- with $effective.customVolumeMounts }} {{- toYaml . | nindent 12 }} {{- end }} # Scratch space outside the read-only root filesystem. @@ -327,7 +329,7 @@ spec: # and postgres containers all already enforce this — the # engine matching them closes the last writable-root gap. readOnlyRootFilesystem: true - {{- if $.Values.engineSpec.podSecurityContext.runAsNonRoot }} + {{- if $effective.podSecurityContext.runAsNonRoot }} runAsNonRoot: true runAsUser: 3473 runAsGroup: 3473 @@ -336,10 +338,10 @@ spec: capabilities: drop: - ALL - {{- if $.Values.engineSpec.uiSidecar }} + {{- if $effective.uiSidecar }} - name: core-ui - image: {{ $.Values.engineSpec.uiSidecarImage.repository }}:{{ $.Values.engineSpec.uiSidecarImage.tag }} - imagePullPolicy: {{ $.Values.engineSpec.uiSidecarImage.pullPolicy }} + image: {{ $effective.uiSidecarImage.repository }}:{{ $effective.uiSidecarImage.tag }} + imagePullPolicy: {{ $effective.uiSidecarImage.pullPolicy }} env: - name: FIREBOLT_CORE_URL # The sidecar's nginx reverse proxy defaults proxy_ssl_verify to off, so @@ -385,10 +387,10 @@ spec: - name: nginx-writable-dir mountPath: /var/run/nginx {{- end }} - {{- if and $.Values.engineSpec.memlockSetup $.Values.engineSpec.podSecurityContext.runAsNonRoot }} + {{- if and $effective.memlockSetup $effective.podSecurityContext.runAsNonRoot }} - name: memlock-setup image: {{ $.Values.utilitiesImage }} - imagePullPolicy: {{ $.Values.engineSpec.image.pullPolicy }} + imagePullPolicy: {{ $effective.image.pullPolicy }} resources: requests: cpu: 50m @@ -417,11 +419,11 @@ spec: - name: data mountPath: /var/lib/firebolt {{- end }} -{{- $hpEnabled := $.Values.engineSpec.hostPathStorageEnabled }} -{{- if or $.Values.engineSpec.customInitContainersTemplate $hpEnabled $.Values.tls.engine.enabled }} +{{- $hpEnabled := eq $effective.storageType "hostPath" }} +{{- if or $effective.customInitContainersTemplate $hpEnabled $.Values.tls.engine.enabled }} initContainers: - {{- if $.Values.engineSpec.customInitContainersTemplate }} - {{- tpl (toYaml $.Values.engineSpec.customInitContainersTemplate) $ | nindent 8 }} + {{- if $effective.customInitContainersTemplate }} + {{- tpl (toYaml $effective.customInitContainersTemplate) $ | nindent 8 }} {{- end }} {{- if $.Values.tls.engine.enabled }} - name: tls-chain-setup @@ -437,7 +439,7 @@ spec: # silently shipping a leaf-only chain that fails self-verification # later with a much less obvious curl/SSL error. image: {{ $.Values.utilitiesImage }} - imagePullPolicy: {{ $.Values.engineSpec.image.pullPolicy }} + imagePullPolicy: {{ $effective.image.pullPolicy }} command: ["/bin/bash", "-c"] args: - | @@ -469,7 +471,7 @@ spec: {{- if $hpEnabled }} - name: host-path-setup image: {{ $.Values.utilitiesImage }} - imagePullPolicy: {{ $.Values.engineSpec.image.pullPolicy }} + imagePullPolicy: {{ $effective.image.pullPolicy }} command: ["/bin/bash", "-c"] args: - | @@ -494,7 +496,7 @@ spec: {{- end }} {{- end }} volumes: - {{- if $.Values.engineSpec.uiSidecar }} + {{- if $effective.uiSidecar }} - name: nginx-writable-dir emptyDir: {} {{- end }} @@ -520,21 +522,25 @@ spec: - name: engine-tls-chain emptyDir: {} {{- end }} - {{- with $.Values.engineSpec.customVolumes }} + {{- with $effective.customVolumes }} {{- toYaml . | nindent 8 }} {{- end }} + {{- if eq $effective.storageType "emptyDir" }} + - name: data + emptyDir: {{ toYaml $effective.emptyDir | nindent 12 }} + {{- end }} {{- if $hpEnabled }} - name: data hostPath: - path: {{ $.Values.engineSpec.storageHostPath.path }} - type: {{ default "DirectoryOrCreate" $.Values.engineSpec.storageHostPath.type }} + path: {{ $effective.storageHostPath.path }} + type: {{ default "DirectoryOrCreate" $effective.storageHostPath.type }} {{- end }} -{{- if not $hpEnabled }} +{{- if eq $effective.storageType "pvc" }} volumeClaimTemplates: - metadata: name: data spec: - {{- $storage := $engine.storage | default $.Values.engineSpec.defaultStorage }} + {{- $storage := $effective.storage }} {{- if hasKey $storage "storageClassName" }} storageClassName: {{ $storage.storageClassName | quote }} {{- end }} diff --git a/helm/templates/gateway-configmap.yaml b/helm/templates/gateway-configmap.yaml index a49f6b2..eaa5285 100644 --- a/helm/templates/gateway-configmap.yaml +++ b/helm/templates/gateway-configmap.yaml @@ -87,30 +87,7 @@ data: "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua default_source_code: inline_string: | - local function is_valid_engine(s) - if s == nil or #s == 0 or #s > 63 then return false end - if not string.match(s, "^[%l%d][-%l%d]*$") then return false end - if string.sub(s, -1) == "-" then return false end - return true - end - - function envoy_on_request(handle) - local engine = handle:headers():get("x-firebolt-engine") - if not is_valid_engine(engine) then - handle:respond({[":status"] = "400"}, "invalid or missing X-Firebolt-Engine header") - return - end - -- TODO: remove advanced_mode once Core supports x-request-id - local path = handle:headers():get(":path") - if path:find("?", 1, true) then - handle:headers():replace(":path", path .. "&advanced_mode=true") - else - handle:headers():replace(":path", path .. "?advanced_mode=true") - end - -- Route to the engine's ready-only Service so Envoy resolves - -- per-pod IPs and load-balances across them itself. - handle:headers():replace(":authority", "{{ include "fbinstance.fullname" . }}-engine-" .. engine .. "-ready.{{ .Release.Namespace }}.svc.cluster.local:3473") - end + {{- tpl (.Files.Get "files/gateway-routing.lua") . | nindent 30 }} - name: envoy.filters.http.dynamic_forward_proxy typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig diff --git a/helm/templates/metadata-service-deployment.yaml b/helm/templates/metadata-service-deployment.yaml index 18ccfdb..7cc6b4d 100644 --- a/helm/templates/metadata-service-deployment.yaml +++ b/helm/templates/metadata-service-deployment.yaml @@ -27,6 +27,9 @@ spec: annotations: checksum/config: {{ (include (print $.Template.BasePath "/metadata-service-configmap.yaml") . | fromYaml).data | toYaml | sha256sum }} checksum/secret: {{ toYaml .Values.postgresql | sha256sum }} + {{- if .Values.postgresql.tls.enabled }} + checksum/postgres-ca-version: {{ (lookup "v1" "Secret" .Release.Namespace .Values.postgresql.tls.caSecret).metadata | default dict | dig "resourceVersion" "" | sha256sum }} + {{- end }} {{- if .Values.postgresql.credentials.existingSecret }} checksum/credentials-version: {{ (lookup "v1" "Secret" .Release.Namespace $pgSecretName).metadata | default dict | dig "resourceVersion" "" | sha256sum }} {{- end }} @@ -117,6 +120,11 @@ spec: - name: postgres-creds mountPath: {{ .Values.postgresql.credentials.mountPath }} readOnly: true + {{- if .Values.postgresql.tls.enabled }} + - name: postgres-ca + mountPath: /secrets/postgres-ca + readOnly: true + {{- end }} # Scratch space outside the read-only root filesystem. - name: tmp mountPath: /tmp @@ -146,6 +154,12 @@ spec: value: {{ .Values.postgresql.credentials.mountPath }}/username - name: POSTGRES_PASSWORD_FILE value: {{ .Values.postgresql.credentials.mountPath }}/password + {{- if .Values.postgresql.tls.enabled }} + - name: PGSSLMODE + value: verify-full + - name: PGSSLROOTCERT + value: /secrets/postgres-ca/ca.crt + {{- end }} {{- with .Values.metadata.podTemplate.extraEnv }} {{- toYaml . | nindent 12 }} {{- end }} @@ -163,6 +177,14 @@ spec: - name: postgres-creds secret: secretName: {{ $pgSecretName }} + {{- if .Values.postgresql.tls.enabled }} + - name: postgres-ca + secret: + secretName: {{ .Values.postgresql.tls.caSecret }} + items: + - key: {{ .Values.postgresql.tls.caKey }} + path: ca.crt + {{- end }} - name: tmp emptyDir: {} {{- with .Values.metadata.podTemplate.volumes }} diff --git a/helm/templates/tests/test-auth.yaml b/helm/templates/tests/test-auth.yaml index 3c324f5..b0aa7bc 100644 --- a/helm/templates/tests/test-auth.yaml +++ b/helm/templates/tests/test-auth.yaml @@ -7,6 +7,7 @@ independent of gateway.enabled / tls.gateway settings. */ -}} {{- if .Values.auth.enabled }} {{- $engine := index .Values.engines 0 }} +{{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} apiVersion: v1 kind: Pod metadata: @@ -36,7 +37,7 @@ spec: - | set -e {{- include "fbinstance.testShellHelpers" . | nindent 10 }} - HOST="{{ include "fbinstance.fullname" . }}-engine-{{ $engine.name }}-node-0-0.{{ include "fbinstance.fullname" . }}-engine-{{ $engine.name }}-hl.{{ .Release.Namespace }}.svc{{ .Values.engineSpec.nodeHostSuffix }}" + HOST="{{ include "fbinstance.fullname" . }}-engine-{{ $engine.name }}-node-0-0.{{ include "fbinstance.fullname" . }}-engine-{{ $engine.name }}-hl.{{ .Release.Namespace }}.svc{{ $effective.nodeHostSuffix }}" {{- if .Values.tls.engine.enabled }} # tls.engine.enabled: the query listener speaks HTTPS. This test only exercises # auth enforcement, not certificate trust (the live TLS path covers that diff --git a/helm/templates/tests/test-engine-dns.yaml b/helm/templates/tests/test-engine-dns.yaml index 9e66f51..7dddbc7 100644 --- a/helm/templates/tests/test-engine-dns.yaml +++ b/helm/templates/tests/test-engine-dns.yaml @@ -18,8 +18,9 @@ spec: set -e {{- include "fbinstance.testShellHelpers" $ | nindent 10 }} {{- range $engine := .Values.engines }} + {{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} {{- range $i := until (int $engine.replicas) }} - FQDN="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc.cluster.local" + FQDN="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $effective.nodeHostSuffix }}" log "DNS: engine {{ $engine.name }} node-{{ $i }}" nslookup "$FQDN" >/dev/null 2>&1 || fail "$FQDN did not resolve" pass "{{ $engine.name }} node-{{ $i }} DNS resolves" diff --git a/helm/templates/tests/test-engine-pods-reachable.yaml b/helm/templates/tests/test-engine-pods-reachable.yaml index 126207f..8f3ea80 100644 --- a/helm/templates/tests/test-engine-pods-reachable.yaml +++ b/helm/templates/tests/test-engine-pods-reachable.yaml @@ -18,8 +18,9 @@ spec: set -e {{- include "fbinstance.testShellHelpers" $ | nindent 10 }} {{- range $engine := .Values.engines }} + {{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} {{- range $i := until (int $engine.replicas) }} - HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc.cluster.local" + HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $effective.nodeHostSuffix }}" log "Reachability: engine {{ $engine.name }} node-{{ $i }} ($HOST:8122)" nc -z -w 5 "$HOST" 8122 || fail "{{ $engine.name }} node-{{ $i }} health port unreachable" pass "{{ $engine.name }} node-{{ $i }} health port reachable" diff --git a/helm/templates/tests/test-engine-ready.yaml b/helm/templates/tests/test-engine-ready.yaml index 023225c..8466a82 100644 --- a/helm/templates/tests/test-engine-ready.yaml +++ b/helm/templates/tests/test-engine-ready.yaml @@ -20,8 +20,9 @@ spec: set -e {{- include "fbinstance.testShellHelpers" $ | nindent 10 }} {{- range $engine := .Values.engines }} + {{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} {{- range $i := until (int $engine.replicas) }} - HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc.cluster.local" + HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $effective.nodeHostSuffix }}" log "Readiness: engine {{ $engine.name }} node-{{ $i }} (GET http://$HOST:8122/health/ready)" HTTP_CODE=$(curl -s -o /dev/null -w '%{http_code}' \ --retry 12 --retry-delay 5 --retry-connrefused \ diff --git a/helm/templates/tests/test-postgres.yaml b/helm/templates/tests/test-postgres.yaml index 8bc9d30..9868be1 100644 --- a/helm/templates/tests/test-postgres.yaml +++ b/helm/templates/tests/test-postgres.yaml @@ -1,4 +1,3 @@ -{{- if .Values.postgresql.local_enabled }} apiVersion: v1 kind: Pod metadata: @@ -9,6 +8,15 @@ metadata: helm.sh/hook-delete-policy: before-hook-creation spec: restartPolicy: Never + {{- if .Values.postgresql.tls.enabled }} + volumes: + - name: postgres-ca + secret: + secretName: {{ .Values.postgresql.tls.caSecret }} + items: + - key: {{ .Values.postgresql.tls.caKey }} + path: ca.crt + {{- end }} containers: - name: check image: {{ .Values.postgresql.image }} @@ -24,22 +32,33 @@ spec: name: {{ include "fbinstance.postgresSecretName" . }} key: password - name: PGDATABASE - valueFrom: - secretKeyRef: - name: {{ include "fbinstance.postgresSecretName" . }} - key: database + value: {{ .Values.postgresql.database | quote }} + {{- if .Values.postgresql.tls.enabled }} + - name: PGSSLMODE + value: verify-full + - name: PGSSLROOTCERT + value: /trust/ca.crt + volumeMounts: + - name: postgres-ca + mountPath: /trust + readOnly: true + {{- end }} command: - sh - -c - | set -e {{- include "fbinstance.testShellHelpers" . | nindent 10 }} + {{- if .Values.postgresql.local_enabled }} HOST="{{ include "fbinstance.fullname" . }}-metadata-pg.{{ .Release.Namespace }}.svc.cluster.local" - log "PostgreSQL readiness (pg_isready $HOST:5432)" - pg_isready -h "$HOST" -p 5432 -U {{ .Values.postgresql.username }} -d {{ .Values.postgresql.database }} \ + {{- else }} + HOST={{ .Values.postgresql.host | quote }} + {{- end }} + export PGPORT={{ .Values.postgresql.port }} + log "PostgreSQL readiness (pg_isready $HOST:$PGPORT)" + pg_isready -h "$HOST" -p "$PGPORT" -U {{ .Values.postgresql.username }} -d {{ .Values.postgresql.database }} \ || fail "PostgreSQL not ready" pass "PostgreSQL is accepting connections" - RESULT=$(psql -h "$HOST" -p 5432 -At -v ON_ERROR_STOP=1 -c "SELECT 1") + RESULT=$(psql -h "$HOST" -p "$PGPORT" -At -v ON_ERROR_STOP=1 -c "SELECT 1") [ "$RESULT" = "1" ] || fail "credential Secret could not execute SELECT 1" pass "configured credentials authenticate to PostgreSQL" -{{- end }} diff --git a/helm/templates/tests/test-sql-query.yaml b/helm/templates/tests/test-sql-query.yaml index ff83c4c..a6473f7 100644 --- a/helm/templates/tests/test-sql-query.yaml +++ b/helm/templates/tests/test-sql-query.yaml @@ -83,6 +83,19 @@ spec: sql_query "SELECT 1" "SELECT 1" "1" + log "SQL: query-parameter Engine selector" + RESPONSE=$(curl --fail --silent $CURL_INSECURE --max-time 30 \ + "$GW/?engine=$ENGINE" \ + {{- if $.Values.auth.enabled }} + -H "Authorization: Bearer $ACCESS_TOKEN" \ + {{- end }} + --data "SELECT 1") + echo "$RESPONSE" | grep -q "1" || fail "query selector did not return SELECT 1" + STATUS=$(curl --silent $CURL_INSECURE --max-time 30 -o /dev/null -w '%{http_code}' \ + "$GW/?engine=$ENGINE&engine=$ENGINE" --data "SELECT 1") + [ "$STATUS" = "400" ] || fail "duplicate selectors returned $STATUS" + pass "query selector accepted and duplicate selectors rejected" + TABLE="helm_storage_smoke_$(printf '%s' "$ENGINE" | tr -c 'A-Za-z0-9_' '_')" sql_query "managed table cleanup" "DROP TABLE IF EXISTS $TABLE" "" sql_query "managed table create" "CREATE TABLE $TABLE (id INT, name TEXT)" "" diff --git a/helm/templates/tests/test-tls.yaml b/helm/templates/tests/test-tls.yaml index dc2f83d..8600891 100644 --- a/helm/templates/tests/test-tls.yaml +++ b/helm/templates/tests/test-tls.yaml @@ -54,8 +54,9 @@ spec: {{- include "fbinstance.testShellHelpers" . | nindent 10 }} {{- if .Values.tls.engine.enabled }} {{- range $engine := .Values.engines }} + {{- $effective := include "fbinstance.effectiveEngine" (dict "root" $ "engine" $engine) | fromYaml }} {{- range $i := until (int $engine.replicas) }} - HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $.Values.engineSpec.nodeHostSuffix }}" + HOST="{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-node-{{ $i }}-0.{{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-hl.{{ $.Release.Namespace }}.svc{{ $effective.nodeHostSuffix }}" log "Verify Engine TLS chain and hostname: $HOST" curl --fail --silent --show-error --max-time 30 \ --cacert /trust/engine/ca.crt "https://$HOST:3473/health/ready" diff --git a/helm/values.schema.json b/helm/values.schema.json index 9b9cf07..4bb5085 100644 --- a/helm/values.schema.json +++ b/helm/values.schema.json @@ -63,6 +63,90 @@ } } }, + "storageType": { + "type": "string", + "enum": [ + "pvc", + "emptyDir", + "hostPath" + ] + }, + "emptyDir": { + "type": "object", + "properties": { + "medium": { + "type": "string", + "enum": [ + "", + "Memory" + ] + }, + "sizeLimit": { + "type": "string" + } + } + }, + "terminationGracePeriodSeconds": { + "type": "integer", + "minimum": 6 + }, + "readiness": { + "type": "boolean" + }, + "uiSidecar": { + "type": "boolean" + }, + "serviceAccount": { + "type": "string" + }, + "hostPathStorageEnabled": { + "type": "boolean" + }, + "memlockSetup": { + "type": "boolean" + }, + "podAnnotations": { + "type": "object" + }, + "podLabels": { + "type": "object" + }, + "resources": { + "type": "object" + }, + "nodeSelector": { + "type": "object" + }, + "affinity": { + "type": "object" + }, + "podSecurityContext": { + "type": "object" + }, + "defaultStorage": { + "type": "object" + }, + "extraEnv": { + "type": "array" + }, + "extraEnvFrom": { + "type": "array" + }, + "customVolumes": { + "type": "array" + }, + "customVolumeMounts": { + "type": "array" + }, + "customInitContainersTemplate": { + "type": "array" + }, + "tolerations": { + "type": "array" + }, + "topologySpreadConstraints": { + "type": "array" + }, "annotations": { "type": "object", "additionalProperties": { @@ -147,6 +231,21 @@ "schema": { "type": "string", "pattern": "^[A-Za-z0-9_.-]+$" + }, + "tls": { + "type": "object", + "properties": { + "enabled": { + "type": "boolean" + }, + "caSecret": { + "type": "string" + }, + "caKey": { + "type": "string", + "minLength": 1 + } + } } } }, @@ -260,11 +359,27 @@ "replicas": { "type": "integer", "minimum": 1 + }, + "customEngineConfig": { + "type": "object", + "properties": { + "instance": { + "type": "object" + } + } + }, + "storage": { + "type": "object" } }, "required": [ "name", "replicas" + ], + "allOf": [ + { + "$ref": "#/definitions/engineOverrides" + } ] } }, @@ -396,6 +511,136 @@ "required": [ "id" ] + }, + "engineOverrides": { + "type": "object", + "properties": { + "image": { + "type": "object", + "properties": { + "repository": { + "type": "string", + "minLength": 1 + }, + "pullPolicy": { + "type": "string", + "enum": [ + "Always", + "Never", + "IfNotPresent" + ] + } + } + }, + "uiSidecarImage": { + "type": "object", + "properties": { + "repository": { + "type": "string", + "minLength": 1 + }, + "tag": { + "type": "string", + "minLength": 1 + }, + "pullPolicy": { + "type": "string", + "enum": [ + "Always", + "Never", + "IfNotPresent" + ] + } + } + }, + "storageType": { + "type": "string", + "enum": [ + "pvc", + "emptyDir", + "hostPath" + ] + }, + "emptyDir": { + "type": "object", + "properties": { + "medium": { + "type": "string", + "enum": [ + "", + "Memory" + ] + }, + "sizeLimit": { + "type": "string" + } + } + }, + "terminationGracePeriodSeconds": { + "type": "integer", + "minimum": 6 + }, + "readiness": { + "type": "boolean" + }, + "uiSidecar": { + "type": "boolean" + }, + "serviceAccount": { + "type": "string" + }, + "hostPathStorageEnabled": { + "type": "boolean" + }, + "memlockSetup": { + "type": "boolean" + }, + "podAnnotations": { + "type": "object" + }, + "podLabels": { + "type": "object" + }, + "resources": { + "type": "object" + }, + "nodeSelector": { + "type": "object" + }, + "affinity": { + "type": "object" + }, + "podSecurityContext": { + "type": "object" + }, + "extraEnv": { + "type": "array" + }, + "extraEnvFrom": { + "type": "array" + }, + "customVolumes": { + "type": "array" + }, + "customVolumeMounts": { + "type": "array" + }, + "customInitContainersTemplate": { + "type": "array" + }, + "tolerations": { + "type": "array" + }, + "topologySpreadConstraints": { + "type": "array" + }, + "annotations": { + "type": "object", + "additionalProperties": { + "type": "string" + } + } + } } } } diff --git a/helm/values.yaml b/helm/values.yaml index de52f64..8a0850a 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -18,7 +18,28 @@ createNamespace: false # -- Shared engine pod defaults applied to all engines unless overridden per-engine. # @default -- {} engineSpec: - # -- Annotations on Engine StatefulSets, for example to opt into an externally installed Secret reload controller. These do not annotate pods. + # -- Engine-local cache backend: pvc (retained by default), emptyDir (ephemeral), or hostPath. Changing backend on an existing StatefulSet requires a migration. + storageType: pvc + # -- Options for storageType=emptyDir, such as sizeLimit or medium: Memory. Local cache is lost when the pod is removed; metadata and object storage remain authoritative. + # @default -- {} + emptyDir: {} + # -- Shared container resource requests and limits, overridden as a whole by engines[].resources. + # @default -- {} + resources: + requests: + cpu: "1" + memory: "4Gi" + limits: + memory: "4Gi" + # -- Default priority class, overridden per Engine (an empty override clears it). + priorityClassName: "" + # -- Default custom pod annotations. checksum/ and firebolt.io/ are reserved. + # @default -- {} + podAnnotations: {} + # -- Default extra pod labels. Chart selector labels remain authoritative. + # @default -- {} + podLabels: {} + # -- Default annotations on Engine StatefulSets, replaced as a whole by engines[].annotations (an empty map clears them). These do not annotate pods. # @default -- {} annotations: {} image: @@ -63,12 +84,11 @@ engineSpec: # inherits any RoleBindings accumulated on `default` from unrelated # installs. # - # Non-empty: the chart references the named SA verbatim and does NOT - # create one — bring your own (the documented IRSA / Pod Identity - # flow at docs/usage/object-storage/amazon-s3.mdx works this way). The chart - # cannot influence `automountServiceAccountToken` on a SA it does - # not own; set it explicitly in your SA manifest if you want the - # same hardening. + # Non-empty: this Engine references the named SA verbatim (bring your own). + # A managed SA is still created if another Engine needs it. Every Engine + # pod sets automountServiceAccountToken: false, including with a supplied SA. + # Webhook-injected workload-identity tokens remain supported; see + # docs/usage/object-storage/amazon-s3.mdx. serviceAccount: "" # -- Tolerations for engine pod scheduling. tolerations: [] @@ -121,9 +141,9 @@ engineSpec: # mountPath: /etc/custom-config # readOnly: true - # -- When true, uses hostPath instead of PVC for engine data. + # -- Legacy alias for storageType=hostPath. An explicit engines[].storageType overrides the inherited alias. Setting both hostPathStorageEnabled=true and storageType=emptyDir on the same Engine is invalid. hostPathStorageEnabled: false - # -- Host path configuration used when `hostPathStorageEnabled` is true. + # -- Host path configuration used with storageType=hostPath or the legacy hostPathStorageEnabled alias. # @default -- {} storageHostPath: # -- Host path for engine data. @@ -269,6 +289,15 @@ metadata: # for an external database. # @default -- {} postgresql: + # -- TLS to external PostgreSQL. The bundled database does not provide a TLS listener. + # @default -- {} + tls: + # -- Require certificate-chain and hostname verification (verify-full). + enabled: false + # -- Secret containing the PostgreSQL server CA bundle, mounted read-only into the Metadata Service. + caSecret: "" + # -- Key containing the PEM CA bundle in caSecret. + caKey: ca.crt # -- When true, deploys a bundled PostgreSQL StatefulSet. Set to false to use an external database. local_enabled: true # -- PostgreSQL image used for the bundled StatefulSet. Pinned to an @@ -605,55 +634,17 @@ tls: # -- Engine definitions. Each entry produces one StatefulSet per node (`replicas` controls # node count), plus a shared headless Service, ClusterIP Service, and ConfigMap. -# Per-engine values override the shared `engineSpec` defaults. +# Per-engine values replace shared `engineSpec` defaults, including explicit false and empty maps/lists. +# Image fields and storage fields inherit individually; use per-engine storage for engineSpec.defaultStorage. # @default -- [] engines: - - # -- Engine name. Used to derive resource names across the chart. + - # -- Engine name. Generated pod and Service names must fit in 63 characters together with the release prefix and node suffix. name: default - # -- Number of nodes in this engine group (one StatefulSet replica per node). + # -- Number of nodes (one single-replica StatefulSet per node). Upgrades and topology changes can interrupt queries. replicas: 1 - # -- Resource requests and limits for engine containers. - # Firebolt Core is memory-bound: more RAM directly improves cache hit rates and query - # throughput. CPU governs parallel query execution threads. - # - # Typical sizing guidance: - # Development / functional testing: 2 vCPU / 8 Gi (request) - # Small production workload: 4 vCPU / 32 Gi - # Medium production workload: 8 vCPU / 64 Gi - # Large production workload: 16 vCPU / 128 Gi - # - # Storage I/O is also significant — use an SSD-backed StorageClass and - # size the PVC to hold your working dataset plus ~30 % headroom. - resources: - requests: - cpu: "1" - memory: "4Gi" - limits: - memory: "4Gi" - # -- PVC storage configuration for this engine. Falls back to `engineSpec.defaultStorage` if omitted. Set `storageClassName` to select a class, or an empty string to disable dynamic provisioning. Existing claim templates cannot be resized or switched by a normal Helm upgrade. - storage: - accessModes: [ReadWriteOnce] - resources: - requests: - storage: 100Gi - # -- Priority class name for engine pods. - priorityClassName: "" - # -- Tolerations for engine pod scheduling. - tolerations: [] - # -- Affinity rules for engine pod scheduling. - affinity: {} - # -- Node selector for engine pod scheduling. - nodeSelector: {} - # -- Topology spread constraints for engine pod scheduling. Overrides - # `engineSpec.topologySpreadConstraints` for this engine when set. - topologySpreadConstraints: [] - # -- Annotations applied to engine pods. - podAnnotations: {} - # -- Extra labels applied to this engine's pod template. Chart-reserved - # keys (`app.kubernetes.io/{name,instance,managed-by}` and - # `firebolt/{component,engine,node}`) are silently dropped from user - # input so the StatefulSet selector cannot be detached by a typo. - podLabels: {} + # -- Optional per-Engine config overlay, merged over customEngineConfig. Installation identity and chart-managed topology/auth/TLS fields cannot be overridden. + # @default -- {} + customEngineConfig: {} # @section -- Global