Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,11 @@ For day-2 operational capabilities such as zero-downtime engine rollouts, autosc
- [Security](./security) records the chart's pod-hardening baseline and the responsibilities it leaves to the platform team.

For a value-by-value reference, see the helm-docs-generated [`helm/README.md`](https://github.com/firebolt-db/firebolt-instance-helm/blob/main/helm/README.md).

## Upgrade behavior

The Gateway stops advertising readiness, waits five seconds for endpoint propagation, then closes query admission and waits for active connections. `gateway.terminationGracePeriodSeconds` bounds the whole shutdown (15 seconds by default). Requests still active at that deadline can be interrupted. A custom `gateway.podTemplate.lifecycle` replaces this default drain behavior.

Engine peer discovery uses a headless Service that publishes not-ready pods for bootstrap. The Gateway uses a separate `-ready` headless Service containing ready pods only, with active health checks in addition to Kubernetes readiness. Neither this filtering nor the Gateway's rolling update strategy coordinates Engine StatefulSet replacements. Engine upgrades and replica changes can interrupt queries.

When upgrading an existing TLS deployment, ensure your Engine certificate includes `<release>-engine-<name>-ready.<namespace>.svc.cluster.local` for every Engine before the Gateway switches to ready-only routing. Chart-managed Certificates include these SANs, but issuance and process restarts are separate operations. Use a maintenance window for the first transition, or pre-provision compatible certificate material and restart Engines before upgrading the Gateway.
15 changes: 14 additions & 1 deletion docs/usage/tls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ tls:
<Note>
`tls.engine`'s Secret requires all three keys — `tls.crt`, `tls.key`, and `ca.crt` — even for a self-signed certificate with no separate CA (set `ca.crt` to a copy of `tls.crt` in that case). The engine's own internal health check and the gateway's upstream connection both need the issuing CA to validate the certificate independently of `tls.crt`.

The certificate's SANs must also cover every engine node's per-node hostname (`<release>-engine-<engine>-node-<i>-0.<release>-engine-<engine>-hl.<namespace>.svc.cluster.local`) and each engine's headless-service hostname (`<release>-engine-<engine>-hl.<namespace>.svc.cluster.local`) — both the engine's self-health-check and the gateway's `auto_san_validation` verify against these names. A missing SAN leaves the affected engine node permanently `NotReady` with no other symptom. `tls.engine.certManager` derives these automatically; a BYO certificate has to include them explicitly.
The certificate's SANs must also cover every engine node's per-node hostname (`<release>-engine-<engine>-node-<i>-0.<release>-engine-<engine>-hl.<namespace>.svc.cluster.local`) and each Engine's ready-only serving hostname (`<release>-engine-<engine>-ready.<namespace>.svc.cluster.local`) — both the engine's self-health-check and the gateway's `auto_san_validation` verify against these names. A missing node SAN can leave the Engine `NotReady`. A missing serving-Service SAN can instead leave Engine pods ready while Gateway queries fail certificate verification. `tls.engine.certManager` derives these automatically; a BYO certificate has to include them explicitly.
</Note>

### Install
Expand All @@ -50,6 +50,19 @@ helm install firebolt ./helm \
-f my-values.yaml
```

### Upgrade from discovery-Service query routing

Older chart versions route Gateway queries through `…-hl`. This chart routes through `…-ready`, so existing Engine certificates must include the new serving hostname before the Gateway switches. This applies to both existing Secrets and cert-manager certificates. Helm does not wait for certificate reissuance or coordinate this transition.

For an existing TLS deployment, schedule a maintenance window and use this sequence:

1. Keep the old Gateway and chart configuration in place. Issue an Engine certificate containing the existing node and `…-hl` SANs plus `…-ready` for every Engine. For cert-manager, add these SANs to the existing Engine `Certificate` and wait for the replacement revision and Secret. Do not run an old-chart reconciliation that removes the added SANs during this step.
2. After the replacement Secret exists, restart all Engine StatefulSets and wait for their rollouts. The init container must rebuild the certificate chain. Check the certificate served by every node against both its node hostname and the new serving hostname.
3. Upgrade to this chart using the complete deployment values. The Gateway can now use the ready-only Service with certificates that already cover it.
4. Run `helm test` and verify SQL through the Gateway before ending the maintenance window. If certificate issuance or verification fails, leave the Gateway on the previous routing configuration.

For offline/GitOps deployments, stage the certificate resource or externally managed Secret, Engine restart, and chart upgrade as separate reconciliations. A successful `helm upgrade --wait` alone does not prove that the Gateway can query Engines using the new TLS identity. Preserve the old SANs while rollback to the previous chart remains possible.

### cert-manager as a certificate source

```yaml
Expand Down
4 changes: 2 additions & 2 deletions helm/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g
- Envoy gateway: `Deployment` + `Service` (configurable type) + `ConfigMap` + optional `PodDisruptionBudget` (`gateway-pdb.yaml`). Routes by `X-Firebolt-Engine` header. Optionally terminates client TLS (`tls.gateway.enabled`) and/or speaks TLS upstream to engines (`tls.engine.enabled`).
- Pensieve metadata: `Deployment` + `Service` + `ConfigMap` + `Secret`.
- PostgreSQL: bundled `StatefulSet` + `Service` when `postgresql.local_enabled: true`; otherwise the chart consumes external connection details / `existingSecret`.
- Engines: **one 1-replica `StatefulSet` per node per engine** (i.e. `engines[i].replicas` separate StatefulSets named `<release>-engine-<name>-node-<i>`), plus a shared headless Service, ClusterIP Service, and ConfigMap. Optionally authenticated (`auth.enabled`) and/or TLS-terminated on the query listener (`tls.engine.enabled`).
- Engines: **one 1-replica `StatefulSet` per node per engine** (i.e. `engines[i].replicas` separate StatefulSets named `<release>-engine-<name>-node-<i>`), plus a bootstrap headless Service, a ready-only serving headless Service, a ClusterIP Service, and ConfigMap. Optionally authenticated (`auth.enabled`) and/or TLS-terminated on the query listener (`tls.engine.enabled`).
- Optional cert-manager `Certificate` resources (`certificates.yaml`) — one per `auth.signingKeys[]` entry, `tls.gateway`, or `tls.engine` that requests a `certManager` block instead of an `existingSecret`. Requires cert-manager and its CRDs in-cluster; absent entirely when every artifact uses `existingSecret`.
- Optional `PodMonitor`s — separate ones for engines (`podmonitor.yaml`) and the gateway (`podmonitor-gateway.yaml`), each gated by `podMonitor.engines.enabled` / `podMonitor.gateway.enabled`. Both require the Prometheus Operator CRDs.
- Optional `Namespace` (`createNamespace: true`).
Expand Down Expand Up @@ -55,7 +55,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g
- `.helmignore` — excludes docs and scripts from the packaged chart.
- `templates/_helpers.tpl` — naming, label, port, engine-config (incl. `instance.auth` / engine TLS `endpoints`), secret-source resolution (`fbinstance.secretSourceName` / `fbinstance.usesCertManager`), engine TLS SAN list (`fbinstance.engineTlsDnsNames`), memlock-script, and test-shell-helper templates.
- `templates/certificates.yaml` — optional cert-manager `Certificate` resources for `auth.signingKeys[]` / `tls.gateway` / `tls.engine` entries that request `certManager` instead of `existingSecret`.
- `templates/gateway-*.yaml` — Envoy Deployment, Service, ConfigMap, PodDisruptionBudget.
- `templates/gateway-*.yaml` — Envoy Deployment, Service, ConfigMap, PodDisruptionBudget. `files/gateway-prestop.sh` drains the INBOUND query listener; probes use the independent metrics listener.
- `templates/metadata-service-*.yaml` — Pensieve Deployment, Service, ConfigMap, Secret.
- `templates/postgresql-*.yaml` — bundled PostgreSQL StatefulSet and Service (rendered only when `postgresql.local_enabled: true`).
- `templates/engine-*.yaml` — per-engine StatefulSet(s), headless Service, ClusterIP Service, ConfigMap. The StatefulSet also mounts auth/TLS secrets and (when `tls.engine.enabled`) runs the `tls-chain-setup` init container.
Expand Down
1 change: 1 addition & 0 deletions helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines
| gateway.service | object | {} | Gateway Service configuration. |
| gateway.service.port | int | `80` | External service port proxied to `containerPort`. |
| gateway.service.type | string | `"ClusterIP"` | Service type. One of `ClusterIP`, `LoadBalancer`, or `NodePort`. |
| gateway.terminationGracePeriodSeconds | int | `15` | Total pod shutdown budget in seconds, including a 5s endpoint-propagation wait and explicit connection draining. Requests still active at the deadline can be interrupted. |
| imagePullSecrets | list | `[]` | Registry credentials. Must be a pre-created docker-registry Secret in the deployment namespace. Leave empty if nodes have ambient registry access (e.g. node IAM role). |
| metadata | object | {} | Metadata service configuration. |
| metadata.annotations | object | {} | Annotations on the Metadata Service Deployment, for example to opt into an externally installed Secret reload controller. These do not annotate pods. |
Expand Down
41 changes: 41 additions & 0 deletions helm/files/gateway-prestop.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# Stop admission after endpoint propagation, then wait for active connections.
set -u
admin() {
local response status
response=$(
exec 3<>/dev/tcp/127.0.0.1/{{ .Values.gateway.adminPort }} || exit 1
printf '%s\r\n' "$1 $2 HTTP/1.1" 'Host: localhost' 'Content-Length: 0' 'Connection: close' '' >&3 || exit 1
cat <&3
) || return 1
status=${response%%$'\n'*}
[[ "$status" == HTTP/1.*" 200 "* ]] || return 1
printf '%s\n' "$response"
}
until admin POST /healthcheck/fail >/dev/null; do sleep 0.1; done
# Readiness is now failing. Keep accepting until the Service has stopped
# routing here; only then close admission. See the propagation note above.
sleep 5
# graceful: in-flight requests finish and their connections close right after
# (drain-time-s is 0). The drain's completion also stops the INBOUND-marked
# listeners, so the query socket refuses new connections while the stats
# listener (not INBOUND) keeps serving probes and metrics. Verified against
# the pinned Envoy: adding skip_exit suppresses that listener stop and the
# socket keeps accepting, and without it the process still survives the
# drain - so skip_exit must stay absent.
until admin POST '/drain_listeners?inboundonly&graceful' >/dev/null; do sleep 0.1; done
while true; do
if response=$(admin GET '/stats?filter=^http[.]gateway[.]downstream_cx_active$'); then
if grep -Fxq 'http.gateway.downstream_cx_active: 0' <<<"$response"; then
exit 0
fi
# TLS provisioning can deliberately omit the public listener entirely.
# Missing statistics alone cannot establish that there are no connections.
if ! grep -q '^http[.]gateway[.]downstream_cx_active:' <<<"$response"; then
if listeners=$(admin GET /listeners) && ! grep -q '^listener::' <<<"$listeners"; then
exit 0
fi
fi
fi
sleep 0.1
done
53 changes: 45 additions & 8 deletions helm/scripts/test_gateway_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
the chart's route and retry policy are exercised unmodified.
"""
import http.client
from concurrent.futures import ThreadPoolExecutor
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import subprocess
Expand All @@ -20,7 +21,9 @@

class RetryRuntimeTests(unittest.TestCase):
def test_delivered_reset_is_not_replayed_but_drained_response_is(self):
calls = {"/reset": 0, "/drained": 0}
calls = {"/reset": 0, "/drained": 0, "/hold": 0}
accepted = threading.Event()
release = threading.Event()

class Backend(BaseHTTPRequestHandler):
def log_message(self, *_args):
Expand All @@ -33,7 +36,10 @@ def do_POST(self):
# The mutation has happened. Close before response headers.
self.close_connection = True
return
drained = calls[self.path] == 1
if self.path == "/hold":
accepted.set()
release.wait(15)
drained = self.path == "/drained" and calls[self.path] == 1
self.send_response(503 if drained else 200)
if drained:
self.send_header("X-Firebolt-Drained", "true")
Expand All @@ -48,6 +54,7 @@ def do_POST(self):
hcm["http_filters"] = [f for f in hcm["http_filters"] if f["name"] in {
"envoy.filters.http.health_check", "envoy.filters.http.router",
}]
admin_cluster = config["static_resources"]["clusters"][-1]
config["static_resources"]["clusters"] = [{
"name": "dynamic_forward_proxy", "type": "LOGICAL_DNS", "connect_timeout": "1s",
"dns_lookup_family": "V4_ONLY",
Expand All @@ -57,8 +64,9 @@ def do_POST(self):
}}}}],
}]},
}]
config["static_resources"]["listeners"] = [listener]
image = resource(render(), "Deployment", "audit-gateway")["spec"]["template"]["spec"]["containers"][0]["image"]
config["static_resources"]["clusters"].append(admin_cluster)
envoy = resource(render(), "Deployment", "audit-gateway")["spec"]["template"]["spec"]["containers"][0]
image = envoy["image"]
container = None
try:
with tempfile.TemporaryDirectory(prefix="firebolt-gateway-test-") as temp:
Expand All @@ -67,13 +75,15 @@ def do_POST(self):
container = subprocess.check_output([
"docker", "run", "-d",
*(["--add-host=host.docker.internal:host-gateway"] if sys.platform == "linux" else []),
"-p", "127.0.0.1::8080", "-v", f"{path}:/etc/envoy/envoy.yaml:ro",
image, "envoy", "-c", "/etc/envoy/envoy.yaml", "--concurrency", "1",
"-p", "127.0.0.1::8080", "-p", "127.0.0.1::9090", "-v", f"{path}:/etc/envoy/envoy.yaml:ro",
image, *envoy["args"], "--concurrency", "1",
], text=True).strip()
port = int(subprocess.check_output(["docker", "port", container, "8080/tcp"], text=True).strip().rsplit(":", 1)[1])

def request(method, path, body=None):
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20)
metrics_port = int(subprocess.check_output(["docker", "port", container, "9090/tcp"], text=True).strip().rsplit(":", 1)[1])

def request(method, path, body=None, target_port=None):
conn = http.client.HTTPConnection("127.0.0.1", target_port or port, timeout=20)
try:
conn.request(method, path, body=body)
response = conn.getresponse()
Expand All @@ -96,6 +106,33 @@ def request(method, path, body=None):
self.assertEqual(calls["/reset"], 1, subprocess.check_output(["docker", "logs", container], text=True, stderr=subprocess.STDOUT))
self.assertEqual(request("POST", "/drained", "mutation"), 200)
self.assertEqual(calls["/drained"], 2, "the pre-work fence should be retried")
# Run the actual chart preStop while a request is active. It must
# retain the connection, close query admission, and leave probes up.
with ThreadPoolExecutor(max_workers=1) as pool:
query = pool.submit(request, "POST", "/hold", "long query")
self.assertTrue(accepted.wait(5))
hook = subprocess.Popen(["docker", "exec", container, *envoy["lifecycle"]["preStop"]["exec"]["command"]], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
deadline = time.monotonic() + 5
while request("GET", "/healthz", target_port=metrics_port) != 503:
self.assertLess(time.monotonic(), deadline)
time.sleep(0.1)
time.sleep(5.2) # The chart's endpoint propagation floor.
self.assertIsNone(hook.poll(), "preStop exited with a request active")
self.assertFalse(query.done())
release.set()
self.assertEqual(query.result(timeout=5), 200)
out, err = hook.communicate(timeout=5)
self.assertEqual(hook.returncode, 0, (out, err))
self.assertEqual(request("GET", "/healthz", target_port=metrics_port), 503)
with self.assertRaises((OSError, http.client.HTTPException)):
request("GET", "/healthz")
finally:
release.set()
if hook.poll() is None:
hook.terminate()
hook.communicate(timeout=5)

finally:
if container:
subprocess.run(["docker", "rm", "-f", container], check=True, stdout=subprocess.DEVNULL)
Expand Down
34 changes: 34 additions & 0 deletions helm/scripts/test_render.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,40 @@ def test_retries_do_not_replay_delivered_requests(self):
self.assertEqual(policy["retriable_headers"], [{"name": "X-Firebolt-Drained", "present_match": True}])


class GatewayLifecycleTests(unittest.TestCase):
def test_discovery_and_serving_endpoints_are_separate(self):
docs = render()
discovery = resource(docs, "Service", "audit-engine-default-hl")["spec"]
serving = resource(docs, "Service", "audit-engine-default-ready")["spec"]
self.assertTrue(discovery["publishNotReadyAddresses"])
self.assertFalse(serving["publishNotReadyAddresses"])
self.assertEqual(serving["clusterIP"], "None")
self.assertEqual(discovery["selector"], serving["selector"])
config = resource(docs, "ConfigMap", "audit-gateway")["data"]["envoy.yaml"]
self.assertIn('-ready.audit.svc.cluster.local:3473', config)
self.assertIn('no_default_search_domain: true', config)

def test_gateway_drain_and_probes_match_listener_names(self):
docs = render()
pod = resource(docs, "Deployment", "audit-gateway")["spec"]["template"]["spec"]
envoy = pod["containers"][0]
self.assertEqual(envoy["livenessProbe"]["httpGet"]["port"], "metrics")
self.assertEqual(envoy["readinessProbe"]["httpGet"]["port"], "metrics")
self.assertIn("--drain-time-s", envoy["args"])
script = envoy["lifecycle"]["preStop"]["exec"]["command"][-1]
self.assertIn("http.gateway.downstream_cx_active", script)
self.assertIn("/drain_listeners?inboundonly&graceful", script)
config = envoy_config()
self.assertEqual(config["static_resources"]["listeners"][0]["traffic_direction"], "INBOUND")
self.assertNotIn("traffic_direction", config["static_resources"]["listeners"][1])

def test_engine_certificate_covers_serving_service(self):
docs = render({"tls": {"engine": {"enabled": True, "certManager": {"issuerRef": {"name": "ca"}}}}})
names = resource(docs, "Certificate", "audit-tls-engine")["spec"]["dnsNames"]
self.assertIn("audit-engine-default-ready.audit.svc.cluster.local", names)
self.assertIn("audit-engine-default-node-0-0.audit-engine-default-hl.audit.svc.cluster.local", names)


class ConfigurationTests(unittest.TestCase):
def test_existing_database_secret_reaches_both_consumers(self):
docs = render({"postgresql": {"credentials": {"existingSecret": "database-creds"}}})
Expand Down
Loading