diff --git a/docs/overview.mdx b/docs/overview.mdx index e718cb1..d759de8 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -45,3 +45,11 @@ For day-2 operational capabilities such as zero-downtime engine rollouts, autosc - [Security](./security) records the chart's pod-hardening baseline and the responsibilities it leaves to the platform team. For a value-by-value reference, see the helm-docs-generated [`helm/README.md`](https://github.com/firebolt-db/firebolt-instance-helm/blob/main/helm/README.md). + +## Upgrade behavior + +The Gateway stops advertising readiness, waits five seconds for endpoint propagation, then closes query admission and waits for active connections. `gateway.terminationGracePeriodSeconds` bounds the whole shutdown (15 seconds by default). Requests still active at that deadline can be interrupted. A custom `gateway.podTemplate.lifecycle` replaces this default drain behavior. + +Engine peer discovery uses a headless Service that publishes not-ready pods for bootstrap. The Gateway uses a separate `-ready` headless Service containing ready pods only, with active health checks in addition to Kubernetes readiness. Neither this filtering nor the Gateway's rolling update strategy coordinates Engine StatefulSet replacements. Engine upgrades and replica changes can interrupt queries. + +When upgrading an existing TLS deployment, ensure your Engine certificate includes `-engine--ready..svc.cluster.local` for every Engine before the Gateway switches to ready-only routing. Chart-managed Certificates include these SANs, but issuance and process restarts are separate operations. Use a maintenance window for the first transition, or pre-provision compatible certificate material and restart Engines before upgrading the Gateway. diff --git a/docs/usage/tls.mdx b/docs/usage/tls.mdx index bba30e3..4d1fd9f 100644 --- a/docs/usage/tls.mdx +++ b/docs/usage/tls.mdx @@ -32,7 +32,7 @@ tls: `tls.engine`'s Secret requires all three keys — `tls.crt`, `tls.key`, and `ca.crt` — even for a self-signed certificate with no separate CA (set `ca.crt` to a copy of `tls.crt` in that case). The engine's own internal health check and the gateway's upstream connection both need the issuing CA to validate the certificate independently of `tls.crt`. -The certificate's SANs must also cover every engine node's per-node hostname (`-engine--node--0.-engine--hl..svc.cluster.local`) and each engine's headless-service hostname (`-engine--hl..svc.cluster.local`) — both the engine's self-health-check and the gateway's `auto_san_validation` verify against these names. A missing SAN leaves the affected engine node permanently `NotReady` with no other symptom. `tls.engine.certManager` derives these automatically; a BYO certificate has to include them explicitly. +The certificate's SANs must also cover every engine node's per-node hostname (`-engine--node--0.-engine--hl..svc.cluster.local`) and each Engine's ready-only serving hostname (`-engine--ready..svc.cluster.local`) — both the engine's self-health-check and the gateway's `auto_san_validation` verify against these names. A missing node SAN can leave the Engine `NotReady`. A missing serving-Service SAN can instead leave Engine pods ready while Gateway queries fail certificate verification. `tls.engine.certManager` derives these automatically; a BYO certificate has to include them explicitly. ### Install @@ -50,6 +50,19 @@ helm install firebolt ./helm \ -f my-values.yaml ``` +### Upgrade from discovery-Service query routing + +Older chart versions route Gateway queries through `…-hl`. This chart routes through `…-ready`, so existing Engine certificates must include the new serving hostname before the Gateway switches. This applies to both existing Secrets and cert-manager certificates. Helm does not wait for certificate reissuance or coordinate this transition. + +For an existing TLS deployment, schedule a maintenance window and use this sequence: + +1. Keep the old Gateway and chart configuration in place. Issue an Engine certificate containing the existing node and `…-hl` SANs plus `…-ready` for every Engine. For cert-manager, add these SANs to the existing Engine `Certificate` and wait for the replacement revision and Secret. Do not run an old-chart reconciliation that removes the added SANs during this step. +2. After the replacement Secret exists, restart all Engine StatefulSets and wait for their rollouts. The init container must rebuild the certificate chain. Check the certificate served by every node against both its node hostname and the new serving hostname. +3. Upgrade to this chart using the complete deployment values. The Gateway can now use the ready-only Service with certificates that already cover it. +4. Run `helm test` and verify SQL through the Gateway before ending the maintenance window. If certificate issuance or verification fails, leave the Gateway on the previous routing configuration. + +For offline/GitOps deployments, stage the certificate resource or externally managed Secret, Engine restart, and chart upgrade as separate reconciliations. A successful `helm upgrade --wait` alone does not prove that the Gateway can query Engines using the new TLS identity. Preserve the old SANs while rollback to the previous chart remains possible. + ### cert-manager as a certificate source ```yaml diff --git a/helm/AGENTS.md b/helm/AGENTS.md index 4cb98fe..1b6e691 100644 --- a/helm/AGENTS.md +++ b/helm/AGENTS.md @@ -14,7 +14,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - Envoy gateway: `Deployment` + `Service` (configurable type) + `ConfigMap` + optional `PodDisruptionBudget` (`gateway-pdb.yaml`). Routes by `X-Firebolt-Engine` header. Optionally terminates client TLS (`tls.gateway.enabled`) and/or speaks TLS upstream to engines (`tls.engine.enabled`). - Pensieve metadata: `Deployment` + `Service` + `ConfigMap` + `Secret`. - PostgreSQL: bundled `StatefulSet` + `Service` when `postgresql.local_enabled: true`; otherwise the chart consumes external connection details / `existingSecret`. - - Engines: **one 1-replica `StatefulSet` per node per engine** (i.e. `engines[i].replicas` separate StatefulSets named `-engine--node-`), plus a shared headless Service, ClusterIP Service, and ConfigMap. Optionally authenticated (`auth.enabled`) and/or TLS-terminated on the query listener (`tls.engine.enabled`). + - Engines: **one 1-replica `StatefulSet` per node per engine** (i.e. `engines[i].replicas` separate StatefulSets named `-engine--node-`), plus a bootstrap headless Service, a ready-only serving headless Service, a ClusterIP Service, and ConfigMap. Optionally authenticated (`auth.enabled`) and/or TLS-terminated on the query listener (`tls.engine.enabled`). - Optional cert-manager `Certificate` resources (`certificates.yaml`) — one per `auth.signingKeys[]` entry, `tls.gateway`, or `tls.engine` that requests a `certManager` block instead of an `existingSecret`. Requires cert-manager and its CRDs in-cluster; absent entirely when every artifact uses `existingSecret`. - Optional `PodMonitor`s — separate ones for engines (`podmonitor.yaml`) and the gateway (`podmonitor-gateway.yaml`), each gated by `podMonitor.engines.enabled` / `podMonitor.gateway.enabled`. Both require the Prometheus Operator CRDs. - Optional `Namespace` (`createNamespace: true`). @@ -55,7 +55,7 @@ The Helm chart itself, packaged and published as `firebolt-instance` to `oci://g - `.helmignore` — excludes docs and scripts from the packaged chart. - `templates/_helpers.tpl` — naming, label, port, engine-config (incl. `instance.auth` / engine TLS `endpoints`), secret-source resolution (`fbinstance.secretSourceName` / `fbinstance.usesCertManager`), engine TLS SAN list (`fbinstance.engineTlsDnsNames`), memlock-script, and test-shell-helper templates. - `templates/certificates.yaml` — optional cert-manager `Certificate` resources for `auth.signingKeys[]` / `tls.gateway` / `tls.engine` entries that request `certManager` instead of `existingSecret`. -- `templates/gateway-*.yaml` — Envoy Deployment, Service, ConfigMap, PodDisruptionBudget. +- `templates/gateway-*.yaml` — Envoy Deployment, Service, ConfigMap, PodDisruptionBudget. `files/gateway-prestop.sh` drains the INBOUND query listener; probes use the independent metrics listener. - `templates/metadata-service-*.yaml` — Pensieve Deployment, Service, ConfigMap, Secret. - `templates/postgresql-*.yaml` — bundled PostgreSQL StatefulSet and Service (rendered only when `postgresql.local_enabled: true`). - `templates/engine-*.yaml` — per-engine StatefulSet(s), headless Service, ClusterIP Service, ConfigMap. The StatefulSet also mounts auth/TLS secrets and (when `tls.engine.enabled`) runs the `tls-chain-setup` init container. diff --git a/helm/README.md b/helm/README.md index fe96e3d..a7ba045 100644 --- a/helm/README.md +++ b/helm/README.md @@ -107,6 +107,7 @@ Firebolt Instance on Kubernetes — Envoy gateway, metadata, auth, and engines | gateway.service | object | {} | Gateway Service configuration. | | gateway.service.port | int | `80` | External service port proxied to `containerPort`. | | gateway.service.type | string | `"ClusterIP"` | Service type. One of `ClusterIP`, `LoadBalancer`, or `NodePort`. | +| gateway.terminationGracePeriodSeconds | int | `15` | Total pod shutdown budget in seconds, including a 5s endpoint-propagation wait and explicit connection draining. Requests still active at the deadline can be interrupted. | | imagePullSecrets | list | `[]` | Registry credentials. Must be a pre-created docker-registry Secret in the deployment namespace. Leave empty if nodes have ambient registry access (e.g. node IAM role). | | metadata | object | {} | Metadata service configuration. | | metadata.annotations | object | {} | Annotations on the Metadata Service Deployment, for example to opt into an externally installed Secret reload controller. These do not annotate pods. | diff --git a/helm/files/gateway-prestop.sh b/helm/files/gateway-prestop.sh new file mode 100644 index 0000000..87a0d92 --- /dev/null +++ b/helm/files/gateway-prestop.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Stop admission after endpoint propagation, then wait for active connections. +set -u +admin() { + local response status + response=$( + exec 3<>/dev/tcp/127.0.0.1/{{ .Values.gateway.adminPort }} || exit 1 + printf '%s\r\n' "$1 $2 HTTP/1.1" 'Host: localhost' 'Content-Length: 0' 'Connection: close' '' >&3 || exit 1 + cat <&3 + ) || return 1 + status=${response%%$'\n'*} + [[ "$status" == HTTP/1.*" 200 "* ]] || return 1 + printf '%s\n' "$response" +} +until admin POST /healthcheck/fail >/dev/null; do sleep 0.1; done +# Readiness is now failing. Keep accepting until the Service has stopped +# routing here; only then close admission. See the propagation note above. +sleep 5 +# graceful: in-flight requests finish and their connections close right after +# (drain-time-s is 0). The drain's completion also stops the INBOUND-marked +# listeners, so the query socket refuses new connections while the stats +# listener (not INBOUND) keeps serving probes and metrics. Verified against +# the pinned Envoy: adding skip_exit suppresses that listener stop and the +# socket keeps accepting, and without it the process still survives the +# drain - so skip_exit must stay absent. +until admin POST '/drain_listeners?inboundonly&graceful' >/dev/null; do sleep 0.1; done +while true; do + if response=$(admin GET '/stats?filter=^http[.]gateway[.]downstream_cx_active$'); then + if grep -Fxq 'http.gateway.downstream_cx_active: 0' <<<"$response"; then + exit 0 + fi + # TLS provisioning can deliberately omit the public listener entirely. + # Missing statistics alone cannot establish that there are no connections. + if ! grep -q '^http[.]gateway[.]downstream_cx_active:' <<<"$response"; then + if listeners=$(admin GET /listeners) && ! grep -q '^listener::' <<<"$listeners"; then + exit 0 + fi + fi + fi + sleep 0.1 +done diff --git a/helm/scripts/test_gateway_runtime.py b/helm/scripts/test_gateway_runtime.py index 6f5e472..654c91a 100644 --- a/helm/scripts/test_gateway_runtime.py +++ b/helm/scripts/test_gateway_runtime.py @@ -5,6 +5,7 @@ the chart's route and retry policy are exercised unmodified. """ import http.client +from concurrent.futures import ThreadPoolExecutor from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import subprocess @@ -20,7 +21,9 @@ class RetryRuntimeTests(unittest.TestCase): def test_delivered_reset_is_not_replayed_but_drained_response_is(self): - calls = {"/reset": 0, "/drained": 0} + calls = {"/reset": 0, "/drained": 0, "/hold": 0} + accepted = threading.Event() + release = threading.Event() class Backend(BaseHTTPRequestHandler): def log_message(self, *_args): @@ -33,7 +36,10 @@ def do_POST(self): # The mutation has happened. Close before response headers. self.close_connection = True return - drained = calls[self.path] == 1 + if self.path == "/hold": + accepted.set() + release.wait(15) + drained = self.path == "/drained" and calls[self.path] == 1 self.send_response(503 if drained else 200) if drained: self.send_header("X-Firebolt-Drained", "true") @@ -48,6 +54,7 @@ def do_POST(self): hcm["http_filters"] = [f for f in hcm["http_filters"] if f["name"] in { "envoy.filters.http.health_check", "envoy.filters.http.router", }] + admin_cluster = config["static_resources"]["clusters"][-1] config["static_resources"]["clusters"] = [{ "name": "dynamic_forward_proxy", "type": "LOGICAL_DNS", "connect_timeout": "1s", "dns_lookup_family": "V4_ONLY", @@ -57,8 +64,9 @@ def do_POST(self): }}}}], }]}, }] - config["static_resources"]["listeners"] = [listener] - image = resource(render(), "Deployment", "audit-gateway")["spec"]["template"]["spec"]["containers"][0]["image"] + config["static_resources"]["clusters"].append(admin_cluster) + envoy = resource(render(), "Deployment", "audit-gateway")["spec"]["template"]["spec"]["containers"][0] + image = envoy["image"] container = None try: with tempfile.TemporaryDirectory(prefix="firebolt-gateway-test-") as temp: @@ -67,13 +75,15 @@ def do_POST(self): container = subprocess.check_output([ "docker", "run", "-d", *(["--add-host=host.docker.internal:host-gateway"] if sys.platform == "linux" else []), - "-p", "127.0.0.1::8080", "-v", f"{path}:/etc/envoy/envoy.yaml:ro", - image, "envoy", "-c", "/etc/envoy/envoy.yaml", "--concurrency", "1", + "-p", "127.0.0.1::8080", "-p", "127.0.0.1::9090", "-v", f"{path}:/etc/envoy/envoy.yaml:ro", + image, *envoy["args"], "--concurrency", "1", ], text=True).strip() port = int(subprocess.check_output(["docker", "port", container, "8080/tcp"], text=True).strip().rsplit(":", 1)[1]) - def request(method, path, body=None): - conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20) + metrics_port = int(subprocess.check_output(["docker", "port", container, "9090/tcp"], text=True).strip().rsplit(":", 1)[1]) + + def request(method, path, body=None, target_port=None): + conn = http.client.HTTPConnection("127.0.0.1", target_port or port, timeout=20) try: conn.request(method, path, body=body) response = conn.getresponse() @@ -96,6 +106,33 @@ def request(method, path, body=None): self.assertEqual(calls["/reset"], 1, subprocess.check_output(["docker", "logs", container], text=True, stderr=subprocess.STDOUT)) self.assertEqual(request("POST", "/drained", "mutation"), 200) self.assertEqual(calls["/drained"], 2, "the pre-work fence should be retried") + # Run the actual chart preStop while a request is active. It must + # retain the connection, close query admission, and leave probes up. + with ThreadPoolExecutor(max_workers=1) as pool: + query = pool.submit(request, "POST", "/hold", "long query") + self.assertTrue(accepted.wait(5)) + hook = subprocess.Popen(["docker", "exec", container, *envoy["lifecycle"]["preStop"]["exec"]["command"]], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + try: + deadline = time.monotonic() + 5 + while request("GET", "/healthz", target_port=metrics_port) != 503: + self.assertLess(time.monotonic(), deadline) + time.sleep(0.1) + time.sleep(5.2) # The chart's endpoint propagation floor. + self.assertIsNone(hook.poll(), "preStop exited with a request active") + self.assertFalse(query.done()) + release.set() + self.assertEqual(query.result(timeout=5), 200) + out, err = hook.communicate(timeout=5) + self.assertEqual(hook.returncode, 0, (out, err)) + self.assertEqual(request("GET", "/healthz", target_port=metrics_port), 503) + with self.assertRaises((OSError, http.client.HTTPException)): + request("GET", "/healthz") + finally: + release.set() + if hook.poll() is None: + hook.terminate() + hook.communicate(timeout=5) + finally: if container: subprocess.run(["docker", "rm", "-f", container], check=True, stdout=subprocess.DEVNULL) diff --git a/helm/scripts/test_render.py b/helm/scripts/test_render.py index fdee2f9..646b580 100644 --- a/helm/scripts/test_render.py +++ b/helm/scripts/test_render.py @@ -44,6 +44,40 @@ def test_retries_do_not_replay_delivered_requests(self): self.assertEqual(policy["retriable_headers"], [{"name": "X-Firebolt-Drained", "present_match": True}]) +class GatewayLifecycleTests(unittest.TestCase): + def test_discovery_and_serving_endpoints_are_separate(self): + docs = render() + discovery = resource(docs, "Service", "audit-engine-default-hl")["spec"] + serving = resource(docs, "Service", "audit-engine-default-ready")["spec"] + self.assertTrue(discovery["publishNotReadyAddresses"]) + self.assertFalse(serving["publishNotReadyAddresses"]) + self.assertEqual(serving["clusterIP"], "None") + self.assertEqual(discovery["selector"], serving["selector"]) + config = resource(docs, "ConfigMap", "audit-gateway")["data"]["envoy.yaml"] + self.assertIn('-ready.audit.svc.cluster.local:3473', config) + self.assertIn('no_default_search_domain: true', config) + + def test_gateway_drain_and_probes_match_listener_names(self): + docs = render() + pod = resource(docs, "Deployment", "audit-gateway")["spec"]["template"]["spec"] + envoy = pod["containers"][0] + self.assertEqual(envoy["livenessProbe"]["httpGet"]["port"], "metrics") + self.assertEqual(envoy["readinessProbe"]["httpGet"]["port"], "metrics") + self.assertIn("--drain-time-s", envoy["args"]) + script = envoy["lifecycle"]["preStop"]["exec"]["command"][-1] + self.assertIn("http.gateway.downstream_cx_active", script) + self.assertIn("/drain_listeners?inboundonly&graceful", script) + config = envoy_config() + self.assertEqual(config["static_resources"]["listeners"][0]["traffic_direction"], "INBOUND") + self.assertNotIn("traffic_direction", config["static_resources"]["listeners"][1]) + + def test_engine_certificate_covers_serving_service(self): + docs = render({"tls": {"engine": {"enabled": True, "certManager": {"issuerRef": {"name": "ca"}}}}}) + names = resource(docs, "Certificate", "audit-tls-engine")["spec"]["dnsNames"] + self.assertIn("audit-engine-default-ready.audit.svc.cluster.local", names) + self.assertIn("audit-engine-default-node-0-0.audit-engine-default-hl.audit.svc.cluster.local", names) + + class ConfigurationTests(unittest.TestCase): def test_existing_database_secret_reaches_both_consumers(self): docs = render({"postgresql": {"credentials": {"existingSecret": "database-creds"}}}) diff --git a/helm/templates/_helpers.tpl b/helm/templates/_helpers.tpl index 09848bd..70824de 100644 --- a/helm/templates/_helpers.tpl +++ b/helm/templates/_helpers.tpl @@ -366,6 +366,7 @@ Usage: {{ include "fbinstance.engineTlsDnsNames" . }} {{- range $engine := $root.Values.engines -}} {{- $baseName := printf "%s-engine-%s" (include "fbinstance.fullname" $root) $engine.name -}} {{- $svcName := printf "%s-hl" $baseName -}} +{{- $names = append $names (printf "%s-ready.%s.svc.cluster.local" $baseName $ns) -}} {{- $names = append $names (printf "%s.%s.svc%s" $svcName $ns $root.Values.engineSpec.nodeHostSuffix) -}} {{- range $i := until (int $engine.replicas) -}} {{- $names = append $names (printf "%s-node-%d-0.%s.%s.svc%s" $baseName $i $svcName $ns $root.Values.engineSpec.nodeHostSuffix) -}} diff --git a/helm/templates/engine-ready-service.yaml b/helm/templates/engine-ready-service.yaml new file mode 100644 index 0000000..917c2b7 --- /dev/null +++ b/helm/templates/engine-ready-service.yaml @@ -0,0 +1,21 @@ +{{- range .Values.engines }} +{{- $engine := . }} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ include "fbinstance.fullname" $ }}-engine-{{ $engine.name }}-ready + namespace: {{ $.Release.Namespace }} + labels: + {{- include "fbinstance.labels" $ | nindent 4 }} + firebolt/component: engine + firebolt/engine: {{ $engine.name }} +spec: + clusterIP: None + publishNotReadyAddresses: false + selector: + {{- include "fbinstance.selectorLabels" $ | nindent 4 }} + firebolt/engine: {{ $engine.name }} + ports: + {{- include "fbinstance.enginePorts" $ | nindent 4 }} +{{- end }} diff --git a/helm/templates/gateway-configmap.yaml b/helm/templates/gateway-configmap.yaml index 95307e4..a49f6b2 100644 --- a/helm/templates/gateway-configmap.yaml +++ b/helm/templates/gateway-configmap.yaml @@ -12,6 +12,7 @@ data: static_resources: listeners: - name: listener + traffic_direction: INBOUND # per_connection_buffer_limit_bytes caps both downstream-receive # and upstream-send buffering on this listener AND is the budget # the router uses when deciding whether to BUFFER a request body @@ -106,9 +107,9 @@ data: else handle:headers():replace(":path", path .. "?advanced_mode=true") end - -- Route to the engine's headless Service so Envoy resolves + -- Route to the engine's ready-only Service so Envoy resolves -- per-pod IPs and load-balances across them itself. - handle:headers():replace(":authority", "{{ include "fbinstance.fullname" . }}-engine-" .. engine .. "-hl.{{ .Release.Namespace }}.svc.cluster.local:3473") + handle:headers():replace(":authority", "{{ include "fbinstance.fullname" . }}-engine-" .. engine .. "-ready.{{ .Release.Namespace }}.svc.cluster.local:3473") end - name: envoy.filters.http.dynamic_forward_proxy typed_config: @@ -202,6 +203,14 @@ data: route: cluster: admin_stats http_filters: + - name: envoy.filters.http.health_check + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck + pass_through_mode: false + headers: + - name: ":path" + string_match: + exact: "/healthz" - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router @@ -275,6 +284,13 @@ data: unhealthy_threshold: 1 http_health_check: path: /health/ready + # Authorities are already fully qualified; do not search infrastructure domains. + typed_dns_resolver_config: + name: envoy.network.dns_resolver.cares + typed_config: + "@type": type.googleapis.com/envoy.extensions.network.dns_resolver.cares.v3.CaresDnsResolverConfig + dns_resolver_options: + no_default_search_domain: true cluster_type: name: envoy.clusters.dynamic_forward_proxy typed_config: diff --git a/helm/templates/gateway-deployment.yaml b/helm/templates/gateway-deployment.yaml index 36cf94f..37159f9 100644 --- a/helm/templates/gateway-deployment.yaml +++ b/helm/templates/gateway-deployment.yaml @@ -67,8 +67,8 @@ spec: # token either; webhook-injected projected tokens (IRSA, Pod Identity) are # separate volumes and unaffected. automountServiceAccountToken: false - # 15s = 8s preStop /healthcheck/fail propagation + ~7s Envoy drain budget. - terminationGracePeriodSeconds: 15 + # Includes the 5s endpoint propagation floor and bounded connection drain. + terminationGracePeriodSeconds: {{ .Values.gateway.terminationGracePeriodSeconds }} {{- with .Values.gateway.podTemplate.serviceAccountName }} serviceAccountName: {{ . }} {{- end }} @@ -111,6 +111,10 @@ spec: - envoy - -c - /etc/envoy/envoy.yaml + - --drain-time-s + - "0" + - --drain-strategy + - immediate ports: - name: http containerPort: {{ .Values.gateway.containerPort }} @@ -124,44 +128,25 @@ spec: {{- else }} preStop: exec: - # Flip Envoy's health filter to 503 via /dev/tcp (no curl in the - # image), then sleep so Cilium drops the pod from Service - # endpoints before SIGTERM. New SYNs stop landing here while - # in-flight streams keep draining. command: - bash - -c - | - exec 3<>/dev/tcp/127.0.0.1/{{ .Values.gateway.adminPort }} - printf 'POST /healthcheck/fail HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\nConnection: close\r\n\r\n' >&3 - cat <&3 >/dev/null - exec 3<&- 3>&- - sleep 8 + {{- tpl (.Files.Get "files/gateway-prestop.sh") . | nindent 20 }} {{- end }} + # Keep probes reachable when the query listener is draining or uses TLS. livenessProbe: httpGet: path: /healthz - port: http - {{- if .Values.tls.gateway.enabled }} - # kubelet HTTPS probes don't verify the server certificate - # chain, so a self-signed/internal-CA cert needs no extra - # trust configuration here. - scheme: HTTPS - {{- end }} + port: metrics initialDelaySeconds: 1 periodSeconds: 15 timeoutSeconds: 3 readinessProbe: httpGet: path: /healthz - port: http - {{- if .Values.tls.gateway.enabled }} - scheme: HTTPS - {{- end }} + port: metrics initialDelaySeconds: 1 - # ~4s worst-case detection (period × failureThreshold) fits inside - # the 8s preStop window while still requiring two consecutive bad - # probes so a single hiccup does not flap the pod out of Service. periodSeconds: 2 timeoutSeconds: 2 failureThreshold: 2 diff --git a/helm/values.schema.json b/helm/values.schema.json index d0c5a98..a698f10 100644 --- a/helm/values.schema.json +++ b/helm/values.schema.json @@ -179,6 +179,10 @@ ] } } + }, + "terminationGracePeriodSeconds": { + "type": "integer", + "minimum": 10 } } }, diff --git a/helm/values.yaml b/helm/values.yaml index 0c0dfb7..bfc37ef 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -331,6 +331,8 @@ postgresql: # the upstream to `{engine}-service:3473` via dynamic forward proxy. # @default -- {} gateway: + # -- Total pod shutdown budget in seconds, including a 5s endpoint-propagation wait and explicit connection draining. Requests still active at the deadline can be interrupted. + terminationGracePeriodSeconds: 15 # -- Set to true to deploy the Envoy gateway proxy. enabled: true # -- Number of gateway replicas.