From eb083264da57b325d65a1c039b38244692a60f2d Mon Sep 17 00:00:00 2001 From: Alberto Arroyo Raygada Date: Mon, 3 Aug 2026 22:22:55 -0500 Subject: [PATCH] feat(infra): a stable cross-cluster address for the Core and its three surfaces (#417) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(infra): give the Core a stable address other clusters can hold Core and Tracker are separate products in separate repositories that talk over HTTP, so validating them in ONE cluster would hand them in-cluster DNS and hide the defects that only appear across a network boundary. Two clusters is the honest topology and already the de-facto one — the Tracker's manifest deploys into its own `evolith-tracker` cluster. What was missing was the link. Everything was reached with `kubectl port-forward`, which is right inside one cluster and wrong between two: it is a host process nothing in another cluster can point at, and it dies on every rollout of the target — so redeploying the Core breaks the Tracker and the symptom looks like a Tracker defect. There are TWO addresses now, and they are not interchangeable: FROM ANOTHER CLUSTER http://-control-plane:30080 FROM THIS MACHINE http://localhost:30080 The cross-cluster one does not use the host mapping at all. Every kind cluster joins the same `kind` Docker network, so the consumer's pod reaches the node container by name and hits the NodePort directly. The host mapping serves the other consumer — browser, curl, CLI — and is bound to 127.0.0.1, which is also precisely why it cannot serve the first case. `host.docker.internal` was the first thing written here and it is WRONG: it fails with `Could not resolve host` inside a kind pod, which resolves through CoreDNS in the node and never sees the Docker Desktop entry. The comments say so, because the wrong URL is the plausible one. Measured on a throwaway pair of clusters with the real core-api image, then torn down: a pod in a second cluster got `{"status":"OK"}` by name and by IP, the host port answered, and both survived `rollout restart`. `local-test.sh url` proves BOTH paths on demand rather than asserting them — they fail independently, so one check would cover for the other. It picks a RUNNING peer: the first version picked a stopped cluster, reported the network as broken, and the real cause was a node container dead for two days. The port mapping is fixed at cluster creation and cannot be added later, so `kind_create` says when the existing cluster predates the config instead of leaving a port that was never bound to be discovered by a refused connection. Co-Authored-By: Claude Opus 5 * feat(infra): extend the stable address to all three surfaces, and fix the probe three times "Evolith Core and its interfaces" is REST, MCP and the agent runtime. The first version gave a stable address to core-api alone, which would have let a consumer reach one surface of three while the docs implied parity — so mcp (30081) and agent-runtime (30082) now carry pinned node ports too, and the kind config maps all three. The probe that proves it needed fixing three times, each a different way of reporting a defect that was not there: - it picked a STOPPED peer cluster, failed, and blamed the Docker network; the node container had been dead for two days; - it matched `"status":"OK"` literally, so mcp and agent-runtime — which answer with a bare `"status":"ok"` rather than the ADR-0073 envelope core-api uses — were reported unreachable while serving fine; - it used `kubectl run -i --rm`, which attaches AFTER creating the pod and loses the stdout of a container that already exited. Two consecutive runs disagreed about which surface was reachable. All three are pinned in the comments, because each produced a confident red that pointed at the wrong layer. Measured, not asserted: three clean clusters' worth of runs with the real images. Three consecutive `url` invocations agreed 3/3 on both paths, and all six checks held after `rollout restart` of all three deployments. Test clusters torn down; `evolith-cluster` untouched (it has been stopped for 47h, exit 137). Worth a look separately: the three surfaces disagree on the /health shape. core-api returns the ADR-0073 envelope, the other two a bare object. Not changed here — that is a contract question, not an infra one. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- .../templates/service.yaml | 6 + .../evolith-agent-runtime/values-local.yaml | 17 ++- .../helm/evolith-agent-runtime/values.yaml | 6 + .../evolith-core-api/templates/service.yaml | 7 + .../helm/evolith-core-api/values-local.yaml | 18 ++- .../infra/helm/evolith-core-api/values.yaml | 6 + .../helm/evolith-mcp/templates/service.yaml | 6 + .../infra/helm/evolith-mcp/values-local.yaml | 19 ++- product/infra/helm/evolith-mcp/values.yaml | 6 + product/infra/helm/local-test.sh | 135 +++++++++++++++++- product/infra/kind/core-cluster.yaml | 86 +++++++++++ 11 files changed, 302 insertions(+), 10 deletions(-) create mode 100644 product/infra/kind/core-cluster.yaml diff --git a/product/infra/helm/evolith-agent-runtime/templates/service.yaml b/product/infra/helm/evolith-agent-runtime/templates/service.yaml index 5addf4daa..95252975d 100644 --- a/product/infra/helm/evolith-agent-runtime/templates/service.yaml +++ b/product/infra/helm/evolith-agent-runtime/templates/service.yaml @@ -11,5 +11,11 @@ spec: targetPort: http protocol: TCP name: http + {{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }} + # PINNED. An unpinned NodePort is allocated at random from 30000-32767 and + # changes on reinstall, so anything configured against it breaks — which is + # the failure a stable cross-cluster address exists to prevent. + nodePort: {{ .Values.service.nodePort }} + {{- end }} selector: app: {{ include "evolith-agent-runtime.name" . }} diff --git a/product/infra/helm/evolith-agent-runtime/values-local.yaml b/product/infra/helm/evolith-agent-runtime/values-local.yaml index 6d0c6959e..d1dfc8983 100644 --- a/product/infra/helm/evolith-agent-runtime/values-local.yaml +++ b/product/infra/helm/evolith-agent-runtime/values-local.yaml @@ -1,5 +1,14 @@ -# Local Kubernetes (Docker Desktop) overrides for evolith-agent-runtime. -# Reach the service with: kubectl -n evolith-local port-forward svc/-evolith-agent-runtime 8082:80 +# Local Kubernetes (Docker Desktop / kind) overrides for evolith-agent-runtime. +# +# Two ways to reach it, and they are not interchangeable: +# +# FROM ANOTHER CLUSTER (the Tracker) — http://-control-plane:30082 +# FROM THIS MACHINE, ad hoc — kubectl -n evolith-local port-forward svc/-evolith-agent-runtime 8082:80 +# or http://localhost:30082 on a kind cluster built with core-cluster.yaml +# +# The cross-cluster one is why `service.type` is NodePort here: a port-forward is +# a host process that dies on every rollout, so it cannot be what a separate +# cluster is configured against. image: repository: evolith-agent-runtime tag: local2 @@ -26,3 +35,7 @@ podDisruptionBudget: networkPolicy: enabled: false + +service: + type: NodePort + nodePort: 30082 diff --git a/product/infra/helm/evolith-agent-runtime/values.yaml b/product/infra/helm/evolith-agent-runtime/values.yaml index 620b09ddd..b907a6a65 100644 --- a/product/infra/helm/evolith-agent-runtime/values.yaml +++ b/product/infra/helm/evolith-agent-runtime/values.yaml @@ -18,6 +18,12 @@ service: type: ClusterIP port: 80 targetPort: 3000 + # Only read when `type: NodePort`. Empty means "let Kubernetes allocate one", + # which is fine for a throwaway and wrong for anything another cluster is + # configured against — see values-local.yaml and kind/core-cluster.yaml. + # ClusterIP stays the default: a node port is a LOCAL cross-cluster + # affordance, never a production one. + nodePort: "" ingressRoute: enabled: true diff --git a/product/infra/helm/evolith-core-api/templates/service.yaml b/product/infra/helm/evolith-core-api/templates/service.yaml index dbd58b440..a146fb2b6 100644 --- a/product/infra/helm/evolith-core-api/templates/service.yaml +++ b/product/infra/helm/evolith-core-api/templates/service.yaml @@ -11,5 +11,12 @@ spec: targetPort: http protocol: TCP name: http + {{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }} + # PINNED on purpose. An unpinned NodePort is allocated at random from + # 30000-32767, so it changes on reinstall — and a cross-cluster consumer + # configured against it would break every time the chart is reinstalled, + # which is the failure `extraPortMappings` exists to prevent. + nodePort: {{ .Values.service.nodePort }} + {{- end }} selector: app: {{ include "evolith-core-api.name" . }} diff --git a/product/infra/helm/evolith-core-api/values-local.yaml b/product/infra/helm/evolith-core-api/values-local.yaml index 0b436abb9..a5dde4e57 100644 --- a/product/infra/helm/evolith-core-api/values-local.yaml +++ b/product/infra/helm/evolith-core-api/values-local.yaml @@ -1,6 +1,16 @@ -# Local Kubernetes (Docker Desktop) overrides for evolith-core-api. +# Local Kubernetes (Docker Desktop / kind) overrides for evolith-core-api. # Use the locally-built image, skip Traefik/IngressRoute and cluster-only addons. -# Reach the service with: kubectl -n evolith-local port-forward svc/-evolith-core-api 8080:80 +# +# Two ways to reach it, and they are not interchangeable: +# +# FROM THIS MACHINE, ad hoc — kubectl -n evolith-local port-forward svc/-evolith-core-api 8080:80 +# FROM ANOTHER CLUSTER (the Tracker) — http://host.docker.internal:30080 +# +# The second is why `service.type` is NodePort here. A port-forward is a process +# on the host that dies on every rollout of the Core, so it cannot be what a +# separate cluster is configured against; the node port is fixed at cluster +# creation and survives. Requires the cluster to have been created with +# `product/infra/kind/core-cluster.yaml` — the mapping cannot be added later. image: repository: evolith-core-api tag: local2 @@ -8,6 +18,10 @@ image: replicaCount: 1 +service: + type: NodePort + nodePort: 30080 + ingressRoute: enabled: false diff --git a/product/infra/helm/evolith-core-api/values.yaml b/product/infra/helm/evolith-core-api/values.yaml index 7b7c69ba2..6971d30bf 100644 --- a/product/infra/helm/evolith-core-api/values.yaml +++ b/product/infra/helm/evolith-core-api/values.yaml @@ -22,6 +22,12 @@ service: type: ClusterIP port: 80 targetPort: 3000 + # Only read when `type: NodePort`. Empty means "let Kubernetes allocate one", + # which is fine for a throwaway and wrong for anything another cluster is + # configured against — see values-local.yaml and kind/core-cluster.yaml. + # ClusterIP stays the default: exposing the Core on a node port is a LOCAL + # cross-cluster affordance, never a production one. + nodePort: "" ingressRoute: enabled: true diff --git a/product/infra/helm/evolith-mcp/templates/service.yaml b/product/infra/helm/evolith-mcp/templates/service.yaml index acfcfb111..8d1d8633c 100644 --- a/product/infra/helm/evolith-mcp/templates/service.yaml +++ b/product/infra/helm/evolith-mcp/templates/service.yaml @@ -11,6 +11,12 @@ spec: targetPort: http protocol: TCP name: http + {{- if and (eq .Values.service.type "NodePort") .Values.service.nodePort }} + # PINNED. An unpinned NodePort is allocated at random from 30000-32767 and + # changes on reinstall, so anything configured against it breaks — which is + # the failure a stable cross-cluster address exists to prevent. + nodePort: {{ .Values.service.nodePort }} + {{- end }} {{- if .Values.opa.enabled }} # GT-551: expose the OPA sidecar's HTTP port so Prometheus can reach its # /metrics endpoint. Without this the sidecar is pod-local only and diff --git a/product/infra/helm/evolith-mcp/values-local.yaml b/product/infra/helm/evolith-mcp/values-local.yaml index 9cab0c99b..e2e46f9ff 100644 --- a/product/infra/helm/evolith-mcp/values-local.yaml +++ b/product/infra/helm/evolith-mcp/values-local.yaml @@ -1,7 +1,16 @@ -# Local Kubernetes (Docker Desktop) overrides for evolith-mcp. +# Local Kubernetes (Docker Desktop / kind) overrides for evolith-mcp. # The OPA sidecar is DISABLED locally (it fetches signed bundles from an in-cluster -# MinIO that does not exist on Docker Desktop). Reach the service with: -# kubectl -n evolith-local port-forward svc/-evolith-mcp 8081:80 +# MinIO that does not exist on Docker Desktop). +# +# Two ways to reach it, and they are not interchangeable: +# +# FROM ANOTHER CLUSTER (the Tracker) — http://-control-plane:30081 +# FROM THIS MACHINE, ad hoc — kubectl -n evolith-local port-forward svc/-evolith-mcp 8081:80 +# or http://localhost:30081 on a kind cluster built with core-cluster.yaml +# +# The cross-cluster one is why `service.type` is NodePort here: a port-forward is +# a host process that dies on every rollout, so it cannot be what a separate +# cluster is configured against. image: repository: evolith-mcp tag: local2 @@ -23,3 +32,7 @@ podDisruptionBudget: networkPolicy: enabled: false + +service: + type: NodePort + nodePort: 30081 diff --git a/product/infra/helm/evolith-mcp/values.yaml b/product/infra/helm/evolith-mcp/values.yaml index 159c41001..f9d6749a1 100644 --- a/product/infra/helm/evolith-mcp/values.yaml +++ b/product/infra/helm/evolith-mcp/values.yaml @@ -19,6 +19,12 @@ service: port: 80 # The real mcp-server listens on 3000 (packages/mcp-server/Dockerfile). targetPort: 3000 + # Only read when `type: NodePort`. Empty means "let Kubernetes allocate one", + # which is fine for a throwaway and wrong for anything another cluster is + # configured against — see values-local.yaml and kind/core-cluster.yaml. + # ClusterIP stays the default: a node port is a LOCAL cross-cluster + # affordance, never a production one. + nodePort: "" # App container environment. The mcp-server requires EVOLITH_API_KEY in production # (injected from a secret below); set EVOLITH_MCP_ALLOW_NO_AUTH=true to relax. diff --git a/product/infra/helm/local-test.sh b/product/infra/helm/local-test.sh index dba722601..ce6e41509 100644 --- a/product/infra/helm/local-test.sh +++ b/product/infra/helm/local-test.sh @@ -8,6 +8,7 @@ # bash product/infra/helm/local-test.sh kind-apps-up # kind + build + load + install apps # bash product/infra/helm/local-test.sh kind-up # kind + build + load + infra + apps # bash product/infra/helm/local-test.sh smoke # port-forward + curl +# bash product/infra/helm/local-test.sh url # cross-cluster URL, and proof it answers # bash product/infra/helm/local-test.sh kind-down # uninstall + delete kind cluster # # Env: @@ -27,16 +28,143 @@ CLUSTER="${KIND_CLUSTER:-evolith}" # dedicated kind cluster name (kind- IMAGES=(evolith-core-api:"$IMAGE_TAG" evolith-mcp:"$IMAGE_TAG" evolith-agent-runtime:"$IMAGE_TAG") +KIND_CONFIG="$ROOT/product/infra/kind/core-cluster.yaml" +CORE_NODE_PORT=30080 +# One entry per interface: "