From fd2def245ce13c2344655c01bcc6ad5d6677b0a8 Mon Sep 17 00:00:00 2001 From: Pedro Gomes Date: Thu, 23 Jul 2026 09:08:53 +0100 Subject: [PATCH] feat(gateway): make the gateway private, fronted by an ingress (ADR-0045) A gateway is no longer its own public edge. It now requires an `ingress:` FK (same scope, singleton) that fronts it: the ingress terminates TLS, enforces the edge security tier (CrowdSec/rate-limit), and reverse-proxies the gateway FQDN to a private gateway over the network, preserving the client IP. Rendered as two servers: a TERMINATION server on the ingress host (ACME + rate limit -> reverse-proxy to the gateway) and a plain-HTTP ROUTING server on the gateway host (nginx.GatewayHTTPPort; set_real_ip_from the ingress + real_ip_header X-Forwarded-For to recover the client, then the derived routes -> mesh egress). Co-located = loopback; split = private IPs resolved in the provider apply. Derivation changes: crowdsecEdgeHosts is ingress-hosts-only (a gateway is never a public edge); the gateway FQDN A record + firewall 443/80 live on the ingress host; the split gateway host opens only GatewayHTTPPort privately; listed-service health relocates to the ingress host. nginx.Render gained a gatewayRoutes slice; IngressProvider.Realize gained gatewayRoutes + gatewayIPs. No wardnet-cloud change: the routing server's real_ip recovery preserves the leftmost-XFF-from-loopback contract client_ip() expects. Breaking change (required ingress: FK) -> targets inforge v6.2.0. ADR-0045; AGENTS.md + website docs updated. go build/test -race/lint clean. --- AGENTS.md | 26 ++- docs/adr/0045-gateway-behind-ingress.md | 99 +++++++++ internal/loader/loader.go | 1 + internal/nginx/config.go | 153 +++++++++----- internal/nginx/config_test.go | 167 ++++++++------- internal/nginx/paths.go | 12 ++ internal/types/types.go | 80 +++++--- internal/validate/validate.go | 81 ++++++-- internal/validate/validate_test.go | 60 +++++- program/crowdsec_test.go | 24 ++- program/gateway_paths_test.go | 103 ++++++---- program/program.go | 202 +++++++++++++------ program/ratelimit_test.go | 2 +- providers/hetzner/deploykey_test.go | 2 +- providers/hetzner/tls.go | 73 ++++++- providers/hetzner/tls_test.go | 66 ++++++ schemas/gateway.json | 2 +- website/docs/configuration/variables-yaml.md | 5 +- website/docs/resources/gateway.md | 154 ++++++++------ website/docs/resources/ingress.md | 7 +- 20 files changed, 962 insertions(+), 357 deletions(-) create mode 100644 docs/adr/0045-gateway-behind-ingress.md create mode 100644 providers/hetzner/tls_test.go diff --git a/AGENTS.md b/AGENTS.md index 9d6936a..f472f08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -634,9 +634,29 @@ feed the gateway's derived routing table (rule `gateway-routes-are-derived-from- ## North-south gateway realization (ADR-0032 daemon edge, ADR-0034 derived routes) -The `gateway` resource (authored: `host` + **`pki`** + `subdomain` + **`services:[names]`** + -optional `health_probes_port`/`health_probe_paths`; scope singleton; validated in `checkGateway` -incl. listed-service `allowed_services` + **same-`pki:`** match) is realized in two halves: +The `gateway` resource (authored: `host` + **`ingress`** + **`pki`** + `subdomain` + +**`services:[names]`** + optional `health_probes_port`/`health_probe_paths`; scope singleton; +validated in `checkGateway` incl. listed-service `allowed_services` + **same-`pki:`** match) is +realized in two halves: + +- **A gateway is PRIVATE, fronted by an ingress (ADR-0045).** The `ingress:` FK is **required** + (same-scope singleton ingress; gateway host and ingress host must share a network). A gateway is + **never a public edge** — it opens no public port and holds no ACME cert. Its FQDN renders as TWO + servers: a **termination server** on the fronting ingress host (ACME TLS + rate limit → + reverse-proxy the whole FQDN to the gateway, appending the real client to XFF) and a **routing + server** on the gateway host (plain HTTP on `nginx.GatewayHTTPPort`, `set_real_ip_from ` + + `real_ip_header X-Forwarded-For` to recover the client, then the derived route locations → mesh + egress). `program.gatewayEdgeByHost` derives both halves (terminations by ingress host, routings by + gateway host) + the cross-host peer-IP map; co-located = loopback, split = private IPs resolved in + the provider apply. **Consequences:** `crowdsecEdgeHosts` derives from **ingress hosts only** (a + gateway is never an edge); the gateway FQDN A record and firewall `443`/`80` live on the **ingress + host**; the split gateway host opens only `GatewayHTTPPort` privately; listed-service health + relocates to the ingress host. **No wardnet-cloud change** — the routing server's `real_ip` + recovery preserves the leftmost-XFF contract `client_ip()` expects. The `nginx.Render` signature + gained a `gatewayRoutes` slice; `IngressProvider.Realize` gained `gatewayRoutes` + `gatewayIPs`. + The bullets below describe the derived routing table and mesh-client half, which are unchanged; + where they say the gateway "terminates" or "opens public 443", read that as the **termination + server on the ingress host** per ADR-0045. - **Edge half (north-south nginx).** The routing table is **derived, never authored** (ADR-0034): `toGatewayNginxRoutes` emits one `IngressGatewayRoute{Pattern, Service}` per (listed service, diff --git a/docs/adr/0045-gateway-behind-ingress.md b/docs/adr/0045-gateway-behind-ingress.md new file mode 100644 index 0000000..0431cf5 --- /dev/null +++ b/docs/adr/0045-gateway-behind-ingress.md @@ -0,0 +1,99 @@ +--- +status: accepted +date: 2026-07-23 +issue: "TBD" +--- + +# The gateway is private, fronted by an ingress + +The north-south daemon `gateway` (ADR-0032/0034) was realized as its **own public +edge**: it opened public `443`/`80` on its host, held its own ACME cert and DNS A +record, and terminated client TLS directly. It was a *sibling* of the ingress, not +fronted by it. Two consequences followed: the edge security tier (ADR-0043 CrowdSec + +rate-limit) had to treat gateway hosts as public edges, and there was no way to run a +gateway that is *not* internet-exposed. + +We want the opposite invariant: **an API gateway is never exposed on the public +network.** All public traffic enters through the **ingress**, which terminates TLS and +enforces the security tier, then reaches a **private** gateway. + +## Decision + +A gateway is **always fronted by an ingress**, declared with a **required** `ingress:` +FK (same scope, the scope's singleton ingress — mirroring `app.ingress`). This is a +breaking change to the gateway schema: every gateway manifest must name its ingress. + +The gateway renders as **two halves** (Model A — the ingress terminates, it does not +L4-passthrough; only termination lets CrowdSec/rate-limit see the request): + +- **Termination server** — on the **fronting ingress host**. ACME-managed TLS on the + gateway FQDN, the rate limit applied here, reverse-proxying the *whole* FQDN to the + gateway's routing server. It **blind-proxies**: it never inspects paths or touches + `X-Mesh-Target` — routing stays gateway business. It participates in ssl_preread + mixed ports exactly like an app (same SNI-on-443 behavior). +- **Routing server** — on the **gateway's own host**, plain HTTP on + `nginx.GatewayHTTPPort` (never public; reached only through the ingress). It holds + the derived `public_paths` routing table and hands each route to the local mesh + egress. The gateway's mesh membership (identity `/gateway`, fixed + `GatewayEgressPort`) is **unchanged** — it still runs on the gateway host. + +Co-located (gateway host == ingress host) is the trivial case: both servers render on +one nginx and the hop is loopback. Split (different hosts, same network — enforced by +`checkGateway`) resolves the peer private IPs at deploy and the firewall opens +`GatewayHTTPPort` to the network CIDR. + +Derivation changes that fall out: + +- **Security tier** — `crowdsecEdgeHosts` derives from **ingress hosts only**; a + gateway is never a public edge. "Security at the ingress" becomes structural, not a + config toggle. The rate limit is stamped on the termination server. +- **DNS** — the gateway FQDN A record points at the **ingress host** (where the cert + lives), not the gateway host. +- **Firewall** — the ingress host opens public `443`/`80` for the gateway; the gateway + host opens `GatewayHTTPPort` privately (split only). No public port on the gateway. +- **Health** — a listed service's health listener + DNS record relocate to the + **ingress host** (like a service-with-an-ingress); the gateway's own `/livez` is + answered by the routing server, reached through the ingress (full-path liveness). + +## Client-IP preservation (the load-bearing detail) + +wardnet-cloud resolves the client IP from the **leftmost `X-Forwarded-For`**, trusted +**only when the socket peer is loopback** (the mesh proxy) — `client_ip()` in +`crates/common/src/proxy_protocol.rs`, "invariant #8". The old single-server gateway +hard-**set** `X-Forwarded-For = $remote_addr` (anti-spoof: it was the internet-facing +first hop). With an ingress in front, `$remote_addr` at the gateway becomes the ingress, +which would leak the ingress IP to the service. + +The fix is entirely inforge-side, **no wardnet-cloud change**: + +- The termination server **appends** the real client (`$proxy_add_x_forwarded_for`) — + it is now the trusted internet-facing hop. +- The routing server recovers the real client with + `set_real_ip_from ; real_ip_header X-Forwarded-For; real_ip_recursive on;` + — trusting **only** the ingress, so a client-forged left entry is stripped + (anti-spoof preserved) and `$remote_addr` becomes the true client. +- The routing server then **sets** `X-Forwarded-For = $remote_addr` toward the mesh; + the callee mesh proxy appends, so the service reads the client as the leftmost entry + — exactly the contract `client_ip()` already expects. + +## Alternatives considered + +- **Optional FK (gateway may be standalone-public).** Rejected: the security model is + "never public," and an optional flag leaves a foot-gun (a public-but-undefended + gateway) and a silent-break path. +- **L4 `ssl_preread` passthrough to a still-terminating gateway (PROXY protocol).** + Rejected: the ingress could not inspect L7, so CrowdSec/rate-limit could not run on + gateway traffic — defeating the purpose. +- **Merge the gateway routing into the ingress nginx.** Rejected: it would move the + gateway's mesh membership/egress onto the ingress host and blur the + north-south/east-west separation; keeping two servers preserves both. + +## Rollout + +Phase 1 lands the model with prd gateways **co-located** with their ingresses (unchanged +topology, loopback hop) — full security model + client-IP correctness at minimal risk. +The actual host-split is later a one-line `host:` change; the cross-host machinery +(real_ip, firewall, DNS, peer-IP resolution) is already in place. + +Shipped in inforge v6.2.0 (breaking: the required `ingress:` FK). Consumers add the FK +in lockstep with picking up the release. diff --git a/internal/loader/loader.go b/internal/loader/loader.go index 11fb35a..9151e1a 100644 --- a/internal/loader/loader.go +++ b/internal/loader/loader.go @@ -449,6 +449,7 @@ func NormalizeGateway(g *types.GatewaySpec) { g.Name = strings.TrimSpace(g.Name) g.Container = strings.TrimSpace(g.Container) g.Host = strings.TrimSpace(g.Host) + g.Ingress = strings.TrimSpace(g.Ingress) g.Pki = strings.TrimSpace(g.Pki) g.Subdomain = strings.TrimSpace(g.Subdomain) trimAll(g.Services) diff --git a/internal/nginx/config.go b/internal/nginx/config.go index 0ca6e9d..637cf9f 100644 --- a/internal/nginx/config.go +++ b/internal/nginx/config.go @@ -52,7 +52,12 @@ type mixedPort struct { // strictly by server_name (the service FQDN) and reverse-proxied to the backend // health port. http{} is emitted when the host terminates TLS for any route, serves // any app, or has any health endpoint; stream{} when it has any forward route. -func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway) (string, error) { +// gateways are the gateway TERMINATION servers this host fronts as an ingress +// (TLS + security → reverse-proxy to the private gateway, ADR-0045); gatewayRoutes +// are the gateway ROUTING servers this host runs as a gateway host (plain-HTTP, +// real_ip recovery → mesh egress). When a gateway is co-located with its ingress, +// the same host receives both for the same gateway. +func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, gatewayRoutes []types.IngressGateway) (string, error) { var terminate, forward []types.IngressRoute for _, r := range routes { // Backend is the resolved upstream address the caller must fill — "127.0.0.1" @@ -114,17 +119,31 @@ func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types } sort.Slice(sortedHealth, func(i, j int) bool { return sortedHealth[i].FQDN < sortedHealth[j].FQDN }) - // Gateways render in a stable order. Each must carry its resolved FQDN — the - // server_name and ACME cert SNI; empty means the program wiring failed. Every - // route pattern is compiled up front (validation guarantees parseability; a - // bad glob reaching this far fails the render loud rather than emitting a - // broken location). + // Gateway TERMINATION servers (on the ingress host) render in a stable order. + // Each must carry its resolved FQDN (server_name + ACME SNI) and the resolved + // backend the ingress reverse-proxies to (the private gateway's HTTP listener) — + // empty means the program wiring failed; fail loud rather than proxy to nowhere. sortedGateways := append([]types.IngressGateway(nil), gateways...) - gatewayRegexOf := map[string]string{} for _, g := range sortedGateways { if g.FQDN == "" { return "", fmt.Errorf("nginx: gateway %q has no FQDN", g.Name) } + if g.Backend == "" || g.HTTPPort <= 0 { + return "", fmt.Errorf("nginx: gateway %q termination has no resolved backend (unresolved private gateway address)", g.Name) + } + } + sort.Slice(sortedGateways, func(i, j int) bool { return sortedGateways[i].FQDN < sortedGateways[j].FQDN }) + + // Gateway ROUTING servers (on the gateway host) render in a stable order. Each + // must carry its bind address, HTTP port, and the ingress source it trusts for + // real-IP recovery. Every route pattern is compiled up front (validation + // guarantees parseability; a bad glob reaching this far fails the render loud). + sortedGatewayRoutes := append([]types.IngressGateway(nil), gatewayRoutes...) + gatewayRegexOf := map[string]string{} + for _, g := range sortedGatewayRoutes { + if g.ListenAddr == "" || g.HTTPPort <= 0 || g.RealIPFrom == "" { + return "", fmt.Errorf("nginx: gateway %q routing server has unresolved wiring (listen/port/real_ip)", g.Name) + } for _, rt := range g.Routes { p, err := pathglob.Parse(rt.Pattern) if err != nil { @@ -133,7 +152,7 @@ func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types gatewayRegexOf[rt.Pattern] = p.Regex() } } - sort.Slice(sortedGateways, func(i, j int) bool { return sortedGateways[i].FQDN < sortedGateways[j].FQDN }) + sort.Slice(sortedGatewayRoutes, func(i, j int) bool { return sortedGatewayRoutes[i].FQDN < sortedGatewayRoutes[j].FQDN }) // Classify ports: a forward whose listen also carries a tls-termination route // or (on 443) an app is "mixed" and needs ssl_preread; the rest are forward-only. @@ -232,8 +251,8 @@ func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types block("events", nil, dir("worker_connections", "1024")), ) - if len(terminate) > 0 || len(sortedApps) > 0 || len(sortedHealth) > 0 || len(sortedGateways) > 0 { - top = append(top, httpBlock(terminate, sortedApps, sortedHealth, healthPort, sortedGateways, gatewayRegexOf, listenDir, len(mixedPorts) > 0)) + if len(terminate) > 0 || len(sortedApps) > 0 || len(sortedHealth) > 0 || len(sortedGateways) > 0 || len(sortedGatewayRoutes) > 0 { + top = append(top, httpBlock(terminate, sortedApps, sortedHealth, healthPort, sortedGateways, sortedGatewayRoutes, gatewayRegexOf, listenDir, len(mixedPorts) > 0)) } if len(forwardOnly) > 0 || len(mixedPorts) > 0 { top = append(top, streamBlock(forwardOnly, mixedPorts)) @@ -256,7 +275,7 @@ func Render(routes []types.IngressRoute, apps []types.IngressApp, health []types // ACME-challenge/redirect server (only when something terminates TLS). // listenDir places a terminating server on its public port or, when that port is // mixed, on its internal loopback port. -func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, gatewayRegexOf map[string]string, listenDir func(int) *crossplane.Directive, anyMixed bool) *crossplane.Directive { +func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, gatewayRoutes []types.IngressGateway, gatewayRegexOf map[string]string, listenDir func(int) *crossplane.Directive, anyMixed bool) *crossplane.Directive { terminatesTLS := len(terminate) > 0 || len(apps) > 0 || len(gateways) > 0 // Content-Type for everything served off disk. Unconditional: nginx's built-in @@ -268,10 +287,10 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ dir("include", mimeTypesPath), dir("default_type", "application/octet-stream"), } - if len(gateways) > 0 { - // WebSocket upgrade support for the gateway's mesh-bound locations: a daemon - // WS handshake crosses gateway → gateway-mesh → callee-mesh, and every hop - // must pass Upgrade through (ADR-0032 realization invariant). + if len(gateways) > 0 || len(gatewayRoutes) > 0 { + // WebSocket upgrade support for the gateway path: a daemon WS handshake crosses + // ingress-termination → gateway-routing → gateway-mesh → callee-mesh, and every + // hop must pass Upgrade through (ADR-0032/0045 realization invariant). children = append(children, block("map", []string{"$http_upgrade", "$connection_upgrade"}, dir("default", "upgrade"), @@ -318,7 +337,10 @@ func httpBlock(terminate []types.IngressRoute, apps []types.IngressApp, health [ children = append(children, appServer(a, listenDir(443))) } for _, g := range gateways { - children = append(children, gatewayServer(g, listenDir(443), gatewayRegexOf)) + children = append(children, gatewayTerminationServer(g, listenDir(443))) + } + for _, g := range gatewayRoutes { + children = append(children, gatewayRoutingServer(g, gatewayRegexOf)) } if len(health) > 0 { children = append(children, healthCatchAllServer(healthPort)) @@ -387,30 +409,64 @@ func appServer(a types.IngressApp, listen *crossplane.Directive) *crossplane.Dir ) } -// gatewayServer renders the north-south daemon gateway server (ADR-0032/0034): -// ACME-managed TLS on the gateway's single FQDN, with one regex location per -// derived route (a listed service's public path glob, compiled by pathglob) -// handing the request to the LOCAL mesh proxy's gateway egress listener. -// The location proxies plain HTTP to loopback; the mesh does the mTLS hop -// presenting the gateway's /gateway leaf, and the callee's mesh stamps -// X-Service-Identity from that cert. Invariants realized here: +// gatewayTerminationServer renders the ingress-side half of a private gateway +// (ADR-0045): ACME-managed TLS on the gateway's FQDN on the FRONTING INGRESS host, +// enforcing the edge security tier (rate limit) and reverse-proxying the whole FQDN +// to the private gateway's plain-HTTP routing server (g.Backend:g.HTTPPort). The +// routing table stays gateway business — this server never inspects paths or +// touches X-Mesh-Target; it blind-proxies. XFF is APPENDED here (the ingress is the +// trusted internet-facing hop): $proxy_add_x_forwarded_for stamps the real client +// ($remote_addr — recovered from proxy_protocol on a mixed port) as the rightmost +// entry, which the routing server then recovers via set_real_ip_from this ingress. +// listen is the pre-computed listen directive (public ssl, or a mixed-port loopback +// ssl listen accepting the PROXY protocol). WebSocket-capable (a daemon WS crosses +// four proxy hops); the daemon Authorization header is forwarded untouched. +func gatewayTerminationServer(g types.IngressGateway, listen *crossplane.Directive) *crossplane.Directive { + loc := append(rlHTTPLimitDirs(g.RateLimit), + dir("proxy_http_version", "1.1"), + dir("proxy_set_header", "Upgrade", "$http_upgrade"), + dir("proxy_set_header", "Connection", "$connection_upgrade"), + dir("proxy_set_header", "Host", "$host"), + dir("proxy_set_header", "X-Forwarded-For", "$proxy_add_x_forwarded_for"), + dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + dir("proxy_read_timeout", "3600s"), + dir("proxy_pass", fmt.Sprintf("http://%s:%d", g.Backend, g.HTTPPort)), + ) + return block("server", nil, + listen, + dir("server_name", g.FQDN), + dir("acme_certificate", acmeIssuer), + dir("ssl_certificate", "$acme_certificate"), + dir("ssl_certificate_key", "$acme_certificate_key"), + dir("ssl_certificate_cache", "max=2"), + block("location", []string{"/"}, loc...), + ) +} + +// gatewayRoutingServer renders the gateway-side half of a private gateway +// (ADR-0032/0034/0045): a plain-HTTP server on the gateway host (never public — it +// listens on g.ListenAddr:g.HTTPPort, reached only through the fronting ingress), +// with one regex location per derived route handing the request to the LOCAL mesh +// proxy's gateway egress listener. The mesh does the mTLS hop presenting the +// gateway's /gateway leaf. Invariants realized here: +// - the client IP is recovered from the ingress-stamped X-Forwarded-For, trusting +// ONLY the fronting ingress (set_real_ip_from g.RealIPFrom, real_ip_recursive) — +// so $remote_addr is the real daemon and a client-forged XFF entry is stripped; // - the target is named out-of-band in X-Mesh-Target, so the PATH IS PRESERVED // byte-for-byte — a daemon's PoP-signed path reaches the service verbatim; // - the daemon's Authorization header is forwarded untouched (the service // validates the JWT, never the gateway); -// - every location is WebSocket-capable ($connection_upgrade map, 1h read -// timeout) — a daemon WS crosses three proxy hops; -// - the real daemon IP is stamped into X-Forwarded-For at this, the TLS edge; -// - the gateway's own health probe paths are answered 200 "ok" by nginx -// itself, over the same TLS path daemons use (edge liveness); -// - a path matching no route is a JSON 404 at the edge, never proxied. +// - X-Forwarded-For toward the mesh is SET to $remote_addr (the recovered client), +// so the callee reads the real client as the leftmost XFF entry; +// - every location is WebSocket-capable; the health probe paths are answered 200 +// "ok" directly (edge liveness, reached over the real public path through the +// ingress); a path matching no route is a JSON 404, never proxied. // -// Regex locations are evaluated in emitted order for every request on this -// server, but validation guarantees the patterns are pairwise non-overlapping -// across the gateway's services (pathglob.Overlaps), so order cannot change -// which route wins. regexOf maps each route's raw pattern to its compiled, -// Render-verified regex. -func gatewayServer(g types.IngressGateway, listen *crossplane.Directive, regexOf map[string]string) *crossplane.Directive { +// Regex locations are evaluated in emitted order, but validation guarantees the +// patterns are pairwise non-overlapping across the gateway's services, so order +// cannot change which route wins. regexOf maps each route's raw pattern to its +// compiled, Render-verified regex. +func gatewayRoutingServer(g types.IngressGateway, regexOf map[string]string) *crossplane.Directive { routes := append([]types.IngressGatewayRoute(nil), g.Routes...) sort.Slice(routes, func(i, j int) bool { if routes[i].Pattern != routes[j].Pattern { @@ -420,12 +476,15 @@ func gatewayServer(g types.IngressGateway, listen *crossplane.Directive, regexOf }) children := crossplane.Directives{ - listen, + dir("listen", fmt.Sprintf("%s:%d", g.ListenAddr, g.HTTPPort)), dir("server_name", g.FQDN), - dir("acme_certificate", acmeIssuer), - dir("ssl_certificate", "$acme_certificate"), - dir("ssl_certificate_key", "$acme_certificate_key"), - dir("ssl_certificate_cache", "max=2"), + // The fronting ingress is the sole trusted source of the client IP: recover the + // real daemon address from its X-Forwarded-For. Only this ingress is trusted, so + // real_ip_recursive stops at the rightmost non-trusted entry — the real client + // the ingress appended — and a client-forged left entry is discarded (anti-spoof). + dir("set_real_ip_from", g.RealIPFrom), + dir("real_ip_header", "X-Forwarded-For"), + dir("real_ip_recursive", "on"), } probes := append([]string(nil), g.HealthProbePaths...) sort.Strings(probes) @@ -436,22 +495,20 @@ func gatewayServer(g types.IngressGateway, listen *crossplane.Directive, regexOf )) } for _, rt := range routes { - loc := append(rlHTTPLimitDirs(g.RateLimit), + children = append(children, block("location", []string{"~", regexOf[rt.Pattern]}, dir("proxy_http_version", "1.1"), dir("proxy_set_header", "Upgrade", "$http_upgrade"), dir("proxy_set_header", "Connection", "$connection_upgrade"), dir("proxy_set_header", "Host", "$host"), - // SET, never append: this is the internet-facing first hop, so a - // daemon-supplied X-Forwarded-For is untrusted input — appending - // ($proxy_add_x_forwarded_for) would forward a forged first entry - // through the mesh and defeat per-IP audit/rate-limit logic. + // SET, never append: $remote_addr is the real client (recovered above from + // the trusted ingress), so this stamps it as the sole XFF entry the mesh + // forwards — the callee reads it as the leftmost value. dir("proxy_set_header", "X-Forwarded-For", "$remote_addr"), - dir("proxy_set_header", "X-Forwarded-Proto", "$scheme"), + dir("proxy_set_header", "X-Forwarded-Proto", "$http_x_forwarded_proto"), dir("proxy_set_header", "X-Mesh-Target", rt.Service), dir("proxy_read_timeout", "3600s"), dir("proxy_pass", fmt.Sprintf("http://127.0.0.1:%d", meshpaths.GatewayEgressPort)), - ) - children = append(children, block("location", []string{"~", regexOf[rt.Pattern]}, loc...)) + )) } children = append(children, jsonNotFoundLocation()) return block("server", nil, children...) diff --git a/internal/nginx/config_test.go b/internal/nginx/config_test.go index f06172b..ccb5042 100644 --- a/internal/nginx/config_test.go +++ b/internal/nginx/config_test.go @@ -71,7 +71,7 @@ stream { } } ` - got, err := Render(routes, nil, nil, 0, nil) + got, err := Render(routes, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Equal(t, want, got) } @@ -87,7 +87,7 @@ func TestRenderRateLimit(t *testing.T) { {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", FQDNs: []string{"api.svc.prd.use1.wardnet.network"}, RateLimit: rl}, } - got, err := Render(routes, nil, nil, 0, nil) + got, err := Render(routes, nil, nil, 0, nil, nil) require.NoError(t, err) // http{} zones, declared once, with the 429 status overrides. @@ -118,7 +118,7 @@ func TestRenderRateLimitExemptsHealthAndACME(t *testing.T) { health := []types.IngressHealth{ {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 9000, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, } - got, err := Render(routes, nil, health, 81, nil) + got, err := Render(routes, nil, health, 81, nil, nil) require.NoError(t, err) // Exactly one limited location — the tls-termination route. Health + ACME are exempt. assert.Equal(t, 1, strings.Count(got, "limit_req zone=rl_edge")) @@ -132,7 +132,7 @@ func TestRenderNoRateLimit(t *testing.T) { {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", FQDNs: []string{"api.svc.prd.use1.wardnet.network"}}, } - got, err := Render(routes, nil, nil, 0, nil) + got, err := Render(routes, nil, nil, 0, nil, nil) require.NoError(t, err) assert.NotContains(t, got, "limit_req") assert.NotContains(t, got, "limit_conn") @@ -222,7 +222,7 @@ stream { } } ` - got, err := Render(routes, nil, nil, 0, nil) + got, err := Render(routes, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Equal(t, want, got) } @@ -237,7 +237,7 @@ func TestRenderMixedWithApp(t *testing.T) { apps := []types.IngressApp{ {Name: "dashboard", FQDN: "my.use1.wardnet.network", Root: "/srv/wardnet/app/dashboard/current", Spa: true}, } - got, err := Render(routes, apps, nil, 0, nil) + got, err := Render(routes, apps, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "listen 127.0.0.1:11443 ssl proxy_protocol;", "app server moves to the loopback terminator") assert.Contains(t, got, "my.use1.wardnet.network 127.0.0.1:11443;", "app FQDN joins the preread map") @@ -326,7 +326,7 @@ http { } } ` - got, err := Render(routes, apps, nil, 0, nil) + got, err := Render(routes, apps, nil, 0, nil, nil) require.NoError(t, err) assert.Equal(t, want, got) } @@ -337,7 +337,7 @@ http { func TestRenderAppOnly(t *testing.T) { got, err := Render(nil, []types.IngressApp{ {Name: "my", FQDN: "my.wardnet.network", Root: "/srv/wardnet/app/my/current", Spa: true}, - }, nil, 0, nil) + }, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "http {") assert.Contains(t, got, "acme_certificate letsencrypt;") @@ -358,7 +358,7 @@ func TestRenderAppOnly(t *testing.T) { func TestRenderIncludesMimeTypes(t *testing.T) { got, err := Render(nil, []types.IngressApp{ {Name: "my", FQDN: "my.wardnet.network", Root: "/srv/wardnet/app/my/current", Spa: true}, - }, nil, 0, nil) + }, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "include "+mimeTypesPath+";") // The fallback for an unmapped extension must be a byte stream, not text — @@ -369,7 +369,7 @@ func TestRenderIncludesMimeTypes(t *testing.T) { // TestRenderAppEmptyRootErrors: an app with no resolved document root fails loud // rather than letting nginx serve the whole filesystem. func TestRenderAppEmptyRootErrors(t *testing.T) { - _, err := Render(nil, []types.IngressApp{{Name: "my", FQDN: "my.wardnet.network", Spa: true}}, nil, 0, nil) + _, err := Render(nil, []types.IngressApp{{Name: "my", FQDN: "my.wardnet.network", Spa: true}}, nil, 0, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "no document root") } @@ -388,7 +388,7 @@ func TestRenderHealthGolden(t *testing.T) { {Service: "web", FQDN: "web.svc.prd.use1.wardnet.network", Target: 3001, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, } - got, err := Render(routes, nil, health, 81, nil) + got, err := Render(routes, nil, health, 81, nil, nil) require.NoError(t, err) assert.Contains(t, got, " server {\n listen 81;\n server_name api.svc.prd.use1.wardnet.network;\n location = /healthz {\n proxy_pass http://127.0.0.1:8081;\n proxy_set_header Host $host;\n }\n location / {\n return 404;\n }\n }") assert.Contains(t, got, " server {\n listen 81;\n server_name web.svc.prd.use1.wardnet.network;\n location = /healthz {\n proxy_pass http://127.0.0.1:3001;\n proxy_set_header Host $host;\n }\n location / {\n return 404;\n }\n }") @@ -404,7 +404,7 @@ func TestRenderHealthCatchAll(t *testing.T) { got, err := Render(nil, nil, []types.IngressHealth{ {Service: "web", FQDN: "web.svc.prd.use1.wardnet.network", Target: 3001, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, - }, 81, nil) + }, 81, nil, nil) require.NoError(t, err) assert.Contains(t, got, " server {\n listen 81 default_server;\n server_name _;\n return 404;\n }") // The catch-all precedes every named health server, so no service's server can @@ -419,7 +419,7 @@ func TestRenderHealthCatchAll(t *testing.T) { func TestRenderNoHealthNoCatchAll(t *testing.T) { got, err := Render([]types.IngressRoute{ {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", FQDNs: []string{"api.example.com"}}, - }, nil, nil, 81, nil) + }, nil, nil, 81, nil, nil) require.NoError(t, err) assert.NotContains(t, got, "default_server") } @@ -435,21 +435,21 @@ func TestRenderForwardOn80TerminatingHostRejected(t *testing.T) { {Service: "relay", Type: types.IngressTypeForward, Listen: 80, Target: 8080, Backend: "127.0.0.1"}, } // Terminating because of an app. - _, err := Render(fwd80, []types.IngressApp{{Name: "dash", FQDN: "my.example.com", Root: "/srv/dash", Spa: true}}, nil, 81, nil) + _, err := Render(fwd80, []types.IngressApp{{Name: "dash", FQDN: "my.example.com", Root: "/srv/dash", Spa: true}}, nil, 81, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "ACME HTTP-01") - // Terminating because of a gateway. - _, err = Render(fwd80, nil, nil, 81, []types.IngressGateway{{Name: "api", FQDN: "api.example.com"}}) + // Terminating because of a gateway termination server. + _, err = Render(fwd80, nil, nil, 81, []types.IngressGateway{{Name: "api", FQDN: "api.example.com", Backend: "127.0.0.1", HTTPPort: GatewayHTTPPort}}, nil) require.Error(t, err) assert.Contains(t, err.Error(), "ACME HTTP-01") // Terminating because of a tls-termination route. _, err = Render(append(append([]types.IngressRoute(nil), fwd80...), types.IngressRoute{Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", FQDNs: []string{"api.example.com"}}, - ), nil, nil, 81, nil) + ), nil, nil, 81, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "ACME HTTP-01") // Nothing terminates TLS: no :80 http server, so the forward owns the port. - got, err := Render(fwd80, nil, nil, 81, nil) + got, err := Render(fwd80, nil, nil, 81, nil, nil) require.NoError(t, err) assert.Contains(t, got, "listen 80;") } @@ -461,7 +461,7 @@ func TestRenderForwardOnHealthPortRejected(t *testing.T) { {Service: "relay", Type: types.IngressTypeForward, Listen: 81, Target: 8080, Backend: "127.0.0.1"}, }, nil, []types.IngressHealth{ {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, - }, 81, nil) + }, 81, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "public health port") } @@ -471,7 +471,7 @@ func TestRenderForwardOnHealthPortRejected(t *testing.T) { func TestRenderHealthOnly(t *testing.T) { got, err := Render(nil, nil, []types.IngressHealth{ {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}, - }, 81, nil) + }, 81, nil, nil) require.NoError(t, err) assert.Contains(t, got, "http {") assert.Contains(t, got, "listen 81;") @@ -488,7 +488,7 @@ func TestRenderHealthPaths(t *testing.T) { got, err := Render(nil, nil, []types.IngressHealth{ {Service: "api", FQDN: "api.svc.prd.use1.wardnet.network", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/readyz", "/healthz"}}, - }, 81, nil) + }, 81, nil, nil) require.NoError(t, err) assert.Contains(t, got, " location = /healthz {\n proxy_pass http://127.0.0.1:8081;\n proxy_set_header Host $host;\n }") assert.Contains(t, got, " location = /readyz {\n proxy_pass http://127.0.0.1:8081;\n proxy_set_header Host $host;\n }") @@ -504,7 +504,7 @@ func TestRenderHealthPaths(t *testing.T) { func TestRenderHealthNoPathsErrors(t *testing.T) { _, err := Render(nil, nil, []types.IngressHealth{ {Service: "api", FQDN: "api.svc", Target: 8081, Backend: "127.0.0.1"}, - }, 81, nil) + }, 81, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "allowlist-only") } @@ -512,7 +512,7 @@ func TestRenderHealthNoPathsErrors(t *testing.T) { // TestRenderHealthNoBackendErrors: a health entry with no resolved backend fails // loud, like a route. func TestRenderHealthNoBackendErrors(t *testing.T) { - _, err := Render(nil, nil, []types.IngressHealth{{Service: "api", FQDN: "api.svc", Target: 8081}}, 81, nil) + _, err := Render(nil, nil, []types.IngressHealth{{Service: "api", FQDN: "api.svc", Target: 8081}}, 81, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "no backend address") } @@ -520,7 +520,7 @@ func TestRenderHealthNoBackendErrors(t *testing.T) { // TestRenderHealthNoPortErrors: health entries with no public health port fail loud // (the program must resolve the ingress's health port, default 81). func TestRenderHealthNoPortErrors(t *testing.T) { - _, err := Render(nil, nil, []types.IngressHealth{{Service: "api", FQDN: "api.svc", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}}, 0, nil) + _, err := Render(nil, nil, []types.IngressHealth{{Service: "api", FQDN: "api.svc", Target: 8081, Backend: "127.0.0.1", Paths: []string{"/healthz"}}}, 0, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "no public health port") } @@ -534,9 +534,9 @@ func TestRenderDeterministic(t *testing.T) { {Service: "c", Type: types.IngressTypeForward, Listen: 853, Target: 3000, Backend: "127.0.0.1"}, } b := []types.IngressRoute{a[2], a[1], a[0]} - ra, err := Render(a, nil, nil, 0, nil) + ra, err := Render(a, nil, nil, 0, nil, nil) require.NoError(t, err) - rb, err := Render(b, nil, nil, 0, nil) + rb, err := Render(b, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Equal(t, ra, rb) } @@ -546,7 +546,7 @@ func TestRenderDeterministic(t *testing.T) { func TestRenderForwardOnly(t *testing.T) { got, err := Render([]types.IngressRoute{ {Service: "bridge", Type: types.IngressTypeForward, Listen: 443, Target: 8080, Backend: "127.0.0.1"}, - }, nil, nil, 0, nil) + }, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "stream {") assert.Contains(t, got, "proxy_protocol on;") @@ -561,7 +561,7 @@ func TestRenderForwardOnly(t *testing.T) { func TestRenderTerminateOnly(t *testing.T) { got, err := Render([]types.IngressRoute{ {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, FQDNs: []string{"api.svc"}, Backend: "127.0.0.1"}, - }, nil, nil, 0, nil) + }, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "http {") assert.Contains(t, got, "acme_certificate letsencrypt;") @@ -579,7 +579,7 @@ func TestRenderCrossHostBackend(t *testing.T) { got, err := Render([]types.IngressRoute{ {Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, FQDNs: []string{"api.svc"}, Backend: "10.0.1.5"}, {Service: "dns", Type: types.IngressTypeForward, Listen: 853, Target: 5353, Backend: "10.0.1.6"}, - }, nil, nil, 0, nil) + }, nil, nil, 0, nil, nil) require.NoError(t, err) assert.Contains(t, got, "proxy_pass http://10.0.1.5:8080;", "cross-host tls-termination proxies to the backend private IP") assert.Contains(t, got, "proxy_pass 10.0.1.6:5353;", "cross-host forward proxies to the backend private IP") @@ -588,7 +588,7 @@ func TestRenderCrossHostBackend(t *testing.T) { // TestRenderUnknownTypeErrors guards the renderer against an unexpected route type. func TestRenderUnknownTypeErrors(t *testing.T) { - _, err := Render([]types.IngressRoute{{Service: "x", Type: "passthrough", Listen: 443, Backend: "127.0.0.1"}}, nil, nil, 0, nil) + _, err := Render([]types.IngressRoute{{Service: "x", Type: "passthrough", Listen: 443, Backend: "127.0.0.1"}}, nil, nil, 0, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "unknown type") } @@ -597,31 +597,51 @@ func TestRenderUnknownTypeErrors(t *testing.T) { // than silently proxying the service to localhost (the program/provider must fill // Backend — "127.0.0.1" co-located, the private IP cross-host). func TestRenderEmptyBackendErrors(t *testing.T) { - _, err := Render([]types.IngressRoute{{Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, FQDNs: []string{"api.svc"}}}, nil, nil, 0, nil) + _, err := Render([]types.IngressRoute{{Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, FQDNs: []string{"api.svc"}}}, nil, nil, 0, nil, nil) require.Error(t, err) assert.Contains(t, err.Error(), "no backend address") } -// TestRenderGateway: the north-south gateway server block — ACME on its single -// FQDN, one regex location per derived route (a listed service's public path -// glob) handing the request to the LOCAL mesh egress at the fixed gateway port -// (target named out-of-band in X-Mesh-Target so the path is preserved -// byte-for-byte), WebSocket-capable, XFF stamped at the edge, and a JSON 404 -// default for undeclared paths (ADR-0034). +// coLocatedGateway builds the termination + routing halves of a co-located +// private gateway (ADR-0045) for the render tests: the ingress-side TLS termination +// (proxy to the gateway's loopback HTTP port) and the gateway-side plain-HTTP routing +// server (real_ip recovery → mesh egress). +func coLocatedGateway(name, fqdn string, routes []types.IngressGatewayRoute, healthPaths []string) (term, route types.IngressGateway) { + term = types.IngressGateway{Name: name, FQDN: fqdn, Backend: "127.0.0.1", HTTPPort: GatewayHTTPPort} + route = types.IngressGateway{ + Name: name, FQDN: fqdn, Routes: routes, HealthProbePaths: healthPaths, + HTTPPort: GatewayHTTPPort, ListenAddr: "127.0.0.1", RealIPFrom: "127.0.0.1", + } + return term, route +} + +// TestRenderGateway: a private gateway (ADR-0045) renders two servers. The ingress +// TERMINATION server holds ACME on the FQDN and reverse-proxies the whole host to the +// gateway's plain-HTTP port, appending the real client to XFF. The gateway ROUTING +// server recovers the client IP from that XFF (trusting only the ingress), then hands +// each derived route to the LOCAL mesh egress (target out-of-band in X-Mesh-Target so +// the path is preserved), WebSocket-capable, with a JSON 404 default (ADR-0034). func TestRenderGateway(t *testing.T) { - got, err := Render(nil, nil, nil, 0, []types.IngressGateway{{ - Name: "api", - FQDN: "api.use1.wardnet.network", - Routes: []types.IngressGatewayRoute{ - {Pattern: "/tunnel/**", Service: "tunneller"}, - {Pattern: "/v*/dns/**", Service: "ddns"}, - {Pattern: "/dns-status", Service: "ddns"}, - }, - }}) + term, route := coLocatedGateway("api", "api.use1.wardnet.network", []types.IngressGatewayRoute{ + {Pattern: "/tunnel/**", Service: "tunneller"}, + {Pattern: "/v*/dns/**", Service: "ddns"}, + {Pattern: "/dns-status", Service: "ddns"}, + }, nil) + got, err := Render(nil, nil, nil, 0, []types.IngressGateway{term}, []types.IngressGateway{route}) require.NoError(t, err) + // Termination server: ACME on the FQDN, blind-proxy to the gateway HTTP port, + // APPEND the real client to XFF (the trusted internet edge). assert.Contains(t, got, "server_name api.use1.wardnet.network;") assert.Contains(t, got, "acme_certificate letsencrypt;") assert.Contains(t, got, "listen 80;", "gateway alone still needs the ACME :80 server") + assert.Contains(t, got, fmt.Sprintf("proxy_pass http://127.0.0.1:%d;", GatewayHTTPPort), "termination blind-proxies to the gateway HTTP port") + assert.Contains(t, got, "proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;", "the ingress edge appends the real client") + // Routing server: recover the client IP from the ingress XFF (trust only it), + // then hand each route to the mesh egress with X-Mesh-Target. + assert.Contains(t, got, fmt.Sprintf("listen 127.0.0.1:%d;", GatewayHTTPPort), "routing server listens plain HTTP on loopback") + assert.Contains(t, got, "set_real_ip_from 127.0.0.1;") + assert.Contains(t, got, "real_ip_header X-Forwarded-For;") + assert.Contains(t, got, "real_ip_recursive on;") assert.Contains(t, got, "map $http_upgrade $connection_upgrade", "WS upgrade map present") assert.Contains(t, got, `location ~ "^/v[^/]+/dns(/.*)?$" {`, "glob compiled to an anchored regex location") assert.Contains(t, got, `location ~ "^/dns-status$" {`, "exact glob compiled to an anchored regex location") @@ -629,12 +649,11 @@ func TestRenderGateway(t *testing.T) { assert.Contains(t, got, "proxy_set_header X-Mesh-Target ddns;") assert.Contains(t, got, "proxy_set_header X-Mesh-Target tunneller;") assert.Contains(t, got, fmt.Sprintf("proxy_pass http://127.0.0.1:%d;", meshpaths.GatewayEgressPort)) - // XFF is SET to the real client at the internet edge, not appended — a - // daemon-supplied X-Forwarded-For must not survive into the mesh. + // Toward the mesh, XFF is SET to the recovered client ($remote_addr) — the callee + // reads it as the leftmost entry. assert.Contains(t, got, "proxy_set_header X-Forwarded-For $remote_addr;") - assert.NotContains(t, got, "$proxy_add_x_forwarded_for", "the edge must not append a client-supplied XFF") assert.Contains(t, got, "proxy_set_header Upgrade $http_upgrade;") - // Undeclared paths are answered with a JSON 404 at the edge, never proxied. + // Undeclared paths are answered with a JSON 404, never proxied. assert.Contains(t, got, "default_type application/json;") assert.Contains(t, got, `return 404 '{"error":"not_found"}';`) // Routes render in sorted-pattern order (/dns-status before /tunnel/** before /v*/dns/**). @@ -644,15 +663,12 @@ func TestRenderGateway(t *testing.T) { } // TestRenderGatewaySelfHealth: the gateway's declared health probe paths are -// answered 200 "ok" by nginx itself on the 443 server — edge liveness over the -// real daemon TLS path — and render before the route locations. +// answered 200 "ok" by nginx itself on the ROUTING server (reached through the +// ingress over the real public path) and render before the route locations. func TestRenderGatewaySelfHealth(t *testing.T) { - got, err := Render(nil, nil, nil, 0, []types.IngressGateway{{ - Name: "api", - FQDN: "api.use1.wardnet.network", - Routes: []types.IngressGatewayRoute{{Pattern: "/tenants/**", Service: "tenants"}}, - HealthProbePaths: []string{"/healthz"}, - }}) + term, route := coLocatedGateway("api", "api.use1.wardnet.network", + []types.IngressGatewayRoute{{Pattern: "/tenants/**", Service: "tenants"}}, []string{"/healthz"}) + got, err := Render(nil, nil, nil, 0, []types.IngressGateway{term}, []types.IngressGateway{route}) require.NoError(t, err) assert.Contains(t, got, "location = /healthz {") assert.Contains(t, got, "default_type text/plain;") @@ -664,33 +680,40 @@ func TestRenderGatewaySelfHealth(t *testing.T) { // fails the render loud (validation guarantees it never happens; the renderer // must not emit a broken location if it does). func TestRenderGatewayBadGlobErrors(t *testing.T) { - _, err := Render(nil, nil, nil, 0, []types.IngressGateway{{ - Name: "api", FQDN: "api.use1.wardnet.network", - Routes: []types.IngressGatewayRoute{{Pattern: "/a/**/b", Service: "tenants"}}, - }}) + _, route := coLocatedGateway("api", "api.use1.wardnet.network", + []types.IngressGatewayRoute{{Pattern: "/a/**/b", Service: "tenants"}}, nil) + _, err := Render(nil, nil, nil, 0, nil, []types.IngressGateway{route}) require.Error(t, err) assert.Contains(t, err.Error(), "invalid path glob") } -// TestRenderGatewayMixedPort: a forward on :443 coexists with the gateway via -// ssl_preread — the gateway server moves to a loopback terminator and its FQDN -// joins the SNI map, exactly like an app. +// TestRenderGatewayMixedPort: a forward on :443 coexists with the gateway +// TERMINATION server via ssl_preread — the termination moves to a loopback +// terminator and its FQDN joins the SNI map, exactly like an app. The routing +// server (plain HTTP on its own port) is unaffected. func TestRenderGatewayMixedPort(t *testing.T) { + term, route := coLocatedGateway("api", "api.use1.wardnet.network", + []types.IngressGatewayRoute{{Pattern: "/ddns/**", Service: "ddns"}}, nil) got, err := Render([]types.IngressRoute{ {Service: "tunneller", Type: types.IngressTypeForward, Listen: 443, Target: 9443, Backend: "127.0.0.1"}, - }, nil, nil, 0, []types.IngressGateway{{ - Name: "api", FQDN: "api.use1.wardnet.network", - Routes: []types.IngressGatewayRoute{{Pattern: "/ddns/**", Service: "ddns"}}, - }}) + }, nil, nil, 0, []types.IngressGateway{term}, []types.IngressGateway{route}) require.NoError(t, err) assert.Contains(t, got, "ssl_preread on;") assert.Contains(t, got, "api.use1.wardnet.network 127.0.0.1:11443;", "gateway FQDN routes to the loopback terminator") - assert.Contains(t, got, "listen 127.0.0.1:11443 ssl proxy_protocol;", "gateway server moved to the loopback terminator") + assert.Contains(t, got, "listen 127.0.0.1:11443 ssl proxy_protocol;", "gateway termination moved to the loopback terminator") } -// TestRenderGatewayNoFQDNErrors: a gateway without a resolved FQDN fails loud. +// TestRenderGatewayNoFQDNErrors: a gateway termination without a resolved FQDN fails loud. func TestRenderGatewayNoFQDNErrors(t *testing.T) { - _, err := Render(nil, nil, nil, 0, []types.IngressGateway{{Name: "api"}}) + _, err := Render(nil, nil, nil, 0, []types.IngressGateway{{Name: "api", Backend: "127.0.0.1", HTTPPort: GatewayHTTPPort}}, nil) require.Error(t, err) assert.Contains(t, err.Error(), "no FQDN") } + +// TestRenderGatewayUnresolvedBackendErrors: a termination missing its resolved +// backend (the private gateway address) fails loud rather than proxying to nowhere. +func TestRenderGatewayUnresolvedBackendErrors(t *testing.T) { + _, err := Render(nil, nil, nil, 0, []types.IngressGateway{{Name: "api", FQDN: "api.svc"}}, nil) + require.Error(t, err) + assert.Contains(t, err.Error(), "unresolved private gateway address") +} diff --git a/internal/nginx/paths.go b/internal/nginx/paths.go index 1a27296..89bfc96 100644 --- a/internal/nginx/paths.go +++ b/internal/nginx/paths.go @@ -67,4 +67,16 @@ const ( // MaxMixedPorts bounds the reserved loopback range (far above any real ingress, // which has a handful of listen ports at most). MaxMixedPorts = 64 + + // GatewayHTTPPort is the plain-HTTP port a PRIVATE gateway's routing server + // listens on (ADR-0045). A gateway is never publicly exposed: its fronting + // ingress terminates TLS and reverse-proxies the gateway FQDN to this port over + // loopback (co-located) or the private network (split; the firewall opens it to + // the network CIDR only). The routing server recovers the real client IP from + // the ingress-stamped X-Forwarded-For (set_real_ip_from the ingress), then hands + // the request to the local mesh egress. It sits outside every other reserved + // range: meshpaths MTLSPort (8443) / egress [9500,9756], and the ssl_preread + // loopback range [LoopbackBase, LoopbackBase+MaxMixedPorts). A co-located + // backend target must avoid it, same as the loopback range. + GatewayHTTPPort = 8442 ) diff --git a/internal/types/types.go b/internal/types/types.go index d13e128..ec7496c 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -65,32 +65,38 @@ type AppSpec struct { Spa bool `yaml:"spa"` // when true, any non-file path serves index.html (SPA deep-link fallback) } -// GatewaySpec is the north-south daemon API gateway (ADR-0032): the public host -// external daemons HTTPS into. It is NOT the east-west router (service↔service -// runs through the derived mesh, not here). Like IngressSpec it is a sibling of -// the workloads that references a compute Host by name in the SAME scope (an FK -// exactly like ingress.host, resolved via resolveComputeHost) and carries no -// provider of its own (it inherits the host's). It is a mesh client with identity -// /gateway: a daemon request is TLS-terminated here, matched against the -// public path globs of the listed Services (the routing table is DERIVED from -// their mesh.public_paths — ADR-0034: the gateway names WHICH services are -// public, the service names WHAT endpoints exist), and handed to the owning -// service THROUGH the mesh (so the gateway is location-transparent and needs no -// service locations). A path matching no public glob is answered 404 (JSON) at -// the edge and never traverses. It does NOT validate the daemon JWT — it -// forwards it for the service to validate. +// GatewaySpec is the north-south daemon API gateway (ADR-0032, ADR-0045): a +// PRIVATE router fronted by an ingress. It is NOT the east-west router +// (service↔service runs through the derived mesh, not here). External daemons +// HTTPS into the referenced Ingress (its public FQDN resolves to the ingress +// host); the ingress terminates TLS, enforces the edge security tier +// (CrowdSec/rate-limit, ADR-0043), and reverse-proxies the gateway FQDN over the +// private network to this gateway, preserving the client IP. A gateway is +// therefore NEVER a public edge — it opens no public port and holds no ACME cert +// of its own. Like IngressSpec it references a compute Host by name in the SAME +// scope (an FK resolved via resolveComputeHost) and carries no provider of its +// own (it inherits the host's). It is a mesh client with identity /gateway: +// a forwarded daemon request is matched against the public path globs of the +// listed Services (the routing table is DERIVED from their mesh.public_paths — +// ADR-0034: the gateway names WHICH services are public, the service names WHAT +// endpoints exist), and handed to the owning service THROUGH the mesh (so the +// gateway is location-transparent and needs no service locations). A path +// matching no public glob is answered 404 (JSON) and never traverses. It does NOT +// validate the daemon JWT — it forwards it for the service to validate. type GatewaySpec struct { Name string `yaml:"name"` Container string `yaml:"container"` Host string `yaml:"host"` // FK -> compute resource name (same scope); reuses the host's provisioning/firewall/SSH + Ingress string `yaml:"ingress"` // FK -> ingress resource (same scope) that fronts this gateway (ADR-0045). REQUIRED: a gateway is never public; the ingress terminates TLS + security and reverse-proxies to it, preserving the client IP. host and ingress.host must share a network. Pki string `yaml:"pki"` // FK -> two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from (required); every listed service must join the same mesh - Subdomain string `yaml:"subdomain"` // public subdomain; the FQDN is composed at realization from scope + base domain + Subdomain string `yaml:"subdomain"` // public subdomain; the FQDN is composed at realization from scope + base domain and resolves to the ingress host Services []string `yaml:"services"` // FKs -> services (same scope) exposed at the edge; the routing table is derived from their mesh.public_paths - HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN; defaults to 81) - HealthProbePaths []string `yaml:"health_probe_paths,omitempty"` // exact paths on the gateway's own 443 server that nginx answers 200 "ok" directly (edge liveness over the real TLS path); optional - // Security opts this gateway edge out of the env-level security tier (ADR-0043) when - // set to false: no rate limiting on its mesh routes (and, slice 2, no CrowdSec on its - // host). Nil/absent = the env policy applies. + HealthProbesPort int `yaml:"health_probes_port,omitempty"` // public port the FRONTING INGRESS host exposes the listed services' health checks on (plain HTTP, Host-demuxed by service FQDN; defaults to 81) + HealthProbePaths []string `yaml:"health_probe_paths,omitempty"` // exact paths nginx answers 200 "ok" directly for edge liveness (served by the gateway, reached through the ingress over the real public path); optional + // Security opts this gateway's forwarded traffic out of the env-level rate limit + // (ADR-0043/0045) at the fronting ingress when set to false. A gateway is never a + // public edge, so this no longer affects CrowdSec host selection. Nil/absent = the + // env policy applies. Security *bool `yaml:"security,omitempty"` } @@ -464,12 +470,29 @@ type IngressHealth struct { // location is the mesh's business. type IngressGateway struct { Name string // gateway resource name (used to name Pulumi command resources) - FQDN string // fully-qualified gateway domain (single SNI / ACME cert) + FQDN string // fully-qualified gateway domain (single SNI / ACME cert on the fronting ingress) Routes []IngressGatewayRoute - HealthProbePaths []string // exact paths on the 443 server nginx answers 200 "ok" directly (edge liveness) - // RateLimit is the resolved rate-limit profile applied to this gateway's mesh-route - // locations (nil = none). The edge health-probe locations are never limited. + HealthProbePaths []string // exact paths the gateway's routing server answers 200 "ok" directly (edge liveness, reached through the ingress) + // RateLimit is the resolved rate-limit profile applied to this gateway's termination + // server on the ingress (nil = none). The edge health-probe locations are never limited. RateLimit *RateLimitProfile + // Model-A private-gateway wiring (ADR-0045). A gateway is fronted by an ingress: + // the ingress renders a TERMINATION server (TLS + security → proxy to the gateway), + // and the gateway host renders a plain-HTTP ROUTING server (real_ip recovery → + // mesh egress). These fields carry the resolved addresses that connect the two. + // + // Backend is the address the ingress termination server proxies to — "127.0.0.1" + // when the gateway is co-located with its ingress, else the gateway host's private + // IP. HTTPPort is the routing server's plain-HTTP port (nginx.GatewayHTTPPort). + // RealIPFrom is the source the routing server trusts for X-Forwarded-For — + // "127.0.0.1" co-located, else the ingress host's private IP. ListenAddr is the + // routing server's bind address — "127.0.0.1" co-located (only the local ingress + // reaches it), else the gateway host's OWN private IP (never the public interface; + // left empty for the provider to fill from the host being realized). + Backend string + HTTPPort int + RealIPFrom string + ListenAddr string } // IngressGatewayRoute is one derived path route on the gateway server: daemon @@ -748,8 +771,15 @@ type DnsProvider interface { // host's cloud-init readiness gate (and any other prerequisites): the provider must // make its first per-host SSH command depend on it so realization never races // deploy_user creation. +// gateways are the gateway TERMINATION servers this host fronts as an ingress +// (ADR-0045); gatewayRoutes are the gateway ROUTING servers this host runs as a +// gateway host. When split (gateway and ingress on different hosts) a cross-host +// private IP is substituted from gatewayIPs, keyed by gateway name — the gateway +// host's IP for a termination's Backend, the ingress host's IP for a routing +// server's real-IP source. Co-located gateways carry "127.0.0.1" already and need +// no entry. type IngressProvider interface { - Realize(ctx *pulumi.Context, hostKey string, host ComputeOutputs, deployUser string, routes []IngressRoute, apps []IngressApp, health []IngressHealth, healthPort int, gateways []IngressGateway, backendIPs map[string]pulumi.StringOutput, env string, dependsOn []pulumi.Resource) error + Realize(ctx *pulumi.Context, hostKey string, host ComputeOutputs, deployUser string, routes []IngressRoute, apps []IngressApp, health []IngressHealth, healthPort int, gateways []IngressGateway, gatewayRoutes []IngressGateway, backendIPs map[string]pulumi.StringOutput, gatewayIPs map[string]pulumi.StringOutput, env string, dependsOn []pulumi.Resource) error } // MeshProvider realizes a host's east-west mesh proxy — the SECOND nginx, private diff --git a/internal/validate/validate.go b/internal/validate/validate.go index 899b612..e789a6d 100644 --- a/internal/validate/validate.go +++ b/internal/validate/validate.go @@ -374,9 +374,12 @@ type regionContext struct { gatewayServiceTargets map[string]bool // gatewayHostKey/gatewayHealthPort are the scope singleton gateway's canonical // host and effective public health port ("" / 0 when the scope has no gateway), - // for the gateway-tier co-location checks in checkService. - gatewayHostKey string - gatewayHealthPort int + // for the gateway-tier co-location checks in checkService. gatewayIngressHostKey is + // the canonical host of the gateway's fronting ingress (ADR-0045), where its TLS + // termination binds :443/:80 (the gateway's own host binds only GatewayHTTPPort). + gatewayHostKey string + gatewayIngressHostKey string + gatewayHealthPort int // servicePkiByName maps each service name in this scope to its pki: mesh membership. // checkGateway requires every route target to join the SAME mesh as the gateway — // a callee's trust bundle only admits callers chaining to its own mesh's @@ -993,6 +996,7 @@ func validateResourceSet(r *reporter, schemaSet map[string]*jsonschema.Schema, b ctx.gatewayServiceTargets[name] = true } ctx.gatewayHostKey = canonicalHost(f.spec.Host, ctx) + ctx.gatewayIngressHostKey = ctx.ingressHost[f.spec.Ingress] // ingress pass ran earlier ctx.gatewayHealthPort = f.spec.EffectiveHealthProbesPort() } // Service name sets (this scope): every service name, and the mesh-member subset, @@ -1144,10 +1148,17 @@ func validateResourceSet(r *reporter, schemaSet map[string]*jsonschema.Schema, b addNginxBind(host, 443, "app TLS listener") } } - // The gateway host binds :443 (daemon edge) and terminates TLS there (-> :80). + // A private gateway (ADR-0045) is fronted by an ingress: its TLS termination binds + // :443 (-> :80 ACME) on the FRONTING INGRESS host, while its plain-HTTP routing + // server binds GatewayHTTPPort on the gateway's OWN host. A co-located backend must + // avoid the routing port (EADDRINUSE with the routing nginx), the same way it must + // avoid an app/route :443 or the loopback range. if ctx.gatewayHostKey != "" { - ctx.tlsTermIngressByHost[ctx.gatewayHostKey] = true - addNginxBind(ctx.gatewayHostKey, 443, "gateway TLS listener") + if ih := ctx.gatewayIngressHostKey; ih != "" { + ctx.tlsTermIngressByHost[ih] = true + addNginxBind(ih, 443, "gateway TLS termination") + } + addNginxBind(ctx.gatewayHostKey, nginx.GatewayHTTPPort, "gateway routing listener") } // Anything terminating TLS on a host provisions an ACME cert, so nginx binds :80 // (HTTP-01 challenge + redirect) there. Its :443 listen is a route listen (already @@ -2483,6 +2494,28 @@ func checkGateway(s types.GatewaySpec, ctx regionContext) (errs, warns []string) if ctx.gatewayScopeCount > 1 { errs = append(errs, "a scope may declare at most one gateway (it is a scope singleton — one public daemon edge per scope)") } + // The gateway is PRIVATE behind an ingress (ADR-0045): the ingress terminates + // TLS + the edge security tier and reverse-proxies the gateway FQDN to it over + // the private network. The FK is required and resolves to the scope's (singleton) + // ingress, exactly like app.ingress. Because the reverse-proxy hop is private, + // the gateway host and the ingress host must share a network (co-located is the + // trivial case: gwHost == ingHost). + var ingHost string + switch { + case s.Ingress == "": + errs = append(errs, "ingress: is required — a gateway is never publicly exposed; name the ingress (same scope) that fronts it") + case strings.HasPrefix(s.Ingress, "global/"): + errs = append(errs, fmt.Sprintf("ingress: %q references a global ingress; a gateway fronted by a global ingress is declared in the global slice itself, not referenced from a region", s.Ingress)) + case !ctx.ingressNames[s.Ingress]: + errs = append(errs, fmt.Sprintf("ingress: %q does not resolve to an ingress resource in this scope", s.Ingress)) + default: + ingHost = ctx.ingressHost[s.Ingress] + if gwHost != "" && ingHost != "" && gwHost != ingHost { + if gn, in := ctx.computeNetwork[gwHost], ctx.computeNetwork[ingHost]; gn != in { + errs = append(errs, fmt.Sprintf("ingress: %q is on network %q, but the gateway host is on network %q — the ingress reverse-proxies to the gateway over the private network, so the two hosts must share a network", s.Ingress, in, gn)) + } + } + } // Listed services (ADR-0034): the gateway's routing table is DERIVED from each // listed service's mesh.public_paths, so each name must resolve to a service in // this scope that permits the gateway (mesh.allowed_services contains @@ -2551,9 +2584,10 @@ func checkGateway(s types.GatewaySpec, ctx regionContext) (errs, warns []string) } } // The gateway's own health probe paths render as exact 200-"ok" locations on - // the 443 server (edge liveness, ADR-0034). Exact paths only, unique, and none - // may be claimed by a listed service's public glob — the exact-match location - // would silently shadow the service's endpoint. + // the gateway's (now plain-HTTP, ingress-fronted) server (edge liveness, + // ADR-0034/0045; reached through the ingress over the real public path). Exact + // paths only, unique, and none may be claimed by a listed service's public glob + // — the exact-match location would silently shadow the service's endpoint. errs = append(errs, checkExactPaths("health_probe_paths", s.HealthProbePaths)...) for _, hp := range s.HealthProbePaths { for _, pp := range patterns { @@ -2562,15 +2596,22 @@ func checkGateway(s types.GatewaySpec, ctx regionContext) (errs, warns []string) } } } - // The public health port the gateway host exposes its listed services' health - // on (plain HTTP, Host-demuxed) — same rules as the ingress twin, plus 443 - // (the gateway terminates daemon TLS there). + // The public health port the FRONTING INGRESS host exposes the gateway's listed + // services' health on (plain HTTP, Host-demuxed) — the gateway is private, so its + // listed-service health follows it behind the ingress (ADR-0045), rendered on the + // ingress host exactly like a service-with-an-ingress. Same rules as the ingress + // twin, plus 443 (the ingress terminates TLS there). Checks target the ingress + // host (== the gateway host when co-located). + healthHost := ingHost + if healthHost == "" { + healthHost = gwHost + } healthPort := s.EffectiveHealthProbesPort() if healthPort == 80 { - errs = append(errs, "health_probes_port: must not be 80 (ACME HTTP-01 owns :80 on the gateway host)") + errs = append(errs, "health_probes_port: must not be 80 (ACME HTTP-01 owns :80 on the fronting ingress host)") } if healthPort == 443 { - errs = append(errs, "health_probes_port: must not be 443 (the gateway terminates daemon TLS there)") + errs = append(errs, "health_probes_port: must not be 443 (the fronting ingress terminates TLS there)") } if inReservedLoopbackRange(healthPort) { errs = append(errs, fmt.Sprintf("health_probes_port: %d falls in the reserved internal range [%d,%d) nginx uses for ssl_preread TLS terminators; pick a health port outside it", healthPort, nginx.LoopbackBase, nginx.LoopbackBase+nginx.MaxMixedPorts)) @@ -2581,15 +2622,15 @@ func checkGateway(s types.GatewaySpec, ctx regionContext) (errs, warns []string) if msg := meshEgressRangeErr("health_probes_port", healthPort); msg != "" { errs = append(errs, msg) } - if gwHost != "" { - if users := ctx.portUsersByHost[gwHost][healthPort]; len(users) > 0 { - errs = append(errs, fmt.Sprintf("health_probes_port: %d collides with a route listen port on the gateway host (used by service(s) %s); pick a distinct health port", healthPort, strings.Join(users, ", "))) + if healthHost != "" { + if users := ctx.portUsersByHost[healthHost][healthPort]; len(users) > 0 { + errs = append(errs, fmt.Sprintf("health_probes_port: %d collides with a route listen port on the ingress host (used by service(s) %s); pick a distinct health port", healthPort, strings.Join(users, ", "))) } // One host renders ONE health listener (Render takes a single healthPort per - // host), so a gateway sharing its host with an ingress must agree on the port. - for _, ingName := range ctx.ingressNamesByHost[gwHost] { + // host), so the gateway's health port must match the fronting ingress's own. + for _, ingName := range ctx.ingressNamesByHost[healthHost] { if ip := ctx.ingressHealthPort[ingName]; ip != 0 && ip != healthPort { - errs = append(errs, fmt.Sprintf("health_probes_port: %d differs from ingress %q's public health port %d on the shared host %q; one host renders one health listener, so the two must match", healthPort, ingName, ip, s.Host)) + errs = append(errs, fmt.Sprintf("health_probes_port: %d differs from ingress %q's public health port %d on host %q; one host renders one health listener, so the two must match", healthPort, ingName, ip, healthHost)) } } } diff --git a/internal/validate/validate_test.go b/internal/validate/validate_test.go index a34fe63..14f8b6f 100644 --- a/internal/validate/validate_test.go +++ b/internal/validate/validate_test.go @@ -823,6 +823,10 @@ func meshCtx() regionContext { c.servicePublicPathsByName = map[string][]string{"ddns": {"/dns/**"}, "tunneller": {"/tunnel/**"}} c.targetUsersByHost = map[string]map[int][]string{} c.portUsersByHost = map[string]map[int][]string{} + // The gateway is fronted by an ingress (ADR-0045); "front" is co-located on the + // gateway's host (bridge), so the same-network check is trivially satisfied. + c.ingressNames = map[string]bool{"front": true} + c.ingressHost = map[string]string{"front": "bridge-01"} return c } @@ -836,16 +840,47 @@ func meshSvc(m *types.MeshSpec) types.ServiceSpec { } func gw(services ...string) types.GatewaySpec { - return types.GatewaySpec{Name: "api", Host: "bridge", Pki: "mesh", Subdomain: "api", Services: services} + return types.GatewaySpec{Name: "api", Host: "bridge", Ingress: "front", Pki: "mesh", Subdomain: "api", Services: services} } -// TestCheckGatewayValid: a gateway on a same-scope vm, sole in scope, listing a -// service that permits the gateway and publishes public paths, passes. +// TestCheckGatewayValid: a gateway on a same-scope vm, sole in scope, fronted by an +// ingress, listing a service that permits the gateway and publishes public paths, passes. func TestCheckGatewayValid(t *testing.T) { errs, _ := checkGateway(gw("ddns"), meshCtx()) assert.Empty(t, errs) } +// TestCheckGatewayRequiresIngress: the ingress FK is mandatory (ADR-0045) — a gateway +// is never a standalone public edge. +func TestCheckGatewayRequiresIngress(t *testing.T) { + g := gw("ddns") + g.Ingress = "" + errs, _ := checkGateway(g, meshCtx()) + require.NotEmpty(t, errs) + assert.Contains(t, strings.Join(errs, "\n"), "ingress: is required") +} + +// TestCheckGatewayUnknownIngressRejected: the ingress FK must resolve in scope. +func TestCheckGatewayUnknownIngressRejected(t *testing.T) { + g := gw("ddns") + g.Ingress = "ghost" + errs, _ := checkGateway(g, meshCtx()) + require.NotEmpty(t, errs) + assert.Contains(t, strings.Join(errs, "\n"), "does not resolve to an ingress") +} + +// TestCheckGatewayCrossNetworkIngressRejected: the gateway host and its fronting +// ingress host must share a network (the ingress reverse-proxies over the private net). +func TestCheckGatewayCrossNetworkIngressRejected(t *testing.T) { + c := twoNetworkMeshCtx() + c.ingressHost = map[string]string{"front": "bridge-01"} // ingress on net-a, gateway on edge/net-b + g := types.GatewaySpec{Name: "api", Host: "edge", Ingress: "front", Pki: "mesh", Subdomain: "api", + Services: []string{"ddns"}} + errs, _ := checkGateway(g, c) + require.NotEmpty(t, errs) + assert.Contains(t, strings.Join(errs, "\n"), "reverse-proxies to the gateway") +} + // TestCheckGatewayGlobalHostRejected: a gateway referencing a global compute is // rejected — like service.host, it is declared in the global slice itself. func TestCheckGatewayGlobalHostRejected(t *testing.T) { @@ -858,7 +893,7 @@ func TestCheckGatewayGlobalHostRejected(t *testing.T) { // TestCheckGatewayUnknownHostRejected: a gateway whose host: does not resolve fails. func TestCheckGatewayUnknownHostRejected(t *testing.T) { c := meshCtx() - errs, _ := checkGateway(types.GatewaySpec{Name: "api", Host: "ghost", Subdomain: "api"}, c) + errs, _ := checkGateway(types.GatewaySpec{Name: "api", Host: "ghost", Ingress: "front", Subdomain: "api"}, c) require.Len(t, errs, 1) assert.Contains(t, errs[0], "does not resolve to a compute") } @@ -1011,6 +1046,9 @@ func twoNetworkMeshCtx() regionContext { "edge-01": "net-b", "edge": "net-b", } c.serviceHostByName = map[string]string{"ddns": "bridge", "tunneller": "bridge"} + // The gateway sits on "edge" here, so co-locate its fronting ingress there (the + // gateway↔ingress same-network check is separate from the target check under test). + c.ingressHost = map[string]string{"front": "edge-01"} return c } @@ -1019,7 +1057,7 @@ func twoNetworkMeshCtx() regionContext { // private IP, unroutable across networks. func TestCheckGatewayCrossNetworkTargetRejected(t *testing.T) { c := twoNetworkMeshCtx() - g := types.GatewaySpec{Name: "api", Host: "edge", Pki: "mesh", Subdomain: "api", + g := types.GatewaySpec{Name: "api", Host: "edge", Ingress: "front", Pki: "mesh", Subdomain: "api", Services: []string{"ddns"}} errs, _ := checkGateway(g, c) require.NotEmpty(t, errs) @@ -1031,7 +1069,7 @@ func TestCheckGatewaySameNetworkTargetOK(t *testing.T) { c := twoNetworkMeshCtx() c.computeNetwork["edge-01"] = "net-a" c.computeNetwork["edge"] = "net-a" - g := types.GatewaySpec{Name: "api", Host: "edge", Pki: "mesh", Subdomain: "api", + g := types.GatewaySpec{Name: "api", Host: "edge", Ingress: "front", Pki: "mesh", Subdomain: "api", Services: []string{"ddns"}} errs, _ := checkGateway(g, c) assert.Empty(t, errs) @@ -1630,6 +1668,16 @@ func TestCheckServiceNginxImplicitBinds(t *testing.T) { Routes: []types.RouteSpec{{Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8081}}} errs, _ = checkService(rs, rc) assert.Contains(t, strings.Join(errs, "|"), "held by the edge nginx") + + // exposed_ports hitting the gateway routing server's port on the gateway host + // (ADR-0045): a co-located backend must avoid GatewayHTTPPort. + gc := baseCtx() + gc.nginxBindsByHost = map[string]map[int]string{"bridge-01": {nginx.GatewayHTTPPort: "gateway routing listener"}} + gs := types.ServiceSpec{Name: "svc", Host: "bridge", Type: "raw", User: "svc", + ExposedPorts: []types.ExposedPort{{Proto: "tcp", Port: nginx.GatewayHTTPPort}}} + errs, _ = checkService(gs, gc) + require.NotEmpty(t, errs) + assert.Contains(t, strings.Join(errs, "|"), "gateway routing listener") } // TestCheckMeshEgressRangeOnPublicBinds: a PUBLIC bind (route listen or the diff --git a/program/crowdsec_test.go b/program/crowdsec_test.go index 9ce37a0..bbe9929 100644 --- a/program/crowdsec_test.go +++ b/program/crowdsec_test.go @@ -17,33 +17,39 @@ func edgeFixture() types.Resources { {Name: "worker", Kind: "vm", InstanceCount: 1}, }, Ingress: []types.IngressSpec{{Name: "main", Host: "edge"}}, - Gateway: []types.GatewaySpec{{Name: "gw", Host: "gwhost"}}, + // The gateway is fronted by the ingress (ADR-0045). Its own host (gwhost) is + // PRIVATE and is never a CrowdSec edge. + Gateway: []types.GatewaySpec{{Name: "gw", Host: "gwhost", Ingress: "main"}}, } } -func TestCrowdsecEdgeHostsIncludesIngressAndGatewayOnly(t *testing.T) { +// The public edge is the INGRESS tier only (ADR-0045): a gateway is never a public +// edge, so its host contributes no CrowdSec host. +func TestCrowdsecEdgeHostsIsIngressOnly(t *testing.T) { res := edgeFixture() canonical := naming.CanonicalComputeKeys(res.Compute) edge := crowdsecEdgeHosts(res, canonical) assert.True(t, edge[canonical["edge"]], "ingress host is an edge") - assert.True(t, edge[canonical["gwhost"]], "gateway host is an edge") + assert.False(t, edge[canonical["gwhost"]], "the gateway host is private, never a CrowdSec edge") assert.False(t, edge[canonical["worker"]], "a plain worker gets no CrowdSec") - assert.Len(t, edge, 2) + assert.Len(t, edge, 1) } -func TestCrowdsecEdgeHostsHonorsOptOut(t *testing.T) { +func TestCrowdsecEdgeHostsHonorsIngressOptOut(t *testing.T) { res := edgeFixture() res.Ingress[0].Security = ptrBool(false) // this ingress opts out canonical := naming.CanonicalComputeKeys(res.Compute) edge := crowdsecEdgeHosts(res, canonical) assert.False(t, edge[canonical["edge"]], "opted-out ingress host is excluded") - assert.True(t, edge[canonical["gwhost"]], "the gateway host is a separate edge and stays in") + assert.Empty(t, edge, "no other public edge remains — the gateway host is never one") } -func TestCrowdsecEdgeHostsExplicitTrueStillIncluded(t *testing.T) { +// The gateway's `security:` field no longer affects CrowdSec host selection: a gateway +// is never a public edge, whatever the flag says. +func TestCrowdsecEdgeHostsIgnoresGatewaySecurityFlag(t *testing.T) { res := edgeFixture() - res.Gateway[0].Security = ptrBool(true) // explicit opt-in == absent + res.Gateway[0].Security = ptrBool(true) canonical := naming.CanonicalComputeKeys(res.Compute) edge := crowdsecEdgeHosts(res, canonical) - assert.True(t, edge[canonical["gwhost"]]) + assert.False(t, edge[canonical["gwhost"]], "the gateway host stays out regardless of its security flag") } diff --git a/program/gateway_paths_test.go b/program/gateway_paths_test.go index a1d4f29..b3d96f1 100644 --- a/program/gateway_paths_test.go +++ b/program/gateway_paths_test.go @@ -5,14 +5,15 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/wardnet/inforge/internal/nginx" "github.com/wardnet/inforge/internal/types" ) -// gatewayPathsRes is the shared fixture for the ADR-0034 derivations: a gateway -// on its own host (edge) listing two services — tenants (co-located with the -// gateway? no: on back, cross-host, with health) and billing (on edge, -// co-located, no health) — plus an ingress-fronted service that must stay out of -// the gateway health tier (D12). +// gatewayPathsRes is the shared fixture for the ADR-0034/0045 derivations: a +// gateway on host `edge`, fronted by the scope ingress `web` on host `back` (a +// SPLIT gateway — the gateway and its ingress are on different hosts), listing two +// services — tenants (on back, with health) and billing (on edge, no health) — +// plus an ingress-fronted service that must stay out of the gateway health tier (D12). func gatewayPathsRes() types.Resources { return types.Resources{ Compute: []types.ComputeSpec{ @@ -21,7 +22,7 @@ func gatewayPathsRes() types.Resources { }, Ingress: []types.IngressSpec{{Name: "web", Host: "back"}}, Gateway: []types.GatewaySpec{{ - Name: "api", Container: "edge", Host: "edge", Pki: "mesh", Subdomain: "api", + Name: "api", Container: "edge", Host: "edge", Ingress: "web", Pki: "mesh", Subdomain: "api", Services: []string{"tenants", "billing"}, HealthProbePaths: []string{"/healthz"}, }}, @@ -54,21 +55,49 @@ func TestToGatewayNginxRoutes(t *testing.T) { }, routes) } -// TestGatewaysByHostThreadsRoutesAndHealth: the derived IngressGateway carries -// the derived routes and the gateway's own health probe paths. -func TestGatewaysByHostThreadsRoutesAndHealth(t *testing.T) { +// TestGatewayEdgeByHostSplitsTerminationAndRouting: a private gateway (ADR-0045) +// derives a TERMINATION server on its fronting ingress host (back-01) and a ROUTING +// server on its own host (edge-01), carrying the derived routes + health paths. Being +// split, each half records the peer host whose private IP the provider resolves. +func TestGatewayEdgeByHostSplitsTerminationAndRouting(t *testing.T) { res := gatewayPathsRes() canonical := map[string]string{"edge": "edge-01", "back": "back-01"} - byHost := gatewaysByHost(res, canonical, "use1", "wardnet.network", "") - require.Len(t, byHost["edge-01"], 1) - g := byHost["edge-01"][0] - assert.Equal(t, []string{"/healthz"}, g.HealthProbePaths) - assert.Len(t, g.Routes, 3) + terms, routings, peers := gatewayEdgeByHost(res, canonical, "use1", "wardnet.network", "") + // Termination on the ingress host; routing on the gateway host. + require.Len(t, terms["back-01"], 1) + require.Len(t, routings["edge-01"], 1) + assert.Empty(t, terms["edge-01"]) + assert.Empty(t, routings["back-01"]) + route := routings["edge-01"][0] + assert.Equal(t, []string{"/healthz"}, route.HealthProbePaths) + assert.Len(t, route.Routes, 3) + assert.Empty(t, route.ListenAddr, "split routing server binds the gateway host's own private IP, filled by the provider") + assert.Empty(t, route.RealIPFrom, "split routing real_ip source resolves from the ingress host's private IP") + assert.Empty(t, terms["back-01"][0].Backend, "split termination backend resolves from the gateway host's private IP") + // Peer-IP needs: the termination (on back-01) needs the gateway host (edge-01); + // the routing (on edge-01) needs the ingress host (back-01). + assert.Equal(t, "edge-01", peers["back-01"]["api"]) + assert.Equal(t, "back-01", peers["edge-01"]["api"]) +} + +// TestGatewayEdgeByHostCoLocated: when the gateway shares its ingress's host, both +// halves render on one host with loopback addresses and no peer-IP resolution. +func TestGatewayEdgeByHostCoLocated(t *testing.T) { + res := gatewayPathsRes() + res.Ingress = []types.IngressSpec{{Name: "web", Host: "edge"}} // move the ingress onto the gateway host + canonical := map[string]string{"edge": "edge-01", "back": "back-01"} + terms, routings, peers := gatewayEdgeByHost(res, canonical, "use1", "wardnet.network", "") + require.Len(t, terms["edge-01"], 1) + require.Len(t, routings["edge-01"], 1) + assert.Equal(t, "127.0.0.1", terms["edge-01"][0].Backend) + assert.Equal(t, "127.0.0.1", routings["edge-01"][0].ListenAddr) + assert.Equal(t, "127.0.0.1", routings["edge-01"][0].RealIPFrom) + assert.Empty(t, peers, "co-located gateways need no cross-host IP resolution") } // TestResolveGatewayHealthServices: only gateway-listed, health-declaring, -// ingress-less services join the gateway health tier (D12), with co-location -// resolved against the gateway host. +// ingress-less services join the gateway health tier (D12). The listener renders on +// the gateway's FRONTING INGRESS host (ADR-0045), and co-location is relative to it. func TestResolveGatewayHealthServices(t *testing.T) { res := gatewayPathsRes() canonical := map[string]string{"edge": "edge-01", "back": "back-01"} @@ -76,41 +105,45 @@ func TestResolveGatewayHealthServices(t *testing.T) { require.Len(t, gsvcs, 1, "billing has no health port; webapp has an ingress (D12)") gs := gsvcs[0] assert.Equal(t, "tenants", gs.svc.Name) - assert.Equal(t, "edge-01", gs.gwHost) + assert.Equal(t, "back-01", gs.gwHost, "health renders on the fronting ingress host, not the gateway host") assert.Equal(t, "back-01", gs.svcHost) - assert.False(t, gs.coLocated) + assert.True(t, gs.coLocated, "tenants shares the ingress host (back), so its health backend is loopback") } -// TestGatewayHealthByHost: the health entries render on the gateway's host with -// the declared probe paths; a cross-host backend is left empty for the provider -// to substitute the private IP. +// TestGatewayHealthByHost: the health entries render on the gateway's FRONTING +// INGRESS host (ADR-0045) with the declared probe paths. tenants shares that host +// (back), so its backend is loopback. func TestGatewayHealthByHost(t *testing.T) { res := gatewayPathsRes() canonical := map[string]string{"edge": "edge-01", "back": "back-01"} byHost := gatewayHealthByHost(resolveGatewayHealthServices(res, canonical), "prd", "use1", "wardnet.network") - require.Len(t, byHost["edge-01"], 1) - h := byHost["edge-01"][0] + require.Len(t, byHost["back-01"], 1) + h := byHost["back-01"][0] assert.Equal(t, "tenants.svc.prd.use1.wardnet.network", h.FQDN) assert.Equal(t, 8081, h.Target) assert.Equal(t, []string{"/livez", "/readyz"}, h.Paths) - assert.Empty(t, h.Backend, "cross-host backend is the provider's to fill") + assert.Equal(t, "127.0.0.1", h.Backend, "tenants is co-located with the ingress host, so its health backend is loopback") } -// TestFirewallPlanByHostGatewayHealth: the gateway host opens its public health -// port (default 81) alongside 443/80; the cross-host backend opens its health -// port privately to the network CIDR. +// TestFirewallPlanByHostGatewayHealth: a private gateway (ADR-0045) opens its public +// pair (443/80) and its health port on the FRONTING INGRESS host (back-01), never on +// the gateway host; the split gateway host (edge-01) opens only its private routing +// port (GatewayHTTPPort) to the network CIDR. func TestFirewallPlanByHostGatewayHealth(t *testing.T) { res := gatewayPathsRes() got := firewallPlanByHost(res, false) - assert.Contains(t, got["edge-01"].Public, 81, "gateway health listener is public on the gateway host") - assert.Contains(t, got["edge-01"].Public, 443) - assert.Contains(t, got["edge-01"].Public, 80) - assert.Contains(t, got["back-01"].Private, 8081, "cross-host backend health port opens privately") + assert.Contains(t, got["back-01"].Public, 443, "gateway TLS termination is public on the ingress host") + assert.Contains(t, got["back-01"].Public, 80, "gateway ACME is public on the ingress host") + assert.Contains(t, got["back-01"].Public, 81, "gateway health listener is public on the ingress host") + assert.NotContains(t, got["edge-01"].Public, 443, "the gateway host is never public") + assert.NotContains(t, got["edge-01"].Public, 80) + assert.Contains(t, got["edge-01"].Private, nginx.GatewayHTTPPort, "the split gateway host opens its routing port privately") } -// TestDerivedRecordsGatewayHealth: a gateway-routed service's ServiceFQDN A -// record points at the GATEWAY host; an ingress-fronted health service's record -// points at the ingress host (D12 keeps the two exclusive). +// TestDerivedRecordsGatewayHealth: a gateway-routed service's ServiceFQDN A record +// points at the gateway's FRONTING INGRESS host (ADR-0045: health follows the gateway +// behind the ingress); an ingress-fronted health service's record points at its own +// ingress host (D12 keeps the two exclusive). Here both resolve to back-01. func TestDerivedRecordsGatewayHealth(t *testing.T) { res := gatewayPathsRes() got := derivedRecords(res, "prd", "use1", "wardnet.network", "") @@ -118,7 +151,7 @@ func TestDerivedRecordsGatewayHealth(t *testing.T) { for _, d := range got { byRecord[d.rec.RecordName] = d.hostKey } - assert.Equal(t, "edge-01", byRecord["tenants.svc.prd.use1"], "gateway-routed health record points at the gateway host") + assert.Equal(t, "back-01", byRecord["tenants.svc.prd.use1"], "gateway-routed health record points at the fronting ingress host") assert.Equal(t, "back-01", byRecord["webapp.svc.prd.use1"], "ingress-fronted health-only service gets its record at the ingress host") _, hasBilling := byRecord["billing.svc.prd.use1"] assert.False(t, hasBilling, "no health port -> no service record") diff --git a/program/program.go b/program/program.go index db03a54..29e43b1 100644 --- a/program/program.go +++ b/program/program.go @@ -31,6 +31,7 @@ import ( "github.com/wardnet/inforge/internal/meshpaths" "github.com/wardnet/inforge/internal/meshplan" "github.com/wardnet/inforge/internal/naming" + "github.com/wardnet/inforge/internal/nginx" "github.com/wardnet/inforge/internal/otelcol" "github.com/wardnet/inforge/internal/pgrole" "github.com/wardnet/inforge/internal/postgres" @@ -1036,9 +1037,12 @@ func provisionCrowdsec(ctx *pulumi.Context, res types.Resources, computeOut map[ return nil } -// crowdsecEdgeHosts returns the set of canonical host keys that run a public edge (an -// ingress or a gateway) and have not opted out of the security tier via `security: false` -// (ADR-0043). A host is included if at least one non-opted-out edge resource lands on it. +// crowdsecEdgeHosts returns the set of public-edge hosts CrowdSec installs on +// (ADR-0043). The public edge is the INGRESS tier only: a gateway is never a +// public edge (ADR-0045 — it is fronted by an ingress and holds no public port), +// so its host contributes nothing here. When a gateway is co-located with its +// ingress, the host is already covered via the ingress. The `security: false` +// opt-out is honored per ingress. func crowdsecEdgeHosts(res types.Resources, canonical map[string]string) map[string]bool { edge := map[string]bool{} for _, ing := range res.Ingress { @@ -1049,14 +1053,6 @@ func crowdsecEdgeHosts(res types.Resources, canonical map[string]string) map[str edge[hk] = true } } - for _, gw := range res.Gateway { - if gw.Security != nil && !*gw.Security { - continue - } - if hk, ok := canonical[gw.Host]; ok { - edge[hk] = true - } - } return edge } @@ -1904,11 +1900,16 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type return fmt.Errorf("ingress: %w", err) } appsByHostKey := ingressAppsByHost(res, canonical, slug, baseDomain, ephemeralSlug) - gatewaysByHostKey := gatewaysByHost(res, canonical, slug, baseDomain, ephemeralSlug) + // A private gateway (ADR-0045) renders as two halves: a TERMINATION server on the + // fronting ingress host and a ROUTING server on the gateway's own host. Co-located + // gateways carry loopback addresses; a split gateway's cross-host IP is resolved + // per host from gatewayPeerByHost below. + gatewayTermByHost, gatewayRoutingByHost, gatewayPeerByHost := gatewayEdgeByHost(res, canonical, slug, baseDomain, ephemeralSlug) // Stamp the env-uniform IP rate limit (ADR-0043) onto every edge server whose edge // has not opted out (`security: false`). Rate limiting is a blanket security floor, - // so the same profile lands on every route/app/gateway — there is no per-route knob. - stampRateLimit(sec, res, routesByHostKey, appsByHostKey, gatewaysByHostKey) + // so the same profile lands on every route/app and — for a gateway — its termination + // server on the ingress (ADR-0045), there is no per-route knob. + stampRateLimit(sec, res, routesByHostKey, appsByHostKey, gatewayTermByHost) // Resolve the ingress-tier services once and feed both derivations below — the // health entries and the cross-host backend set — so the service list is walked a // single time per host realization. @@ -1952,13 +1953,14 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type // orders for the hostname. The route-vs-route half of this rule lives in // ingressRoutesByHost; this closes the app-vs-route/app-vs-app half so the // guarantee holds even when `up` runs without a prior `validate`. - if err := checkAppSNICollisions(routesByHostKey, appsByHostKey, gatewaysByHostKey); err != nil { + if err := checkAppSNICollisions(routesByHostKey, appsByHostKey, gatewayTermByHost); err != nil { return fmt.Errorf("ingress: %w", err) } - // nginx is installed on an ingress host iff at least one route, app, health - // endpoint, OR gateway targets it; an app-only, health-only, or gateway-only - // host still realizes — its server blocks and ACME certs provision alone. - hostKeys := ingressHostUnion(routesByHostKey, appsByHostKey, healthByHostKey, gatewaysByHostKey) + // nginx is installed on a host iff at least one route, app, health endpoint, + // gateway TERMINATION (ingress host), or gateway ROUTING server (gateway host) + // targets it; a gateway-only host (split, ADR-0045) still realizes its plain-HTTP + // routing server alone. + hostKeys := ingressHostUnion(routesByHostKey, appsByHostKey, healthByHostKey, gatewayTermByHost, gatewayRoutingByHost) if len(hostKeys) == 0 { return nil } @@ -1985,6 +1987,18 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type } backendIPs[svcName] = backend.PrivateIP } + // A split gateway (ADR-0045) needs its peer host's private IP: the gateway + // host's IP for a termination server on this ingress host, or the ingress + // host's IP for a routing server on this gateway host. Co-located gateways + // record no peer, so gatewayIPs stays empty and the config renders synchronously. + gatewayIPs := map[string]pulumi.StringOutput{} + for gwName, peerHostKey := range gatewayPeerByHost[hostKey] { + peer, ok := computeOut[peerHostKey] + if !ok { + return fmt.Errorf("ingress: host %q: gateway %q peer host %q has no compute output (available: %v)", hostKey, gwName, peerHostKey, sortedKeys(computeOut)) + } + gatewayIPs[gwName] = peer.PrivateIP + } // The realization SSHes the ingress host, so it waits on that host's // cloud-init gate (shared with service provisioning when co-located). gate, err := cloudInitGate(ctx, gates, hostKey, host, deployPrivateKey, env, slug) @@ -1999,7 +2013,7 @@ func realizeIngress(ctx *pulumi.Context, reg registry.ProviderRegistry, res type if healthPort == 0 { healthPort = types.DefaultHealthProbesPort } - if err := ip.Realize(ctx, hostKey, host, deployUserByCompute[hostKey], routesByHostKey[hostKey], appsByHostKey[hostKey], healthByHostKey[hostKey], healthPort, gatewaysByHostKey[hostKey], backendIPs, env, deps); err != nil { + if err := ip.Realize(ctx, hostKey, host, deployUserByCompute[hostKey], routesByHostKey[hostKey], appsByHostKey[hostKey], healthByHostKey[hostKey], healthPort, gatewayTermByHost[hostKey], gatewayRoutingByHost[hostKey], backendIPs, gatewayIPs, env, deps); err != nil { return err } } @@ -2206,15 +2220,17 @@ func ingressAppsByHost(res types.Resources, canonical map[string]string, slug, b // consume so the three can never drift (the gateway analogue of resolveIngressApps; // rule mesh-host-grouping-is-single-sourced names gateway realization a consumer). type resolvedGateway struct { - gw types.GatewaySpec - host string // canonical compute specKey - fqdn string // naming.AppFQDN(subdomain, …) — server_name, ACME cert, and DNS record + gw types.GatewaySpec + host string // canonical compute specKey — the gateway's own host (runs the plain-HTTP routing server + mesh egress) + ingHost string // canonical specKey of the fronting ingress's host (ADR-0045): terminates TLS + security, and where the FQDN/cert/DNS live + fqdn string // naming.AppFQDN(subdomain, …) — server_name, ACME cert (on the ingress), and DNS record (→ ingress host) } // resolveGateways resolves each authored gateway to its canonical host and public // FQDN once. An unresolved host FK is skipped (validation rejects it long before // this), so every consumer sees the same host/FQDN pair. func resolveGateways(res types.Resources, canonical map[string]string, slug, baseDomain, ephemeralSlug string) []resolvedGateway { + ingHosts := ingressHostsByName(res, canonical) out := make([]resolvedGateway, 0, len(res.Gateway)) for _, gw := range res.Gateway { host, ok := canonical[gw.Host] @@ -2222,31 +2238,79 @@ func resolveGateways(res types.Resources, canonical map[string]string, slug, bas continue } out = append(out, resolvedGateway{ - gw: gw, - host: host, - fqdn: naming.AppFQDN(gw.Subdomain, slug, baseDomain, ephemeralSlug), + gw: gw, + host: host, + ingHost: ingHosts[gw.Ingress], // "" if the ingress FK is unresolved (validation rejects that) + fqdn: naming.AppFQDN(gw.Subdomain, slug, baseDomain, ephemeralSlug), }) } return out } -// gatewaysByHost groups the scope's north-south gateway (a scope singleton) under -// its canonical host as the derived types.IngressGateway the public nginx renders -// (ADR-0032/0034), via the shared resolveGateways derivation. The routing table -// is DERIVED, not authored: one route per (listed service, public path glob), -// carrying the raw pattern and the owning service name (the X-Mesh-Target -// value); the mesh resolves the target's location, so no backend is resolved here. -func gatewaysByHost(res types.Resources, canonical map[string]string, slug, baseDomain, ephemeralSlug string) map[string][]types.IngressGateway { - byHost := map[string][]types.IngressGateway{} +// gatewayEdgeByHost derives the two halves of every private gateway (ADR-0045), +// via the shared resolveGateways derivation. A gateway is fronted by an ingress: +// - the TERMINATION server (grouped by the fronting INGRESS host) terminates the +// gateway FQDN's TLS, enforces the security tier, and reverse-proxies to the +// gateway's routing server; +// - the ROUTING server (grouped by the gateway's OWN host) is plain HTTP, recovers +// the client IP from the ingress-stamped XFF, and hands each derived route to the +// local mesh egress. The routing table is DERIVED, not authored: one route per +// (listed service, public path glob). +// +// Co-located gateways carry loopback addresses ("127.0.0.1") and need no IP +// resolution. A split gateway leaves the cross-host field empty and records the peer +// host whose private IP fills it: gatewayPeerByHost[hostKey][gatewayName] is the peer +// host — the gateway host's IP for a termination's Backend, the ingress host's IP for +// a routing server's RealIPFrom. realizeIngress resolves those to compute private IPs. +func gatewayEdgeByHost(res types.Resources, canonical map[string]string, slug, baseDomain, ephemeralSlug string) (terminations, routings map[string][]types.IngressGateway, gatewayPeerByHost map[string]map[string]string) { + terminations = map[string][]types.IngressGateway{} + routings = map[string][]types.IngressGateway{} + gatewayPeerByHost = map[string]map[string]string{} + addPeer := func(host, gwName, peer string) { + if gatewayPeerByHost[host] == nil { + gatewayPeerByHost[host] = map[string]string{} + } + gatewayPeerByHost[host][gwName] = peer + } for _, rg := range resolveGateways(res, canonical, slug, baseDomain, ephemeralSlug) { - byHost[rg.host] = append(byHost[rg.host], types.IngressGateway{ + if rg.ingHost == "" { + continue // unresolved ingress FK — validation rejects it; skip defensively + } + coLocated := rg.host == rg.ingHost + term := types.IngressGateway{Name: rg.gw.Name, FQDN: rg.fqdn, HTTPPort: nginx.GatewayHTTPPort} + if coLocated { + term.Backend = "127.0.0.1" + } else { + addPeer(rg.ingHost, rg.gw.Name, rg.host) // termination needs the gateway host's private IP + } + terminations[rg.ingHost] = append(terminations[rg.ingHost], term) + + route := types.IngressGateway{ Name: rg.gw.Name, FQDN: rg.fqdn, Routes: toGatewayNginxRoutes(rg.gw.Services, res.Service), HealthProbePaths: append([]string(nil), rg.gw.HealthProbePaths...), - }) + HTTPPort: nginx.GatewayHTTPPort, + } + if coLocated { + route.ListenAddr = "127.0.0.1" + route.RealIPFrom = "127.0.0.1" + } else { + // Split: the routing server binds the gateway host's OWN private IP (the + // provider fills the empty ListenAddr from the host being realized) — never + // 0.0.0.0, so the port is not exposed on the public interface even if the + // firewall CIDR rule is ever wrong. RealIPFrom trusts the ingress host's IP. + addPeer(rg.host, rg.gw.Name, rg.ingHost) // routing needs the ingress host's private IP + } + routings[rg.host] = append(routings[rg.host], route) } - return byHost + for _, gs := range terminations { + sort.Slice(gs, func(i, j int) bool { return gs[i].FQDN < gs[j].FQDN }) + } + for _, gs := range routings { + sort.Slice(gs, func(i, j int) bool { return gs[i].FQDN < gs[j].FQDN }) + } + return terminations, routings, gatewayPeerByHost } // toGatewayNginxRoutes derives the gateway's provider-facing nginx routes from @@ -2296,7 +2360,7 @@ func ingressHealthByHost(svcs []ingressService, env, slug, baseDomain string) ma // address per service, or the ServiceFQDN A record would derive at two hosts). type gatewayHealthService struct { svc types.ServiceSpec - gwHost string // canonical specKey of the gateway's host (where the health server renders) + gwHost string // canonical specKey of the FRONTING INGRESS host (where the health server renders, ADR-0045) svcHost string // canonical specKey of the service's own host coLocated bool gwHealthPort int // the gateway's effective public health port — carried here so nginx, firewall, and DNS read ONE derivation @@ -2313,6 +2377,9 @@ func resolveGatewayHealthServices(res types.Resources, canonical map[string]stri seen := map[string]bool{} var out []gatewayHealthService for _, rg := range resolveGateways(res, canonical, "", "", "") { + if rg.ingHost == "" { + continue // unresolved ingress FK — validation rejects it + } for _, name := range rg.gw.Services { svc, ok := svcByName[name] if !ok || svc.HealthProbesPort == 0 || svc.Ingress != "" || seen[name] { @@ -2323,11 +2390,14 @@ func resolveGatewayHealthServices(res types.Resources, canonical map[string]stri continue } seen[name] = true + // A private gateway's listed-service health follows it behind the ingress + // (ADR-0045): the public health listener renders on the FRONTING INGRESS + // host, not the gateway host. coLocated is relative to that ingress host. out = append(out, gatewayHealthService{ svc: svc, - gwHost: rg.host, + gwHost: rg.ingHost, svcHost: svcHost, - coLocated: svcHost == rg.host, + coLocated: svcHost == rg.ingHost, gwHealthPort: rg.gw.EffectiveHealthProbesPort(), }) } @@ -2422,10 +2492,11 @@ func ingressCrossHostBackends(svcs []ingressService) map[string]map[string]strin return out } -// ingressHostUnion returns the sorted union of the three ingress-host maps' keys — -// the hosts that have routes, apps, and/or health endpoints — so realization visits -// each host exactly once in a stable order. -func ingressHostUnion(routes map[string][]types.IngressRoute, apps map[string][]types.IngressApp, health map[string][]types.IngressHealth, gateways map[string][]types.IngressGateway) []string { +// ingressHostUnion returns the sorted union of the five host maps' keys — the hosts +// that have routes, apps, health endpoints, gateway termination servers, and/or +// gateway routing servers (ADR-0045) — so realization visits each host exactly once +// in a stable order. +func ingressHostUnion(routes map[string][]types.IngressRoute, apps map[string][]types.IngressApp, health map[string][]types.IngressHealth, gatewayTerms, gatewayRoutes map[string][]types.IngressGateway) []string { set := map[string]bool{} for k := range routes { set[k] = true @@ -2436,7 +2507,10 @@ func ingressHostUnion(routes map[string][]types.IngressRoute, apps map[string][] for k := range health { set[k] = true } - for k := range gateways { + for k := range gatewayTerms { + set[k] = true + } + for k := range gatewayRoutes { set[k] = true } out := make([]string, 0, len(set)) @@ -2560,15 +2634,24 @@ func firewallPlanByHost(res types.Resources, meshPublic bool) map[string]types.F addPublic(ia.ingHost, 443) addPublic(ia.ingHost, 80) } - // The north-south gateway host opens the same public pair: daemons HTTPS in - // on 443, and :80 serves ACME HTTP-01 for the gateway cert. The gateway's - // backends are reached THROUGH the mesh (its own loopback egress → the - // callee's MTLSPort, already covered by the mesh rules) — no private rule. - // Slug/baseDomain don't affect the host resolution the firewall needs, so the - // empty-args resolveGateways call shares the FK resolution with nginx + DNS. + // A private gateway (ADR-0045) is fronted by an ingress: the daemon-facing public + // pair (443 for TLS, 80 for ACME HTTP-01 of the gateway cert) opens on the FRONTING + // INGRESS host, not the gateway host. When the gateway is split onto its own host, + // that host opens the gateway's plain-HTTP routing port (nginx.GatewayHTTPPort) to + // the network CIDR only (the ingress reaches it privately); co-located needs no rule + // (loopback). The gateway's own mesh backends are reached THROUGH the mesh (already + // covered by the mesh MTLSPort rules) — no other rule. Slug/baseDomain don't affect + // host resolution, so the empty-args resolveGateways call shares FK resolution with + // nginx + DNS. for _, rg := range resolveGateways(res, canonical, "", "", "") { - addPublic(rg.host, 443) - addPublic(rg.host, 80) + if rg.ingHost == "" { + continue + } + addPublic(rg.ingHost, 443) + addPublic(rg.ingHost, 80) + if rg.host != rg.ingHost { + addPrivate(rg.host, nginx.GatewayHTTPPort) + } } // The gateway health tier (ADR-0034): the gateway host opens its public health // port when >=1 listed (ingress-less) service declares a backend health port, @@ -2904,11 +2987,16 @@ func derivedRecords(res types.Resources, env, slug, baseDomain, ephemeralSlug st fqdn := naming.AppFQDN(ia.app.Subdomain, slug, baseDomain, ephemeralSlug) dedupAdd(fqdn, ia.app.Container, ia.ingHost) } - // The gateway's FQDN is a grey-cloud A record at its own host (where its nginx - // terminates daemon TLS) — the SAME resolveGateways derivation gatewaysByHost - // feeds the server_name/ACME cert from, so record and cert can never drift. + // The gateway's FQDN is a grey-cloud A record at its FRONTING INGRESS host + // (ADR-0045): the ingress terminates the gateway's daemon TLS and holds its ACME + // cert, so the record points there — the SAME resolveGateways derivation + // gatewayEdgeByHost feeds the termination server_name/ACME from, so record and + // cert can never drift. for _, rg := range resolveGateways(res, canonical, slug, baseDomain, ephemeralSlug) { - dedupAdd(rg.fqdn, rg.gw.Container, rg.host) + if rg.ingHost == "" { + continue + } + dedupAdd(rg.fqdn, rg.gw.Container, rg.ingHost) } // The scope's ingress gets a stable DNS name — `ingress.` (global) / // `ingress..` (regional) — pointing at its host, so consumers can diff --git a/program/ratelimit_test.go b/program/ratelimit_test.go index e38b434..f556e30 100644 --- a/program/ratelimit_test.go +++ b/program/ratelimit_test.go @@ -55,7 +55,7 @@ func TestStampRateLimitIngressOptOut(t *testing.T) { stampRateLimit(enabledSec(), res, routes, apps, gws) assert.Nil(t, routes["h"][0].RateLimit, "route on an opted-out ingress is not limited") assert.Nil(t, apps["h"][0].RateLimit, "app on an opted-out ingress is not limited") - assert.NotNil(t, gws["h"][0].RateLimit, "the gateway is a separate edge and stays limited") + assert.NotNil(t, gws["h"][0].RateLimit, "the gateway's own edge (its ingress) did not opt out, so its termination stays limited") } func TestStampRateLimitGatewayOptOut(t *testing.T) { diff --git a/providers/hetzner/deploykey_test.go b/providers/hetzner/deploykey_test.go index c711ffe..528ea84 100644 --- a/providers/hetzner/deploykey_test.go +++ b/providers/hetzner/deploykey_test.go @@ -26,7 +26,7 @@ func TestRealizeRequiresDeployKey(t *testing.T) { err := h.Realize(ctx, "edge-01", host, "deploy", []types.IngressRoute{{Service: "api", Type: types.IngressTypeTLSTermination, Listen: 443, Target: 8080, Backend: "127.0.0.1", FQDNs: []string{"api.wardnet.network"}}}, - nil, nil, 0, nil, nil, "prd", nil) + nil, nil, 0, nil, nil, nil, nil, "prd", nil) require.Error(t, err) assert.Contains(t, err.Error(), "no deploy private key configured") return nil diff --git a/providers/hetzner/tls.go b/providers/hetzner/tls.go index 12d80e7..8f3c9dc 100644 --- a/providers/hetzner/tls.go +++ b/providers/hetzner/tls.go @@ -57,7 +57,9 @@ func (h *HetznerTLS) Realize( health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, + gatewayRoutes []types.IngressGateway, backendIPs map[string]pulumi.StringOutput, + gatewayIPs map[string]pulumi.StringOutput, env string, dependsOn []pulumi.Resource, ) error { @@ -74,7 +76,7 @@ func (h *HetznerTLS) Realize( return fmt.Errorf("ingress %q: host has no deploy_user; inforge needs one to SSH and realize the ingress proxy", hostKey) } - writeScript, err := h.renderWriteScript(hostKey, routes, apps, health, healthPort, gateways, backendIPs) + writeScript, err := h.renderWriteScript(hostKey, routes, apps, health, healthPort, gateways, gatewayRoutes, backendIPs, gatewayIPs, host.PrivateIP) if err != nil { return fmt.Errorf("ingress %q: %w", hostKey, err) } @@ -130,29 +132,51 @@ func (h *HetznerTLS) Realize( // Backend with the resolved IP before Render — so the upstream addresses are the // real private IPs, resolved at deploy time. In preview a cross-host backend IP is // unknown, so the apply (and the command that consumes it) is skipped entirely. -func (h *HetznerTLS) renderWriteScript(hostKey string, routes []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, backendIPs map[string]pulumi.StringOutput) (pulumi.StringInput, error) { - if len(backendIPs) == 0 { - cfg, err := nginx.Render(routes, apps, health, healthPort, gateways) +func (h *HetznerTLS) renderWriteScript(hostKey string, routes []types.IngressRoute, apps []types.IngressApp, health []types.IngressHealth, healthPort int, gateways []types.IngressGateway, gatewayRoutes []types.IngressGateway, backendIPs map[string]pulumi.StringOutput, gatewayIPs map[string]pulumi.StringOutput, hostPrivateIP pulumi.StringOutput) (pulumi.StringInput, error) { + // Fast path: everything is co-located (routes carry "127.0.0.1", gateway + // termination/routing fields are pre-filled), so the whole config renders + // synchronously with no apply over unknown private IPs. + if len(backendIPs) == 0 && len(gatewayIPs) == 0 { + cfg, err := nginx.Render(routes, apps, health, healthPort, gateways, gatewayRoutes) if err != nil { return nil, err } return pulumi.String(iremote.WriteFileScript(nginx.ConfigPath, cfg)), nil } + // A split gateway (or cross-host route/health) needs its peer host's private IP + // resolved at deploy time. Await backend IPs (keyed by service) and gateway IPs + // (keyed by gateway name) together, then substitute the empty fields before Render. svcNames := make([]string, 0, len(backendIPs)) for n := range backendIPs { svcNames = append(svcNames, n) } sort.Strings(svcNames) - outs := make([]any, len(svcNames)) - for i, n := range svcNames { - outs[i] = backendIPs[n] + gwNames := make([]string, 0, len(gatewayIPs)) + for n := range gatewayIPs { + gwNames = append(gwNames, n) } + sort.Strings(gwNames) + outs := make([]any, 0, len(svcNames)+len(gwNames)+1) + for _, n := range svcNames { + outs = append(outs, backendIPs[n]) + } + for _, n := range gwNames { + outs = append(outs, gatewayIPs[n]) + } + // The gateway routing server (on THIS host, when it is a split gateway host) binds + // this host's own private IP, resolved last. + outs = append(outs, hostPrivateIP) return pulumi.All(outs...).ApplyT(func(args []any) (string, error) { resolved := make(map[string]string, len(svcNames)) for i, n := range svcNames { resolved[n], _ = args[i].(string) } + gwResolved := make(map[string]string, len(gwNames)) + for i, n := range gwNames { + gwResolved[n], _ = args[len(svcNames)+i].(string) + } + hostIP, _ := args[len(svcNames)+len(gwNames)].(string) rendered := make([]types.IngressRoute, len(routes)) for i, r := range routes { if r.Backend == "" { @@ -175,7 +199,40 @@ func (h *HetznerTLS) renderWriteScript(hostKey string, routes []types.IngressRou } renderedHealth[i] = hh } - cfg, err := nginx.Render(rendered, apps, renderedHealth, healthPort, gateways) + // A split gateway's TERMINATION server (on this ingress host) proxies to the + // gateway host's private IP; fill any empty Backend. + renderedGw := make([]types.IngressGateway, len(gateways)) + for i, g := range gateways { + if g.Backend == "" { + ip := gwResolved[g.Name] + if ip == "" { + return "", fmt.Errorf("ingress %q: split gateway %q has no resolved gateway-host private IP", hostKey, g.Name) + } + g.Backend = ip + } + renderedGw[i] = g + } + // A split gateway's ROUTING server (on this gateway host) binds this host's own + // private IP and trusts the ingress host's private IP for real-IP recovery; fill + // any empty ListenAddr / RealIPFrom. + renderedGwRoutes := make([]types.IngressGateway, len(gatewayRoutes)) + for i, g := range gatewayRoutes { + if g.ListenAddr == "" { + if hostIP == "" { + return "", fmt.Errorf("ingress %q: split gateway %q has no resolved gateway-host private IP for its listen address", hostKey, g.Name) + } + g.ListenAddr = hostIP + } + if g.RealIPFrom == "" { + ip := gwResolved[g.Name] + if ip == "" { + return "", fmt.Errorf("ingress %q: split gateway %q has no resolved ingress-host private IP", hostKey, g.Name) + } + g.RealIPFrom = ip + } + renderedGwRoutes[i] = g + } + cfg, err := nginx.Render(rendered, apps, renderedHealth, healthPort, renderedGw, renderedGwRoutes) if err != nil { return "", err } diff --git a/providers/hetzner/tls_test.go b/providers/hetzner/tls_test.go new file mode 100644 index 0000000..8b466ea --- /dev/null +++ b/providers/hetzner/tls_test.go @@ -0,0 +1,66 @@ +package hetzner + +import ( + "sync" + "testing" + + "github.com/pulumi/pulumi/sdk/v3/go/pulumi" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "github.com/wardnet/inforge/internal/nginx" + "github.com/wardnet/inforge/internal/types" +) + +// awaitScript resolves a rendered write-script (a plain string on the co-located +// fast path, a StringOutput on the cross-host apply path) and returns its content. +// ApplyT is asynchronous, so the test must WAIT for it or race the resolution. +func awaitScript(t *testing.T, in pulumi.StringInput) string { + t.Helper() + var got string + var wg sync.WaitGroup + wg.Add(1) + in.ToStringOutput().ApplyT(func(s string) string { got = s; wg.Done(); return s }) + wg.Wait() + return got +} + +// TestRenderWriteScriptGateway exercises both halves of a private gateway (ADR-0045) +// through renderWriteScript: the co-located FAST path (loopback addresses, synchronous) +// and the SPLIT apply path (the termination's Backend, the routing server's own-host +// ListenAddr, and its ingress-trusting RealIPFrom all resolved from private IPs). A +// non-empty result proves the fill logic resolved every address and nginx.Render +// succeeded — Render/renderWriteScript return an error if any address is unresolved. +func TestRenderWriteScriptGateway(t *testing.T) { + fqdn := "api.use1.wardnet.network" + routes := []types.IngressGatewayRoute{{Pattern: "/x/**", Service: "svc"}} + + err := pulumi.RunErr(func(ctx *pulumi.Context) error { + h := NewTLS("deploy-key", "use1") + + // Fast path: co-located gateway — every address is loopback, so the config + // renders synchronously with no apply over unknown private IPs. + coTerm := types.IngressGateway{Name: "api", FQDN: fqdn, Backend: "127.0.0.1", HTTPPort: nginx.GatewayHTTPPort} + coRoute := types.IngressGateway{Name: "api", FQDN: fqdn, HTTPPort: nginx.GatewayHTTPPort, + ListenAddr: "127.0.0.1", RealIPFrom: "127.0.0.1", Routes: routes} + fast, err := h.renderWriteScript("edge-01", nil, nil, nil, 0, + []types.IngressGateway{coTerm}, []types.IngressGateway{coRoute}, nil, nil, + pulumi.String("10.0.0.5").ToStringOutput()) + require.NoError(t, err) + assert.NotEmpty(t, awaitScript(t, fast), "co-located gateway renders synchronously") + + // Split path: the termination's Backend and the routing's RealIPFrom resolve from + // the gateway's peer IP; the routing's ListenAddr resolves from the host's own + // private IP. Empty fields force the apply branch that fills them. + splitTerm := types.IngressGateway{Name: "api", FQDN: fqdn, HTTPPort: nginx.GatewayHTTPPort} + splitRoute := types.IngressGateway{Name: "api", FQDN: fqdn, HTTPPort: nginx.GatewayHTTPPort, Routes: routes} + gwIPs := map[string]pulumi.StringOutput{"api": pulumi.String("10.0.0.9").ToStringOutput()} + split, err := h.renderWriteScript("edge-01", nil, nil, nil, 0, + []types.IngressGateway{splitTerm}, []types.IngressGateway{splitRoute}, nil, gwIPs, + pulumi.String("10.0.0.5").ToStringOutput()) + require.NoError(t, err) + assert.NotEmpty(t, awaitScript(t, split), "split gateway resolves every private IP and renders") + return nil + }, pulumi.WithMocks("project", "stack", &testMocks{})) + require.NoError(t, err) +} diff --git a/schemas/gateway.json b/schemas/gateway.json index 1205689..eb1d5ec 100644 --- a/schemas/gateway.json +++ b/schemas/gateway.json @@ -1 +1 @@ -{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}},"security":{"type":"boolean","description":"Opt this gateway edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} +{"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"gateway.json","title":"Gateway (north-south daemon API gateway) resource","type":"object","additionalProperties":false,"required":["name","container","host","ingress","pki","subdomain","services"],"properties":{"name":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"},"container":{"type":"string","minLength":1,"description":"Logical container/grouping the gateway belongs to (same field as other resources)."},"host":{"type":"string","minLength":1,"description":"Name of the compute resource (in the same scope) this gateway runs on. The gateway reuses the host's provisioning/firewall/SSH and inherits its provider; it declares no provider of its own."},"ingress":{"type":"string","minLength":1,"description":"Name of the ingress resource (in the same scope) that fronts this gateway (ADR-0045). REQUIRED: a gateway is never publicly exposed — the ingress terminates TLS and the edge security tier and reverse-proxies the gateway FQDN to it over the private network, preserving the client IP. This gateway's host and the ingress's host must share a network."},"pki":{"type":"string","minLength":1,"description":"Name of the two-tier (mesh) PKI in pki.enc.yaml the gateway's client leaf (/gateway) mints from. Every listed service must declare the same pki (a callee only trusts callers chaining to its own mesh)."},"subdomain":{"type":"string","minLength":1,"description":"Public subdomain daemons connect to. The fully-qualified domain is composed at realization from the scope and base domain."},"services":{"type":"array","minItems":1,"description":"names of the services (same scope) exposed at the internet edge. The gateway's routing table is DERIVED from each listed service's mesh.public_paths (ADR-0034); a request matching no public glob is answered 404 (JSON) at the edge. Each listed service must permit the gateway (list \"gateway\" in its mesh.allowed_services) and declare at least one public path.","items":{"type":"string","pattern":"^[a-z][a-z0-9_-]*$"}},"health_probes_port":{"type":"integer","minimum":1,"maximum":65535,"description":"public port the gateway host exposes its listed services' health checks on (plain HTTP, Host-demuxed by service FQDN). Defaults to 81. Opened only when a listed service declares its own health_probes_port."},"health_probe_paths":{"type":"array","minItems":1,"description":"exact paths on the gateway's own 443 server that nginx answers 200 \"ok\" directly — edge liveness over the real daemon path (DNS+cert+TLS). Must not overlap any listed service's public_paths.","items":{"type":"string","pattern":"^/"}},"security":{"type":"boolean","description":"Opt this gateway edge out of the env-level security tier (ADR-0043) when false: no rate limiting (and no CrowdSec). Absent = the env security policy applies."}}} diff --git a/website/docs/configuration/variables-yaml.md b/website/docs/configuration/variables-yaml.md index aba80d1..6cc055c 100644 --- a/website/docs/configuration/variables-yaml.md +++ b/website/docs/configuration/variables-yaml.md @@ -21,9 +21,10 @@ observability: # optional — Grafana Cloud integration otlp_endpoint: https://otlp-gateway-....grafana.net/otlp # host/DB metrics collector grafana_url: https://myorg.grafana.net # dashboards + alerts target -security: # optional — edge security tier (public ingress + gateway hosts) +security: # optional — edge security tier (public ingress hosts; a gateway is + # fronted by an ingress and enforced there, ADR-0045) crowdsec: - enabled: true # IP banning (nftables) + free community blocklist on edges + enabled: true # IP banning (nftables) + free community blocklist on ingress edges rate_limit: enabled: true # one blanket IP-based limit on every public edge server requests_per_second: 20 diff --git a/website/docs/resources/gateway.md b/website/docs/resources/gateway.md index 25f733d..ca05a2c 100644 --- a/website/docs/resources/gateway.md +++ b/website/docs/resources/gateway.md @@ -4,30 +4,37 @@ sidebar_position: 8 # Gateway -The **gateway** is the north-south public edge external **daemons** HTTPS into — the API front door -of an environment. It TLS-terminates the daemon connection on its own public FQDN, matches the -request path against a **derived routing table** — the gateway lists which `services:` are public; -each listed service declares *what* is public via its -[`mesh.public_paths`](./service#path-level-exposure) — and hands the request to the owning service -**through the east-west mesh** (it is a mesh client with identity `/gateway`). It holds no -service locations, does not validate the daemon's JWT (it forwards it — the service validates), and -never rewrites the request path. - -It is distinct from the [ingress](./ingress) (which fronts apps and per-service web/SNI routes) and -from the mesh itself (service↔service traffic never passes through the gateway). +The **gateway** is the north-south API front door for external **daemons** — but it is **private**, +fronted by an [ingress](./ingress) (ADR-0045). A daemon connects to the gateway's public FQDN, which +resolves to the **ingress host**; the ingress TLS-terminates, enforces the edge +[security tier](/configuration/variables-yaml#security) (CrowdSec + rate limiting), and reverse-proxies +to the gateway over the private network, preserving the client IP. The gateway itself opens **no +public port** and holds **no ACME cert**. It matches the request path against a **derived routing +table** — the gateway lists which `services:` are public; each listed service declares *what* is +public via its [`mesh.public_paths`](./service#path-level-exposure) — and hands the request to the +owning service **through the east-west mesh** (it is a mesh client with identity `/gateway`). +It holds no service locations, does not validate the daemon's JWT (it forwards it — the service +validates), and never rewrites the request path. + +Because a gateway is never a public edge, the security tier applies at its **ingress**, not on the +gateway host. It is distinct from the mesh itself (service↔service traffic never passes through the +gateway). ```yaml title="regional/gateway/api/manifest.yaml" name: api container: edge -host: bridge # compute host (same scope) the gateway nginx runs on +host: bridge # compute host (same scope) the gateway routing nginx runs on +ingress: web # REQUIRED — the ingress (same scope) that fronts this gateway (ADR-0045); + # it terminates TLS + security and reverse-proxies here. host and the + # ingress's host must share a network. pki: wardnet-mesh # the mesh the gateway joins as a CLIENT (must match its listed services) -subdomain: api # -> api.. (regional) / api. (global) +subdomain: api # -> api.. (regional) / api. (global); resolves to the ingress host services: # the services exposed at the edge — routing is DERIVED from - ddns # each service's mesh.public_paths (one location per glob) - tunneller -health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" on the 443 server +health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" (reached through the ingress) - /livez -# security: false # optional — opt this edge out of the env security tier (rate limiting) +# security: false # optional — opt this gateway's traffic out of rate limiting at the ingress ``` ## Fields @@ -36,16 +43,17 @@ health_probe_paths: # optional — edge liveness: nginx answers 200 "ok" o |-------|------|----------|-------------| | `name` | string | Yes | Gateway resource name (unique per scope). | | `container` | string | Yes | Logical container/grouping, like every resource. | -| `host` | string | Yes | **Name** of the compute resource (same scope) the gateway runs on. Single-instance; the gateway reuses the host's provisioning/firewall/SSH and inherits its provider. | +| `host` | string | Yes | **Name** of the compute resource (same scope) the gateway's plain-HTTP routing nginx runs on. Single-instance; the gateway reuses the host's provisioning/firewall/SSH and inherits its provider. It opens no public port — its routing server is reached only through the fronting ingress. | +| `ingress` | string | **Yes** | **Name** of the [ingress](./ingress) (same scope) that fronts this gateway (ADR-0045). Required — a gateway is never publicly exposed: the ingress terminates TLS and the security tier and reverse-proxies the gateway FQDN here over the private network, preserving the client IP. This gateway's `host` and the ingress's host must **share a network** (co-located is the trivial case). | | `pki` | string | Yes | Name of the **two-tier (mesh) PKI** in `pki.enc.yaml` the gateway's client leaf (`CN=/gateway`) mints from. Every listed service must declare the **same** `pki:` — a callee only trusts callers chaining to its own mesh. | | `subdomain` | string | Yes | Public subdomain daemons connect to. The FQDN is the flat app-style form: `..` regional, `.` global (an ephemeral env inserts its slug). Must not collide with an app subdomain in the scope, and must not be the reserved `ingress` label (the scope's [ingress host record](./ingress#the-ingress-dns-name)). | | `services` | array | Yes (min 1) | The services (same scope) exposed at the internet edge. The routing table is **derived**: one nginx regex location per (listed service, [`mesh.public_paths`](./service#path-level-exposure) glob), target named in `X-Mesh-Target`. A request matching no public glob is answered at the edge with an `application/json` `404` body `{"error":"not_found"}` — it never traverses to any service. | -| `health_probes_port` | int | No | Public **plain-HTTP** port the gateway host exposes its listed services' health checks on (default `81`), demuxed by request `Host` (the service's FQDN) — the gateway twin of the ingress [health port](./ingress#health-probes). Opened to the internet only when a listed service declares its own [`health_probes_port`](./service#health-probes). Must not be `80` or `443`, and must **match** a co-hosted ingress's health port (one public health port per host). | -| `health_probe_paths` | array | No | Exact paths on the gateway's **own `443` server** that nginx answers `200 "ok"` directly — edge liveness proving the real daemon path (DNS + cert + TLS) without touching any backend. `inforge validate` rejects a path that any listed service's public glob claims. | -| `security` | bool | No | Set `false` to opt this gateway out of the env-level [security tier](/configuration/variables-yaml#security) — its mesh-route locations get no rate limiting. Absent/`true` = the env policy applies. Health-probe locations are never rate-limited regardless. | +| `health_probes_port` | int | No | Public **plain-HTTP** port the listed services' health checks are exposed on (default `81`), demuxed by request `Host` (the service's FQDN). Because the gateway is private, this listener renders on the **fronting ingress host** (ADR-0045), not the gateway host. Opened only when a listed service declares its own [`health_probes_port`](./service#health-probes). Must not be `80` or `443`, and must **match** the fronting ingress's health port (one public health port per host). | +| `health_probe_paths` | array | No | Exact paths the gateway answers `200 "ok"` directly — edge liveness over the real public path (reached **through the ingress**), proving DNS + cert + termination + the gateway routing server without touching any backend. `inforge validate` rejects a path that any listed service's public glob claims. | +| `security` | bool | No | Set `false` to opt this gateway's traffic out of the env-level [security tier](/configuration/variables-yaml#security)'s rate limiting at the fronting ingress. A gateway is never a public edge, so this no longer affects CrowdSec. Absent/`true` = the env policy applies. Health-probe locations are never rate-limited regardless. | -**A gateway is a scope singleton** — at most one per scope: it is the scope's one public daemon -edge. It may only list services in the **same scope**. +**A gateway is a scope singleton** — at most one per scope: it is the scope's one daemon API front +door. It may only list services in the **same scope**, and must be fronted by the scope's ingress. **The two sides must agree**: a listed service must include `gateway` in its [`mesh.allowed_services`](./service#east-west-service-mesh) — otherwise the callee's mesh proxy @@ -61,8 +69,11 @@ identity). ```mermaid flowchart TB daemon["daemon
(JWT + Ed25519 PoP)"] - subgraph gwhost ["gateway host"] - gwnginx["public nginx — TLS terminates (ACME)
regex location per public glob → X-Mesh-Target: ddns
no match → JSON 404 at the edge
path preserved · Authorization forwarded · XFF stamped"] + subgraph inghost ["ingress host (the public edge)"] + term["ingress nginx — TLS terminates (ACME)
CrowdSec + rate limit
blind-proxy the FQDN · append real client to XFF"] + end + subgraph gwhost ["gateway host (private)"] + gwnginx["routing nginx (plain HTTP)
recover client IP from XFF (trust the ingress)
regex location per public glob → X-Mesh-Target: ddns
no match → JSON 404 · path preserved · Authorization forwarded"] gwmesh["mesh proxy (egress)
presents leaf CN=use1/gateway"] gwnginx -- "plain HTTP, loopback :9756" --> gwmesh end @@ -71,13 +82,18 @@ flowchart TB svc["service
(validates the forwarded JWT)"] svcmesh -- "plain HTTP, loopback :<mesh.port>" --> svc end - daemon -- "HTTPS api.use1.example.com/ddns/api/foo" --> gwnginx + daemon -- "HTTPS api.use1.example.com/ddns/api/foo" --> term + term -- "plain HTTP :8442 (private net;
loopback when co-located)" --> gwnginx gwmesh -- "mTLS :8443 (private net;
public only at the global mesh gateway)" --> svcmesh ``` +- **Private gateway**: the FQDN resolves to the **ingress host**, which terminates TLS and the + security tier and reverse-proxies to the gateway. The gateway opens no public port; the client IP + is recovered from the ingress-stamped `X-Forwarded-For` (trusting only the ingress, so a + client-forged entry is stripped) — the service still sees the real daemon IP. - **Path-preserving**: the daemon signs `/ddns/api/foo` (Ed25519 PoP covers the path); the service receives `/ddns/api/foo` byte-for-byte. Author `public_paths` globs against the same absolute - paths the service serves — the gateway never strips or rewrites anything. + paths the service serves — neither the ingress nor the gateway strips or rewrites anything. - **Fail-closed at the edge**: a request matching no listed service's public glob gets an `application/json` `404` body `{"error":"not_found"}` from the gateway nginx itself — zero backend traffic. An undeclared endpoint is unreachable from the internet until its path is added @@ -88,90 +104,96 @@ flowchart TB ## Topology shapes -The gateway's `host:` decides the shape. Both are the same three logical hops — the difference is -only whether the public nginx is shared. +Whether the gateway shares its ingress's host decides the shape. Either way the gateway is private +and the ingress is the public edge — the difference is only whether the ingress→gateway hop is +loopback or crosses the private network. -### Co-located with the ingress (one host, one public nginx) +### Co-located with the ingress (one host, one nginx) -Authoring the gateway on the **same host** as an ingress merges them: apps, service routes, and the -gateway are server blocks of the **one** public nginx on that host (they share the ACME issuer and, -when a `forward` route shares `:443`, the same `ssl_preread` demux). Zero extra processes or hops — -the cheapest shape, right for small environments. +Authoring the gateway on the **same host** as its ingress renders both halves on the **one** public +nginx there: apps, service routes, the gateway's TLS **termination** server (which blind-proxies over +loopback to the gateway's plain-HTTP **routing** server), and the routing server itself. Zero extra +servers — the cheapest shape, right for small environments. The gateway is still private: it never +gets a public listener of its own; only the ingress's `:443`/`:80` are public. ```mermaid flowchart TB - subgraph bridge ["bridge host"] + subgraph bridge ["bridge host (gateway.host == ingress.host)"] direction TB subgraph nginx1 ["public nginx (north-south, :443/:80)"] app["app · my.use1.example.com
(static files)"] route["route · api.svc.…
(web/SNI → 127.0.0.1:8080)"] - gw["GATEWAY · api.use1.example.com
(daemon edge)"] + term["GATEWAY termination · api.use1.example.com
(TLS + security → loopback :8442)"] + gwroute["GATEWAY routing (plain HTTP :8442)
real_ip recover → X-Mesh-Target"] end mesh["mesh proxy (east-west, private :8443)"] svcs["services (tenants, ddns, …)"] - gw -- "loopback :9756" --> mesh + term -- "loopback :8442" --> gwroute + gwroute -- "loopback :9756" --> mesh mesh -- "loopback :<mesh.port>" --> svcs end peers(("other hosts")) mesh <-- "mTLS" --> peers ``` -### Own host (isolated edge) +### Split (gateway on its own private host) -Authoring a dedicated `host:` gives the gateway its own public nginx and its own (egress-only) mesh -proxy. The daemon edge is isolated from the app/web tier — a compromise or overload of one does not -touch the other — at the cost of one more server. Traffic still crosses the same three hops; the -mesh hop just always leaves the host. +Authoring the gateway on a **different host** from its ingress (same network) isolates the daemon +routing tier: the ingress host stays the sole public edge (it runs the gateway's termination server), +while the gateway host is **private** — it runs only the plain-HTTP routing server (reached over the +network on `:8442`, opened to the network CIDR by the firewall) and its own egress-only mesh proxy. +A compromise or overload of the app/web tier does not touch the daemon tier, at the cost of one more +server. The client IP survives the extra hop via the ingress-stamped XFF the routing server recovers. ```mermaid flowchart LR - subgraph edge ["edge host (dedicated)"] + subgraph bridge ["ingress host (public edge)"] direction TB - gwnginx["public nginx
GATEWAY · api.use1.…"] - gwmesh["mesh proxy
(egress-only)"] - gwnginx -- "loopback :9756" --> gwmesh + appnginx["public nginx (:443/:80)
apps + web routes +
GATEWAY termination (TLS + security)"] end - subgraph bridge ["bridge host"] + subgraph edge ["gateway host (private)"] direction TB - appnginx["public nginx
(apps + web routes)"] - mesh["mesh proxy
(private :8443)"] - svcs["services
(tenants, ddns, …)"] - mesh -- "loopback :<mesh.port>" --> svcs + gwnginx["routing nginx (plain HTTP :8442)
real_ip recover → X-Mesh-Target"] + gwmesh["mesh proxy (egress-only)"] + gwnginx -- "loopback :9756" --> gwmesh end - subgraph global ["global host"] - gmesh["mesh proxy
(the global mesh gateway, public :8443)"] + subgraph svchost ["service host"] + mesh["mesh proxy (private :8443)"] + svcs["services (tenants, ddns, …)"] + mesh -- "loopback :<mesh.port>" --> svcs end + appnginx -- "plain HTTP :8442 (private net)" --> gwnginx gwmesh -- "mTLS (private net)" --> mesh - gwmesh -. "cross-scope: regional → global
mTLS over the internet" .-> gmesh ``` ## Health -The gateway participates in the health tier on both sides: +The gateway participates in the health tier on both sides — but because the gateway is private, the +public listeners render on the **fronting ingress host** (ADR-0045): - **Listed services' health** — when a listed service declares its own [`health_probes_port`](./service#health-probes) and has **no `ingress:`**, the Host-demuxed - plain-HTTP health server renders on the **gateway's** host: the gateway exposes its + plain-HTTP health server renders on the gateway's **fronting ingress host**: it exposes the `health_probes_port` (default `81`) publicly and reverse-proxies each probe to the right backend, demuxed by the service's `.svc` FQDN, exactly like the [ingress health tier](./ingress#health-probes). Only the service's declared [`health_probe_paths`](./service#health-probes) are proxied — anything else is `404`. A service - with **both** an ingress and a gateway listing keeps its health at the **ingress** (one canonical - health address). A listed service **co-located on the gateway host** must keep its backend - `health_probes_port` off the gateway's public binds — the gateway's health port, `443` (daemon - TLS), and `80` (ACME) — `inforge validate` rejects the collision. + with **both** an ingress and a gateway listing keeps its health at that ingress (one canonical + health address). The health port must **match** the fronting ingress's health port (one public + health listener per host); `inforge validate` rejects a mismatch or a collision with a route port. - **The gateway's own liveness** — `health_probe_paths` on the gateway spec are exact paths the - **`443` server itself** answers with `200 "ok"`, no backend involved. Because the probe rides the - real daemon path (DNS resolution, the ACME cert, TLS termination), a green probe proves the edge - end to end. A liveness path claimed by a listed service's public glob is a validation error. + gateway's **routing server** answers with `200 "ok"`, no backend involved, reached **through the + ingress** over the real public path. Because the probe rides DNS resolution → the ACME cert on the + ingress → TLS termination → the private hop → the routing server, a green probe proves the edge end + to end. A liveness path claimed by a listed service's public glob is a validation error. ## Certificates and renewal -The gateway's public cert is ACME (Let's Encrypt) on its FQDN, exactly like an app. Its **mesh -client leaf** (`CN=/gateway`, minted from its `pki:`) is delivered like every mesh leaf: -SSH-pushed into the host's `leaf.age` by `inforge deploy` (baseline) and -[`inforge pki renew`](/cli/pki), which also reload-or-restarts the mesh proxy. Nothing to run -per gateway. +The gateway's public cert is ACME (Let's Encrypt) on its FQDN, issued and held by the **fronting +ingress** (which terminates its TLS) — the gateway itself has no cert. Its **mesh client leaf** +(`CN=/gateway`, minted from its `pki:`) is delivered like every mesh leaf: SSH-pushed into the +gateway host's `leaf.age` by `inforge deploy` (baseline) and [`inforge pki renew`](/cli/pki), which +also reload-or-restarts the mesh proxy. Nothing to run per gateway. ## See also diff --git a/website/docs/resources/ingress.md b/website/docs/resources/ingress.md index 26735b1..52f6c98 100644 --- a/website/docs/resources/ingress.md +++ b/website/docs/resources/ingress.md @@ -11,9 +11,10 @@ like `service.host`) and reuses that host's provisioning, firewall, cloud-init, **no provider** of its own — it inherits its host's. The ingress is one of two **north-south** (external-facing) tiers. Its sibling is the -[Gateway](./gateway) — the public edge external **daemons** HTTPS into, which routes each path to a -service through the east-west mesh. Both are public nginx server blocks and can share one host (they -merge into a single nginx there); the ingress fronts apps/web, the gateway fronts the daemon API. +[Gateway](./gateway) — the daemon API front door, which routes each path to a service through the +east-west mesh. A gateway is **private, fronted by an ingress** (ADR-0045): this ingress terminates +its TLS and the security tier and reverse-proxies to it. The ingress fronts apps/web (and any +gateway); the gateway holds the daemon routing table. An ingress lives in a folder under `regional/ingress//` (or `global/ingress//`):