From d560444adaa2fe96c8680818b9de649fec848320 Mon Sep 17 00:00:00 2001 From: Timo Derstappen Date: Mon, 19 Oct 2026 20:44:32 +0200 Subject: [PATCH 1/3] chore(agentgateway): run 2.5.0, the stable of 2.5.0-rc.4 agentgateway 2.5.0 is the stable of 2.5.0-rc.4 on the same commit: the 2.5.0-rc.2 the line ran, plus every address of the gateway counted local for a tcp rule and a policy denial answered 403 naming the destination and where it is allowed. A stable Substrate runs a stable agentgateway. --- .circleci/config.yml | 2 +- FORK.md | 2 +- charts/substrate/values.yaml | 2 +- docs/upgrade.md | 9 +++++---- .../components/agentgateway/kustomization.yaml | 4 ++-- 5 files changed, 10 insertions(+), 9 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 67b1f56367..bb848075cb 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -47,7 +47,7 @@ parameters: images. Moves only together with the chart default, and only with the pin (FORK.md, "Re-pin"); never bumped by Renovate on its own. type: string - default: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0-rc.2 + default: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0 jobs: release-tag: diff --git a/FORK.md b/FORK.md index 43f2825cd0..295002c584 100644 --- a/FORK.md +++ b/FORK.md @@ -30,7 +30,7 @@ reasoning instead. |---|---| | Upstream tag | **v0.5.0-alpha2** (2026-10-08; tag commit `9c4b1fb5`, "Let every Helm chart container set resource requests and limits (#54)"; 109 commits past v0.4.0-alpha1, which is not its ancestor, so the re-pin replays every carried commit). Previous pins: v0.0.26 (2026-09-10 → 2026-09-14, releases `v0.0.27-gs.1`–`gs.9`), v0.0.29 (2026-09-14 → 2026-09-22, releases `v0.0.30-gs.1`–`gs.5`, `1.0.0`, `1.0.1`; `release-1.0` stays on it), v0.2.0-beta4 (2026-09-22 → 2026-09-24, dev builds only), v0.2.0-beta5 (2026-09-24 → 2026-10-05, releases `1.1.0`–`1.3.0` and the `1.4.0` candidates; `1.4.0` is cut from `81abb52e`, the last candidate before the re-pin), v0.3.0-alpha3 (2026-10-05 → 2026-10-08, releases `1.4.0`–`1.5.x`), v0.4.0-alpha1 (2026-10-08 → 2026-10-12, releases `1.6.0`–`1.8.x`) | | Why this one | upstream kagent-dev/kagent pins `github.com/kagent-dev/substrate v0.5.0-alpha2` in `go/go.mod` (the `replace` of `github.com/agent-substrate/substrate`) since kagent-dev/kagent#3116 (`5eb9b633`, "Bump Substrate to 0.5.0-alpha2"), and at `main` `92dd6caa` (2026-10-11), the head the kagent line re-pins onto; `giantswarm/kagent-upstream` replaces the same path with this line's release. Since kagent-dev/kagent#2708 kagent sets up its PostgreSQL through `pkg/postgressetup`, which v0.5.0-alpha2 exports. v0.5.0-alpha2 changes the contracts between the components: `SetWorkerCapacity` is `RegisterWorker` on `WorkerService` and `AteomSupport`, with the worker's `HardwareIdentity` (agent-substrate/substrate#2282); `MintActorJWTRequest.actor_uid` is gone (#2207); `SnapshotConfig.on_pause` is gone and a pause captures the `on_commit` scope (#2309); an actor's node is `ActorStatus.assigned_node`, and `LocalSnapshot.content_scope` moves onto field 2, where the line's node list was; the control API answers through `internal/apierror`. The persisted Actor rows of a 1.8 release are migrated on read (see its ledger file), the PostgreSQL schema gains nothing, so the move upgrades a 1.8.x installation in place. During the roll an ateom of the old release retries its capacity report against the renamed RPC until its pool rolls; the workers it registered keep their capacity. The ate-api gRPC contract between kagent's client and Substrate's server must match, so the two lines move together (2026-09-22 onto beta4, 2026-09-24 onto beta5, 2026-10-05 onto alpha3, 2026-10-08 onto v0.4.0-alpha1, 2026-10-12 onto v0.5.0-alpha2). | -| agentgateway it runs | the `agentgateway-image` parameter of `.circleci/config.yml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema) and to the image, by digest, of `manifests/ate-install/components/agentgateway`, which the e2e installs (`hack/verify/agentgateway-image.sh` fails a pull request on any drift; before the 2026-10-08 re-pin that component ran upstream's alpha build, so the e2e never ran the line's agentgateway): a release of the agentgateway line, published by that line from CircleCI under its own gsoci path and versioned on its own — **`gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0-rc.2`** (tag `v2.5.0-rc.2` on `46019685` = 2.5.0-rc.1 + the Substrate egress applying a path-prefixed credential header to a path with an encoded slash (`%2F` read as a separator yields no dot or empty segment), giantswarm/agentgateway-upstream#106 for giantswarm/agentgateway-upstream#105, the rule this line's `egresspolicy.ReplaceHeadersFor` follows since #483; pinned here by #485; before it 2.5.0-rc.1, tag `v2.5.0-rc.1` on `76f469bc` = 2.4.0 + the Substrate egress relaying TCP a `tcp` egress rule allows, decided on the address and port the actor connected to, giantswarm/agentgateway-upstream#101 for giantswarm/substrate#164, and the agentgateway line's carried fixes since: a stateful MCP session's pin to a gone endpoint dropped, test isolation of the AWS SDK config and the proxy test configs; pinned here by the pull request that adds the `tcp` rule; before it 2.4.0, tag `v2.4.0` on `d117e601`, the stable of `v2.4.0-rc.2` on the same commit = 2.3.6 + the Substrate egress injecting an `actor_jwt` entry of `replace_headers` with a JWT minted through `Control.MintActorJWT`, giantswarm/agentgateway-upstream#80 for giantswarm/agentgateway-upstream#79 and #82, which calls the mint by the method name Substrate serves (2.4.0-rc.1 asked for `MintActorJwt` and was answered `Unimplemented`), the agentgateway dataplane's half of the actor-JWT injection this line carries (its ledger file); pinned here as the stable 2.4.0 by #429, after #421 pinned its release candidate; before it 2.3.6, tag `v2.3.6` on `df3072c3` = 2.3.5 + a request the upstream refused before processing it (a refused HTTP/2 stream, or one above the upstream's `GOAWAY`) retried once on a fresh connection, giantswarm/agentgateway-upstream#78 for #401, without which an actor's `git-upload-pack` to github.com was answered `503` when a pooled connection the upstream was closing refused its stream, and the JWKS refetch awaiting the refresh in flight; proven as the egress of 1.8.5 in agentlab (`skills-test`, `agents-test`); pinned here by #411; before it 2.3.5, tag `v2.3.5` on `d5987a97` = 2.3.0 + the agentgateway line's carried patches since: the JWKS refetch on an unknown key id, the gRPC status of a trailers-only refusal logged, Go 1.27.2 and `golang.org/x/net` v0.61.0, and the Substrate egress applying a `replace_headers` entry by its `path_prefix` (giantswarm/agentgateway-upstream#65 for giantswarm/agentgateway-upstream#62), which the path-scoped credentials of this line's egress policy need on the agentgateway dataplane; pinned here by #355; before it 2.3.0, tag `v2.3.0` on `97b142d6` = the agentgateway line's 2026-10-05 re-pin onto upstream v1.6.0 (`ea560864`) plus its 40 carried commits; v1.6.0 contains agentgateway#3677 (`8d6be5fb`), the egress gateway authorizing from the `ateom-for-actor` SPIFFE URI of agent-substrate/substrate#1809 that alpha3's ateom presents and 2.2.2 refuses, and agentgateway#3689 (`c4143926`), the squash of `50999825`, the build upstream's v0.4.0-alpha1 chart pins as `ghcr.io/kagent-dev/substrate/agentgateway:50999825cb55` (alpha1 commit `50e20d5e`), so the v0.4.0-alpha1 re-pin needs no new agentgateway release; pinned here by `a07cc154`; before it 2.2.2, tag `v2.2.2` on `149d9b40` = 2.2.1 + an egress policy call that never reached its policy service answered `503 unavailable` instead of `403 denied`, giantswarm/agentgateway-upstream#35, − the named buffer error on an MCP upstream's response, reverted by giantswarm/agentgateway-upstream#36; 2.2.1 is tag `v2.2.1` on `13325b82` = 2.2.0 + a resume refusal the control plane marks not resumable answered at once with its own status, giantswarm/agentgateway-upstream#29, which the golden refusal of #80 needs; 2.2.0 is tag `v2.2.0` on `71f1adf6`, the agentgateway line's 2026-09-24 re-pin onto upstream `main` @ `3528a428` plus its carried patches and the grpc pin; before it `2.1.2`, tag `v2.1.2` on `39023841`), which descends from the dev build the 2026-09-22 re-pin chose (`0.0.0-dev.giantswarm.2026-09-22.11-31-30.h6afcce9` = upstream agentgateway `main` @ `08f4563b`, 2026-09-21, ≥ `8dba3989`: #3411's egress credential injection and #3524's provider contract, which beta4's and beta5's charts pin as `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.8dba3989`) plus the line's carried patches; the agentgateway line moved to `main` @ `3528a428` on 2026-09-24 without a release yet, so this line ran 2.1.2 until that release, `2.2.0`, existed (a released build, as the platform's image policy requires). Since 2.1.1 a file resource is read from disk on every managed fetch, so an egress CA switch reloads `tls.crt` and `tls.key` as one pair: the dev build and upstream's alpha reload them as two events and serve leaves signed with the other file's key in between, which fails every handshake of that window. Since 2.1.2 the files a parse reads are watched as well. `release-1.0` pins the same release. v0.5.0-alpha2's chart pins upstream's nightly `cr.agentgateway.dev/agentgateway:v0.0.0-alpha.168b05d6` (agent-substrate/substrate#2158) for the TLS passthrough listener and the credential providers on the HTTP and HTTPS egress policies, which the line's chart has rendered for 2.3.0 since before the pin, so the line stays on 2.3.0. | +| agentgateway it runs | the `agentgateway-image` parameter of `.circleci/config.yml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema) and to the image, by digest, of `manifests/ate-install/components/agentgateway`, which the e2e installs (`hack/verify/agentgateway-image.sh` fails a pull request on any drift; before the 2026-10-08 re-pin that component ran upstream's alpha build, so the e2e never ran the line's agentgateway): a release of the agentgateway line, published by that line from CircleCI under its own gsoci path and versioned on its own — **`gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0`** (tag `v2.5.0` on `913dbe92`, the stable of `v2.5.0-rc.4` on the same commit = 2.5.0-rc.2 + every address of the gateway counted local for a `tcp` rule, giantswarm/agentgateway-upstream#108 (`v2.5.0-rc.3`), and a policy denial answered `403` with a status line and body naming the destination and where it is allowed, giantswarm/agentgateway-upstream#109; pinned here by PRNUM; before it 2.5.0-rc.2, tag `v2.5.0-rc.2` on `46019685` = 2.5.0-rc.1 + the Substrate egress applying a path-prefixed credential header to a path with an encoded slash (`%2F` read as a separator yields no dot or empty segment), giantswarm/agentgateway-upstream#106 for giantswarm/agentgateway-upstream#105, the rule this line's `egresspolicy.ReplaceHeadersFor` follows since #483; pinned here by #485; before it 2.5.0-rc.1, tag `v2.5.0-rc.1` on `76f469bc` = 2.4.0 + the Substrate egress relaying TCP a `tcp` egress rule allows, decided on the address and port the actor connected to, giantswarm/agentgateway-upstream#101 for giantswarm/substrate#164, and the agentgateway line's carried fixes since: a stateful MCP session's pin to a gone endpoint dropped, test isolation of the AWS SDK config and the proxy test configs; pinned here by the pull request that adds the `tcp` rule; before it 2.4.0, tag `v2.4.0` on `d117e601`, the stable of `v2.4.0-rc.2` on the same commit = 2.3.6 + the Substrate egress injecting an `actor_jwt` entry of `replace_headers` with a JWT minted through `Control.MintActorJWT`, giantswarm/agentgateway-upstream#80 for giantswarm/agentgateway-upstream#79 and #82, which calls the mint by the method name Substrate serves (2.4.0-rc.1 asked for `MintActorJwt` and was answered `Unimplemented`), the agentgateway dataplane's half of the actor-JWT injection this line carries (its ledger file); pinned here as the stable 2.4.0 by #429, after #421 pinned its release candidate; before it 2.3.6, tag `v2.3.6` on `df3072c3` = 2.3.5 + a request the upstream refused before processing it (a refused HTTP/2 stream, or one above the upstream's `GOAWAY`) retried once on a fresh connection, giantswarm/agentgateway-upstream#78 for #401, without which an actor's `git-upload-pack` to github.com was answered `503` when a pooled connection the upstream was closing refused its stream, and the JWKS refetch awaiting the refresh in flight; proven as the egress of 1.8.5 in agentlab (`skills-test`, `agents-test`); pinned here by #411; before it 2.3.5, tag `v2.3.5` on `d5987a97` = 2.3.0 + the agentgateway line's carried patches since: the JWKS refetch on an unknown key id, the gRPC status of a trailers-only refusal logged, Go 1.27.2 and `golang.org/x/net` v0.61.0, and the Substrate egress applying a `replace_headers` entry by its `path_prefix` (giantswarm/agentgateway-upstream#65 for giantswarm/agentgateway-upstream#62), which the path-scoped credentials of this line's egress policy need on the agentgateway dataplane; pinned here by #355; before it 2.3.0, tag `v2.3.0` on `97b142d6` = the agentgateway line's 2026-10-05 re-pin onto upstream v1.6.0 (`ea560864`) plus its 40 carried commits; v1.6.0 contains agentgateway#3677 (`8d6be5fb`), the egress gateway authorizing from the `ateom-for-actor` SPIFFE URI of agent-substrate/substrate#1809 that alpha3's ateom presents and 2.2.2 refuses, and agentgateway#3689 (`c4143926`), the squash of `50999825`, the build upstream's v0.4.0-alpha1 chart pins as `ghcr.io/kagent-dev/substrate/agentgateway:50999825cb55` (alpha1 commit `50e20d5e`), so the v0.4.0-alpha1 re-pin needs no new agentgateway release; pinned here by `a07cc154`; before it 2.2.2, tag `v2.2.2` on `149d9b40` = 2.2.1 + an egress policy call that never reached its policy service answered `503 unavailable` instead of `403 denied`, giantswarm/agentgateway-upstream#35, − the named buffer error on an MCP upstream's response, reverted by giantswarm/agentgateway-upstream#36; 2.2.1 is tag `v2.2.1` on `13325b82` = 2.2.0 + a resume refusal the control plane marks not resumable answered at once with its own status, giantswarm/agentgateway-upstream#29, which the golden refusal of #80 needs; 2.2.0 is tag `v2.2.0` on `71f1adf6`, the agentgateway line's 2026-09-24 re-pin onto upstream `main` @ `3528a428` plus its carried patches and the grpc pin; before it `2.1.2`, tag `v2.1.2` on `39023841`), which descends from the dev build the 2026-09-22 re-pin chose (`0.0.0-dev.giantswarm.2026-09-22.11-31-30.h6afcce9` = upstream agentgateway `main` @ `08f4563b`, 2026-09-21, ≥ `8dba3989`: #3411's egress credential injection and #3524's provider contract, which beta4's and beta5's charts pin as `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.8dba3989`) plus the line's carried patches; the agentgateway line moved to `main` @ `3528a428` on 2026-09-24 without a release yet, so this line ran 2.1.2 until that release, `2.2.0`, existed (a released build, as the platform's image policy requires). Since 2.1.1 a file resource is read from disk on every managed fetch, so an egress CA switch reloads `tls.crt` and `tls.key` as one pair: the dev build and upstream's alpha reload them as two events and serve leaves signed with the other file's key in between, which fails every handshake of that window. Since 2.1.2 the files a parse reads are watched as well. `release-1.0` pins the same release. v0.5.0-alpha2's chart pins upstream's nightly `cr.agentgateway.dev/agentgateway:v0.0.0-alpha.168b05d6` (agent-substrate/substrate#2158) for the TLS passthrough listener and the credential providers on the HTTP and HTTPS egress policies, which the line's chart has rendered for 2.3.0 since before the pin, so the line stays on 2.3.0. | | When it moves | only together with kagent's pin, proven in agentlab first (`agentlab configure --defaults --chart-branch poc/kagent-main && agentlab up` and the proofs) — see "Re-pin". Not on a schedule. | | Derived how | the nearest upstream `v*` tag reachable from the line, prereleases included (`git tag --merged` against `git ls-remote --tags upstream`, nearest by commit distance — a version sort would place a prerelease behind its release): the line's own tags version the package (see "Versions") and share the `v*` namespace, so they never count — and upstream `main`, rebased onto agent-substrate, has no release tag as an ancestor, so a merge-base with it cannot serve. The sync workflow computes it on every run, weekly ones included, and fails with the reason when no upstream tag is reachable; until 2026-10-08 it counted stable tags only, so every weekly run from 2026-09-28 on failed at this step while the line pinned v0.2.0-beta5, and the mirror stayed where the last run had left it (#274, #276). `.circleci/config.yml` states it once, as the annotation `io.giantswarm.upstream.version` every image carries. | diff --git a/charts/substrate/values.yaml b/charts/substrate/values.yaml index d18914b808..d7308f9190 100644 --- a/charts/substrate/values.yaml +++ b/charts/substrate/values.yaml @@ -431,7 +431,7 @@ images: # versions, as this chart's are). Its egress config (atenet-egress.yaml) is # written for this build's config schema, so the two move together; the # publish pipeline stamps AGENTGATEWAY_IMAGE here and refuses a drift. - agentgateway: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0-rc.2 + agentgateway: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0 # Not rendered by any template of this chart; kept as upstream keeps it. busybox: gsoci.azurecr.io/giantswarm/busybox:1.36 # The root container of every gVisor sandbox (`spec.pauseImage` of the diff --git a/docs/upgrade.md b/docs/upgrade.md index 911f659931..94c9aa2967 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -1179,10 +1179,11 @@ conversations and durable volumes survive it. - **Egress contract:** `v0.4.0-alpha1+path-prefix+tcp`. A 1.15 `kubectl ate` and a 1.16 ate-api refuse each other's egress-policy writes; use the `kubectl ate` of the release the server runs. -- **Chart:** `images.agentgateway` is 2.5.0-rc.2, which relays a stream a - `tcp` rule allows and applies a path-prefixed credential to a path with - a percent-encoded `/`; the agentgateway egress config gains its TCP - listener. The Envoy dataplane keeps closing such streams. +- **Chart:** `images.agentgateway` is 2.5.0, which relays a stream a + `tcp` rule allows, applies a path-prefixed credential to a path with a + percent-encoded `/`, and answers a denied request with a 403 that names + the destination and where it is allowed; the agentgateway egress config + gains its TCP listener. The Envoy dataplane keeps closing such streams. - **During the roll:** a `tcp` rule takes effect once the egress gateway runs the new agentgateway; until then such a stream is closed as before. - **Credential path prefix:** a request path with a percent-encoded `/` diff --git a/manifests/ate-install/components/agentgateway/kustomization.yaml b/manifests/ate-install/components/agentgateway/kustomization.yaml index 7634605d9b..8806196727 100644 --- a/manifests/ate-install/components/agentgateway/kustomization.yaml +++ b/manifests/ate-install/components/agentgateway/kustomization.yaml @@ -42,7 +42,7 @@ patches: path: /spec/template/spec/containers/0 value: name: agentgateway - image: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0-rc.2@sha256:c8bf536727c533708f1866b61c332e142d6bd8e7d6d6ec281e15fd84d63b7528 + image: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0@sha256:e48f9d4f2811f45e22a9530273b9e8b3832da2ae7eb6c541f5dcd9dbad81ac2c args: - -f - /etc/agentgateway/config.yaml @@ -118,7 +118,7 @@ patches: path: /spec/template/spec/containers/0 value: name: agentgateway - image: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0-rc.2@sha256:c8bf536727c533708f1866b61c332e142d6bd8e7d6d6ec281e15fd84d63b7528 + image: gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0@sha256:e48f9d4f2811f45e22a9530273b9e8b3832da2ae7eb6c541f5dcd9dbad81ac2c args: - -f - /etc/agentgateway/config.yaml From c74ae730f4128b7c914f904e7bcd58852079117a Mon Sep 17 00:00:00 2001 From: Timo Derstappen Date: Mon, 19 Oct 2026 20:44:42 +0200 Subject: [PATCH 2/3] docs(fork): the agentgateway pin names its pull request --- FORK.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/FORK.md b/FORK.md index 295002c584..37b85190f5 100644 --- a/FORK.md +++ b/FORK.md @@ -30,7 +30,7 @@ reasoning instead. |---|---| | Upstream tag | **v0.5.0-alpha2** (2026-10-08; tag commit `9c4b1fb5`, "Let every Helm chart container set resource requests and limits (#54)"; 109 commits past v0.4.0-alpha1, which is not its ancestor, so the re-pin replays every carried commit). Previous pins: v0.0.26 (2026-09-10 → 2026-09-14, releases `v0.0.27-gs.1`–`gs.9`), v0.0.29 (2026-09-14 → 2026-09-22, releases `v0.0.30-gs.1`–`gs.5`, `1.0.0`, `1.0.1`; `release-1.0` stays on it), v0.2.0-beta4 (2026-09-22 → 2026-09-24, dev builds only), v0.2.0-beta5 (2026-09-24 → 2026-10-05, releases `1.1.0`–`1.3.0` and the `1.4.0` candidates; `1.4.0` is cut from `81abb52e`, the last candidate before the re-pin), v0.3.0-alpha3 (2026-10-05 → 2026-10-08, releases `1.4.0`–`1.5.x`), v0.4.0-alpha1 (2026-10-08 → 2026-10-12, releases `1.6.0`–`1.8.x`) | | Why this one | upstream kagent-dev/kagent pins `github.com/kagent-dev/substrate v0.5.0-alpha2` in `go/go.mod` (the `replace` of `github.com/agent-substrate/substrate`) since kagent-dev/kagent#3116 (`5eb9b633`, "Bump Substrate to 0.5.0-alpha2"), and at `main` `92dd6caa` (2026-10-11), the head the kagent line re-pins onto; `giantswarm/kagent-upstream` replaces the same path with this line's release. Since kagent-dev/kagent#2708 kagent sets up its PostgreSQL through `pkg/postgressetup`, which v0.5.0-alpha2 exports. v0.5.0-alpha2 changes the contracts between the components: `SetWorkerCapacity` is `RegisterWorker` on `WorkerService` and `AteomSupport`, with the worker's `HardwareIdentity` (agent-substrate/substrate#2282); `MintActorJWTRequest.actor_uid` is gone (#2207); `SnapshotConfig.on_pause` is gone and a pause captures the `on_commit` scope (#2309); an actor's node is `ActorStatus.assigned_node`, and `LocalSnapshot.content_scope` moves onto field 2, where the line's node list was; the control API answers through `internal/apierror`. The persisted Actor rows of a 1.8 release are migrated on read (see its ledger file), the PostgreSQL schema gains nothing, so the move upgrades a 1.8.x installation in place. During the roll an ateom of the old release retries its capacity report against the renamed RPC until its pool rolls; the workers it registered keep their capacity. The ate-api gRPC contract between kagent's client and Substrate's server must match, so the two lines move together (2026-09-22 onto beta4, 2026-09-24 onto beta5, 2026-10-05 onto alpha3, 2026-10-08 onto v0.4.0-alpha1, 2026-10-12 onto v0.5.0-alpha2). | -| agentgateway it runs | the `agentgateway-image` parameter of `.circleci/config.yml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema) and to the image, by digest, of `manifests/ate-install/components/agentgateway`, which the e2e installs (`hack/verify/agentgateway-image.sh` fails a pull request on any drift; before the 2026-10-08 re-pin that component ran upstream's alpha build, so the e2e never ran the line's agentgateway): a release of the agentgateway line, published by that line from CircleCI under its own gsoci path and versioned on its own — **`gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0`** (tag `v2.5.0` on `913dbe92`, the stable of `v2.5.0-rc.4` on the same commit = 2.5.0-rc.2 + every address of the gateway counted local for a `tcp` rule, giantswarm/agentgateway-upstream#108 (`v2.5.0-rc.3`), and a policy denial answered `403` with a status line and body naming the destination and where it is allowed, giantswarm/agentgateway-upstream#109; pinned here by PRNUM; before it 2.5.0-rc.2, tag `v2.5.0-rc.2` on `46019685` = 2.5.0-rc.1 + the Substrate egress applying a path-prefixed credential header to a path with an encoded slash (`%2F` read as a separator yields no dot or empty segment), giantswarm/agentgateway-upstream#106 for giantswarm/agentgateway-upstream#105, the rule this line's `egresspolicy.ReplaceHeadersFor` follows since #483; pinned here by #485; before it 2.5.0-rc.1, tag `v2.5.0-rc.1` on `76f469bc` = 2.4.0 + the Substrate egress relaying TCP a `tcp` egress rule allows, decided on the address and port the actor connected to, giantswarm/agentgateway-upstream#101 for giantswarm/substrate#164, and the agentgateway line's carried fixes since: a stateful MCP session's pin to a gone endpoint dropped, test isolation of the AWS SDK config and the proxy test configs; pinned here by the pull request that adds the `tcp` rule; before it 2.4.0, tag `v2.4.0` on `d117e601`, the stable of `v2.4.0-rc.2` on the same commit = 2.3.6 + the Substrate egress injecting an `actor_jwt` entry of `replace_headers` with a JWT minted through `Control.MintActorJWT`, giantswarm/agentgateway-upstream#80 for giantswarm/agentgateway-upstream#79 and #82, which calls the mint by the method name Substrate serves (2.4.0-rc.1 asked for `MintActorJwt` and was answered `Unimplemented`), the agentgateway dataplane's half of the actor-JWT injection this line carries (its ledger file); pinned here as the stable 2.4.0 by #429, after #421 pinned its release candidate; before it 2.3.6, tag `v2.3.6` on `df3072c3` = 2.3.5 + a request the upstream refused before processing it (a refused HTTP/2 stream, or one above the upstream's `GOAWAY`) retried once on a fresh connection, giantswarm/agentgateway-upstream#78 for #401, without which an actor's `git-upload-pack` to github.com was answered `503` when a pooled connection the upstream was closing refused its stream, and the JWKS refetch awaiting the refresh in flight; proven as the egress of 1.8.5 in agentlab (`skills-test`, `agents-test`); pinned here by #411; before it 2.3.5, tag `v2.3.5` on `d5987a97` = 2.3.0 + the agentgateway line's carried patches since: the JWKS refetch on an unknown key id, the gRPC status of a trailers-only refusal logged, Go 1.27.2 and `golang.org/x/net` v0.61.0, and the Substrate egress applying a `replace_headers` entry by its `path_prefix` (giantswarm/agentgateway-upstream#65 for giantswarm/agentgateway-upstream#62), which the path-scoped credentials of this line's egress policy need on the agentgateway dataplane; pinned here by #355; before it 2.3.0, tag `v2.3.0` on `97b142d6` = the agentgateway line's 2026-10-05 re-pin onto upstream v1.6.0 (`ea560864`) plus its 40 carried commits; v1.6.0 contains agentgateway#3677 (`8d6be5fb`), the egress gateway authorizing from the `ateom-for-actor` SPIFFE URI of agent-substrate/substrate#1809 that alpha3's ateom presents and 2.2.2 refuses, and agentgateway#3689 (`c4143926`), the squash of `50999825`, the build upstream's v0.4.0-alpha1 chart pins as `ghcr.io/kagent-dev/substrate/agentgateway:50999825cb55` (alpha1 commit `50e20d5e`), so the v0.4.0-alpha1 re-pin needs no new agentgateway release; pinned here by `a07cc154`; before it 2.2.2, tag `v2.2.2` on `149d9b40` = 2.2.1 + an egress policy call that never reached its policy service answered `503 unavailable` instead of `403 denied`, giantswarm/agentgateway-upstream#35, − the named buffer error on an MCP upstream's response, reverted by giantswarm/agentgateway-upstream#36; 2.2.1 is tag `v2.2.1` on `13325b82` = 2.2.0 + a resume refusal the control plane marks not resumable answered at once with its own status, giantswarm/agentgateway-upstream#29, which the golden refusal of #80 needs; 2.2.0 is tag `v2.2.0` on `71f1adf6`, the agentgateway line's 2026-09-24 re-pin onto upstream `main` @ `3528a428` plus its carried patches and the grpc pin; before it `2.1.2`, tag `v2.1.2` on `39023841`), which descends from the dev build the 2026-09-22 re-pin chose (`0.0.0-dev.giantswarm.2026-09-22.11-31-30.h6afcce9` = upstream agentgateway `main` @ `08f4563b`, 2026-09-21, ≥ `8dba3989`: #3411's egress credential injection and #3524's provider contract, which beta4's and beta5's charts pin as `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.8dba3989`) plus the line's carried patches; the agentgateway line moved to `main` @ `3528a428` on 2026-09-24 without a release yet, so this line ran 2.1.2 until that release, `2.2.0`, existed (a released build, as the platform's image policy requires). Since 2.1.1 a file resource is read from disk on every managed fetch, so an egress CA switch reloads `tls.crt` and `tls.key` as one pair: the dev build and upstream's alpha reload them as two events and serve leaves signed with the other file's key in between, which fails every handshake of that window. Since 2.1.2 the files a parse reads are watched as well. `release-1.0` pins the same release. v0.5.0-alpha2's chart pins upstream's nightly `cr.agentgateway.dev/agentgateway:v0.0.0-alpha.168b05d6` (agent-substrate/substrate#2158) for the TLS passthrough listener and the credential providers on the HTTP and HTTPS egress policies, which the line's chart has rendered for 2.3.0 since before the pin, so the line stays on 2.3.0. | +| agentgateway it runs | the `agentgateway-image` parameter of `.circleci/config.yml`, equal to the chart default `images.agentgateway` (the publish refuses a drift — the chart must install unstamped, and its egress config is written for this build's schema) and to the image, by digest, of `manifests/ate-install/components/agentgateway`, which the e2e installs (`hack/verify/agentgateway-image.sh` fails a pull request on any drift; before the 2026-10-08 re-pin that component ran upstream's alpha build, so the e2e never ran the line's agentgateway): a release of the agentgateway line, published by that line from CircleCI under its own gsoci path and versioned on its own — **`gsoci.azurecr.io/giantswarm/agentgateway-upstream/agentgateway:2.5.0`** (tag `v2.5.0` on `913dbe92`, the stable of `v2.5.0-rc.4` on the same commit = 2.5.0-rc.2 + every address of the gateway counted local for a `tcp` rule, giantswarm/agentgateway-upstream#108 (`v2.5.0-rc.3`), and a policy denial answered `403` with a status line and body naming the destination and where it is allowed, giantswarm/agentgateway-upstream#109; pinned here by [#488](https://github.com/giantswarm/substrate/pull/488); before it 2.5.0-rc.2, tag `v2.5.0-rc.2` on `46019685` = 2.5.0-rc.1 + the Substrate egress applying a path-prefixed credential header to a path with an encoded slash (`%2F` read as a separator yields no dot or empty segment), giantswarm/agentgateway-upstream#106 for giantswarm/agentgateway-upstream#105, the rule this line's `egresspolicy.ReplaceHeadersFor` follows since #483; pinned here by #485; before it 2.5.0-rc.1, tag `v2.5.0-rc.1` on `76f469bc` = 2.4.0 + the Substrate egress relaying TCP a `tcp` egress rule allows, decided on the address and port the actor connected to, giantswarm/agentgateway-upstream#101 for giantswarm/substrate#164, and the agentgateway line's carried fixes since: a stateful MCP session's pin to a gone endpoint dropped, test isolation of the AWS SDK config and the proxy test configs; pinned here by the pull request that adds the `tcp` rule; before it 2.4.0, tag `v2.4.0` on `d117e601`, the stable of `v2.4.0-rc.2` on the same commit = 2.3.6 + the Substrate egress injecting an `actor_jwt` entry of `replace_headers` with a JWT minted through `Control.MintActorJWT`, giantswarm/agentgateway-upstream#80 for giantswarm/agentgateway-upstream#79 and #82, which calls the mint by the method name Substrate serves (2.4.0-rc.1 asked for `MintActorJwt` and was answered `Unimplemented`), the agentgateway dataplane's half of the actor-JWT injection this line carries (its ledger file); pinned here as the stable 2.4.0 by #429, after #421 pinned its release candidate; before it 2.3.6, tag `v2.3.6` on `df3072c3` = 2.3.5 + a request the upstream refused before processing it (a refused HTTP/2 stream, or one above the upstream's `GOAWAY`) retried once on a fresh connection, giantswarm/agentgateway-upstream#78 for #401, without which an actor's `git-upload-pack` to github.com was answered `503` when a pooled connection the upstream was closing refused its stream, and the JWKS refetch awaiting the refresh in flight; proven as the egress of 1.8.5 in agentlab (`skills-test`, `agents-test`); pinned here by #411; before it 2.3.5, tag `v2.3.5` on `d5987a97` = 2.3.0 + the agentgateway line's carried patches since: the JWKS refetch on an unknown key id, the gRPC status of a trailers-only refusal logged, Go 1.27.2 and `golang.org/x/net` v0.61.0, and the Substrate egress applying a `replace_headers` entry by its `path_prefix` (giantswarm/agentgateway-upstream#65 for giantswarm/agentgateway-upstream#62), which the path-scoped credentials of this line's egress policy need on the agentgateway dataplane; pinned here by #355; before it 2.3.0, tag `v2.3.0` on `97b142d6` = the agentgateway line's 2026-10-05 re-pin onto upstream v1.6.0 (`ea560864`) plus its 40 carried commits; v1.6.0 contains agentgateway#3677 (`8d6be5fb`), the egress gateway authorizing from the `ateom-for-actor` SPIFFE URI of agent-substrate/substrate#1809 that alpha3's ateom presents and 2.2.2 refuses, and agentgateway#3689 (`c4143926`), the squash of `50999825`, the build upstream's v0.4.0-alpha1 chart pins as `ghcr.io/kagent-dev/substrate/agentgateway:50999825cb55` (alpha1 commit `50e20d5e`), so the v0.4.0-alpha1 re-pin needs no new agentgateway release; pinned here by `a07cc154`; before it 2.2.2, tag `v2.2.2` on `149d9b40` = 2.2.1 + an egress policy call that never reached its policy service answered `503 unavailable` instead of `403 denied`, giantswarm/agentgateway-upstream#35, − the named buffer error on an MCP upstream's response, reverted by giantswarm/agentgateway-upstream#36; 2.2.1 is tag `v2.2.1` on `13325b82` = 2.2.0 + a resume refusal the control plane marks not resumable answered at once with its own status, giantswarm/agentgateway-upstream#29, which the golden refusal of #80 needs; 2.2.0 is tag `v2.2.0` on `71f1adf6`, the agentgateway line's 2026-09-24 re-pin onto upstream `main` @ `3528a428` plus its carried patches and the grpc pin; before it `2.1.2`, tag `v2.1.2` on `39023841`), which descends from the dev build the 2026-09-22 re-pin chose (`0.0.0-dev.giantswarm.2026-09-22.11-31-30.h6afcce9` = upstream agentgateway `main` @ `08f4563b`, 2026-09-21, ≥ `8dba3989`: #3411's egress credential injection and #3524's provider contract, which beta4's and beta5's charts pin as `ghcr.io/agentgateway/agentgateway:v0.0.0-alpha.8dba3989`) plus the line's carried patches; the agentgateway line moved to `main` @ `3528a428` on 2026-09-24 without a release yet, so this line ran 2.1.2 until that release, `2.2.0`, existed (a released build, as the platform's image policy requires). Since 2.1.1 a file resource is read from disk on every managed fetch, so an egress CA switch reloads `tls.crt` and `tls.key` as one pair: the dev build and upstream's alpha reload them as two events and serve leaves signed with the other file's key in between, which fails every handshake of that window. Since 2.1.2 the files a parse reads are watched as well. `release-1.0` pins the same release. v0.5.0-alpha2's chart pins upstream's nightly `cr.agentgateway.dev/agentgateway:v0.0.0-alpha.168b05d6` (agent-substrate/substrate#2158) for the TLS passthrough listener and the credential providers on the HTTP and HTTPS egress policies, which the line's chart has rendered for 2.3.0 since before the pin, so the line stays on 2.3.0. | | When it moves | only together with kagent's pin, proven in agentlab first (`agentlab configure --defaults --chart-branch poc/kagent-main && agentlab up` and the proofs) — see "Re-pin". Not on a schedule. | | Derived how | the nearest upstream `v*` tag reachable from the line, prereleases included (`git tag --merged` against `git ls-remote --tags upstream`, nearest by commit distance — a version sort would place a prerelease behind its release): the line's own tags version the package (see "Versions") and share the `v*` namespace, so they never count — and upstream `main`, rebased onto agent-substrate, has no release tag as an ancestor, so a merge-base with it cannot serve. The sync workflow computes it on every run, weekly ones included, and fails with the reason when no upstream tag is reachable; until 2026-10-08 it counted stable tags only, so every weekly run from 2026-09-28 on failed at this step while the line pinned v0.2.0-beta5, and the mirror stayed where the last run had left it (#274, #276). `.circleci/config.yml` states it once, as the annotation `io.giantswarm.upstream.version` every image carries. | From da593c8d3dccdc07e4d7d4fb05c08e5d2d1e094e Mon Sep 17 00:00:00 2001 From: Timo Derstappen Date: Mon, 19 Oct 2026 20:46:07 +0200 Subject: [PATCH 3/3] docs(fork): the tcp rule file names the agentgateway 2.5.0 pin --- fork/egress-tcp-rule-allows-tcp-neither-tls-nor-http.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fork/egress-tcp-rule-allows-tcp-neither-tls-nor-http.md b/fork/egress-tcp-rule-allows-tcp-neither-tls-nor-http.md index 8814dcc8a1..230a1d2bf5 100644 --- a/fork/egress-tcp-rule-allows-tcp-neither-tls-nor-http.md +++ b/fork/egress-tcp-rule-allows-tcp-neither-tls-nor-http.md @@ -9,6 +9,7 @@ commits: - 'fix(egresspolicy): the line''s egress policy contract carries the tcp rule' - 'chore(agentgateway): run 2.5.0-rc.1, which relays a tcp rule' - 'test(ate-setup): the agentgateway egress overlay has the tcp listener' + - 'chore(agentgateway): run 2.5.0, the stable of 2.5.0-rc.4' upstream: outcome: to-file refs: [] @@ -20,7 +21,7 @@ verified: 2026-10-19 ## Change -`EgressRule` gains a `tcp` member (`TCPRule`, field 4 of the `rule` oneof): exact hostnames and CIDRs with a required port selection. The gateway decides a connection by its first bytes: a TLS ClientHello by the `tls_passthrough` and `https` rules, an HTTP request by the `http` rules, anything else by the `tcp` rules, on the address and port the actor connected to. A hostname matches when the gateway resolves it to that address; the actor's own resolution plays no part. The addresses of the gateway and its node (loopback, link-local, the clouds' metadata and platform services, the gateway's own address, and their NAT64 forms) are reached only through a CIDR inside their range, never through a hostname or a wider CIDR. Admission requires at least one hostname or CIDR and refuses a CIDR with host bits past its prefix. The agentgateway egress config gains the TCP listener the rule's streams are relayed through; the relay itself is agentgateway's (the agentgateway line's release this line pins, giantswarm/agentgateway-upstream#101). The Envoy dataplane has no `tcp` chain and keeps closing such connections. `kubectl ate` reads and writes the rule. Unit tests in `internal/egresspolicy`, `apivalidation` and `kubectl-ate`; the networking e2e (`TestActorEgressTCPByAddress`) relays a client-first exchange to a listed origin and sees a connection to an unlisted one closed with no bytes; the egress demo gains `POST /tcp`, the e2e test server a `tcp-echo` mode. +`EgressRule` gains a `tcp` member (`TCPRule`, field 4 of the `rule` oneof): exact hostnames and CIDRs with a required port selection. The gateway decides a connection by its first bytes: a TLS ClientHello by the `tls_passthrough` and `https` rules, an HTTP request by the `http` rules, anything else by the `tcp` rules, on the address and port the actor connected to. A hostname matches when the gateway resolves it to that address; the actor's own resolution plays no part. The addresses of the gateway and its node (loopback, link-local, the clouds' metadata and platform services, the gateway's own address, and their NAT64 forms) are reached only through a CIDR inside their range, never through a hostname or a wider CIDR. Admission requires at least one hostname or CIDR and refuses a CIDR with host bits past its prefix. The agentgateway egress config gains the TCP listener the rule's streams are relayed through; the relay itself is agentgateway's (the agentgateway line's release this line pins, giantswarm/agentgateway-upstream#101; since 2.5.0, [#488](https://github.com/giantswarm/substrate/pull/488), every address of the gateway counts as local for the rule, giantswarm/agentgateway-upstream#108). The Envoy dataplane has no `tcp` chain and keeps closing such connections. `kubectl ate` reads and writes the rule. Unit tests in `internal/egresspolicy`, `apivalidation` and `kubectl-ate`; the networking e2e (`TestActorEgressTCPByAddress`) relays a client-first exchange to a listed origin and sees a connection to an unlisted one closed with no bytes; the egress demo gains `POST /tcp`, the e2e test server a `tcp-echo` mode. ## Reason