Add the ProxyAdminService proto

[liam-lowe]

Summary

Adds the ProxyAdminService protobuf service boilerplate.

The service starts with one RPC, DescribeClusterConnections, which reports a proxy and the cluster connections it has to other proxies.

Routing

Routing is interesting:

Each proxy (already) defines in config the desired cluster connections it reaches - these will be the connections reported on.

For each cluster connection, we will report on reachability and configuration. is the cluster connection reachable - can we connect, is it stable? how was the proxy configured, what dynamic config does the cluster have, what version is the cluster on? etc.

When hitting the DescribeClusterConnections endpoint, the call will describe itself. Ie - DescribeClusterConnections(target = self). In order to reach another cluster connection (one defined in the self's config) we set a grpc metadata fors2s-proxy-target. This will forward the call to that particular proxy. Ie - DescribeClusterConnections(target = oneOfTheCounterpartyClusterConnections)

In order to forward the call, we reach the counterparty proxy through its own existing mux connection. If the endpoint can't be reached, the cluster connection is considered not reachable. We don't expose a public API per proxy because: 1. we already have the mux connection; 2. that would required increased networking and security surface area.

Once a cluster is reached, the response is determined, and returned to the called. Determining the response is interesting. We need to ensure: 1. all pods serving that cluster connection are on the same page (deployments, rollouts, etc) and 2. some response values are aggregate across all pods (ie total connected muxes). Thus, we need to hit all pods in the cluster connection and aggregate their response.

Once a cluster connection is reached by an external cluster connection - it performs a fanout to all proxy pods within its pool. This is controlled by another gRPC metadata - s2s-proxy-scope. With scope - the pod will either response for member (itself) or group where it does a fanout to all members (now passing scope as member and aggregating the result).

I opted for using metadata instead of request attributes - as this functionality will need to be built into more ProxyAdminService methods in the future - and this should scale more easily

To determine members of a pool, we now provide memberlist configuration on startup. We are keeping this configuration extensible, to support numerous approaches to memberlists. For now, a basic dns discovery will be supported.

As diagrams

some diagrams to visualize the routing a little easier:

1. s2s-proxy-scope: how many members of this group answer.

1. is it just member (just itself) the node hit

flowchart LR
  subgraph M["s2s-proxy-scope: member"]
    direction LR
    O1(["operator"]) -->|call| A1["member-1"]
    A1 -->|"answers for itself"| R1(["one member"])
  end

Loading

2. Or is it the whole group via fanout. with the originally hit member aggregating.

flowchart LR
  subgraph G["s2s-proxy-scope: group (the default when absent)"]
    direction LR
    O2(["operator"]) -->|call| B1["member-1"]
    B1 -.->|fan out| B2["member-2"]
    B1 -.->|fan out| B3["member-3"]
    B2 -.->|reply| B1
    B3 -.->|reply| B1
    B1 -->|"one merged answer"| R2(["whole group"])
  end

Loading

2. s2s-proxy-target: which cluster connection answers the query? is it itself, or forwarded to a counterparty ( a cluster connection listed in the config)

flowchart LR
  O(["operator"]) -->|"target: prod-migration"| A1["group A<br/>member-1"]
  A1 ==>|"forwarded once,<br/>over that connection's mux"| B1["group B<br/>counterparty"]
  B1 -.->|group scope| B2["member-2"]
  B2 -.-> B1
  B1 ==>|"B's answer"| A1
  A1 --> O

Loading

s2s-proxy-target names a cluster connection. The call is forwarded once to that connection's counterparty, which answers at group scope for its own group. grpc metadata is not propagated to outgoing calls - so calls will be forwarded exactly once.

DescribeClusterConnections

The request is empty - at the moment this call just describes the state and everything the proxy knows.

The response is a single list. Each cluster connection carries its members, and each member reports its respective information.

Worked example

Requested at the default group scope against a three-member group with three cluster connections configured.

Request. Empty. Everything that varies travels in metadata.

grpcurl -plaintext \
  -H 's2s-proxy-scope: group' \
  localhost:6061 \
  temporal.s2sproxy.proxyadmin.v1.ProxyAdminService/DescribeClusterConnections

{}

Response. Every connection in config is listed, whether or not it is up.

{
  "clusterConnections": [
    {
      "name": "prod-migration",
      "state": "CONNECTION_STATE_CONNECTED",
      "muxSessionsConnected": 30,
      "muxSessionsTotal": 30,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        }
      ]
    },
    {
      "name": "dr-standby",
      "state": "CONNECTION_STATE_ERROR",
      "muxSessionsTotal": 30,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_ERROR",
          "muxSessionsTotal": 10
        }
      ]
    },
    {
      "name": "eu-migration",
      "state": "CONNECTION_STATE_ERROR",
      "muxSessionsConnected": 20,
      "muxSessionsTotal": 20,
      "members": [
        {
          "identity": {
            "id": "s2s-proxy-0",
            "self": true,
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:02Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-1",
            "address": "10.4.1.19:9234",
            "version": "1.42.0",
            "startTime": "2026-08-24T09:14:07Z"
          },
          "state": "CONNECTION_STATE_CONNECTED",
          "muxSessionsConnected": 10,
          "muxSessionsTotal": 10
        },
        {
          "identity": {
            "id": "s2s-proxy-2",
            "address": "10.4.1.23:9234",
            "version": "1.41.2",
            "startTime": "2026-08-21T18:02:55Z"
          },
          "state": "CONNECTION_STATE_ERROR"
        }
      ]
    }
  ]
}

[JayChung0258]

How do we determine CONNECTED at the group level?

For example, suppose muxCount is configured as 30 and the expected deployment has three pods. The happy path would be:

3 pods × 10 sessions = 30 connected / 30 total / 30 target

Would two pods with 15 sessions each still be considered CONNECTED because the aggregate is 30 / 30 / 30? What about one pod holding all 30 sessions?

We have seen cases where one pod holds all sessions but replication still gets stuck. Should CONNECTED also require the expected number of members and a healthy session distribution across them, rather than only matching the aggregate target?