Skip to content

[Bug] Data plane is hard IPv4 #246

Open

Description

@ixcans

Expected Behavior

IPv6 only cluster support

Actual Behavior

Envoy DNS resolver pinned V4_ONLY (atenet/.../xds.go:241), 0.0.0.0 binds, and the actor interior net is hard-coded 169.254.17.x/30 forced through .To4() which rejects non-IPv4.

Activity

  1. EItanya commented on Jun 15, 2026

    @EItanya
    Collaborator

    I agree that ipv6 only support is super important, but I also think we have so many open networking questions we're looking to answer that maybe this should be a fast follow from those.
    Some examples:

  2. bowei commented on Jun 15, 2026

    @bowei
    Collaborator

    I think that would be good to have as a first step is a way to start the Kind setup with IPv6 single-stack and IPv6 dual stack. Otherwise, if we get patches to fix IPv6 support, we would not be able to test the fixes.

  3. BenTheElder commented on Jun 15, 2026

    @BenTheElder
    Collaborator

    I think that would be good to have as a first step is a way to start the Kind setup with IPv6 single-stack and IPv6 dual stack. Otherwise, if we get patches to fix IPv6 support, we would not be able to test the fixes.

    We have to be careful about how much actions time we use up on every PR as long as we're operating with the free default resources. It can add up every quickly covering every dimension.

    Also FYI single-stack ipv6 kind has some gaps currently, in general we mostly see ipv4 or dual-stack demand and contributions ...

    We're not even running ipv6 kind on every k8s PR currently. We run it periodically post-merge.

  4. bowei commented on Jun 15, 2026

    @bowei
    Collaborator

    I'd be ok with starting with dual-stack for local testing but not in the CI.

  5. bowei commented on Jun 22, 2026

    @bowei
    Collaborator

    Ok, some concrete steps towards chipping away at this:

    • PR to add starting up Kind in dual-stack mode, installing Substrate on the cluster
    • Fixing the Envoy bind to include IPv6 addresses
    • Test (probably draft PR) to show what else is broken right now.
  6. ixcans commented on Jul 31, 2026

    @ixcans
    Author

    Hello folks, Bowei Du (@bowei), Benjamin Elder (@BenTheElder), Eitan Yarmush (@EItanya),

    Following up after PR #559 merged: the pod-IP:80 → actor-veth:80 DNAT is gone with no fallback, so atunnel's mTLS listener is now the only actor ingress path. The listen address itself is flag-configurable, but the egress redirect in internal/atunnel/original_dst_linux.go is SO_ORIGINAL_DST-only (IPv6 variant still a TODO), and internal/ateomnet's interior actor network is still hard IPv4 after the #548 refactor moved it into a shared package.

    I was previously able to fork-patch around this to run Substrate on an IPv6-only cluster; with atunnel now mandatory and no fallback, that's no longer straightforward. Could we raise the priority on IPv6/dual-stack support, and what's actually blocking it at this point? Please let me know I could help here.

    Thanks!

  7. bowei commented on Jul 31, 2026

    @bowei
    Collaborator

    See my comment: first step is to enable creating an ipv6 enabled kind cluster so so ipv6 support can be tested.

  8. 41 remaining items

  9. go-faustino commented on Oct 21, 2026

    @go-faustino

    Adding a downstream data point, in case it helps with the "where does IPv6 demand come from" question earlier in this thread.

    Setup: EKS with an IPv6 service plane. Services are ipFamilyPolicy: SingleStack and ipFamilies: [IPv6], so cluster DNS returns AAAA records only. Substrate is v0.6.0-alpha3 from the kagent-dev fork, with the gVisor sandbox class. kagent is v1.0.0-alpha16 on top.

    What happens: Since kagent-dev/kagent#2939, the kagent runtime inside the actor saves A2A task state over gRPC to the kagent controller Service (runtimeDefaults.controlPlaneUrl). On this cluster that Service only has an IPv6 ClusterIP. The actor resolves it fine, because the DNS relay forwards the query and the AAAA record comes back. The dial then fails inside the guest before the model is ever called:

    failed to save task state: rpc error: code = Unavailable
    desc = connection error: desc = "transport: Error while dialing:
    dial tcp [<ipv6-clusterip>]:8083: connect: network is unreachable"
    

    This matches the current actornet shape. The actor only gets 169.254.17.2/30 with an IPv4 default route, and the atunnel redirect is only installed in the IPv4 nftables table. So there's no route for an IPv6 destination, and #945 would drop the traffic further along anyway. Agent readiness and snapshot checks all stay green; the failure only shows up on the first session input. Every kagent-on-Substrate chat on an IPv6-service EKS cluster fails the same way.

    Offer: we'd be glad to test the #1057 / #1116 stack on a real IPv6 EKS cluster (gVisor) once it's in a testable state, and report back on the per-family paths it touches, mainly actor → in-cluster IPv6 Service. If there's a branch or image tag you'd like exercised, or specific scenarios that would help get the fixes past the IPv6 e2e gate, let us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions