Multiple flaky networking tests

Lior Lieberman (@LiorLieberman)

TestActorArbitraryPortAccess
TestActorDirectAccess
TestActorEgress
TestActorEgressHTTPS

/cc Lior Lieberman (@LiorLieberman) Bowei Du (@bowei) Keith Mattix II (@keithmattix)

failing on main post merge:
https://github.com/agent-substrate/substrate/actions/runs/32200086998/job/95911826556

#993 adds some retries to TestActorArbitraryPortAccess

thank you! everything passed when that merged, watching for if the next commit does as well. unfortunately we're starting to get throttled so it may be a bit. will close if we don't see more.

From https://github.com/agent-substrate/substrate/actions/runs/32200086998/job/95911826556,

=== RUN   TestActorArbitraryPortAccess
    arbitraryport_test.go:51: creating actor networking-e2e/arbitraryport-1787098383986053833
    arbitraryport_test.go:51: resumed actor networking-e2e/arbitraryport-1787098383986053833
=== RUN   TestActorArbitraryPortAccess/regression:_default_port_still_reachable
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:58: default-port access through ingress returned HTTP 503; retrying...
    arbitraryport_test.go:61: default-port access through ingress succeeded; body: ok
=== RUN   TestActorArbitraryPortAccess/arbitrary_port_reachable
    arbitraryport_test.go:73: tunneled request returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
=== RUN   TestActorArbitraryPortAccess/unlisted_port_rejected
    arbitraryport_test.go:130: tunneled request to an unlisted port correctly returned HTTP 503; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorArbitraryPortAccess (42.69s)
    --- PASS: TestActorArbitraryPortAccess/regression:_default_port_still_reachable (30.50s)
    --- FAIL: TestActorArbitraryPortAccess/arbitrary_port_reachable (5.23s)
    --- PASS: TestActorArbitraryPortAccess/unlisted_port_rejected (5.04s)
=== RUN   TestActorDirectAccess
    networking_test.go:53: creating actor networking-e2e/direct-1787098426678550099
    networking_test.go:53: resumed actor networking-e2e/direct-1787098426678550099
=== RUN   TestActorDirectAccess/direct
    networking_test.go:58: direct Actor access through ate-demo-counter/counter-679c74d7df-m5hlk:80 was blocked as expected: the server is currently unable to handle the request (get pods counter-679c74d7df-m5hlk:80)
=== RUN   TestActorDirectAccess/via_ingress
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503; retrying...
    networking_test.go:62: Actor access through ingress returned HTTP 503 after 30s; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorDirectAccess (36.40s)
    --- PASS: TestActorDirectAccess/direct (0.01s)
    --- FAIL: TestActorDirectAccess/via_ingress (35.54s)
=== RUN   TestActorEgress
    networking_test.go:77: creating actor networking-e2e/egress-1787098463077668393
    networking_test.go:77: resumed actor networking-e2e/egress-1787098463077668393
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:82: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:84: Actor egress fetch returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorEgress (35.44s)
=== RUN   TestActorEgressHTTPS
    networking_test.go:96: creating actor networking-e2e/egress-https-1787098498518949911
    networking_test.go:96: resumed actor networking-e2e/egress-https-1787098498518949911
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:105: fetch through egress Actor returned HTTP 503; retrying...
    networking_test.go:107: Actor HTTPS egress fetch returned HTTP 503, want 200; body: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
--- FAIL: TestActorEgressHTTPS (35.44s)
FAIL
FAIL	github.com/agent-substrate/substrate/internal/e2e/suites/networking	150.107s

I will take a look

haiyanmeng

haiyanmeng Looks like that link is actually an old run from before my PR and #669 which fixes a lot of the 503s. Is there a newer failure I can take a look at?

Keith Mattix II (@keithmattix)

Keith Mattix II (@keithmattix) , I glanced the failures at the top of https://github.com/agent-substrate/substrate/actions/workflows/pr-workflow.yaml?query=is%3Afailure, but have not found one showing the flaky networking tests.

I wish we could have some dashboard showing the green/red status of each test in each run.

Keith Mattix II (@keithmattix)

Keith Mattix II (@keithmattix) Newer failure:https://github.com/agent-substrate/substrate/actions/runs/32333892880

=== RUN   TestActorArbitraryPortAccess/regression:_default_port_still_reachable
arbitraryport_test.go:60: default-port access through ingress returned HTTP 503   (×6)
=== RUN   TestActorArbitraryPortAccess/arbitrary_port_reachable
arbitraryport_test.go:67: tunneled request to port 9090 returned HTTP 503         (×3)
arbitraryport_test.go:88: default-port response body = "upstream connect error or
disconnect/reset before headers. reset reason: connection timeout"
 --- FAIL: TestActorArbitraryPortAccess (59.57s)

Benjamin Elder (@BenTheElder)

The rootcause is kubernetes-sigs/kind#4215 — kindnetd's 50Mi limit. Fix is merged upstream, but no release carries it yet.

I reproduced it on an x86_64 VM matching the CI runner (same kernel, 6.17.0-1022-azure) and ran A/B experiments to confirm it's the same defect, that it's what produces our 503s. With the limits removed, the failures go away.

What happens. kindnetd's steady-state working set measured 44–67MB (median ~58MB) across 14 runs under our e2e load, against a 50Mi cap. It's never OOMKilled (restartCount=0) — it just thrashes its own text pages, stops draining the NFQUEUE, and NetworkPolicy verdicts stop arriving. Queued SYNs are neither accepted nor dropped (queue_dropped=0), so the router waits out its 5s connect timeout and Envoy returns the 503.

Evidence.

26 identical CI rounds on one host, alternating only kindnetd's memory limit: 512Mi → 0/11 failed, 50Mi → 5/12 (Fisher one-sided p = 0.024).
Our patch run verbatim: 0/4 on the same host, under the pressure that made a 40Mi arm fail with all four tests.
On a host that was actively black-holing, raising memory.max with no pod restart took router→pod reachability from 0/10 to 10/10 in under 20 seconds. In another round the whole chain landed on one 5s-sampled timeline — reclaim climbed, NFQUEUE depth went 0 → 117 and plateaued, all four tests failed together. CPU throttling stayed flat at 82%, so it isn't the CPU limit.

Why it's intermittent. Two things have to coincide: kindnetd's own cgroup has to be charged for its ~34MB of text (memcg charges whoever faults first — on a fresh node containerd holds them and kindnet's cgroup sits at ~15MB, nowhere near the cap), and the node has to be under page-cache pressure. Hence ~17% in CI and never locally.

Next step:

Cut a kind release with the fix and we'll bump to it. cc Benjamin Elder (@BenTheElder)
Land test(e2e): require 502 for rejected ports #1103 alongside this — TestActorArbitraryPortAccess only asserted != 200, so these 503s passed as "correctly rejected" in 6 runs.

fix: land the three root-caused flaky-test fixes (identity, parking, …

No failures for TestActorEgressHTTPS (e2e-gvisor) across 441 runs in the last 7 days — the rolling window has fully aged out; marking resolved on the flake dashboard.

Navigation Menu

Repository navigation

Multiple flaky networking tests #1050

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Multiple flaky networking tests #1050

Description

Activity

keithmattix commented on Aug 19, 2026

BenTheElder commented on Aug 19, 2026

haiyanmeng commented on Aug 19, 2026

keithmattix commented on Aug 19, 2026

keithmattix commented on Aug 19, 2026

haiyanmeng commented on Aug 19, 2026

haiyanmeng commented on Aug 19, 2026

orangeCatDeveloper commented on Aug 20, 2026

orangeCatDeveloper commented on Aug 21, 2026

aditya-shantanu commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions