Skip to content

test(live): ask redis who is subscribed, not the relay's own map - #261

Merged
houko merged 2 commits into
mainfrom
fix/relay-subscription-readiness
Sep 19, 2026
Merged

houko merged 2 commits into
mainfrom
fix/relay-subscription-readiness

Conversation

@houko

@houko houko commented Sep 19, 2026

Copy link
Copy Markdown
Member

The failure

TestAChangeOnOneServerReachesTheOther fails on CI roughly one run in eight, always the same way:

--- FAIL: TestAChangeOnOneServerReachesTheOther (10.01s)
    relay_test.go:131: the change never crossed to the other server: 0 sync frame(s) arrived in ten seconds, each rejected as []

It hit #254 and #258 during this batch. Neither touches internal/live; #258 does not touch Go at all.

Why

Relay.Subscribe records a subscription in r.subscriptions and only afterwards waits for Redis to confirm it, because client.Subscribe merely queues the command — Receive is what sends it:

subscription := r.client.Subscribe(ctx, documentChannel(name))
r.subscriptions[name] = subscription          // relay.go:140 — visible here
r.mu.Unlock()

if _, err := subscription.Receive(ctx); err != nil {   // relay.go:144 — confirmed only here

waitForSubscription polled that map, so it returned during the window between those two statements. Pub/sub keeps no backlog, so the change the test publishes in that window is not delayed, it is dropped, and the test then waits out its full ten seconds for a frame that no longer exists.

The relay's own ordering is correct — it confirms before it publishes — so nothing in relay.go changes here.

0 frames is what points at this rather than at a slow machine: every server publishes SyncStep1 and QueryAwareness from Subscribe itself, so a server that were subscribed would have received those too. Receiving nothing at all means it was not subscribed when the other side published.

The change

Ask Redis instead, with PUBSUB NUMSUB, which only counts a subscriber once the SUBSCRIBE has actually landed. Both servers share one Redis, so one question covers the pair.

Verification

  • The internal/live suite passes, including under CPU contention and -race.
  • Not verified: the original failure did not reproduce locally in 60 runs under load, on the old code or the new. This closes a window that is certain from reading the code, not one demonstrated by a local repro. If the test still flakes after this lands, the root cause is elsewhere and this reasoning should be discarded rather than patched over.

TestAChangeOnOneServerReachesTheOther fails on CI roughly one run in eight with
"0 sync frame(s) arrived in ten seconds" -- the failure the test's own comment
describes, from before the last attempt to close it.

The readiness probe was the hole. Relay.Subscribe records a subscription in
r.subscriptions and only then waits for Redis to confirm it, because Subscribe
merely queues the command. waitForSubscription polled that map, so it returned
during the window where the far server is registered locally but not yet with
Redis. Pub/sub keeps no backlog, so the change published in that window is not
delayed, it is dropped, and the test then waits out its full ten seconds for a
frame that no longer exists.

Redis is now asked directly with PUBSUB NUMSUB, which only counts a subscriber
once the SUBSCRIBE has actually landed. The relay itself is unchanged: its own
ordering is correct, it confirms before it publishes.

Verified: the live suite passes, including under CPU contention and -race. Not
verified: the original failure did not reproduce locally in 60 runs under load,
so this closes a window that is certain from reading the code rather than one
demonstrated by a local repro.
@houko
houko force-pushed the fix/relay-subscription-readiness branch from c39aa95 to 5cc95cc Compare September 19, 2026 14:33
@houko
houko merged commit 3e0d1ed into main Sep 19, 2026
14 checks passed
@houko
houko deleted the fix/relay-subscription-readiness branch September 19, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant