test(live): ask redis who is subscribed, not the relay's own map - #261
Merged
Merged
Conversation
TestAChangeOnOneServerReachesTheOther fails on CI roughly one run in eight with "0 sync frame(s) arrived in ten seconds" -- the failure the test's own comment describes, from before the last attempt to close it. The readiness probe was the hole. Relay.Subscribe records a subscription in r.subscriptions and only then waits for Redis to confirm it, because Subscribe merely queues the command. waitForSubscription polled that map, so it returned during the window where the far server is registered locally but not yet with Redis. Pub/sub keeps no backlog, so the change published in that window is not delayed, it is dropped, and the test then waits out its full ten seconds for a frame that no longer exists. Redis is now asked directly with PUBSUB NUMSUB, which only counts a subscriber once the SUBSCRIBE has actually landed. The relay itself is unchanged: its own ordering is correct, it confirms before it publishes. Verified: the live suite passes, including under CPU contention and -race. Not verified: the original failure did not reproduce locally in 60 runs under load, so this closes a window that is certain from reading the code rather than one demonstrated by a local repro.
houko
force-pushed
the
fix/relay-subscription-readiness
branch
from
September 19, 2026 14:33
c39aa95 to
5cc95cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
TestAChangeOnOneServerReachesTheOtherfails on CI roughly one run in eight, always the same way:It hit #254 and #258 during this batch. Neither touches
internal/live; #258 does not touch Go at all.Why
Relay.Subscriberecords a subscription inr.subscriptionsand only afterwards waits for Redis to confirm it, becauseclient.Subscribemerely queues the command —Receiveis what sends it:waitForSubscriptionpolled that map, so it returned during the window between those two statements. Pub/sub keeps no backlog, so the change the test publishes in that window is not delayed, it is dropped, and the test then waits out its full ten seconds for a frame that no longer exists.The relay's own ordering is correct — it confirms before it publishes — so nothing in
relay.gochanges here.0 framesis what points at this rather than at a slow machine: every server publishes SyncStep1 and QueryAwareness fromSubscribeitself, so a server that were subscribed would have received those too. Receiving nothing at all means it was not subscribed when the other side published.The change
Ask Redis instead, with
PUBSUB NUMSUB, which only counts a subscriber once theSUBSCRIBEhas actually landed. Both servers share one Redis, so one question covers the pair.Verification
internal/livesuite passes, including under CPU contention and-race.