Repository navigation
Live show leader re-election can race when the same user has two open tabs #1424
Description
Background
Split from #1414 (flaky Firefox E2E: `13-live-show` "follower receives leader page navigation"). That issue's flake was root-caused and fixed on the test side (the E2E spec logged both the leader and follower browser contexts in as the same admin user, which put the follower's WS session into the leader's own re-election candidate pool — see the fix PR for the full mechanism). Fixing the test by giving the follower a distinct user account eliminates the flake, but it surfaced a real, narrower race in the server-side leader re-election logic that's worth tracking separately.
The mechanism
In `server/controllers/ws_controller.py`:
-
`on_close()` (~line 171-226): when the WS connection that is the current live-session leader closes, it re-elects a new leader via:
next_session: Session = session.scalars( select(Session).where(Session.user_id == live_session.user_id) ).first()
This picks any other open `Session` row belonging to the same user as the one who started the live session — with no `ORDER BY`, so the choice is DB-order-dependent, not deterministic.
-
If that same user has two tabs open (e.g. a stage manager with a laptop and a backup tablet, both logged in as the same account), and the leader tab reloads (e.g. via `RELOAD_CLIENT`, broadcast by "Jump To Page" / "Reload Clients" — see `LIVE_SHOW_JUMP_TO_PAGE` and `RELOAD_CLIENTS` handlers), both tabs' WebSocket connections can close and reconnect at nearly the same time.
-
`on_close`'s `.first()` query can then non-deterministically pick whichever tab's `Session` row happens to exist at that instant — which is not necessarily the original leader tab reconnecting. If it picks the other tab, that tab silently becomes the new leader (`ShowSession.client_internal_id` is reassigned, `last_client_internal_id` is cleared), even though the docs (`docs/pages/live_show.md`) describe the leader tab as reclaiming leadership automatically on reconnect via `REFRESH_CLIENT` matching `last_client_internal_id` — that self-healing path only works if `last_client_internal_id` hasn't already been overwritten by a premature promotion of the other tab.
-
`server/test/controllers/test_ws_controller.py::test_websocket_close_elects_leader` already demonstrates the underlying promotion behavior deterministically (same-user second session gets promoted on leader close) — it's the timing of a concurrent reload race that's the open question, not the promotion logic itself.
Suggested follow-up
- Consider only promoting a same-user session that isn't itself about to also disconnect/reconnect (e.g. a short grace period before re-electing, or having the reconnecting leader's `REFRESH_CLIENT` take priority over an in-flight promotion).
- Add a backend WS test that drives the two-tabs-same-user-reload interleaving explicitly (connect ws1 as leader, connect ws2 same user, close ws1, immediately have ws1 reconnect via `REFRESH_CLIENT` before ws2's promotion completes, and assert the leader ends up back on ws1) to pin down and fix the exact race.
Not a regression from #1414's fix
This is a pre-existing behavior in `ws_controller.py`; the E2E fix for #1414 works around it by using two different user accounts for leader/follower in the test (matching realistic usage), which is why the flake in that spec is resolved without touching this code path.
Activity
Tim020 commentedon Sep 24, 2026
Correction to the "Suggested follow-up" reproduction sketch above: promotion in `on_close` runs synchronously (no await between the `.first()` query and reassigning `client_internal_id`), so a single leader-tab reload racing against the follower's own `REFRESH_CLIENT` reconnect can't interleave the way I described — a lone reload with the other tab's original connection still open is just the designed handoff, and `test_websocket_close_elects_leader` already covers that path correctly (not a bug).
The actual sticky interleaving needs both tabs reloading (which is exactly what happens here, since `RELOAD_CLIENT` is broadcast to every client):
on_closefires, itselect_live_leaderblock queriesSessionrows foruser_id == live_session.user_idand finds F's still-open row, promoting F:client_internal_id = F,last_client_internal_id = None.on_closethen fires (or already had, order doesn't matter for this step) — since F is not the elected leader at the time its own connection closes,entry.live_sessionis only truthy for whichever session currently holdsclient_internal_id, so this step doesn't necessarily re-trigger election again for F specifically; what matters is F's reconnect.REFRESH_CLIENTwith its persisted UUID. Sinceclient_internal_idalready equals F's UUID (from step 2), F is already "the leader" from the DB's point of view — no correction needed, but this is now the wrong client owning leadership from the user's perspective (the person who reloaded expecting to still be leader is now a follower).REFRESH_CLIENTwith its own persisted UUID.last_client_internal_idisNone(cleared in step 2), so theshow_session.last_client_internal_id == new_uuidcheck at ws_controller.py lines ~380-398 fails, and L's reconnect does not reclaim leadership. L stays a follower.So the actual bug is simpler than I first described: step 2's promotion is unconditional and immediate on any other same-user session existing at close time, with no way for the original leader to reassert itself once that's happened, because
last_client_internal_idwas already spent/cleared by the promotion. Confirmed the swapped-header outcome directly by A/B-testing the E2E fix in #1425 (temporarily reverting the follower back to the admin login and re-running the full Firefox suite): 3 of 4 runs failed with the leader's own header showing "Following" after the reload, matching this mechanism exactly.Updating the suggested follow-up: a fix likely needs the reconnecting original leader's
REFRESH_CLIENT/NEW_CLIENTto be able to reclaim leadership from whoever was opportunistically promoted during the reload window, not just whenlast_client_internal_idstill points at it.Test real line-position sync, not just the leader/follower flag
Tim020 commentedon Sep 24, 2026
One more correction to my step 3 above (the rest of the writeup stands): I said F's `on_close` wouldn't re-trigger election because F wasn't the leader "at the time its own connection closes" — that's backwards. By the time F's socket closes, F is the leader (step 2 already set `client_internal_id = F`), so `entry.live_session` is truthy for F, and F's `on_close` does run the election block again: it sets `last_client_internal_id = F` and, finding no other authenticated same-user session at that point, hits `NO_LEADER`.
That's what actually makes the swap stick: F's subsequent `REFRESH_CLIENT` reconnect matches `last_client_internal_id == F` (lines ~380-398) and reclaims leadership for itself — regardless of whether L's `NEW_CLIENT` got there first, since `REFRESH_CLIENT`'s reclaim doesn't check who currently holds `client_internal_id`, and `NEW_CLIENT` never clears `last_client_internal_id`. So F ends up leader twice over: once from L's original promotion, and again self-confirmed via its own close/reconnect cycle. L's own `REFRESH_CLIENT` reconnect has nothing to match against (`last_client_internal_id` is already spent on F) and never reclaims.
Net effect is the same as I described — L stays a follower after reconnecting — just via this corrected path. Whoever picks this up should verify the exact interleaving with a backend test before fixing it, since I've now gotten a detail wrong twice from code reading alone.
Fix flaky Firefox E2E: eliminate leader/follower race in live-show re…
Tim020 commentedon Sep 24, 2026
Closing as a duplicate of #1419. Both come from the same root cause:
on_closefinalises a departing uuid's state (it deletes the Session row and re-elects the leader) before a reloading client'sREFRESH_CLIENTcan reclaim it. The corrected two-tab interleaving and the A/B evidence from #1425 have been copied into #1419 (see the consolidation comment), and #1419's acceptance checklist includes the two-tab leader case. The fix will be tracked there.Hold WS session state through a reconnect grace window (fixes #1419, #…
Tim020 commentedon Oct 19, 2026
Resolved by #1427 (merged into
devas d9718b8), which fixed the consolidated issue #1419. In the two-tab reload race described here, the original leader tab now keeps leadership in both close orderings:test_two_tab_reload_original_leader_wins_leader_closes_first/..._follower_closes_first. This issue was already closed as a duplicate of #1419.