fix(ws): stash + replay data frames during resubscribe window (#176)#187
Conversation
Fixes silent message loss when the server sends data frames on a freshly-assigned sid before the corresponding subscribe ack reaches `_wait_for_response`. Previously these frames were non-matching from the wait's perspective and discarded with a debug log; under market-burst reconnects on ticker / trade / fill channels, the SDK could drop tens of messages per reconnect. SubscriptionManager now stashes non-matching data frames in a per-sid bounded deque (stash_maxlen=1000 default) for the duration of resubscribe_all. _handle_reconnect drains the stash through _process_frame after resubscribe completes so the frames flow through the normal dispatch path -- seq tracker advances correctly, orderbook manager applies, iterators receive them in arrival order. Coordinates with #139's seq-gap tracking: replayed frames go through seq_tracker.track exactly once, so the first live frame after resubscribe sees the right watermark and doesn't trip a spurious gap. Stash is bounded; overflow evicts oldest via deque maxlen with a single WARNING per fill event. Frames whose sid wasn't re-mapped (per-sub resubscribe failure, F-P-01 path from #77) are dropped on drain with a debug log. 5 unit tests cover the stash mechanics directly (collect-by-sid, no-sid drop, maxlen eviction, take_stash atomicity, default-off flag). 3 integration tests cover the dispatcher integration: - drain replays through dispatch into the iterator queue - drain advances seq_tracker on sequenced channels (orderbook) - drain skips frames whose sid didn't get re-mapped The race-engineered end-to-end test (server sends pre-ack data during a reconnect) is intentionally not included: race timing is hardware-dependent and the value is the post-drain dispatch path, which the direct tests cover deterministically. Existing reconnect tests in this file exercise the close -> reconnect -> resubscribe pipeline.
Code Review — PR #187: stash + replay data frames during resubscribe windowOverviewSolid fix for a real correctness bug. The race is accurately diagnosed: between Bug:
|
Bot caught a real bug: `_maybe_stash`'s `elif len(bucket) == self._stash_maxlen` check is True on EVERY append once the deque is full, so the WARNING fired on every frame after the fill, not once as the comment and PR description claimed. Under a prolonged stall on a high-volume sid, this is per-frame log spam. Fix: track warned sids in a `_stash_warned: set[int]` attribute. The warning fires once per (sid, resubscribe cycle); `take_stash()` clears the set so the next resubscribe gets fresh warnings. Tightened the test: - Renamed to test_maybe_stash_maxlen_evicts_oldest_with_one_warning_per_fill - Pushes 10 frames into a maxlen=3 deque, asserts EXACTLY one WARNING (not >= 1 via `any(...)` as before) - Verifies take_stash() reset by running a second 5-frame cycle and asserting the total warning count goes from 1 -> 2 (not 1+5=6) Also moved `import logging` from inside the test body to module level. asyncio.get_event_loop() deprecation (channels.py:111,113) noted by bot but left alone — pre-existing, out of scope here.
|
Real bug. Fixed in 6eafeab. The overfire — you're right: Fix — added elif len(bucket) == self._stash_maxlen and sid not in self._stash_warned:
self._stash_warned.add(sid)
logger.warning("Stash for sid %d is full (%d frames); …", sid, self._stash_maxlen)
bucket.append(raw)
# in take_stash()
stash, self._stash = self._stash, {}
self._stash_warned.clear()
return stashTightened test — renamed to Other notes:
Verification: 2125 unit tests pass ( |
Code Review — PR #187: stash + replay data frames during resubscribe windowOverviewSolid fix for a real correctness bug. The race is accurately diagnosed: between The Issue 1 — Pre-existing:
|
| Design | ✅ Sound — bounded stash, atomic take, drain before mark_streaming is the right ordering |
_stash_warned fix |
✅ Correctly gates overflow warnings to once per (sid, cycle) |
| Seq-gap coordination | ✅ Drain through _process_frame ensures seq watermark is set before live frames arrive |
asyncio.get_event_loop() |
|
| Stash leak on reconnect failure | clear() call in the finally block as a defensive measure |
| Tests | ✅ Good coverage of mechanics and integration |
The core fix is correct and the PR description is thorough. Address issues 1 and 2 before merge — both are small changes.
Round-2 bot review on PR #187: 1. asyncio.get_event_loop() in _wait_for_response was deprecated since 3.10. Pre-existing, but since this PR already touches the function body it's in-scope. Swapped both calls to asyncio.get_running_loop().time() which is the correct API inside an async def with a guaranteed-running loop. 2. Stash state could survive a resubscribe_all exception. Today the outer _handle_reconnect's except sets _running=False so no next reconnect happens on the same instance, but the invariant should be defensive: if a future code path ever calls resubscribe_all on a manager with prior failed-cycle state, stale stashed frames would replay alongside fresh ones. Added a clear() of both _stash and _stash_warned at the start of resubscribe_all (before _stashing = True). take_stash() still clears them on the happy path; this is purely the failure-mode guard. Regression test added (test_resubscribe_clears_stale_stash_at_start): inject stale _stash + _stash_warned state, run a clean resubscribe_all, verify both are empty regardless of what the per-sub loop did. 2126 unit tests pass. mypy strict + ruff clean.
|
Both addressed in 9767e57. You were right that since this PR already touches 1. 2. Defensive stash clear on Your analysis is right that within the current instance, Added a regression test ( Verification: 2126 unit tests pass. |
Code Review — PR #187: WS resubscribe-window frame stashingOverall: This is a well-reasoned fix for a real race condition. The stash/replay pattern is clean, the memory bound is explicit, and the seq-tracker coordination with #139 is handled correctly. A few small issues below, none of them blockers. What the PR does
CorrectnessThe core logic is correct:
Issues1. Redundant
# tests/ws/test_recv_loop_hardening.py — inside test_resubscribe_stash_skips_unmapped_sids
import logging # ← redundant; already imported at file top2. PR-number comment in production code (CLAUDE.md style violation)
# Defensive reset: if a prior resubscribe cycle raised before
# _drain_resubscribe_stash ... (#187 review)Per CLAUDE.md: "Don't reference the current task, fix, or callers … those belong in the PR description and rot as the codebase evolves." The explanation is valuable — just drop the 3. Incidental The diff silently migrates two calls in - deadline = asyncio.get_event_loop().time() + timeout
+ deadline = asyncio.get_running_loop().time() + timeout
Test coverageGood. The 8 tests cover:
The decision to test drain mechanics directly rather than writing a timing-dependent race test is correct — race tests are flaky by nature and this approach gives full deterministic coverage of the dispatch path. One gap worth noting: there is no test verifying that a normal (non-resubscribe) subscribe still silently drops non-matching frames (i.e., that Performance / memoryMemory bound is SummaryTwo small fixes before merge:
The |
…in CHANGELOG Round-3 bot review on PR #187: 1. Redundant `import logging` inside test_resubscribe_stash_skips_unmapped_sids removed. The module-level import at the top of the file already covers it. 2. Dropped the `(#187 review)` tag from the channels.py defensive-clear comment. The text explanation stays — it's the rationale that matters, not the PR number, which becomes a meaningless reference once merged. Issue numbers (#176, #172, #77) are durable and stay; PR numbers in production code rot. 3. Added a one-line CHANGELOG note for the asyncio.get_event_loop() -> get_running_loop() migration in _wait_for_response so the drive-by doesn't get lost in code review. The bot also suggested an explicit test that _stashing=False keeps the discard path intact. The existing test_stashing_flag_off_by_default already locks in the default state, and the structural invariant (_maybe_stash only called inside the `if self._stashing:` branch in _wait_for_response) is visible in the code. Adding a standalone test for the discard path would require mocking _wait_for_response and the underlying recv() to feed it a non-matching frame; the contract is already locked by the existing test plus code structure. Not adding. 2126 unit tests pass. ruff + mypy strict clean.
|
Addressed in 6b4db52:
Not actioned: the suggested test for " Verification: 2126 unit tests pass. |
Code Review — PR #187: fix(ws): stash + replay data frames during resubscribe windowOverviewThis PR fixes a real, observable bug: data frames sent by the server on a freshly-assigned sid, during the window between What's GoodCorrectness of the core fix
Memory bound
Test coverage
Issues1. Untyped In if not isinstance(sid, int):
logger.debug("Stash mode: dropping frame with non-int sid: type=%s sid=%r", data.get("type"), sid)
returnNot a blocker — the Kalshi protocol always sends integer sids — but it's a cheap defensive check. 2. Self-referential
PR numbers in test docstrings become stale quickly and aren't meaningful to future readers. The what and why are already captured in the assertion comments. Worth trimming before merge. 3. except Exception:
logger.warning("Failed to replay stashed frame for sid %d", sid, exc_info=True)Catching 4. Theoretical: server sid collision across resubscribes — edge case If the server reuses a numeric sid value across two subscriptions (unlikely — sids are typically monotonically increasing), frames stashed under that value could be associated with the wrong subscription after the SummaryThe fix is correct, well-bounded, and thoroughly tested. Issues 2 and 4 are cosmetic/theoretical. Issue 1 is a cheap defensive improvement. Issue 3 is a trade-off judgment call that's reasonably decided in the current direction. Verdict: approve with minor suggestions. None of these block merge; the core race fix is sound. |
Round-4 bot review on PR #187: 1. Untyped sid from data.get("sid"). The stash is typed as `dict[int, ...]` but data is `dict[str, Any]`, so a malformed frame with a non-int sid would silently corrupt the stash. Replaced `if sid is None:` with `if not isinstance(sid, int):` to catch both None and string/float/etc. Added a comment noting the Kalshi protocol's monotonic-int-sid assumption and the within-cycle uniqueness assumption. Renamed the test `test_maybe_stash_drops_frame_without_sid` -> `test_maybe_stash_drops_frame_with_non_int_sid` and added two more cases: string sid and float sid. All drop with empty stash result. 2. Dropped `(#187 review fix)` from `test_maybe_stash_maxlen_evicts_oldest_with_one_warning_per_fill` and `#187 review:` from `test_resubscribe_clears_stale_stash_at_start`. PR refs rot once merged; the test docstrings already explain the what and why without the tag. Not actioned: - Bot's "broad except Exception in _drain_resubscribe_stash" — bot itself noted this is "a trade-off judgment call that's reasonably decided in the current direction." Leaving as-is; a malformed replayed frame shouldn't abort the entire reconnect orchestration. - Bot's "theoretical sid collision across resubscribes" — added a comment in _maybe_stash documenting the within-cycle uniqueness assumption. No code change needed. 2126 unit tests pass. ruff + mypy strict clean.
|
Round 4 addressed in 2dfebc1. 1.
Renamed the test 2. Not actioned:
Verification: 2126 unit tests pass. This is the 4th review round. Each one has found something real (overfire bug → sentinel divergence → close() exception safety → simultaneous-completion → defensive stash clear → now the sid type guard); the implementation is meaningfully tighter than the original push. Happy to keep iterating if there are more substantive items, but otherwise this looks ready to merge. |
Code Review — PR #187: WS resubscribe-window frame stashingOverviewThis fixes a real, observable signal-loss bug: frames sent by the server on a freshly-assigned sid, before the subscribe ack lands in Correctness ✅The core logic is correct:
Issues1. Comment verbosity violates CLAUDE.md conventions (minor, but consistent) CLAUDE.md specifies: "Never write multi-paragraph docstrings or multi-line comment blocks — one short line max." Several new additions exceed this:
The PR description and commit message carry this context better. Suggest trimming each to a single 2. No validation on
3. except Exception:
logger.warning("Failed to replay stashed frame for sid %d", sid, exc_info=True)Continuing on a per-frame failure is the right choice, but if Test Coverage ✅Eight new tests are well-scoped:
The decision to skip a race-engineered end-to-end test is called out and justified. The One gap: no test for Performance ✅Stash is a Security ✅No new attack surface. Stash is internal, bounded, and cleared each cycle. SummaryThe fix is correct and well-tested. Two issues worth addressing before merge:
The per-frame exception handling in |
Bot round-2 found one item to action and one based on a misreading of
convention.
Actioned:
* ROADMAP.md "Open trackers" trimmed from four lines of tombstone prose
("None. The previous entries here (#45, #53, #106) are all closed:
... ") down to "None." The why-they-closed history will rot by the
next release; anyone curious can run `gh issue view 45` or check git
history.
Not actioned:
* Bot suggested changing CHANGELOG.md "Pre-release docs audit (#179)"
to "(#188)" on the claim that "every other CHANGELOG entry uses the
PR number." This is backwards. Existing CHANGELOG entries
(#175, #176, #177, #178, #183) are all *issue* numbers; the PRs that
closed them were #185, #187, #186, #182, #184 respectively. Verified
via `gh issue view <N> --json closedByPullRequestsReferences`. The
convention is "(#<issue-number>)" — using #179 is correct.
mkdocs --strict still clean.
Closes #176.
The race
SubscriptionManager.resubscribe_allclears_sid_to_clientat the start (#77 F-P-03 — prevents stale-sid mis-routing) and rebuilds it as each new sid lands in_wait_for_response. Between those steps, the server can send data frames on the just-assigned sid. The client's_wait_for_responsereads them off the wire, sees a non-matchingid, logs a debug line, and drops them. Under market-burst reconnects onticker/trade/fill, the SDK could lose tens of messages per reconnect with no visible signal.The drop site is
_wait_for_response, not the dispatcher — the recv loop is blocked inside_handle_reconnectduring resubscribe, so_wait_for_responseis the only path reading the socket. The issue body's "reach the dispatcher" framing is a touch off, but the bug is real.The fix
SubscriptionManagergains a per-sid bounded stash, toggled on for the duration ofresubscribe_all:_wait_for_responsebranches on the flag — outside resubscribe (_stashing=False) the existing debug-drop behavior is unchanged; inside resubscribe (_stashing=True) non-matching frames go toself._stash[sid]. Frames without asid(control envelopes) are still dropped — there's nowhere to replay them to.After
resubscribe_allreturns (try/finally guarantees_stashing=False),KalshiWebSocket._handle_reconnectdrains the stash through_process_frameso each frame flows through the normal pipeline:_drain_resubscribe_stashcallstake_stash()(atomic return-and-clear), then for each(sid, raw_frames):await self._process_frame(raw)for each, in arrival order.Going through
_process_frame(not directly through_dispatcher.dispatch) keeps seq tracking and orderbook state consistent — the replayed frames update the seq watermark exactly once, so the next live frame doesn't trip a spurious gap.Bound + degradation
Per-sid deque uses
collections.deque(maxlen=self._stash_maxlen). On overflow:Memory worst-case:
stash_maxlen × len(active_subs) × avg_frame_size. With defaults (1000 per sid), this is ~1 MB per active subscription assuming ~1 KB frames — bounded and proportional to the user's existing subscribe footprint.Coordination with #139 (seq-gap correctness)
The original issue specifically called out that "a stashed-then-replayed frame must still flow through the seq tracker correctly." The drain routes through
_process_framewhich callsseq_tracker.track(sid, seq, channel)exactly once per frame. Becauseseq_tracker.reset_all()is called at the start of_handle_reconnect, the first stashed frame for each sid establishes the baseline watermark. Subsequent live frames track normally against that baseline. No spurious 0 → N gap.Verified by
test_stash_drain_advances_seq_tracker_on_sequenced_channels(orderbook_delta channel; ticker isn't sequenced).Tests
5 unit tests in
tests/ws/test_channels.py::TestResubscribeStashcover the mechanics:test_maybe_stash_collects_by_sid— per-sid arrival-order dequestest_maybe_stash_drops_frame_without_sid— control envelopes have nowhere to replaytest_maybe_stash_maxlen_evicts_oldest— deque maxlen + single WARNING per filltest_take_stash_returns_and_clears— atomic return-and-cleartest_stashing_flag_off_by_default— default state guards against accidental stash3 integration tests in
tests/ws/test_recv_loop_hardening.py::TestResubscribeStashIntegrationcover the dispatcher integration:test_stash_drain_replays_through_dispatch— frame reaches iterator queuetest_stash_drain_advances_seq_tracker_on_sequenced_channels— fix(ws): roll back seq on backpressure + reset all tickers on multi-ticker gap #139 coordinationtest_resubscribe_stash_skips_unmapped_sids— failed-resubscribe sids drop gracefullyThe race-engineered end-to-end test (server sends pre-ack data during a reconnect) is intentionally not included: race timing is hardware-dependent and the value is the post-drain dispatch path, which the direct tests cover deterministically. Existing tests in
test_recv_loop_hardening.pyalready exercise the close → reconnect → resubscribe pipeline (TestResubscribeIsolation,TestReconnectHoldsSubscribeLock).Verification
uv run pytest tests/ --ignore=tests/integration→ 2125 passed, 0 warnings.uv run ruff check .→ clean.uv run mypy kalshi/→Success: no issues found in 76 source files.Soft-breaking footprint
None.
SubscriptionManager.__init__gains a keyword-onlystash_maxlen=1000parameter; positional usage (SubscriptionManager(connection)) is unchanged. New behavior only fires duringresubscribe_all, which is internal-only.Backlog after this lands
Two open issues remaining: #179 (ROADMAP cleanup, pure docs) and whatever the next nightly catches. The
run_forever()hardening trio (#175, #177) + this stash fix make the WS recv path correctness-complete against the F-P / F-R audit findings.