fix(ws): recv-loop overhaul — reconnect races, narrowed exceptions, log scrub, ConnectionManager API#138
Conversation
…ption handling Unified rewrite of the WebSocket recv-loop exception-handling region. The five reconnect/resubscribe race fixes (#77) and the narrowed broad-except (#83) all touch the same code region, so they land together. #77 — five race fixes: - F-P-01 `resubscribe_all` now isolates per-sub failures: each resub is wrapped in try/except; on failure the failed sub gets a sentinel, is removed from `_subscriptions`, and the loop continues. Previously one failed resub aborted the whole reconnect and silently terminated every iterator. - F-P-03 the reconnect path now acquires `_subscribe_lock` for the entire reconnect-and-resubscribe sequence, so concurrent user-level `subscribe_*` calls can't race the sid-remap and mis-route in-flight messages. `_sid_to_client` is cleared before any resubscribe is sent. - F-P-04 the recv loop now uses `asyncio.shield` around the recv->dispatch critical section. A pause/cancel during dispatch no longer drops the in-flight frame; the shielded coroutine completes before the outer cancellation is honored. - F-P-05 `_wait_for_response` catches `websockets.ConnectionClosed` and re-raises as `KalshiConnectionError`, matching the documented SDK exception hierarchy. - F-P-08 `unsubscribe` pushes a sentinel into the sub's queue before cleaning up mappings, so any held iterator exits cleanly via `StopAsyncIteration` instead of hanging on `queue.get()` forever. #83 — narrowed recv-loop exception handling: - Re-raise / break+sentinel-broadcast for `KalshiBackpressureError` and `KalshiSubscriptionError`. These signal real consumer-visible problems and should propagate. - Log+continue with `exc_info=True` only for the genuinely-non-fatal classes: `json.JSONDecodeError`, `pydantic.ValidationError`, `KeyError`. Tracebacks are now visible for debugging. - `asyncio.CancelledError` propagates so the loop exits cleanly. Regression coverage: 7 new tests in `tests/ws/test_recv_loop_hardening.py`. Race scenarios use `asyncio.Event` barriers, not sleeps, so they're deterministic. Closes #77 Closes #83 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
httpx exception strings include the full request URL with query string.
Interpolating them into `KalshiError(f"...: {e}")` leaks any token-like
query parameter into uncaught-exception sinks (Sentry, stderr, log
infra). Anyone constructing a URL with a sensitive query value would
have it surface in error reports.
Fix: drop the `{e}` interpolation. Use a fixed message and rely on
`raise ... from e` so the underlying exception is still available via
`__cause__` for debugging. Most error trackers serialize `__cause__`
anyway.
Touched:
- `kalshi/_base_client.py`: `KalshiError("Request timed out")` and
`KalshiError("HTTP error")` for sync + async transports.
- `kalshi/ws/connection.py`: `KalshiConnectionError("WebSocket
connection failed")` in `connect()`.
Regression tests assert that a sensitive-looking query token and the
hostname are NOT in `str(error)`.
Refs #84 (F-O-09 only; F-O-05 — dispatch.py `exc_info=True` trade-data
leakage — is in the dispatcher-correctness sibling branch's scope and
must close #84 fully when both branches land).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The recv loop's reconnect path was calling `self._connection._set_state( ConnectionState.STREAMING)` directly, reaching into ConnectionManager's name-mangled internal state machine. Leaky abstraction — a refactor that renamed `_set_state` would silently break the streaming-state report, and `on_state_change` fired from the "wrong" caller. Fix: add a public `ConnectionManager.mark_streaming()` that performs the CONNECTED -> STREAMING transition. The recv loop calls that instead. Closes #88 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code Review — PR #138 (Wave 3 recv-loop overhaul)Overall this is a well-scoped PR with clear issue traceability, good test coverage, and clean 🔴 Correctness —
|
Per bot review on PR #137: - dispatch.py: `if "error" in data:` -> `if data.get("error") is not None:` so a legitimate `"error": null` field on a typed envelope doesn't trip the channel-level error path. - dispatch.py _surface_channel_error: when ErrorMessage.model_validate fails, fall back to ErrorMessage.model_construct(...) so a registered on_error handler still fires with the raw payload. Log level escalates WARNING -> ERROR (we almost missed an alert). - tests/ws/test_dispatch.py: * Module-level imports for `logging` and `SequenceTracker` (was inline inside test methods). * FakeSubManager.add() now accepts an optional `client_id` so tests can exercise the production case where server-assigned `sid !=` client-side id. * Added test_server_unsubscribe_with_distinct_client_id covering the two-step lookup that was previously hidden by sid == client_id. * Added test_null_error_field_not_misrouted for the get/!=None guard. * Added test_channel_level_error_validation_failure_still_fires_handler for the model_construct fallback. - CHANGELOG.md: [Unreleased] -> Changed entry calling out the #80 fan-out as a behavioral change so users upgrading see the signal. Deferred (tracked elsewhere): - _sid_to_client / _subscriptions private-access layer violation: bot rightly suggests SubscriptionManager.remove_by_sid(); channels.py is the recv-loop branch's territory (#138). Track in the post-Wave-3 integration cleanup. Verify: tests/ws/test_dispatch.py 30 passed. ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per bot review on PR #137: - dispatch.py: `if "error" in data:` -> `if data.get("error") is not None:` so a legitimate `"error": null` field on a typed envelope doesn't trip the channel-level error path. - dispatch.py _surface_channel_error: when ErrorMessage.model_validate fails, fall back to ErrorMessage.model_construct(...) so a registered on_error handler still fires with the raw payload. Log level escalates WARNING -> ERROR (we almost missed an alert). - tests/ws/test_dispatch.py: * Module-level imports for `logging` and `SequenceTracker` (was inline inside test methods). * FakeSubManager.add() now accepts an optional `client_id` so tests can exercise the production case where server-assigned `sid !=` client-side id. * Added test_server_unsubscribe_with_distinct_client_id covering the two-step lookup that was previously hidden by sid == client_id. * Added test_null_error_field_not_misrouted for the get/!=None guard. * Added test_channel_level_error_validation_failure_still_fires_handler for the model_construct fallback. - CHANGELOG.md: [Unreleased] -> Changed entry calling out the #80 fan-out as a behavioral change so users upgrading see the signal. Deferred (tracked elsewhere): - _sid_to_client / _subscriptions private-access layer violation: bot rightly suggests SubscriptionManager.remove_by_sid(); channels.py is the recv-loop branch's territory (#138). Track in the post-Wave-3 integration cleanup. Verify: tests/ws/test_dispatch.py 30 passed. ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per bot review on PR #138: 🔴 Real correctness bug: - recv loop's `asyncio.shield(self._process_frame(raw))` created a detached background task. On outer cancel, `CancelledError` bubbled, the loop `break`'d, and dispatch continued running orphaned — racing any subsequent subscribe. Fixed by holding the inner task in a local and `await inner` on cancel before breaking, so loop exit is strictly post-dispatch. 🟡 Observability: - `_handle_reconnect` "Reconnect failed: %s" log now uses `exc_info=True` so the traceback isn't swallowed on TCP timeouts inside resubscribe_all etc. 🟡 F-O-09 debug context: - KalshiError("Request timed out") -> KalshiError(f"Request timed out: {method.upper()} {path}"). Same for "HTTP error". method+path carry no secrets and are essential debug context. Tightened existing tests to assert "POST" and "/portfolio/orders" (etc.) ARE in the message while SUPER_SECRET_TOKEN and kalshi.com are NOT. - KalshiConnectionError("WebSocket connection failed") -> includes the ws path (urlparse strips query) for the same reason. 🟡 Inline imports: - channels.py: `from kalshi.errors import KalshiSubscriptionError` was repeated 4× inside method bodies (in `subscribe`, `unsubscribe`, `resubscribe_all`, `_handle_server_unsubscribe`). Moved to top of file alongside KalshiConnectionError. 🟠 Dead indirection: - Dropped `_resume_recv_loop()` — pure one-liner wrapper around `_ensure_recv_loop()`. Call site in `_do_subscribe` calls `_ensure_recv_loop()` directly. Test updated. Skipped (optional polish): - mark_streaming() docstring tweak (works from any state today; adding a state guard would be a behavior change, not just doc fix). - Replacing 3 remaining `asyncio.sleep(0.05)` test syncs with events — flagged for future cleanup, low CI-flake risk today. Verify: 1631 passed, 48 skipped. ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Five items addressed in latest commit:
Skipped (justified):
Verify: 1631 passed, 48 skipped. |
Code Review — PR #138: recv-loop overhaul (Wave 3)Overall: This is a well-scoped, well-documented fix for five real races and two security/observability issues. The asyncio patterns (shield + explicit 🔴 Correctness (low probability, real edge case)
inner = asyncio.ensure_future(self._process_frame(raw))
try:
await asyncio.shield(inner)
except asyncio.CancelledError:
await inner # <-- re-raises any exception from _process_frame
breakIf Fix: except asyncio.CancelledError:
try:
await inner
except Exception:
pass # best-effort; loop is exiting anyway
break🟡 Observability
except Exception as e:
logger.warning(
"Resubscribe failed for client_id=%d channel=%s: %s",
client_id, sub.channel, e,
)The Suggested: logger.warning(
"Resubscribe failed for client_id=%d channel=%s",
client_id, sub.channel,
exc_info=True,
)🟢 Minor style / doc nits
Python name mangling applies only to Extra blank line in if data.get("type") == "error":
# <-- extra blank line
error_msg = data.get("msg", {})Minor inconsistency with surrounding style. 🟢 Testing (acknowledged, low urgency)
✅ Confirmed correct
The shield edge case (🔴) is the only item worth addressing before merge. Everything else is polish. |
…aceback Per second-pass bot review on PR #138: 🔴 Rare exception path inside cancel handler: - `await inner` inside `except asyncio.CancelledError` could re-raise a dispatch exception (json.JSONDecodeError, pydantic.ValidationError, etc.) right back into the cancel handler — no handler, recv loop crashes mid-shutdown instead of cleanly breaking. Wrapped the `await inner` in a best-effort try/except, logging at DEBUG. Timing window is narrow (malformed frame + concurrent pause during dispatch) but genuine. 🟡 resubscribe_all log loses traceback on unexpected exceptions: - Was `: %s, e` which only prints the message — fine for the expected KalshiSubscriptionError, but an AttributeError / programming bug would have no traceback. Switched to `exc_info=True` matching the pattern already used in `_handle_reconnect`. 🟢 Doc terminology: - mark_streaming() docstring said "name-mangled `_set_state`". Single-underscore is private-by-convention, not mangled (mangling is only for __double_underscore). Changed to "private `_set_state`". Skipped (justified): - Extra blank line in channels.py — pure style, no behavior signal. - Replacing 3 `asyncio.sleep(0.05)` test syncs with event barriers — already acknowledged in the prior commit as a low-urgency future sweep. Verify: tests/ws/ 215 passed. ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Three items addressed:
Skipped (justified):
Verify: |
Code Review — PR #138: recv-loop overhaulOverall this is a high-quality, well-scoped PR. The races are each fixed with the right primitive (lock, shield, sentinel, isolation), the exception-narrowing is long overdue, and the test coverage is solid. A few things worth discussing before merging. Issues1. Unexpected exceptions from
|
…tion 🔴 Real bug, third bot pass: The recv loop's exception ladder caught KalshiBackpressureError / KalshiSubscriptionError / json.JSONDecodeError / ValidationError / KeyError. Anything else (AttributeError from a user callback, etc.) propagated out of the loop and the recv task died WITHOUT broadcasting sentinels — consumers hung forever on their queues. Added a final `except Exception:` after the explicit handlers: - Logs at ERROR with traceback - Iterates active subscriptions, pushes sentinel on each queue - Re-raises so the task failure is still visible to anyone awaiting it Regression test (test_unexpected_exception_broadcasts_sentinels_before_raising): monkey-patches dispatcher.dispatch to raise AttributeError, asserts the iterator sees StopAsyncIteration (sentinel) AND the recv_task ends with an AttributeError exception. 🟦 Style cleanup (4 sites): channels.py had 4 orphaned blank lines at the start of `if` blocks — artifacts of removing the inline `from kalshi.errors import KalshiSubscriptionError` imports last commit. Fixed in `_wait_for_response`, `subscribe`, `update_subscription`, and `resubscribe_all`. Skipped (justified): - `assert self._dispatcher is not None` vs runtime guard — bot called judgment call; assert matches the file's pattern, change is unnecessary risk in a hot path. - "Reconnect failure leaves WS permanently dead" — true and intentional; surfaced through `_running = False` + `if not self._running: break`. Will doc in CHANGELOG at release-cut. Verify: tests/ws/test_recv_loop_hardening.py 8 passed (+1). ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Three items addressed:
Skipped (justified):
Verify: |
Code Review — PR #138: recv-loop overhaulOverall this is a solid, well-scoped PR. The issue decomposition is clear, the commit boundaries are logical, and the test coverage is strong. A few things worth addressing before merge. What's working well
Issues1.
|
…reliability Per fourth-pass bot review on PR #138: 🔴 Sentinel gap in CancelledError cleanup: Last pass added best-effort try/except Exception around `await inner` inside the CancelledError handler. But if dispatch raises KalshiBackpressureError or KalshiSubscriptionError DURING cancellation, they were silently swallowed without broadcasting sentinels — consumers hung. The sentinel-before-exit invariant must hold unconditionally. Split the except into two: the fatal SDK errors still broadcast sentinels even on the cancel path; only generic Exception is best-effort logged. 🟦 ensure_future -> create_task: Consistency with the rest of the file (line 131 uses create_task). ensure_future has been soft-deprecated since 3.10 in favor of create_task for the create-task-from-coroutine case. 🟦 Test reliability — F-P-03 lock test: test_handle_reconnect_acquires_subscribe_lock used `await asyncio.sleep(0.05)` before asserting `not reconnect_task.done()`, which would pass vacuously on a heavily-loaded runner where the task never even ran. Replaced with a deterministic poll on `session._subscribe_lock._waiters`, wrapped in asyncio.wait_for(2.0), so a real regression fails loudly within bounded time rather than passing by accident. Skipped (justified): - asyncio.get_event_loop().time() -> get_running_loop().time() — pre-existing in channels.py:_wait_for_response, flagged for a future sweep. Not introduced by this PR. - assert vs runtime guard — matches file convention. - Comment about Exception correctly excluding CancelledError — covered by the existing block comment above the except handlers. Verify: tests/ws/test_recv_loop_hardening.py 8 passed. ruff + mypy --strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Three items addressed:
Skipped (justified):
Verify: |
Code Review — PR #138: recv-loop overhaulThis is a large, high-impact PR fixing real concurrency hazards. The overall engineering quality is high: the races are correctly identified, the fixes are targeted, and the tests are genuinely deterministic (events/barriers rather than sleeps in most cases). Below is specific feedback by area.
|
…r envelope surface (#137) * fix(ws): fan out messages to both callback and iterator on same channel When a user registered an @on() callback AND an iterator subscription for the same channel, the dispatcher routed exclusively to the callback and the iterator's queue silently stayed empty (async for ... would hang on queue.get() forever). Change MessageDispatcher.dispatch() to fan out: deliver to the callback (if any) AND the subscription queue. Add a WARNING at register_callback() when an active subscription already exists for the channel so the fan-out behavior is visible. Closes #80 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ws): reap sid mappings on server-initiated unsubscribe When the server unilaterally dropped a subscription (admin action, session expiry, channel close), the dispatcher logged the "unsubscribed" envelope and dropped it on the floor. SubscriptionManager._sid_to_client and _subscriptions retained stale entries; iterators blocked on the queue forever; long-running sessions slowly leaked. Add MessageDispatcher._handle_server_unsubscribe(): pop the sid mapping and the subscription, reset SequenceTracker state for the sid (when injected), and push a sentinel so any held async-for iterator exits cleanly. Accept the envelope in either shape: sid at the top level or nested under msg.sid. The SequenceTracker reset path is gated on an optional constructor injection (seq_tracker=None by default). The client.py wiring to pass the live tracker in is owned by the sibling recv-loop branch (#77/#83/#84/#86/#88) per Wave 3 boundaries; this commit makes the dispatcher ready to receive it. Closes #81 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ws): surface channel-level error envelopes via on_error AsyncAPI permits server errors to ride on a typed message envelope (an `error` field on an otherwise-typed payload) rather than the top-level type="error" form. The dispatcher previously dropped these on the floor through the unknown-type log path, leaving the user no signal. Add MessageDispatcher._surface_channel_error(): when a non-control envelope carries an `error` field, coerce it to ErrorMessage and route to the registered on_error handler. If no handler is registered, log the full envelope at WARNING (was: silently dropped). Errors with a sid the dispatcher no longer recognizes are surfaced the same way. Closes #82 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review(#137): tighten error-routing guard + on_error fallback + tests Per bot review on PR #137: - dispatch.py: `if "error" in data:` -> `if data.get("error") is not None:` so a legitimate `"error": null` field on a typed envelope doesn't trip the channel-level error path. - dispatch.py _surface_channel_error: when ErrorMessage.model_validate fails, fall back to ErrorMessage.model_construct(...) so a registered on_error handler still fires with the raw payload. Log level escalates WARNING -> ERROR (we almost missed an alert). - tests/ws/test_dispatch.py: * Module-level imports for `logging` and `SequenceTracker` (was inline inside test methods). * FakeSubManager.add() now accepts an optional `client_id` so tests can exercise the production case where server-assigned `sid !=` client-side id. * Added test_server_unsubscribe_with_distinct_client_id covering the two-step lookup that was previously hidden by sid == client_id. * Added test_null_error_field_not_misrouted for the get/!=None guard. * Added test_channel_level_error_validation_failure_still_fires_handler for the model_construct fallback. - CHANGELOG.md: [Unreleased] -> Changed entry calling out the #80 fan-out as a behavioral change so users upgrading see the signal. Deferred (tracked elsewhere): - _sid_to_client / _subscriptions private-access layer violation: bot rightly suggests SubscriptionManager.remove_by_sid(); channels.py is the recv-loop branch's territory (#138). Track in the post-Wave-3 integration cleanup. Verify: tests/ws/test_dispatch.py 30 passed. ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Wave 3, the largest unit of work in v1.2. Three commits:
39aa3bdc0f9897KalshiError.__str__no longer leaks request URLsa4dc672ConnectionManager.mark_streaming()public API replaces recv-loop's_set_statereach-throughTest delta: 1620 → 1631 passing (+11). 48 skipped.
ruffclean,mypy --strictclean.Unified #77 + #83 fix (
39aa3bd)5 races (#77) — each addressed inside one cohesive recv-loop rewrite:
resubscribe_allper-sub isolation — each resub wrapped in try/except, failures push sentinel + drop the sub + continue (was: abort whole reconnect on first failure)._handle_reconnectholds_subscribe_lockfor the entire reconnect+resubscribe sequence; concurrent usersubscribe_*can no longer race the sid-remap._sid_to_clientcleared before sends.asyncio.shieldaround the recv→dispatch critical section (_process_frame). Cancel during dispatch no longer drops the in-flight frame._wait_for_responsecatcheswebsockets.ConnectionClosedand re-raises asKalshiConnectionError.unsubscribepushes sentinel before mapping cleanup.Narrowed exception handling (#83):
KalshiBackpressureError,KalshiSubscriptionErrorasyncio.CancelledErrorjson.JSONDecodeError,pydantic.ValidationError,KeyErrorexc_info=Trueexcept Exception)exc_info=Truedumps trade data;KalshiErrorstr() includes URLs #84 F-O-05 (dispatch.py exc_info=Trueleaking trade data) — lives inkalshi/ws/dispatch.py, owned by the sibling dispatcher-correctness branch. This PR's commit isRefs #84, notCloses #84, so the issue stays open until the sibling branch lands its half. The hardcoded URL leak (F-O-09) IS closed here.dispatch.py. The recv loop already has the parseddatadict ready to forward; the sibling branch can take it from here. Issue stays open.Wave 4 compatibility
_recv_loopalready surfacesKalshiBackpressureErrorvia break+sentinel — Wave 4 extends this._process_frameper sid; the shielded structure is per-frame so Wave 4 can change the gap handler without races.Wave 3 boundaries
No edits to
kalshi/ws/dispatch.pyorkalshi/ws/orderbook.py(sibling branches). One narrow exception: the recv loop calls intoOrderbookManager.apply_*and the dispatcher — no signature changes, those sibling branches can evolve freely.Closes #77
Closes #83
Refs #84
Closes #88
Test plan
tests/ws/test_recv_loop_hardening.py(new) covering all 5 racestests/test_client.py+tests/ws/test_connection.pyfor URL-leakage regressiontests/ws/test_connection.pyformark_streaming()public surfaceuv run ruff check .cleanuv run mypy kalshi/clean (strict)