Summary
Intermittent heap corruption (Windows fatal exception: code 0xc0000374, STATUS_HEAP_CORRUPTION) when a multi-instrument TradingSuite (with features=["orderbook"]) disconnects — i.e. on reconnect, restart, or the daily CME session-boundary feed rollover. The process dies hard; a scheduler/supervisor sees a non-zero exit and respawns.
Version: project-x-py 3.5.9 (Windows, Python 3.14). Config: TradingSuite.create(instruments=[...4 index futures...], timeframes=["1min","5min","15min","30min","1hr"], features=["orderbook"]).
Faulthandler stack at the crash
bounded_statistics.py: get_statistics
bounded_statistics.py: _cleanup_counters
bounded_statistics.py: cleanup_bounded_statistics
realtime_data_manager/core.py: cleanup # gated by `if self.use_bounded_statistics:`
trading_suite.py: _disconnect_single_context
Concurrently, other threads in the same dump are inside realtime_data_manager/data_processing.py _update_timeframe_data / _process_tick_data (i.e. live SignalR tick dispatch).
Root cause (teardown ordering race)
At disconnect, RealtimeDataManager.cleanup() runs while the shared SignalR reader thread is still alive, because the realtime transport is stopped last:
TradingSuite.disconnect() → _disconnect_instrument_contexts() runs each context's data.cleanup(), which (a) calls cleanup_bounded_statistics() → _cleanup_counters() sweeping the per-counter stats under an asyncio lock, and (b) clears the Polars frames / tick deques.realtime.disconnect() (which actually stops the SignalR user+market hubs and joins the receive thread) is only awaited after those per-context cleanups.
An asyncio lock does not coordinate the SignalR worker threads, so a trade/quote can be dispatched into a manager whose stats/frames are being torn down → free-during-read → native heap corruption. It surfaces intermittently (only when a tick lands in the teardown window) and always at a disconnect.
Suggested fix
Stop the realtime transport first, so the reader threads are joined before any per-context state is freed:
- In
TradingSuite.disconnect(), await self.realtime.disconnect() (or the equivalent hub-stop) before _disconnect_instrument_contexts() / the per-context data.cleanup(). realtime.disconnect() preserves callbacks/subscriptions, so moving it earlier is state-safe.
A secondary hardening: the disconnect-time cleanup_bounded_statistics() sweep does no useful work at teardown (the counters are about to be GC'd) — skipping it there removes one more concurrent reader of shared state. The periodic in-session cleanup (via the CleanupScheduler) is a separate path and can stay.
Workaround (for anyone else hitting this)
Downstream, we invert the order in our own broker wrapper — await suite.realtime.disconnect() before suite.disconnect() — and separately monkeypatch BoundedStatisticsMixin.cleanup_bounded_statistics to drop the disconnect-time counter sweep. Both eliminate the crash in our testing across repeated reconnects/rollovers.
Happy to open a PR with the reorder if that's useful.
Summary
Intermittent heap corruption (
Windows fatal exception: code 0xc0000374,STATUS_HEAP_CORRUPTION) when a multi-instrumentTradingSuite(withfeatures=["orderbook"]) disconnects — i.e. on reconnect, restart, or the daily CME session-boundary feed rollover. The process dies hard; a scheduler/supervisor sees a non-zero exit and respawns.Version: project-x-py 3.5.9 (Windows, Python 3.14). Config:
TradingSuite.create(instruments=[...4 index futures...], timeframes=["1min","5min","15min","30min","1hr"], features=["orderbook"]).Faulthandler stack at the crash
Concurrently, other threads in the same dump are inside
realtime_data_manager/data_processing.py_update_timeframe_data/_process_tick_data(i.e. live SignalR tick dispatch).Root cause (teardown ordering race)
At disconnect,
RealtimeDataManager.cleanup()runs while the shared SignalR reader thread is still alive, because the realtime transport is stopped last:TradingSuite.disconnect()→_disconnect_instrument_contexts()runs each context'sdata.cleanup(), which (a) callscleanup_bounded_statistics()→_cleanup_counters()sweeping the per-counter stats under an asyncio lock, and (b) clears the Polars frames / tick deques.realtime.disconnect()(which actually stops the SignalR user+market hubs and joins the receive thread) is only awaited after those per-context cleanups.An asyncio lock does not coordinate the SignalR worker threads, so a trade/quote can be dispatched into a manager whose stats/frames are being torn down → free-during-read → native heap corruption. It surfaces intermittently (only when a tick lands in the teardown window) and always at a disconnect.
Suggested fix
Stop the realtime transport first, so the reader threads are joined before any per-context state is freed:
TradingSuite.disconnect(),await self.realtime.disconnect()(or the equivalent hub-stop) before_disconnect_instrument_contexts()/ the per-contextdata.cleanup().realtime.disconnect()preserves callbacks/subscriptions, so moving it earlier is state-safe.A secondary hardening: the disconnect-time
cleanup_bounded_statistics()sweep does no useful work at teardown (the counters are about to be GC'd) — skipping it there removes one more concurrent reader of shared state. The periodic in-session cleanup (via theCleanupScheduler) is a separate path and can stay.Workaround (for anyone else hitting this)
Downstream, we invert the order in our own broker wrapper —
await suite.realtime.disconnect()beforesuite.disconnect()— and separately monkeypatchBoundedStatisticsMixin.cleanup_bounded_statisticsto drop the disconnect-time counter sweep. Both eliminate the crash in our testing across repeated reconnects/rollovers.Happy to open a PR with the reorder if that's useful.