Realtime WS tick stream silently freezes (get_current_price stale for minutes) while REST /History/retrieveBars stays live — TopStepX/ProjectX Gateway

[finiflex]

Environment

project-x-py	3.5.9 (installed from PyPI)
Python	3.12.10
OS	macOS 13 (darwin 22.6.0)
Broker / gateway	TopStepX (ProjectX Gateway)
Instrument	ES / CON.F.US.EP.U26 (front-month E-mini S&P)
Feature set	TradingSuite.create(instruments=["ESU26"], timeframes=["1min"])

Summary

During live trading sessions the market-data WebSocket tick stream silently stops updating for extended periods (observed 90 s to 6+ minutes at a time) while the connection reports itself healthy. During these windows:

• await data_manager.get_current_price() keeps returning the exact same value (frozen tick), because current_tick_data stops receiving updates.
• The realtime bar cache (get_data("1min")) also stops advancing / produces flat bars.
• No exception is raised and no disconnect callback fires — the feed looks connected.
• Meanwhile the REST endpoint is completely fresh: client.get_bars(..., partial=True) (which POSTs to /History/retrieveBars with live: False) returns up-to-the-second bars for the same contract.

This makes any strategy that relies on get_current_price() / realtime bars go "blind" and miss valid signals, even though the data is clearly available from the same account via REST.

Evidence (from a multi-day run)

• 848 log events where the live tick was detected frozen (value unchanged ≥ 90 s during active RTH/Globex hours).

• 432 feed-heal attempts and 2805 reconnect-related log events across ~112k log lines.

• Representative log line during a freeze (RTH, liquid market):

  Scan 13:33 ET | bar 13:25 ET | flat | 7568.75 | entry blocked — frozen live tick (93s unchanged @ 7568.75)
  

• At the same wall-clock moments, a fresh TradingSuite / client.get_bars() call returned bars that had moved several points away from the frozen value — i.e. the market was actively trading, only the WS stream was stuck.

Minimal reproduction

The freeze is intermittent (broker-side), but the divergence is trivial to observe: compare the WS price to a REST fetch for the same contract in the same process. When the stream is frozen, ws_px stays pinned while rest advances.

import asyncio
from project_x_py import TradingSuite

async def main():
    suite = await TradingSuite.create(instruments=["ESU26"], timeframes=["1min"])
    dm = suite["ESU26"].data
    for _ in range(120):  # ~10 min
        ws_px = await dm.get_current_price()                      # WS tick / bar-close fallback
        df = await suite.client.get_bars(                         # REST /History/retrieveBars
            "ESU26", days=1, interval=1, unit=2, limit=2, partial=True
        )
        rest_px = float(df["close"][-1]) if df is not None and not df.is_empty() else None
        print(f"ws={ws_px}  rest={rest_px}  diff={None if rest_px is None else round(rest_px-ws_px,2)}")
        await asyncio.sleep(5)
    await suite.disconnect()

asyncio.run(main())

Expected: ws and rest track each other.
Observed during a freeze: ws is pinned to one value for minutes while rest keeps moving (diff grows to several points), then ws eventually "jumps" to catch up after a reconnect.

Questions / asks

1. Is there a supported way to detect a silent WS stall from the SDK (e.g. a "last tick received" timestamp, a staleness/heartbeat callback, or a health property) rather than diffing get_current_price() ourselves?
2. Does the SDK expose (or could it expose) a hook to auto-fail-over to REST for get_current_price() when the WS stream stalls?
3. When a reconnect happens, is current_tick_data / the bar cache guaranteed to be backfilled so get_data() is gap-free, or should we force a REST reload after every reconnect?
4. Any recommended TradingSuite / connection-management settings (keepalive, ping interval, resubscribe policy) to reduce these silent stalls with the ProjectX Gateway?

Workaround we implemented (in case it helps others)

• A frozen-tick detector: track the last value + wall-clock of the last change from get_current_price(); if it hasn't changed for N seconds during active hours, treat the feed as stalled.
• A REST-polling price fallback: while the tick is frozen, source the chart/price from client.get_bars(contract, interval=1, unit=2, partial=True) (throttled + rate-limited) so the strategy keeps evaluating on fresh data.
• An external auto-restart watchdog for cases where the stall persists past a max threshold.

Happy to provide anonymized logs or run additional diagnostics. Thanks for maintaining the library.