Skip to content

Warm combo boards in serve and name the legs a venue lacks - #40

Merged
YukiC0804 merged 1 commit into
codex/order-lifecycle-storagefrom
sp/combo-warm
Oct 22, 2026
Merged

YukiC0804 merged 1 commit into
codex/order-lifecycle-storagefrom
sp/combo-warm

Conversation

@YukiC0804

Copy link
Copy Markdown
Collaborator

Experimental line only (codex/order-lifecycle-storage), not for main.

What changes

  • Warm combo boards. serve builds every combo venue's board in the background at start-up and every refresh_interval_s ([combo_board] warm, refresh_interval_s = 60, pause_s = 0.1). It keeps the last good board, so the board route never waits on a build. While the first build runs, the route answers status: "building" at once.
  • The board in parts. New GET /trading/combos/board/categories for the category rail. The board route now takes category, cursor, limit, q and sort; the whole-board call still works.
  • The whole catalogue. Polymarket's combo catalogue is about 214,000 markets (214 pages), not 5,000. Gamma reads are cached between builds (BoardCache), and a first build publishes partial boards after page 5 and then every 40 pages.
  • Clear skip reasons. An RFQ group matches every leg before ruling a venue out. Each skip lists missing_legs (market_id, label, code, mapped_id, detail), with codes missing_leg, matching_error, ineligible_leg, and the new outside_collections for Kalshi. Labels come from the warm board ("Al Fayha to win"), else a shortened catalogue question.
  • The build doesn't stall trading. Its CPU steps run in worker threads that hand back the GIL every 100 items, the GIL switch interval is 0.5 ms while boards are warmed, and board objects are frozen out of the cyclic collector (gc.freeze) as they are built. Profiling showed full garbage collections over the board's objects were the main stall (0.3-1.4 s each).

No public Python interface from #39 is renamed or used differently here.

Measured on the real Polymarket catalogue

Result
Categories call 2 ms, ~1 KB
One 40-event page 3-25 ms
First partial board about 25 s after start-up
Full build 213,880 markets, 28,813 events; 489 s, peak memory 2.3 GB
Refresh using the cache 77 s
Serve requests during a build before this fix 1-6 s, so the terminal reported "Engine offline"; after it, worst 0.71 s over 5 minutes, no dropouts

Concerns

  • Peak memory is 2.3 GB while the whole board builds, too much for a small host. [combo_board] warm = false turns warming off.
  • A terminal backend without the paged board would pull the whole 22 MB board.

Tests

  • Full suite: 3,267 passed, 199 skipped.
  • New tests/test_combo_warmer.py (16 tests). Its Gamma rows are real, saved in tests/samples/polymarket_combo_skip_legs_live.json.

The terminal side is on Synpath-ai/prediction_market#104.

🤖 Generated with Claude Code

serve now builds every combo venue's board in the background at start-up and
again every refresh_interval_s ([combo_board] warm, refresh_interval_s,
pause_s), keeping the last good board, so the board route never waits on a
build. The board is served in parts: GET /trading/combos/board/categories
for the rail, and category / cursor / limit / q / sort pages on the board
route. Polymarket's catalogue is about 214,000 markets; Gamma reads are
cached between builds, and a first build publishes partial boards.

The build stays off the event loop that answers health checks and orders:
its CPU steps run in worker threads that hand back the GIL every 100 items,
the GIL switch interval drops to 0.5 ms while boards are warmed, and the
board's objects are frozen out of the cyclic collector as they are built.
Requests stay under 0.8 s during a whole-catalogue build (before: 1-6 s).

An RFQ group now matches every leg before skipping a venue, and each skip
lists missing_legs (market_id, label, code, mapped_id, detail), with codes
missing_leg, matching_error, ineligible_leg and outside_collections; labels
come from the warm board, else a shortened catalogue question.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@YukiC0804
YukiC0804 merged commit 8501387 into codex/order-lifecycle-storage Oct 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant