fix(data): chunk poll_once's catch-up request under Coinbase's candle cap

[eaitbrahim]

The bug

poll_once requested its entire catch-up range in a single get_candles call:

fetch_start = last_ts + gran_sec if last_ts is not None else latest_closed
fetched = client.get_candles(product_id, granularity, fetch_start, latest_closed)

Coinbase rejects any range over ~350 candles:

400 INVALID_ARGUMENT ... "number of candles requested should be less than 350"

Why it can't recover on its own

The request size is a function of how stale the product is, and staleness only grows while the poll is failing. Every subsequent poll therefore asks for a strictly larger range and fails the same way — there is no path back to a legal request without intervention.

The live instance: ZEC-USD at ONE_HOUR sat 552 hours stale, i.e. a ~552-candle request, and had been failing hourly for two days, widening by ~24 candles a day. The other 18 products were current, so nothing else masked or explained it.

Note the empty-repo case was never affected: with last_ts is None, fetch_start = latest_closed, so exactly one candle is requested. Wedging requires cached-but-stale candles.

The fix

poll_once now pages [fetch_start, latest_closed] forward in windows of at most MAX_CANDLES_PER_REQUEST candles, importing the existing constant from keel.data.history rather than declaring a second one — a duplicated venue limit is the same bug class being fixed here.

The new _poll_catch_up helper mirrors history._fill_forward's windowing idiom so the two read the same way, with two deliberate differences:

• windows are sized (MAX - 1) * gran_sec, so the inclusive [start, end] range is at most MAX candles (_fill_forward's + MAX * step is one over, harmless under the ~350 cap but not a property worth copying);
• an empty window does not break the loop, since a mid-history hole must not block catch-up of newer candles.

Each window is upserted as it arrives, so a failure partway through leaves the earlier windows persisted and the next poll resumes further along instead of restarting.

cb_client.get_candles is deliberately unchanged — it correctly passes start/end through, and batching belongs in the caller.

Preserved exactly: the small-range path (still a single request), the empty-repo path (still one candle at latest_closed), the last_ts >= latest_closed no-op, and the return value's meaning.

Tests (written first, confirmed failing for the right reason)

Against the old code the three new chunking tests fail on the oversized single call — assert 552 <= 300 and assert 1 > 1 — not on any import or fixture error:

FAILED test_poll_once_chunks_a_gap_larger_than_the_coinbase_cap
FAILED test_poll_once_never_requests_more_than_the_candle_cap
FAILED test_poll_once_chunk_windows_are_contiguous_and_non_overlapping
3 failed, 12 passed

They reproduce the real 552-hour ZEC-USD gap and assert that catch-up issues multiple calls and persists the complete contiguous series, that no request exceeds MAX_CANDLES_PER_REQUEST candles (asserted on the start/end args the fake client actually received), and that the windows tile the range exactly — no duplicate and no gap at a chunk boundary. A fourth test guards that a small gap still takes exactly one request.

Gates

uv run ruff check keel tests packages   All checks passed!
uv run mypy                             Success: no issues found in 224 source files
uv run pytest -q                        2732 passed, 1 skipped

Baseline reconciliation, measured directly rather than assumed: the FULL suite on main at
29d9b20 is 2727 passed, 1 skipped. This branch adds 5 tests (4 chunking + 1 pinning that
an empty window must not stop catch-up), giving 2732.

An earlier version of this section cited "2716", which was a scoped baseline measured with
--ignore=tests/data/test_market_feed.py, not the full-suite figure. It reconciled correctly
(2716 + 15 in that file = 2731) but read as if it were the whole suite, and two later readers
tripped on it. The full-suite number is 2727.

🤖 Generated with Claude Code