Skip to content

fix(pricing): page through the whole catalog in pricing sync - #104

Merged
andreacappelletti97 merged 1 commit into
mainfrom
fix/sync-paging
Sep 29, 2026
Merged

andreacappelletti97 merged 1 commit into
mainfrom
fix/sync-paging

Conversation

@andreacappelletti97

Copy link
Copy Markdown
Contributor

Bug

The documented air-gapped workflow didn't work. c3x pricing sync reported "Offline ready", but c3x estimate --offline against the synced cache priced an m5.xlarge at $0 (the stub).

Cause: the product enumerator requests pages of 5,000. It treated any page shorter than that as the last one, and stepped the offset by the requested size. c3x-pricing-api#60 capped requests at 1,000 products (MAX_PRODUCTS_PER_REQUEST), so every page comes back short and the sync stopped after the first 1,000 products of every service and region. The live API confirms this: asking for limit:5000 returns 1000.

Fix

  • Advance the offset by the rows actually returned, and stop only on an empty page. A server cap of any size can no longer truncate a sync.
  • Regression test: a stub server capped at 3 per page with 7 products. The old code saw 3; the new code sees all 7.

Evidence

Live pricing.c3x.dev, pricing sync --providers aws --regions us-east-1, then the same config priced offline and live:

before after
prices synced 4,455 22,364
cache size 1.6 MB 9.1 MB
offline m5.xlarge $0 (stub) $140.16
offline total vs live $11.50 vs $334.16 $334.16 = $334.16

Released CLIs still have the old paging. c3x-pricing-api is raising its default cap to 5,000 in a separate PR, so their one-page requests are complete again. That also cuts the sync time (7m56s at 1,000 per page).

The enumerator stopped at the first page shorter than it requested and stepped the offset by the requested size. Once the pricing API capped requests at 1,000 products, every page was short, so a sync kept only the first 1,000 products per service and region and offline estimates fell back to the stub. Advance by the rows returned; stop on an empty page.
@andreacappelletti97
andreacappelletti97 merged commit 6cf69e2 into main Sep 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant