Skip to content

fix(eia): an oversized unlimited request returns the newest page instead of failing - #7761

Open
artcashin wants to merge 3 commits into
openbq-org:developfrom
artcashin:fix/eia-widget-newest-page-defaults
Open

artcashin wants to merge 3 commits into
openbq-org:developfrom
artcashin:fix/eia-widget-newest-page-defaults

Conversation

@artcashin

@artcashin artcashin commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

An EIA request with no filters and no limit fails when it matches more rows than MAX_AUTO_ROWS (500,000):

The unfiltered request matches N rows, above the automatic pagination ceiling of 500000. Narrow the request with facet filters or dates, or set an explicit limit.

27 dataset widgets open on such a request, so each one fails on first open: AEO, IEO, International, SEDS, Crude Oil Imports, Total Energy, three electricity datasets, four electricity grid datasets, two nuclear outage datasets and twelve petroleum datasets. The set grows as datasets grow: Crude Oil Imports and Total Energy crossed the ceiling in the last month.

Change

The default lives in the query layer (utils/api_query.py::paginate), as asked in review. When an unlimited request matches more than MAX_AUTO_ROWS, paginate returns the newest page (5,000 rows) and warns, instead of raising:

The unfiltered request matches N rows, above the automatic pagination ceiling of 500000. Returning the newest 5000 rows. Narrow the request with facet filters or dates, or set an explicit limit, to get more.

  • If the request is sorted ascending, the newest page costs one more request; if descending, the first page is already it.
  • The rows come back in the order the caller asked for.
  • A request with an explicit limit, or under the ceiling, behaves as before.
  • Every caller of paginate gets this, including the data browser. The limit descriptions say so.

Nothing changes in eia_router.py and no dataset is listed anywhere, so a dataset that grows past the ceiling needs no code change.

Behaviour change

For the REST API and the Python interface, an oversized unlimited request used to be an error and is now 5,000 rows plus a warning. The bulk files remain the way to get a full dataset.

Tests

  • test_auto_ceiling_raises_without_limit is replaced by test_auto_ceiling_returns_newest_page_without_limit (ascending: two requests, the second descending; descending: one request; both warn and return the newest page). It fails on develop.
  • tests/test_api_query.py, tests/test_data_browser.py and tests/test_eia_router.py: 260 passed (Python 3.12).
  • Whole provider suite in my container: 1,856 passed, 208 failed. The same 208 fail on develop there: 206 fetcher tests have no API credential and 2 are in test_generate_catalog.py. None is in code this PR touches.

Not included

  • generate_catalog is unchanged. The catalog records facet counts, not row counts, so a per-dataset default there would need a data request per dataset at generation time and would go stale between regenerations. Handling it at request time needs neither. Happy to add a catalog field if you would rather have it recorded.
  • I have not checked end to end that the warning reaches OBBject.warnings through the REST API; it uses warnings.warn as short_term_energy_outlook.py does.

History

The first two commits set widget defaults in the router; the third removes them and moves the behaviour into paginate. The net diff against develop is three files.

🤖 Generated with Claude Code

Unfiltered, 25 EIA datasets match more rows than MAX_AUTO_ROWS (500,000),
and the provider refuses an unbounded request over that ceiling ("Narrow
the request with facet filters or dates, or set an explicit limit"). Their
widgets opened with no limit, so every one of them failed on first open.

Those widgets now default to limit=5000 and a hidden sort=desc, so they open
on the newest page of data. Only the widget metadata changes: the REST
defaults (full pagination, ascending) are unchanged, and the user can still
change the limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@dlee-bq dlee-bq left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, @artcashin.

I agree that some endpoints require practical default values, the bulk data sources and spreadsheet files are more appropriate for mining the full dataset.

These types of considerations should get baked in to the generate_catalog and api_query utilities layer instead of patching at the router level.

artcashin and others added 2 commits November 18, 2026 03:33
…page

Both datasets have passed MAX_AUTO_ROWS (500,000), so their unfiltered widgets were refused. They join the widgets that open on the newest 5,000 rows.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… defaults

Review feedback: the default for an oversized request belongs in the query layer, not in per-widget router config. paginate() now returns the newest 5,000 rows with a warning when an unlimited request matches more than MAX_AUTO_ROWS, instead of raising. The router list and the widget params are removed, so no dataset has to be listed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@artcashin artcashin changed the title fix(eia): open oversized dataset widgets on the newest 5,000 rows fix(eia): an oversized unlimited request returns the newest page instead of failing Nov 18, 2026
@artcashin

Copy link
Copy Markdown
Contributor Author

Thanks, @dlee-bq, and apologies for the slow turnaround on this one. I also pushed a commit earlier today that added two more widgets to the router list before I had read your review; that was a mistake and it is undone.

Reworked in a03d959: the router is back to develop, and the behaviour is in api_query.paginate. An unlimited request that matches more than MAX_AUTO_ROWS now returns the newest 5,000 rows with a warning, instead of raising. No dataset is listed anywhere, so one that grows past the ceiling (Crude Oil Imports and Total Energy both did this month) needs no change.

Two things for you to decide:

  • This changes the API as well as the widgets: that request was an error and is now data plus a warning. If you would rather keep the error for the API and only default the widgets, say so and I'll split it.
  • I left generate_catalog alone. It records facet counts, not row counts, so a per-dataset default there needs a data request per dataset at generation time. If you want it in the catalog anyway, I'll add it.

The description is updated with the tests and what I could not run locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants