Repository navigation
Conversation
Unfiltered, 25 EIA datasets match more rows than MAX_AUTO_ROWS (500,000),
and the provider refuses an unbounded request over that ceiling ("Narrow
the request with facet filters or dates, or set an explicit limit"). Their
widgets opened with no limit, so every one of them failed on first open.
Those widgets now default to limit=5000 and a hidden sort=desc, so they open
on the newest page of data. Only the widget metadata changes: the REST
defaults (full pagination, ascending) are unchanged, and the user can still
change the limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for the PR, @artcashin.
I agree that some endpoints require practical default values, the bulk data sources and spreadsheet files are more appropriate for mining the full dataset.
These types of considerations should get baked in to the generate_catalog and api_query utilities layer instead of patching at the router level.
…page Both datasets have passed MAX_AUTO_ROWS (500,000), so their unfiltered widgets were refused. They join the widgets that open on the newest 5,000 rows. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… defaults Review feedback: the default for an oversized request belongs in the query layer, not in per-widget router config. paginate() now returns the newest 5,000 rows with a warning when an unlimited request matches more than MAX_AUTO_ROWS, instead of raising. The router list and the widget params are removed, so no dataset has to be listed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Thanks, @dlee-bq, and apologies for the slow turnaround on this one. I also pushed a commit earlier today that added two more widgets to the router list before I had read your review; that was a mistake and it is undone. Reworked in a03d959: the router is back to Two things for you to decide:
The description is updated with the tests and what I could not run locally. |
Problem
An EIA request with no filters and no
limitfails when it matches more rows thanMAX_AUTO_ROWS(500,000):27 dataset widgets open on such a request, so each one fails on first open: AEO, IEO, International, SEDS, Crude Oil Imports, Total Energy, three electricity datasets, four electricity grid datasets, two nuclear outage datasets and twelve petroleum datasets. The set grows as datasets grow: Crude Oil Imports and Total Energy crossed the ceiling in the last month.
Change
The default lives in the query layer (
utils/api_query.py::paginate), as asked in review. When an unlimited request matches more thanMAX_AUTO_ROWS,paginatereturns the newest page (5,000 rows) and warns, instead of raising:limit, or under the ceiling, behaves as before.paginategets this, including the data browser. Thelimitdescriptions say so.Nothing changes in
eia_router.pyand no dataset is listed anywhere, so a dataset that grows past the ceiling needs no code change.Behaviour change
For the REST API and the Python interface, an oversized unlimited request used to be an error and is now 5,000 rows plus a warning. The bulk files remain the way to get a full dataset.
Tests
test_auto_ceiling_raises_without_limitis replaced bytest_auto_ceiling_returns_newest_page_without_limit(ascending: two requests, the second descending; descending: one request; both warn and return the newest page). It fails ondevelop.tests/test_api_query.py,tests/test_data_browser.pyandtests/test_eia_router.py: 260 passed (Python 3.12).developthere: 206 fetcher tests have no API credential and 2 are intest_generate_catalog.py. None is in code this PR touches.Not included
generate_catalogis unchanged. The catalog records facet counts, not row counts, so a per-dataset default there would need a data request per dataset at generation time and would go stale between regenerations. Handling it at request time needs neither. Happy to add a catalog field if you would rather have it recorded.OBBject.warningsthrough the REST API; it useswarnings.warnasshort_term_energy_outlook.pydoes.History
The first two commits set widget defaults in the router; the third removes them and moves the behaviour into
paginate. The net diff againstdevelopis three files.🤖 Generated with Claude Code