Repository navigation
Define the behavior of result set fetches after an error #1225
Description
Use case
What a result set does after a fetch raises is unspecified, and the backends differ:
fetchmany()/fetchall()ofAthenaResultSetandAthenaAioResultSetdrop the rows that they took from the buffer before a failed page request, while counting them inrownumber; a later fetch continues after them. This is older than Take buffered rows in bulk in fetchmany() and fetchall() #1187, which made the bulk take: the previous loop overfetchone()lost them too.- The GetQueryResults result sets advance the token of the next page before converting its rows, so a fetch after a failed conversion skips that page.
- The Arrow result set consumes a record batch before converting it, the S3FS result set consumes each CSV row before converting it, and the pandas and Polars result sets exhaust their row iterator on an error (their tests expect it).
PEP 249 does not define the state of a cursor after an exception. dbt-athena and SQLAlchemy call the fetch methods in loops, and the async SQLAlchemy adapter fetches all rows at execute() and closes the cursor when that fetch fails (pyathena/aio/sqlalchemy/base.py).
A partial fix was tried in #1222 and removed: returning the taken rows to the buffer and advancing the token after conversion. Review found that it miscounted rownumber when an interrupt arrived between taking and counting the rows, could refill the buffer of a result set closed meanwhile (#1197 defines that a closed result set fetches nothing), covered only the GetQueryResults result sets, and made a deterministic conversion error repeat the same request on every later fetch.
Proposed change
Decide one contract for all result sets and implement it everywhere. Two options were proposed in a design consultation:
- (b) A failed fetch has no effect. A transient failure (a request error after retries, an interrupt, a cancellation) leaves the result set as before the call, and a later fetch continues without losing or duplicating rows; a conversion failure raises again on a later fetch and skips nothing. Counting the rows once on success, instead of per buffered take, removes the miscount; checking
is_closedafter returning rows keeps a closed result set empty. Arrow could return the batch to its iterator; S3FS, pandas, and Polars would need their readers to support re-reading or would have to be documented as exceptions. - (c) A failed fetch breaks the result set. The original error propagates, and later fetches raise a distinct "broken" error before any request, distinguished from a closed result set, with argument validation errors left outside. This is simpler to make uniform across backends and to test, at the cost of resumability.
Either way, cover every override (fetchone, fetchmany, fetchall, iteration, the aio variants, and the pandas, Polars, Arrow, and S3FS result sets) and the async cancellation boundaries (asyncio.to_thread() cancellation can leave consumption running).
Validation plan (if implementing)
Unit tests without AWS for each result set: request failure, conversion failure, an interrupt between taking and counting rows, an aio cancellation, a close() during a failed request, the behavior of later fetches (continuation for (b), rejection without a request for (c)), and a successful fresh execution afterwards; then the cursor and SQLAlchemy test suites.
Activity
laughingman7743 commentedon Oct 17, 2026
Decision: option (c), a failed fetch breaks the result set
fetchone,fetchmany,fetchall, iteration, the aio variants, and the pandas, Polars, Arrow, and S3FS result sets) raiseProgrammingErrorbefore any request, with a message distinct from the closed result set.size, do not break the result set.Option (b) was not chosen because the S3FS CSV reader and the pandas and Polars chunk readers cannot resume after failing mid-read, so it could not be uniform without rewriting those readers.
Implementation
This is implemented together with #1206: the operation scope of the result set guard marks the result set broken when an exception leaves it. One PR closes both issues.
Expect ProgrammingError after a failed binary CSV chunk read
Break result sets after a failed fetch and drop the cross-thread guards
Expect ProgrammingError after a failed binary CSV chunk read