Skip to content

Define the behavior of result set fetches after an error #1225

Closed

Description

@laughingman7743

Use case

What a result set does after a fetch raises is unspecified, and the backends differ:

  • fetchmany()/fetchall() of AthenaResultSet and AthenaAioResultSet drop the rows that they took from the buffer before a failed page request, while counting them in rownumber; a later fetch continues after them. This is older than Take buffered rows in bulk in fetchmany() and fetchall() #1187, which made the bulk take: the previous loop over fetchone() lost them too.
  • The GetQueryResults result sets advance the token of the next page before converting its rows, so a fetch after a failed conversion skips that page.
  • The Arrow result set consumes a record batch before converting it, the S3FS result set consumes each CSV row before converting it, and the pandas and Polars result sets exhaust their row iterator on an error (their tests expect it).

PEP 249 does not define the state of a cursor after an exception. dbt-athena and SQLAlchemy call the fetch methods in loops, and the async SQLAlchemy adapter fetches all rows at execute() and closes the cursor when that fetch fails (pyathena/aio/sqlalchemy/base.py).

A partial fix was tried in #1222 and removed: returning the taken rows to the buffer and advancing the token after conversion. Review found that it miscounted rownumber when an interrupt arrived between taking and counting the rows, could refill the buffer of a result set closed meanwhile (#1197 defines that a closed result set fetches nothing), covered only the GetQueryResults result sets, and made a deterministic conversion error repeat the same request on every later fetch.

Proposed change

Decide one contract for all result sets and implement it everywhere. Two options were proposed in a design consultation:

  • (b) A failed fetch has no effect. A transient failure (a request error after retries, an interrupt, a cancellation) leaves the result set as before the call, and a later fetch continues without losing or duplicating rows; a conversion failure raises again on a later fetch and skips nothing. Counting the rows once on success, instead of per buffered take, removes the miscount; checking is_closed after returning rows keeps a closed result set empty. Arrow could return the batch to its iterator; S3FS, pandas, and Polars would need their readers to support re-reading or would have to be documented as exceptions.
  • (c) A failed fetch breaks the result set. The original error propagates, and later fetches raise a distinct "broken" error before any request, distinguished from a closed result set, with argument validation errors left outside. This is simpler to make uniform across backends and to test, at the cost of resumability.

Either way, cover every override (fetchone, fetchmany, fetchall, iteration, the aio variants, and the pandas, Polars, Arrow, and S3FS result sets) and the async cancellation boundaries (asyncio.to_thread() cancellation can leave consumption running).

Validation plan (if implementing)

Unit tests without AWS for each result set: request failure, conversion failure, an interrupt between taking and counting rows, an aio cancellation, a close() during a failed request, the behavior of later fetches (continuation for (b), rejection without a request for (c)), and a successful fresh execution afterwards; then the cursor and SQLAlchemy test suites.

Activity

  1. laughingman7743 commented on Oct 17, 2026

    @laughingman7743
    MemberAuthor

    Decision: option (c), a failed fetch breaks the result set

    • The original error propagates unchanged.
    • Later fetches (fetchone, fetchmany, fetchall, iteration, the aio variants, and the pandas, Polars, Arrow, and S3FS result sets) raise ProgrammingError before any request, with a message distinct from the closed result set.
    • Argument validation errors raised before the operation starts, such as a non-integer size, do not break the result set.
    • Interrupts and asyncio cancellations break it, because what the interrupted read consumed is unknown.
    • This is a breaking change and needs a release note: today the pandas and Polars result sets return no rows after an error, and the other result sets continue after skipped or dropped rows.

    Option (b) was not chosen because the S3FS CSV reader and the pandas and Polars chunk readers cannot resume after failing mid-read, so it could not be uniform without rewriting those readers.

    Implementation

    This is implemented together with #1206: the operation scope of the result set guard marks the result set broken when an exception leaves it. One PR closes both issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions