Skip to content

State the current reasons for the Polars chunk reader factories #1271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laughingman7743 merged 4 commits into from
Oct 18, 2026
Merged

State the current reasons for the Polars chunk reader factories #1271

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0174353
State the current reasons for the Polars chunk reader factories
laughingman7743 Oct 18, 2026
cf186d9
Name the async future and the Polars scan in the chunk reader docstring
laughingman7743 Oct 18, 2026
088b79e
Qualify the schema use of the Polars chunk scan
laughingman7743 Oct 18, 2026
df1d35b
Refer to _iter_parquet_chunks() for the use of the Parquet schema
laughingman7743 Oct 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions pyathena/polars/result_set.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,9 +456,7 @@ def _get_parquet_storage_options(connection: Connection[Any]) -> dict[str, Any]:
fsspec. The credentials are frozen when the options are created.

Args:
connection: The connection whose session and region the reads use. The
chunk readers give the connection that they took when they were
created, as ``close()`` releases the result set's connection.
connection: The connection whose session and region the reads use.

@laughingman7743 laughingman7743 Oct 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 2 (claims, callers): CLEAN

Base e086eac7bb2120e434c08c2eababccd9d0167fb0, head 0174353d12172a03995a4420da46a5c9d5caeb83.

  • "execute() raises their errors": the result set is constructed inside execute() for PolarsCursor and AioPolarsCursor (in its thread), and inside the future of AsyncPolarsCursor.execute(), whose result() raises them. In all three the errors surface before the first fetch, which is the distinction Decide whether the Polars chunk readers keep the state they take at creation #1249 is about.
  • PR body claims: Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238 reverted the pandas Parquet reader change and kept the Polars split (pandas _iter_parquet_chunks() now takes no state arguments, pyathena/pandas/result_set.py:1564); merging the CSV pair would move _is_csv_readable() into the generator body and so to the first fetch.
  • The connection parameter is still passed by the readers; this PR keeps that structure as option 1 decided.


Returns:
Dictionary with AWS credentials and region for S3 access.
Expand Down Expand Up @@ -606,9 +604,12 @@ def _create_dataframe_iterator(self) -> PolarsDataFrameIterator:
For unload queries, it replaces the metadata with the schema of the Parquet
files, as ``_as_polars()`` does, before the chunks are read.

The chunk readers take the state that they need, such as the output location,
the column names, and the connection, when they are created. The credentials
are frozen at the first read.
The result set calls this method when it is created, in ``execute()`` or in the
future that ``AsyncPolarsCursor.execute()`` returns. The requests that prepare
the reads are made then, before the first fetch, and raise their errors there:
the size check of a CSV result file, and the data manifest and the Parquet
schema of an UNLOAD result. The schema sets the metadata and is passed to
``_iter_parquet_chunks()``. The chunks themselves are read lazily.

Returns:
PolarsDataFrameIterator that reads each chunk lazily.
Expand Down
Loading