Skip to content

Closing a PolarsCursor result set during the first chunked CSV read makes that read raise ProgrammingError #1188

Closed

Description

@laughingman7743

Summary

Closing a chunked PolarsCursor result set while another thread is in the first read of a CSV result makes that read raise ProgrammingError: output_location is not available. instead of returning its chunk.

AthenaPolarsResultSet.close() (pyathena/polars/result_set.py:1023) calls super().close() first, which clears the query execution, so output_location becomes None.
The CSV chunk generator _iter_csv_chunks() (pyathena/polars/result_set.py:880) reads that state lazily in its first next().
It calls _is_csv_readable(), whose _get_content_length() sends an S3 request, and then checks self.output_location (pyathena/polars/result_set.py:896).
A close that lands before or during that request leaves the read without the output location.

This happens with AioPolarsCursor: a fetch runs in asyncio.to_thread() and keeps reading after its task is cancelled, while leaving async with closes the result set through close_in_thread() (pyathena/aio/common.py).
A synchronous cursor closed from another thread while it is read is outside threadsafety = 2.

Line numbers refer to 38dc8f86.
Before #1182, the close itself raised ValueError: generator already executing when the generator was running (#1172); after #1182, the close succeeds and the reader fails instead.
Found by an independent review of #1182.

Expected behavior

As in #1172, a fetch that is still reading when the result set is closed ends with the chunk it is reading.
The state that the chunk readers need (the output location, the CSV parameters, the column names) should not depend on the result set staying open until their first read.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions