Skip to content

Close objects built for cancelled asyncio constructions #1208

Closed

Description

@laughingman7743

Part of #1205 (step 4; see R4 and model 4 there).

Problem

pyathena.aio.util.create_in_thread() closes an object that is built after its caller was cancelled, but only the result sets of the aio pandas, Arrow, Polars, and S3FS cursors use it. Other constructions leave orphans:

  1. AioConnection.create() (pyathena/aio/connection.py:62), cancelled: the connection is built and never closed.
  2. The documented await asyncio.to_thread(conn.cursor) for AioSparkCursor (pyathena/aio/spark/cursor.py:36-41), cancelled: the Spark session goes IDLE and is never terminated.
  3. AioS3FileSystem.open_async() (pyathena/filesystem/s3_async.py:902), cancelled: the file is built and left to the garbage collector. In xb mode, collection emits a misleading "was not committed" ResourceWarning.
  4. AsyncCursor / AsyncPandasCursor: a result set built for a running future that the caller cancels is never closed (pyathena/async_cursor.py:274, pyathena/pandas/async_cursor.py:274). AsyncCursor._description (pyathena/async_cursor.py:139) builds a result set and never closes it.
  5. A KeyboardInterrupt during S3FS result set construction leaves the CSV file open until garbage collection, because cleanup runs only on Exception (pyathena/s3fs/result_set.py:210-217).

Reproduction

The stub reproductions are summarized in #1205. They use no AWS calls.

Environment

  • PyAthena master at b2d7aa2
  • Python 3.15

Proposed fix (optional)

  • Every asyncio construction of a closable object goes through create_in_thread(). That covers connections, Spark cursors (with an async factory instead of the documented to_thread), files from open_async(), and result sets.
  • Sync constructors release what they opened on BaseException.
  • AsyncCursor owns the result sets of its futures and closes the ones that were not taken when the cursor closes.

Validate with unit tests that cancel each construction and check that the object is closed and, for Spark, that the session is terminated.

Decisions (2026-10-18)

Re-checked on master at b8cec9d: all five problems still reproduce with the stub reproductions. The other result sets (pandas, Arrow, Polars) already release their files on BaseException, so item 5 is specific to S3FS.

  • Item 2: the async factory is await AioSparkCursor.create(connection, **kwargs). It is an async classmethod, like AioConnection.create() and AthenaAioResultSet.create(). It builds the cursor through connection.cursor(), so cursor_kwargs apply. A cursor built after a cancellation terminates the session it started. The docs switch from asyncio.to_thread(conn.cursor) to this factory.
  • Items 2 and 3 close with coroutines, so create_in_thread() accepts a synchronous release for an object built after a cancellation. A file from a cancelled open_async() is discarded, not committed, and its threads are released.
  • Item 4 (revised 2026-10-18): AsyncCursor.close() closes the result sets of queries that finish after the close, before their futures resolve. A result set whose future completed before the close belongs to the caller. The first plan was to close every result set not taken with result(). It was dropped because a future cannot tell whether its result set was taken: Python 3.15's asyncio.wrap_future() reads the outcome with the internal Future._get_snapshot() instead of result(). _description() closes the result set it builds.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions