Repository navigation
Close objects built for cancelled asyncio constructions #1208
Copy link
Copy link
Closed
Labels
Description
Part of #1205 (step 4; see R4 and model 4 there).
Problem
pyathena.aio.util.create_in_thread() closes an object that is built after its caller was cancelled, but only the result sets of the aio pandas, Arrow, Polars, and S3FS cursors use it. Other constructions leave orphans:
AioConnection.create()(pyathena/aio/connection.py:62), cancelled: the connection is built and never closed.- The documented
await asyncio.to_thread(conn.cursor)forAioSparkCursor(pyathena/aio/spark/cursor.py:36-41), cancelled: the Spark session goes IDLE and is never terminated. AioS3FileSystem.open_async()(pyathena/filesystem/s3_async.py:902), cancelled: the file is built and left to the garbage collector. Inxbmode, collection emits a misleading "was not committed"ResourceWarning.AsyncCursor/AsyncPandasCursor: a result set built for a running future that the caller cancels is never closed (pyathena/async_cursor.py:274,pyathena/pandas/async_cursor.py:274).AsyncCursor._description(pyathena/async_cursor.py:139) builds a result set and never closes it.- A
KeyboardInterruptduring S3FS result set construction leaves the CSV file open until garbage collection, because cleanup runs only onException(pyathena/s3fs/result_set.py:210-217).
Reproduction
The stub reproductions are summarized in #1205. They use no AWS calls.
Environment
- PyAthena master at b2d7aa2
- Python 3.15
Proposed fix (optional)
- Every asyncio construction of a closable object goes through
create_in_thread(). That covers connections, Spark cursors (with an async factory instead of the documentedto_thread), files fromopen_async(), and result sets. - Sync constructors release what they opened on
BaseException. AsyncCursorowns the result sets of its futures and closes the ones that were not taken when the cursor closes.
Validate with unit tests that cancel each construction and check that the object is closed and, for Spark, that the session is terminated.
Decisions (2026-10-18)
Re-checked on master at b8cec9d: all five problems still reproduce with the stub reproductions. The other result sets (pandas, Arrow, Polars) already release their files on BaseException, so item 5 is specific to S3FS.
- Item 2: the async factory is
await AioSparkCursor.create(connection, **kwargs). It is an async classmethod, likeAioConnection.create()andAthenaAioResultSet.create(). It builds the cursor throughconnection.cursor(), socursor_kwargsapply. A cursor built after a cancellation terminates the session it started. The docs switch fromasyncio.to_thread(conn.cursor)to this factory. - Items 2 and 3 close with coroutines, so
create_in_thread()accepts a synchronous release for an object built after a cancellation. A file from a cancelledopen_async()is discarded, not committed, and its threads are released. - Item 4 (revised 2026-10-18):
AsyncCursor.close()closes the result sets of queries that finish after the close, before their futures resolve. A result set whose future completed before the close belongs to the caller. The first plan was to close every result set not taken withresult(). It was dropped because a future cannot tell whether its result set was taken: Python 3.15'sasyncio.wrap_future()reads the outcome with the internalFuture._get_snapshot()instead ofresult()._description()closes the result set it builds.
Reactions are currently unavailable
Activity
Move the AsyncCursor ownership docs after the closed cursor rules (GH…
Close only the AsyncCursor result sets created after the close (GH-1208)
Take a zero-argument factory in create_in_thread() (GH-1208)
Say in the AsyncCursor docs which result sets closing the cursor clos…
Tighten the cleanup tests and narrow the AsyncCursor close wording (G…
State the AsyncCursor close boundary that the worker check guarantees (…
Qualify the remaining AsyncCursor close statements with the finishing…
Merge pull request #1279 from pyathena-dev/fix/1208-cancelled-constru…