You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark: define cleanup on Connection and AioConnection context exit #796
Spark sessions are stateful resources, but Connection.close() currently performs no cleanup and Connection.cursor() does not retain returned cursors for cleanup. AioConnection.__aexit__() calls that same synchronous close method.
A caller using a connection context and creating a Spark cursor without a separate cursor context can therefore leave the Spark session running after the connection exits.
Agree on the connection's responsibility for Spark resources, following the session-ownership decision in the sibling issue.
Specify cleanup on explicit close and both context-manager variants, including outstanding calculations, multiple cursors, shared or borrowed sessions, and cleanup errors.
Decide how asynchronous cleanup is exposed without blocking the event loop or leaving unawaited coroutines.
Do not predetermine a new aclose() API or extend cleanup changes to unrelated cursor backends without a separate rationale.
If cleanup remains cursor-only by design, make that limitation explicit in Spark examples and verify the documented usage actually releases resources.
Validation plan (if implementing)
Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.
Use connection-only and nested connection/cursor contexts across all Spark variants, including exceptional exits.
Verify resource behavior against the agreed ownership contract, and preserve existing non-Spark connection and SQLAlchemy close behavior.
Any public async-connection changes need direct-caller and async SQLAlchemy compatibility checks plus relevant real-AWS lifecycle coverage.
Disposition: session cleanup stays the responsibility of the Spark cursor.
Connection and AioConnection are unchanged: they do not track the cursors they create, and closing them (including through their context managers) does not terminate Spark sessions. #844 documents this in the Spark "Session lifecycle" section, converts the Spark examples to close their cursors (context managers, or try/finally with await cursor.close()), and notes that a session left running is terminated by Athena after its idle timeout. The same PR defines which sessions close() terminates (#793) through the new terminate_session_on_close argument.
Activity
laughingman7743 commentedon Sep 26, 2026
Disposition: session cleanup stays the responsibility of the Spark cursor.
ConnectionandAioConnectionare unchanged: they do not track the cursors they create, and closing them (including through their context managers) does not terminate Spark sessions. #844 documents this in the Spark "Session lifecycle" section, converts the Spark examples to close their cursors (context managers, ortry/finallywithawait cursor.close()), and notes that a session left running is terminated by Athena after its idle timeout. The same PR defines which sessionsclose()terminates (#793) through the newterminate_session_on_closeargument.This issue will close with #844.