You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark: define ownership when reusing externally supplied or shared sessions #793
A caller can pass an existing session_id to a Spark cursor, including a session managed by another component or shared with another cursor.
Currently, closing any of SparkCursor, AsyncSparkCursor, or AioSparkCursor calls terminate_session for that supplied ID.
Closing one cursor can therefore end a session another component still needs.
The current documentation describes termination on cursor close without distinguishing ownership.
Changing this behavior requires an explicit compatibility decision.
Proposed change
Agree on an ownership contract before implementation:
Distinguish sessions created by the library from sessions supplied and managed externally.
Define whether and how ownership can be transferred explicitly.
Define sharing between cursors, including separate connections, and who may terminate the session.
Separate the session's owner from the owner of a calculation submitted to it.
Preserve external sessions on borrower close under the agreed contract.
Choose the smallest implementation that satisfies that contract; connection-wide reference counting is not predetermined.
Document any compatibility impact and migration path for callers relying on the current close behavior.
Validation plan (if implementing)
Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.
Cover owned, borrowed, and shared sessions across synchronous, thread-based, and asyncio cursors.
Verify that closing a borrower leaves the externally managed session usable, and that owned resources are released according to the agreed policy.
Include real-AWS verification with isolated sessions, explicit ownership and guaranteed final cleanup, in addition to deterministic local tests.
Activity