You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark: clean up newly created sessions when cursor initialization fails #794
A cursor can create a remote Spark session and then fail before returning the cursor to its caller.
The current initialization path does not recover that newly created session:
_start_session() obtains an ID, then _wait_for_idle_session() raises on an API failure or interruption.
Session creation and readiness succeed, then construction of the S3 client raises.
The caller receives no cursor on which to invoke close().
The session can remain allocated until service-side expiry or external cleanup.
Mock start_session to return a synthetic session ID and make readiness waiting raise OperationalError.
Alternatively, return a synthetic newly created session from _start_session and make connection.session.client("s3", ...) raise.
Both propagate the injected error with zero calls to terminate_session.
These probes establish missing cleanup in local control flow, not an observed live session leak.
Environment
Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.
Proposed fix (optional)
Retain the new session ID as soon as it is allocated and provide exception-safe initialization cleanup.
Coordinate the ownership boundary with the ownership sub-issue: a failed attempt to attach to an external session must not terminate it.
Preserve the original failure and make any cleanup failure visible, with enough information to recover the session.
Avoid treating every API error as proof that the session is already gone.
Acceptance:
Cover readiness API failures, interruption, and later local initialization failures for all three variants.
Attempt cleanup only for sessions owned by the failed initialization.
Cover cleanup failure and preserve both the initiating error and recovery information.
Verify normal construction and actual session cleanup in an isolated AWS environment.
Activity