Skip to content

Spark: define cleanup on Connection and AioConnection context exit #796

Closed

Description

@laughingman7743

Parent: #791

Use case

Spark sessions are stateful resources, but Connection.close() currently performs no cleanup and Connection.cursor() does not retain returned cursors for cleanup.
AioConnection.__aexit__() calls that same synchronous close method.
A caller using a connection context and creating a Spark cursor without a separate cursor context can therefore leave the Spark session running after the connection exits.

Evidence: cursor creation, connection close, synchronous context exit, and asyncio context exit.
The no-op behavior is documented today, so this is a lifecycle contract decision rather than a claim that an existing connection-cleanup guarantee works incorrectly.

Proposed change

Agree on the connection's responsibility for Spark resources, following the session-ownership decision in the sibling issue.
Specify cleanup on explicit close and both context-manager variants, including outstanding calculations, multiple cursors, shared or borrowed sessions, and cleanup errors.
Decide how asynchronous cleanup is exposed without blocking the event loop or leaving unawaited coroutines.
Do not predetermine a new aclose() API or extend cleanup changes to unrelated cursor backends without a separate rationale.

If cleanup remains cursor-only by design, make that limitation explicit in Spark examples and verify the documented usage actually releases resources.

Validation plan (if implementing)

Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.

Use connection-only and nested connection/cursor contexts across all Spark variants, including exceptional exits.
Verify resource behavior against the agreed ownership contract, and preserve existing non-Spark connection and SQLAlchemy close behavior.
Any public async-connection changes need direct-caller and async SQLAlchemy compatibility checks plus relevant real-AWS lifecycle coverage.

Ownership contract: #793.

Activity

  1. added this to the 4.0.0 milestone on Sep 26, 2026
  2. laughingman7743 commented on Sep 26, 2026

    @laughingman7743
    MemberAuthor

    Disposition: session cleanup stays the responsibility of the Spark cursor.

    Connection and AioConnection are unchanged: they do not track the cursors they create, and closing them (including through their context managers) does not terminate Spark sessions. #844 documents this in the Spark "Session lifecycle" section, converts the Spark examples to close their cursors (context managers, or try/finally with await cursor.close()), and notes that a session left running is terminated by Athena after its idle timeout. The same PR defines which sessions close() terminates (#793) through the new terminate_session_on_close argument.

    This issue will close with #844.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions