Skip to content

Spark: define ownership when reusing externally supplied or shared sessions #793

Closed

Description

@laughingman7743

Parent: #791

Use case

A caller can pass an existing session_id to a Spark cursor, including a session managed by another component or shared with another cursor.
Currently, closing any of SparkCursor, AsyncSparkCursor, or AioSparkCursor calls terminate_session for that supplied ID.
Closing one cursor can therefore end a session another component still needs.

Evidence: supplied-session constructor path, base close, close entrypoint, and asyncio close.
Offline probes for all three variants confirmed a termination request for a supplied synthetic session ID.

The current documentation describes termination on cursor close without distinguishing ownership.
Changing this behavior requires an explicit compatibility decision.

Proposed change

Agree on an ownership contract before implementation:

  • Distinguish sessions created by the library from sessions supplied and managed externally.
  • Define whether and how ownership can be transferred explicitly.
  • Define sharing between cursors, including separate connections, and who may terminate the session.
  • Separate the session's owner from the owner of a calculation submitted to it.
  • Preserve external sessions on borrower close under the agreed contract.

Choose the smallest implementation that satisfies that contract; connection-wide reference counting is not predetermined.
Document any compatibility impact and migration path for callers relying on the current close behavior.

Validation plan (if implementing)

Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.

Cover owned, borrowed, and shared sessions across synchronous, thread-based, and asyncio cursors.
Verify that closing a borrower leaves the externally managed session usable, and that owned resources are released according to the agreed policy.
Include real-AWS verification with isolated sessions, explicit ownership and guaranteed final cleanup, in addition to deterministic local tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions