Skip to content

Reuse Athena Spark sessions across Spark tests #849

Closed as not planned
Closed as not planned
Reuse Athena Spark sessions across Spark tests#849
Labels
enhancement

Description

@laughingman7743
laughingman7743
opened on Sep 26, 2026

Use case

Spark tests cost about $20 of the September bill (58 DPU-hours; see #834).

  • The spark_cursor and async_spark_cursor fixtures in tests/pyathena/conftest.py and aio_spark_cursor in tests/pyathena/aio/conftest.py are function-scoped.
  • Every Spark test starts a new Athena Spark session and waits for it to become idle.
  • The cursor's close() terminates the session.

So the DPU cost of starting a session is paid once per test.

Since #837, Spark tests run on pull requests only when Spark code or tests change, plus weekly and on dispatch. The Spark lifecycle work tracked in #791 changes those files, so they run often while that work is in progress.

Proposed change

Reuse Spark sessions across tests where isolation allows it:

  • Provide a module- or worker-scoped Spark connection/session fixture for tests that only run calculations.
  • Keep function-scoped sessions for tests that exercise session lifecycle: creation failure, termination, cancellation, and close() behavior.
  • Check how this interacts with the Improve Spark cursor session ownership and lifecycle handling #791 changes (for example, terminate_session_on_close) so the tests still cover the lifecycle contracts.

Validation plan (if implementing)

  • Run the Spark tests (tests/pyathena/spark/, tests/pyathena/aio/spark/) against the pyathena-spark work group, and confirm the same results as before.
  • Compare the number of sessions started (ListSessions for the work group during the run) and the DPU-hours before and after.

Activity

  1. laughingman7743
    added
    enhancement
    on Sep 26, 2026
  2. laughingman7743
    added a parent issue on Sep 26, 2026
  3. laughingman7743
    added this to the 4.0.0 milestone on Sep 26, 2026

  4. laughingman7743 commented on Sep 28, 2026

    @laughingman7743
    laughingman7743
    on Sep 28, 2026
    MemberAuthor

    Closing as not planned. Measurements on 2026-09-28 show that the remaining saving no longer justifies the change.

    Spark DPU cost (Cost Explorer, USW2-CodeExecutionInDPUHours):

    Period Per day
    2026-09-23 to 2026-09-26 (every Python version, #791 in progress) $2.8–3.4 (8–10 DPU-hours)
    2026-09-27 (after #863: pull requests test Python 3.14 only) $0.65 (1.87 DPU-hours)
    2026-09-28 $0

    One Spark test run now (the latest CI run, for #861, on Python 3.14): 15 sessions with 345 session-seconds in total. The engine configuration is CoordinatorDpuSize: 1, MaxConcurrentDpus: 2, DefaultExecutorDpuSize: 1.

    Remaining saving

    What the change would cost

    • Moving the Spark tests onto one worker, which means switching the whole PyAthena suite to --dist loadgroup.
    • Sharing sessions between tests: a failed or stuck calculation could leave the session BUSY for the following tests.
    • Separate owned-session fixtures for the lifecycle tests.

    If Spark work picks up again and the Spark DPU cost becomes significant, this can be reopened. The cost is tracked in #846.

  5. laughingman7743
    closed this as not plannedon Sep 28, 2026
  6. laughingman7743
    removed this from the 4.0.0 milestone on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions