Skip to content

Replace the cross-thread cancel() example in the Spark documentation #1248

Closed

Description

@laughingman7743

Use case

The Cancellation section of docs/spark.md (around line 280) shows a threading.Timer that calls cursor.cancel on the same SparkCursor while execute() runs:

with conn.cursor() as cursor:
    timer = threading.Timer(60, cursor.cancel)
    timer.start()
    try:
        cursor.execute(...)
    finally:
        timer.cancel()

PyAthena declares threadsafety = 2. #1206 (PR #1238) documented that calling the methods of one cursor from several threads, including close() and cancel(), is not supported, and removed the equivalent query example from docs/usage.md. The Spark example now contradicts that contract.

Unlike SQL queries, Spark calculations have no start callback such as on_start_query_execution, so the calculation ID is not available to another thread before execute() returns. The SQL replacement, stopping a query by its ID, does not carry over directly.

Proposed change

Remove the Timer example and document a supported way to stop a long calculation, for example one of:

  • KeyboardInterrupt with kill_on_interrupt enabled, which the same section already describes.
  • AsyncSparkCursor, whose execute() returns the calculation ID with a future, and AsyncSparkCursor.cancel(calculation_id).
  • AioSparkCursor with asyncio.wait_for(), if its task cancellation stops the calculation as the SQL asyncio cursors do. This needs to be checked against pyathena/aio/spark/cursor.py before it is documented.

Whether Spark cursors should also get a start callback is a separate decision and is not part of this change.

Validation plan (if implementing)

just docs lint and just docs build. Check each documented alternative against the Spark cursor code, and against the Spark work group if a code path is unclear.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions