You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Replace the cross-thread cancel() example in the Spark documentation #1248
The Cancellation section of docs/spark.md (around line 280) shows a threading.Timer that calls cursor.cancel on the same SparkCursor while execute() runs:
PyAthena declares threadsafety = 2. #1206 (PR #1238) documented that calling the methods of one cursor from several threads, including close() and cancel(), is not supported, and removed the equivalent query example from docs/usage.md. The Spark example now contradicts that contract.
Unlike SQL queries, Spark calculations have no start callback such as on_start_query_execution, so the calculation ID is not available to another thread before execute() returns. The SQL replacement, stopping a query by its ID, does not carry over directly.
Proposed change
Remove the Timer example and document a supported way to stop a long calculation, for example one of:
KeyboardInterrupt with kill_on_interrupt enabled, which the same section already describes.
AsyncSparkCursor, whose execute() returns the calculation ID with a future, and AsyncSparkCursor.cancel(calculation_id).
AioSparkCursor with asyncio.wait_for(), if its task cancellation stops the calculation as the SQL asyncio cursors do. This needs to be checked against pyathena/aio/spark/cursor.py before it is documented.
Whether Spark cursors should also get a start callback is a separate decision and is not part of this change.
Validation plan (if implementing)
just docs lint and just docs build. Check each documented alternative against the Spark cursor code, and against the Spark work group if a code path is unclear.
Activity