Repository navigation
Replace the cross-thread cancel example in the Spark documentation - #1270
Conversation
Calling cancel() from another thread on a cursor whose execute() is running is not supported under threadsafety = 2. List the supported ways to stop a long calculation instead, and show asyncio.wait_for() with AioSparkCursor. Closes #1248 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| import threading | ||
| from pyathena import connect | ||
| from pyathena.spark.cursor import SparkCursor | ||
| As `pyathena.threadsafety = 2` declares, calling `cancel()` from another thread while `execute()` runs on the same cursor is not supported. |
There was a problem hiding this comment.
Self-review round 1 (implementation behavior): CLEAN
Base e086eac7bb2120e434c08c2eababccd9d0167fb0, head 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19. Scope: docs/spark.md (Cancellation and AioSparkCursor sections).
- The removed Timer example was the only cross-thread
cancel()usage in the docs (grepfor Timer and "another thread" acrossdocs/andpyathena/spark,pyathena/aio/spark).docs/sqlalchemy.mdcancels from another thread through a new cursor on the shared connection, whichthreadsafety = 2allows. - Each listed alternative is checked against the code:
SparkCursor._calculate/_poll(KeyboardInterrupt),AsyncSparkCursor.execute()returning(calculation_id, future)andcancel(query_id)(pyathena/spark/async_cursor.py:205-253), andAioSparkCursor._calculate/_poll(pyathena/aio/spark/cursor.py:98-257), which cancel the calculation on task cancellation withkill_on_interrupt. - The existing AioSparkCursor
create_task+await cursor.cancel()example runs on one event loop thread, so it is not affected and is kept. - The tests that call
SparkCursor.cancel()from another thread (tests/pyathena/spark/test_spark_cursor.py:162) are kept, as decided in Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238.
| If the task is cancelled before `execute()` begins the request, the request is never sent. | ||
| Cancelling the task again during this wait raises `asyncio.CancelledError` at once without cancelling the calculation. | ||
|
|
||
| With `kill_on_interrupt` enabled, a timeout from `asyncio.wait_for()` while `execute()` starts or waits for the calculation therefore cancels it and raises `asyncio.TimeoutError`. |
There was a problem hiding this comment.
Self-review round 2 (claims, callers, documentation reader): CLEAN
Base e086eac7bb2120e434c08c2eababccd9d0167fb0, head 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19.
Claims checked:
- "a timeout from
asyncio.wait_for()... cancels it and raisesasyncio.TimeoutError": verified offline with anAioSparkCursorwhose first status request blocks (mocked_calculate,_poll_until_terminal,_cancel). On Python 3.11 (wherewait_forruns the coroutine in a separate task) and 3.14 (same task viaasyncio.timeout), the result wasTimeoutError,_cancelawaited once with the calculation ID,calculation_id == "calculation_id", andstate == "CANCELED". The example'scursor.calculation_idandcursor.statetherefore print the cancelled calculation.requires-python = ">=3.11". - The start-phase case (timeout while
StartCalculationExecutionruns) follows the existing paragraph above, so it is not repeated. {ref}targetsasync-spark-cursorandaio-spark-cursorresolve in a worktreesphinx-build(spark.html).- The example mirrors the SQL one in
docs/aio.md(Task cancellation) and uses the sameaio_connect/asyncio.to_thread(conn.cursor)pattern as the other AioSparkCursor examples.
Limits: no live Spark run, since this is a documentation-only change. The live task-cancellation path is covered by tests/pyathena/aio/spark/test_cursor.py::test_cancel.
| If the task is cancelled before `execute()` begins the request, the request is never sent. | ||
| Cancelling the task again during this wait raises `asyncio.CancelledError` at once without cancelling the calculation. | ||
|
|
||
| With `kill_on_interrupt` enabled, a timeout from `asyncio.wait_for()` while `execute()` starts or waits for the calculation therefore cancels it and raises `asyncio.TimeoutError`. |
There was a problem hiding this comment.
Independent review (relayed): Codex CLI 0.163.0 (codex exec -s read-only, session 01a14cf6-50bd-78c2-a423-58dafce47c29), static review on a detached snapshot of 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19, base e086eac7bb2120e434c08c2eababccd9d0167fb0. The prompt contained only the diff, the threadsafety contract, and the review request (no PR text or prior findings). The snapshot and the PR worktree were unchanged afterwards.
Result: CLEAN.
Covered: both changed hunks; SparkCursor and the shared start, poll, interrupt, cancel, and state handling; AsyncSparkCursor's calculation ID/future return and ID-based cancellation; AioSparkCursor's timeout handling, defaults, cursor creation, context managers, and reported state; related usage, asyncio, SQLAlchemy, cursor/API docs, cross-references, and relevant tests.
Non-actionable notes: the example follows the page's top-level-await convention (a script needs an async function and asyncio.run()), and since cancellation is best effort, the wait can exceed 60 seconds while the cancellation finishes. Both match the rest of the page and the existing "Cancellation" rules, so there is no change.
WHAT
threading.Timerexample from the Spark "Cancellation" section, which calledcancel()on aSparkCursorfrom another thread whileexecute()ran.threadsafety = 2, and list the supported ways to stop a long calculation:KeyboardInterruptwithkill_on_interrupt,AsyncSparkCursor.cancel(calculation_id), andAioSparkCursorwithasyncio.wait_for().asyncio.wait_for()example to the AioSparkCursor section, mirroring the SQL asyncio cursor documentation indocs/aio.md.WHY
Closes #1248. #1206 (PR #1238) documented that the methods of one cursor, including
cancel(), must not be called from several threads, and removed the equivalent SQL example fromdocs/usage.md. The Spark example contradicted that contract.The
wait_for()behavior was checked againstpyathena/aio/spark/cursor.py: withkill_on_interruptenabled, a task cancellation in_poll()requestsStopCalculationExecution, waits for a terminal state via_cancel_and_wait(), and re-raises;_calculate()does the same for a cancellation during the start request.asyncio.wait_for()turns that cancellation intoasyncio.TimeoutError, andcalculation_idandstatekeep the cancelled calculation.The tests that call
SparkCursor.cancel()from another thread are kept, as decided in #1238.TEST
just docs lint: 0 errors.just docs build: succeeded. It builds git refs, so the change was also built from the worktree withuv run sphinx-build -b html docs <tmp>:spark.htmlrenders the new list and example, and theasync-spark-cursorandaio-spark-cursorreferences resolve. The build reports no warning fordocs/spark.md; its docutils errors come from existing API docstrings.🤖 Generated with Claude Code