Skip to content

Replace the cross-thread cancel example in the Spark documentation - #1270

Merged
laughingman7743 merged 1 commit into
masterfrom
docs/1248-spark-cancel-example
Oct 18, 2026
Merged

laughingman7743 merged 1 commit into
masterfrom
docs/1248-spark-cancel-example

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

WHAT

  • Remove the threading.Timer example from the Spark "Cancellation" section, which called cancel() on a SparkCursor from another thread while execute() ran.
  • State that this cross-thread use is not supported under threadsafety = 2, and list the supported ways to stop a long calculation: KeyboardInterrupt with kill_on_interrupt, AsyncSparkCursor.cancel(calculation_id), and AioSparkCursor with asyncio.wait_for().
  • Add an asyncio.wait_for() example to the AioSparkCursor section, mirroring the SQL asyncio cursor documentation in docs/aio.md.

WHY

Closes #1248. #1206 (PR #1238) documented that the methods of one cursor, including cancel(), must not be called from several threads, and removed the equivalent SQL example from docs/usage.md. The Spark example contradicted that contract.

The wait_for() behavior was checked against pyathena/aio/spark/cursor.py: with kill_on_interrupt enabled, a task cancellation in _poll() requests StopCalculationExecution, waits for a terminal state via _cancel_and_wait(), and re-raises; _calculate() does the same for a cancellation during the start request. asyncio.wait_for() turns that cancellation into asyncio.TimeoutError, and calculation_id and state keep the cancelled calculation.

The tests that call SparkCursor.cancel() from another thread are kept, as decided in #1238.

TEST

  • just docs lint: 0 errors.
  • just docs build: succeeded. It builds git refs, so the change was also built from the worktree with uv run sphinx-build -b html docs <tmp>: spark.html renders the new list and example, and the async-spark-cursor and aio-spark-cursor references resolve. The build reports no warning for docs/spark.md; its docutils errors come from existing API docstrings.
  • Documentation only; no AWS tests are needed.

🤖 Generated with Claude Code

Calling cancel() from another thread on a cursor whose execute() is running
is not supported under threadsafety = 2. List the supported ways to stop a
long calculation instead, and show asyncio.wait_for() with AioSparkCursor.

Closes #1248

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/spark.md
import threading
from pyathena import connect
from pyathena.spark.cursor import SparkCursor
As `pyathena.threadsafety = 2` declares, calling `cancel()` from another thread while `execute()` runs on the same cursor is not supported.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 1 (implementation behavior): CLEAN

Base e086eac7bb2120e434c08c2eababccd9d0167fb0, head 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19. Scope: docs/spark.md (Cancellation and AioSparkCursor sections).

  • The removed Timer example was the only cross-thread cancel() usage in the docs (grep for Timer and "another thread" across docs/ and pyathena/spark, pyathena/aio/spark). docs/sqlalchemy.md cancels from another thread through a new cursor on the shared connection, which threadsafety = 2 allows.
  • Each listed alternative is checked against the code: SparkCursor._calculate/_poll (KeyboardInterrupt), AsyncSparkCursor.execute() returning (calculation_id, future) and cancel(query_id) (pyathena/spark/async_cursor.py:205-253), and AioSparkCursor._calculate/_poll (pyathena/aio/spark/cursor.py:98-257), which cancel the calculation on task cancellation with kill_on_interrupt.
  • The existing AioSparkCursor create_task + await cursor.cancel() example runs on one event loop thread, so it is not affected and is kept.
  • The tests that call SparkCursor.cancel() from another thread (tests/pyathena/spark/test_spark_cursor.py:162) are kept, as decided in Break result sets after failed or cancelled fetches and drop the cross-thread guards #1238.

Comment thread docs/spark.md
If the task is cancelled before `execute()` begins the request, the request is never sent.
Cancelling the task again during this wait raises `asyncio.CancelledError` at once without cancelling the calculation.

With `kill_on_interrupt` enabled, a timeout from `asyncio.wait_for()` while `execute()` starts or waits for the calculation therefore cancels it and raises `asyncio.TimeoutError`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round 2 (claims, callers, documentation reader): CLEAN

Base e086eac7bb2120e434c08c2eababccd9d0167fb0, head 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19.

Claims checked:

  • "a timeout from asyncio.wait_for() ... cancels it and raises asyncio.TimeoutError": verified offline with an AioSparkCursor whose first status request blocks (mocked _calculate, _poll_until_terminal, _cancel). On Python 3.11 (where wait_for runs the coroutine in a separate task) and 3.14 (same task via asyncio.timeout), the result was TimeoutError, _cancel awaited once with the calculation ID, calculation_id == "calculation_id", and state == "CANCELED". The example's cursor.calculation_id and cursor.state therefore print the cancelled calculation. requires-python = ">=3.11".
  • The start-phase case (timeout while StartCalculationExecution runs) follows the existing paragraph above, so it is not repeated.
  • {ref} targets async-spark-cursor and aio-spark-cursor resolve in a worktree sphinx-build (spark.html).
  • The example mirrors the SQL one in docs/aio.md (Task cancellation) and uses the same aio_connect / asyncio.to_thread(conn.cursor) pattern as the other AioSparkCursor examples.

Limits: no live Spark run, since this is a documentation-only change. The live task-cancellation path is covered by tests/pyathena/aio/spark/test_cursor.py::test_cancel.

Comment thread docs/spark.md
If the task is cancelled before `execute()` begins the request, the request is never sent.
Cancelling the task again during this wait raises `asyncio.CancelledError` at once without cancelling the calculation.

With `kill_on_interrupt` enabled, a timeout from `asyncio.wait_for()` while `execute()` starts or waits for the calculation therefore cancels it and raises `asyncio.TimeoutError`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI 0.163.0 (codex exec -s read-only, session 01a14cf6-50bd-78c2-a423-58dafce47c29), static review on a detached snapshot of 32dbdd45bf46285c1fc5c77d3a55bc02795e9c19, base e086eac7bb2120e434c08c2eababccd9d0167fb0. The prompt contained only the diff, the threadsafety contract, and the review request (no PR text or prior findings). The snapshot and the PR worktree were unchanged afterwards.

Result: CLEAN.

Covered: both changed hunks; SparkCursor and the shared start, poll, interrupt, cancel, and state handling; AsyncSparkCursor's calculation ID/future return and ID-based cancellation; AioSparkCursor's timeout handling, defaults, cursor creation, context managers, and reported state; related usage, asyncio, SQLAlchemy, cursor/API docs, cross-references, and relevant tests.

Non-actionable notes: the example follows the page's top-level-await convention (a script needs an async function and asyncio.run()), and since cancellation is best effort, the wait can exceed 60 seconds while the cancellation finishes. Both match the rest of the page and the existing "Cancellation" rules, so there is no change.

@laughingman7743
laughingman7743 marked this pull request as ready for review October 18, 2026 03:17
@laughingman7743
laughingman7743 merged commit 9530737 into master Oct 18, 2026
3 checks passed
@laughingman7743
laughingman7743 deleted the docs/1248-spark-cancel-example branch October 18, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the cross-thread cancel() example in the Spark documentation

1 participant