Skip to content

Spark: an interrupt during StartCalculationExecution leaves the calculation running #841

Closed

Description

@laughingman7743

Problem

The Spark cursors handle an interrupt only while they poll a calculation.
SparkCursor.execute() and AioSparkCursor.execute() first call _calculate(), which sends StartCalculationExecution, and only then call _poll():

  • Synchronous: a KeyboardInterrupt raised while StartCalculationExecution is in flight (including retry_api_call backoff sleeps) propagates before the cursor knows the calculation ID.
  • asyncio: async_retry_api_call runs the request in asyncio.to_thread(); cancelling the awaiting task does not stop the thread, so the request can still create the calculation after the task has been cancelled.
  • AsyncSparkCursor.execute() also calls _calculate() on the caller's thread before submitting the poll.

If Athena accepted the request, the calculation keeps running without a stop request, and calculation_id is not set on the cursor.
While it runs, the session is BUSY: in the #797 measurement, StartCalculationExecution on a BUSY session failed with InvalidRequestException, so subsequent execute() calls on the same cursor fail until the orphaned calculation finishes.
Plain-Python code on the driver is not interrupted even by a stop request (#797), so the orphaned calculation can hold the session for its full runtime.

Evidence (master at c01c56f): AioSparkCursor.execute, SparkCursor.execute, and async_retry_api_call.
Raised by the independent review of #833; out of scope there.

Reproduction

Not reproduced against Athena.
Deterministically: make start_calculation_execution block on an event, cancel the awaiting asyncio task (or interrupt the sync call), release the event so the request "succeeds", and observe that no StopCalculationExecution is sent and calculation_id stays unset.

Environment

master at c01c56f; SparkCursor, AsyncSparkCursor, AioSparkCursor.

Proposed fix (optional)

Investigate first:

  • Whether StartCalculationExecution with a caller-supplied ClientRequestToken is idempotent in practice, so that the cursor could learn the calculation ID after an interrupted start and stop it.
  • For asyncio, whether shielding the start request and then stopping the calculation it created is sufficient.

Keep kill_on_interrupt semantics consistent with #833 (stop request, bounded wait, re-raise the original interrupt).
The SQL cursors have the same gap around StartQueryExecution; decide separately whether to address it.

Related: #791, #797, #833.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions