You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark: an interrupt during StartCalculationExecution leaves the calculation running #841
The Spark cursors handle an interrupt only while they poll a calculation. SparkCursor.execute() and AioSparkCursor.execute() first call _calculate(), which sends StartCalculationExecution, and only then call _poll():
Synchronous: a KeyboardInterrupt raised while StartCalculationExecution is in flight (including retry_api_call backoff sleeps) propagates before the cursor knows the calculation ID.
asyncio: async_retry_api_call runs the request in asyncio.to_thread(); cancelling the awaiting task does not stop the thread, so the request can still create the calculation after the task has been cancelled.
AsyncSparkCursor.execute() also calls _calculate() on the caller's thread before submitting the poll.
If Athena accepted the request, the calculation keeps running without a stop request, and calculation_id is not set on the cursor.
While it runs, the session is BUSY: in the #797 measurement, StartCalculationExecution on a BUSY session failed with InvalidRequestException, so subsequent execute() calls on the same cursor fail until the orphaned calculation finishes.
Plain-Python code on the driver is not interrupted even by a stop request (#797), so the orphaned calculation can hold the session for its full runtime.
Not reproduced against Athena.
Deterministically: make start_calculation_execution block on an event, cancel the awaiting asyncio task (or interrupt the sync call), release the event so the request "succeeds", and observe that no StopCalculationExecution is sent and calculation_id stays unset.
Environment
master at c01c56f; SparkCursor, AsyncSparkCursor, AioSparkCursor.
Proposed fix (optional)
Investigate first:
Whether StartCalculationExecution with a caller-supplied ClientRequestToken is idempotent in practice, so that the cursor could learn the calculation ID after an interrupted start and stop it.
For asyncio, whether shielding the start request and then stopping the calculation it created is sufficient.
Keep kill_on_interrupt semantics consistent with #833 (stop request, bounded wait, re-raise the original interrupt).
The SQL cursors have the same gap around StartQueryExecution; decide separately whether to address it.
Activity