Spark: verify best-effort calculation cancellation without mandatory session termination #797

Parent: #791

Use case

The current cancellation tests call both cancel() and close() before checking the outcome, so they do not establish what calculation cancellation achieves while the session remains alive.
The documentation also says cancellation appears to require session termination.

Evidence: synchronous test, thread-based test, asyncio test, and documentation.

AWS defines StopCalculationExecution as a best-effort cancellation request.
Acceptance of that request does not guarantee an immediate or eventual CANCELED result.
Session termination is a broader operation and must respect ownership.
This issue is an investigation and coverage gap; no new live-AWS cancellation failure has been reproduced in this session.

Proposed change

Measure the behavior of calculation-only cancellation for all three cursor variants and establish the supported contract.
Separate requesting cancellation, observing a terminal state, and terminating a session.
Clarify the behavior of synchronous KeyboardInterrupt, thread futures, and native asyncio task cancellation, including kill_on_interrupt.
Define what happens when cancellation cannot complete, polling fails, or close overlaps with execution; do not promise unconditional CANCELED or indefinite waits.

Validation plan (if implementing)

At revision a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e, only source and test inspection has been performed for this issue.

• Run isolated real-AWS experiments that record calculation and session IDs, observed states, timings, and stop/terminate calls.
• Attempt calculation-only cancellation and verify a subsequent calculation on the same session when it remains usable.
• Cover already-terminal calculations and cancellation racing with completion.
• Check interruption, polling failure, and close/execute races with deterministic local tests, then relevant service validation.
• Bound experiment duration and clean up test-owned sessions explicitly after measurement; record fallback termination separately from calculation-only success.
• Update the tests and documentation to reflect observed service behavior and the agreed contract.

Serialize live Spark runs and use contributor-controlled resources; no AWS execution was performed while filing these issues.