Skip to content

Spark: stop readiness polling on failed, terminated, or degraded session states #792

Closed

Description

@laughingman7743

Parent: #791

Problem

SparkBaseCursor._wait_for_idle_session() checks session_status.state for IDLE, but compares the status object itself with the string constants for TERMINATED, DEGRADED, and FAILED.
That error branch cannot match an AthenaSessionStatus instance.
As long as the API continues returning one of those states, cursor construction keeps polling instead of reporting the failure reason.
This affects all three Spark cursor variants through their shared constructor.

Evidence: readiness loop, callers, and status model.
AWS documents these session states.

Reproduction

Use a real AthenaSessionStatus with Status.State set to FAILED and a StateChangeReason.
Return it from a mocked _get_session_status() and intercept time.sleep() so the probe cannot hang.
The current implementation reaches sleep instead of raising OperationalError.
The same probe reproduced this for TERMINATED and DEGRADED; no AWS calls are needed.

Environment

Checked at a03bc2b4ea5dddcbf6230aafda0a359dce12ef0e with Python 3.13.1.
Evidence below is from source inspection and offline probes with AWS calls replaced by mocks; no live Athena validation was performed.

Proposed fix (optional)

Compare the state value in both branches and preserve the reported failure reason.
Keep this fix focused on the existing failure-state check; any new startup deadline is a separate policy decision.

Acceptance:

  • IDLE completes the wait; FAILED, TERMINATED, and DEGRADED raise with the available reason without sleeping again.
  • CREATING/BUSY followed by IDLE still works.
  • New-session and supplied-session paths use the corrected behavior.
  • Add deterministic regression coverage without invoking the AWS-dependent pytest session setup unnecessarily.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions