Skip to content

fix(scheduler): fail an AQE job that loses a finished stage's output instead of hanging - #2887

Merged
andygrove merged 7 commits into
apache:mainfrom
abstractdog:aqe-rerun-lost-stage
Nov 19, 2026
Merged

andygrove merged 7 commits into
apache:mainfrom
abstractdog:aqe-rerun-lost-stage

Conversation

@abstractdog

@abstractdog abstractdog commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #1986. Addresses the hang reported in #2539, which this turns into a job failure; see below for what it leaves open.

Rationale for this change

With AQE, a job hangs forever once it loses the shuffle output of a stage that has already finished. I hit this on main (6ee64d6) running TPC-DS SF10000 on 12 executors, with push-staged scheduling: several executors were evicted mid-query, and q40 then sat with no running tasks until a client-side timeout cancelled it 27 minutes later. The scheduler log matches #2539:

WARN  state::aqe: Need to resubmit the current running Stage 2 and its map Stage 0 due to FetchPartitionError from task TID 038FAGQ4P8000/2.0/6
WARN  state::aqe: Reset 1 tasks for successful job/stage 038FAGQ4P8000/1 on lost Executor b152409e-...
WARN  state::task_manager: Failed to update task statuses for job 038FAGQ4P8000, skipping its updates: DataFusion error: Execution error: Can't find active stage to update stage outputs

So the hang does not depend on pull-staged mode.

The cause: when a stage finishes, the AdaptivePlanner drops it from runnable_stage_cache and runnable_stage_output. Losing that stage's output, through a lost executor (reset_stages_on_lost_executor) or a reduce task's FetchPartitionError, makes the graph call rerun_successful_stage. That turns the stage back into a running one in the graph only. The first rerun task to report then fails in update_exchange_locations with "Can't find active stage to update stage outputs". The task manager drops the whole status update on that error, so the stage never completes. Even without that error, update_stage_progress only adds stages the graph does not have yet, so the rolled-back reader stage would never be rebuilt.

Rerunning a finished stage under AQE is the missing recovery that #1986 tracks. Until it exists, failing the job with a clear message is better than hanging it: the client gets an error it can act on, and the job releases its resources.

What changes are included in this PR?

  • reset_stages_on_lost_executor returns an error when the lost executor held output that a finished stage's readers still need. TaskManager::executor_lost already fails the job when this returns an error.
  • Any other rollback a lost executor would cause also fails the job, because the adaptive graph cannot resolve a rolled-back stage again: its stages only come resolved from the planner. losing_the_input_of_a_stage_fails_the_rollback covers it. This wasn't in the initial patch set, added later.
  • A fetch failure on the output of a finished stage fails the reading stage, as a fetch failure on lost cache data already does, so the job fails through the usual path. The rollback and resubmit bookkeeping only that path fed is removed.
  • Both report the same message: the stage's output was lost, AQE cannot rerun a finished stage yet, so retry the query or set ballista.planner.adaptive.enabled to false to use the static planner, which recovers lost output.
  • AdaptiveExecutionGraph::rerun_successful_stage stays, as the ExecutionGraph trait requires it, but the adaptive graph no longer calls it.
  • Tests in aqe/test/job_failure.rs for both paths. losing_a_shared_stage_output_rolls_back_the_stage_once in aqe/test/reuse_exchange.rs recorded the old behaviour; it is renamed to losing_a_shared_stage_output_fails_the_rollback and now expects the error. All three fail without the change.

Not addressed here: actually rerunning the stage (#1986), and the stale tasks #2539 reports in pull-staged mode, which a rolled-back stage cannot cancel.

Are there any user-facing changes?

Yes. An AQE job that loses the output of a finished stage now fails with the error above instead of hanging. The AQE limitations in the tuning guide say so. Losing only the output of running stages is handled as before.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 17, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @abstractdog, this is a nice fix. The root-cause write-up and the log excerpt make the hang really easy to follow. Failing the job with a message that points at the static planner is much better than hanging forever. The new tests fail without the change, which is great to see. A couple of small suggestions inline, none of them blocking.

Comment thread ballista/scheduler/src/state/aqe/mod.rs Outdated
for stage_id in mem::take(&mut updated_stages.resubmit_successful_stages) {
updated_stages
.failed_stages
.insert(stage_id, cannot_rerun_finished_stage(stage_id));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small wording thing. On the fetch-failure path the job error ends up as Job failed due to stage 0 failed: the output of finished stage 0 was lost .... Stage 0 actually succeeded. It was the reader stage whose task failed. Would it read better to key this entry on the reader stage, or to adjust the message so it doesn't say stage 0 failed?

let task = graph.pop_next_task(&b.id)?.unwrap();
let status = mock_failed_task(task, fetch_partition_failure("a", 0, 0));

graph.update_task_status(&b, vec![status], 4, 4)?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this also assert that update_task_status returns a QueryStageSchedulerEvent::JobRunningFailed? That event is what actually aborts the job and cancels its tasks. Checking it would guard the "no longer hangs" part directly, since the status alone could be set without the event going out.

abstractdog and others added 4 commits November 18, 2026 16:03
…instead of hanging

When an executor holding the shuffle output of a finished stage is lost,
or a reduce task fails to fetch that output, the adaptive graph turned
the stage back into a running one to rerun it. Its planner had already
dropped the stage when it finished, though, so the first rerun task to
report failed with "Can't find active stage to update stage outputs".
The task manager drops the whole status update on that error, and the
rolled-back reader stage is never rebuilt either, so the job hung with
no running tasks until cancelled (apache#2539).

Rerunning a finished stage under AQE is not supported yet (apache#1986), so
fail the job instead, with a message that points to retrying the query
or to the static planner, which recovers lost output:

- reset_stages_on_lost_executor returns an error when the lost executor
  held output a finished stage's readers still need; the task manager
  already fails the job on that error.
- A fetch failure that would resubmit a finished stage fails that stage,
  and with it the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review comments:

- On a fetch failure, the job error named the finished map stage as the
  one that failed ("stage 0 failed: the output of finished stage 0 was
  lost"), though its reader was the stage whose task failed. Fail the
  reading stage at the fetch failure instead, as the cache-data-lost case
  above it already does.
- The fetch-failure test also checks that update_task_status returns the
  JobRunningFailed event, which is what aborts the job and cancels its
  tasks, and that the message names the reading stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er uses

A fetch failure now fails its reading stage, so nothing fills
rollback_running_stages or resubmit_successful_stages in
AdaptiveExecutionGraph::update_task_status any more. Remove the two maps
and the loop that marked a resubmitted finished stage's tasks failed,
along with the imports only that loop used. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abstractdog and others added 3 commits November 18, 2026 16:19
reset_stages_internal rolled back every stage whose input was on the lost
executor, and returned an error only when a finished stage of this graph
had to be rerun. A rolled-back stage is unresolved, and nothing in the
adaptive graph resolves a stage again: stages only come resolved from the
planner. A rollback without that error, for an input that is not a
finished stage of this graph, would therefore have left the job to hang.

Fail on any such rollback instead. The rollback code after the checks can
no longer run, so it goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_task_status declared reset_running_stages as an empty, immutable
map, so the check that a successful stage was not in it always passed
and the loop resetting its tasks never ran. The static graph, which this
code was copied from, fills it on late fetch failures; the adaptive graph
has no such path. Noted on apache#1986. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apache#2730 added recovers_external_inputs and resolve_external_input between
resolve_stage and its doc comment, which left "Convert unresolved stage
to be resolved" on recovers_external_inputs. Move it back, and say what
recovers_external_inputs does for the static graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abstractdog

abstractdog commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

thanks @andygrove for the review! added a few more commits, addressing your comments among others, like:
73730ae , which handles one more case: the AQE graph can't rebuild any stage that's rolled back to UnResolved, so any rollback under AQE, not just a finished-stage re-run, should fail the job

origin of most code in AQE is the static planner; I hope this patch can stabilize AQE until further improvements

@andygrove
andygrove added this pull request to the merge queue Nov 19, 2026
@andygrove

Copy link
Copy Markdown
Member

LGTM. Thanks @abstractdog!

Merged via the queue into apache:main with commit 126ea5f Nov 19, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants