Skip to content

refactor(core): share job setup and status handling in DistributedQueryExec - #2884

Open
marvelefe wants to merge 2 commits into
apache:mainfrom
marvelefe:main
Open

marvelefe wants to merge 2 commits into
apache:mainfrom
marvelefe:main

Conversation

@marvelefe

@marvelefe marvelefe commented Nov 17, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #2598.

Rationale for this change

execute_query_pull and execute_query_push duplicated connection setup, job-success handling and status messages. The Substrait example had a third copy of the status loop. This is a refactor; pull and push are both kept, as agreed in the issue thread.

What changes are included in this PR?

  • distributed_query.rs: adds connect_scheduler, finish_successful_job, log_job_wait and job_failed. PipelinedFetch::for_job bundles the fetch settings, which removes the too_many_arguments allows. fetch_partition takes PipelinedFetch.
  • distributed_query.rs and mod.rs: adds the public poll_job_until_successful helper and exports SchedulerClient.
  • standalone-substrait.rs: replaces the private poll_job with the helper.
  • Tests: scripted scheduler covering success (timing gauges), failure and polling for pull and push.

Are there any user-facing changes?

No behavior change for DistributedQueryExec. The new public helper is additive. The Substrait example now sends gRPC metadata interceptor headers (empty by default) and returns an error for pipelined jobs.

Verification

  • Lint: Lint, Clippy, Check for unused dependencies, Check Cargo.toml formatting, Check generated config docs are in sync, Check vendored DataFusion proto is in sync, cargo doc (CI, passing)
  • distributed_query and ballista-core tests: test linux crates, macos test (CI, passing)
  • Substrait example run (--features ballista-scheduler/substrait): runs end to end, returns 8 rows
  • ballista client tests with ballista.client.pull=false (default): test linux ballista (CI), and locally
  • ballista client tests with ballista.client.pull=true: passes locally

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @marvelefe, this is a really nice cleanup. The pull and push loops are much easier to follow now, and I checked that the error and log messages, poll timing, and fetch settings all come through unchanged. The new FakeScheduler tests are great too. One thing I'd like to see before approving is inline on poll_job_until_successful, and the rest are small. If it's easy, could you also run the ballista client tests once with ballista.client.pull=true and the Substrait example end to end, and tick those boxes in the description?

..
}),
) if !pipelined_output_locations.is_empty() => {
return Err(DataFusionError::NotImplemented(format!(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this sees a running pipelined job it returns an error, but the job is still running on the cluster. Since nobody reads the final stage, it holds its vcores until the producer hits ballista.pipelined.registration_timeout_seconds. Could we call cancel_job before returning in the Running case? FakeScheduler already records cancellations, so poll_job_until_successful_rejects_pipelined_jobs could assert on scheduler.cancelled() too.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. poll_job_until_successful now cancels a running pipelined job before returning its error. Finished pipelined jobs are not cancelled. poll_job_until_successful_rejects_pipelined_jobs now asserts the cancellation through FakeScheduler::cancelled().

///
/// Pipelined jobs are not supported here, because their results stream while
/// the job runs. Use [DistributedQueryExec] to read them.
pub async fn poll_job_until_successful(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question on the new public surface. poll_job_until_successful only accepts the intercepted SchedulerClient, which is why the Substrait example had to switch to with_interceptor. Would it be worth making it generic over the client's transport, so embedders with their own SchedulerGrpcClient<T> can use it too? Happy either way if you'd rather keep it simple for now.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm leaning towards keeping poll_job_until_successful on SchedulerClient for now. It already carries Ballista's metadata interceptor, and from my checks, nothing in the repo needs a generic version yet.

Happy to add it in a follow-up if you'd like - I'll create its own issue. Just let me know

io_retries_times,
io_retry_wait_time_ms,
registration_timeout,
..fetch.clone()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tiny one. This arm uses ..fetch.clone() but it breaks right after, so ..fetch should compile just like the Successful arm. Same at line 1217.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed and updated now. Both pipelined arms now use ..fetch

Some(job_status::Status::Failed(err)) => {
return Err(job_failed(&job_id, err.error));
}
Some(job_status::Status::Successful(successful)) => return Ok(successful),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal, but the Substrait example used to log Job ... finished executing in ..., and this helper doesn't log on success. Could it log here like finish_successful_job does?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. The helper logs Job ... finished executing in ... on success, matching the stage-based path.

@marvelefe

Copy link
Copy Markdown
Author

Thanks @andygrove. I've addressed the inline comments. I ran the ballista client tests with ballista.client.pull=true and the Substrait example end to end, and both pass. I've ticked those boxes in the description

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deduplicate execute_query_pull and execute_query_push in DistributedQueryExec

2 participants