Skip to content

perf: start the next local partition when a running one finishes - #7693

Open
viirya wants to merge 1 commit into
apache:mainfrom
viirya:local-exec-free-slots
Open

viirya wants to merge 1 commit into
apache:mainfrom
viirya:local-exec-free-slots

Conversation

@viirya

@viirya viirya commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #6499. Follow-up to #6883, found while benchmarking #7692.

Rationale for this change

A local query runs at most as many partitions at once as Spark runs tasks. Under the current scheduler, the next partition starts only when the first undelivered one is delivered. So a partition that finishes before its turn keeps its slot until it is delivered, and one slow partition leaves the slots behind it idle. Spark instead starts the next task whenever any task ends.

Range partitions are uniform, so this rarely shows with the ranges #6883 admits. Scan partitions are not: their splits differ in size and some read a file footer. With #7692, collecting a Parquet scan of 320 small partitions on local[8] took about 45% longer locally than with ordinary Comet. Per-partition timing of that case showed:

  • Planning and starting a partition took about 50 µs.
  • Running a partition took a median of 340 µs and up to 2.9 ms.
  • Finished partitions then waited a mean of 1.4 ms for delivery while keeping their slots.
  • On average only 2.9 of the 8 slots were busy.

What changes are included in this PR?

  • When a partition's task finishes, it drops the partition's stream, which releases its operators and their memory consumers, and wakes the reader so that another partition starts. The bound on running partitions now counts only partitions that have not finished. It still bounds how many partitions share the memory pool.
  • A finished partition waits for its turn holding the batches it has not delivered. Once as many partitions as may run at once are started and not delivered, another starts only while the undelivered batches total less than the read-ahead limit (spark.driver.maxResultSize). Without this bound, finished partitions could pile up their batches behind a slow one; in the experiment without it, up to 57 were waiting at once.
  • local-execution.md describes the new rule.

How are these changes tested?

  • New tests in native/local:
    • a_partition_that_finishes_before_its_turn_gives_up_its_slot: while the first partition runs, each later partition that finishes drops its stream and lets the next one start.
    • finished_partitions_wait_for_their_turn_within_the_read_ahead_limit: with no read-ahead, a finished partition does not let a partition start beyond those that may run at once.
  • Two tests were rewritten because they asserted the old rule, that a partition starts only after the first is delivered: partition_order_runs_at_most_the_given_number_of_partitions and partition_order_follows_a_changing_running_limit. They now use partitions that run until the test lets them finish, so the bound is checked against running partitions. The first also replaces partitions_from_a_start_function_are_planned_as_they_start, since it plans its partitions with a start function.
  • Without the fix, a_partition_that_finishes_before_its_turn_gives_up_its_slot and the rewritten partition_order_runs_at_most_the_given_number_of_partitions fail. The read-ahead test passes either way, since it guards the bound this change adds.

Benchmark (collect, local[8], Apple M4 Max, release build, 8M rows in 64 Parquet files). The "before" build is #7692 and the "after" build is #7692 with this change; the two differ only in the native library, and the runs alternate between them to cancel drift on the machine. The selective filter cases come from three runs of each build at 30 iterations, in the order before/after, after/before, before/after. The other cases come from two full runs of each build at 10 iterations, in the order before/after, before/after. Times are averages in ms, before → after:

Case Partitions Spark Comet Comet local
Selective filter (8K rows returned) 320 94 → 91 71 → 71 96 → 39
Selective filter (8K rows returned) 8 65 → 63 33 → 33 31 → 31
All rows 320 1158 → 1237 1097 → 1154 1108 → 1059
All rows 8 1258 → 1330 1116 → 1199 1051 → 1227
All rows, half in one file 8 1713 → 1674 1556 → 1426 1140 → 1056
Range, all rows 8 527 → 510 534 → 490 423 → 449
Range, selective filter 8 9 → 8 15 → 16 16 → 15

Spark and ordinary Comet do not run the code this change touches, so their columns show the noise between runs. With many small partitions, local execution goes from about 35% slower than ordinary Comet to about 1.8x faster, and that case varied by less than 3 ms across runs. The other cases move within the noise that Spark and ordinary Comet show. The largest move is local execution of all rows over 8 partitions, but its single runs ranged from 964 to 1283 ms, and with 8 partitions all of them start at once, so this change does not alter their scheduling. CometLocalExecutionSuite also passes (35 tests) with this change applied on top of #7692.

A local query started the next partition only when the first undelivered one was delivered. A
partition that finished ahead of its turn kept its slot, so one slow partition left the slots of
the finished partitions behind it idle. A finished partition now drops its stream, which releases
its operators, and gives up its slot. Beyond the partitions that may run at once, another starts
only while the undelivered batches total less than the read-ahead limit, so finished partitions
do not pile up their batches behind a slow one.

Co-authored-by: Isaac <no-reply@databricks.com>

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scheduling change reads correctly to me. The finished waker is registered before the partitions are counted, the task drops the stream outside the slot lock, and #7692's turn = next - running.len() still holds since running keeps every started partition that is not delivered. Two points below, one on the error path and one on what the new read-ahead test proves.

Comment thread native/local/src/lib.rs
Comment on lines +351 to +359
let mut running = self
.running
.iter()
.filter(|partition| !partition.finished.load(Ordering::Acquire))
.count();
while running < limit
&& self.next < partitions
&& (self.running.len() < limit
|| self.read_ahead.held.load(Ordering::Acquire) < self.read_ahead.limit)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A partition that sends an error now also sets finished (the task returns from the error arm and reaches the drop at the end), so it frees its slot and start_partitions keeps starting the partitions behind it, up to the read-ahead bound. None of those can be delivered: when the failed partition's turn comes, fail clears running, and a consumer that stops earlier never reaches them either. Before this change the failed partition kept its slot, so at most max_running - 1 others ran behind it. With the default 1 GiB spark.driver.maxResultSize, I expect a fast failure behind a slow first partition to now run most of the remaining partitions for nothing. Could the task record the failure on ReadAhead (for example a failed flag set before finished), and start_partitions return early once it is set? Every partition before the failed one is already started, so stopping there cannot stall delivery.

Comment on lines +637 to +643
let reader = tokio::spawn(gated.query(2, 0).execute().unwrap().try_collect());
gated.wait_started(2).await;
// The second partition finishes, but it holds a batch beyond the read-ahead limit, so no
// partition starts beyond the two that are not delivered.
gated.open(1);
gated.wait_dropped(1).await;
gated.wait_started(2).await;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With read_ahead 0, held < limit is never true, so the third partition cannot start whether or not partition 1's batch is counted in held. The test pins the running.len() < limit gate, but not what the comment on line 639 describes. With a limit of 1 byte, the test would fail if the finished partition's batch were left out of held.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants