Repository navigation
Conversation
A local query started the next partition only when the first undelivered one was delivered. A partition that finished ahead of its turn kept its slot, so one slow partition left the slots of the finished partitions behind it idle. A finished partition now drops its stream, which releases its operators, and gives up its slot. Beyond the partitions that may run at once, another starts only while the undelivered batches total less than the read-ahead limit, so finished partitions do not pile up their batches behind a slow one. Co-authored-by: Isaac <no-reply@databricks.com>
comphead
left a comment
There was a problem hiding this comment.
The scheduling change reads correctly to me. The finished waker is registered before the partitions are counted, the task drops the stream outside the slot lock, and #7692's turn = next - running.len() still holds since running keeps every started partition that is not delivered. Two points below, one on the error path and one on what the new read-ahead test proves.
| let mut running = self | ||
| .running | ||
| .iter() | ||
| .filter(|partition| !partition.finished.load(Ordering::Acquire)) | ||
| .count(); | ||
| while running < limit | ||
| && self.next < partitions | ||
| && (self.running.len() < limit | ||
| || self.read_ahead.held.load(Ordering::Acquire) < self.read_ahead.limit) |
There was a problem hiding this comment.
A partition that sends an error now also sets finished (the task returns from the error arm and reaches the drop at the end), so it frees its slot and start_partitions keeps starting the partitions behind it, up to the read-ahead bound. None of those can be delivered: when the failed partition's turn comes, fail clears running, and a consumer that stops earlier never reaches them either. Before this change the failed partition kept its slot, so at most max_running - 1 others ran behind it. With the default 1 GiB spark.driver.maxResultSize, I expect a fast failure behind a slow first partition to now run most of the remaining partitions for nothing. Could the task record the failure on ReadAhead (for example a failed flag set before finished), and start_partitions return early once it is set? Every partition before the failed one is already started, so stopping there cannot stall delivery.
| let reader = tokio::spawn(gated.query(2, 0).execute().unwrap().try_collect()); | ||
| gated.wait_started(2).await; | ||
| // The second partition finishes, but it holds a batch beyond the read-ahead limit, so no | ||
| // partition starts beyond the two that are not delivered. | ||
| gated.open(1); | ||
| gated.wait_dropped(1).await; | ||
| gated.wait_started(2).await; |
There was a problem hiding this comment.
With read_ahead 0, held < limit is never true, so the third partition cannot start whether or not partition 1's batch is counted in held. The test pins the running.len() < limit gate, but not what the comment on line 639 describes. With a limit of 1 byte, the test would fail if the finished partition's batch were left out of held.
Which issue does this PR close?
Part of #6499. Follow-up to #6883, found while benchmarking #7692.
Rationale for this change
A local query runs at most as many partitions at once as Spark runs tasks. Under the current scheduler, the next partition starts only when the first undelivered one is delivered. So a partition that finishes before its turn keeps its slot until it is delivered, and one slow partition leaves the slots behind it idle. Spark instead starts the next task whenever any task ends.
Range partitions are uniform, so this rarely shows with the ranges #6883 admits. Scan partitions are not: their splits differ in size and some read a file footer. With #7692, collecting a Parquet scan of 320 small partitions on
local[8]took about 45% longer locally than with ordinary Comet. Per-partition timing of that case showed:What changes are included in this PR?
spark.driver.maxResultSize). Without this bound, finished partitions could pile up their batches behind a slow one; in the experiment without it, up to 57 were waiting at once.local-execution.mddescribes the new rule.How are these changes tested?
native/local:a_partition_that_finishes_before_its_turn_gives_up_its_slot: while the first partition runs, each later partition that finishes drops its stream and lets the next one start.finished_partitions_wait_for_their_turn_within_the_read_ahead_limit: with no read-ahead, a finished partition does not let a partition start beyond those that may run at once.partition_order_runs_at_most_the_given_number_of_partitionsandpartition_order_follows_a_changing_running_limit. They now use partitions that run until the test lets them finish, so the bound is checked against running partitions. The first also replacespartitions_from_a_start_function_are_planned_as_they_start, since it plans its partitions with a start function.a_partition_that_finishes_before_its_turn_gives_up_its_slotand the rewrittenpartition_order_runs_at_most_the_given_number_of_partitionsfail. The read-ahead test passes either way, since it guards the bound this change adds.Benchmark (
collect,local[8], Apple M4 Max, release build, 8M rows in 64 Parquet files). The "before" build is #7692 and the "after" build is #7692 with this change; the two differ only in the native library, and the runs alternate between them to cancel drift on the machine. The selective filter cases come from three runs of each build at 30 iterations, in the order before/after, after/before, before/after. The other cases come from two full runs of each build at 10 iterations, in the order before/after, before/after. Times are averages in ms, before → after:Spark and ordinary Comet do not run the code this change touches, so their columns show the noise between runs. With many small partitions, local execution goes from about 35% slower than ordinary Comet to about 1.8x faster, and that case varied by less than 3 ms across runs. The other cases move within the noise that Spark and ordinary Comet show. The largest move is local execution of all rows over 8 partitions, but its single runs ranged from 964 to 1283 ms, and with 8 partitions all of them start at once, so this change does not alter their scheduling.
CometLocalExecutionSuitealso passes (35 tests) with this change applied on top of #7692.