Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions docs/source/contributor-guide/local-execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,17 +71,22 @@ single partition. That partition's task creates the native graph:
task, so a start that waits, for instance for a memory grant, does not hold up the pull, and a
partition's error, including a failure to start, is reported when its turn comes.
- At most as many partitions run at once as Spark runs tasks, `spark.executor.cores` divided by
`spark.task.cpus`, and the next starts when the first is delivered. A partition's operators
register their memory consumers when it starts, so this also bounds how many share the task's
memory pool. Whenever another partition would start, the bound is further divided among the
executor's tasks running native plans, this one included: Comet's pools give each of those
tasks an equal share of Spark's memory, so a partition gets what a task would. Partitions
already running keep running, so a job that starts after a local query has started all its
partitions still leaves those partitions less than a task's share, until they end.
`spark.task.cpus`, and another starts whenever one finishes, as Spark starts a task whenever
one ends. A partition's operators register their memory consumers when it starts and are
dropped when it finishes, so this also bounds how many share the task's memory pool. A
finished partition waits for its turn with the batches it has not delivered, so a slow
partition does not hold up the others. Whenever another partition would start, the bound is
further divided among the executor's tasks running native plans, this one included: Comet's
pools give each of those tasks an equal share of Spark's memory, so a partition gets what a
task would. Partitions already running keep running, so a job that starts after a local query
has started all its partitions still leaves those partitions less than a task's share, until
they end.
- The running partitions do not wait for their turn to produce batches, so they all make
progress, as Spark's tasks do. What they hold ahead of delivery is bounded by
`spark.driver.maxResultSize`, the most a collect may bring to the driver; past it, each holds
at most two batches until the earlier partitions are delivered. These batches are not reserved
at most two batches until the earlier partitions are delivered. Once as many partitions as
run at once are started and not delivered, another starts only below that bound, so finished
partitions do not pile up their batches behind a slow one. These batches are not reserved
in the memory pool: a reservation could not be given back when an operator later needs the
memory, so it would fail queries that ordinary execution runs. Spark's own collect also holds
its task results outside the execution pool, on the driver.
Expand Down Expand Up @@ -161,8 +166,9 @@ make core
```

The native tests cover partition-order delivery, the bound on running partitions and its
changes, partition starts that wait, read-ahead, errors reported in a partition's turn,
cancellation with busy workers, the pull that waits on its own thread, and runtime shutdown. `CometLocalExecutionSuite` compares admitted queries
changes, partitions that finish before their turn, partition starts that wait, read-ahead,
errors reported in a partition's turn, cancellation with busy workers, the pull that waits on
its own thread, and runtime shutdown. `CometLocalExecutionSuite` compares admitted queries
with Spark's results and order, and covers ranges Spark would overflow, declined expressions,
AQE, reported orderings, limits left to ordinary execution, iterators, tails, many partitions with sorts in a small memory pool, caching and other RDD consumers, the result size limit, job and task cancellation, and native errors. After
each test it checks that no native query, imported Arrow memory or result slot is left.
53 changes: 45 additions & 8 deletions native/local/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ use datafusion::physical_plan::{
ExecutionPlan, ExecutionPlanProperties, RecordBatchStream, SendableRecordBatchStream,
};
use futures::future::poll_fn;
use futures::task::AtomicWaker;
use futures::{FutureExt, Stream, StreamExt};
use tokio::sync::Notify;
use tokio::task::JoinHandle;
Expand Down Expand Up @@ -69,10 +70,16 @@ impl LocalQuery {
/// Takes ownership of a fresh graph and its query-scoped context.
///
/// The root partitions are delivered one after another, in partition order, as Spark returns
/// the partitions of a result. Up to `max_running` partitions run at once, and the next one
/// starts when the first is delivered, as Spark runs at most as many tasks as it has cores.
/// A partition's operators register their memory consumers when it starts, so this also
/// bounds how many share the memory pool.
/// the partitions of a result. Up to `max_running` partitions run at once, and another one
/// starts whenever one finishes, as Spark runs at most as many tasks as it has cores and
/// starts the next whenever one ends. A partition's operators register their memory
/// consumers when it starts and are dropped when it finishes, so this also bounds how many
/// share the memory pool.
///
/// A finished partition waits for its turn with the batches it has not delivered. Once
/// `max_running` partitions are started and not delivered, another starts only while the
/// batches they all hold total less than `read_ahead` bytes, so that partitions which finish
/// ahead of a slow one do not pile up their batches.
///
/// A partition starts its operators on its own task, so a start that waits, for instance to
/// register a memory consumer, does not hold up delivery. A failure to start is reported in
Expand Down Expand Up @@ -156,6 +163,7 @@ fn in_partition_order(query: &LocalQuery) -> SendableRecordBatchStream {
limit: query.read_ahead,
held: AtomicUsize::new(0),
delivered: Notify::new(),
finished: AtomicWaker::new(),
}),
running: VecDeque::new(),
};
Expand All @@ -174,6 +182,8 @@ struct ReadAhead {
held: AtomicUsize,
/// Notified whenever a batch is delivered.
delivered: Notify,
/// Woken whenever a partition finishes, so that another can start.
finished: AtomicWaker,
}

/// One root partition running on its own Tokio task, ahead of delivery.
Expand All @@ -186,6 +196,8 @@ struct Partition {
/// task ends, and a runtime that shuts down cancels its tasks, so only this tells that
/// every batch arrived.
ended: Arc<AtomicBool>,
/// Set once the task has dropped the partition's stream, however it ended.
finished: Arc<AtomicBool>,
/// The partition's stream. The task locks it only while polling it, so dropping the
/// partition can take it whenever the task is waiting.
stream: Arc<Mutex<Slot>>,
Expand All @@ -197,7 +209,8 @@ enum Slot {
/// The task is starting the partition's operators.
Starting,
Running(SendableRecordBatchStream),
/// The partition was dropped. A task that started the stream afterwards drops it.
/// The partition was dropped, or its task finished. A task that started the stream after
/// the partition was dropped drops it.
Closed,
}

Expand All @@ -206,9 +219,14 @@ impl Partition {
let (sender, batches) = tokio::sync::mpsc::unbounded_channel();
let queued = Arc::new(AtomicUsize::new(0));
let ended = Arc::new(AtomicBool::new(false));
let finished = Arc::new(AtomicBool::new(false));
let stream = Arc::new(Mutex::new(Slot::Starting));
let (task_queued, task_ended, polled) =
(Arc::clone(&queued), Arc::clone(&ended), Arc::clone(&stream));
let (task_queued, task_ended, task_finished, polled) = (
Arc::clone(&queued),
Arc::clone(&ended),
Arc::clone(&finished),
Arc::clone(&stream),
);
let read_ahead = Arc::clone(&order.read_ahead);
let (start, context) = (Arc::clone(&order.start), Arc::clone(&order.context));
let task = order.runtime.spawn(async move {
Expand Down Expand Up @@ -276,11 +294,17 @@ impl Partition {
"Local query partition panicked".into(),
)));
}
// Release the operators, and their memory, before another partition takes the slot.
let stream = std::mem::replace(&mut *lock(&polled), Slot::Closed);
let _ = std::panic::catch_unwind(AssertUnwindSafe(|| drop(stream)));
task_finished.store(true, Ordering::Release);
read_ahead.finished.wake();
});
Self {
batches,
queued,
ended,
finished,
stream,
task,
}
Expand Down Expand Up @@ -324,7 +348,17 @@ impl PartitionOrder {
.running_limit
.as_ref()
.map_or(self.max_running, |limit| limit().clamp(1, self.max_running));
while self.running.len() < limit && self.next < partitions {
let mut running = self
.running
.iter()
.filter(|partition| !partition.finished.load(Ordering::Acquire))
.count();
while running < limit
&& self.next < partitions
&& (self.running.len() < limit
|| self.read_ahead.held.load(Ordering::Acquire) < self.read_ahead.limit)
Comment on lines +351 to +359

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A partition that sends an error now also sets finished (the task returns from the error arm and reaches the drop at the end), so it frees its slot and start_partitions keeps starting the partitions behind it, up to the read-ahead bound. None of those can be delivered: when the failed partition's turn comes, fail clears running, and a consumer that stops earlier never reaches them either. Before this change the failed partition kept its slot, so at most max_running - 1 others ran behind it. With the default 1 GiB spark.driver.maxResultSize, I expect a fast failure behind a slow first partition to now run most of the remaining partitions for nothing. Could the task record the failure on ReadAhead (for example a failed flag set before finished), and start_partitions return early once it is set? Every partition before the failed one is already started, so stopping there cannot stall delivery.

{
running += 1;
let partition = Partition::spawn(self, self.next);
self.running.push_back(partition);
self.next += 1;
Expand All @@ -343,6 +377,9 @@ impl Stream for PartitionOrder {

fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
loop {
// Registered before the partitions are counted, so a partition that finishes after
// they are counted wakes this reader to start another.
self.read_ahead.finished.register(cx.waker());
self.start_partitions();
let Some(partition) = self.running.front_mut() else {
return Poll::Ready(None);
Expand Down
Loading