diff --git a/docs/source/contributor-guide/local-execution.md b/docs/source/contributor-guide/local-execution.md index 4fc9470ff8d..0d9a7897d25 100644 --- a/docs/source/contributor-guide/local-execution.md +++ b/docs/source/contributor-guide/local-execution.md @@ -71,17 +71,22 @@ single partition. That partition's task creates the native graph: task, so a start that waits, for instance for a memory grant, does not hold up the pull, and a partition's error, including a failure to start, is reported when its turn comes. - At most as many partitions run at once as Spark runs tasks, `spark.executor.cores` divided by - `spark.task.cpus`, and the next starts when the first is delivered. A partition's operators - register their memory consumers when it starts, so this also bounds how many share the task's - memory pool. Whenever another partition would start, the bound is further divided among the - executor's tasks running native plans, this one included: Comet's pools give each of those - tasks an equal share of Spark's memory, so a partition gets what a task would. Partitions - already running keep running, so a job that starts after a local query has started all its - partitions still leaves those partitions less than a task's share, until they end. + `spark.task.cpus`, and another starts whenever one finishes, as Spark starts a task whenever + one ends. A partition's operators register their memory consumers when it starts and are + dropped when it finishes, so this also bounds how many share the task's memory pool. A + finished partition waits for its turn with the batches it has not delivered, so a slow + partition does not hold up the others. Whenever another partition would start, the bound is + further divided among the executor's tasks running native plans, this one included: Comet's + pools give each of those tasks an equal share of Spark's memory, so a partition gets what a + task would. Partitions already running keep running, so a job that starts after a local query + has started all its partitions still leaves those partitions less than a task's share, until + they end. - The running partitions do not wait for their turn to produce batches, so they all make progress, as Spark's tasks do. What they hold ahead of delivery is bounded by `spark.driver.maxResultSize`, the most a collect may bring to the driver; past it, each holds - at most two batches until the earlier partitions are delivered. These batches are not reserved + at most two batches until the earlier partitions are delivered. Once as many partitions as + run at once are started and not delivered, another starts only below that bound, so finished + partitions do not pile up their batches behind a slow one. These batches are not reserved in the memory pool: a reservation could not be given back when an operator later needs the memory, so it would fail queries that ordinary execution runs. Spark's own collect also holds its task results outside the execution pool, on the driver. @@ -161,8 +166,9 @@ make core ``` The native tests cover partition-order delivery, the bound on running partitions and its -changes, partition starts that wait, read-ahead, errors reported in a partition's turn, -cancellation with busy workers, the pull that waits on its own thread, and runtime shutdown. `CometLocalExecutionSuite` compares admitted queries +changes, partitions that finish before their turn, partition starts that wait, read-ahead, +errors reported in a partition's turn, cancellation with busy workers, the pull that waits on +its own thread, and runtime shutdown. `CometLocalExecutionSuite` compares admitted queries with Spark's results and order, and covers ranges Spark would overflow, declined expressions, AQE, reported orderings, limits left to ordinary execution, iterators, tails, many partitions with sorts in a small memory pool, caching and other RDD consumers, the result size limit, job and task cancellation, and native errors. After each test it checks that no native query, imported Arrow memory or result slot is left. diff --git a/native/local/src/lib.rs b/native/local/src/lib.rs index dee31918320..03bcc3908cf 100644 --- a/native/local/src/lib.rs +++ b/native/local/src/lib.rs @@ -39,6 +39,7 @@ use datafusion::physical_plan::{ ExecutionPlan, ExecutionPlanProperties, RecordBatchStream, SendableRecordBatchStream, }; use futures::future::poll_fn; +use futures::task::AtomicWaker; use futures::{FutureExt, Stream, StreamExt}; use tokio::sync::Notify; use tokio::task::JoinHandle; @@ -69,10 +70,16 @@ impl LocalQuery { /// Takes ownership of a fresh graph and its query-scoped context. /// /// The root partitions are delivered one after another, in partition order, as Spark returns - /// the partitions of a result. Up to `max_running` partitions run at once, and the next one - /// starts when the first is delivered, as Spark runs at most as many tasks as it has cores. - /// A partition's operators register their memory consumers when it starts, so this also - /// bounds how many share the memory pool. + /// the partitions of a result. Up to `max_running` partitions run at once, and another one + /// starts whenever one finishes, as Spark runs at most as many tasks as it has cores and + /// starts the next whenever one ends. A partition's operators register their memory + /// consumers when it starts and are dropped when it finishes, so this also bounds how many + /// share the memory pool. + /// + /// A finished partition waits for its turn with the batches it has not delivered. Once + /// `max_running` partitions are started and not delivered, another starts only while the + /// batches they all hold total less than `read_ahead` bytes, so that partitions which finish + /// ahead of a slow one do not pile up their batches. /// /// A partition starts its operators on its own task, so a start that waits, for instance to /// register a memory consumer, does not hold up delivery. A failure to start is reported in @@ -156,6 +163,7 @@ fn in_partition_order(query: &LocalQuery) -> SendableRecordBatchStream { limit: query.read_ahead, held: AtomicUsize::new(0), delivered: Notify::new(), + finished: AtomicWaker::new(), }), running: VecDeque::new(), }; @@ -174,6 +182,8 @@ struct ReadAhead { held: AtomicUsize, /// Notified whenever a batch is delivered. delivered: Notify, + /// Woken whenever a partition finishes, so that another can start. + finished: AtomicWaker, } /// One root partition running on its own Tokio task, ahead of delivery. @@ -186,6 +196,8 @@ struct Partition { /// task ends, and a runtime that shuts down cancels its tasks, so only this tells that /// every batch arrived. ended: Arc, + /// Set once the task has dropped the partition's stream, however it ended. + finished: Arc, /// The partition's stream. The task locks it only while polling it, so dropping the /// partition can take it whenever the task is waiting. stream: Arc>, @@ -197,7 +209,8 @@ enum Slot { /// The task is starting the partition's operators. Starting, Running(SendableRecordBatchStream), - /// The partition was dropped. A task that started the stream afterwards drops it. + /// The partition was dropped, or its task finished. A task that started the stream after + /// the partition was dropped drops it. Closed, } @@ -206,9 +219,14 @@ impl Partition { let (sender, batches) = tokio::sync::mpsc::unbounded_channel(); let queued = Arc::new(AtomicUsize::new(0)); let ended = Arc::new(AtomicBool::new(false)); + let finished = Arc::new(AtomicBool::new(false)); let stream = Arc::new(Mutex::new(Slot::Starting)); - let (task_queued, task_ended, polled) = - (Arc::clone(&queued), Arc::clone(&ended), Arc::clone(&stream)); + let (task_queued, task_ended, task_finished, polled) = ( + Arc::clone(&queued), + Arc::clone(&ended), + Arc::clone(&finished), + Arc::clone(&stream), + ); let read_ahead = Arc::clone(&order.read_ahead); let (start, context) = (Arc::clone(&order.start), Arc::clone(&order.context)); let task = order.runtime.spawn(async move { @@ -276,11 +294,17 @@ impl Partition { "Local query partition panicked".into(), ))); } + // Release the operators, and their memory, before another partition takes the slot. + let stream = std::mem::replace(&mut *lock(&polled), Slot::Closed); + let _ = std::panic::catch_unwind(AssertUnwindSafe(|| drop(stream))); + task_finished.store(true, Ordering::Release); + read_ahead.finished.wake(); }); Self { batches, queued, ended, + finished, stream, task, } @@ -324,7 +348,17 @@ impl PartitionOrder { .running_limit .as_ref() .map_or(self.max_running, |limit| limit().clamp(1, self.max_running)); - while self.running.len() < limit && self.next < partitions { + let mut running = self + .running + .iter() + .filter(|partition| !partition.finished.load(Ordering::Acquire)) + .count(); + while running < limit + && self.next < partitions + && (self.running.len() < limit + || self.read_ahead.held.load(Ordering::Acquire) < self.read_ahead.limit) + { + running += 1; let partition = Partition::spawn(self, self.next); self.running.push_back(partition); self.next += 1; @@ -343,6 +377,9 @@ impl Stream for PartitionOrder { fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { loop { + // Registered before the partitions are counted, so a partition that finishes after + // they are counted wakes this reader to start another. + self.read_ahead.finished.register(cx.waker()); self.start_partitions(); let Some(partition) = self.running.front_mut() else { return Poll::Ready(None); diff --git a/native/local/tests/execution.rs b/native/local/tests/execution.rs index 4cf4660b457..f1a7efaef81 100644 --- a/native/local/tests/execution.rs +++ b/native/local/tests/execution.rs @@ -474,74 +474,174 @@ async fn partition_order_cancel_releases_running_partitions() { wait_released(&refs).await; } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn partition_order_runs_at_most_the_given_number_of_partitions() { - let (plan, observations) = observe(ordered_input(6, 3)); - let executed = || observations.executions.lock().unwrap().clone(); - let mut stream = LocalQuery::new(plan, context(), 2, usize::MAX) - .execute() - .unwrap(); - // The first two partitions start with the query; the rest wait for their turn. - wait_started(&observations, 2).await; - for _ in 0..3 { - timeout(DEADLINE, stream.next()) - .await - .unwrap() - .unwrap() - .unwrap(); +/// Partitions planned by a start function, each of which delivers one batch once its gate opens. +struct Gated { + started: Mutex>, + gates: Vec>, + /// Held by each partition's stream until it is dropped. + streams: Vec>, +} + +impl Gated { + fn new(partitions: usize) -> Arc { + Arc::new(Self { + started: Mutex::new(Vec::new()), + gates: (0..partitions) + .map(|_| Arc::new(tokio::sync::Semaphore::new(0))) + .collect(), + streams: (0..partitions).map(|_| Arc::new(())).collect(), + }) } - assert_eq!(executed(), vec![0, 1]); - // Finishing the first partition starts the third. - timeout(DEADLINE, stream.next()) + + fn query(self: &Arc, max_running: usize, read_ahead: usize) -> LocalQuery { + let gated = Arc::clone(self); + let start: PartitionStart = Arc::new(move |partition, _| { + gated.started.lock().unwrap().push(partition); + let gate = Arc::clone(&gated.gates[partition]); + let held = Arc::clone(&gated.streams[partition]); + let base = partition as i64 * 4; + let stream = futures::stream::once(async move { + gate.acquire().await.unwrap().forget(); + Ok(batch(base..base + 4)) + }) + .inspect(move |_| { + let _ = &held; + }); + Ok(Box::pin(RecordBatchStreamAdapter::new(schema(), stream))) + }); + LocalQuery::from_partitions( + schema(), + self.gates.len(), + start, + context(), + max_running, + read_ahead, + ) + } + + fn open(&self, partition: usize) { + self.gates[partition].add_permits(1); + } + + fn started(&self) -> Vec { + let mut started = self.started.lock().unwrap().clone(); + started.sort_unstable(); + started + } + + async fn wait_started(&self, partitions: usize) { + timeout(DEADLINE, async { + while self.started.lock().unwrap().len() < partitions { + tokio::task::yield_now().await; + } + }) .await - .unwrap() - .unwrap() - .unwrap(); - wait_started(&observations, 3).await; - let rest: Vec = timeout(DEADLINE, stream.try_collect()) + .expect("partitions did not start"); + // Long enough for a partition started beyond the bound to show up. + tokio::time::sleep(Duration::from_millis(50)).await; + assert_eq!(self.started(), (0..partitions).collect::>()); + } + + /// Opens every gate and checks that `reader` read every partition in order. + async fn read_in_order(&self, reader: tokio::task::JoinHandle>>) { + for partition in 0..self.gates.len() { + self.open(partition); + } + let batches = timeout(DEADLINE, reader).await.unwrap().unwrap().unwrap(); + let expected: Vec = (0..self.gates.len() as i64 * 4).collect(); + assert_eq!(ids(&batches), expected); + assert_eq!(self.started(), (0..self.gates.len()).collect::>()); + } + + async fn wait_dropped(&self, partition: usize) { + timeout(DEADLINE, async { + while Arc::strong_count(&self.streams[partition]) > 1 { + tokio::task::yield_now().await; + } + }) + .await + .expect("a finished partition kept its stream"); + } +} + +fn ids(batches: &[RecordBatch]) -> Vec { + batches + .iter() + .flat_map(|batch| { + batch + .column(1) + .as_any() + .downcast_ref::() + .unwrap() + .values() + .to_vec() + }) + .collect() +} + +async fn collect_in_order( + gated: &Gated, + mut stream: impl futures::Stream> + Unpin, + from: usize, +) { + for partition in from..gated.gates.len() { + gated.open(partition); + } + let rest: Vec = timeout(DEADLINE, stream.by_ref().try_collect()) .await .unwrap() .unwrap(); - assert_eq!(rest.len(), 6 * 3 - 4); - assert_executed_once(&observations, 6); + assert_eq!( + ids(&rest), + (from as i64 * 4..gated.gates.len() as i64 * 4).collect::>() + ); + assert_eq!(gated.started(), (0..gated.gates.len()).collect::>()); } #[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn partitions_from_a_start_function_are_planned_as_they_start() { - let planned = Arc::new(Mutex::new(Vec::new())); - let (plans, source) = (Arc::clone(&planned), ordered_input(6, 1)); - let start: PartitionStart = Arc::new(move |partition, context| { - plans.lock().unwrap().push(partition); - source.execute(partition, context) - }); - let mut stream = LocalQuery::from_partitions(schema(), 6, start, context(), 2, usize::MAX) - .execute() - .unwrap(); - let planned_now = || planned.lock().unwrap().clone(); - timeout(DEADLINE, async { - while planned_now().len() < 2 { - tokio::task::yield_now().await; - } - }) - .await - .expect("the first two partitions were not planned"); - timeout(DEADLINE, stream.next()) +async fn partition_order_runs_at_most_the_given_number_of_partitions() { + let gated = Gated::new(6); + let mut stream = gated.query(2, usize::MAX).execute().unwrap(); + // Partitions are planned only as they start, and two run at once. + gated.wait_started(2).await; + // Finishing the first partition starts the third. + gated.open(0); + let first = timeout(DEADLINE, stream.next()) .await .unwrap() .unwrap() .unwrap(); - // The third partition is planned only once the first has been delivered. - let mut first = planned_now(); - first.sort_unstable(); - assert_eq!(first, vec![0, 1]); - let rest: Vec = timeout(DEADLINE, stream.try_collect()) - .await - .unwrap() - .unwrap(); - assert_eq!(rest.len(), 5); - let mut all = planned_now(); - all.sort_unstable(); - assert_eq!(all, (0..6).collect::>()); + assert_eq!(ids(&[first]), (0..4).collect::>()); + gated.wait_started(3).await; + collect_in_order(&gated, stream, 1).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_partition_that_finishes_before_its_turn_gives_up_its_slot() { + let gated = Gated::new(6); + let reader = tokio::spawn(gated.query(2, usize::MAX).execute().unwrap().try_collect()); + gated.wait_started(2).await; + // While the first partition runs, each later one that finishes drops its operators and + // lets the next one start, and its batch waits for its turn. + for partition in 1..4 { + gated.open(partition); + gated.wait_dropped(partition).await; + gated.wait_started(partition + 2).await; + } + gated.read_in_order(reader).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn finished_partitions_wait_for_their_turn_within_the_read_ahead_limit() { + let gated = Gated::new(6); + let reader = tokio::spawn(gated.query(2, 0).execute().unwrap().try_collect()); + gated.wait_started(2).await; + // The second partition finishes, but it holds a batch beyond the read-ahead limit, so no + // partition starts beyond the two that are not delivered. + gated.open(1); + gated.wait_dropped(1).await; + gated.wait_started(2).await; + gated.read_in_order(reader).await; } /// A first partition that never delivers, and a second one of ten batches. @@ -583,33 +683,28 @@ async fn partition_order_holds_a_few_batches_beyond_the_limit() { #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn partition_order_follows_a_changing_running_limit() { - let (plan, observations) = observe(ordered_input(6, 1)); + let gated = Gated::new(6); let limit = Arc::new(AtomicUsize::new(1)); let current = Arc::clone(&limit); - let mut stream = LocalQuery::new(plan, context(), 3, usize::MAX) + let mut stream = gated + .query(3, usize::MAX) .with_running_limit(Arc::new(move || current.load(Ordering::Acquire))) .execute() .unwrap(); - let executed = || observations.executions.lock().unwrap().len(); - wait_started(&observations, 1).await; + gated.wait_started(1).await; // A limit of zero still runs one partition at a time. limit.store(0, Ordering::Release); - for partition in 0..2 { - timeout(DEADLINE, stream.next()) - .await - .unwrap() - .unwrap() - .unwrap(); - // The next partition starts only once this one ends. - tokio::time::sleep(Duration::from_millis(50)).await; - assert_eq!(executed(), partition + 1, "started beyond the limit"); - } - // A higher limit applies to the partitions started from then on. - limit.store(usize::MAX, Ordering::Release); - let rest: Vec = timeout(DEADLINE, stream.try_collect()) + gated.open(0); + timeout(DEADLINE, stream.next()) .await .unwrap() + .unwrap() .unwrap(); - assert_eq!(rest.len(), 4); - assert_executed_once(&observations, 6); + assert!(futures::poll!(stream.next()).is_pending()); + gated.wait_started(2).await; + // A higher limit applies to the partitions started from then on, up to the maximum. + limit.store(usize::MAX, Ordering::Release); + assert!(futures::poll!(stream.next()).is_pending()); + gated.wait_started(4).await; + collect_in_order(&gated, stream, 1).await; }