From 1d65400a4da37bd5c9aaae758a745dbde465673c Mon Sep 17 00:00:00 2001 From: Liang-Chi Hsieh Date: Tue, 17 Nov 2026 17:42:17 -0800 Subject: [PATCH] feat: run admitted native Parquet scans as one local graph Local execution now admits a native block whose leaf is a native Parquet scan, not only a range, so scan/filter/project queries run as one DataFusion graph whether or not AQE is enabled. - Admission declines a scan that reports partitioning (bucketed) or reads encrypted files, and any expression with a physical subquery, such as a scalar subquery in a scan's filters. - A scan's file partitions are listed when the query runs, as ordinary execution lists them. A collect with no partitions or more than 1024 runs as ordinary execution. - The JNI call takes each partition's serialized NativeScan with its files, and a partition's files are injected into a copy of the plan when the partition is planned. - native/local plans and starts each partition on the runtime's blocking pool, since creating an S3 object store blocks on Comet's runtime, which panics on a worker. - A file read error names the failing partition's files, as ordinary execution names its task's. - The local result carries the block's logical link, which AQE needs when spark.sql.adaptive.forceApply wraps the plan. Co-authored-by: Isaac --- .../contributor-guide/local-execution.md | 59 +++- native/core/src/local.rs | 277 +++++++++++++++++- native/local/src/handle.rs | 19 +- native/local/src/lib.rs | 64 ++-- native/local/tests/execution.rs | 68 +++++ .../apache/comet/local/CometLocalExec.scala | 34 ++- .../comet/local/CometLocalResultExec.scala | 20 +- .../apache/comet/local/CometLocalRule.scala | 81 +++-- .../comet/local/LocalQueryIterator.scala | 17 +- .../org/apache/comet/local/NativeLocal.scala | 10 +- .../local/CometLocalExecutionSuite.scala | 255 +++++++++++++++- 11 files changed, 818 insertions(+), 86 deletions(-) diff --git a/docs/source/contributor-guide/local-execution.md b/docs/source/contributor-guide/local-execution.md index 4fc9470ff8d..69e0d6f5aaa 100644 --- a/docs/source/contributor-guide/local-execution.md +++ b/docs/source/contributor-guide/local-execution.md @@ -39,15 +39,26 @@ admitted when: - the converted plan is one native block, a tree of Comet native operators whose root carries the block's serialized plan, with no JVM input, broadcast or shuffle inside it; - the block's only leaf is a `CometRangeExec`, so the range must be converted to Comet - (`spark.comet.exec.range.enabled`); -- its expressions are deterministic, contain no subquery, and do not call back into the JVM, - which the serialized plan is also checked for; + (`spark.comet.exec.range.enabled`), or a `CometNativeScanExec`, Comet's native Parquet scan; +- a scan reports no partitioning, so a bucketed scan, which reports its buckets' hash + partitioning, is not admitted, and its files are not encrypted, since decrypting them needs + the JVM; +- its expressions are deterministic, contain no subquery, including a scalar subquery or dynamic + pruning in a scan's filters, and do not call back into the JVM, which the serialized plan is + also checked for; - it has between 1 and 1024 partitions. +A range's partitions are known when the plan is admitted. A scan's partitions are its file +partitions, which ordinary execution lists only when the query runs, so a `collect` lists them +then, as ordinary execution would, and runs as ordinary execution when there are none, as for a +scan whose partitions are all pruned, or more than 1024. It has the same file partitions as +ordinary execution, so the result has Spark's row order. + A root `CollectLimitExec`, which Spark plans for `take`, `head`, `show` and `limit`, is not admitted: it collects through `executeTake`, which does not run locally, so such a query keeps -ordinary Comet execution. Admitted plans have no exchange, so AQE does not wrap them and they are -admitted whether or not AQE is enabled. +ordinary Comet execution. Admitted plans have no exchange or subquery, so AQE does not wrap them +and they are admitted whether or not AQE is enabled. When `spark.sql.adaptive.forceApply` wraps +one anyway, it is admitted when AQE plans its final stage. ## Execution @@ -66,10 +77,16 @@ single partition. That partition's task creates the native graph: what they copy from the serialized plan, such as a string literal. Partition 0 is planned up front, for the output schema and so that a plan the planner rejects fails before the query starts, and runs that plan when it starts. +- A scan's partition reads that partition's files. The JNI call takes each partition's serialized + `NativeScan` with its files, as a Spark task receives them, and a partition's files are + injected into a copy of the plan when the partition is planned, so only the files of all + partitions are held for the whole query. - `native/local` runs the partitions on Comet's Tokio runtime and delivers them in partition - order, so the result has Spark's row order. Each partition starts its operators on its own - task, so a start that waits, for instance for a memory grant, does not hold up the pull, and a - partition's error, including a failure to start, is reported when its turn comes. + order, so the result has Spark's row order. Each partition plans and starts its operators on + the runtime's blocking pool, since planning can block: creating an S3 object store waits on + Comet's runtime, which panics on a worker. So a start that waits, for instance for a memory + grant, holds up neither the pull nor a worker, and a partition's error, including a failure to + start, is reported when its turn comes. - At most as many partitions run at once as Spark runs tasks, `spark.executor.cores` divided by `spark.task.cpus`, and the next starts when the first is delivered. A partition's operators register their memory consumers when it starts, so this also bounds how many share the task's @@ -98,8 +115,14 @@ partition, or when the partitions are ranges of a prefix of that ordering, as a are. Sorting within partitions, for example, reports no ordering. `CometLocalResultExec` reports the block's own partitioning and ordering, which hold for the partitions it returns. -Explain shows the native block under `CometLocalExec`. Its native operators do not report SQL -metrics; `CometLocalExec` counts the rows and batches it delivers. +Explain shows the native block under `CometLocalExec`, with the number of partitions for a range; +a scan's are known only once the query runs. Its native operators do not report SQL metrics, and +a scan reports no input metrics to the task; `CometLocalExec` counts the rows and batches it +delivers. + +A file that fails to read is named as ordinary execution names it. When the native error does +not carry the file's path, ordinary execution names the files of the failing task. A local query +records the partition whose turn it was when it failed, and names that partition's files. ## Result delivery @@ -155,14 +178,20 @@ Build native code, including the JNI library, before the JVM suite, and never us ```shell cd native cargo test -p datafusion-comet-local +cargo test -p datafusion-comet --lib local:: cd .. make core ./mvnw test -Dsuites=org.apache.comet.local.CometLocalExecutionSuite ``` The native tests cover partition-order delivery, the bound on running partitions and its -changes, partition starts that wait, read-ahead, errors reported in a partition's turn, -cancellation with busy workers, the pull that waits on its own thread, and runtime shutdown. `CometLocalExecutionSuite` compares admitted queries -with Spark's results and order, and covers ranges Spark would overflow, declined expressions, -AQE, reported orderings, limits left to ordinary execution, iterators, tails, many partitions with sorts in a small memory pool, caching and other RDD consumers, the result size limit, job and task cancellation, and native errors. After -each test it checks that no native query, imported Arrow memory or result slot is left. +changes, partition starts that wait, block or panic, read-ahead, errors reported in a +partition's turn and the partition they came from, cancellation with busy workers, the pull that +waits on its own thread, runtime shutdown, and scan partitions that each read their own files. +`CometLocalExecutionSuite` compares admitted queries with Spark's results and order, and covers +ranges Spark would overflow, Parquet scans split into many partitions, partition pruning, scans +of no files or of more partitions than run locally, declined scans and expressions, AQE, +reported orderings, limits left to ordinary execution, iterators, tails, many partitions with +sorts in a small memory pool, caching and other RDD consumers, the result size limit, job and +task cancellation, unreadable files, and native errors. After each test it checks that no native +query, imported Arrow memory or result slot is left. diff --git a/native/core/src/local.rs b/native/core/src/local.rs index 6b20679bf35..34bfa1d1a1e 100644 --- a/native/core/src/local.rs +++ b/native/core/src/local.rs @@ -31,12 +31,14 @@ use datafusion::physical_plan::{ExecutionPlan, ExecutionPlanProperties}; use datafusion_comet_jni_bridge::{jni_new_global_ref, JVMClasses}; use datafusion_comet_local::handle::{QueryHandle, QueryPoll}; use datafusion_comet_local::{LocalQuery, PartitionStart}; -use datafusion_comet_proto::spark_operator::Operator; +use datafusion_comet_proto::spark_operator::operator::OpStruct; +use datafusion_comet_proto::spark_operator::{NativeScan, Operator, SparkFilePartition}; use jni::objects::{JByteArray, JLongArray, JObject, JObjectArray, JString, ReleaseMode}; use jni::sys::{jboolean, jint, jlong, JNI_FALSE}; use jni::{Env, EnvUnowned}; use log::warn; use parking_lot::Mutex; +use prost::Message; use crate::errors::{try_unwrap_or_throw, CometError, CometResult}; use crate::execution::jni_api::{ @@ -78,6 +80,10 @@ static QUERIES: LazyLock>>> = LazyLock::new(Mutex: static NEXT_ID: AtomicI64 = AtomicI64::new(1); const MAX_LIVE_QUERIES: usize = 1024; +// The partition whose turn it was when a query failed, kept after the failure closes the query +// until the JVM reads it or closes the query, so that a file read error can name its files. +static FAILED_PARTITIONS: LazyLock>> = LazyLock::new(Mutex::default); + fn close(id: i64) { let entry = QUERIES.lock().remove(&id); if let Some(entry) = entry { @@ -88,11 +94,16 @@ fn close(id: i64) { /// Creates and starts a local query over `partitions` partitions of a serialized Comet plan /// without JVM inputs. Its memory pool and DataFusion session are those `Native.createPlan` /// creates for a plan of the calling Spark task. +/// +/// When the plan's leaf is a native file scan, `scan_partitions` holds each partition's +/// serialized `NativeScan` carrying that partition's files, as a Spark task injects them, and is +/// otherwise empty. #[no_mangle] pub extern "system" fn Java_org_apache_comet_local_NativeLocal_createQuery( e: EnvUnowned, _: JObject, serialized_query: JByteArray, + scan_partitions: JObjectArray, serialized_spark_configs: JByteArray, partitions: jint, columns: jint, @@ -122,6 +133,12 @@ pub extern "system" fn Java_org_apache_comet_local_NativeLocal_createQuery( ); JVMClasses::init(env); let serialized_plan = env.convert_byte_array(serialized_query)?; + let mut scan_partitions_vec = Vec::with_capacity(scan_partitions.len(env)?); + for i in 0..scan_partitions.len(env)? { + let bytes = scan_partitions.get_element(env, i)?; + let bytes = unsafe { JByteArray::from_raw(&*env, bytes.into_raw()) }; + scan_partitions_vec.push(env.convert_byte_array(bytes)?); + } let task_memory_manager = Arc::new(jni_new_global_ref!(env, comet_task_memory_manager_obj)?); let memory_pool_type = memory_pool_type.try_to_string(env)?; @@ -152,7 +169,7 @@ pub extern "system" fn Java_org_apache_comet_local_NativeLocal_createQuery( let context = create_execution_context(inputs, |config| { create_memory_pool(config, task_memory_manager, task_attempt_id) })?; - let (schema, start) = local_plan(&context, partitions as usize)?; + let (schema, start) = local_plan(&context, partitions as usize, scan_partitions_vec)?; if schema.fields().len() != columns as usize { return Err(CometError::Internal( "Local output schema width mismatch".into(), @@ -197,15 +214,21 @@ fn task_slots(cores: usize, task_cpus: usize) -> usize { /// planned only when it starts, as a Spark task plans only when it runs, so the plans held at once, /// and whatever each copies from the serialized plan, are those of the running partitions. /// +/// A file scan's partition reads the files in `scan_partitions` for that partition, which are +/// injected into a copy of the plan when it is planned, as a Spark task injects them before it +/// creates its plan. Only the files of all partitions are held for the whole query. +/// /// Partition 0 is planned here, for the output schema and so that a plan the planner rejects /// fails the query before it starts, and that plan is the one partition 0 runs. fn local_plan( context: &ExecutionContext, partitions: usize, + scan_partitions: Vec>, ) -> CometResult<(SchemaRef, PartitionStart)> { let planner = context.planner(0); let spark_plan = Arc::new(context.spark_plan.clone()); - let first = plan_partition(&planner, &spark_plan, 0, partitions)?; + let files = Arc::new(ScanFiles::new(&spark_plan, scan_partitions, partitions)?); + let first = plan_partition(&planner, &spark_plan, &files, 0, partitions)?; let schema = first.schema(); let expected = Arc::clone(&schema); let first = Mutex::new(Some(first)); @@ -217,7 +240,7 @@ fn local_plan( }; let plan = match planned { Some(plan) => plan, - None => plan_partition(&planner, &spark_plan, partition, partitions)?, + None => plan_partition(&planner, &spark_plan, &files, partition, partitions)?, }; if plan.schema() != expected { return Err(DataFusionError::Internal(format!( @@ -229,17 +252,92 @@ fn local_plan( Ok((schema, start)) } +/// The files each partition of a plan's native file scan reads, if it has one. +struct ScanFiles { + /// Each partition's serialized `NativeScan`, which carries only its files. + partitions: Vec>, +} + +impl ScanFiles { + fn new( + spark_plan: &Operator, + partitions: Vec>, + partition_count: usize, + ) -> CometResult { + let scans = count_scans_without_files(spark_plan); + let valid = if partitions.is_empty() { + scans == 0 + } else { + scans == 1 && partitions.len() == partition_count + }; + if !valid { + return Err(CometError::Internal(format!( + "Local query has {scans} file scans without files and file lists for {} of its \ + {partition_count} partitions", + partitions.len() + ))); + } + Ok(Self { partitions }) + } + + /// `spark_plan` with the files of `partition` in its file scan, or `spark_plan` itself + /// when it has no file scan. + fn plan_for(&self, spark_plan: &Arc, partition: usize) -> CometResult> { + let Some(bytes) = self.partitions.get(partition) else { + return Ok(Arc::clone(spark_plan)); + }; + let files = NativeScan::decode(bytes.as_slice()) + .map_err(|e| CometError::Internal(format!("Invalid local scan partition: {e}")))? + .file_partition + .ok_or_else(|| CometError::Internal("Local scan partition has no files".into()))?; + let mut plan = Operator::clone(spark_plan); + inject_files(&mut plan, files); + Ok(Arc::new(plan)) + } +} + +fn count_scans_without_files(op: &Operator) -> usize { + let own = match &op.op_struct { + Some(OpStruct::NativeScan(scan)) if scan.file_partition.is_none() => 1, + _ => 0, + }; + own + op + .children + .iter() + .map(count_scans_without_files) + .sum::() +} + +/// Gives the plan's one file scan without files `files`. [`ScanFiles::new`] checked that it +/// has exactly one. +fn inject_files(op: &mut Operator, files: SparkFilePartition) -> Option { + if let Some(OpStruct::NativeScan(scan)) = &mut op.op_struct { + if scan.file_partition.is_none() { + scan.file_partition = Some(files); + return None; + } + } + let mut files = Some(files); + for child in &mut op.children { + files = inject_files(child, files?); + } + files +} + /// The operators of `partition`, whose single output partition is that partition of the graph. fn plan_partition( planner: &PhysicalPlanner, - spark_plan: &Operator, + spark_plan: &Arc, + files: &ScanFiles, partition: usize, partitions: usize, ) -> CometResult> { - let (scans, shuffle_scans, root) = - planner - .for_partition(partition as i32) - .create_plan(spark_plan, &mut vec![], partitions)?; + let spark_plan = files.plan_for(spark_plan, partition)?; + let (scans, shuffle_scans, root) = planner.for_partition(partition as i32).create_plan( + &spark_plan, + &mut vec![], + partitions, + )?; if !scans.is_empty() || !shuffle_scans.is_empty() || root.native_plan.output_partitioning().partition_count() != 1 @@ -305,6 +403,12 @@ pub extern "system" fn Java_org_apache_comet_local_NativeLocal_nextBatch( ) -> jlong { try_unwrap_or_throw(&e, |env| { let result = next(env, id, arrays, schemas); + if result.is_err() { + let partition = QUERIES.lock().get(&id).map(|entry| entry.query.partition()); + if let Some(partition) = partition { + FAILED_PARTITIONS.lock().insert(id, partition); + } + } if result.is_err() || matches!(result, Ok(-1)) { close(id); } @@ -320,21 +424,172 @@ pub extern "system" fn Java_org_apache_comet_local_NativeLocal_close( ) { try_unwrap_or_throw(&e, |_| { close(id); + FAILED_PARTITIONS.lock().remove(&id); Ok(()) }) } +/// The partition whose turn it was when the query failed, or -1. Reading it forgets it. +#[no_mangle] +pub extern "system" fn Java_org_apache_comet_local_NativeLocal_failedPartition( + e: EnvUnowned, + _: JObject, + id: jlong, +) -> jint { + try_unwrap_or_throw(&e, |_| { + Ok(FAILED_PARTITIONS + .lock() + .remove(&id) + .map_or(-1, |partition| partition as jint)) + }) +} + #[no_mangle] pub extern "system" fn Java_org_apache_comet_local_NativeLocal_activeQueries( e: EnvUnowned, _: JObject, ) -> jlong { - try_unwrap_or_throw(&e, |_| Ok(QUERIES.lock().len() as jlong)) + // A failure kept for the JVM counts until it is read, so that tests catch one left behind. + try_unwrap_or_throw(&e, |_| { + Ok((QUERIES.lock().len() + FAILED_PARTITIONS.lock().len()) as jlong) + }) } #[cfg(test)] mod tests { - use super::task_slots; + use std::collections::HashMap; + use std::fs::File; + use std::sync::Arc; + + use arrow::array::{Array, Int64Array, RecordBatch}; + use arrow::datatypes::{DataType, Field, Schema}; + use datafusion::physical_plan::common::collect; + use datafusion::prelude::SessionContext; + use datafusion_comet_proto::spark_expression::{ + data_type::DataTypeId, DataType as SparkDataType, + }; + use datafusion_comet_proto::spark_operator::operator::OpStruct; + use datafusion_comet_proto::spark_operator::{ + NativeScan, NativeScanCommon, Operator, SparkFilePartition, SparkPartitionedFile, + SparkStructField, + }; + use parquet::arrow::ArrowWriter; + use prost::Message; + + use super::{plan_partition, task_slots, ScanFiles}; + use crate::execution::planner::PhysicalPlanner; + + /// Writes `ids` to a Parquet file and returns the file as a Spark partitioned file. + fn parquet_file(dir: &std::path::Path, name: &str, ids: &[i64]) -> SparkPartitionedFile { + let path = dir.join(name); + let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); + let batch = RecordBatch::try_new( + Arc::clone(&schema), + vec![Arc::new(Int64Array::from(ids.to_vec()))], + ) + .unwrap(); + let mut writer = ArrowWriter::try_new(File::create(&path).unwrap(), schema, None).unwrap(); + writer.write(&batch).unwrap(); + writer.close().unwrap(); + let size = std::fs::metadata(&path).unwrap().len() as i64; + SparkPartitionedFile { + file_path: format!("file://{}", path.display()), + start: 0, + length: size, + file_size: size, + partition_values: vec![], + } + } + + fn scan() -> Operator { + let id = SparkStructField { + name: "id".to_string(), + data_type: Some(SparkDataType { + type_id: DataTypeId::Int64 as i32, + type_info: None, + }), + nullable: false, + metadata: HashMap::new(), + }; + Operator { + op_struct: Some(OpStruct::NativeScan(NativeScan { + common: Some(NativeScanCommon { + required_schema: vec![id.clone()], + data_schema: vec![id], + projection_vector: vec![0], + session_timezone: "UTC".to_string(), + ..Default::default() + }), + ..Default::default() + })), + ..Default::default() + } + } + + /// A partition's serialized `NativeScan` carrying only its files, as the JVM sends it. + fn files(partitioned_file: Vec) -> Vec { + NativeScan { + file_partition: Some(SparkFilePartition { partitioned_file }), + ..Default::default() + } + .encode_to_vec() + } + + #[tokio::test] + async fn each_partition_reads_its_own_files() { + let dir = tempfile::tempdir().unwrap(); + let groups = vec![ + vec![ + parquet_file(dir.path(), "a.parquet", &[1, 2]), + parquet_file(dir.path(), "b.parquet", &[3]), + ], + vec![], + vec![parquet_file(dir.path(), "c.parquet", &[4, 5, 6])], + ]; + let plan = Arc::new(scan()); + let scan_files = ScanFiles::new(&plan, groups.into_iter().map(files).collect(), 3).unwrap(); + let session = Arc::new(SessionContext::new()); + let planner = PhysicalPlanner::new(Arc::clone(&session), 0); + let mut ids = vec![]; + for partition in 0..3 { + let native = plan_partition(&planner, &plan, &scan_files, partition, 3).unwrap(); + let batches = collect(native.execute(0, session.task_ctx()).unwrap()) + .await + .unwrap(); + let read: Vec = batches + .iter() + .flat_map(|batch| { + let column = batch.column(0); + let values = column.as_any().downcast_ref::().unwrap(); + values.values().to_vec() + }) + .collect(); + ids.push(read); + } + assert_eq!(ids, vec![vec![1, 2, 3], vec![], vec![4, 5, 6]]); + // The injected files live only in each partition's copy. + match &plan.op_struct { + Some(OpStruct::NativeScan(scan)) => assert!(scan.file_partition.is_none()), + _ => unreachable!(), + } + } + + #[test] + fn file_lists_must_match_the_plan_and_its_partitions() { + let plan = scan(); + let error = |result: super::CometResult| result.err().unwrap().to_string(); + // A file scan needs a file list for every partition. + assert!(error(ScanFiles::new(&plan, vec![], 2)).contains("1 file scans")); + assert!(error(ScanFiles::new(&plan, vec![files(vec![])], 2)).contains("for 1 of its 2")); + // A plan without a file scan takes none. + let range = Operator::default(); + assert!(ScanFiles::new(&range, vec![], 2).is_ok()); + assert!(error(ScanFiles::new(&range, vec![files(vec![])], 1)).contains("0 file scans")); + // Nor does a plan whose scan already carries its files. + let mut injected = scan(); + super::inject_files(&mut injected, SparkFilePartition::default()); + assert!(ScanFiles::new(&injected, vec![], 1).is_ok()); + } #[test] fn task_slots_divide_cores_by_task_cpus() { diff --git a/native/local/src/handle.rs b/native/local/src/handle.rs index 9ca0dd20c23..f476936c046 100644 --- a/native/local/src/handle.rs +++ b/native/local/src/handle.rs @@ -18,7 +18,7 @@ //! Delivery to blocking callers, with cancellation that does not wait for a reader's wait. use std::panic::AssertUnwindSafe; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::{Arc, Mutex, TryLockError}; use std::task::{Context, Poll, Wake, Waker}; use std::thread::Thread; @@ -50,6 +50,8 @@ pub struct QueryHandle { reader: Mutex>, cancelled: AtomicBool, runtime: Handle, + /// The partition whose turn it is; see [`LocalQueryStream::partition`]. + turn: Arc, /// Set when a reader is about to park, so that a test can cancel a parked reader. #[cfg(test)] parking: AtomicBool, @@ -70,7 +72,9 @@ impl QueryHandle { let _guard = runtime.enter(); query.execute()? }; + let turn = Arc::clone(&stream.turn); Ok(Self { + turn, stream: Mutex::new(Some(stream)), reader: Mutex::new(None), cancelled: AtomicBool::new(false), @@ -80,6 +84,12 @@ impl QueryHandle { }) } + /// The partition whose turn it is: the one the last batch or error came from, or that is + /// being waited for. Readable while another thread polls. + pub fn partition(&self) -> usize { + self.turn.load(Ordering::Acquire) + } + /// A bounded wait lets a JVM caller check Spark task interruption between polls. /// Only one reader may call this at a time. pub fn poll(&self, wait: Duration) -> Result { @@ -195,10 +205,11 @@ mod tests { let thread = std::thread::spawn(move || { sender.send(reader.poll(Duration::from_secs(60))).unwrap(); }); - // Cancel only once the reader has passed its last check before parking, so that only - // the wakeup can end its wait. + // Cancel only once both partitions have started, on the runtime's blocking pool, and the + // reader has passed its last check before parking, so that only the wakeup can end its + // wait. The plan and each started stream hold a reference. let waiting = Instant::now(); - while !query.parking.load(Ordering::Acquire) { + while refs.strong_count() < 3 || !query.parking.load(Ordering::Acquire) { assert!( waiting.elapsed() < Duration::from_secs(5), "the reader never parked" diff --git a/native/local/src/lib.rs b/native/local/src/lib.rs index dee31918320..937e9f1e1d8 100644 --- a/native/local/src/lib.rs +++ b/native/local/src/lib.rs @@ -74,9 +74,9 @@ impl LocalQuery { /// A partition's operators register their memory consumers when it starts, so this also /// bounds how many share the memory pool. /// - /// A partition starts its operators on its own task, so a start that waits, for instance to - /// register a memory consumer, does not hold up delivery. A failure to start is reported in - /// the partition's turn. + /// A partition starts its operators on the runtime's blocking pool, so a start that waits, + /// for instance to register a memory consumer, holds up neither delivery nor a worker. A + /// failure to start is reported in the partition's turn. /// /// The running partitions do not wait for their turn: each runs ahead of delivery while the /// batches they all hold total less than `read_ahead` bytes, and up to a few batches beyond @@ -99,7 +99,8 @@ impl LocalQuery { /// As [`Self::new`], for a graph whose root partitions `start` plans as each one starts, so /// that only the running partitions hold their operators. Each stream must have `schema`. - /// `start` runs on a Tokio worker of the caller's runtime, so it must not block on it. + /// `start` runs on the blocking pool of the caller's runtime, so it may block, including on + /// a runtime, as creating a file scan's object store can. pub fn from_partitions( schema: SchemaRef, partitions: usize, @@ -134,17 +135,20 @@ impl LocalQuery { DataFusionError::Execution("LocalQuery requires an active Tokio runtime".into()) })?; let schema = Arc::clone(&self.schema); - let stream = in_partition_order(&self); + let turn = Arc::new(AtomicUsize::new(0)); + let stream = in_partition_order(&self, Arc::clone(&turn)); Ok(LocalQueryStream { stream: Some(stream), query: Some(self), schema, + turn, }) } } -fn in_partition_order(query: &LocalQuery) -> SendableRecordBatchStream { +fn in_partition_order(query: &LocalQuery, turn: Arc) -> SendableRecordBatchStream { let mut order = PartitionOrder { + turn, partitions: query.partitions, start: Arc::clone(&query.start), context: Arc::clone(&query.context), @@ -213,18 +217,30 @@ impl Partition { let (start, context) = (Arc::clone(&order.start), Arc::clone(&order.context)); let task = order.runtime.spawn(async move { let result = AssertUnwindSafe(async { - let started = start(partition, context); - drop(start); - match started { - Ok(started) => { - let mut slot = lock(&polled); - if matches!(*slot, Slot::Closed) { - drop(slot); - drop(started); - return; - } - *slot = Slot::Running(started); + // Off the workers: planning a partition can block, and a worker that blocks on + // a runtime panics. The started stream goes into the slot on the same thread, so + // there is no wait in which dropping the partition could not take it. + let starting = Arc::clone(&polled); + let started = tokio::task::spawn_blocking(move || { + let started = start(partition, context)?; + let mut slot = lock(&starting); + if matches!(*slot, Slot::Closed) { + drop(slot); + drop(started); + return Ok(false); } + *slot = Slot::Running(started); + Ok(true) + }) + .await + .unwrap_or_else(|_| { + Err(DataFusionError::Execution( + "Local query partition panicked while starting".into(), + )) + }); + match started { + Ok(true) => {} + Ok(false) => return, Err(error) => { let _ = sender.send(Err(error)); return; @@ -312,6 +328,8 @@ struct PartitionOrder { read_ahead: Arc, /// Started partitions, the first of which is being delivered. running: VecDeque, + /// The partition whose turn it is, shared with the query's stream. + turn: Arc, } impl PartitionOrder { @@ -344,6 +362,11 @@ impl Stream for PartitionOrder { fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { loop { self.start_partitions(); + if self.running.is_empty() { + return Poll::Ready(None); + } + let turn = self.next - self.running.len(); + self.turn.store(turn, Ordering::Release); let Some(partition) = self.running.front_mut() else { return Poll::Ready(None); }; @@ -383,9 +406,16 @@ pub struct LocalQueryStream { stream: Option, query: Option, schema: SchemaRef, + turn: Arc, } impl LocalQueryStream { + /// The partition whose turn it is: the one the last batch or error came from, or that is + /// being waited for. + pub fn partition(&self) -> usize { + self.turn.load(Ordering::Acquire) + } + /// Stops result delivery and requests cancellation of the whole native execution. /// Calling this again, or polling afterwards, is harmless. pub fn cancel(&mut self) { diff --git a/native/local/tests/execution.rs b/native/local/tests/execution.rs index 4cf4660b457..994944ec443 100644 --- a/native/local/tests/execution.rs +++ b/native/local/tests/execution.rs @@ -544,6 +544,74 @@ async fn partitions_from_a_start_function_are_planned_as_they_start() { assert_eq!(all, (0..6).collect::>()); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_partition_start_may_block_on_the_runtime() { + // Creating a file scan's object store can block on a runtime, which panics on a worker. + let source = ordered_input(3, 1); + let start: PartitionStart = Arc::new(move |partition, context| { + tokio::runtime::Handle::current().block_on(tokio::task::yield_now()); + source.execute(partition, context) + }); + let batches: Vec = timeout( + DEADLINE, + LocalQuery::from_partitions(schema(), 3, start, context(), 2, usize::MAX) + .execute() + .unwrap() + .try_collect(), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(batches.len(), 3); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_partition_start_that_panics_fails_in_its_turn() { + let start: PartitionStart = Arc::new(|_, _| panic!("start panicked")); + let error = timeout( + DEADLINE, + LocalQuery::from_partitions(schema(), 2, start, context(), 2, usize::MAX) + .execute() + .unwrap() + .next(), + ) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(error.to_string().contains("panicked"), "{error}"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn the_stream_reports_the_partition_whose_turn_it_is() { + let source = ordered_input(3, 1); + let start: PartitionStart = Arc::new(move |partition, context| { + if partition == 2 { + Err(DataFusionError::Execution("partition 2 failed".into())) + } else { + source.execute(partition, context) + } + }); + let mut stream = LocalQuery::from_partitions(schema(), 3, start, context(), 3, usize::MAX) + .execute() + .unwrap(); + for partition in 0..2 { + timeout(DEADLINE, stream.next()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(stream.partition(), partition); + } + let error = timeout(DEADLINE, stream.next()) + .await + .unwrap() + .unwrap() + .unwrap_err(); + assert!(error.to_string().contains("partition 2 failed")); + assert_eq!(stream.partition(), 2); +} + /// A first partition that never delivers, and a second one of ten batches. fn behind_a_blocked_partition() -> (Arc, Arc) { let blocking = Arc::new(BlockingExec::new(schema(), 1)); diff --git a/spark/src/main/scala/org/apache/comet/local/CometLocalExec.scala b/spark/src/main/scala/org/apache/comet/local/CometLocalExec.scala index 40297dee2be..29602ebd6ea 100644 --- a/spark/src/main/scala/org/apache/comet/local/CometLocalExec.scala +++ b/spark/src/main/scala/org/apache/comet/local/CometLocalExec.scala @@ -25,29 +25,43 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.{Attribute, SortOrder} import org.apache.spark.sql.catalyst.plans.QueryPlan import org.apache.spark.sql.catalyst.plans.physical.{Partitioning, UnknownPartitioning} -import org.apache.spark.sql.execution.{ColumnarToRowExec, LeafExecNode, SparkPlan} +import org.apache.spark.sql.comet.CometNativeExec +import org.apache.spark.sql.execution.{ColumnarToRowExec, LeafExecNode} import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} import org.apache.spark.sql.types.DataType import org.apache.spark.sql.vectorized.ColumnarBatch -/** What the result task needs to create the native graph: no plan objects. */ +/** + * What the result task needs to create the native graph: no plan objects. For a native file scan, + * `scanPartitions` holds each partition's serialized `NativeScan` with its files, and `filePaths` + * each partition's file paths, which name the files of a partition that fails to read one. + */ private[local] case class LocalQuerySpec( plan: Array[Byte], partitions: Int, - outputTypes: Seq[DataType]) { + outputTypes: Seq[DataType], + scanPartitions: Array[Array[Byte]] = Array.empty, + filePaths: Array[Seq[String]] = Array.empty) { def columns: Int = outputTypes.size } /** * A whole query run as one native graph by the single Spark task of its result. `block` is the - * Comet native block the graph is created from, kept for explain only. + * Comet native block the graph is created from, which explain shows. */ case class CometLocalExec private[local] ( override val output: Seq[Attribute], - spec: LocalQuerySpec, ordering: Seq[SortOrder], - block: SparkPlan) + block: CometNativeExec) extends LeafExecNode { + + /** + * The graph's spec, made when the query runs, as ordinary execution lists a file scan's files + * only then. `None` when the block has no partitions or more than local execution runs, and the + * result then comes from ordinary execution. + */ + @transient private[local] lazy val spec: Option[LocalQuerySpec] = CometLocalRule.spec(block) + override def supportsColumnar: Boolean = true // The result is one partition, but its rows are not clustered by anything. @@ -58,7 +72,11 @@ case class CometLocalExec private[local] ( override def innerChildren: Seq[QueryPlan[_]] = Seq(block) - override def stringArgs: Iterator[Any] = Iterator(output, s"partitions=${spec.partitions}") + // A file scan's partitions are known only once its files are listed, when the query runs. + override def stringArgs: Iterator[Any] = { + val partitions = block.collectLeaves().head.outputPartitioning.numPartitions + if (partitions > 0) Iterator(output, s"partitions=$partitions") else Iterator(output) + } override lazy val metrics: Map[String, SQLMetric] = Map( "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output rows"), @@ -69,6 +87,8 @@ case class CometLocalExec private[local] ( override protected def doExecuteColumnar(): RDD[ColumnarBatch] = { val rows = longMetric("numOutputRows") val batches = longMetric("numOutputBatches") + val spec = this.spec.getOrElse( + throw new IllegalStateException("A local query without a local spec does not run locally")) new LocalQueryRDD(sparkContext, spec).mapPartitions { input => input.map { batch => rows += batch.numRows().toLong diff --git a/spark/src/main/scala/org/apache/comet/local/CometLocalResultExec.scala b/spark/src/main/scala/org/apache/comet/local/CometLocalResultExec.scala index 6c0fa10bf52..d408e5778e1 100644 --- a/spark/src/main/scala/org/apache/comet/local/CometLocalResultExec.scala +++ b/spark/src/main/scala/org/apache/comet/local/CometLocalResultExec.scala @@ -54,17 +54,25 @@ case class CometLocalResultExec(child: SparkPlan) extends UnaryExecNode { override protected def doExecute(): RDD[InternalRow] = ordinary.execute() + // A file scan's partitions are known only now. With none, or more than run locally, the + // collect runs as ordinary execution, before any native output. override def executeCollect(): Array[InternalRow] = - if (sparkContext.isLocal) LocalResultHandoff.collect(child.execute(), maxResultSize) - else super.executeCollect() + if (sparkContext.isLocal && localExec.forall(_.spec.isDefined)) { + LocalResultHandoff.collect(child.execute(), maxResultSize) + } else if (sparkContext.isLocal) { + ordinary.executeCollect() + } else { + super.executeCollect() + } + + @transient private lazy val localExec: Option[CometLocalExec] = + child.collectFirst { case exec: CometLocalExec => exec } // The block as ordinary planning prepares it, which runs after this node is created: the // columnar-to-row transition Comet picks, or Spark's RangeExec for a bare range. @transient private[local] lazy val ordinary: SparkPlan = - child - .collectFirst { case local: CometLocalExec => - EliminateRedundantTransitions(session)(ColumnarToRowExec(local.block)) - } + localExec + .map(exec => EliminateRedundantTransitions(session)(ColumnarToRowExec(exec.block))) .getOrElse(child) // Spark enforces this on serialized task results; this boundary enforces it on copied rows. diff --git a/spark/src/main/scala/org/apache/comet/local/CometLocalRule.scala b/spark/src/main/scala/org/apache/comet/local/CometLocalRule.scala index 59180453f79..62aab32941c 100644 --- a/spark/src/main/scala/org/apache/comet/local/CometLocalRule.scala +++ b/spark/src/main/scala/org/apache/comet/local/CometLocalRule.scala @@ -22,9 +22,9 @@ package org.apache.comet.local import scala.jdk.CollectionConverters._ import org.apache.spark.sql.SparkSession -import org.apache.spark.sql.catalyst.expressions.{SortOrder, SubqueryExpression} -import org.apache.spark.sql.catalyst.plans.physical.RangePartitioning -import org.apache.spark.sql.comet.{CometNativeExec, CometRangeExec} +import org.apache.spark.sql.catalyst.expressions.{PlanExpression, SortOrder} +import org.apache.spark.sql.catalyst.plans.physical.{RangePartitioning, UnknownPartitioning} +import org.apache.spark.sql.comet.{CometNativeExec, CometNativeScanExec, CometRangeExec} import org.apache.spark.sql.execution.{QueryExecution, SparkPlan} import org.apache.spark.sql.internal.SQLConf @@ -39,6 +39,10 @@ import org.apache.comet.shims.ShimCometStreaming * Admission of whole queries to local execution. A query is admitted only as a whole, from the * plan Comet's ordinary conversion produces: one native block with no JVM input, whose operators * then run for all partitions in one native graph. Everything else keeps ordinary execution. + * + * The block's leaf is a range or a native Parquet scan. A scan's partitions are its file + * partitions, which ordinary execution lists only when the query runs, so their number is checked + * then; see [[spec]]. */ private[comet] object CometLocalRule { @@ -69,25 +73,46 @@ private[comet] object CometLocalRule { } convert(plan) match { case block: CometNativeExec if admits(block) => - val partitions = block.collectLeaves().head.outputPartitioning.numPartitions - val spec = LocalQuerySpec( - block.serializedPlanOpt.plan.get, - partitions, - block.output.map(_.dataType)) - CometLocalResultExec( - CometLocalExec(block.output, spec, resultOrdering(block, partitions), block)) + val result = + CometLocalResultExec(CometLocalExec(block.output, resultOrdering(block), block)) + // AQE, when `spark.sql.adaptive.forceApply` wraps the plan, makes the result its last + // stage, which needs the logical plan the block came from. + block.logicalLink.foreach(result.setLogicalLink) + result case converted => converted } } + /** + * The spec of an admitted block's graph, made when the query runs. A scan's files are listed + * then, as ordinary execution lists them, after its subqueries are resolved. `None` when the + * block has no partitions, such as a scan of no files, or more than [[MaxPartitions]]. + */ + private[local] def spec(block: CometNativeExec): Option[LocalQuerySpec] = { + val plan = block.serializedPlanOpt.plan.get + val types = block.output.map(_.dataType) + val spec = block.collectLeaves().head match { + case scan: CometNativeScanExec => + scan.ensureSubqueriesResolved() + scan.sendDriverMetrics() + val files = scan.perPartitionData + LocalQuerySpec(plan, files.length, types, files, scan.perPartitionFilePaths) + case leaf => + LocalQuerySpec(plan, leaf.outputPartitioning.numPartitions, types) + } + Some(spec).filter(spec => spec.partitions >= 1 && spec.partitions <= MaxPartitions) + } + /** * The ordering of the whole result. Spark's `outputOrdering` holds within each partition, and * the result concatenates the partitions in order, so it holds for the whole result only with * one partition, or when the partitions are ranges of a prefix of that ordering, as a range's - * slices are. + * slices are. A scan's partitions are unknown until the query runs, so its ordering is not + * reported; a scan local execution admits reports none anyway. */ - private def resultOrdering(block: CometNativeExec, partitions: Int): Seq[SortOrder] = { + private def resultOrdering(block: CometNativeExec): Seq[SortOrder] = { val ordering = block.outputOrdering + val partitions = block.collectLeaves().head.outputPartitioning.numPartitions block.outputPartitioning match { case _ if partitions == 1 => ordering case RangePartitioning(ranges, _) if SortOrder.orderingSatisfies(ordering, ranges) => @@ -113,25 +138,43 @@ private[comet] object CometLocalRule { /** * A single native block that a Spark task could run for each partition on its own: its root * carries the serialized plan, every other operator is native and part of the same block, its - * only leaf is a range, and its expressions are deterministic and do not call back into the - * JVM. Running all partitions in one graph then produces what the Spark tasks would. + * only leaf is a range or a native Parquet scan (see [[admitsLeaf]]), and its expressions are + * deterministic, contain no subquery, and do not call back into the JVM. Running all partitions + * in one graph then produces what the Spark tasks would. */ private def admits(block: CometNativeExec): Boolean = { block.serializedPlanOpt.isDefined && block.output.nonEmpty && - block.collectLeaves().map(_.getClass) == Seq(classOf[CometRangeExec]) && + (block.collectLeaves() match { + case Seq(leaf) => admitsLeaf(leaf) + case _ => false + }) && block.collectFirst { case node: CometNativeExec if (node ne block) && node.serializedPlanOpt.isDefined => node case node if !node.isInstanceOf[CometNativeExec] => node }.isEmpty && + // `PlanExpression` covers the subqueries of physical plans, such as a scalar subquery or + // dynamic pruning in a scan's filters, which are not in the serialized plan. !block.exists(_.expressions.exists(e => - !e.deterministic || e.exists(_.isInstanceOf[SubqueryExpression]))) && { - val partitions = block.collectLeaves().head.outputPartitioning.numPartitions - partitions >= 1 && partitions <= MaxPartitions - } && + !e.deterministic || e.exists(_.isInstanceOf[PlanExpression[_]]))) && withoutJvmCallbacks(block.nativeOp) } + /** + * A range with 1 to [[MaxPartitions]] partitions, or a native Parquet scan that reports no + * partitioning and reads unencrypted files. A bucketed scan reports its buckets' hash + * partitioning, which local execution does not rely on. Decrypting a file needs the JVM. + */ + private def admitsLeaf(leaf: SparkPlan): Boolean = leaf match { + case range: CometRangeExec => + val partitions = range.outputPartitioning.numPartitions + partitions >= 1 && partitions <= MaxPartitions + case scan: CometNativeScanExec => + scan.outputPartitioning.isInstanceOf[UnknownPartitioning] && + !scan.nativeOp.getNativeScan.getCommon.getEncryptionEnabled + case _ => false + } + private def withoutJvmCallbacks(message: Message): Boolean = { val accepted = message match { case expression: Expr => !JvmCallbacks.contains(expression.getExprStructCase) diff --git a/spark/src/main/scala/org/apache/comet/local/LocalQueryIterator.scala b/spark/src/main/scala/org/apache/comet/local/LocalQueryIterator.scala index 21f27a1d740..74b9863b26c 100644 --- a/spark/src/main/scala/org/apache/comet/local/LocalQueryIterator.scala +++ b/spark/src/main/scala/org/apache/comet/local/LocalQueryIterator.scala @@ -19,6 +19,8 @@ package org.apache.comet.local +import scala.util.control.NonFatal + import org.apache.spark.{SparkEnv, TaskContext, TaskKilledException} import org.apache.spark.sql.comet.CometExec import org.apache.spark.sql.vectorized.ColumnarBatch @@ -52,6 +54,7 @@ private[local] class LocalQueryIterator(spec: LocalQuerySpec, context: TaskConte val maxResultSize = SparkEnv.get.conf.getSizeAsBytes("spark.driver.maxResultSize", "1g") id = native.createQuery( spec.plan, + spec.scanPartitions, CometExecIterator.serializeCometSQLConfs(), spec.partitions, spec.columns, @@ -80,9 +83,21 @@ private[local] class LocalQueryIterator(spec: LocalQuerySpec, context: TaskConte } private def executionFailure(failure: Throwable): Throwable = { + // A file read error that does not name its file names the failing partition's files, as + // ordinary execution names its task's. Read before close, which forgets the partition. + val filePaths = failure match { + case _: CometQueryExecutionException if id != 0L => + val partition = + try native.failedPartition(id) + catch { case NonFatal(_) => -1 } + if (partition >= 0 && partition < spec.filePaths.length) spec.filePaths(partition) + else Seq.empty[String] + case _ => Seq.empty[String] + } closeAfterFailure(failure) failure match { - case e: CometQueryExecutionException => SparkErrorConverter.convertToSparkException(e) + case e: CometQueryExecutionException => + SparkErrorConverter.convertToSparkException(e, filePaths) case other => other } } diff --git a/spark/src/main/scala/org/apache/comet/local/NativeLocal.scala b/spark/src/main/scala/org/apache/comet/local/NativeLocal.scala index 7c7e11df839..0cfdefa5e01 100644 --- a/spark/src/main/scala/org/apache/comet/local/NativeLocal.scala +++ b/spark/src/main/scala/org/apache/comet/local/NativeLocal.scala @@ -32,11 +32,13 @@ private[local] class NativeLocal extends NativeBase { * Creates and starts a local query over `partitions` partitions of a serialized Comet plan * without JVM inputs, with the memory pool and session that `Native.createPlan` creates for the * calling Spark task. Its running partitions read ahead of delivery up to `readAheadBytes`. - * Returns its handle. + * When the plan's leaf is a native file scan, `scanPartitions` holds each partition's + * serialized `NativeScan` carrying its files, and is otherwise empty. Returns its handle. */ // scalastyle:off argcount @native def createQuery( plan: Array[Byte], + scanPartitions: Array[Array[Byte]], configs: Array[Byte], partitions: Int, columns: Int, @@ -57,5 +59,11 @@ private[local] class NativeLocal extends NativeBase { @native def close(id: Long): Unit + /** + * The partition whose turn it was when the query failed, or -1. Reading it forgets it, as does + * `close`. + */ + @native def failedPartition(id: Long): Int + @native def activeQueries(): Long } diff --git a/spark/src/test/scala/org/apache/comet/local/CometLocalExecutionSuite.scala b/spark/src/test/scala/org/apache/comet/local/CometLocalExecutionSuite.scala index 910cc79af37..0976211c19b 100644 --- a/spark/src/test/scala/org/apache/comet/local/CometLocalExecutionSuite.scala +++ b/spark/src/test/scala/org/apache/comet/local/CometLocalExecutionSuite.scala @@ -19,6 +19,10 @@ package org.apache.comet.local +import java.io.File +import java.nio.charset.StandardCharsets.UTF_8 +import java.nio.file.{Files, StandardOpenOption} +import java.util.Base64 import java.util.concurrent.{Callable, Executors, TimeUnit} import scala.jdk.CollectionConverters._ @@ -27,9 +31,14 @@ import scala.reflect.ClassTag import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach} import org.scalatest.funsuite.AnyFunSuite +import org.apache.commons.io.FileUtils +import org.apache.parquet.crypto.keytools.PropertiesDrivenCryptoFactory +import org.apache.parquet.crypto.keytools.mocks.InMemoryKMS import org.apache.spark.{SparkEnv, SparkException, TaskContext} import org.apache.spark.sql.{DataFrame, Row, SparkSession} +import org.apache.spark.sql.comet.CometNativeScanExec import org.apache.spark.sql.execution.{ColumnarToRowExec, RangeExec, SparkPlan} +import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, QueryStageExec} import org.apache.spark.sql.execution.columnar.InMemoryRelation import org.apache.spark.sql.execution.exchange.ShuffleExchangeLike import org.apache.spark.sql.expressions.Window @@ -109,6 +118,34 @@ class CometLocalExecutionSuite assert(actual.collect().toSeq == expected(query)) } + /** Writes `data` as Parquet, with Spark, into a temporary directory and passes its path. */ + private def withParquet[T](data: => DataFrame, partitionBy: String*)(f: String => T): T = { + val dir = Files.createTempDirectory("comet-local").toFile + val path = new File(dir, "table").getCanonicalPath + try { + withConf("spark.comet.enabled", "false") { + data.write.partitionBy(partitionBy: _*).parquet(path) + } + f(path) + } finally FileUtils.deleteDirectory(dir) + } + + /** Six files of 2000 rows in all, and a partition column `p` when partitioned. */ + private def table: DataFrame = + spark + .range(0, 2000, 1, 6) + .selectExpr("id", "CAST(id % 7 AS STRING) AS name", "id * 1.5 AS v", "id % 3 AS p") + + /** Splits files into many small partitions, as a large table would be. */ + private def withSmallSplits[T](f: => T): T = + withConf("spark.sql.files.maxPartitionBytes", "2k") { + withConf("spark.sql.files.openCostInBytes", "1")(f) + } + + /** The partitions of the graph that ran `df`'s last collect, if it ran locally. */ + private def localPartitions(df: DataFrame): Option[Int] = + localNodes(df).head.spec.map(_.partitions) + /** Runs `f` in a Spark task, as the local query iterator does. */ private def inTask[T: ClassTag](f: TaskContext => T): T = spark.sparkContext @@ -482,15 +519,212 @@ class CometLocalExecutionSuite } } + test("native Parquet scans run locally, each partition reading its own files") { + withParquet(table) { path => + withSmallSplits { + // Each query is planned afresh, with Comet and without. + def scan = spark.read.parquet(path) + for (query <- Seq( + () => scan, + () => scan.filter("id % 3 = 1").selectExpr("id + 1 AS x", "name", "v * 2 AS w"), + () => scan.filter("name = '3' AND v > 100").select("v", "id"))) { + checkLocal(query()) + assert(localNodes(query()).head.block.exists(_.isInstanceOf[CometNativeScanExec])) + // Ordinary execution reads the same partitions, so Spark's row order is kept. + assert(localPartitions(query()) == Some(scan.rdd.getNumPartitions)) + assert(localPartitions(query()).get > 6) + } + } + // The default splits pack the six small files into fewer partitions. + checkLocal(spark.read.parquet(path).selectExpr("id", "v")) + } + } + + test("partitioned tables are pruned as ordinary execution prunes them") { + withParquet(table, "p") { path => + withSmallSplits { + def scan = spark.read.parquet(path) + checkLocal(scan) + checkLocal(scan.filter("p = 1 AND id > 100").select("p", "id", "name")) + checkLocal(scan.filter("p <> 2").selectExpr("id + p AS q")) + } + } + } + + test("a scan of no files and one of more partitions than run locally collect as ordinary") { + withParquet(table, "p") { path => + // Every partition directory is pruned, so the scan has no partitions. + val none = spark.read.parquet(path).filter("p = 7") + assert(localNodes(none).size == 1) + assert(none.collect().isEmpty) + assert(localPartitions(none).isEmpty) + } + // One file of about 2 MiB, split into 1 KiB partitions. + withParquet(spark.range(0, 60000, 1, 1).selectExpr("id", "uuid() AS u")) { path => + withConf("spark.sql.files.maxPartitionBytes", "1k") { + withConf("spark.sql.files.openCostInBytes", "1") { + val query = spark.read.parquet(path).select("id") + assert(localNodes(query).size == 1) + assert(query.collect().toSeq == expected(spark.read.parquet(path).select("id"))) + assert(localPartitions(query).isEmpty) + assert(spark.read.parquet(path).rdd.getNumPartitions > 1024) + } + } + } + } + + test("bucketed scans and scan subqueries keep ordinary execution") { + withConf("spark.sql.sources.bucketing.autoBucketedScan.enabled", "false") { + try { + withConf("spark.comet.enabled", "false") { + table.write.bucketBy(4, "id").saveAsTable("comet_local_bucketed") + } + val bucketed = () => spark.table("comet_local_bucketed").select("id", "name") + assert(localNodes(bucketed()).isEmpty, bucketed().queryExecution.executedPlan.toString) + assert( + bucketed().collect().toSeq.sortBy(_.getLong(0)) == + expected(bucketed()).sortBy(_.getLong(0))) + } finally spark.sql("DROP TABLE IF EXISTS comet_local_bucketed") + } + withParquet(table) { path => + spark.read.parquet(path).createOrReplaceTempView("comet_local_scan") + try { + val subquery = () => + spark.sql( + "SELECT id FROM comet_local_scan WHERE id > (SELECT max(id) - 5 FROM range(2000))") + assert(localNodes(subquery()).isEmpty, subquery().queryExecution.executedPlan.toString) + assert(subquery().collect().toSeq == expected(subquery())) + for (query <- Seq( + () => spark.read.parquet(path).selectExpr("id", "spark_partition_id() AS s"), + () => spark.read.parquet(path).selectExpr("id", "rand(7) AS r"), + () => + spark.read.parquet(path).selectExpr("id", "monotonically_increasing_id() AS m"))) { + assert(localNodes(query()).isEmpty, query().queryExecution.executedPlan.toString) + assert(query().collect().toSeq == expected(query())) + } + } finally spark.catalog.dropTempView("comet_local_scan") + } + } + + test("encrypted files keep ordinary execution, which reads them through the JVM") { + val key = Base64.getEncoder.encodeToString("1234567890123450".getBytes(UTF_8)) + withConf("parquet.crypto.factory.class", classOf[PropertiesDrivenCryptoFactory].getName) { + withConf("parquet.encryption.kms.client.class", classOf[InMemoryKMS].getName) { + withConf("parquet.encryption.key.list", s"key1: $key") { + val dir = Files.createTempDirectory("comet-local").toFile + val path = new File(dir, "encrypted").getCanonicalPath + try { + withConf("spark.comet.enabled", "false") { + table.write.option("parquet.encryption.uniform.key", "key1").parquet(path) + } + val query = () => spark.read.parquet(path).select("id", "name") + assert( + query().queryExecution.executedPlan.exists(_.isInstanceOf[CometNativeScanExec])) + assert(localNodes(query()).isEmpty, query().queryExecution.executedPlan.toString) + assert(query().collect().toSeq == expected(query())) + } finally FileUtils.deleteDirectory(dir) + } + } + } + } + + test("scans run locally whether AQE is enabled, disabled or forced") { + withParquet(table) { path => + withSmallSplits { + for (adaptive <- Seq("true", "false")) { + withConf("spark.sql.adaptive.enabled", adaptive) { + checkLocal(spark.read.parquet(path).filter("id % 5 = 0").select("id", "v")) + } + } + withConf("spark.sql.adaptive.forceApply", "true") { + for (query <- Seq( + () => spark.read.parquet(path).filter("id % 5 = 0").select("id", "v"), + () => spark.range(0, 500, 1, 5).selectExpr("id * 3 AS v"))) { + val df = query() + assert(df.collect().toSeq == expected(query())) + val plan = df.queryExecution.executedPlan + assert(plan.isInstanceOf[AdaptiveSparkPlanExec], plan.toString) + def local(plan: SparkPlan): Seq[CometLocalExec] = plan match { + case p: CometLocalExec => Seq(p) + case p: AdaptiveSparkPlanExec => local(p.executedPlan) + case p: QueryStageExec => local(p.plan) + case p => p.children.flatMap(local) + } + assert(local(plan).size == 1, plan.toString) + } + } + } + } + } + + test("a cached scan and its RDD keep the partitions of ordinary execution") { + withParquet(table) { path => + withSmallSplits { + val ordinaryPartitions = + withConf(enabled, "false")(spark.read.parquet(path).rdd.getNumPartitions) + val query = spark.read.parquet(path).select("id", "p") + assert(localNodes(query).size == 1) + assert(query.rdd.getNumPartitions == ordinaryPartitions) + query.cache() + try { + val relations = query.groupBy("p").count().queryExecution.withCachedData.collect { + case relation: InMemoryRelation => relation + } + assert( + relations.head.cacheBuilder.cachedColumnBuffers.getNumPartitions == + ordinaryPartitions) + assert(query.groupBy("p").count().collect().map(_.getLong(1)).sum == 2000L) + } finally query.unpersist(blocking = true) + val limited = spark.read.parquet(path).select("id") + assert(localNodes(limited.limit(5)).isEmpty) + assert( + limited.take(5).toSeq == withConf("spark.comet.enabled", "false")( + spark.read.parquet(path).select("id").take(5).toSeq)) + } + } + } + + test("a file a partition fails to read is named, as ordinary execution names it") { + withParquet(table) { path => + val files = + new File(path).listFiles().filter(_.getName.endsWith(".parquet")).sortBy(_.getName) + val corrupt = files.last + Files.write(corrupt.toPath, Array.fill[Byte](4096)(7), StandardOpenOption.TRUNCATE_EXISTING) + withSmallSplits { + // The schema is given, so that inferring it does not read the corrupt file. + val query = () => spark.read.schema(table.schema).parquet(path).select("id") + assert(localNodes(query()).size == 1) + val local = intercept[SparkException](query().collect()) + val ordinary = intercept[SparkException](withConf(enabled, "false")(query().collect())) + for (error <- Seq(local, ordinary)) { + assert(error.getMessage.contains(corrupt.getName), error.getMessage) + } + assert(local.getClass == ordinary.getClass) + assert(Option(local.getErrorClass) == Option(ordinary.getErrorClass)) + } + } + } + + test("explain shows a scan's block before its files are listed") { + withParquet(table) { path => + val query = spark.read.parquet(path).select("id") + val local = localNodes(query).head + assert(!local.toString.contains("partitions="), local.toString) + assert(query.queryExecution.executedPlan.treeString.contains("CometNativeScan")) + query.collect() + assert(local.metrics("numOutputRows").value == 2000) + } + } + test("native creation and output errors fail without leaking handles") { - val spec = localNodes(spark.range(10, 20, 1, 1).toDF()).head.spec + val spec = localNodes(spark.range(10, 20, 1, 1).toDF()).head.spec.get val (failures, live) = inTask(CometLocalExecutionSuite.creationAndOutputErrors(spec, _)) - assert(failures == Seq(true, true, true)) + assert(failures == Seq(true, true, true, true)) assert(live == 0) } test("imported Arrow batches outlive native close") { - val spec = localNodes(spark.range(10, 20, 1, 1).toDF()).head.spec + val spec = localNodes(spark.range(10, 20, 1, 1).toDF()).head.spec.get assert(inTask(CometLocalExecutionSuite.readAfterClose(spec, _)) == (10L until 20L)) } @@ -525,10 +759,12 @@ object CometLocalExecutionSuite { spec: LocalQuerySpec, context: TaskContext, partitions: Int, - columns: Int): Long = { + columns: Int, + scanPartitions: Option[Array[Array[Byte]]] = None): Long = { val memory = CometExecIterator.getMemoryConfig(SparkEnv.get.conf) native.createQuery( spec.plan, + scanPartitions.getOrElse(spec.scanPartitions), CometExecIterator.serializeCometSQLConfs(), partitions, columns, @@ -557,11 +793,20 @@ object CometLocalExecutionSuite { val native = new NativeLocal val noPartitions = fails(createQuery(native, spec, context, partitions = 0, columns = 1)) val wrongWidth = fails(createQuery(native, spec, context, partitions = 1, columns = 2)) + // A range has no file scan to give files to. + val strayFiles = fails( + createQuery( + native, + spec, + context, + partitions = 1, + columns = 1, + scanPartitions = Some(Array(Array.emptyByteArray)))) val id = createQuery(native, spec, context, partitions = 1, columns = 1) val nullOutput = fails(native.nextBatch(id, Array(0L), Array(0L))) native.close(id) native.close(id) - (Seq(noPartitions, wrongWidth, nullOutput), native.activeQueries()) + (Seq(noPartitions, wrongWidth, strayFiles, nullOutput), native.activeQueries()) } /** The values of the first batch, read after its native query is closed. */