Repository navigation
Conversation
Local execution now admits a native block whose leaf is a native Parquet scan, not only a range, so scan/filter/project queries run as one DataFusion graph whether or not AQE is enabled. - Admission declines a scan that reports partitioning (bucketed) or reads encrypted files, and any expression with a physical subquery, such as a scalar subquery in a scan's filters. - A scan's file partitions are listed when the query runs, as ordinary execution lists them. A collect with no partitions or more than 1024 runs as ordinary execution. - The JNI call takes each partition's serialized NativeScan with its files, and a partition's files are injected into a copy of the plan when the partition is planned. - native/local plans and starts each partition on the runtime's blocking pool, since creating an S3 object store blocks on Comet's runtime, which panics on a worker. - A file read error names the failing partition's files, as ordinary execution names its task's. - The local result carries the block's logical link, which AQE needs when spark.sql.adaptive.forceApply wraps the plan. Co-authored-by: Isaac <no-reply@databricks.com>
|
Benchmark of Setup: Apple M4 Max,
A second run showed the same pattern.
The benchmark ( |
comphead
left a comment
There was a problem hiding this comment.
I read the admission rules, the per-partition file injection and the fallback to ordinary collect for a scan with no partitions or more than 1024. The local graph gets the same file partitions as ordinary execution, so Spark's row order holds. The comments below are about the JNI loop, the failed-partition bookkeeping and one branch that can no longer run. This PR and #7693 conflict textually only in local-execution.md.
| let mut scan_partitions_vec = Vec::with_capacity(scan_partitions.len(env)?); | ||
| for i in 0..scan_partitions.len(env)? { | ||
| let bytes = scan_partitions.get_element(env, i)?; | ||
| let bytes = unsafe { JByteArray::from_raw(&*env, bytes.into_raw()) }; | ||
| scan_partitions_vec.push(env.convert_byte_array(bytes)?); | ||
| } |
There was a problem hiding this comment.
Each get_element result stays a live local reference until createQuery returns, so a scan of 1024 partitions holds 1024 of them in one JNI frame. The jni 0.22 docs for delete_local_ref recommend freeing references in loops like this, and credential_bridge.rs does it for the same reason (env.delete_local_ref(element), with the comment about the local frame). Converting with env.convert_byte_array(&bytes) and then calling env.delete_local_ref(bytes), or using .auto(), avoids that. Reading scan_partitions.len(env)? once also saves the second JNI call.
| if result.is_err() { | ||
| let partition = QUERIES.lock().get(&id).map(|entry| entry.query.partition()); | ||
| if let Some(partition) = partition { | ||
| FAILED_PARTITIONS.lock().insert(id, partition); | ||
| } | ||
| } | ||
| if result.is_err() || matches!(result, Ok(-1)) { | ||
| close(id); |
There was a problem hiding this comment.
LocalQueryIterator.executionFailure reads the failed partition and then closes the query, and the task completion listener closes it on every other path. If nextBatch closed only at end of stream and left a failed query in QUERIES until close, failedPartition could read entry.query.partition() from the live entry. FAILED_PARTITIONS, its removal in close and the extra count in activeQueries would then not be needed. PartitionOrder::fail already drops the running partitions on an error, so I expect the entry to hold no running partition while it waits for close.
| if self.running.is_empty() { | ||
| return Poll::Ready(None); | ||
| } | ||
| let turn = self.next - self.running.len(); | ||
| self.turn.store(turn, Ordering::Release); | ||
| let Some(partition) = self.running.front_mut() else { | ||
| return Poll::Ready(None); | ||
| }; |
There was a problem hiding this comment.
With this check, the else of the let Some(partition) = self.running.front_mut() else two lines below can no longer run, so one of the two exits can go. self.next - self.running.len() cannot underflow on an empty queue, and turn is read only right after an error.
Which issue does this PR close?
Part of #6499 (step 2). Builds on #6883.
Rationale for this change
#6883 runs a query as one local DataFusion graph only when its leaf is a range. Step 2 of #6499 admits exchange-free plans over Comet's native Parquet scan (scan, filter, project), judged from the plan
CometExecRuleproduces, whether or not AQE is enabled.What changes are included in this PR?
Admission (
CometLocalRule)CometNativeScanExec. The scan is declined when it reports partitioning (a bucketed or key-grouped scan) or reads encrypted files, since decryption needs the JVM.PlanExpression, for example a scalar subquery or dynamic pruning in a scan's filters) is declined. These are not in the serialized plan, so the existing JVM-callback check on the plan does not see them.CometLocalExecbuilds its spec at execution time, after resolving the scan's subqueries. When there are no partitions, or more than 1024,collectruns as ordinary Comet execution.spark.sql.adaptive.forceApplyfailed with an internal error when AQE made the result its final stage. This also affected range plans from feat: run admitted local queries as one native graph #6883.Native
createQuerytakes each partition's serializedNativeScanwith its files, which is the same data a Spark task receives. A partition's files are injected into a copy of the plan when that partition is planned. The ordinary planner and the protobuf are unchanged.native/localnow plans and starts each partition on the runtime's blocking pool. This is needed because creating an S3 object store blocks on Comet's runtime, which panics on a worker. The started stream is stored from the blocking thread, so cancellation still drops it synchronously.The contributor guide page
local-execution.mdis updated. One known gap is documented there: a local scan reports no task input metrics.How are these changes tested?
native/local: new tests for a partition start that blocks on the runtime, a start that panics, and the partition reported for an error.native/core(local::): each partition reads only its own Parquet files, an empty file group yields an empty partition, and file lists that do not match the plan are rejected.CometLocalExecutionSuite: Parquet scans split into many partitions are compared with Spark's results and row order. Further tests cover:forceApply(range and scan);