Skip to content

feat: run admitted native Parquet scans as one local graph - #7692

Open
viirya wants to merge 1 commit into
apache:mainfrom
viirya:local-exec-scan
Open

viirya wants to merge 1 commit into
apache:mainfrom
viirya:local-exec-scan

Conversation

@viirya

@viirya viirya commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #6499 (step 2). Builds on #6883.

Rationale for this change

#6883 runs a query as one local DataFusion graph only when its leaf is a range. Step 2 of #6499 admits exchange-free plans over Comet's native Parquet scan (scan, filter, project), judged from the plan CometExecRule produces, whether or not AQE is enabled.

What changes are included in this PR?

Admission (CometLocalRule)

  • The block's single leaf may now be a CometNativeScanExec. The scan is declined when it reports partitioning (a bucketed or key-grouped scan) or reads encrypted files, since decryption needs the JVM.
  • A plan whose expressions contain a physical subquery (PlanExpression, for example a scalar subquery or dynamic pruning in a scan's filters) is declined. These are not in the serialized plan, so the existing JVM-callback check on the plan does not see them.
  • A scan's file partitions are listed only when the query runs, as in ordinary execution. So CometLocalExec builds its spec at execution time, after resolving the scan's subqueries. When there are no partitions, or more than 1024, collect runs as ordinary Comet execution.
  • The result node copies the block's logical link. Without it, spark.sql.adaptive.forceApply failed with an internal error when AQE made the result its final stage. This also affected range plans from feat: run admitted local queries as one native graph #6883.

Native

  • createQuery takes each partition's serialized NativeScan with its files, which is the same data a Spark task receives. A partition's files are injected into a copy of the plan when that partition is planned. The ordinary planner and the protobuf are unchanged.
  • native/local now plans and starts each partition on the runtime's blocking pool. This is needed because creating an S3 object store blocks on Comet's runtime, which panics on a worker. The started stream is stored from the blocking thread, so cancellation still drops it synchronously.
  • The query records which partition's turn it was when it failed. The JVM uses this to name that partition's files when a native read error carries no file path, as ordinary execution names the failing task's files.

The contributor guide page local-execution.md is updated. One known gap is documented there: a local scan reports no task input metrics.

How are these changes tested?

  • native/local: new tests for a partition start that blocks on the runtime, a start that panics, and the partition reported for an error.
  • native/core (local::): each partition reads only its own Parquet files, an empty file group yields an empty partition, and file lists that do not match the plan are rejected.
  • CometLocalExecutionSuite: Parquet scans split into many partitions are compared with Spark's results and row order. Further tests cover:
    • partition pruning;
    • scans of no files or of more than 1024 partitions;
    • declined bucketed, encrypted and subquery plans;
    • AQE on, off and forceApply (range and scan);
    • cached and RDD partition counts;
    • a corrupt file named in the error;
    • explain before files are listed.

Local execution now admits a native block whose leaf is a native
Parquet scan, not only a range, so scan/filter/project queries run as
one DataFusion graph whether or not AQE is enabled.

- Admission declines a scan that reports partitioning (bucketed) or
  reads encrypted files, and any expression with a physical subquery,
  such as a scalar subquery in a scan's filters.
- A scan's file partitions are listed when the query runs, as ordinary
  execution lists them. A collect with no partitions or more than 1024
  runs as ordinary execution.
- The JNI call takes each partition's serialized NativeScan with its
  files, and a partition's files are injected into a copy of the plan
  when the partition is planned.
- native/local plans and starts each partition on the runtime's
  blocking pool, since creating an S3 object store blocks on Comet's
  runtime, which panics on a worker.
- A file read error names the failing partition's files, as ordinary
  execution names its task's.
- The local result carries the block's logical link, which AQE needs
  when spark.sql.adaptive.forceApply wraps the plan.

Co-authored-by: Isaac <no-reply@databricks.com>
@viirya

viirya commented Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

Benchmark of collect for the plans this PR admits, comparing Spark, ordinary Comet execution and Comet local execution.

Setup: Apple M4 Max, local[8], JDK 17, Spark 4.1, native library built with --release -Ctarget-cpu=native, 4 GB off-heap. Table: 8M rows (id long, a int, b double, s short string) in 64 Parquet files. Each case runs 10 iterations. Times are averages in ms; best times are in parentheses.

Case Partitions Rows returned Spark Comet Comet local
All rows 8 8M 1414 (1225) 1178 (1089) 1254 (1056)
All rows 320 8M 1164 (1069) 1279 (1129) 1235 (1026)
Selective filter 8 8K 62 (57) 34 (33) 33 (31)
Selective filter 320 8K 93 (87) 74 (71) 107 (101)
All rows, half in one file 8 8M 1791 (1590) 1574 (1397) 1124 (1027)
Range, all rows 8 8M 494 (396) 541 (422) 439 (237)
Range, selective filter 8 8K 7 (7) 17 (9) 16 (9)

A second run showed the same pattern.

  • When files are uniform, local execution is within noise of ordinary Comet. These plans have no exchange, so the only work it removes is per-task overhead and the encoding of task results.
  • When one partition holds half the rows, local execution is about 30% faster than ordinary Comet. Ordinary execution encodes and compresses that partition's result in one task and decodes it on the driver, all on one thread. Local execution copies the rows straight into the driver's result slot.
  • Many tiny partitions are slower: with 320 partitions returning 8K rows in all, local execution takes about 45% longer than ordinary Comet, and the result was stable across both runs.

The benchmark (CometLocalExecutionBenchmark) is not part of this PR.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the admission rules, the per-partition file injection and the fallback to ordinary collect for a scan with no partitions or more than 1024. The local graph gets the same file partitions as ordinary execution, so Spark's row order holds. The comments below are about the JNI loop, the failed-partition bookkeeping and one branch that can no longer run. This PR and #7693 conflict textually only in local-execution.md.

Comment thread native/core/src/local.rs
Comment on lines +136 to +141
let mut scan_partitions_vec = Vec::with_capacity(scan_partitions.len(env)?);
for i in 0..scan_partitions.len(env)? {
let bytes = scan_partitions.get_element(env, i)?;
let bytes = unsafe { JByteArray::from_raw(&*env, bytes.into_raw()) };
scan_partitions_vec.push(env.convert_byte_array(bytes)?);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each get_element result stays a live local reference until createQuery returns, so a scan of 1024 partitions holds 1024 of them in one JNI frame. The jni 0.22 docs for delete_local_ref recommend freeing references in loops like this, and credential_bridge.rs does it for the same reason (env.delete_local_ref(element), with the comment about the local frame). Converting with env.convert_byte_array(&bytes) and then calling env.delete_local_ref(bytes), or using .auto(), avoids that. Reading scan_partitions.len(env)? once also saves the second JNI call.

Comment thread native/core/src/local.rs
Comment on lines +406 to 413
if result.is_err() {
let partition = QUERIES.lock().get(&id).map(|entry| entry.query.partition());
if let Some(partition) = partition {
FAILED_PARTITIONS.lock().insert(id, partition);
}
}
if result.is_err() || matches!(result, Ok(-1)) {
close(id);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LocalQueryIterator.executionFailure reads the failed partition and then closes the query, and the task completion listener closes it on every other path. If nextBatch closed only at end of stream and left a failed query in QUERIES until close, failedPartition could read entry.query.partition() from the live entry. FAILED_PARTITIONS, its removal in close and the extra count in activeQueries would then not be needed. PartitionOrder::fail already drops the running partitions on an error, so I expect the entry to hold no running partition while it waits for close.

Comment thread native/local/src/lib.rs
Comment on lines +365 to 372
if self.running.is_empty() {
return Poll::Ready(None);
}
let turn = self.next - self.running.len();
self.turn.store(turn, Ordering::Release);
let Some(partition) = self.running.front_mut() else {
return Poll::Ready(None);
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this check, the else of the let Some(partition) = self.running.front_mut() else two lines below can no longer run, so one of the two exits can go. self.next - self.running.len() cannot underflow on an empty queue, and turn is read only right after an error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants