Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr_build_linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -688,6 +688,7 @@ jobs:
org.apache.comet.CometJsonJvmSuite
org.apache.comet.SparkErrorConverterSuite
org.apache.comet.serde.SerdeRegistrationSuite
org.apache.comet.serde.CometExpressionContribSuite
org.apache.comet.expressions.conditional.CometIfSuite
org.apache.comet.expressions.conditional.CometCoalesceSuite
org.apache.comet.expressions.conditional.CometCaseWhenSuite
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/pr_build_macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,7 @@ jobs:
org.apache.comet.CometJsonJvmSuite
org.apache.comet.SparkErrorConverterSuite
org.apache.comet.serde.SerdeRegistrationSuite
org.apache.comet.serde.CometExpressionContribSuite
org.apache.comet.expressions.conditional.CometIfSuite
org.apache.comet.expressions.conditional.CometCoalesceSuite
org.apache.comet.expressions.conditional.CometCaseWhenSuite
Expand Down
28 changes: 26 additions & 2 deletions contrib/delta-spark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,17 @@ Native Delta Lake reads for Comet. Delta tables are scanned through Comet's
existing native Parquet reader, so they get row-group pruning, page-index
pruning, and filter pushdown, with deletion vectors applied inside the scan.

Support is experimental and explicitly opt-in. Two things are required:
The jar also gives Comet a Rust implementation of Delta's `InterleaveBits`
(the Z-order key) for one or two integer inputs. It is on whenever the jar is
present; `spark.comet.expression.InterleaveBits.enabled=false` turns it off.

Scan support is experimental and explicitly opt-in. Two things are required:

1. This module's jar (`comet-contrib-delta-spark`) on the classpath, alongside
`delta-spark`. It is never bundled into `comet-spark`; without it, Comet
has no Delta surface at all.
2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so the jar
alone does nothing.
alone leaves Delta table scans to Spark.

Unsupported tables and features fall back to Spark's reader. See the
[user guide](https://datafusion.apache.org/comet/user-guide/latest/delta.html)
Expand Down Expand Up @@ -68,3 +72,23 @@ no Docker daemon is reachable; setting `COMET_DELTA_S3_REQUIRED=1`, as the

`dev/` contains a benchmark script (`bench_delta_comet.py`) and a harness for
running Delta's own test suites against Comet (`run-delta-regression.sh`).

### InterleaveBits benchmark

Run from the repository root with JDK 17 and the Rust toolchain available:

```shell
(cd native && cargo build --release)
./mvnw -Pspark-4.1,delta -Prelease install -DskipTests
COMET_BENCHMARK_JVM_ARGS=$(./mvnw -q -Pspark-4.1 help:evaluate -DforceStdout -Dexpression=extraJavaTestArgs)
COMET_CONF_DIR="$PWD/conf" \
./mvnw -f contrib/delta-spark/pom.xml -Pspark-4.1,delta -Prelease \
exec:exec -Dexec.executable=java -Dexec.classpathScope=test \
"-Dexec.args=-Xmx6g $COMET_BENCHMARK_JVM_ARGS -Djava.library.path=$PWD/native/target/release -classpath %classpath org.apache.spark.sql.benchmark.CometDeltaInterleaveBitsBenchmark"
```

This compares one and two integer inputs over one million Parquet rows, including
nulls in the second input, with Delta's fast implementation enabled and disabled.
Both engines consume the projected keys. Check the output for fallback warnings
before interpreting timings. Results include scanning and projection; they do
not measure `OPTIMIZE` or the kernel in isolation.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
org.apache.comet.contrib.delta.DeltaExpressionContrib
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.comet.contrib.delta

import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.delta.expressions.InterleaveBits
import org.apache.spark.sql.types.{BinaryType, IntegerType}

import org.apache.comet.serde.{CometExpressionSerde, Compatible, SupportLevel, Unsupported}
import org.apache.comet.serde.ExprOuterClass.Expr
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctionExprToProtoWithReturnType}

/**
* Delta's `InterleaveBits`, the Z-order key `OPTIMIZE ... ZORDER BY` sorts on, through the native
* `delta_interleave_bits` kernel. The kernel takes one or two int inputs; other shapes fall back.
*
* A null input counts as 0 and the result is never null, so the call is not wrapped in a null
* short circuit.
*/
object CometDeltaInterleaveBits extends CometExpressionSerde[InterleaveBits] {

private val MaxInputs = 2

override def getSupportLevel(expr: InterleaveBits): SupportLevel = {
val arity = expr.children.length
if (arity < 1 || arity > MaxInputs) {
Unsupported(Some(s"InterleaveBits supports 1 or $MaxInputs inputs, got $arity"))
} else {
expr.children.find(_.dataType != IntegerType) match {
case Some(child) =>
Unsupported(
Some(s"InterleaveBits supports only int inputs, got ${child.dataType.catalogString}"))
case None => Compatible(None)
}
}
}

override def convert(
expr: InterleaveBits,
inputs: Seq[Attribute],
binding: Boolean): Option[Expr] = {
val args = expr.children.map(exprToProtoInternal(_, inputs, binding))
scalarFunctionExprToProtoWithReturnType("delta_interleave_bits", BinaryType, false, args: _*)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.comet.contrib.delta

import org.apache.spark.sql.catalyst.expressions.Expression
import org.apache.spark.sql.delta.expressions.InterleaveBits

import org.apache.comet.serde.{CometExpressionContrib, CometExpressionSerde}

/**
* Serdes for Delta expressions, discovered by core's ServiceLoader (see
* `META-INF/services/org.apache.comet.serde.CometExpressionContrib`).
*/
class DeltaExpressionContrib extends CometExpressionContrib {

override def expressionSerdes: Map[Class[_ <: Expression], CometExpressionSerde[_]] =
Map(classOf[InterleaveBits] -> CometDeltaInterleaveBits)
}
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ object DeltaScanConf {
"planned by delta-spark are executed through Comet's native Parquet scan, " +
"inheriting row-group pruning, page-index pruning, and filter pushdown, with " +
"deletion vectors applied inside the scan. Experimental: defaults to false, so " +
"adding the contrib jar does not by itself change how any query is read.")
"adding the contrib jar does not by itself change how any Delta table is scanned.")
.booleanConf
.createWithDefault(false)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ class DeltaScanContrib extends CometScanContrib with Logging {

if (!DeltaScanConf.scanEnabled) {
// Deliberate deviation from the "own but cannot handle => claim" contract: a
// user-disabled contrib must be fully inert (the jar alone changes nothing) and must
// user-disabled contrib must be fully inert (the jar alone changes no scan) and must
// not shadow another registered Delta contrib. Tag the opt-in hint for EXPLAIN, pass.
withFallbackReason(
scanExec,
Expand Down
Loading