Repository navigation
Conversation
|
@ErikBPF thanks for opening this. Could you add me as a collaborator on ErikBPF/datafusion-comet so I can push to |
|
Here's how I'd split the work so each piece stays reviewable:
I'll open the first two from my fork, link them here, and add you as a collaborator on it so you can push to those branches. Once you've added me on yours, I'll move the serde and suite into this PR and hand it back to you. |
Adds delta_interleave_bits, a scalar function that builds Delta Lake's Z-order key for one or two Int32 inputs: bits are taken from the most significant down, alternating between inputs with the first input in the higher position, a null input counts as zero, and the result is four bytes per input and never null. The tests check the output against values produced by Delta's own InterleaveBits on both its default and fast code paths, including signed extrema and nulls. Nothing calls the function from Spark yet; the Delta serde that uses it comes separately.
|
@dwsmith1983 Thanks. I sent a collaborator invitation for The proposed split works for us:
I will keep the manual probe in this PR as the reference baseline until the suite supersedes it. The only open dependency is the expression-admission design in #7573; we will not add core Delta class matching on our side unless that is the maintainers' preference. |
Adds CometExpressionContrib, a ServiceLoader hook beside CometScanContrib, so a contrib can register a serde for an expression class core never names. Delta's InterleaveBits is the first user: the class lives in delta-spark, which core does not depend on. QueryPlanSerde.serdeFor now looks in core's map first and the contrib registry second, and both lookup sites use it. A contrib entry for a class core already handles is ignored, a class claimed by two contribs is dropped from both since ServiceLoader order is unspecified, and a contrib that throws is skipped. Default builds ship no service file, so nothing changes for them. The scan suite's service-file test helper moves into a shared test utility that both suites use.
Registers a serde for Delta's InterleaveBits through the contrib expression hook, so the Z-order key for one or two integer inputs runs in Comet's native kernel instead of falling back. Other forms keep falling back with a reason, and spark.comet.expression.InterleaveBits.enabled turns it off. The new suite checks exact bytes for column and literal inputs, nulls, signed extrema and generated pairs against Spark under both of Delta's interleave code paths, across several batches and on an empty table. It replaces the standalone reference checks under dev/repros/delta. The Delta user guide gains a section on Z-order keys, and the scan flag's description now says only scans are left to Spark by default.
Arrow builders leave zero under a null slot, so the existing null tests passed even without the kernel's null check. The new test puts -1 under a null and expects the bytes of a zero input.
|
@ErikBPF pushed the Delta serde and the expression suite (dcbbdec). Until the two pieces underneath merge, this diff also carries them:
After those merge and this branch merges It's yours from here for cleanup and extending. Open items I left:
|
|
@ErikBPF the core hook is up as draft #7686, the same |
Which issue does this PR close?
Refs #7573. This remains a draft pending agreement on the expression-registration seam.
Rationale for this change
Delta's
InterleaveBitsfalls back to Spark. This adds a Delta contrib serde and expression suite for the one- and two-inputINTshapes scoped in #7573, backed by a Rust kernel.The work is split into three pieces:
CometExpressionContribServiceLoader hook. It is open as a draft; the design question in Support Delta InterleaveBits for one or two INT inputs #7573 still awaits maintainer feedback.The diff currently carries the two prerequisites. Once they merge, merging
maininto this branch will leave the Delta integration changes. The kernel (#7600) and hook (#7686) can land in either order; this PR lands last.OPTIMIZE ... ZORDER BYstill falls back for key construction because its range-partition inputs are unsupported. Hilbert ordering and broader clustering operations remain out of scope.What changes are included in this PR?
OPTIMIZEregression with the exact ordered["a","b"]history assertion.How are these changes tested?
Observed on Apollo with Spark 4.1.3 / Delta 4.3.1 and JDK 17:
cargo test -p datafusion-comet-spark-expr --lib delta_funcs: 10 passed.CometExpressionContribSuiteandSerdeRegistrationSuite: 10 passed.CometDeltaInterleaveBitsSuite: 9 passed.-Pspark-4.1,delta -DskipTests, native release build, Rust formatting, Spotless and suite-registration checks passed.Benchmark: 1,048,576 Parquet rows, deterministic integers with nulls in the second column,
local[1], Xeon E5-2690 v4, standard release profile. Best times:These measure Parquet scan plus projection, not the kernel in isolation or
OPTIMIZEthroughput. Reproduction commands are incontrib/delta-spark/README.md.These are local results. Cross-version CI and strict Spark 3.5 compilation were not rerun for this cleanup. The Spark 4.2 / Delta pairing remains separate work in #7593.