Skip to content

feat(delta): run Delta InterleaveBits natively - #7597

Draft
ErikBPF wants to merge 12 commits into
apache:mainfrom
ErikBPF:feat/7573-interleave-handoff
Draft

ErikBPF wants to merge 12 commits into
apache:mainfrom
ErikBPF:feat/7573-interleave-handoff

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Refs #7573. This remains a draft pending agreement on the expression-registration seam.

Rationale for this change

Delta's InterleaveBits falls back to Spark. This adds a Delta contrib serde and expression suite for the one- and two-input INT shapes scoped in #7573, backed by a Rust kernel.

The work is split into three pieces:

  1. feat: add a native kernel for Delta's InterleaveBits #7600 owns the Rust kernel and its unit tests.
  2. feat: let contribs register expression serdes #7686 owns the core CometExpressionContrib ServiceLoader hook. It is open as a draft; the design question in Support Delta InterleaveBits for one or two INT inputs #7573 still awaits maintainer feedback.
  3. This PR owns the Delta serde, integration suite, documentation and benchmark.

The diff currently carries the two prerequisites. Once they merge, merging main into this branch will leave the Delta integration changes. The kernel (#7600) and hook (#7686) can land in either order; this PR lands last.

OPTIMIZE ... ZORDER BY still falls back for key construction because its range-partition inputs are unsupported. Hilbert ordering and broader clustering operations remain out of scope.

What changes are included in this PR?

  • Delta contrib registration and serde, preserving fallback for unsupported types and arities.
  • Expression tests against both Delta implementations: exact bytes, nulls, signed extrema, generated pairs, column/literal combinations, constant-folding control, empty tables, config disable and fallback reasons.
  • An OPTIMIZE regression with the exact ordered ["a","b"] history assertion.
  • Current-behavior documentation and a reproducible contrib benchmark with a separate JVM runner.
  • Prerequisite core registration hook and Rust kernel, including direct zero-length batch tests. The same empty-batch coverage is included in feat: add a native kernel for Delta's InterleaveBits #7600.

How are these changes tested?

Observed on Apollo with Spark 4.1.3 / Delta 4.3.1 and JDK 17:

  • cargo test -p datafusion-comet-spark-expr --lib delta_funcs: 10 passed.
  • CometExpressionContribSuite and SerdeRegistrationSuite: 10 passed.
  • CometDeltaInterleaveBitsSuite: 9 passed.
  • Full JVM install with -Pspark-4.1,delta -DskipTests, native release build, Rust formatting, Spotless and suite-registration checks passed.
  • Independent review completed; the benchmark runner was corrected to avoid an in-process Maven/Hadoop shutdown-classloader error. The final child-JVM run completed cleanly with no fallback warnings, and process mappings confirmed the freshly built release library was loaded.

Benchmark: 1,048,576 Parquet rows, deterministic integers with nulls in the second column, local[1], Xeon E5-2690 v4, standard release profile. Best times:

Inputs Delta fast path Spark Comet Relative
One Enabled 118 ms 97 ms 1.2x
Two Enabled 171 ms 92 ms 1.9x
One Disabled 255 ms 90 ms 2.8x
Two Disabled 365 ms 89 ms 4.1x

These measure Parquet scan plus projection, not the kernel in isolation or OPTIMIZE throughput. Reproduction commands are in contrib/delta-spark/README.md.

These are local results. Cross-version CI and strict Spark 3.5 compilation were not rerun for this cleanup. The Spark 4.2 / Delta pairing remains separate work in #7593.

@dwsmith1983

Copy link
Copy Markdown
Contributor

@ErikBPF thanks for opening this. Could you add me as a collaborator on ErikBPF/datafusion-comet so I can push to feat/7573-interleave-handoff? "Allow edits by maintainers" doesn't cover me, since I don't have write access here. I'll bring over only the Morton interleave for InterleaveBits; Hilbert will get its own issue and PR.

@dwsmith1983

Copy link
Copy Markdown
Contributor

Here's how I'd split the work so each piece stays reviewable:

  1. A core PR adding a CometExpressionContrib hook, so the Delta contrib can register a serde for InterleaveBits. It waits on the maintainers' answer in Support Delta InterleaveBits for one or two INT inputs #7573.
  2. A Rust-only PR with the delta_interleave_bits kernel, tested against vectors generated from both of Delta's code paths.
  3. This PR on top of both, with the Delta serde and an expression suite that takes over the checks from your probe.

I'll open the first two from my fork, link them here, and add you as a collaborator on it so you can push to those branches. Once you've added me on yours, I'll move the serde and suite into this PR and hand it back to you.

Adds delta_interleave_bits, a scalar function that builds Delta Lake's
Z-order key for one or two Int32 inputs: bits are taken from the most
significant down, alternating between inputs with the first input in
the higher position, a null input counts as zero, and the result is
four bytes per input and never null.

The tests check the output against values produced by Delta's own
InterleaveBits on both its default and fast code paths, including
signed extrema and nulls. Nothing calls the function from Spark yet;
the Delta serde that uses it comes separately.
@dwsmith1983

Copy link
Copy Markdown
Contributor

The native kernel is up as #7600 (draft). The core hook PR follows once the design question in #7573 is settled.

@ErikBPF

ErikBPF commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@dwsmith1983 Thanks. I sent a collaborator invitation for ErikBPF/datafusion-comet with push access; accept it and you can push directly to feat/7573-interleave-handoff. No separate branch is needed.

The proposed split works for us:

  1. Core CometExpressionContrib hook once Support Delta InterleaveBits for one or two INT inputs #7573 gets a maintainer answer.
  2. Rust-only delta_interleave_bits kernel checked against both Delta reference paths.
  3. This PR on top with the Delta serde and the expression suite.

I will keep the manual probe in this PR as the reference baseline until the suite supersedes it. The only open dependency is the expression-admission design in #7573; we will not add core Delta class matching on our side unless that is the maintainers' preference.

Adds CometExpressionContrib, a ServiceLoader hook beside CometScanContrib,
so a contrib can register a serde for an expression class core never
names. Delta's InterleaveBits is the first user: the class lives in
delta-spark, which core does not depend on.

QueryPlanSerde.serdeFor now looks in core's map first and the contrib
registry second, and both lookup sites use it. A contrib entry for a
class core already handles is ignored, a class claimed by two contribs
is dropped from both since ServiceLoader order is unspecified, and a
contrib that throws is skipped. Default builds ship no service file,
so nothing changes for them.

The scan suite's service-file test helper moves into a shared test
utility that both suites use.
Registers a serde for Delta's InterleaveBits through the contrib
expression hook, so the Z-order key for one or two integer inputs runs
in Comet's native kernel instead of falling back. Other forms keep
falling back with a reason, and spark.comet.expression.InterleaveBits.enabled
turns it off.

The new suite checks exact bytes for column and literal inputs, nulls,
signed extrema and generated pairs against Spark under both of Delta's
interleave code paths, across several batches and on an empty table.
It replaces the standalone reference checks under dev/repros/delta.

The Delta user guide gains a section on Z-order keys, and the scan
flag's description now says only scans are left to Spark by default.
Arrow builders leave zero under a null slot, so the existing null tests
passed even without the kernel's null check. The new test puts -1 under
a null and expects the bytes of a zero input.
@dwsmith1983

Copy link
Copy Markdown
Contributor

@ErikBPF pushed the Delta serde and the expression suite (dcbbdec). Until the two pieces underneath merge, this diff also carries them:

After those merge and this branch merges main, what's left here is the serde, the suite and a "Z-Order Keys" section in the Delta user guide. I removed the probe under dev/repros/delta because each of its checks now has a test in CometDeltaInterleaveBitsSuite, under both Delta paths; bring it back if you'd rather keep it as a baseline.

It's yours from here for cleanup and extending. Open items I left:

  • The OPTIMIZE test checks zOrderBy by substring; an exact match on ["a","b"] would be tighter.
  • delta.md says the OPTIMIZE path "does not benefit yet"; the docs style guide keeps plans out of the user guide.
  • The empty-table test writes no batches, so the kernel's zero-length-batch path isn't exercised.
  • No benchmark against Spark yet. It would also tell us whether a faster bit-spread interleave is worth adding to the kernel.

@ErikBPF ErikBPF changed the title feat(delta): prepare InterleaveBits integration feat(delta): run Delta InterleaveBits natively Nov 11, 2026
@dwsmith1983

Copy link
Copy Markdown
Contributor

@ErikBPF the core hook is up as draft #7686, the same CometExpressionContrib change this branch already carries, with current main merged in. No one answered the design question in #7573, so I went ahead with the ServiceLoader interface. The order is #7600 and #7686 in either order, then this PR. The description here still says the hook awaits maintainer agreement, so you could swap that line for #7686.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants