Skip to content

feat: contrib Delta serde + native exec — end-to-end native reads [Delta contrib split, part 4b] - #8

Draft
schenksj wants to merge 3 commits into
pr/delta-A4a-scala-claimfrom
pr/delta-A4b-scala-exec
Draft

schenksj wants to merge 3 commits into
pr/delta-A4a-scala-claimfrom
pr/delta-A4b-scala-exec

Conversation

@schenksj

Copy link
Copy Markdown
Owner

Fork-local review draft (Delta-contrib PR split, part 4b). Base is pr/delta-A4a-scala-claim so the diff shows only A.4b. The red-to-green unit; stacks on parts 1 (apache#4700), 2, 3a, 3b, 4a. Tracking umbrella: apache#4366.

What this part is

The serde + native exec — the moment a -Pcontrib-delta build does end-to-end native Delta reads. CometExecRule's scanHandler lookup (wired in part 2) now resolves: the serde converts the CometDeltaScanMarker (planted by part 4a's DeltaScanRule) into a CometDeltaNativeScanExec that reads through delta-kernel-rs (parts 3a/3b).

  • CometDeltaNativeScan — serde (marker → native scan operator: schema annotation, column mapping, row tracking, partitioning). The CDF path (convertCdf) is deferred to part 5.
  • CometDeltaNativeScanExec — the exec (CometScanWithPlanData): file partitions from kernel tasks + DPP pruning. Interim error semantics until part 8 (a read failure surfaces as a generic CometNativeException; the FAILED_READ_FILE path lands later).
  • Native.scala — JNI declarations binding the part-3a Rust entry points.
  • DeltaPlanDataInjector — registers under OpStruct::DELTA_SCAN; part 1's reflective registry picks it up for per-partition injection.

Why it's contained

No core / earlier-unit edits — the reflective wiring already reaches the serde + injector the moment these classes land. A.4b touches only contrib/delta/src. Default builds still carry zero Delta surface (gate-verify confirms 0 Delta symbols).

Verification

gated JVM test-compile, 60 contrib tests across 6 suites (CometDeltaNativeSuite 19, ColumnMapping 5, Features 8, Coverage 24, PhysicalNameRepro 1, Marker 3) — end-to-end native reads — spotless/scalastyle, check-suites, and the gate-verify script — all green.


🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.

schenksj added a commit that referenced this pull request Jun 21, 2026
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 2dc47b1 to dd9a0d8 Compare June 29, 2026 13:29
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 8d14cdb to 0ee301f Compare June 29, 2026 13:29
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from dd9a0d8 to ea990f9 Compare August 16, 2026 01:28
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 0ee301f to ebe0c4f Compare August 16, 2026 01:28
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from ea990f9 to 8ce571c Compare August 18, 2026 15:16
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from ebe0c4f to 36298c6 Compare August 18, 2026 15:16
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 8ce571c to 01961d4 Compare August 18, 2026 18:09
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 36298c6 to c5c2ed1 Compare August 18, 2026 18:09
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 01961d4 to 3844041 Compare August 18, 2026 19:44
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from c5c2ed1 to f4ba3c7 Compare August 18, 2026 19:44
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 3844041 to 5429c62 Compare August 19, 2026 03:12
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from f4ba3c7 to 33898ea Compare August 19, 2026 03:12
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 5429c62 to c573e20 Compare August 19, 2026 10:59
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 33898ea to 54e37a6 Compare August 19, 2026 10:59
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from c573e20 to e24a437 Compare August 20, 2026 14:31
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 54e37a6 to cd36d7c Compare August 20, 2026 14:31
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from e24a437 to ff99360 Compare August 20, 2026 14:51
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch 2 times, most recently from b788dfd to 6de9be6 Compare August 20, 2026 16:58
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch 2 times, most recently from 0684389 to b266c44 Compare August 20, 2026 17:47
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 6de9be6 to 224d992 Compare August 20, 2026 17:47
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from b266c44 to 1c6c838 Compare August 20, 2026 20:01
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 224d992 to 403300d Compare August 20, 2026 20:01
schenksj and others added 3 commits August 20, 2026 21:33
…lta contrib split, part 4b]

Part 4b of the Delta Lake contrib PR breakup (tracking: apache#4366). The red-to-green moment: a
`-Pcontrib-delta` build now does END-TO-END native Delta reads. `CometExecRule`'s scanHandler
lookup (wired in part 2) now resolves -- the serde converts the `CometDeltaScanMarker` (planted
by part 4a's DeltaScanRule) into a `CometDeltaNativeScanExec` that reads through delta-kernel-rs
(parts 3a/3b).

- `CometDeltaNativeScan.scala` — the serde: marker -> native scan operator (schema annotation,
  column mapping, row tracking, partition handling). CDF conversion is deferred to part 5 (the
  `convertCdf` path is carved out here to avoid a compile dependency on `CometDeltaCdfScanExec`).
  `ScanImpl` is not redefined — part 4a moved it to `DeltaScanMetadata`.
- `CometDeltaNativeScanExec.scala` — the exec (`CometScanWithPlanData`): synthesises file
  partitions from kernel scan tasks, applies DPP pruning. Interim error semantics (until part 8):
  the `perPartitionFilePaths` / `FAILED_READ_FILE` plumbing is omitted, so a Delta read failure
  surfaces as a generic `CometNativeException` (the `CometExecRDD` param defaults to empty).
- `Native.scala` — JNI declarations binding the part-3a Rust entry points.
- `DeltaPlanDataInjector.scala` — registers under `OpStruct::DELTA_SCAN`; part 1's reflective
  registry picks it up, so per-partition Delta data is injected at execution.

No core / earlier-unit edits — the reflective wiring already reaches the serde + injector the
moment these classes land.

Tests (gated, end-to-end native reads): CometDeltaNativeSuite (19), CometDeltaColumnMappingSuite (5),
CometDeltaFeaturesSuite (8), CometDeltaCoverageSuite (24), CometDeltaColumnMappingPhysicalNameReproSuite
(1) — all pass. CometDeltaTestBase re-gains the native-read helpers (kept part 4a's marker helpers
that are still used). CometDeltaMarkerSuite updated: with the serde present, a claimed scan now
engages `CometDeltaNativeScanExec` (it no longer leaves the marker in the plan), so its assertions
moved from marker-presence to native engagement.

Verification: gated JVM test-compile, 60 contrib tests across 6 suites, spotless/scalastyle,
check-suites, gate-verify (default build still 0 Delta symbols) — all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… RowId metadata column

A Delta table may legitimately have a user column named `row_id`. When row
tracking is disabled that column is ordinary data, and `dataReadSchemaJson`
already says so -- its strip list is gated on `rowTrackingActive`, and
`convert`'s `emitRowId` flag is too.

`synthesizeReadSchemaJson` was not. Its `isRowId` helper matched the bare name
`row_id` unconditionally and stamped kernel's `delta.metadataSpec = row_id` on
the field, so the projected read schema shipped to delta-kernel was:

  {"name":"row_id","type":"long","metadata":{"delta.metadataSpec":"row_id"}}

delta-kernel's `validate_metadata_columns` rejects a RowId metadata column when
`enable_row_tracking != true`, failing log replay with "Row ids are not enabled
on this table". Every read of such a table lost the native path. Gate `isRowId`
on `rowTrackingActive`, threaded in from `convert`'s existing
`rowTrackingEnabled`; the `_row-id-col-*` materialised names only exist when row
tracking is on, so gating both is safe.

Two declines also became visible rather than silent. Core's
`reportUnexplainedFallback` (apache#5236) now fails the query
under `spark.comet.strictFallbackReasons` when an operator is left unconverted
with no recorded reason, and both of these declined with only a log line:

  - the delta-kernel log-replay failure above, and
  - the deletion-vector maintenance read that requests both
    `__delta_internal_row_index` and `_tmp_metadata_row_index`
    (`useMetadataRowIndex=false`), which is a correct decline.

Both now call `withFallbackReason`. That is what kept the row-id bug hidden: a
log-only decline never reaches `EXPLAIN`, so the scan silently fell back.

Repros: CometDeltaRowIdColumnCollisionReproSuite (row-id collision, now reads
natively), CometDeltaDeleteWithDVReproSuite (DV decline, now explained).
Contrib battery on Spark 3.5 / delta 3.3.2: 165 passed, 0 failed, 33 suites
(was 163 passed / 2 failed).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 1c6c838 to e758983 Compare August 21, 2026 01:33
@schenksj
schenksj force-pushed the pr/delta-A4b-scala-exec branch from 403300d to 26ffb81 Compare August 21, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant