Repository navigation
Conversation
schenksj
added a commit
that referenced
this pull request
Jun 21, 2026
…ed (end-to-end native reads, fork #8)
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
June 29, 2026 13:29
2dc47b1 to
dd9a0d8
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
June 29, 2026 13:29
8d14cdb to
0ee301f
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 16, 2026 01:28
dd9a0d8 to
ea990f9
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 16, 2026 01:28
0ee301f to
ebe0c4f
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 15:16
ea990f9 to
8ce571c
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 18, 2026 15:16
ebe0c4f to
36298c6
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 18:09
8ce571c to
01961d4
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 18, 2026 18:09
36298c6 to
c5c2ed1
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 19:44
01961d4 to
3844041
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 18, 2026 19:44
c5c2ed1 to
f4ba3c7
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 19, 2026 03:12
3844041 to
5429c62
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 19, 2026 03:12
f4ba3c7 to
33898ea
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 19, 2026 10:59
5429c62 to
c573e20
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 19, 2026 10:59
33898ea to
54e37a6
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 20, 2026 14:31
c573e20 to
e24a437
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 20, 2026 14:31
54e37a6 to
cd36d7c
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 20, 2026 14:51
e24a437 to
ff99360
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
2 times, most recently
from
August 20, 2026 16:58
b788dfd to
6de9be6
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
2 times, most recently
from
August 20, 2026 17:47
0684389 to
b266c44
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 20, 2026 17:47
6de9be6 to
224d992
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 20, 2026 20:01
b266c44 to
1c6c838
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 20, 2026 20:01
224d992 to
403300d
Compare
…lta contrib split, part 4b] Part 4b of the Delta Lake contrib PR breakup (tracking: apache#4366). The red-to-green moment: a `-Pcontrib-delta` build now does END-TO-END native Delta reads. `CometExecRule`'s scanHandler lookup (wired in part 2) now resolves -- the serde converts the `CometDeltaScanMarker` (planted by part 4a's DeltaScanRule) into a `CometDeltaNativeScanExec` that reads through delta-kernel-rs (parts 3a/3b). - `CometDeltaNativeScan.scala` — the serde: marker -> native scan operator (schema annotation, column mapping, row tracking, partition handling). CDF conversion is deferred to part 5 (the `convertCdf` path is carved out here to avoid a compile dependency on `CometDeltaCdfScanExec`). `ScanImpl` is not redefined — part 4a moved it to `DeltaScanMetadata`. - `CometDeltaNativeScanExec.scala` — the exec (`CometScanWithPlanData`): synthesises file partitions from kernel scan tasks, applies DPP pruning. Interim error semantics (until part 8): the `perPartitionFilePaths` / `FAILED_READ_FILE` plumbing is omitted, so a Delta read failure surfaces as a generic `CometNativeException` (the `CometExecRDD` param defaults to empty). - `Native.scala` — JNI declarations binding the part-3a Rust entry points. - `DeltaPlanDataInjector.scala` — registers under `OpStruct::DELTA_SCAN`; part 1's reflective registry picks it up, so per-partition Delta data is injected at execution. No core / earlier-unit edits — the reflective wiring already reaches the serde + injector the moment these classes land. Tests (gated, end-to-end native reads): CometDeltaNativeSuite (19), CometDeltaColumnMappingSuite (5), CometDeltaFeaturesSuite (8), CometDeltaCoverageSuite (24), CometDeltaColumnMappingPhysicalNameReproSuite (1) — all pass. CometDeltaTestBase re-gains the native-read helpers (kept part 4a's marker helpers that are still used). CometDeltaMarkerSuite updated: with the serde present, a claimed scan now engages `CometDeltaNativeScanExec` (it no longer leaves the marker in the plan), so its assertions moved from marker-presence to native engagement. Verification: gated JVM test-compile, 60 contrib tests across 6 suites, spotless/scalastyle, check-suites, gate-verify (default build still 0 Delta symbols) — all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lision guard [apache#30 + themeA, folded into A.4b]
… RowId metadata column
A Delta table may legitimately have a user column named `row_id`. When row
tracking is disabled that column is ordinary data, and `dataReadSchemaJson`
already says so -- its strip list is gated on `rowTrackingActive`, and
`convert`'s `emitRowId` flag is too.
`synthesizeReadSchemaJson` was not. Its `isRowId` helper matched the bare name
`row_id` unconditionally and stamped kernel's `delta.metadataSpec = row_id` on
the field, so the projected read schema shipped to delta-kernel was:
{"name":"row_id","type":"long","metadata":{"delta.metadataSpec":"row_id"}}
delta-kernel's `validate_metadata_columns` rejects a RowId metadata column when
`enable_row_tracking != true`, failing log replay with "Row ids are not enabled
on this table". Every read of such a table lost the native path. Gate `isRowId`
on `rowTrackingActive`, threaded in from `convert`'s existing
`rowTrackingEnabled`; the `_row-id-col-*` materialised names only exist when row
tracking is on, so gating both is safe.
Two declines also became visible rather than silent. Core's
`reportUnexplainedFallback` (apache#5236) now fails the query
under `spark.comet.strictFallbackReasons` when an operator is left unconverted
with no recorded reason, and both of these declined with only a log line:
- the delta-kernel log-replay failure above, and
- the deletion-vector maintenance read that requests both
`__delta_internal_row_index` and `_tmp_metadata_row_index`
(`useMetadataRowIndex=false`), which is a correct decline.
Both now call `withFallbackReason`. That is what kept the row-id bug hidden: a
log-only decline never reaches `EXPLAIN`, so the scan silently fell back.
Repros: CometDeltaRowIdColumnCollisionReproSuite (row-id collision, now reads
natively), CometDeltaDeleteWithDVReproSuite (DV decline, now explained).
Contrib battery on Spark 3.5 / delta 3.3.2: 165 passed, 0 failed, 33 suites
(was 163 passed / 2 failed).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 21, 2026 01:33
1c6c838 to
e758983
Compare
schenksj
force-pushed
the
pr/delta-A4b-scala-exec
branch
from
August 21, 2026 01:33
403300d to
26ffb81
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this part is
The serde + native exec — the moment a
-Pcontrib-deltabuild does end-to-end native Delta reads.CometExecRule's scanHandler lookup (wired in part 2) now resolves: the serde converts theCometDeltaScanMarker(planted by part 4a'sDeltaScanRule) into aCometDeltaNativeScanExecthat reads through delta-kernel-rs (parts 3a/3b).CometDeltaNativeScan— serde (marker → native scan operator: schema annotation, column mapping, row tracking, partitioning). The CDF path (convertCdf) is deferred to part 5.CometDeltaNativeScanExec— the exec (CometScanWithPlanData): file partitions from kernel tasks + DPP pruning. Interim error semantics until part 8 (a read failure surfaces as a genericCometNativeException; theFAILED_READ_FILEpath lands later).Native.scala— JNI declarations binding the part-3a Rust entry points.DeltaPlanDataInjector— registers underOpStruct::DELTA_SCAN; part 1's reflective registry picks it up for per-partition injection.Why it's contained
No core / earlier-unit edits — the reflective wiring already reaches the serde + injector the moment these classes land. A.4b touches only
contrib/delta/src. Default builds still carry zero Delta surface (gate-verify confirms 0 Delta symbols).Verification
gated JVM test-compile, 60 contrib tests across 6 suites (
CometDeltaNativeSuite19, ColumnMapping 5, Features 8, Coverage 24, PhysicalNameRepro 1, Marker 3) — end-to-end native reads — spotless/scalastyle, check-suites, and the gate-verify script — all green.🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.