Skip to content

feat: contrib Delta JVM claim/decline + scan marker [Delta contrib split, part 4a] - #7

Draft
schenksj wants to merge 2 commits into
pr/delta-A3b-rust-executorfrom
pr/delta-A4a-scala-claim
Draft

schenksj wants to merge 2 commits into
pr/delta-A3b-rust-executorfrom
pr/delta-A4a-scala-claim

Conversation

@schenksj

Copy link
Copy Markdown
Owner

Fork-local review draft (Delta-contrib PR split, part 4a). Base is pr/delta-A3b-rust-executor so the diff shows only A.4a. First of the JVM units; stacks on parts 1 (apache#4700), 2, 3a, 3b. Tracking umbrella: apache#4366.

What this part is

The JVM claim/decline layer. DeltaScanRule recognises a V1 Delta scan and plants a CometDeltaScanMarker wrapping the original scan. This "activates" the reflective path part 2 already wired (DeltaIntegration looks up DeltaScanRule$), but stays inert end to end: with no serde yet (part 4b), CometExecRule's scanHandler lookup returns None, so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on this build: Delta reads still run on vanilla Spark — so this is safe to land before the read path exists.

  • DeltaScanRule — claim/decline (declines input_file_name, encryption, …; plants the marker otherwise).
  • CometDeltaScanMarker — leaf exec wrapping the original FileSourceScanExec; doExecute delegates to it (the fallback). FQN matches part 2's DeltaIntegration.MarkerClass.
  • DeltaScanMetadata — planning info on the marker; now also home to the ScanImpl constant (moved off the not-yet-present serde so the rule can name it; 4b re-points its serde here).
  • DeltaReflection (CDF members inert until part 5), RowTrackingAugmentedFileIndex.
  • Tests: CometDeltaTestBase (claim/marker helpers; the native-read helpers move to 4b) + a new CometDeltaMarkerSuite.
  • dev/ci/check-suites.py — exempt contrib suites (they run under the dedicated Delta workflow).

Why it's contained

No core / part-2 edits — the reflective wiring already reaches this rule + marker. A.4a touches only contrib/delta/src + check-suites.py. The contrib Scala compiles only under -Pcontrib-delta; default builds carry zero Delta surface (gate-verify confirms 0 Delta symbols, only the DeltaIntegration bridge class).

Verification

gated JVM test-compile, CometDeltaMarkerSuite 3/3 (marker planted on a plain read — red on a part-2-only build, green here; fallback result-correct; input_file_name() declines with no marker), check-suites, spotless + scalastyle, and the gate-verify script — all green.


🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.

@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 38e2312 to 7c1773f Compare June 21, 2026 14:14
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from ae3ec8d to 2dc47b1 Compare June 21, 2026 14:14
schenksj added a commit that referenced this pull request Jun 21, 2026
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 7c1773f to 0fab506 Compare June 29, 2026 13:29
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 2dc47b1 to dd9a0d8 Compare June 29, 2026 13:29
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 0fab506 to 4295269 Compare August 16, 2026 01:28
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from dd9a0d8 to ea990f9 Compare August 16, 2026 01:28
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 4295269 to 1749771 Compare August 18, 2026 15:16
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from ea990f9 to 8ce571c Compare August 18, 2026 15:16
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 1749771 to f5ea150 Compare August 18, 2026 18:09
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 8ce571c to 01961d4 Compare August 18, 2026 18:09
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from f5ea150 to 18dc0af Compare August 18, 2026 19:44
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 01961d4 to 3844041 Compare August 18, 2026 19:44
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 18dc0af to 0910217 Compare August 19, 2026 03:12
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 3844041 to 5429c62 Compare August 19, 2026 03:12
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 0910217 to ccc38f4 Compare August 19, 2026 10:59
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch 2 times, most recently from c573e20 to e24a437 Compare August 20, 2026 14:31
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from ccc38f4 to 03c0391 Compare August 20, 2026 14:31
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from e24a437 to ff99360 Compare August 20, 2026 14:51
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from b3e4c5d to 98ee61f Compare August 20, 2026 16:58
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from ff99360 to 0684389 Compare August 20, 2026 16:58
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 98ee61f to fd30e9e Compare August 20, 2026 17:47
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch 2 times, most recently from b266c44 to 1c6c838 Compare August 20, 2026 20:01
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from fd30e9e to 02a4c10 Compare August 20, 2026 20:01
schenksj and others added 2 commits August 20, 2026 21:33
…lit, part 4a]

Part 4a of the Delta Lake contrib PR breakup (tracking: apache#4366). The JVM claim/decline
layer: `DeltaScanRule` recognises a V1 Delta scan and plants a `CometDeltaScanMarker`
wrapping the original scan. This "activates" the reflective path A.2 already wired
(`DeltaIntegration` looks up `DeltaScanRule$`), but stays INERT end to end: with no serde
yet (`CometDeltaNativeScan`, part 4b), `CometExecRule`'s `scanHandler` lookup returns None,
so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on
this build: Delta reads still run on vanilla Spark.

- `DeltaScanRule.scala` - claim/decline rule (declines input_file_name, encryption, etc.;
  plants the marker otherwise).
- `CometDeltaScanMarker.scala` - leaf exec node wrapping the original `FileSourceScanExec`;
  `doExecute` delegates to it (the vanilla fallback). FQN matches A.2's `DeltaIntegration.
  MarkerClass` string.
- `DeltaScanMetadata.scala` - planning info carried on the marker; now also home to the
  `ScanImpl` constant (moved off the not-yet-present serde so the rule can name it).
- `DeltaReflection.scala` - reflective Delta accessors (CDF members inert until part 5).
- `RowTrackingAugmentedFileIndex.scala` - row-tracking file index used by the rule.
- Tests: `CometDeltaTestBase` (trimmed of the serde/exec-dependent native-read helpers, which
  move to 4b; added marker-claim helpers) + a new `CometDeltaMarkerSuite` asserting the marker
  is planted on a plain read, the fallback is result-correct, and a declined projection plants
  no marker.
- `dev/ci/check-suites.py` - exempt contrib test suites (they run under the dedicated Delta
  workflow, not the default pr_build matrix).

No core / A.2 edits needed -- the reflective wiring already reaches this rule + marker.
Required edit per the split plan: `CometDeltaNativeScan.ScanImpl` -> `DeltaScanMetadata.ScanImpl`
(4b re-points the serde at it).

Verification: gated JVM test-compile, `CometDeltaMarkerSuite` 3/3 (marker planted is red on the
A.2 build / green here), check-suites, spotless + scalastyle, gate-verify (default build still
0 Delta symbols, only the DeltaIntegration bridge class) -- all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…P0+themeB DeltaReflection, folded into A.4a]
@schenksj
schenksj force-pushed the pr/delta-A4a-scala-claim branch from 1c6c838 to e758983 Compare August 21, 2026 01:33
@schenksj
schenksj force-pushed the pr/delta-A3b-rust-executor branch from 02a4c10 to 250e69e Compare August 21, 2026 01:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant