Repository navigation
Conversation
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
June 21, 2026 14:14
38e2312 to
7c1773f
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
June 21, 2026 14:14
ae3ec8d to
2dc47b1
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
June 29, 2026 13:29
7c1773f to
0fab506
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
June 29, 2026 13:29
2dc47b1 to
dd9a0d8
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 16, 2026 01:28
0fab506 to
4295269
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 16, 2026 01:28
dd9a0d8 to
ea990f9
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 18, 2026 15:16
4295269 to
1749771
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 15:16
ea990f9 to
8ce571c
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 18, 2026 18:09
1749771 to
f5ea150
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 18:09
8ce571c to
01961d4
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 18, 2026 19:44
f5ea150 to
18dc0af
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 18, 2026 19:44
01961d4 to
3844041
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 19, 2026 03:12
18dc0af to
0910217
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 19, 2026 03:12
3844041 to
5429c62
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 19, 2026 10:59
0910217 to
ccc38f4
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
2 times, most recently
from
August 20, 2026 14:31
c573e20 to
e24a437
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 20, 2026 14:31
ccc38f4 to
03c0391
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 20, 2026 14:51
e24a437 to
ff99360
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 20, 2026 16:58
b3e4c5d to
98ee61f
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 20, 2026 16:58
ff99360 to
0684389
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 20, 2026 17:47
98ee61f to
fd30e9e
Compare
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
2 times, most recently
from
August 20, 2026 20:01
b266c44 to
1c6c838
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 20, 2026 20:01
fd30e9e to
02a4c10
Compare
…lit, part 4a] Part 4a of the Delta Lake contrib PR breakup (tracking: apache#4366). The JVM claim/decline layer: `DeltaScanRule` recognises a V1 Delta scan and plants a `CometDeltaScanMarker` wrapping the original scan. This "activates" the reflective path A.2 already wired (`DeltaIntegration` looks up `DeltaScanRule$`), but stays INERT end to end: with no serde yet (`CometDeltaNativeScan`, part 4b), `CometExecRule`'s `scanHandler` lookup returns None, so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on this build: Delta reads still run on vanilla Spark. - `DeltaScanRule.scala` - claim/decline rule (declines input_file_name, encryption, etc.; plants the marker otherwise). - `CometDeltaScanMarker.scala` - leaf exec node wrapping the original `FileSourceScanExec`; `doExecute` delegates to it (the vanilla fallback). FQN matches A.2's `DeltaIntegration. MarkerClass` string. - `DeltaScanMetadata.scala` - planning info carried on the marker; now also home to the `ScanImpl` constant (moved off the not-yet-present serde so the rule can name it). - `DeltaReflection.scala` - reflective Delta accessors (CDF members inert until part 5). - `RowTrackingAugmentedFileIndex.scala` - row-tracking file index used by the rule. - Tests: `CometDeltaTestBase` (trimmed of the serde/exec-dependent native-read helpers, which move to 4b; added marker-claim helpers) + a new `CometDeltaMarkerSuite` asserting the marker is planted on a plain read, the fallback is result-correct, and a declined projection plants no marker. - `dev/ci/check-suites.py` - exempt contrib test suites (they run under the dedicated Delta workflow, not the default pr_build matrix). No core / A.2 edits needed -- the reflective wiring already reaches this rule + marker. Required edit per the split plan: `CometDeltaNativeScan.ScanImpl` -> `DeltaScanMetadata.ScanImpl` (4b re-points the serde at it). Verification: gated JVM test-compile, `CometDeltaMarkerSuite` 3/3 (marker planted is red on the A.2 build / green here), check-suites, spotless + scalastyle, gate-verify (default build still 0 Delta symbols, only the DeltaIntegration bridge class) -- all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…P0+themeB DeltaReflection, folded into A.4a]
schenksj
force-pushed
the
pr/delta-A4a-scala-claim
branch
from
August 21, 2026 01:33
1c6c838 to
e758983
Compare
schenksj
force-pushed
the
pr/delta-A3b-rust-executor
branch
from
August 21, 2026 01:33
02a4c10 to
250e69e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this part is
The JVM claim/decline layer.
DeltaScanRulerecognises a V1 Delta scan and plants aCometDeltaScanMarkerwrapping the original scan. This "activates" the reflective path part 2 already wired (DeltaIntegrationlooks upDeltaScanRule$), but stays inert end to end: with no serde yet (part 4b),CometExecRule'sscanHandlerlookup returnsNone, so the marker is left in the plan and executes as a vanilla Delta fallback. Net behavior on this build: Delta reads still run on vanilla Spark — so this is safe to land before the read path exists.DeltaScanRule— claim/decline (declinesinput_file_name, encryption, …; plants the marker otherwise).CometDeltaScanMarker— leaf exec wrapping the originalFileSourceScanExec;doExecutedelegates to it (the fallback). FQN matches part 2'sDeltaIntegration.MarkerClass.DeltaScanMetadata— planning info on the marker; now also home to theScanImplconstant (moved off the not-yet-present serde so the rule can name it; 4b re-points its serde here).DeltaReflection(CDF members inert until part 5),RowTrackingAugmentedFileIndex.CometDeltaTestBase(claim/marker helpers; the native-read helpers move to 4b) + a newCometDeltaMarkerSuite.dev/ci/check-suites.py— exempt contrib suites (they run under the dedicated Delta workflow).Why it's contained
No core / part-2 edits — the reflective wiring already reaches this rule + marker. A.4a touches only
contrib/delta/src+check-suites.py. The contrib Scala compiles only under-Pcontrib-delta; default builds carry zero Delta surface (gate-verify confirms 0 Delta symbols, only theDeltaIntegrationbridge class).Verification
gated JVM test-compile,
CometDeltaMarkerSuite3/3 (marker planted on a plain read — red on a part-2-only build, green here; fallback result-correct;input_file_name()declines with no marker), check-suites, spotless + scalastyle, and the gate-verify script — all green.🤖 AI disclosure: this PR was prepared with assistance from Claude Code (Claude Opus 4.8), under the submitter's review and direction.