Repository navigation
Conversation
copyScanWithFilters found the filters by searching the constructor arguments for requiredSchema and writing at fixed offsets from it. Move the Spark 3 versus Spark 4 layout difference into a ShimFileSourceScanExec in the spark-3.x and spark-4.x source sets, and check that each slot holds the scan's current filter list before replacing it, so an unexpected layout fails with a clear error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
comphead
reviewed
Nov 18, 2026
comphead
left a comment
Contributor
There was a problem hiding this comment.
The constructor positions match Spark 3.5.8 (3, 6) and 4.0.1 and 4.1.1 (4, 7). One point on the new test.
Comment on lines
+43
to
+53
| test("FileSourceScanExec shim filter indices match the constructor") { | ||
| val output = Seq(AttributeReference("id", IntegerType)()) | ||
| val scan = ExtendedFileSourceScanExec( | ||
| output, | ||
| partitionFilters = Seq(GreaterThan(output.head, Literal(1))), | ||
| dataFilters = Seq(LessThan(output.head, Literal(10)))) | ||
|
|
||
| assert( | ||
| scan.productElement(ShimFileSourceScanExec.partitionFiltersIndex) == scan.partitionFilters) | ||
| assert(scan.productElement(ShimFileSourceScanExec.dataFiltersIndex) == scan.dataFilters) | ||
| } |
Contributor
There was a problem hiding this comment.
This test pins what scan canonicalization preserves runtime fields and Spark normalization already pins. That test builds ExtendedFileSourceScanExec with non-empty partition and data filters and goes through canonicalizeOriginalPlan, which calls copyScanWithFilters. A wrong shim index makes replaceArg throw IllegalStateException there. Could this one be removed?
comphead
approved these changes
Nov 18, 2026
comphead
left a comment
Contributor
There was a problem hiding this comment.
Please have a look into findings
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue. Follow-up to #7508.
Stacked on #7508. Until #7508 merges, this branch also carries its commits. Only the last commit (
refactor: locate FileSourceScanExec filter arguments through a shim) is new here. I'll rebase ontomainonce #7508 lands.Rationale for this change
#7508 rebuilds a
FileSourceScanExecthroughmakeCopyso that a runtime with extra constructor fields, such as Amazon EMR, keeps them. To find the filter arguments, it searches the constructor arguments forrequiredSchemaby identity and writes at offsets +1 and +4 from it. That handles the extrastreamargument in Spark 4 at runtime. Comet already splits Spark 3 and Spark 4 differences into thespark-3.xandspark-4.xsource sets, so that is where this layout difference belongs.The offsets are also written without checking what is there. So if a future Spark or a vendor runtime inserts a field, the scan is rebuilt with an argument in the wrong position instead of failing.
What changes are included in this PR?
ShimFileSourceScanExecobjects inspark-3.xandspark-4.xgive the positions ofpartitionFiltersanddataFiltersin theFileSourceScanExecconstructor. They are 3 and 6 on Spark 3.5, and 4 and 7 on Spark 4.0, 4.1 and 4.2. I checked these against the Spark 3.4.3, 4.1.1 sources and the 4.2.0 jar.CometScanUtils.copyScanWithFiltersuses those positions. Before replacing each slot, it checks that the slot holds the scan's current filter list, and it throws anIllegalStateExceptionthat names the unexpected layout if not.makeCopypath from fix: preserve runtime fields when copying file scan filters #7508 is unchanged, so fields that a runtime appends are still carried through.How are these changes tested?
A new test in
CometPlanEqualitySuitechecks that the shim positions hold the scan's filter lists on whichever Spark profile it runs. The existing #7508 tests cover the fallback,make_dateand canonicalization paths.Locally,
CometPlanEqualitySuiteandCometExecRuleSuitepass on the default Spark 4.1 profile and on-Pspark-3.5. Main and test sources compile on-Pspark-4.2.🤖 Generated with Claude Code