Skip to content

refactor: locate FileSourceScanExec filter arguments through a shim - #7673

Open
andygrove wants to merge 6 commits into
apache:mainfrom
andygrove:scan-filter-shim
Open

andygrove wants to merge 6 commits into
apache:mainfrom
andygrove:scan-filter-shim

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue. Follow-up to #7508.

Stacked on #7508. Until #7508 merges, this branch also carries its commits. Only the last commit (refactor: locate FileSourceScanExec filter arguments through a shim) is new here. I'll rebase onto main once #7508 lands.

Rationale for this change

#7508 rebuilds a FileSourceScanExec through makeCopy so that a runtime with extra constructor fields, such as Amazon EMR, keeps them. To find the filter arguments, it searches the constructor arguments for requiredSchema by identity and writes at offsets +1 and +4 from it. That handles the extra stream argument in Spark 4 at runtime. Comet already splits Spark 3 and Spark 4 differences into the spark-3.x and spark-4.x source sets, so that is where this layout difference belongs.

The offsets are also written without checking what is there. So if a future Spark or a vendor runtime inserts a field, the scan is rebuilt with an argument in the wrong position instead of failing.

What changes are included in this PR?

  • New ShimFileSourceScanExec objects in spark-3.x and spark-4.x give the positions of partitionFilters and dataFilters in the FileSourceScanExec constructor. They are 3 and 6 on Spark 3.5, and 4 and 7 on Spark 4.0, 4.1 and 4.2. I checked these against the Spark 3.4.3, 4.1.1 sources and the 4.2.0 jar.
  • CometScanUtils.copyScanWithFilters uses those positions. Before replacing each slot, it checks that the slot holds the scan's current filter list, and it throws an IllegalStateException that names the unexpected layout if not.
  • The makeCopy path from fix: preserve runtime fields when copying file scan filters #7508 is unchanged, so fields that a runtime appends are still carried through.

How are these changes tested?

A new test in CometPlanEqualitySuite checks that the shim positions hold the scan's filter lists on whichever Spark profile it runs. The existing #7508 tests cover the fallback, make_date and canonicalization paths.

Locally, CometPlanEqualitySuite and CometExecRuleSuite pass on the default Spark 4.1 profile and on -Pspark-3.5. Main and test sources compile on -Pspark-4.2.

🤖 Generated with Claude Code

danielwei816 and others added 6 commits November 16, 2026 15:20
copyScanWithFilters found the filters by searching the constructor
arguments for requiredSchema and writing at fixed offsets from it. Move
the Spark 3 versus Spark 4 layout difference into a
ShimFileSourceScanExec in the spark-3.x and spark-4.x source sets, and
check that each slot holds the scan's current filter list before
replacing it, so an unexpected layout fails with a clear error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added enhancement New feature or request area:scan Parquet scan / data reading labels Nov 17, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constructor positions match Spark 3.5.8 (3, 6) and 4.0.1 and 4.1.1 (4, 7). One point on the new test.

Comment on lines +43 to +53
test("FileSourceScanExec shim filter indices match the constructor") {
val output = Seq(AttributeReference("id", IntegerType)())
val scan = ExtendedFileSourceScanExec(
output,
partitionFilters = Seq(GreaterThan(output.head, Literal(1))),
dataFilters = Seq(LessThan(output.head, Literal(10))))

assert(
scan.productElement(ShimFileSourceScanExec.partitionFiltersIndex) == scan.partitionFilters)
assert(scan.productElement(ShimFileSourceScanExec.dataFiltersIndex) == scan.dataFilters)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test pins what scan canonicalization preserves runtime fields and Spark normalization already pins. That test builds ExtendedFileSourceScanExec with non-empty partition and data filters and goes through canonicalizeOriginalPlan, which calls copyScanWithFilters. A wrong shim index makes replaceArg throw IllegalStateException there. Could this one be removed?

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have a look into findings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants