Skip to content

fix: preserve runtime fields when copying file scan filters - #7508

Open
danielwei816 wants to merge 6 commits into
apache:mainfrom
danielwei816:codex/fix-scan-copy-emr
Open

danielwei816 wants to merge 6 commits into
apache:mainfrom
danielwei816:codex/fix-scan-copy-emr

Conversation

@danielwei816

@danielwei816 danielwei816 commented Nov 5, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7393.

Rationale for this change

We hit a NoSuchMethodError on EMR 8.1.0 when Spark canonicalizes a Comet scan. EMR adds two fields to FileSourceScanExec, so its copy() signature differs from the one Comet was compiled against.

What changes are included in this PR?

We’re reusing the approach from #207, which addressed the same copy() signature problem reported on Databricks. It reads the scan’s actual constructor fields at runtime, replaces the field being changed, and lets Spark’s makeCopy() rebuild the scan. #207 used this to replace the relation; this PR applies it to partition and data filters in canonicalization, native scan fallback, and the make_date planning rule.

All other fields are carried through, including EMR’s extra fields. Stock Spark keeps the same filter and canonicalization behavior. The helper locates the filters relative to the schema to handle the constructor difference between Spark 3 and Spark 4, without hardcoding EMR’s signature.

We’ve checked this on stock Spark 3.5 and 4.1 and the EMR 8.1.0 image. The approach assumes the existing relative order of the schema and filter fields; other constructor or API changes in a vendor runtime would still need validation.

How are these changes tested?

  • Regression tests cover extra scan fields in native and make_date fallback, plus canonicalization with an extended scan. The new make_date regression fails with the old copy() call.
  • Focused Spark 4.1 suites: 56 tests passed, including date-handling and partition-pruning tests. Spark 3.5 with strict warnings: 5 tests passed. The Spark 4.2-only test was skipped on both.
  • Spark 4.1 SQL data-source shard (dev/local-ci.sh spark sql_core-4): 5,430 tests passed, 0 failed.
  • Public ARM64 EMR 8.1.0 image: verified the current canonicalization and make_date code preserve the real 12-field scan. The original caching reproducer also passed with the copy fix applied to Comet 1.1.0.

@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Nov 5, 2026
@danielwei816

Copy link
Copy Markdown
Author

Hi @kazuyukitanimura and @viirya, could you take a look at this? It follows the makeCopy() approach from #207, with validation on Spark 3.5, Spark 4.1, and the public EMR 8.1.0 image. Thanks!

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @danielwei816, reading the filters' positions from the live constructor is a reasonable way to survive a vendor FileSourceScanExec.

One call site is still left on the compiled copy signature. MakeDateEvaluation.preserve builds the Spark form of a CometScanExec with scan.wrapped.copy(partitionFilters = scan.partitionFilters, dataFilters = scan.dataFilters) (MakeDateEvaluation.scala:234). preserve runs from CometExecRule on every plan while spark.comet.exec.makeDate.operatorFallback.enabled is on, which is the default, and it computes that value for every node it visits. So I expect a plan with a CometScanExec to hit the same NoSuchMethodError on EMR. I have not run it there. Could that line use CometScanUtils.copyScanWithFilters(scan.wrapped, scan.partitionFilters, scan.dataFilters) too?

test("scan canonicalization retains Spark normalization and removes original DPP filters") {
val output = Seq(AttributeReference("id", IntegerType)())
val staticFilter = GreaterThan(output.head, Literal(1))
val scan = ExtendedFileSourceScanExec(output).copy(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy on an ExtendedFileSourceScanExec returns a plain FileSourceScanExec, since the case-class copy builds the base class, as the comment in ExtendedFileSourceScanExec says. So this test canonicalizes a stock scan, and copyScanWithFilters never sees the extra fields on this path. Could ExtendedFileSourceScanExec.apply take the partition and data filters, so the scan here is the extended one?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks fixed both! MakeDateEvaluation uses the shared helper now and the canonicalization test constructs the extended scan with its filters directly.

@danielwei816
danielwei816 requested a review from comphead November 6, 2026 00:06

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

Using the live constructor through makeCopy is a sensible way to survive a vendor FileSourceScanExec, and the MakeDateEvaluation call site comphead pointed out is now covered. I checked the offsets in CometScanUtils.copyScanWithFilters against the FileSourceScanExec constructors in the Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 and 4.2.0 jars. The schema is at index 2 on 3.x and 3 on 4.x, and partitionFilters and dataFilters sit at +1 and +4 from it on every one of them. I also compared against branch-1.2. The three call sites use the same copy pattern there, so on stock Spark the new helper should give the same scans, filters and canonical form as the release.

The one thing I would address is the canonicalization test, scan canonicalization retains Spark normalization and removes original DPP filters in CometPlanEqualitySuite. The assertion compares against scan.copy(partitionFilters = ...).doCanonicalize(), and Spark's FileSourceScanExec.doCanonicalize builds a plain FileSourceScanExec in both cases. That means the result drops the extra fields whether or not copyScanWithFilters preserved them, so I expect this test to also pass with the old originalPlan.copy(...) call in canonicalizeOriginalPlan. The PR description says the canonicalization path is covered by an extended scan, but the extended fields cannot be observed through it today. Could ExtendedFileSourceScanExec override doCanonicalize to rebuild itself (for example through makeCopy with the normalized arguments), so the test fails if the copy step loses vendorFlag and vendorOutput? Alternatively, a direct test of copyScanWithFilters that checks the result class and the last two fields would pin the behavior the fix is about.

I did not find any other FileSourceScanExec.copy( calls left in spark/src/main. The remaining .copy( calls on scans are on Comet's own case classes. No blocking concerns otherwise.

@danielwei816

Copy link
Copy Markdown
Author

@andygrove appreciate the review! I fixed the test so it actually catches the vendor fields getting dropped during canonicalization and I confirmed it fails with the old copy() call and passes with the fix on Spark 3.5 and 4.1.

@comphead looping you back in too. Would appreciate another look from either of you when you get a chance!

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @danielwei816, overriding doCanonicalize in the test scan is a nice way to make the vendor fields observable, and checking the class and the last two fields pins exactly what the fix is about. One thing before merge: CI (fork) reports that Actions aren't enabled on your fork, so no CI has run on any commit here. Could you enable them and push again? Canonicalization sits under plan reuse for every scan, so I'd like to see the suites green.

@danielwei816
danielwei816 force-pushed the codex/fix-scan-copy-emr branch from d5eb92a to ea4a9d8 Compare November 16, 2026 23:21
@danielwei816

Copy link
Copy Markdown
Author

Thanks @andygrove! Fork CI is green now. My scan fix is unchanged since your approval; I rebased onto main to pick up an existing Rust CI fix.

Could you approve the pending workflows on the Apache repo when you get a chance? Appreciate the backport label too!

Comment on lines +42 to +43
// Spark 4 adds a stream argument before the schema; the filters keep the same offsets
// after it. Locate the existing schema rather than assuming a Spark-specific arity.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably have handled this in the shim layer

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will create follow on PR

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier points are addressed. MakeDateEvaluation now goes through copyScanWithFilters, and the canonicalization test builds the extended scan so the vendor fields are observable. One behavior difference from the old copy call is noted inline.

val restoredScan =
originalPlan.copy(partitionFilters = partitionFilters, dataFilters = dataFilters)
CometScanUtils.copyScanWithFilters(originalPlan, partitionFilters, dataFilters)
originalPlan.logicalLink.foreach(restoredScan.setLogicalLink)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy did not carry tags to the new node, but makeCopy ends with res.copyTagsFrom(this) in Spark 3.5.8 and 4.1.1. So restoredScan now starts with originalPlan's tags, including the logical plan link, and this setLogicalLink call has nothing left to do. Can it be dropped?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I removed the extra setLogicalLink call and verified that makeCopy preserves the logical link when Comet falls back to Spark.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 18, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, great PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Comet 1.1.0: FileSourceScanExec.copy NoSuchMethodError on EMR Spark 4.1.1

3 participants