Repository navigation
fix: preserve runtime fields when copying file scan filters - #7508
danielwei816 wants to merge 6 commits into
Conversation
|
Hi @kazuyukitanimura and @viirya, could you take a look at this? It follows the |
comphead
left a comment
There was a problem hiding this comment.
Thanks @danielwei816, reading the filters' positions from the live constructor is a reasonable way to survive a vendor FileSourceScanExec.
One call site is still left on the compiled copy signature. MakeDateEvaluation.preserve builds the Spark form of a CometScanExec with scan.wrapped.copy(partitionFilters = scan.partitionFilters, dataFilters = scan.dataFilters) (MakeDateEvaluation.scala:234). preserve runs from CometExecRule on every plan while spark.comet.exec.makeDate.operatorFallback.enabled is on, which is the default, and it computes that value for every node it visits. So I expect a plan with a CometScanExec to hit the same NoSuchMethodError on EMR. I have not run it there. Could that line use CometScanUtils.copyScanWithFilters(scan.wrapped, scan.partitionFilters, scan.dataFilters) too?
| test("scan canonicalization retains Spark normalization and removes original DPP filters") { | ||
| val output = Seq(AttributeReference("id", IntegerType)()) | ||
| val staticFilter = GreaterThan(output.head, Literal(1)) | ||
| val scan = ExtendedFileSourceScanExec(output).copy( |
There was a problem hiding this comment.
copy on an ExtendedFileSourceScanExec returns a plain FileSourceScanExec, since the case-class copy builds the base class, as the comment in ExtendedFileSourceScanExec says. So this test canonicalizes a stock scan, and copyScanWithFilters never sees the extra fields on this path. Could ExtendedFileSourceScanExec.apply take the partition and data filters, so the scan here is the extended one?
There was a problem hiding this comment.
Thanks fixed both! MakeDateEvaluation uses the shared helper now and the canonicalization test constructs the extended scan with its filters directly.
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Using the live constructor through makeCopy is a sensible way to survive a vendor FileSourceScanExec, and the MakeDateEvaluation call site comphead pointed out is now covered. I checked the offsets in CometScanUtils.copyScanWithFilters against the FileSourceScanExec constructors in the Spark 3.4.3, 3.5.8, 4.0.1, 4.1.1 and 4.2.0 jars. The schema is at index 2 on 3.x and 3 on 4.x, and partitionFilters and dataFilters sit at +1 and +4 from it on every one of them. I also compared against branch-1.2. The three call sites use the same copy pattern there, so on stock Spark the new helper should give the same scans, filters and canonical form as the release.
The one thing I would address is the canonicalization test, scan canonicalization retains Spark normalization and removes original DPP filters in CometPlanEqualitySuite. The assertion compares against scan.copy(partitionFilters = ...).doCanonicalize(), and Spark's FileSourceScanExec.doCanonicalize builds a plain FileSourceScanExec in both cases. That means the result drops the extra fields whether or not copyScanWithFilters preserved them, so I expect this test to also pass with the old originalPlan.copy(...) call in canonicalizeOriginalPlan. The PR description says the canonicalization path is covered by an extended scan, but the extended fields cannot be observed through it today. Could ExtendedFileSourceScanExec override doCanonicalize to rebuild itself (for example through makeCopy with the normalized arguments), so the test fails if the copy step loses vendorFlag and vendorOutput? Alternatively, a direct test of copyScanWithFilters that checks the result class and the last two fields would pin the behavior the fix is about.
I did not find any other FileSourceScanExec.copy( calls left in spark/src/main. The remaining .copy( calls on scans are on Comet's own case classes. No blocking concerns otherwise.
|
@andygrove appreciate the review! I fixed the test so it actually catches the vendor fields getting dropped during canonicalization and I confirmed it fails with the old copy() call and passes with the fix on Spark 3.5 and 4.1. @comphead looping you back in too. Would appreciate another look from either of you when you get a chance! |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @danielwei816, overriding doCanonicalize in the test scan is a nice way to make the vendor fields observable, and checking the class and the last two fields pins exactly what the fix is about. One thing before merge: CI (fork) reports that Actions aren't enabled on your fork, so no CI has run on any commit here. Could you enable them and push again? Canonicalization sits under plan reuse for every scan, so I'd like to see the suites green.
d5eb92a to
ea4a9d8
Compare
|
Thanks @andygrove! Fork CI is green now. My scan fix is unchanged since your approval; I rebased onto main to pick up an existing Rust CI fix. Could you approve the pending workflows on the Apache repo when you get a chance? Appreciate the backport label too! |
| // Spark 4 adds a stream argument before the schema; the filters keep the same offsets | ||
| // after it. Locate the existing schema rather than assuming a Spark-specific arity. |
There was a problem hiding this comment.
We should probably have handled this in the shim layer
comphead
left a comment
There was a problem hiding this comment.
Both earlier points are addressed. MakeDateEvaluation now goes through copyScanWithFilters, and the canonicalization test builds the extended scan so the vendor fields are observable. One behavior difference from the old copy call is noted inline.
| val restoredScan = | ||
| originalPlan.copy(partitionFilters = partitionFilters, dataFilters = dataFilters) | ||
| CometScanUtils.copyScanWithFilters(originalPlan, partitionFilters, dataFilters) | ||
| originalPlan.logicalLink.foreach(restoredScan.setLogicalLink) |
There was a problem hiding this comment.
copy did not carry tags to the new node, but makeCopy ends with res.copyTagsFrom(this) in Spark 3.5.8 and 4.1.1. So restoredScan now starts with originalPlan's tags, including the logical plan link, and this setLogicalLink call has nothing left to do. Can it be dropped?
There was a problem hiding this comment.
Thanks! I removed the extra setLogicalLink call and verified that makeCopy preserves the logical link when Comet falls back to Spark.
Which issue does this PR close?
Closes #7393.
Rationale for this change
We hit a
NoSuchMethodErroron EMR 8.1.0 when Spark canonicalizes a Comet scan. EMR adds two fields toFileSourceScanExec, so itscopy()signature differs from the one Comet was compiled against.What changes are included in this PR?
We’re reusing the approach from #207, which addressed the same
copy()signature problem reported on Databricks. It reads the scan’s actual constructor fields at runtime, replaces the field being changed, and lets Spark’smakeCopy()rebuild the scan. #207 used this to replace the relation; this PR applies it to partition and data filters in canonicalization, native scan fallback, and themake_dateplanning rule.All other fields are carried through, including EMR’s extra fields. Stock Spark keeps the same filter and canonicalization behavior. The helper locates the filters relative to the schema to handle the constructor difference between Spark 3 and Spark 4, without hardcoding EMR’s signature.
We’ve checked this on stock Spark 3.5 and 4.1 and the EMR 8.1.0 image. The approach assumes the existing relative order of the schema and filter fields; other constructor or API changes in a vendor runtime would still need validation.
How are these changes tested?
make_datefallback, plus canonicalization with an extended scan. The newmake_dateregression fails with the oldcopy()call.dev/local-ci.sh spark sql_core-4): 5,430 tests passed, 0 failed.make_datecode preserve the real 12-field scan. The original caching reproducer also passed with the copy fix applied to Comet 1.1.0.