Repository navigation
fix(cast): [branch-1.2] fall back for duplicate struct field names (#6671) - #7653
Merged
Merged
Conversation
* fix(cast): fall back for duplicate struct fields Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com> * test(cast): cover duplicate fields in SQL fixture * fix(cast): reject duplicate targets through null paths --------- Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com> (cherry picked from commit bb76f12)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #6671 to
branch-1.2. Closes #6591 on this branch.Rationale for this change
Spark accepts a cast to a struct type with repeated field names, but Arrow Java collapses the fields by name when the native result is imported, so the query fails with
ArrowArray struct has 2 children (expected 1)(#6591,priority:high).branch-1.2has the same cast serde and the same import path. The fix routes that shape through fallback before it crosses Arrow FFI. See #6671 for the full analysis.What changes are included in this PR?
Cherry-pick of bb76f12 (
git cherry-pick -x).CometCast.isSupportedrejects a target that contains a struct with duplicate field names, at any depth through arrays, maps and nested structs, and a null literal of such a type is declined too. It comes with the source PR's SQL file test and its tests inCometNativeCastSuiteandCometLiteralSuite.One adaptation: the import block of
CometCast.scalaconflicted becausemainandbranch-1.2list differentorg.apache.comet.serdenames. The pick keepsbranch-1.2'sserdeimport line and takes only the changedDataTypeSupportimport line, which is the only line the source commit touches there. The diff against the source commit is otherwise identical apart from line numbers and blob hashes.Other release branches:
branch-1.2is the newest release branch, so there is no newer one to port to. Whetherbranch-1.1needs a backport is not part of this PR.How are these changes tested?
On this branch, with JDK 17 and the default Spark profile:
./mvnw test-compilesucceeds. The new test inCometNativeCastSuite,CometLiteralSuitein full and theexpressions/cast/cast_duplicate_struct_fields.sqlfile test pass (7 tests, 0 failures) against a native library built from this branch. No native source changed. Withspark/src/mainreverted tobranch-1.2's version (keeping only the reason constant so the tests compile),cast to a struct with duplicate field names is unsupported,a null literal with duplicate struct field names is unsupportedand the SQL file test fail, so the branch has the bug and the tests catch it.