Repository navigation
fix(cast): fall back for duplicate struct field names - #6671
Conversation
Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com>
…cate-struct-fields
sunchao
left a comment
There was a problem hiding this comment.
Summary
- Prior state and problem: Spark accepts repeated target struct field names, but Arrow Java can collapse those children and fail when importing native cast results.
- Design approach:
CometCast.isSupportedreuseshasDuplicateFieldNamesto returnUnsupportedbefore native execution. - Correctness / compatibility analysis: Spark sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 confirm positional struct casting. The dispatcher also rejects duplicate-name output types, allowing Spark fallback. Focused tests passed.
- Key design decisions: The guard uses the existing helper and recursive support checks. It adds a planning-time field-name scan without changing execution kernels or FFI ownership.
- Implementation sketch: One guard and a shared fallback reason in
CometCast, plus regression coverage inCometNativeCastSuite. - Behavioral changes worth calling out: Compared with
branch-1.1, affected duplicate-name casts intentionally fall back instead of failing during Arrow import. Unique and case-distinct field names retain native execution in the tested cases. - Suggested improvements: No introduced P1/P2 issues found within this review.
Reviewed the entire two-file diff at 0f298851f93469affb649a20945a43808f47c167 against 8a7e6644139c55a10b8f3ee1b1c367bf7eac7d91. Existing reviews, issue comments, inline comments and review threads were empty. Routed skills: review-comet-pr, review-comet-expression-pr, and review-comet-ffi-pr.
Exact-head CI: Comet CI, CodeQL and Check PR Title remain action_required. Label automation passed. There is no exact-head CI test verdict.
Validation: The current JVM sources compiled, and the added regression passed on Spark 4.1.3/JDK 17 using the specified base’s unchanged native artifact. A disposable suite also passed 28 Spark/Comet comparisons covering nested structs, arrays, map keys and values, nulls, empty collections, ANSI settings, CAST, TRY_CAST, and unique/case-distinct names. git diff --check passed and the working tree remains clean. Full cast/Spark SQL suites and runtime tests on other Spark versions were not run.
comphead
left a comment
There was a problem hiding this comment.
The guard covers struct, array element and map key and value targets through the existing recursion in CometCast.isSupported. One comment on where the end-to-end half of the test could live.
| .range(3) | ||
| .selectExpr("named_struct('p', CAST(id AS INT), 'q', CAST(id + 10 AS INT)) AS s") | ||
|
|
||
| checkSparkAnswerAndFallbackReason(input.select(col("s").cast(duplicateTarget)), reason) |
There was a problem hiding this comment.
Could the end-to-end half of this test move to a SQL file fixture under sql-tests/expressions/cast/? create_map.sql:36 already uses query expect_fallback(...) for the same duplicate-name struct shape, and CometSqlFileTestSuite runs ExpectFallback through checkSparkAnswerAndFallbackReason. A fixture could run the three spellings from #6591 (CAST(s AS STRUCT<x: INT, x: INT>) and the ARRAY<...> and MAP<STRING, ...> wrappers) as real queries, plus a unique-name query that stays native. Today only the struct spelling runs as a query, and the array and map spellings are checked only through isSupported. I have not run it, but from reading the harness I expect the SQL CAST syntax from the issue to work as written.
|
Addressed the review by moving the end-to-end cast coverage into Validation: Pushed head: |
andygrove
left a comment
There was a problem hiding this comment.
CI hasn't run here yet because the workflows are waiting for approval, so I ran this locally on top of main with Spark 4.1. The new fixture and the new CometNativeCastSuite test pass, along with the rest of the cast and struct fixtures and CometNativeCastSuite. Both new tests fail with the original error when the guard is removed.
Besides the guard placement I left inline, one more shape from #6591 still crashes, though it never reaches CometCast. Spark's NullPropagation folds CAST(NULL AS STRUCT<x: INT, x: INT>) into a null Literal. The null branch of CometLiteral.getSupportLevel checks it with supportedDataType(expr.dataType, allowComplex = true), where allowDuplicateStructFieldNames defaults to true. So SELECT s, CAST(NULL AS STRUCT<x: INT, x: INT>) FROM t runs as a native CometProject and fails on import. SELECT CAST(s AS STRUCT<x: INT, x: INT>) FROM t UNION ALL SELECT NULL FROM t fails too. This PR makes the first branch fall back, but the NULL branch still runs natively. Passing allowDuplicateStructFieldNames = false in that branch fixes both. Since this PR closes #6591, could that go in here with a fixture query? If you'd rather keep the PR to CometCast, I'll open a separate issue for it.
| if (hasDuplicateFieldNames(to_struct.fields)) { | ||
| return Unsupported(Some(duplicateStructFieldNamesReason)) | ||
| } |
There was a problem hiding this comment.
This check only runs when the source is a struct too. isSupported returns Compatible() for a NullType source and for an ARRAY<NULL> source before reaching it, and a struct field whose source is NULL takes the NullType branch on the way down. So these still crash with ArrowArray struct has 2 children (expected 1) on this branch, with default configs:
SELECT CAST(named_struct('a', NULL, 'b', b) AS STRUCT<a: STRUCT<x: INT, x: INT>, b: INT>) FROM t
SELECT a, CAST(e AS STRUCT<x: INT, x: INT>) FROM t LATERAL VIEW explode(array(NULL, NULL)) v AS eCould we check the whole target type once near the top of isSupported instead? It would walk struct fields, array elements, and map keys and values the same way hasCaseInsensitiveDuplicateFieldNames does, and could live next to it in DataTypeSupport:
if (containsDuplicateFieldNames(toType)) {
return Unsupported(Some(duplicateStructFieldNamesReason))
}I tried that locally together with the CometLiteral change from the review body. Both queries then match Spark, and CometNativeCastSuite, CometLiteralSuite and the cast, struct, array, map and conditional fixtures still pass. Could you add these queries to the fixture too?
|
Addressed the null-source correctness gap from the latest review:
Validation on JDK 17:
The SQL-file suite could not validate fallback annotations locally because no native Comet artifact is present; it stopped on the pre-existing first fixture query before reaching the newly added cases. CI is now running on head |
* fix(cast): fall back for duplicate struct fields * test(cast): cover duplicate fields in SQL fixture * fix(cast): reject duplicate targets through null paths --------- (cherry picked from commit bb76f12) Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com> Co-authored-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com>
Which issue does this PR close?
Closes #6591.
Rationale for this change
Spark accepts casts to structs with repeated field names, but importing the native result through Java Arrow collapses fields by name and fails because the child count no longer matches. Comet should route this shape through fallback before it crosses Arrow FFI.
What changes are included in this PR?
CometCast.isSupportedreject target structs with duplicate field names.The repository's
implement-comet-expressionskill was used to follow the existing expression workflow.How are these changes tested?
make core./mvnw ... test -Dsuites=org.apache.comet.CometNativeCastSuite -Dtest=none(190 passed, 0 failed, 8 ignored)make format-checkgit diff --check