Skip to content

fix(cast): fall back for duplicate struct field names - #6671

Merged
andygrove merged 5 commits into
apache:mainfrom
Samearth17:fix/6591-cast-duplicate-struct-fields
Oct 25, 2026
Merged

andygrove merged 5 commits into
apache:mainfrom
Samearth17:fix/6591-cast-duplicate-struct-fields

Conversation

@Samearth17

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6591.

Rationale for this change

Spark accepts casts to structs with repeated field names, but importing the native result through Java Arrow collapses fields by name and fails because the child count no longer matches. Comet should route this shape through fallback before it crosses Arrow FFI.

What changes are included in this PR?

  • Make CometCast.isSupported reject target structs with duplicate field names.
  • Cover direct struct casts plus existing recursive array/map handling.
  • Verify a unique-name target remains natively compatible and the duplicate-name query reports the cast fallback reason.

The repository's implement-comet-expression skill was used to follow the existing expression workflow.

How are these changes tested?

  • make core
  • ./mvnw ... test -Dsuites=org.apache.comet.CometNativeCastSuite -Dtest=none (190 passed, 0 failed, 8 ignored)
  • Focused duplicate-field regression on the merged head (1 passed)
  • Guard-removal mutation check (the focused regression failed as expected when the production guard was temporarily removed)
  • make format-check
  • git diff --check

Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com>
@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Oct 5, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: Spark accepts repeated target struct field names, but Arrow Java can collapse those children and fail when importing native cast results.
  • Design approach: CometCast.isSupported reuses hasDuplicateFieldNames to return Unsupported before native execution.
  • Correctness / compatibility analysis: Spark sources for 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 confirm positional struct casting. The dispatcher also rejects duplicate-name output types, allowing Spark fallback. Focused tests passed.
  • Key design decisions: The guard uses the existing helper and recursive support checks. It adds a planning-time field-name scan without changing execution kernels or FFI ownership.
  • Implementation sketch: One guard and a shared fallback reason in CometCast, plus regression coverage in CometNativeCastSuite.
  • Behavioral changes worth calling out: Compared with branch-1.1, affected duplicate-name casts intentionally fall back instead of failing during Arrow import. Unique and case-distinct field names retain native execution in the tested cases.
  • Suggested improvements: No introduced P1/P2 issues found within this review.

Reviewed the entire two-file diff at 0f298851f93469affb649a20945a43808f47c167 against 8a7e6644139c55a10b8f3ee1b1c367bf7eac7d91. Existing reviews, issue comments, inline comments and review threads were empty. Routed skills: review-comet-pr, review-comet-expression-pr, and review-comet-ffi-pr.

Exact-head CI: Comet CI, CodeQL and Check PR Title remain action_required. Label automation passed. There is no exact-head CI test verdict.

Validation: The current JVM sources compiled, and the added regression passed on Spark 4.1.3/JDK 17 using the specified base’s unchanged native artifact. A disposable suite also passed 28 Spark/Comet comparisons covering nested structs, arrays, map keys and values, nulls, empty collections, ANSI settings, CAST, TRY_CAST, and unique/case-distinct names. git diff --check passed and the working tree remains clean. Full cast/Spark SQL suites and runtime tests on other Spark versions were not run.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The guard covers struct, array element and map key and value targets through the existing recursion in CometCast.isSupported. One comment on where the end-to-end half of the test could live.

.range(3)
.selectExpr("named_struct('p', CAST(id AS INT), 'q', CAST(id + 10 AS INT)) AS s")

checkSparkAnswerAndFallbackReason(input.select(col("s").cast(duplicateTarget)), reason)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the end-to-end half of this test move to a SQL file fixture under sql-tests/expressions/cast/? create_map.sql:36 already uses query expect_fallback(...) for the same duplicate-name struct shape, and CometSqlFileTestSuite runs ExpectFallback through checkSparkAnswerAndFallbackReason. A fixture could run the three spellings from #6591 (CAST(s AS STRUCT<x: INT, x: INT>) and the ARRAY<...> and MAP<STRING, ...> wrappers) as real queries, plus a unique-name query that stays native. Today only the struct spelling runs as a query, and the array and map spellings are checked only through isSupported. I have not run it, but from reading the harness I expect the SQL CAST syntax from the issue to work as written.

@Samearth17

Copy link
Copy Markdown
Contributor Author

Addressed the review by moving the end-to-end cast coverage into cast_duplicate_struct_fields.sql. The fixture exercises duplicate targets through direct struct, array-of-struct, and map-value struct casts with the expected fallback reason, plus a distinct-name native-path control. The low-level recursive CometCast.isSupported checks remain in the Scala suite.

Validation: ./mvnw spotless:check -DskipTests passed and git diff --check passed. Focused native SQL execution is unavailable in this macOS checkout because no Comet native library is built under native/target/*-apple-darwin; CI will provide that end-to-end check.

Pushed head: e112b0f045815bf0f8e4103a0b4db9c68655518c.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI hasn't run here yet because the workflows are waiting for approval, so I ran this locally on top of main with Spark 4.1. The new fixture and the new CometNativeCastSuite test pass, along with the rest of the cast and struct fixtures and CometNativeCastSuite. Both new tests fail with the original error when the guard is removed.

Besides the guard placement I left inline, one more shape from #6591 still crashes, though it never reaches CometCast. Spark's NullPropagation folds CAST(NULL AS STRUCT<x: INT, x: INT>) into a null Literal. The null branch of CometLiteral.getSupportLevel checks it with supportedDataType(expr.dataType, allowComplex = true), where allowDuplicateStructFieldNames defaults to true. So SELECT s, CAST(NULL AS STRUCT<x: INT, x: INT>) FROM t runs as a native CometProject and fails on import. SELECT CAST(s AS STRUCT<x: INT, x: INT>) FROM t UNION ALL SELECT NULL FROM t fails too. This PR makes the first branch fall back, but the NULL branch still runs natively. Passing allowDuplicateStructFieldNames = false in that branch fixes both. Since this PR closes #6591, could that go in here with a fixture query? If you'd rather keep the PR to CometCast, I'll open a separate issue for it.

Comment on lines +276 to +278
if (hasDuplicateFieldNames(to_struct.fields)) {
return Unsupported(Some(duplicateStructFieldNamesReason))
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check only runs when the source is a struct too. isSupported returns Compatible() for a NullType source and for an ARRAY<NULL> source before reaching it, and a struct field whose source is NULL takes the NullType branch on the way down. So these still crash with ArrowArray struct has 2 children (expected 1) on this branch, with default configs:

SELECT CAST(named_struct('a', NULL, 'b', b) AS STRUCT<a: STRUCT<x: INT, x: INT>, b: INT>) FROM t
SELECT a, CAST(e AS STRUCT<x: INT, x: INT>) FROM t LATERAL VIEW explode(array(NULL, NULL)) v AS e

Could we check the whole target type once near the top of isSupported instead? It would walk struct fields, array elements, and map keys and values the same way hasCaseInsensitiveDuplicateFieldNames does, and could live next to it in DataTypeSupport:

if (containsDuplicateFieldNames(toType)) {
  return Unsupported(Some(duplicateStructFieldNamesReason))
}

I tried that locally together with the CometLiteral change from the review body. Both queries then match Spark, and CometNativeCastSuite, CometLiteralSuite and the cast, struct, array, map and conditional fixtures still pass. Could you add these queries to the fixture too?

@Samearth17

Copy link
Copy Markdown
Contributor Author

Addressed the null-source correctness gap from the latest review:

  • validate duplicate struct names recursively across the full cast target before null/identity shortcuts
  • reject typed null literals whose declared type contains duplicate struct fields
  • cover direct NullType, ARRAY<NULL>, nested-null, exploded-null, and typed-null cases
  • merged current upstream/main normally (no rebase/force push)

Validation on JDK 17:

  • focused CometNativeCastSuite + CometLiteralSuite: 2 passed
  • ./mvnw spotless:check -DskipTests: passed
  • git diff --check: passed

The SQL-file suite could not validate fallback annotations locally because no native Comet artifact is present; it stopped on the pre-existing first fixture query before reaching the newly added cases. CI is now running on head f6d518f22.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Samearth17. LGTM.

@andygrove
andygrove enabled auto-merge October 25, 2026 11:03
@andygrove
andygrove added this pull request to the merge queue Oct 25, 2026
Merged via the queue into apache:main with commit bb76f12 Oct 25, 2026
43 checks passed
andygrove added a commit that referenced this pull request Nov 19, 2026
* fix(cast): fall back for duplicate struct fields



* test(cast): cover duplicate fields in SQL fixture

* fix(cast): reject duplicate targets through null paths

---------


(cherry picked from commit bb76f12)

Signed-off-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com>
Co-authored-by: Samarth Mukhija <139998526+Samearth17@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Casting to a struct type with repeated field names fails with "ArrowArray struct has 2 children (expected 1)"

4 participants