You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
arrays_zip with two same-named inputs fails with "ArrowArray struct has 2 children (expected 1)" #6251
arrays_zip over two inputs with the same name fails the task when Comet runs it natively. Spark names each struct field after its input, so arrays_zip(a, a) has type array<struct<a: string, a: string>>, and Comet's native projection returns that struct fine. Importing the result on the JVM then fails, because Java Arrow keys struct children by name and the two a children collapse into one:
java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1)
at org.apache.arrow.util.Preconditions.checkState(Preconditions.java:562)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:92)
at org.apache.arrow.c.ArrayImporter.importChild(ArrayImporter.java:83)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:101)
at org.apache.arrow.c.ArrayImporter.importArray(ArrayImporter.java:68)
at org.apache.arrow.c.ArrowImporter.importVector(ArrowImporter.java:62)
at org.apache.comet.vector.NativeUtil.importVector(NativeUtil.scala:264)
at org.apache.comet.vector.NativeUtil.getNextBatch(NativeUtil.scala:211)
at org.apache.comet.CometExecIterator.getNextBatch(CometExecIterator.scala:237)
arrays_zip(a, b) over the same table works. Spark returns the zipped rows for all three queries below.
This is the same Java Arrow limitation as #1015 and #5605. CometCreateNamedStruct.getSupportLevel already returns Unsupported when names has duplicates, and DataTypeSupport rejects such structs in operator schemas, but CometArraysZip.getSupportLevel only checks the input types and never looks at expr.names. Two same-named inputs come up naturally after a join, for example arrays_zip(t1.tags, t2.tags).
Steps to reproduce
Default configs, reproduced on main at f7952de73 with Spark 4.1.3:
spark.range(4)
.selectExpr("id", "array(cast(id as string), 'x') as a", "array(id, id + 1) as b")
.write.parquet(path)
valdf= spark.read.parquet(path)
df.selectExpr("id", "arrays_zip(a, a) AS r").collect() // IllegalStateException
df.selectExpr("id", "arrays_zip(b, b) AS r").collect() // IllegalStateException
df.selectExpr("id", "arrays_zip(a, b) AS r").collect() // matches Spark
The plan is CometProject over CometNativeScan.
Expected behavior
Either match Spark or fall back. The smallest fix is probably for CometArraysZip.getSupportLevel to return Unsupported when expr.names has duplicates, the same way CometCreateNamedStruct does.
Additional context
Found while reviewing #6036, but unrelated to that PR's change.
Hi @andygrove, I would like to take this up and implement the fix!
Root Cause & Proposed Fix
The issue occurs because Java Arrow's ArrayImporter indexes struct children by field name during JVM vector import, causing duplicate struct field names (e.g., from arrays_zip(a, a)) to collapse and throw java.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1).
Following the established pattern in CometCreateNamedStruct:
SerDe Validation: Update CometArraysZip.getSupportLevel in spark/src/main/scala/org/apache/comet/serde/arrays.scala to evaluate expr.names and check for duplicate field names:
valnames= expr.names.map(_.eval(EmptyRow))
if (names.length != names.distinct.length) {
returnUnsupported(Some("`arrays_zip` with duplicate field names is not supported"))
}
Activity
mohitgurav20 commentedon Sep 27, 2026
Hi @andygrove, I would like to take this up and implement the fix!
Root Cause & Proposed Fix
The issue occurs because Java Arrow's
ArrayImporterindexes struct children by field name during JVM vector import, causing duplicate struct field names (e.g., fromarrays_zip(a, a)) to collapse and throwjava.lang.IllegalStateException: ArrowArray struct has 2 children (expected 1).Following the established pattern in
CometCreateNamedStruct:CometArraysZip.getSupportLevelinspark/src/main/scala/org/apache/comet/serde/arrays.scalato evaluateexpr.namesand check for duplicate field names:mohitgurav20 commentedon Sep 27, 2026
take
comphead commentedon Sep 28, 2026
Thanks @mohitgurav20
mohitgurav20 commentedon Sep 28, 2026
thanks for assigning me here will soon raise a pr , feel free to review it