Repository navigation
fix: cast a non-INT GetArrayItem index to INT like Spark - #7552
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for the fix. I checked it against Spark's GetArrayItem, which takes any IntegralType ordinal and narrows it with intValue(). The legacy cast to INT does the same thing, and the bounds check in ListExtract runs on the narrowed index, so the ANSI error reports the same index as Spark. Compared with 1.2.0, the only change is that a non-INT index no longer fails natively. INT indexes take the same path as before.
The BIGINT wrap-around cases and the ANSI error index are well covered. One combination I don't see is a literal array with a BIGINT column index, for example SELECT array(1, 2, 3)[l] FROM test_get_array_item_ordinals. The existing literal-array case uses an INT index. A scalar child with a cast ordinal takes a different native path from a column child, so could you add that case to both get_array_item.sql and get_array_item_ansi.sql?
|
Good catch, thanks! I added |
andygrove
left a comment
There was a problem hiding this comment.
Thanks for the fix and for adding the literal-array cases. I checked the legacy cast against Spark's intValue() narrowing and the ANSI error index, and both match. This looks good to me.
Which issue does this PR close?
Closes #7551.
Rationale for this change
arr[i]fails in Comet wheniis not an INT, while Spark accepts any integer type there (for examplesort_array(collect_list(x))[floor(count(*) / 2)], sincefloorreturns a BIGINT).What changes are included in this PR?
If the index is not an INT,
CometGetArrayItemcasts it to INT before passing it toListExtract. The cast is the legacy one even in ANSI mode, because Spark's conversion (intValue()) never fails: it keeps the low 32 bits of a BIGINT and widens TINYINT and SMALLINT, and the legacy cast does the same. INT indexes are unchanged. The bounds check still runs on the converted index, so in ANSI mode Comet reports the same out-of-range index as Spark.How are these changes tested?
New cases in the
get_array_item.sqlandget_array_item_ansi.sqlSQL file tests, compared with Spark:4294967296→ 0, the BIGINT minimum → 0) or an invalid one (2147483648,4294967295, the BIGINT maximum → -1), the invalid ones giving NULL with ANSI offINVALID_ARRAY_INDEXerror reports index -1 for4294967295, like SparkBoth files fail without the change. With the change on top of main (891008e), all
expressions/arraySQL file tests andCometArrayExpressionSuitepass with the Spark 4.1 profile, andspotless:checkis clean.Generated-by: Claude Code (Claude Opus 5.5)
I reviewed the change and ran the tests myself.