Skip to content

fix: cast a non-INT GetArrayItem index to INT like Spark - #7552

Merged
andygrove merged 4 commits into
apache:mainfrom
akupchinskiy:getarrayitem-long-ordinal
Nov 14, 2026
Merged

andygrove merged 4 commits into
apache:mainfrom
akupchinskiy:getarrayitem-long-ordinal

Conversation

@akupchinskiy

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7551.

Rationale for this change

arr[i] fails in Comet when i is not an INT, while Spark accepts any integer type there (for example sort_array(collect_list(x))[floor(count(*) / 2)], since floor returns a BIGINT).

What changes are included in this PR?

If the index is not an INT, CometGetArrayItem casts it to INT before passing it to ListExtract. The cast is the legacy one even in ANSI mode, because Spark's conversion (intValue()) never fails: it keeps the low 32 bits of a BIGINT and widens TINYINT and SMALLINT, and the legacy cast does the same. INT indexes are unchanged. The bounds check still runs on the converted index, so in ANSI mode Comet reports the same out-of-range index as Spark.

How are these changes tested?

New cases in the get_array_item.sql and get_array_item_ansi.sql SQL file tests, compared with Spark:

  • TINYINT, SMALLINT and BIGINT indexes read from Parquet, with NULL indexes and NULL arrays
  • BIGINT values whose low 32 bits are a valid index (4294967296 → 0, the BIGINT minimum → 0) or an invalid one (2147483648, 4294967295, the BIGINT maximum → -1), the invalid ones giving NULL with ANSI off
  • with ANSI on, the INVALID_ARRAY_INDEX error reports index -1 for 4294967295, like Spark
  • the median query above

Both files fail without the change. With the change on top of main (891008e), all expressions/array SQL file tests and CometArrayExpressionSuite pass with the Spark 4.1 profile, and spotless:check is clean.

Generated-by: Claude Code (Claude Opus 5.5)

I reviewed the change and ran the tests myself.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 7, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I checked it against Spark's GetArrayItem, which takes any IntegralType ordinal and narrows it with intValue(). The legacy cast to INT does the same thing, and the bounds check in ListExtract runs on the narrowed index, so the ANSI error reports the same index as Spark. Compared with 1.2.0, the only change is that a non-INT index no longer fails natively. INT indexes take the same path as before.

The BIGINT wrap-around cases and the ANSI error index are well covered. One combination I don't see is a literal array with a BIGINT column index, for example SELECT array(1, 2, 3)[l] FROM test_get_array_item_ordinals. The existing literal-array case uses an INT index. A scalar child with a cast ordinal takes a different native path from a column child, so could you add that case to both get_array_item.sql and get_array_item_ansi.sql?

@akupchinskiy

Copy link
Copy Markdown
Contributor Author

Good catch, thanks! I added array(...)[b], [s] and [l] to both files, plus an ANSI case where the BIGINT index wraps to -1 and should error. I also checked that the new cases fail without the fix.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix and for adding the literal-array cases. I checked the legacy cast against Spark's intValue() narrowing and the ANSI error index, and both match. This looks good to me.

@kazuyukitanimura
kazuyukitanimura added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@andygrove
andygrove added this pull request to the merge queue Nov 14, 2026
Merged via the queue into apache:main with commit c40c8c2 Nov 14, 2026
42 of 44 checks passed
andygrove added a commit that referenced this pull request Nov 19, 2026
…7654)

* fix: cast a non-INT `GetArrayItem` index to INT like Spark

* test: literal array with a non-INT column index in GetArrayItem

(cherry picked from commit c40c8c2)

Co-authored-by: Artem Kupchinskiy <akupchinskiy@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GetArrayItem fails on a non-INT index: could not cast array of type Int64 ...

3 participants