Skip to content

fix(spark): cast array_repeat's count to INT as Spark does - #357

Merged
andygrove merged 4 commits into
apache:mainfrom
andygrove:array-repeat-int-count
Nov 19, 2026
Merged

andygrove merged 4 commits into
apache:mainfrom
andygrove:array-repeat-int-count

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Spark's ArrayRepeat takes an INT count. Here the count was coerced to BIGINT, or to UInt64 for an unsigned type, which DataFusion's array_repeat then cannot read:

Query Before This PR
array_repeat('a', arrow_cast(2, 'UInt32')) Internal error: could not cast array of type UInt64 to ...Int64Type [a, a]
array_repeat('a', 3000000000) array_repeat: offset 3000000000 exceeds the maximum value for offset type a cast error, since the count is outside the INT range
array_repeat('a', '2') count must be an integer type [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The second parameter of `array_repeat` requires the "INT" type, however it has the type "STRING". SQLSTATE: 42K09

What changes are included in this PR?

  • coerce_types casts an integer count of any type to INT, and rejects other types with Spark's UNEXPECTED_INPUT_TYPE error.
  • The count is cast back to BIGINT before calling DataFusion's array_repeat, which reads a BIGINT count.

Spark rejects a TINYINT, SMALLINT or BIGINT count, rather than casting it. DataFusion types a bare integer literal such as 2 as BIGINT, so rejecting BIGINT would break array_repeat('a', 2). This follows array_insert and stack, which accept every integer type for their INT arguments for the same reason. I wrote in #353 that array_repeat could use SparkInputTypes with [Any, Integer], but that would also cast strings and decimals, which Spark rejects.

Are these changes tested?

Yes. New records in array/array_repeat.slt cover each integer type, an unsigned count, a column with a NULL count, a count outside the INT range, the errors for other types, and ANSI mode. Most of them fail without this change. I checked the file against Spark 4.2.0 with the PySpark runner. Every record matches except the three TINYINT, SMALLINT and BIGINT counts, which Spark rejects as described above. The runner does not compare error messages, so I checked them against Spark by hand.

Are there any user-facing changes?

array_repeat accepts an unsigned integer count, a count outside the INT range is an error, and a count of another type fails with Spark's error.

This PR was prepared with AI assistance (Claude Code) and reviewed by me.

Spark's ArrayRepeat takes an INT count. Cast any integer count to INT,
so an unsigned count no longer fails inside DataFusion's array_repeat
and a count outside the INT range fails the cast. Other types fail with
Spark's UNEXPECTED_INPUT_TYPE error.

Part of apache#353.

@coderfender coderfender left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! Two non-blocking nits inline.

// literals as BIGINT, so accept every integer type and cast to INT. A
// value outside the INT range then fails the cast instead of being
// silently truncated.
if !(second_type.is_integer() || second_type.is_null()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nit: a dictionary-encoded integer count isn't unwrapped here, since is_integer() is false for e.g. Dictionary(Int32, Int64). Because spark_sql_type unwraps dictionaries, the error then reads requires the "INT" type, however it has the type "BIGINT", which is a bit confusing given a plain BIGINT is accepted. It was rejected before this PR too, so not a regression. Matching on the dictionary's value type would line it up with the convention from #349.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. A dictionary count now has the type of its values, so Dictionary(Int32, Int64) is accepted and a dictionary of strings still reports "STRING". Fixed in 96666c8, with tests in array_repeat.slt.

[]

# A BIGINT count outside the INT range fails the cast to INT.
query error Cast error

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: Spark rejects this one at analysis with UNEXPECTED_INPUT_TYPE (the literal is BIGINT), rather than a cast error. Since coerce_types only sees types, the cast error makes sense here; a short note in the comment above might help future readers.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note above the test saying that Spark rejects the BIGINT literal at analysis, and that we get a cast error because coerce_types only sees types. In 96666c8.

Spark has no dictionary types, so a dictionary count has the type of its
values, as in the shared coercion rules. Also note why an out-of-range
BIGINT literal fails the cast rather than being rejected at analysis.
@andygrove
andygrove merged commit 9d802fe into apache:main Nov 19, 2026
10 checks passed
@andygrove
andygrove deleted the array-repeat-int-count branch November 19, 2026 15:03
andygrove added a commit that referenced this pull request Nov 19, 2026
#378)

#115 added a CI check that `docs/functions` matches the `#[user_doc]`
attributes. It merged after #329 and #357, which changed those
attributes, so the Lint workflow fails on main. This reruns
`dev/update_function_docs.sh`. It only changes generated docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants