Repository navigation
fix(spark): cast array_repeat's count to INT as Spark does - #357
Conversation
Spark's ArrayRepeat takes an INT count. Cast any integer count to INT, so an unsigned count no longer fails inside DataFusion's array_repeat and a count outside the INT range fails the cast. Other types fail with Spark's UNEXPECTED_INPUT_TYPE error. Part of apache#353.
coderfender
left a comment
There was a problem hiding this comment.
LGTM, thanks! Two non-blocking nits inline.
| // literals as BIGINT, so accept every integer type and cast to INT. A | ||
| // value outside the INT range then fails the cast instead of being | ||
| // silently truncated. | ||
| if !(second_type.is_integer() || second_type.is_null()) { |
There was a problem hiding this comment.
Non-blocking nit: a dictionary-encoded integer count isn't unwrapped here, since is_integer() is false for e.g. Dictionary(Int32, Int64). Because spark_sql_type unwraps dictionaries, the error then reads requires the "INT" type, however it has the type "BIGINT", which is a bit confusing given a plain BIGINT is accepted. It was rejected before this PR too, so not a regression. Matching on the dictionary's value type would line it up with the convention from #349.
There was a problem hiding this comment.
Good catch. A dictionary count now has the type of its values, so Dictionary(Int32, Int64) is accepted and a dictionary of strings still reports "STRING". Fixed in 96666c8, with tests in array_repeat.slt.
| [] | ||
|
|
||
| # A BIGINT count outside the INT range fails the cast to INT. | ||
| query error Cast error |
There was a problem hiding this comment.
Optional: Spark rejects this one at analysis with UNEXPECTED_INPUT_TYPE (the literal is BIGINT), rather than a cast error. Since coerce_types only sees types, the cast error makes sense here; a short note in the comment above might help future readers.
There was a problem hiding this comment.
Added a note above the test saying that Spark rejects the BIGINT literal at analysis, and that we get a cast error because coerce_types only sees types. In 96666c8.
Spark has no dictionary types, so a dictionary count has the type of its values, as in the shared coercion rules. Also note why an out-of-range BIGINT literal fails the cast rather than being rejected at analysis.
Which issue does this PR close?
Rationale for this change
Spark's
ArrayRepeattakes an INT count. Here the count was coerced to BIGINT, or toUInt64for an unsigned type, which DataFusion'sarray_repeatthen cannot read:array_repeat('a', arrow_cast(2, 'UInt32'))Internal error: could not cast array of type UInt64 to ...Int64Type[a, a]array_repeat('a', 3000000000)array_repeat: offset 3000000000 exceeds the maximum value for offset typearray_repeat('a', '2')count must be an integer type[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The second parameter of `array_repeat` requires the "INT" type, however it has the type "STRING". SQLSTATE: 42K09What changes are included in this PR?
coerce_typescasts an integer count of any type to INT, and rejects other types with Spark'sUNEXPECTED_INPUT_TYPEerror.array_repeat, which reads a BIGINT count.Spark rejects a TINYINT, SMALLINT or BIGINT count, rather than casting it. DataFusion types a bare integer literal such as
2as BIGINT, so rejecting BIGINT would breakarray_repeat('a', 2). This followsarray_insertandstack, which accept every integer type for their INT arguments for the same reason. I wrote in #353 thatarray_repeatcould useSparkInputTypeswith[Any, Integer], but that would also cast strings and decimals, which Spark rejects.Are these changes tested?
Yes. New records in
array/array_repeat.sltcover each integer type, an unsigned count, a column with a NULL count, a count outside the INT range, the errors for other types, and ANSI mode. Most of them fail without this change. I checked the file against Spark 4.2.0 with the PySpark runner. Every record matches except the three TINYINT, SMALLINT and BIGINT counts, which Spark rejects as described above. The runner does not compare error messages, so I checked them against Spark by hand.Are there any user-facing changes?
array_repeataccepts an unsigned integer count, a count outside the INT range is an error, and a count of another type fails with Spark's error.This PR was prepared with AI assistance (Claude Code) and reviewed by me.