Repository navigation
feat(spark): add Spark-compatible array_insert function - #70
Conversation
comphead
left a comment
There was a problem hiding this comment.
Thanks @andygrove. The place() index math matches Spark in both modes, the size limit equals ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH, and the row building through one MutableArrayData with bulk NULL padding looks good. This conflicts with #63 in array/mod.rs.
| } else if let Some(common) = binary_numeric_coercion(elem_type, item) { | ||
| common | ||
| } else { | ||
| return exec_err!( |
There was a problem hiding this comment.
Any element and item types that differ and are not both numeric end here. binary_numeric_coercion returns None for non-numeric pairs, so array_insert(col, 1, 'x') fails when col holds Utf8View elements (the parquet default) and the literal is Utf8. The same goes for Utf8 and LargeUtf8, different timestamp units, and nested types that differ only in nullability. Spark's findTightestCommonType accepts these. Could we fall back to type_union_resolution(&[elem_type.clone(), item.clone()]) after the numeric case? A Utf8View case in the .slt would cover it. I have not run it.
There was a problem hiding this comment.
Thanks. Applied in 6122a04, with one change: type_union_resolution on its own coerces INT plus STRING to STRING, so array_insert(array(1), 1, 'x') would succeed where Spark 4.2.0 raises DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES. The fallback now runs after the numeric case, with a check that rejects any pair mixing string or binary with a different kind, at any nesting depth. Utf8View/LargeUtf8, timestamp units and nullability-only differences are now accepted. Added .slt cases for a Utf8View literal and column, LargeUtf8, and the INT/STRING error.
There was a problem hiding this comment.
The fallback now runs after the numeric case, with a check that rejects any pair mixing string or binary with a different kind, at any nesting depth.
The fallback also accepts structs that Spark rejects, because type_union_resolution matches struct fields by name. With this PR, array_insert(make_array(named_struct('a', 1::INT, 'b', 2::INT)), 1, named_struct('b', 3::INT, 'a', 4::INT)) returns [{b: 3, a: 4}, {b: 2, a: 1}]. Spark 4.2.0 raises ARRAY_FUNCTION_DIFF_TYPES for the same query in both ANSI modes, because findTypeForComplex pairs struct fields by position. Struct field types also widen here: named_struct('a', 1::INT) with named_struct('a', 2::BIGINT) gives List(Struct("a": Int64)). Spark widens them only with ANSI on, and array_position on main rejects them to follow ANSI off.
Could we drop spark_rejects_pair and the DataFusion fallback and call tightest_common_type from array_position.rs, as #330 does? It already implements findTightestCommonType for these types and pairs struct fields by position, so the two functions would accept the same types. My review of #332 proposes moving it, with the Spark type names that spark_type_name here renders, into one shared module. It needs one fix first. Its Map arm recurses into the entries struct, whose arm requires identical field types, so map<int, int> and map<bigint, bigint> have no common type. Spark widens the key and the value in both ANSI modes, and the existing "Maps whose key and value types widen" case in array_insert.slt checks it. A .slt case with reordered struct fields would cover the struct point.
There was a problem hiding this comment.
Done in 918dc42. spark_rejects_pair, the DataFusion fallback and the local type-name helpers are gone. coerce_types calls tightest_common_type(elem, item, false) from #345 (which moved it, with the map key and value widening fix, into function/type_coercion.rs), together with its non-ANSI struct_field_types_match, as array_position and array_remove do. New .slt cases cover reordered struct fields and an INT vs BIGINT struct field, both ARRAY_FUNCTION_DIFF_TYPES. The existing map widening case passes. I confirmed all three on PySpark 4.2.0. This PR is now stacked on #345.
|
|
||
| // Expected values are taken from Comet's Spark-verified array_insert tests. | ||
| #[test] | ||
| fn positive_positions() -> Result<()> { |
There was a problem hiding this comment.
positive_positions, negative_positions, the NULL item test, zero_index_is_an_error and the i32::MAX half of the size limit test repeat what array_insert.slt asserts. CI runs only cargo test -p datafusion-spark-sqllogictest, not cargo test -p datafusion-spark, so the Rust unit tests here do not run in CI. Could they move to .slt, keeping Rust tests only for legacy mode and i32::MIN? Cases that neither file covers are legacy mode with a position before the start (-5 on [1] should give 6 elements: the item, four NULLs, then 1), the exact boundary -(len + 1), NULL padding for string or nested elements, and a sliced list.
There was a problem hiding this comment.
Done in 6122a04. The duplicated Rust tests are removed and the missing cases moved to .slt. Added the -(len + 1) boundary, -5 on [1], and NULL padding for string and nested elements. The Rust tests that remain cover legacy mode (including -5 on [1] giving 6 elements), i32::MIN in both modes, and a new sliced-list test, since SQL can't produce a sliced list.
Port Comet's array_insert as a scalar UDF.
Fall back to type_union_resolution when the element and item types are not both numeric, so Utf8View/LargeUtf8 elements accept a Utf8 item, timestamps of different units unify, and nested types differing only in nullability are accepted. Pairs that mix a string or binary type with another kind are still rejected, since type_union_resolution would coerce INT and STRING to STRING while Spark raises ARRAY_FUNCTION_DIFF_TYPES. Move the Rust tests that duplicated array_insert.slt into the .slt, keeping Rust tests only for legacy mode, i32::MIN and a sliced list. Add slt coverage for the -(len + 1) boundary, NULL padding with string and nested elements, Utf8View and LargeUtf8 elements, and a mismatched item type. All new expected values were checked against Spark 4.2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
76bc4e8 to
6122a04
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Reject element/item pairs Spark's findTightestCommonType has no type for: decimal with float, decimals of different precision or scale, and an integer with a decimal that cannot hold its range. Recurse into maps too. - Prefix type errors with Spark's DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES and UNEXPECTED_INPUT_TYPE classes and Spark type names. - Make with_legacy_negative_index a chainable builder. - Return a scalar for all-scalar input, size the child buffer exactly, and avoid a panic on an unexpected index type. - Add slt coverage for decimals, maps, DATE with TIMESTAMP, SMALLINT/TINYINT and out-of-range BIGINT positions, and a column position over the size limit; assert full error text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) ## Summary Addresses #46 by migrating the Spark 4 `encode` charset compatibility fix to the dedicated `datafusion-spark` repository, following maintainer instructions on apache/datafusion#27260 and #87. In Spark 4.0, `encode` tightened charset validation and error handling in two key ways: 1. **Canonical Charset Whitelist:** Rejects Java charset aliases (`UTF8`, `LATIN1`, `ASCII`, `UTF-32BE`, `UTF-32LE`, etc.) with `[INVALID_PARAMETER_VALUE.CHARSET]`, accepting only the 7 canonical Spark charsets: `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16`, `UTF-16BE`, `UTF-16LE`, and `UTF-32`. 2. **Unmappable Character Handling:** Raises `[MALFORMED_CHARACTER_CODING] ... SQLSTATE: 22000` instead of silently replacing unmappable characters with `?`. This PR implements Spark 4.0 strict semantics by default while providing programmatic builder methods to restore pre-Spark 4.0 legacy behavior: - `SparkEncode::with_legacy_java_charsets(true)` restores legacy Java charset alias resolution (Spark 3.5 behavior). - `SparkEncode::with_legacy_coding_error_action(true)` restores legacy replacement of unmappable characters with `?` (Spark 3.5 behavior). - `SparkEncode::with_options(legacy_java_charsets, legacy_coding_error_action)` configures both switches. ## Architectural & Configuration Compatibility Per #92 and #93, `datafusion-spark` is moving toward local configuration extensions and version-derived flags. Following the precedent established in PR #70 (`SparkArrayInsert::with_legacy_negative_index`), `SparkEncode` defines explicit builder methods on the struct. This allows downstream consumers (such as Comet) to configure legacy behavior directly, and provides the exact hooks needed once `SparkOptions` (#92) is merged into the crate. ## Changes - **`crates/spark/src/function/string/encode.rs`**: - Enforce canonical 7-charset whitelist in strict mode; return `[INVALID_PARAMETER_VALUE.CHARSET]` for non-canonical names. - Case-insensitive matching for canonical names. - Raise `[MALFORMED_CHARACTER_CODING]` (SQLSTATE 22000) for unmappable characters in `US-ASCII` and `ISO-8859-1`. - Provide `with_legacy_java_charsets`, `with_legacy_coding_error_action`, and `with_options` builder methods. - Validate scalar charsets upfront so invalid charsets fail immediately even if the input is empty. - Update `#[user_doc]` documentation and examples. - Add 10 comprehensive unit tests covering canonical sets, case insensitivity, alias rejection, legacy mode acceptance, unmappable characters, independent flags, nulls, and array inputs. - **`crates/sqllogictest/test_files/string/encode.slt`**: - Update test cases to expect Spark 4 strict errors (`INVALID_PARAMETER_VALUE.CHARSET` for `utf-32be`, `utf-32le`, `EBCDIC`; `MALFORMED_CHARACTER_CODING` for `é` in `us-ascii` and `Ā` in `iso-8859-1`). ## Validation - `cargo fmt --all -- --check` (clean) - `cargo clippy --all-targets --all-features -- -D warnings` (clean, 0 warnings) - `cargo test -p datafusion-spark -- encode` (10/10 focused unit tests passed) - `cargo test -p datafusion-spark` (341/341 crate tests passed) - `cargo test -p datafusion-spark-sqllogictest -- string/encode` (passed) - `cargo test -p datafusion-spark-sqllogictest` (245/245 test files passed, 100%) - `git diff --check` (clean) Closes #46. --- *AI assistance disclosure: This implementation was prepared and verified with the assistance of Claude Code / Antigravity.* --------- Signed-off-by: Subodh Bansode <subodh171011@gmail.com>
# Conflicts: # crates/spark/src/function/array/mod.rs
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. The index math and the size-limit handling match Spark 4.2.0, including legacy mode.
| if pos == 0 { | ||
| return exec_err!( | ||
| "[INVALID_INDEX_OF_ZERO] The index 0 is invalid. An index shall be either < 0 or > 0 (the first element has index 1)." | ||
| ); | ||
| } | ||
| let n = len as i64; | ||
| let pos = i64::from(pos); | ||
| // (original elements before the item, leading NULLs, trailing NULLs, new length) | ||
| let (before, leading, trailing, new_len) = if pos > 0 { | ||
| let idx = pos - 1; | ||
| if idx <= n { | ||
| (idx, 0, 0, n + 1) | ||
| } else { | ||
| (n, 0, idx - n, pos) | ||
| } | ||
| } else if -pos > n { | ||
| let new_len = -pos + i64::from(legacy); | ||
| (0, new_len - n - 1, 0, new_len) | ||
| } else { | ||
| (n + pos + i64::from(!legacy), 0, 0, n + 1) | ||
| }; | ||
| if new_len > MAX_ROUNDED_ARRAY_LENGTH { | ||
| return exec_err!( | ||
| "[COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] Can't create array with {new_len} elements which exceeding the array size limit {MAX_ROUNDED_ARRAY_LENGTH}, unsuccessful try to create arrays in the function `array_insert`." | ||
| ); | ||
| } |
There was a problem hiding this comment.
Spark 4.2.0 ends both of these messages with the SQLSTATE: ... (the first element has index 1). SQLSTATE: 22003 and ... in the function `array_insert`. SQLSTATE: 54000. Other hand-built runtime errors in the crate include it, for example encode. Could we append SQLSTATE: 22003 and SQLSTATE: 54000? The .slt patterns for the size limit don't anchor the end, so they keep passing, and the INVALID_INDEX_OF_ZERO cases could assert the full text.
There was a problem hiding this comment.
Done in 918dc42, with 22003 and 54000, and the INVALID_INDEX_OF_ZERO and size-limit cases now assert the full text. The two planning errors also end with SQLSTATE: 42K09 now, as in Spark.
| let values = list.values().to_data(); | ||
| let item_data = item.to_data(); | ||
| let mut out = MutableArrayData::with_capacities( | ||
| vec![&values, &item_data], | ||
| true, | ||
| Capacities::Array(total), | ||
| ); |
There was a problem hiding this comment.
MutableArrayData asserts that all of its inputs have the same type, so this panics when the item's type differs from the element type. coerce_types prevents that in SQL, but Comet calls invoke_with_args() without coercion. Calling invoke_with_args with a List(Int32) array and an Int64 item panics with Arrays with inconsistent types passed to MutableArrayData. Comet's version checks the types and returns an internal error. Could we keep that check, with a unit test for it?
| let values = list.values().to_data(); | |
| let item_data = item.to_data(); | |
| let mut out = MutableArrayData::with_capacities( | |
| vec![&values, &item_data], | |
| true, | |
| Capacities::Array(total), | |
| ); | |
| let values = list.values().to_data(); | |
| let item_data = item.to_data(); | |
| if values.data_type() != item_data.data_type() { | |
| return internal_err!( | |
| "array_insert expects the item type {} to match the element type {}", | |
| item_data.data_type(), | |
| values.data_type() | |
| ); | |
| } | |
| let mut out = MutableArrayData::with_capacities( | |
| vec![&values, &item_data], | |
| true, | |
| Capacities::Array(total), | |
| ); |
There was a problem hiding this comment.
Applied in 918dc42, with mismatched_item_type_is_an_error calling invoke_with_args with a List(Int32) array and an Int64 item.
Move tightest_common_type from array_position into function/type_coercion, with an ANSI flag, a findTypeForComplex that widens map keys and values, Spark's isOrderable (which also rejects INTERVAL), and Spark type names for error messages. Move float_normalize up to function/ so functions outside array can use it. array_position now reports a map with different key or value types as INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries without collecting, keep complex_type private, and move the non-ANSI struct field check and same_type into type_coercion beside the rules they qualify.
Replace the DataFusion coercion fallback, which paired struct fields by name and widened their types, with the shared tightest_common_type and the non-ANSI struct field check, as array_position and array_remove do. End both runtime errors with Spark's SQLSTATE, and return an error rather than panic when the item type differs from the element type.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. This approves what #70 adds on top of #345. I have nothing to add beyond the existing threads.
) ## Which issue does this PR close? - Relates to #332, #331, #330 and #70, which each carry their own copy of these rules. Requested in [this review comment](#332 (comment)). ## Rationale for this change Functions that need one type for several arguments (`array_position` now, `greatest`, `least`, `array_remove`, `array_max`, `array_min` and `array_insert` in open PRs) each carry a copy of Spark's common-type rules. The copies have already drifted apart. This PR gives them one shared copy, so fixes land once. It also fixes two `array_position` differences from Spark 4.2.0: | Query | Before | Spark 4.2.0 | |---|---|---| | `array_position(array(map(1, 1)), map(1L, 1L))` | `ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT, BIGINT>"` | | `array_position(array(make_interval(1)), make_interval(2))` | `1` | `INVALID_ORDERING_TYPE` on `"INTERVAL"` | ## What changes are included in this PR? - New `function/type_coercion.rs` with: - `tightest_common_type`: Spark's `findTightestCommonType`. It takes an ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to DOUBLE. - `complex_type`: Spark's `findTypeForComplex`. It now widens map keys and values, matched by position because Spark's `MapType` has no field names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ, because that cast can return NULL. The old copy required identical entry types. - `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects `Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`. - `spark_sql_type`: the names Spark's `toSQLType` gives, for error messages. This includes `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO SECOND` and `TIME(6)`, and struct field names quoted the way Spark quotes them. - A dictionary-encoded argument, such as a Parquet string column, has the type of its values, since Spark has no dictionary types. Broader handling of dictionary and run-end-encoded arguments is #349. - `struct_field_types_match`: with ANSI mode off, Spark's implicit cast cannot change a struct's field types, so a function that casts its arguments to their tightest common type rejects structs whose field types differ. - `float_normalize` moves from `function/array/` to `function/`, so functions outside `array` can use it. - `array_position` uses the shared module and passes `ansi_mode = false`, because Spark's `ArrayPosition` calls the non-ANSI `TypeCoercion`. Its errors now name Spark types and end with the SQLSTATE, as Spark's do. It also calls `struct_field_types_match`, which keeps its current behavior for structs. - For two string or binary layouts, `array_position` now picks the layout that holds both (for example `LargeUtf8` over `Utf8`) instead of the left one. Spark has one string type, so this has no effect a user can see. The wider-type functions (`findWiderTypeWithoutStringPromotion`, `findWiderTypeForDecimal`) are left out of this PR because nothing on `main` calls them yet. #332 adds them when it moves to this module. ## Are these changes tested? Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in `array/array_position.slt` cover map widening, flat and nested intervals, and the full error text. I checked them against Spark 4.2.0 with the PySpark runner. The runner flags two records, and both mismatches were already there on `main`: line 129 (Spark's 0-based `[]`) and line 308 (struct names that differ only in case, already documented in the file). ## Are there any user-facing changes? `array_position` error messages change as described above, and it rejects INTERVAL elements. There are no API changes. This PR was prepared with AI assistance (Claude Code) and reviewed by me.
# Conflicts: # crates/spark/src/function/mod.rs # crates/spark/src/function/type_coercion.rs # crates/sqllogictest/test_files/array/array_position.slt
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
Closes #146.
Stacked on #345, which moves the common-type rules and Spark type names into
function/type_coercion.rs. It is merged into this branch, so review the commits after the merge.Summary
array_insert(array, pos, item), ported from DataFusion Comet'sarray_funcs/array_insert.rsas a scalar UDF with type coercion.-1appends). A position past the end pads with NULLs. A negative position before the start puts the item first, then NULL padding, then the original elements.0fails withINVALID_INDEX_OF_ZERO.SparkArrayInsert::new().with_legacy_negative_index(true)givesspark.sql.legacy.negativeIndexInArrayInsertbehavior:-1inserts before the last element, and a position before the start pads one more NULL. Default is Spark 4's behavior. OnceSparkOptionslands (Move SparkOptions into datafusion-spark as a config extension #92) a session could read the Spark config instead.Differences from Comet's version
COLLECTION_SIZE_LIMIT_EXCEEDEDinstead of allocating billions of NULLs.i32::MINreports a 2147483648-element result (2147483649 in legacy mode). This matches Spark 4.2.0 (SPARK-56567); Spark 4.1.x and older overflow.Type rules
findTightestCommonTypedoes. Numeric types widen, and same-family types unify (Utf8/Utf8View, timestamp units, DATE with TIMESTAMP, nested types differing only in nullability). Spark has no common type for some pairs that DataFusion would coerce, and these are rejected: string or binary with another kind, decimal with float, two different decimal types, and an integer with a decimal too narrow for its range. Nested lists, structs and maps are checked the same way. The rules are the sharedtightest_common_typefrom refactor(spark): share Spark's common-type rules between functions #345, called with ANSI mode off as Spark'sArrayInsertdoes, plus its non-ANSI struct check, so struct fields pair by position and their types are not widened (Spark widens them only with ANSI on). Map keys and values widen.array_positionandarray_removeuse the same rules.[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] Input to `array_insert` should have been "ARRAY" followed by a value with same element type, but it's ["ARRAY<INT>", "STRING"]. SQLSTATE: 42K09The message leaves out Spark'sCannot resolve "<expr>"prefix, since the expression text isn't available here.UNEXPECTED_INPUT_TYPE.Testing
array/array_insert.slt. Expected values were verified with pyspark==4.2.0 on 2026-11-15, including legacy mode. The repo's PySpark runner skips most of the file because Spark has nomake_array/arrow_typeof, so the queries were checked as equivalent Spark SQL usingarray().cargo fmt --all,cargo clippy --all-targets --all-features -- -D warnings,cargo test -p datafusion-sparkandcargo test -p datafusion-spark-sqllogictestpass.🤖 Generated with Claude Code