Repository navigation
refactor(spark): share Spark's common-type rules between functions - #345
Conversation
Move tightest_common_type from array_position into function/type_coercion, with an ANSI flag, a findTypeForComplex that widens map keys and values, Spark's isOrderable (which also rejects INTERVAL), and Spark type names for error messages. Move float_normalize up to function/ so functions outside array can use it. array_position now reports a map with different key or value types as INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries without collecting, keep complex_type private, and move the non-ANSI struct field check and same_type into type_coercion beside the rules they qualify.
…common-type rules Spark has no dictionary types, so a dictionary-encoded argument, such as a Parquet string column, now has the tightest common type of its values, and Spark type names render it as its value type.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove, this is the shared module I was hoping for. I compared tightest_common_type with Spark 4.2.0's TypeCoercion.findTightestCommonType and AnsiTypeCoercion.findTightestCommonType through py4j over 2,304 type pairs in both ANSI modes, and is_orderable with OrderUtils.isOrderable over the same 48 types.
| if left == right { | ||
| return Some(left.clone()); | ||
| } | ||
| // Spark has no dictionary types. A dictionary-encoded argument, such as a | ||
| // Parquet string column, has the type of its values. | ||
| if let Dictionary(_, value) = left { | ||
| return tightest_common_type(value, right, ansi_mode); | ||
| } | ||
| if let Dictionary(_, value) = right { | ||
| return tightest_common_type(left, value, ansi_mode); | ||
| } |
There was a problem hiding this comment.
When both arguments have the same dictionary type, the left == right check returns the dictionary before the unwrapping below it runs, so the dictionary reaches the function as it is. normalize_array doesn't look inside dictionaries, so -0.0 and 0.0 stay unequal. SELECT array_position(make_array(-0.0, 1.0), 0.0) returns 1, as Spark does, but the same query with both sides cast to Dictionary(Int32, Float64) (arrow_cast(make_array(-0.0, 1.0), 'List(Dictionary(Int32, Float64))') and arrow_cast(0.0, 'Dictionary(Int32, Float64)')) returns 0. The new .slt case at array_position.slt passes because its strings don't need normalizing. Could the dictionary checks come first, with a float case for two dictionaries?
| if left == right { | |
| return Some(left.clone()); | |
| } | |
| // Spark has no dictionary types. A dictionary-encoded argument, such as a | |
| // Parquet string column, has the type of its values. | |
| if let Dictionary(_, value) = left { | |
| return tightest_common_type(value, right, ansi_mode); | |
| } | |
| if let Dictionary(_, value) = right { | |
| return tightest_common_type(left, value, ansi_mode); | |
| } | |
| // Spark has no dictionary types. A dictionary-encoded argument, such as a | |
| // Parquet string column, has the type of its values. | |
| if let Dictionary(_, value) = left { | |
| return tightest_common_type(value, right, ansi_mode); | |
| } | |
| if let Dictionary(_, value) = right { | |
| return tightest_common_type(left, value, ansi_mode); | |
| } | |
| if left == right { | |
| return Some(left.clone()); | |
| } |
There was a problem hiding this comment.
Thanks, fixed in 73b600c. The dictionary checks now come first. I added your float case with two dictionaries to array_position.slt, and a unit test for two Dictionary(Int32, Float64) arguments.
There was a problem hiding this comment.
The dictionary checks now come first.
That covers a dictionary at the top level. A dictionary one level down still reaches array_position when both sides have the same type, because the left == right shortcut returns the nested type unchanged. These two queries return 0, and Spark 4.2.0 returns 1 for the same values without the arrow_cast calls:
SELECT array_position(arrow_cast(array(array(CAST('-0.0' AS DOUBLE))), 'List(List(Dictionary(Int32, Float64)))'), arrow_cast(array(CAST(0.0 AS DOUBLE)), 'List(Dictionary(Int32, Float64))'));
SELECT array_position(arrow_cast(array(named_struct('a', CAST('-0.0' AS DOUBLE))), 'List(Struct("a": Dictionary(Int32, Float64)))'), arrow_cast(named_struct('a', CAST(0.0 AS DOUBLE)), 'Struct("a": Dictionary(Int32, Float64))'));When only one side is a dictionary, complex_type unwraps it and the result is 1. Could the shortcut skip nested types, so complex_type rebuilds them and unwraps the dictionaries inside?
// Nested types are widened field by field below, which also unwraps nested dictionaries.
if left == right && !left.is_nested() {
return Some(left.clone());
}With this, both queries return 1. Two equal Union or list-view types then have no common type, and Spark has neither. .slt records for the two queries would cover it.
There was a problem hiding this comment.
Fixed in 6042452 with your !left.is_nested() check. I added unit asserts for two equal List(Dictionary(Int32, Float64)) and two equal Struct(Dictionary(Int32, Float64)) types, and an .slt record with both of your queries, which now return 1.
| let ([lk, lv], [rk, rv]) = (&lkv[..], &rkv[..]) else { | ||
| return None; | ||
| }; | ||
| let key = find_type(lk.data_type(), rk.data_type())?; |
There was a problem hiding this comment.
Spark's findTypeForComplex drops a widened map key type when Cast.forceNullable says the cast to it can return NULL (TypeCoercionHelper.scala). Of the widenings tightest_common_type does, only DATE to TIMESTAMP_NTZ hits that (case (DateType, _) => true). So Spark finds no common type for MAP<DATE, INT> and MAP<TIMESTAMP_NTZ, INT> in either ANSI mode, while this returns MAP<TIMESTAMP_NTZ, INT>. Values aren't filtered, so MAP<INT, DATE> and MAP<INT, TIMESTAMP_NTZ> do widen in Spark.
This changes array_position from main, where the Map arm required identical key and value types. In Spark 4.2.0, in both ANSI modes:
> SELECT array_position(array(map(DATE '2020-01-01', 1)), map(CAST('2020-01-01' AS TIMESTAMP_NTZ), 1));
[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] ... but it's ["ARRAY<MAP<DATE, INT>>", "MAP<TIMESTAMP_NTZ, INT>"]. SQLSTATE: 42K09
With this PR, the same query (with arrow_cast(..., 'Timestamp(Microsecond, None)') for the TIMESTAMP_NTZ) fails with INVALID_ORDERING_TYPE on "MAP<TIMESTAMP_NTZ, BIGINT>". array_insert in #70 doesn't check orderability, so it would accept arguments that Spark's array_insert rejects with ARRAY_FUNCTION_DIFF_TYPES.
Could we filter the key the way Spark does? With a helper next to is_orderable:
/// Spark's `Cast.forceNullable`, for the casts that these rules produce. Of those, Spark
/// marks only DATE to TIMESTAMP_NTZ as one that can return NULL.
fn force_nullable(from: &DataType, to: &DataType) -> bool {
matches!((from, to), (DataType::Date32, DataType::Timestamp(_, None)))
}| let key = find_type(lk.data_type(), rk.data_type())?; | |
| // Spark rejects a key type that the cast can make NULL (`Cast.forceNullable`). | |
| let key = find_type(lk.data_type(), rk.data_type()).filter(|key| { | |
| !force_nullable(lk.data_type(), key) | |
| && !force_nullable(rk.data_type(), key) | |
| })?; |
A unit test for both orders of the two maps, and an .slt record for the array_position query above, would cover it.
There was a problem hiding this comment.
Good catch, fixed in 73b600c with your force_nullable helper. The unit tests cover both orders in both ANSI modes, a map value that still widens, and a DATE key with a TIMESTAMP map key that still widens. I added .slt records for your query, which now fails with ARRAY_FUNCTION_DIFF_TYPES, and for the value case. Both messages match Spark 4.2.0.
There was a problem hiding this comment.
Good catch, fixed in 73b600c with your
force_nullablehelper.
force_nullable gets the key type before tightest_common_type unwraps it, so a dictionary-encoded DATE key still widens: tightest_common_type(&map(Dictionary(Int32, Date32), Int32), &map(Timestamp(Microsecond, None), Int32), false) returns MAP<TIMESTAMP_NTZ, INT>. Could it unwrap the dictionary too?
fn force_nullable(from: &DataType, to: &DataType) -> bool {
match from {
DataType::Dictionary(_, value) => force_nullable(value, to),
_ => matches!((from, to), (DataType::Date32, DataType::Timestamp(_, None))),
}
}A dictionary key in map_keys_do_not_widen_from_date_to_timestamp_ntz would cover it.
There was a problem hiding this comment.
Fixed in 6042452 with your force_nullable change. map_keys_do_not_widen_from_date_to_timestamp_ntz now also checks a Dictionary(Int32, Date32) key.
| Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(), | ||
| List(field) | LargeList(field) | FixedSizeList(field, _) => { | ||
| format!("ARRAY<{}>", spark_sql_type(field.data_type())) | ||
| } | ||
| Struct(fields) => format!( | ||
| "STRUCT<{}>", | ||
| fields | ||
| .iter() | ||
| .map(|f| format!( | ||
| "{}: {}{}", | ||
| f.name(), | ||
| spark_sql_type(f.data_type()), | ||
| if f.is_nullable() { "" } else { " NOT NULL" } |
There was a problem hiding this comment.
spark_sql_type prints Arrow names for three Spark types that Comet maps to Arrow and this crate produces (make_dt_interval returns Duration(Microsecond), make_time returns Time64(Nanosecond)). It also doesn't quote struct field names the way StructField.sql does with quoteIfNeeded. All four reach array_position's error text:
| Query | This PR | Spark 4.2.0 |
|---|---|---|
array_position(array(make_dt_interval(1)), 'x') |
"ARRAY<Duration(µs)>" |
"ARRAY<INTERVAL DAY TO SECOND>" |
array_position(array(make_ym_interval(1)), 'x') |
"ARRAY<Interval(YearMonth)>" |
"ARRAY<INTERVAL YEAR TO MONTH>" |
array_position(array(make_time(1, 2, 3)), 'x') |
"ARRAY<Time64(ns)>" |
"ARRAY<TIME(6)>" (with the TIME type enabled) |
array_position(array(named_struct('a b', 1)), 'x') |
"ARRAY<STRUCT<a b: BIGINT>>" |
"ARRAY<STRUCT<`a b`: INT NOT NULL>>" |
The last row also differs in nullability and the literal's type, which this PR doesn't control. The first column for make_ym_interval used arrow_cast(1, 'Interval(YearMonth)'), since the crate has no make_ym_interval. Could we add these arms, and a helper for the quoting?
/// Spark's `QuotingUtils.quoteIfNeeded`.
fn quote_if_needed(name: &str) -> String {
let mut chars = name.chars();
let plain = chars
.next()
.is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
&& chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
if plain {
name.to_string()
} else {
format!("`{}`", name.replace('`', "``"))
}
}| Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(), | |
| List(field) | LargeList(field) | FixedSizeList(field, _) => { | |
| format!("ARRAY<{}>", spark_sql_type(field.data_type())) | |
| } | |
| Struct(fields) => format!( | |
| "STRUCT<{}>", | |
| fields | |
| .iter() | |
| .map(|f| format!( | |
| "{}: {}{}", | |
| f.name(), | |
| spark_sql_type(f.data_type()), | |
| if f.is_nullable() { "" } else { " NOT NULL" } | |
| Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(), | |
| Interval(IntervalUnit::YearMonth) => "INTERVAL YEAR TO MONTH".to_string(), | |
| Duration(_) => "INTERVAL DAY TO SECOND".to_string(), | |
| // Arrow has no TIME precision. Spark's `make_time` returns TIME(6). | |
| Time64(_) => "TIME(6)".to_string(), | |
| List(field) | LargeList(field) | FixedSizeList(field, _) => { | |
| format!("ARRAY<{}>", spark_sql_type(field.data_type())) | |
| } | |
| Struct(fields) => format!( | |
| "STRUCT<{}>", | |
| fields | |
| .iter() | |
| .map(|f| format!( | |
| "{}: {}{}", | |
| quote_if_needed(f.name()), | |
| spark_sql_type(f.data_type()), | |
| if f.is_nullable() { "" } else { " NOT NULL" } |
There was a problem hiding this comment.
Fixed in 73b600c with your arms and quote_if_needed. I added .slt records for all four rows. For the struct I used a CAST(NULL AS INT) field, so the full message matches Spark 4.2.0 exactly.
…the common-type rules - Unwrap dictionaries before the equality check, so two dictionary arguments reach the function as their value type and floats are normalized. - Reject a map key type that the cast can make NULL, as Spark's findTypeForComplex does with Cast.forceNullable (DATE to TIMESTAMP_NTZ). - Name YEAR TO MONTH and DAY TO SECOND intervals and TIME(6) the way Spark does, and quote struct field names with Spark's quoteIfNeeded.
…map keys The left == right shortcut returned two equal nested types unchanged, so a dictionary inside them reached the function and -0.0 did not match 0.0. Nested types now go through complex_type, which unwraps them. force_nullable now looks through a dictionary key, so a dictionary-encoded DATE key no longer widens to TIMESTAMP_NTZ.
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
# Conflicts: # crates/spark/src/function/mod.rs
…errors Name interval, duration and time types as Spark does, quote struct field names that need it, and use Arrow names for types Spark lacks. Co-Authored-By: Claude <noreply@anthropic.com>
…350) ## Which issue does this PR close? - Part of #346. ## Rationale for this change Spark casts a function's arguments to the input types the function declares, in one place. Here each function picks its accepted types in a hand-written `Signature`, so some arguments that Spark casts are rejected, and a rejected type gives DataFusion's error instead of Spark's. `sha2` shows both problems. Spark declares `(BINARY, INT)`: | Query | Before | Spark 4.2.0 and this PR | |---|---|---| | `sha2('Spark', '256')` | `Function 'sha2' requires Int32, but received String` | the SHA-256 hash | | `sha2('Spark', 256.7::DOUBLE)` | `requires Int32, but received Float64` | the SHA-256 hash | | `sha2('Spark', 256.0::DECIMAL(10, 2))` | `requires Int32, but received Decimal(10, 2)` | the SHA-256 hash | | `sha2(1::INT, 256)` | `requires Binary, but received Int32` | `` [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha2` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09 `` | | `sha2('Spark')` | `Function 'sha2' expects 2 arguments but received 1` | `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha2` requires 2 parameters but the actual number is 1. ... `` | Spark's message also names the argument's SQL text (`"1"`). `coerce_types` sees only the types, so the message here says "it has the type" instead. ## What changes are included in this PR? - New `function/implicit_cast.rs`: - `SparkInputType` mirrors the `AbstractDataType`s Spark functions declare: the concrete scalar types, `Numeric`, `Integral`, `Decimal`, `Array` and `OneOf` (Spark's `TypeCollection`). - `implicit_cast` ports the private `implicitCast(DataType, AbstractDataType)` from both `TypeCoercion` (ANSI mode off) and `AnsiTypeCoercion` (ANSI mode on), with `Cast.canANSIStoreAssign` and `DecimalType.forType`. The two differ, for example a string becomes a BOOLEAN only with ANSI mode on. - `SparkInputTypes` holds the declared types, the ANSI flag and the function's `user_defined` `Signature`. A function returns its `signature()`, calls `coerce` from `coerce_types`, and calls `with_config` from `with_updated_config`. `coerce` returns Spark's `UNEXPECTED_INPUT_TYPE` and `WRONG_NUM_ARGS` errors with the SQLSTATE. - A string cast to BINARY keeps its layout (`Utf8View` becomes `BinaryView`). Spark has no dictionary or fixed-size binary types, so a dictionary argument has the type of its values, as in #345, and a `FixedSizeBinary` argument is cast to BINARY. Before this PR `sha2` rejected `FixedSizeBinary`. #349 covers encodings more broadly. - `sha2` uses `Signature::user_defined` with Spark's `(BINARY, INT)` declaration. Left for later PRs: - DATE and TIMESTAMP targets need the session time zone. They can come with the first function that declares them. - Struct and map types are left out for now. Spark functions often declare the bare `StructType` or `MapType` (any struct or map, for example `to_json` and `GetArrayStructFields`), and some declare `AbstractMapType(key, value)`. They can be added when a function moving onto this module needs them. Until then a struct or map argument to a scalar declared type is rejected with `UNEXPECTED_INPUT_TYPE`, as in Spark. - The module is in its own file so it does not conflict with #345, which adds `function/type_coercion.rs`. `spark_sql_type` is copied from #345, and whichever PR lands second should use the other's copy. The two files could merge once both land. - The cast that runs is DataFusion's. With ANSI mode off Spark returns NULL for `sha2('Spark', '256x')` and wraps `sha2('Spark', 4294967552)` to 256, and both raise a cast error here. That is part of the CAST work in #88. - Most `SparkInputType` variants are unused until more functions move over, so the enum has a scoped `allow(dead_code)`. ## Are these changes tested? Yes. Unit tests in `implicit_cast.rs` cover each rule in both modes, `OneOf`, arrays, dictionaries and the error text. New records in `hash/sha2.slt` cover the casts, the errors and ANSI mode. I checked `sha2.slt` against Spark 4.2.0 with the PySpark runner, and all 45 records match. Most of the new records fail without this change. ## Are there any user-facing changes? `sha2` accepts STRING, FLOAT, DOUBLE and DECIMAL bit lengths, and its errors for other types and for the wrong number of arguments now match Spark's. There are no public API changes, since the new module is `pub(crate)`. This PR was prepared with AI assistance (Claude Code) and reviewed by me. --------- Co-authored-by: Claude <noreply@anthropic.com>
Closes #146. **Stacked on #345**, which moves the common-type rules and Spark type names into `function/type_coercion.rs`. It is merged into this branch, so review the commits after the merge. ## Summary - Adds Spark's `array_insert(array, pos, item)`, ported from DataFusion Comet's `array_funcs/array_insert.rs` as a scalar UDF with type coercion. - 1-based index; negative counts from the end (`-1` appends). A position past the end pads with NULLs. A negative position before the start puts the item first, then NULL padding, then the original elements. - NULL array or NULL position gives NULL; a NULL item is inserted as a NULL element; index `0` fails with `INVALID_INDEX_OF_ZERO`. - `SparkArrayInsert::new().with_legacy_negative_index(true)` gives `spark.sql.legacy.negativeIndexInArrayInsert` behavior: `-1` inserts before the last element, and a position before the start pads one more NULL. Default is Spark 4's behavior. Once `SparkOptions` lands (#92) a session could read the Spark config instead. ## Differences from Comet's version - The size limit is checked before any NULL padding is allocated, so a huge index fails with `COLLECTION_SIZE_LIMIT_EXCEEDED` instead of allocating billions of NULLs. - Index arithmetic is 64-bit, so `i32::MIN` reports a 2147483648-element result (2147483649 in legacy mode). This matches Spark 4.2.0 (SPARK-56567); Spark 4.1.x and older overflow. ## Type rules - The item and elements unify the way Spark's `findTightestCommonType` does. Numeric types widen, and same-family types unify (Utf8/Utf8View, timestamp units, DATE with TIMESTAMP, nested types differing only in nullability). Spark has no common type for some pairs that DataFusion would coerce, and these are rejected: string or binary with another kind, decimal with float, two different decimal types, and an integer with a decimal too narrow for its range. Nested lists, structs and maps are checked the same way. The rules are the shared `tightest_common_type` from #345, called with ANSI mode off as Spark's `ArrayInsert` does, plus its non-ANSI struct check, so struct fields pair by position and their types are not widened (Spark widens them only with ANSI on). Map keys and values widen. `array_position` and `array_remove` use the same rules. - Type errors use Spark's error classes and type names, e.g. `` [DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] Input to `array_insert` should have been "ARRAY" followed by a value with same element type, but it's ["ARRAY<INT>", "STRING"]. SQLSTATE: 42K09 `` The message leaves out Spark's `Cannot resolve "<expr>"` prefix, since the expression text isn't available here. - Spark requires an INT index, but DataFusion types bare integer literals as BIGINT. Any integer type is accepted and cast to INT, so a BIGINT outside the INT range fails the cast and doesn't wrap. Spark rejects every BIGINT index. Non-integer indexes fail with `UNEXPECTED_INPUT_TYPE`. - The result's element field is always nullable, since padding can add NULLs even when neither input has any. ## Testing - Unit tests plus `array/array_insert.slt`. Expected values were verified with pyspark==4.2.0 on 2026-11-15, including legacy mode. The repo's PySpark runner skips most of the file because Spark has no `make_array`/`arrow_typeof`, so the queries were checked as equivalent Spark SQL using `array()`. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, `cargo test -p datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Closes #142. **Stacked on #345**, which moves the common-type rules, the orderability check and Spark type names into `function/type_coercion.rs`. It is merged into this branch, so review the commits after the merge. ## Summary Adds a Spark-compatible `array_remove(array, element)` (`SparkArrayRemove`), ported from DataFusion Comet's `spark_array_remove`, and registers it as `array_remove`. It replaces the DataFusion built-in of the same name when the Spark functions are registered. - It removes every element equal to `element`. NULL elements in the array are always kept. - It removes every matching element, where the DataFusion built-in removes only the first one. It returns NULL when the array or the element is NULL, as the built-in does. - Equality follows Spark's ordering-based `genEqual`/`ordering.equiv` at any depth of arrays and structs: `NaN == NaN`, `-0.0 == 0.0`, and NULLs nested inside the compared values are equal. A constant element that is not nested is compared with every value at once (in place for floats) and the kept elements are found with bitmap operations. Otherwise the comparison runs on float-normalized copies (`float_normalize::normalize_array`, as in `array_position`). The kept elements come from the original values, so `array_remove(array(-0.0, 1.0), 1.0)` returns `[-0.0]`, as in Spark. - It supports `List` and `LargeList`, sliced inputs, and both scalar and column arguments. A constant non-nested element is compared with arrow's vectorized `eq` kernel. Nested elements and column elements use `make_comparator`. Elements of NULL rows are not carried into the result's values. - `invoke_with_args` does the work itself and does not rely on `simplify`. ## Differences from Comet's version - Comet's `spark_array_remove` is only routed for float-containing element types: other types use DataFusion's `array_remove_all`. It is registered under a different name and uses `Signature::any(2)` with no coercion, relying on Spark to have cast the arguments already. This version handles every element type and does Spark's type coercion itself (see below). - Comet supports only 32-bit `List`. This version also supports `LargeList`, and accepts `FixedSizeList` by coercing it to `List`. - Comet has hand-written float kernels (a one-pass constant path that reads the mask a word at a time, plus a per-row float path) and depends on its `float_semantics`/`nested_comparison` helpers. This version has a bitmap path for a constant element that is not nested, like Comet's constant path, and otherwise reuses this crate's `normalize_array` plus arrow's `make_comparator`. `benches/array_remove.rs` shows the bitmap path is about 3.5x faster than the comparator path at 8192 rows. The per-row float path is not ported. - Comet keeps the elements of NULL rows in the child values (hidden by the list's null buffer). This version drops them. - Comet's unit tests are adapted rather than copied verbatim, because they depend on Comet-only test helpers: float edge values as a column and as a constant, nulls, structs with float fields, and scalar float32. Tests for sliced lists, `LargeList`, and the coercion rules are added. ## Type rules These are Spark's `ArrayRemove.inputTypes` / `checkInputDataTypes`. The tightest-common-type logic, the non-ANSI struct field check and the orderability check are shared with `array_position` through `function/type_coercion.rs` (#345). - The array element and the element are cast to `findTightestCommonType`, and the result is an array of that type. For example `array_remove(array(1, 2), 1.5D)` is `array<double>` and `array_remove(array(1, 2), CAST(1 AS DECIMAL(12,2)))` is `array<decimal(12,2)>`. - Numbers widen. - An integral type widens to a decimal only if the decimal can hold all its values. - DATE widens to TIMESTAMP. - Pairs with no common type are an error: string and number, two different decimals, decimal and float, and structs whose field names or types differ. - All of these raise `[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES]`. - A first argument that is not an array raises `ARRAY_FUNCTION_DIFF_TYPES`, as in Spark. This includes an untyped `NULL`. - A NULL-typed common element type is accepted, unlike `array_position`: `array_remove(array(), NULL)` returns NULL with type `array<void>`. - Map and INTERVAL elements, at any depth, raise `[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE]`. - The error messages name Spark's types and end with the SQLSTATE, as Spark's do. - Struct field types are not widened, which matches Spark with ANSI mode off. With ANSI on (Spark 4's default), Spark widens them: `array_remove(array(named_struct('a', 1)), named_struct('a', 1L))` returns `[]` in Spark with ANSI on, and is an error here. This is the same choice `array_position` makes. ## Testing - Unit tests in `array_remove.rs` cover: - float edge values (both zeros, two NaN bit patterns, ±Inf, NULL) as a column and as a constant, checking that the kept elements keep their bits - NULL rows, NULL elements and a NULL element - sliced lists - `LargeList` scalars - struct elements with float fields - the coercion rules - New `crates/sqllogictest/test_files/array/array_remove.slt` covers: - basics, NULL semantics, and error cases - float equality, including nested and in structs - nested NULL equality - type coercion - column inputs, including an `OFFSET`-sliced batch - `LargeList` - Verified with pyspark 4.2.0: - `pyspark_slt_runner.py` on `array_remove.slt`: 71 records pass with no differences. The 6 records that use `arrow_typeof` are skipped, and their result types (bigint, bigint, int, double, decimal(12,2)) were checked by hand with `typeof`/schema in a local SparkSession. - About 50 ad-hoc queries using Spark's `array()` were also checked by hand, including the error classes for an untyped NULL array, decimal and string mismatches, and maps. - Commands, all passing: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-spark` (341 unit tests + doc tests) - `cargo test -p datafusion-spark-sqllogictest` (250 of 250 files)
Use spark_sql_type (apache#345) for the "not supported" and "not implemented yet" errors, so they read "cast from BIGINT to TIMESTAMP_NTZ" rather than "cast from Int64 to Timestamp(µs)", and check that parse_spark_type round-trips with spark_sql_type.
…ernel (#317) ## Which issue does this PR close? Part of #88: the "Rework the `spark_cast` UDF" item. ## Rationale for this change `spark_cast(expr, 'timestamp')` only supported a `timestamp` target and integer or float input, with its own conversion code. To test the Spark-compatible cast kernel from SQL and SLT as each family is ported, the function needs to accept any Spark type name and call the kernel. ## What changes are included in this PR? - **Target type parser** (`cast/data_type.rs`): parses the second argument with Spark's `dataType` grammar from `SqlBaseParser.g4`, case-insensitively. - **Primitive types:** `boolean`, `tinyint`/`byte`, `smallint`/`short`, `int`/`integer`, `bigint`/`long`, `float`/`real`, `double`, `date`, `binary` and `string`. - **Decimals:** `decimal`/`dec`/`numeric`, with an optional precision and scale. A bare `decimal` is `decimal(10, 0)`, and precision and scale are validated. - **Character types:** `char(n)` and `varchar(n)` become `STRING`, which is what Spark casts them to by default. A missing length raises `DATATYPE_MISSING_SIZE`, as in Spark. - **Timestamps:** - `timestamp`, `timestamp_ltz` and `timestamp with local time zone` give a `TIMESTAMP` tagged with the session time zone. That matches Spark's default `spark.sql.timestampType`. - `timestamp_ntz` and `timestamp without time zone` give a `TIMESTAMP_NTZ`. - **Complex types:** `array<…>`, `map<…, …>` (with Spark's `ArrowUtils` field names `entries`, `key` and `value`), and `struct<…>`. Struct fields can be backquoted, `NOT NULL` or have a `COMMENT`, and duplicate field names are allowed, as in Spark. - **Unsupported types:** intervals, `TIME`, `VARIANT`, `VOID`, geospatial types and collated strings get an "unsupported" error. Malformed type names get a parse error that names the problem. - **`spark_cast` UDF:** - accepts any input type - requires the target type to be a string literal - computes its return field from the parsed type - calls the kernel with `EvalMode::from_config` and the session time zone as a `SparkTimeZone` The old conversion code in the UDF is removed. - **Error messages:** the kernel's "not supported" and "not implemented yet" errors name types the Spark way, using `spark_sql_type` from #345. For example, they now say `cast from BIGINT to TIMESTAMP_NTZ` instead of `cast from Int64 to Timestamp(µs)`. - **Integer and float to `TIMESTAMP`:** ported into the kernel (`cast/numeric.rs`). Spark has no cast from a number to `TIMESTAMP_NTZ`, so that cast is rejected. Float input follows Spark 4.2.0's `doubleToTimestamp` and `doubleToTimestampAnsi`: | Mode | NaN or infinite | Out of range | |---|---|---| | Legacy | NULL | saturates, like Scala's `toLong` | | ANSI | `CAST_INVALID_INPUT` | `CAST_OVERFLOW`, such as `1.0E25D` from `DOUBLE` to `BIGINT` | | TRY | NULL | NULL | The previous UDF already saturated in Legacy mode. Comet returns NULL there, which differs from Spark. ## Are these changes tested? Yes. - **Unit tests:** - the type parser: every primitive and complex type, invalid names and unsupported types - integer and float to timestamp in each eval mode, including saturation, a NaN under a null, `FLOAT` widening and the error messages - a round trip between `parse_spark_type` and `spark_sql_type` for each canonical Spark type name - the UDF: return fields, a non-literal target type, array and scalar input, and reading ANSI mode from the config - **`#[user_doc]`** for `spark_cast`, which documents the target-type syntax and the two kinds of error. - **New SLT file** `conversion/cast_target_type.slt`: return types from `arrow_typeof`, the session time zone on `timestamp`, casts that already work through Arrow, invalid and unsupported targets, and casts that are not implemented yet. - **Existing SLT files:** `cast_int_to_timestamp.slt` and `cast_float_to_timestamp.slt` still pass, with these changes: - The old "unsupported target" test used `spark_cast(100, 'string')`, which now works, so it now uses `interval` instead. - It also now checks that a number can't be cast to `timestamp_ntz`. - The ANSI errors are checked against Spark's messages. - It has a Legacy saturation test that casts the result back to `bigint`. The old file said this couldn't be shown in SLT. ## Are there any user-facing changes? Yes. `spark_cast` accepts any Spark type name as its target and any input type. - **Newly working:** casts that are already Spark-compatible through Arrow now succeed. One example is `spark_cast(42, 'string')`. - **Errors for casts not ported yet:** those return an explicit "not supported" or "not implemented yet" error rather than a result that differs from Spark's. A cast that Spark supports, such as TIMESTAMP or DATE to a number, says "not implemented yet"; "not supported" is only for casts Spark does not have. Integer narrowing is one example. The unsuffixed literal `1` is a `BIGINT`, so even `spark_cast(1, 'int')` needs that port. - **Error message format:** in ANSI mode, errors for NaN and out-of-range input now use Spark's `[CAST_INVALID_INPUT]` and `[CAST_OVERFLOW]` messages. _This PR was written by an LLM (Claude) from the Comet and Spark sources, and reviewed before opening._ 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #140. **Stacked on #345**, which moves the orderability check and Spark type names into `function/type_coercion.rs`. It is merged into this branch, so review the commits after the merge. ## Summary Adds Spark-compatible `array_max` and `array_min`, ported from Comet's `native/spark-expr/src/array_funcs/array_extrema.rs`. - NULL elements are skipped. A NULL array, an empty array, or an array of only NULL elements returns NULL. - Elements are compared with Spark's ordering: - NaN is greater than every other value, including infinity, and all NaNs are equal. - `-0.0` equals `0.0`. On a tie the first element wins and is returned unchanged, so `array_max(array(-0.0, 0.0))` is `-0.0` and `array_max(array(0.0, -0.0))` is `0.0`. The NaN payload bits are kept as well. - These float rules also apply inside nested arrays and structs. - Strings and binary compare bytewise. - Arrays compare element by element and then by length. Structs compare field by field. A NULL nested inside an array or struct sorts first. - Supported element types: integers, floats, decimals, booleans, strings, binary, dates, timestamps, and nested arrays and structs of these. `List` and `LargeList` are supported, and `FixedSizeList` is coerced to `List`. - An untyped `NULL` argument and `array()` are accepted and return a NULL of the NULL type, as in Spark (`void`). - Planning errors use Spark's error classes: - `INVALID_ORDERING_TYPE` for map and INTERVAL elements at any depth. - `UNEXPECTED_INPUT_TYPE` for an argument that is not an array. - The messages name Spark's types and end with the SQLSTATE, as Spark's do. - How it works: - Primitive elements that Spark produces (integers, floats, decimals, dates, timestamps) take a typed path: one pass over the raw value buffer with no copy, no type-erased comparator and no `take`. Floats compare with `partial_cmp`, with NaN ranked above everything else. - All other types (strings, binary, booleans, nested arrays and structs, intervals) take a generic path. The values are canonicalized with the existing `normalize_array` helper (NaN to the canonical NaN, `-0.0` to `0.0`, at any depth) and compared with arrow's `make_comparator`, nulls first. The winner is then taken from the original, un-normalized values. - `invoke_with_args` does the work itself and does not depend on `simplify`. - The orderability check and the Spark type names come from `function/type_coercion.rs` (#345). - Both functions are documented with `#[user_doc]`. ## Differences from DataFusion's built-in `array_max` and `array_min` These replace the built-ins when the Spark functions are registered. The built-ins: - order `-0.0` before `0.0`, so `array_max(array(-0.0, 0.0))` is `0.0` and `array_min(array(0.0, -0.0))` is `-0.0`, where Spark returns the first of the tied elements, - sort a NULL nested inside an array or struct after other values, so `array_max(array(array(1, NULL), array(1, 2)))` is `[1, NULL]`, where Spark returns `[1, 2]`, - reject a NULL argument at planning, where Spark returns NULL, - fail at execution on map elements, where Spark rejects them at planning. Each of these has a test in `array/array_extrema.slt`, with the built-in's result in a comment. ## Differences from Comet's version - Comet wraps DataFusion's built-in `array_max`/`array_min`: it reuses their signature and return type, and hands non-float, non-nested element types to the built-in. This version has its own signature and `coerce_types`, and one code path for all element types. - Comet has a dedicated float scan. This version has a typed scan for every primitive type, floats included. - Comet also has a hand-written `take` for nested values, which keeps child buffer capacity bounded when a large list loses. This version uses arrow's `take` on the generic path, so `array_max` over arrays of large arrays can briefly over-reserve child capacity. Both produce the same values. - Comet supports only 32-bit `List`. This version also supports `LargeList`, and coerces `FixedSizeList` to `List`. - Comet does not check types. This version rejects map elements and non-array arguments with Spark's error classes. ## Testing - New unit tests in `array_extrema.rs`, adapted from Comet's tests, plus: - a randomized check that the typed float path picks bit-identical results to the generic path, over NaN payloads (including negative NaN), signed zeros, infinities, NULLs and a sliced list. Swapping in IEEE `total_cmp` makes it fail. - decimal precision/scale and timestamp time zone are kept by the typed path - The original tests: - first-winner float semantics for f32 and f64: NaN payloads, signed zeros, infinities, long lists - scalar and sliced input - strings and binary - `LargeList` - nested lists and structs - NULL-typed input - type rejection - New `array/array_extrema.slt` (34 records). All 34 match Spark when run with the PySpark SLT runner on `pyspark==4.2.0`. I also checked the following directly in PySpark 4.2.0: - error messages for maps, non-arrays and the wrong number of arguments - the `void` result type for `NULL` and `array()` - that the zero sign is kept - The sign of `-0.0` is checked with `CAST(... AS STRING)`, because the test runner prints both zeros as `0`. `atan2` cannot be used for this: Spark adds `0.0` to its arguments, which turns `-0.0` into `0.0`. - Not covered in the `.slt` file, because of literal typing (#97): - `array_max(array(1.0, 2))`: Spark types the array as `DECIMAL(11,1)` and returns `2.0`, while DataFusion returns a `DOUBLE`. - Not tested: - Interval elements. Spark orders `DAY TO SECOND` intervals, and PySpark cannot collect `YEAR TO MONTH` intervals. - The `ArrayMax`/`ArrayMin` definitions in the Spark 3.5.8 and 4.1.1 sources have the same logic. I did not do a full multi-version audit. - New benchmark `benches/array_extrema.rs`: 8192 rows, lists of 0 to 31 elements, 10% NULL elements. Typed path vs generic path on this machine: | Benchmark | Generic | Typed | Change | |---|---|---|---| | `array_max` int64 | 443 µs | 184 µs | −57% | | `array_max` float64 | 491 µs | 298 µs | −39% | | `array_min` float64 | 485 µs | 297 µs | −38% | | `array_max` utf8 (generic path, control) | 897 µs | 903 µs | no change | Canonicalizing floats and comparing with `total_cmp` instead was slower (393 µs), so the `partial_cmp` version was kept. - Commands, all passing: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-spark`: 347 passed - `cargo test -p datafusion-spark-sqllogictest`: 250 of 250 files passed
…de (#360) ## Which issue does this PR close? - Relates to #88 and #346. ## Rationale for this change #89, #304, #345 and #350 added a shared way to declare a function's input types, find a common type, cast a value and read ANSI mode. `CONTRIBUTING.md` does not mention any of them, so a new function is likely to hand-write a `Signature`, call Arrow's `cast`, or take a `bool` for ANSI mode instead. ## What changes are included in this PR? - A new "Input types, casts and ANSI mode" section in `CONTRIBUTING.md`. It says when to use `SparkInputTypes`, the rules in `type_coercion.rs`, `spark_cast()` and `EvalMode`, and points to the module docs for details. - The audit skill's "Signature and coercion" and "ANSI handling" checks now point at the guide. The section says that `SparkInputTypes` only chooses target types and that `spark_cast()` returns "not implemented" for casts that are not ported yet (#88). I kept that wording on purpose, so it can be relaxed as the cast port progresses. ## Are these changes tested? Documentation only. I checked each name, path and example against `apache/main` (`sha2.rs`, `parse_url.rs`, `array_position.rs`, `implicit_cast.rs`, `type_coercion.rs`, `eval_mode.rs`, `cast/mod.rs`). ## Are there any user-facing changes? No. This PR was written with the help of an LLM (Claude Code).
Which issue does this PR close?
Rationale for this change
Functions that need one type for several arguments (
array_positionnow,greatest,least,array_remove,array_max,array_minandarray_insertin open PRs) each carry a copy of Spark's common-type rules. The copies have already drifted apart. This PR gives them one shared copy, so fixes land once.It also fixes two
array_positiondifferences from Spark 4.2.0:array_position(array(map(1, 1)), map(1L, 1L))ARRAY_FUNCTION_DIFF_TYPESINVALID_ORDERING_TYPEon"MAP<BIGINT, BIGINT>"array_position(array(make_interval(1)), make_interval(2))1INVALID_ORDERING_TYPEon"INTERVAL"What changes are included in this PR?
function/type_coercion.rswith:tightest_common_type: Spark'sfindTightestCommonType. It takes an ANSI flag, becauseAnsiTypeCoercionwidens an integer and a FLOAT to DOUBLE.complex_type: Spark'sfindTypeForComplex. It now widens map keys and values, matched by position because Spark'sMapTypehas no field names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ, because that cast can return NULL. The old copy required identical entry types.is_orderable: Spark'sOrderUtils.isOrderable. It now also rejectsInterval(MonthDayNano), which is Spark'sCalendarIntervalType.spark_sql_type: the names Spark'stoSQLTypegives, for error messages. This includesINTERVAL YEAR TO MONTH,INTERVAL DAY TO SECONDandTIME(6), and struct field names quoted the way Spark quotes them.struct_field_types_match: with ANSI mode off, Spark's implicit cast cannot change a struct's field types, so a function that casts its arguments to their tightest common type rejects structs whose field types differ.float_normalizemoves fromfunction/array/tofunction/, so functions outsidearraycan use it.array_positionuses the shared module and passesansi_mode = false, because Spark'sArrayPositioncalls the non-ANSITypeCoercion. Its errors now name Spark types and end with the SQLSTATE, as Spark's do. It also callsstruct_field_types_match, which keeps its current behavior for structs.array_positionnow picks the layout that holds both (for exampleLargeUtf8overUtf8) instead of the left one. Spark has one string type, so this has no effect a user can see.The wider-type functions (
findWiderTypeWithoutStringPromotion,findWiderTypeForDecimal) are left out of this PR because nothing onmaincalls them yet. #332 adds them when it moves to this module.Are these changes tested?
Yes. There are unit tests in
type_coercion.rs, and new.sltcases inarray/array_position.sltcover map widening, flat and nested intervals, and the full error text. I checked them against Spark 4.2.0 with the PySpark runner. The runner flags two records, and both mismatches were already there onmain: line 129 (Spark's 0-based[]) and line 308 (struct names that differ only in case, already documented in the file).Are there any user-facing changes?
array_positionerror messages change as described above, and it rejects INTERVAL elements. There are no API changes.This PR was prepared with AI assistance (Claude Code) and reviewed by me.