Repository navigation
feat(spark): add Spark-compatible array_max and array_min - #331
Conversation
Ports array_max and array_min from Comet (array_funcs/array_extrema.rs). NULL elements are skipped, and a NULL, empty or all-NULL array returns NULL. Elements are compared with Spark's ordering: NaN is greater than every other value, -0.0 equals 0.0 (at any depth), strings and binary compare bytewise, and NULLs nested in arrays and structs sort first. On a tie the first element is returned unchanged. Map elements at any depth are rejected with INVALID_ORDERING_TYPE, and non-array input with UNEXPECTED_INPUT_TYPE. Supports List, LargeList and FixedSizeList. Closes apache#140.
…array_min Primitive element types scan the value buffer directly instead of copying the values, using arrow's type-erased comparator and taking the winners. Adds a benchmark and a randomized test that the typed float path matches the generic path.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. The float, NULL, and nested ordering match Spark 4.2.0 in every case I tried, including which tied element is returned.
| /// <https://spark.apache.org/docs/latest/api/sql/index.html#array_max> | ||
| #[user_doc( | ||
| doc_section(label = "Array Functions"), | ||
| description = "Returns the maximum element of the array, using Spark's ordering. NULL \ |
There was a problem hiding this comment.
DataFusion has its own array_max and array_min (min_max.rs), and with_spark_features replaces them with these. CONTRIBUTING.md asks the description to say how a function differs from the built-in of the same name. Could both descriptions (here and at line 101) say that, as #332 does for greatest and least, with a test for each difference?
There was a problem hiding this comment.
Done in 1728af8. Both descriptions now say how they differ from DataFusion's built-ins, which I checked by running them directly. The built-ins order -0.0 before 0.0 instead of returning the first tied element, sort a nested NULL after other values, reject a NULL argument at planning, and fail at execution on map elements. Each difference already had an slt test, and those tests now note the built-in's result. NaN is the greatest value in both, so it isn't a difference.
| if contains_map(&element) { | ||
| return plan_err!( | ||
| "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{name}` does not support \ | ||
| ordering on type {element}" | ||
| ); | ||
| } |
There was a problem hiding this comment.
Spark can't order CalendarIntervalType. This crate's make_interval returns it as Interval(MonthDayNano), and Comet maps it to the same Arrow type (serde.rs). OrderUtils.isOrderable accepts only atomic types and arrays and structs of them, and ArrayMax checks it. For SELECT array_max(array(make_interval(1))), Spark 4.2.0 raises:
[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] ... The `array_max` does not support ordering on type "INTERVAL".
Here array_max(array(make_interval(0, 0, 0, 40), make_interval(0, 1))) returns 1 mons, because Arrow orders by months, then days, then nanoseconds. Could the check reject Interval(MonthDayNano) at any depth, not only maps, with an slt error case? #332's is_orderable needs the same fix. My review of #332 proposes one shared module for the orderability check, the type coercion, and the type names, so the check could live there.
| return plan_err!( | ||
| "[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{name}` does not support \ | ||
| ordering on type {element}" | ||
| ); | ||
| } | ||
| Ok(vec![coerced]) | ||
| } | ||
|
|
||
| fn unexpected_input_type<T>(name: &str, data_type: &DataType) -> Result<T> { | ||
| plan_err!( | ||
| "[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `{name}` \ | ||
| requires the \"ARRAY\" type, however it has the type {data_type}" | ||
| ) |
There was a problem hiding this comment.
These messages print the Arrow type where Spark prints its SQL type name. SELECT array_max(1) gives ... however it has the type Int64, and array_max(array(map(1, 2))) gives ... on type Map("entries": non-null Struct("key": non-null Int64, "value": Int64), unsorted). Spark 4.2.0 gives ... however "1" has the type "INT". SQLSTATE: 42K09 and ... on type "MAP<INT, INT>". SQLSTATE: 42K09. Could we render the type the way Spark does and end with the period and SQLSTATE, as in the templates? The hand-built messages on main include the SQLSTATE, for example encode.rs. #332 adds spark_sql_type to render Spark's type names, and my review of #332 proposes moving it into a shared module that this function could use.
There was a problem hiding this comment.
Done in 1728af8 with spark_sql_type from #345. array_max(1) now gives ... however it has the type "BIGINT". SQLSTATE: 42K09 and maps give ... on type "MAP<BIGINT, BIGINT>". SQLSTATE: 42K09. The message says "it" rather than Spark's quoted expression, because the function only sees the argument's type.
| DataType::UInt8 => ordered!(UInt8Type), | ||
| DataType::UInt16 => ordered!(UInt16Type), | ||
| DataType::UInt32 => ordered!(UInt32Type), | ||
| DataType::UInt64 => ordered!(UInt64Type), | ||
| DataType::Decimal128(_, _) => ordered!(Decimal128Type), | ||
| DataType::Decimal256(_, _) => ordered!(Decimal256Type), | ||
| DataType::Date32 => ordered!(Date32Type), | ||
| DataType::Date64 => ordered!(Date64Type), | ||
| DataType::Timestamp(TimeUnit::Second, _) => ordered!(TimestampSecondType), | ||
| DataType::Timestamp(TimeUnit::Millisecond, _) => { | ||
| ordered!(TimestampMillisecondType) | ||
| } | ||
| DataType::Timestamp(TimeUnit::Microsecond, _) => { | ||
| ordered!(TimestampMicrosecondType) | ||
| } | ||
| DataType::Timestamp(TimeUnit::Nanosecond, _) => { | ||
| ordered!(TimestampNanosecondType) | ||
| } | ||
| DataType::Float16 => primitive!(Float16Type, spark_float_cmp), | ||
| DataType::Float32 => primitive!(Float32Type, spark_float_cmp), | ||
| DataType::Float64 => primitive!(Float64Type, spark_float_cmp), |
There was a problem hiding this comment.
Spark never produces UInt8 through UInt64, Float16, Date64, or Decimal256 (see "Supported types" in CONTRIBUTING.md), and no test reaches these arms. generic_extrema already handles them. Could we drop them, so that every typed arm is one the tests check against the generic path?
There was a problem hiding this comment.
Dropped the unsigned, Float16, Date64 and Decimal256 arms in 1728af8. Those types now go to generic_extrema.
Move tightest_common_type from array_position into function/type_coercion, with an ANSI flag, a findTypeForComplex that widens map keys and values, Spark's isOrderable (which also rejects INTERVAL), and Spark type names for error messages. Move float_normalize up to function/ so functions outside array can use it. array_position now reports a map with different key or value types as INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries without collecting, keep complex_type private, and move the non-ANSI struct field check and same_type into type_coercion beside the rules they qualify.
# Conflicts: # crates/spark/src/function/array/array_position.rs
…ay_min errors Use the shared isOrderable, which also rejects INTERVAL, and render the error types the way Spark does, with its SQLSTATE. Say how the functions differ from DataFusion's built-ins and note the built-in result beside each test of a difference. Drop the typed arms for types Spark never produces.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. I checked each fix against PySpark 4.2.0 and against DataFusion's built-in array_max and array_min.
| let coerced = match array { | ||
| DataType::Null | DataType::List(_) | DataType::LargeList(_) => array.clone(), | ||
| DataType::FixedSizeList(field, _) => DataType::List(Arc::new(Field::new( | ||
| field.name(), | ||
| field.data_type().clone(), | ||
| true, | ||
| ))), | ||
| other => return unexpected_input_type(name, other), | ||
| }; |
There was a problem hiding this comment.
A list of dictionary-encoded doubles keeps its Dictionary element type here, so it goes to generic_extrema. normalize_array doesn't look inside dictionaries, and the comparator then orders -0.0 below 0.0:
SELECT CAST(array_max(c) AS VARCHAR), CAST(array_min(d) AS VARCHAR), CAST(array_max(e) AS VARCHAR), CAST(array_min(f) AS VARCHAR)
FROM (SELECT arrow_cast(array('-0.0'::double, 0.0::double), 'List(Dictionary(Int32, Float64))') AS c,
arrow_cast(array(0.0::double, '-0.0'::double), 'List(Dictionary(Int32, Float64))') AS d,
array('-0.0'::double, 0.0::double) AS e,
array(0.0::double, '-0.0'::double) AS f)
-- 0.0 -0.0 -0.0 0.0
The plain lists e and f return the first tied element, as Spark does, and the dictionary lists c and d don't. Spark has no dictionary types, but engines that build on these functions pass dictionary-encoded columns, for example Sail's rtrim. Could coerce_array replace a dictionary element type with its value type, the way #345's tightest_common_type treats a dictionary argument, with a float case in array_extrema.slt?
There was a problem hiding this comment.
Fixed in 4dcf746. coerce_types now casts dictionary types to their value type at any depth of list and struct, so dictionary floats compare the same way as plain ones. This follows the rule #345 and #350 use, where a dictionary has the type of its values. The nested case will follow whatever #349 settles on. I added float cases for List, LargeList, FixedSizeList, a nested list and a struct field to array_extrema.slt, plus a dictionary string case.
…ray_max and array_min A list of dictionary-encoded floats kept its Dictionary element type, so it took the generic path, where normalize_array does not look inside dictionaries and -0.0 ordered below 0.0. coerce_types now casts dictionary types, at any depth of list and struct, to their value type.
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
) ## Which issue does this PR close? - Relates to #332, #331, #330 and #70, which each carry their own copy of these rules. Requested in [this review comment](#332 (comment)). ## Rationale for this change Functions that need one type for several arguments (`array_position` now, `greatest`, `least`, `array_remove`, `array_max`, `array_min` and `array_insert` in open PRs) each carry a copy of Spark's common-type rules. The copies have already drifted apart. This PR gives them one shared copy, so fixes land once. It also fixes two `array_position` differences from Spark 4.2.0: | Query | Before | Spark 4.2.0 | |---|---|---| | `array_position(array(map(1, 1)), map(1L, 1L))` | `ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT, BIGINT>"` | | `array_position(array(make_interval(1)), make_interval(2))` | `1` | `INVALID_ORDERING_TYPE` on `"INTERVAL"` | ## What changes are included in this PR? - New `function/type_coercion.rs` with: - `tightest_common_type`: Spark's `findTightestCommonType`. It takes an ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to DOUBLE. - `complex_type`: Spark's `findTypeForComplex`. It now widens map keys and values, matched by position because Spark's `MapType` has no field names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ, because that cast can return NULL. The old copy required identical entry types. - `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects `Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`. - `spark_sql_type`: the names Spark's `toSQLType` gives, for error messages. This includes `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO SECOND` and `TIME(6)`, and struct field names quoted the way Spark quotes them. - A dictionary-encoded argument, such as a Parquet string column, has the type of its values, since Spark has no dictionary types. Broader handling of dictionary and run-end-encoded arguments is #349. - `struct_field_types_match`: with ANSI mode off, Spark's implicit cast cannot change a struct's field types, so a function that casts its arguments to their tightest common type rejects structs whose field types differ. - `float_normalize` moves from `function/array/` to `function/`, so functions outside `array` can use it. - `array_position` uses the shared module and passes `ansi_mode = false`, because Spark's `ArrayPosition` calls the non-ANSI `TypeCoercion`. Its errors now name Spark types and end with the SQLSTATE, as Spark's do. It also calls `struct_field_types_match`, which keeps its current behavior for structs. - For two string or binary layouts, `array_position` now picks the layout that holds both (for example `LargeUtf8` over `Utf8`) instead of the left one. Spark has one string type, so this has no effect a user can see. The wider-type functions (`findWiderTypeWithoutStringPromotion`, `findWiderTypeForDecimal`) are left out of this PR because nothing on `main` calls them yet. #332 adds them when it moves to this module. ## Are these changes tested? Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in `array/array_position.slt` cover map widening, flat and nested intervals, and the full error text. I checked them against Spark 4.2.0 with the PySpark runner. The runner flags two records, and both mismatches were already there on `main`: line 129 (Spark's 0-based `[]`) and line 308 (struct names that differ only in case, already documented in the file). ## Are there any user-facing changes? `array_position` error messages change as described above, and it rejects INTERVAL elements. There are no API changes. This PR was prepared with AI assistance (Claude Code) and reviewed by me.
# Conflicts: # crates/spark/src/function/mod.rs # crates/spark/src/function/type_coercion.rs # crates/sqllogictest/test_files/array/array_position.slt
… results The runner prints floats at full precision since apache#333, including the sign of -0.0.
# Conflicts: # crates/spark/Cargo.toml # crates/spark/src/function/array/mod.rs
Closes #140.
Stacked on #345, which moves the orderability check and Spark type names into
function/type_coercion.rs. It is merged into this branch, so review the commits after the merge.Summary
Adds Spark-compatible
array_maxandarray_min, ported from Comet'snative/spark-expr/src/array_funcs/array_extrema.rs.elements returns NULL.
-0.0equals0.0. On a tie the first element wins and is returned unchanged, soarray_max(array(-0.0, 0.0))is-0.0andarray_max(array(0.0, -0.0))is0.0.The NaN payload bits are kept as well.
A NULL nested inside an array or struct sorts first.
timestamps, and nested arrays and structs of these.
ListandLargeListaresupported, and
FixedSizeListis coerced toList.NULLargument andarray()are accepted and return a NULL of the NULL type,as in Spark (
void).INVALID_ORDERING_TYPEfor map and INTERVAL elements at any depth.UNEXPECTED_INPUT_TYPEfor an argument that is not an array.take a typed path: one
pass over the raw value buffer with no copy, no type-erased comparator and no
take.Floats compare with
partial_cmp, with NaN ranked above everything else.generic path. The values are canonicalized with the existing
normalize_arrayhelper (NaNto the canonical NaN,
-0.0to0.0, at any depth) and compared with arrow'smake_comparator, nulls first. The winner is then taken from the original, un-normalizedvalues.
invoke_with_argsdoes the work itself and does not depend onsimplify.function/type_coercion.rs(refactor(spark): share Spark's common-type rules between functions #345).#[user_doc].Differences from DataFusion's built-in
array_maxandarray_minThese replace the built-ins when the Spark functions are registered. The built-ins:
-0.0before0.0, soarray_max(array(-0.0, 0.0))is0.0andarray_min(array(0.0, -0.0))is-0.0, where Spark returns the first of the tied elements,array_max(array(array(1, NULL), array(1, 2)))is[1, NULL], where Spark returns[1, 2],Each of these has a test in
array/array_extrema.slt, with the built-in's result in a comment.Differences from Comet's version
array_max/array_min: it reuses their signature andreturn type, and hands non-float, non-nested element types to the built-in. This version
has its own signature and
coerce_types, and one code path for all element types.floats included.
takefor nested values, which keeps child buffer capacitybounded when a large list loses. This version uses arrow's
takeon the generic path, soarray_maxover arrays of large arrays can briefly over-reserve child capacity. Both producethe same values.
List. This version also supportsLargeList, and coercesFixedSizeListtoList.Spark's error classes.
Testing
New unit tests in
array_extrema.rs, adapted from Comet's tests, plus:path, over NaN payloads (including negative NaN), signed zeros, infinities, NULLs and a
sliced list. Swapping in IEEE
total_cmpmakes it fail.The original tests:
long lists
LargeListNew
array/array_extrema.slt(34 records). All 34 match Spark when run with the PySparkSLT runner on
pyspark==4.2.0. I also checked the following directly in PySpark 4.2.0:voidresult type forNULLandarray()The sign of
-0.0is checked withCAST(... AS STRING), because the test runner prints bothzeros as
0.atan2cannot be used for this: Spark adds0.0to its arguments, which turns-0.0into0.0.Not covered in the
.sltfile, because of literal typing (Type numeric literals as Spark does in the sqllogictest files #97):array_max(array(1.0, 2)): Spark types the array asDECIMAL(11,1)and returns2.0,while DataFusion returns a
DOUBLE.Not tested:
DAY TO SECONDintervals, and PySpark cannot collectYEAR TO MONTHintervals.The
ArrayMax/ArrayMindefinitions in the Spark 3.5.8 and 4.1.1 sources have the samelogic. I did not do a full multi-version audit.
New benchmark
benches/array_extrema.rs: 8192 rows, lists of 0 to 31 elements, 10% NULLelements. Typed path vs generic path on this machine:
array_maxint64array_maxfloat64array_minfloat64array_maxutf8 (generic path, control)Canonicalizing floats and comparing with
total_cmpinstead was slower (393 µs), so thepartial_cmpversion was kept.Commands, all passing:
cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-spark: 347 passedcargo test -p datafusion-spark-sqllogictest: 250 of 250 files passed