Skip to content

feat(spark): add Spark-compatible array_max and array_min - #331

Merged
andygrove merged 11 commits into
apache:mainfrom
andygrove:port-array-extrema
Nov 18, 2026
Merged

andygrove merged 11 commits into
apache:mainfrom
andygrove:port-array-extrema

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #140.

Stacked on #345, which moves the orderability check and Spark type names into function/type_coercion.rs. It is merged into this branch, so review the commits after the merge.

Summary

Adds Spark-compatible array_max and array_min, ported from Comet's
native/spark-expr/src/array_funcs/array_extrema.rs.

  • NULL elements are skipped. A NULL array, an empty array, or an array of only NULL
    elements returns NULL.
  • Elements are compared with Spark's ordering:
    • NaN is greater than every other value, including infinity, and all NaNs are equal.
    • -0.0 equals 0.0. On a tie the first element wins and is returned unchanged, so
      array_max(array(-0.0, 0.0)) is -0.0 and array_max(array(0.0, -0.0)) is 0.0.
      The NaN payload bits are kept as well.
    • These float rules also apply inside nested arrays and structs.
    • Strings and binary compare bytewise.
    • Arrays compare element by element and then by length. Structs compare field by field.
      A NULL nested inside an array or struct sorts first.
  • Supported element types: integers, floats, decimals, booleans, strings, binary, dates,
    timestamps, and nested arrays and structs of these. List and LargeList are
    supported, and FixedSizeList is coerced to List.
  • An untyped NULL argument and array() are accepted and return a NULL of the NULL type,
    as in Spark (void).
  • Planning errors use Spark's error classes:
    • INVALID_ORDERING_TYPE for map and INTERVAL elements at any depth.
    • UNEXPECTED_INPUT_TYPE for an argument that is not an array.
    • The messages name Spark's types and end with the SQLSTATE, as Spark's do.
  • How it works:
    • Primitive elements that Spark produces (integers, floats, decimals, dates, timestamps)
      take a typed path: one
      pass over the raw value buffer with no copy, no type-erased comparator and no take.
      Floats compare with partial_cmp, with NaN ranked above everything else.
    • All other types (strings, binary, booleans, nested arrays and structs, intervals) take a
      generic path. The values are canonicalized with the existing normalize_array helper (NaN
      to the canonical NaN, -0.0 to 0.0, at any depth) and compared with arrow's
      make_comparator, nulls first. The winner is then taken from the original, un-normalized
      values.
    • invoke_with_args does the work itself and does not depend on simplify.
  • The orderability check and the Spark type names come from function/type_coercion.rs (refactor(spark): share Spark's common-type rules between functions #345).
  • Both functions are documented with #[user_doc].

Differences from DataFusion's built-in array_max and array_min

These replace the built-ins when the Spark functions are registered. The built-ins:

  • order -0.0 before 0.0, so array_max(array(-0.0, 0.0)) is 0.0 and array_min(array(0.0, -0.0)) is -0.0, where Spark returns the first of the tied elements,
  • sort a NULL nested inside an array or struct after other values, so array_max(array(array(1, NULL), array(1, 2))) is [1, NULL], where Spark returns [1, 2],
  • reject a NULL argument at planning, where Spark returns NULL,
  • fail at execution on map elements, where Spark rejects them at planning.

Each of these has a test in array/array_extrema.slt, with the built-in's result in a comment.

Differences from Comet's version

  • Comet wraps DataFusion's built-in array_max/array_min: it reuses their signature and
    return type, and hands non-float, non-nested element types to the built-in. This version
    has its own signature and coerce_types, and one code path for all element types.
  • Comet has a dedicated float scan. This version has a typed scan for every primitive type,
    floats included.
  • Comet also has a hand-written take for nested values, which keeps child buffer capacity
    bounded when a large list loses. This version uses arrow's take on the generic path, so
    array_max over arrays of large arrays can briefly over-reserve child capacity. Both produce
    the same values.
  • Comet supports only 32-bit List. This version also supports LargeList, and coerces
    FixedSizeList to List.
  • Comet does not check types. This version rejects map elements and non-array arguments with
    Spark's error classes.

Testing

  • New unit tests in array_extrema.rs, adapted from Comet's tests, plus:

    • a randomized check that the typed float path picks bit-identical results to the generic
      path, over NaN payloads (including negative NaN), signed zeros, infinities, NULLs and a
      sliced list. Swapping in IEEE total_cmp makes it fail.
    • decimal precision/scale and timestamp time zone are kept by the typed path
  • The original tests:

    • first-winner float semantics for f32 and f64: NaN payloads, signed zeros, infinities,
      long lists
    • scalar and sliced input
    • strings and binary
    • LargeList
    • nested lists and structs
    • NULL-typed input
    • type rejection
  • New array/array_extrema.slt (34 records). All 34 match Spark when run with the PySpark
    SLT runner on pyspark==4.2.0. I also checked the following directly in PySpark 4.2.0:

    • error messages for maps, non-arrays and the wrong number of arguments
    • the void result type for NULL and array()
    • that the zero sign is kept
  • The sign of -0.0 is checked with CAST(... AS STRING), because the test runner prints both
    zeros as 0. atan2 cannot be used for this: Spark adds 0.0 to its arguments, which turns
    -0.0 into 0.0.

  • Not covered in the .slt file, because of literal typing (Type numeric literals as Spark does in the sqllogictest files #97):

    • array_max(array(1.0, 2)): Spark types the array as DECIMAL(11,1) and returns 2.0,
      while DataFusion returns a DOUBLE.
  • Not tested:

    • Interval elements. Spark orders DAY TO SECOND intervals, and PySpark cannot collect
      YEAR TO MONTH intervals.
  • The ArrayMax/ArrayMin definitions in the Spark 3.5.8 and 4.1.1 sources have the same
    logic. I did not do a full multi-version audit.

  • New benchmark benches/array_extrema.rs: 8192 rows, lists of 0 to 31 elements, 10% NULL
    elements. Typed path vs generic path on this machine:

    Benchmark Generic Typed Change
    array_max int64 443 µs 184 µs −57%
    array_max float64 491 µs 298 µs −39%
    array_min float64 485 µs 297 µs −38%
    array_max utf8 (generic path, control) 897 µs 903 µs no change

    Canonicalizing floats and comparing with total_cmp instead was slower (393 µs), so the
    partial_cmp version was kept.

  • Commands, all passing:

    • cargo fmt --all
    • cargo clippy --all-targets --all-features -- -D warnings
    • cargo test -p datafusion-spark: 347 passed
    • cargo test -p datafusion-spark-sqllogictest: 250 of 250 files passed

Ports array_max and array_min from Comet (array_funcs/array_extrema.rs).
NULL elements are skipped, and a NULL, empty or all-NULL array returns
NULL. Elements are compared with Spark's ordering: NaN is greater than
every other value, -0.0 equals 0.0 (at any depth), strings and binary
compare bytewise, and NULLs nested in arrays and structs sort first. On
a tie the first element is returned unchanged. Map elements at any depth
are rejected with INVALID_ORDERING_TYPE, and non-array input with
UNEXPECTED_INPUT_TYPE. Supports List, LargeList and FixedSizeList.

Closes apache#140.
…array_min

Primitive element types scan the value buffer directly instead of copying
the values, using arrow's type-erased comparator and taking the winners.
Adds a benchmark and a randomized test that the typed float path matches
the generic path.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The float, NULL, and nested ordering match Spark 4.2.0 in every case I tried, including which tied element is returned.

/// <https://spark.apache.org/docs/latest/api/sql/index.html#array_max>
#[user_doc(
doc_section(label = "Array Functions"),
description = "Returns the maximum element of the array, using Spark's ordering. NULL \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataFusion has its own array_max and array_min (min_max.rs), and with_spark_features replaces them with these. CONTRIBUTING.md asks the description to say how a function differs from the built-in of the same name. Could both descriptions (here and at line 101) say that, as #332 does for greatest and least, with a test for each difference?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1728af8. Both descriptions now say how they differ from DataFusion's built-ins, which I checked by running them directly. The built-ins order -0.0 before 0.0 instead of returning the first tied element, sort a nested NULL after other values, reject a NULL argument at planning, and fail at execution on map elements. Each difference already had an slt test, and those tests now note the built-in's result. NaN is the greatest value in both, so it isn't a difference.

Comment on lines +189 to +194
if contains_map(&element) {
return plan_err!(
"[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{name}` does not support \
ordering on type {element}"
);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark can't order CalendarIntervalType. This crate's make_interval returns it as Interval(MonthDayNano), and Comet maps it to the same Arrow type (serde.rs). OrderUtils.isOrderable accepts only atomic types and arrays and structs of them, and ArrayMax checks it. For SELECT array_max(array(make_interval(1))), Spark 4.2.0 raises:

[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] ... The `array_max` does not support ordering on type "INTERVAL".

Here array_max(array(make_interval(0, 0, 0, 40), make_interval(0, 1))) returns 1 mons, because Arrow orders by months, then days, then nanoseconds. Could the check reject Interval(MonthDayNano) at any depth, not only maps, with an slt error case? #332's is_orderable needs the same fix. My review of #332 proposes one shared module for the orderability check, the type coercion, and the type names, so the check could live there.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. This now uses is_orderable from #345, which rejects Interval(MonthDayNano) at any depth. 1728af8 adds error cases for a flat interval and one inside a struct, and Spark 4.2.0 raises the same errors.

Comment on lines +190 to +202
return plan_err!(
"[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE] The `{name}` does not support \
ordering on type {element}"
);
}
Ok(vec![coerced])
}

fn unexpected_input_type<T>(name: &str, data_type: &DataType) -> Result<T> {
plan_err!(
"[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `{name}` \
requires the \"ARRAY\" type, however it has the type {data_type}"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These messages print the Arrow type where Spark prints its SQL type name. SELECT array_max(1) gives ... however it has the type Int64, and array_max(array(map(1, 2))) gives ... on type Map("entries": non-null Struct("key": non-null Int64, "value": Int64), unsorted). Spark 4.2.0 gives ... however "1" has the type "INT". SQLSTATE: 42K09 and ... on type "MAP<INT, INT>". SQLSTATE: 42K09. Could we render the type the way Spark does and end with the period and SQLSTATE, as in the templates? The hand-built messages on main include the SQLSTATE, for example encode.rs. #332 adds spark_sql_type to render Spark's type names, and my review of #332 proposes moving it into a shared module that this function could use.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1728af8 with spark_sql_type from #345. array_max(1) now gives ... however it has the type "BIGINT". SQLSTATE: 42K09 and maps give ... on type "MAP<BIGINT, BIGINT>". SQLSTATE: 42K09. The message says "it" rather than Spark's quoted expression, because the function only sees the argument's type.

Comment on lines +270 to +290
DataType::UInt8 => ordered!(UInt8Type),
DataType::UInt16 => ordered!(UInt16Type),
DataType::UInt32 => ordered!(UInt32Type),
DataType::UInt64 => ordered!(UInt64Type),
DataType::Decimal128(_, _) => ordered!(Decimal128Type),
DataType::Decimal256(_, _) => ordered!(Decimal256Type),
DataType::Date32 => ordered!(Date32Type),
DataType::Date64 => ordered!(Date64Type),
DataType::Timestamp(TimeUnit::Second, _) => ordered!(TimestampSecondType),
DataType::Timestamp(TimeUnit::Millisecond, _) => {
ordered!(TimestampMillisecondType)
}
DataType::Timestamp(TimeUnit::Microsecond, _) => {
ordered!(TimestampMicrosecondType)
}
DataType::Timestamp(TimeUnit::Nanosecond, _) => {
ordered!(TimestampNanosecondType)
}
DataType::Float16 => primitive!(Float16Type, spark_float_cmp),
DataType::Float32 => primitive!(Float32Type, spark_float_cmp),
DataType::Float64 => primitive!(Float64Type, spark_float_cmp),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark never produces UInt8 through UInt64, Float16, Date64, or Decimal256 (see "Supported types" in CONTRIBUTING.md), and no test reaches these arms. generic_extrema already handles them. Could we drop them, so that every typed arm is one the tests check against the generic path?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped the unsigned, Float16, Date64 and Decimal256 arms in 1728af8. Those types now go to generic_extrema.

Move tightest_common_type from array_position into function/type_coercion,
with an ANSI flag, a findTypeForComplex that widens map keys and values,
Spark's isOrderable (which also rejects INTERVAL), and Spark type names
for error messages. Move float_normalize up to function/ so functions
outside array can use it.

array_position now reports a map with different key or value types as
INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types
and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently
still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries
without collecting, keep complex_type private, and move the non-ANSI
struct field check and same_type into type_coercion beside the rules
they qualify.
# Conflicts:
#	crates/spark/src/function/array/array_position.rs
…ay_min errors

Use the shared isOrderable, which also rejects INTERVAL, and render the
error types the way Spark does, with its SQLSTATE. Say how the functions
differ from DataFusion's built-ins and note the built-in result beside
each test of a difference. Drop the typed arms for types Spark never
produces.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. I checked each fix against PySpark 4.2.0 and against DataFusion's built-in array_max and array_min.

Comment on lines +184 to +192
let coerced = match array {
DataType::Null | DataType::List(_) | DataType::LargeList(_) => array.clone(),
DataType::FixedSizeList(field, _) => DataType::List(Arc::new(Field::new(
field.name(),
field.data_type().clone(),
true,
))),
other => return unexpected_input_type(name, other),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A list of dictionary-encoded doubles keeps its Dictionary element type here, so it goes to generic_extrema. normalize_array doesn't look inside dictionaries, and the comparator then orders -0.0 below 0.0:

SELECT CAST(array_max(c) AS VARCHAR), CAST(array_min(d) AS VARCHAR), CAST(array_max(e) AS VARCHAR), CAST(array_min(f) AS VARCHAR)
FROM (SELECT arrow_cast(array('-0.0'::double, 0.0::double), 'List(Dictionary(Int32, Float64))') AS c,
             arrow_cast(array(0.0::double, '-0.0'::double), 'List(Dictionary(Int32, Float64))') AS d,
             array('-0.0'::double, 0.0::double) AS e,
             array(0.0::double, '-0.0'::double) AS f)
-- 0.0 -0.0 -0.0 0.0

The plain lists e and f return the first tied element, as Spark does, and the dictionary lists c and d don't. Spark has no dictionary types, but engines that build on these functions pass dictionary-encoded columns, for example Sail's rtrim. Could coerce_array replace a dictionary element type with its value type, the way #345's tightest_common_type treats a dictionary argument, with a float case in array_extrema.slt?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4dcf746. coerce_types now casts dictionary types to their value type at any depth of list and struct, so dictionary floats compare the same way as plain ones. This follows the rule #345 and #350 use, where a dictionary has the type of its values. The nested case will follow whatever #349 settles on. I added float cases for List, LargeList, FixedSizeList, a nested list and a struct field to array_extrema.slt, plus a dictionary string case.

…ray_max and array_min

A list of dictionary-encoded floats kept its Dictionary element type, so it
took the generic path, where normalize_array does not look inside
dictionaries and -0.0 ordered below 0.0. coerce_types now casts dictionary
types, at any depth of list and struct, to their value type.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

andygrove added a commit that referenced this pull request Nov 18, 2026
)

## Which issue does this PR close?

- Relates to #332, #331, #330 and #70, which each carry their own copy
of these rules. Requested in [this review
comment](#332 (comment)).

## Rationale for this change

Functions that need one type for several arguments (`array_position`
now, `greatest`, `least`, `array_remove`, `array_max`, `array_min` and
`array_insert` in open PRs) each carry a copy of Spark's common-type
rules. The copies have already drifted apart. This PR gives them one
shared copy, so fixes land once.

It also fixes two `array_position` differences from Spark 4.2.0:

| Query | Before | Spark 4.2.0 |
|---|---|---|
| `array_position(array(map(1, 1)), map(1L, 1L))` |
`ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT,
BIGINT>"` |
| `array_position(array(make_interval(1)), make_interval(2))` | `1` |
`INVALID_ORDERING_TYPE` on `"INTERVAL"` |

## What changes are included in this PR?

- New `function/type_coercion.rs` with:
- `tightest_common_type`: Spark's `findTightestCommonType`. It takes an
ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to
DOUBLE.
- `complex_type`: Spark's `findTypeForComplex`. It now widens map keys
and values, matched by position because Spark's `MapType` has no field
names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ,
because that cast can return NULL. The old copy required identical entry
types.
- `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects
`Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`.
- `spark_sql_type`: the names Spark's `toSQLType` gives, for error
messages. This includes `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO
SECOND` and `TIME(6)`, and struct field names quoted the way Spark
quotes them.
- A dictionary-encoded argument, such as a Parquet string column, has
the type of its values, since Spark has no dictionary types. Broader
handling of dictionary and run-end-encoded arguments is #349.
- `struct_field_types_match`: with ANSI mode off, Spark's implicit cast
cannot change a struct's field types, so a function that casts its
arguments to their tightest common type rejects structs whose field
types differ.
- `float_normalize` moves from `function/array/` to `function/`, so
functions outside `array` can use it.
- `array_position` uses the shared module and passes `ansi_mode =
false`, because Spark's `ArrayPosition` calls the non-ANSI
`TypeCoercion`. Its errors now name Spark types and end with the
SQLSTATE, as Spark's do. It also calls `struct_field_types_match`, which
keeps its current behavior for structs.
- For two string or binary layouts, `array_position` now picks the
layout that holds both (for example `LargeUtf8` over `Utf8`) instead of
the left one. Spark has one string type, so this has no effect a user
can see.

The wider-type functions (`findWiderTypeWithoutStringPromotion`,
`findWiderTypeForDecimal`) are left out of this PR because nothing on
`main` calls them yet. #332 adds them when it moves to this module.

## Are these changes tested?

Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in
`array/array_position.slt` cover map widening, flat and nested
intervals, and the full error text. I checked them against Spark 4.2.0
with the PySpark runner. The runner flags two records, and both
mismatches were already there on `main`: line 129 (Spark's 0-based `[]`)
and line 308 (struct names that differ only in case, already documented
in the file).

## Are there any user-facing changes?

`array_position` error messages change as described above, and it
rejects INTERVAL elements. There are no API changes.

This PR was prepared with AI assistance (Claude Code) and reviewed by
me.
# Conflicts:
#	crates/spark/src/function/mod.rs
#	crates/spark/src/function/type_coercion.rs
#	crates/sqllogictest/test_files/array/array_position.slt
… results

The runner prints floats at full precision since apache#333, including the sign
of -0.0.
# Conflicts:
#	crates/spark/Cargo.toml
#	crates/spark/src/function/array/mod.rs
@andygrove
andygrove merged commit 7ec919f into apache:main Nov 18, 2026
8 checks passed
@andygrove
andygrove deleted the port-array-extrema branch November 18, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port array_max and array_min from Comet

2 participants