Skip to content

refactor(spark): share Spark's common-type rules between functions - #345

Merged
andygrove merged 7 commits into
apache:mainfrom
andygrove:spark-common-type
Nov 18, 2026
Merged

andygrove merged 7 commits into
apache:mainfrom
andygrove:spark-common-type

Conversation

@andygrove

@andygrove andygrove commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Functions that need one type for several arguments (array_position now, greatest, least, array_remove, array_max, array_min and array_insert in open PRs) each carry a copy of Spark's common-type rules. The copies have already drifted apart. This PR gives them one shared copy, so fixes land once.

It also fixes two array_position differences from Spark 4.2.0:

Query Before Spark 4.2.0
array_position(array(map(1, 1)), map(1L, 1L)) ARRAY_FUNCTION_DIFF_TYPES INVALID_ORDERING_TYPE on "MAP<BIGINT, BIGINT>"
array_position(array(make_interval(1)), make_interval(2)) 1 INVALID_ORDERING_TYPE on "INTERVAL"

What changes are included in this PR?

  • New function/type_coercion.rs with:
    • tightest_common_type: Spark's findTightestCommonType. It takes an ANSI flag, because AnsiTypeCoercion widens an integer and a FLOAT to DOUBLE.
    • complex_type: Spark's findTypeForComplex. It now widens map keys and values, matched by position because Spark's MapType has no field names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ, because that cast can return NULL. The old copy required identical entry types.
    • is_orderable: Spark's OrderUtils.isOrderable. It now also rejects Interval(MonthDayNano), which is Spark's CalendarIntervalType.
    • spark_sql_type: the names Spark's toSQLType gives, for error messages. This includes INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND and TIME(6), and struct field names quoted the way Spark quotes them.
    • A dictionary-encoded argument, such as a Parquet string column, has the type of its values, since Spark has no dictionary types. Broader handling of dictionary and run-end-encoded arguments is Decide how functions handle dictionary and run-end-encoded arguments #349.
    • struct_field_types_match: with ANSI mode off, Spark's implicit cast cannot change a struct's field types, so a function that casts its arguments to their tightest common type rejects structs whose field types differ.
  • float_normalize moves from function/array/ to function/, so functions outside array can use it.
  • array_position uses the shared module and passes ansi_mode = false, because Spark's ArrayPosition calls the non-ANSI TypeCoercion. Its errors now name Spark types and end with the SQLSTATE, as Spark's do. It also calls struct_field_types_match, which keeps its current behavior for structs.
  • For two string or binary layouts, array_position now picks the layout that holds both (for example LargeUtf8 over Utf8) instead of the left one. Spark has one string type, so this has no effect a user can see.

The wider-type functions (findWiderTypeWithoutStringPromotion, findWiderTypeForDecimal) are left out of this PR because nothing on main calls them yet. #332 adds them when it moves to this module.

Are these changes tested?

Yes. There are unit tests in type_coercion.rs, and new .slt cases in array/array_position.slt cover map widening, flat and nested intervals, and the full error text. I checked them against Spark 4.2.0 with the PySpark runner. The runner flags two records, and both mismatches were already there on main: line 129 (Spark's 0-based []) and line 308 (struct names that differ only in case, already documented in the file).

Are there any user-facing changes?

array_position error messages change as described above, and it rejects INTERVAL elements. There are no API changes.

This PR was prepared with AI assistance (Claude Code) and reviewed by me.

Move tightest_common_type from array_position into function/type_coercion,
with an ANSI flag, a findTypeForComplex that widens map keys and values,
Spark's isOrderable (which also rejects INTERVAL), and Spark type names
for error messages. Move float_normalize up to function/ so functions
outside array can use it.

array_position now reports a map with different key or value types as
INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types
and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently
still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries
without collecting, keep complex_type private, and move the non-ANSI
struct field check and same_type into type_coercion beside the rules
they qualify.
…common-type rules

Spark has no dictionary types, so a dictionary-encoded argument, such as
a Parquet string column, now has the tightest common type of its values,
and Spark type names render it as its value type.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove, this is the shared module I was hoping for. I compared tightest_common_type with Spark 4.2.0's TypeCoercion.findTightestCommonType and AnsiTypeCoercion.findTightestCommonType through py4j over 2,304 type pairs in both ANSI modes, and is_orderable with OrderUtils.isOrderable over the same 48 types.

Comment on lines +26 to +36
if left == right {
return Some(left.clone());
}
// Spark has no dictionary types. A dictionary-encoded argument, such as a
// Parquet string column, has the type of its values.
if let Dictionary(_, value) = left {
return tightest_common_type(value, right, ansi_mode);
}
if let Dictionary(_, value) = right {
return tightest_common_type(left, value, ansi_mode);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When both arguments have the same dictionary type, the left == right check returns the dictionary before the unwrapping below it runs, so the dictionary reaches the function as it is. normalize_array doesn't look inside dictionaries, so -0.0 and 0.0 stay unequal. SELECT array_position(make_array(-0.0, 1.0), 0.0) returns 1, as Spark does, but the same query with both sides cast to Dictionary(Int32, Float64) (arrow_cast(make_array(-0.0, 1.0), 'List(Dictionary(Int32, Float64))') and arrow_cast(0.0, 'Dictionary(Int32, Float64)')) returns 0. The new .slt case at array_position.slt passes because its strings don't need normalizing. Could the dictionary checks come first, with a float case for two dictionaries?

Suggested change
if left == right {
return Some(left.clone());
}
// Spark has no dictionary types. A dictionary-encoded argument, such as a
// Parquet string column, has the type of its values.
if let Dictionary(_, value) = left {
return tightest_common_type(value, right, ansi_mode);
}
if let Dictionary(_, value) = right {
return tightest_common_type(left, value, ansi_mode);
}
// Spark has no dictionary types. A dictionary-encoded argument, such as a
// Parquet string column, has the type of its values.
if let Dictionary(_, value) = left {
return tightest_common_type(value, right, ansi_mode);
}
if let Dictionary(_, value) = right {
return tightest_common_type(left, value, ansi_mode);
}
if left == right {
return Some(left.clone());
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 73b600c. The dictionary checks now come first. I added your float case with two dictionaries to array_position.slt, and a unit test for two Dictionary(Int32, Float64) arguments.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dictionary checks now come first.

That covers a dictionary at the top level. A dictionary one level down still reaches array_position when both sides have the same type, because the left == right shortcut returns the nested type unchanged. These two queries return 0, and Spark 4.2.0 returns 1 for the same values without the arrow_cast calls:

SELECT array_position(arrow_cast(array(array(CAST('-0.0' AS DOUBLE))), 'List(List(Dictionary(Int32, Float64)))'), arrow_cast(array(CAST(0.0 AS DOUBLE)), 'List(Dictionary(Int32, Float64))'));
SELECT array_position(arrow_cast(array(named_struct('a', CAST('-0.0' AS DOUBLE))), 'List(Struct("a": Dictionary(Int32, Float64)))'), arrow_cast(named_struct('a', CAST(0.0 AS DOUBLE)), 'Struct("a": Dictionary(Int32, Float64))'));

When only one side is a dictionary, complex_type unwraps it and the result is 1. Could the shortcut skip nested types, so complex_type rebuilds them and unwraps the dictionaries inside?

    // Nested types are widened field by field below, which also unwraps nested dictionaries.
    if left == right && !left.is_nested() {
        return Some(left.clone());
    }

With this, both queries return 1. Two equal Union or list-view types then have no common type, and Spark has neither. .slt records for the two queries would cover it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6042452 with your !left.is_nested() check. I added unit asserts for two equal List(Dictionary(Int32, Float64)) and two equal Struct(Dictionary(Int32, Float64)) types, and an .slt record with both of your queries, which now return 1.

let ([lk, lv], [rk, rv]) = (&lkv[..], &rkv[..]) else {
return None;
};
let key = find_type(lk.data_type(), rk.data_type())?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark's findTypeForComplex drops a widened map key type when Cast.forceNullable says the cast to it can return NULL (TypeCoercionHelper.scala). Of the widenings tightest_common_type does, only DATE to TIMESTAMP_NTZ hits that (case (DateType, _) => true). So Spark finds no common type for MAP<DATE, INT> and MAP<TIMESTAMP_NTZ, INT> in either ANSI mode, while this returns MAP<TIMESTAMP_NTZ, INT>. Values aren't filtered, so MAP<INT, DATE> and MAP<INT, TIMESTAMP_NTZ> do widen in Spark.

This changes array_position from main, where the Map arm required identical key and value types. In Spark 4.2.0, in both ANSI modes:

> SELECT array_position(array(map(DATE '2020-01-01', 1)), map(CAST('2020-01-01' AS TIMESTAMP_NTZ), 1));
[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] ... but it's ["ARRAY<MAP<DATE, INT>>", "MAP<TIMESTAMP_NTZ, INT>"]. SQLSTATE: 42K09

With this PR, the same query (with arrow_cast(..., 'Timestamp(Microsecond, None)') for the TIMESTAMP_NTZ) fails with INVALID_ORDERING_TYPE on "MAP<TIMESTAMP_NTZ, BIGINT>". array_insert in #70 doesn't check orderability, so it would accept arguments that Spark's array_insert rejects with ARRAY_FUNCTION_DIFF_TYPES.

Could we filter the key the way Spark does? With a helper next to is_orderable:

/// Spark's `Cast.forceNullable`, for the casts that these rules produce. Of those, Spark
/// marks only DATE to TIMESTAMP_NTZ as one that can return NULL.
fn force_nullable(from: &DataType, to: &DataType) -> bool {
    matches!((from, to), (DataType::Date32, DataType::Timestamp(_, None)))
}
Suggested change
let key = find_type(lk.data_type(), rk.data_type())?;
// Spark rejects a key type that the cast can make NULL (`Cast.forceNullable`).
let key = find_type(lk.data_type(), rk.data_type()).filter(|key| {
!force_nullable(lk.data_type(), key)
&& !force_nullable(rk.data_type(), key)
})?;

A unit test for both orders of the two maps, and an .slt record for the array_position query above, would cover it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 73b600c with your force_nullable helper. The unit tests cover both orders in both ANSI modes, a map value that still widens, and a DATE key with a TIMESTAMP map key that still widens. I added .slt records for your query, which now fails with ARRAY_FUNCTION_DIFF_TYPES, and for the value case. Both messages match Spark 4.2.0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in 73b600c with your force_nullable helper.

force_nullable gets the key type before tightest_common_type unwraps it, so a dictionary-encoded DATE key still widens: tightest_common_type(&map(Dictionary(Int32, Date32), Int32), &map(Timestamp(Microsecond, None), Int32), false) returns MAP<TIMESTAMP_NTZ, INT>. Could it unwrap the dictionary too?

fn force_nullable(from: &DataType, to: &DataType) -> bool {
    match from {
        DataType::Dictionary(_, value) => force_nullable(value, to),
        _ => matches!((from, to), (DataType::Date32, DataType::Timestamp(_, None))),
    }
}

A dictionary key in map_keys_do_not_widen_from_date_to_timestamp_ntz would cover it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6042452 with your force_nullable change. map_keys_do_not_widen_from_date_to_timestamp_ntz now also checks a Dictionary(Int32, Date32) key.

Comment on lines +226 to +238
Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
List(field) | LargeList(field) | FixedSizeList(field, _) => {
format!("ARRAY<{}>", spark_sql_type(field.data_type()))
}
Struct(fields) => format!(
"STRUCT<{}>",
fields
.iter()
.map(|f| format!(
"{}: {}{}",
f.name(),
spark_sql_type(f.data_type()),
if f.is_nullable() { "" } else { " NOT NULL" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark_sql_type prints Arrow names for three Spark types that Comet maps to Arrow and this crate produces (make_dt_interval returns Duration(Microsecond), make_time returns Time64(Nanosecond)). It also doesn't quote struct field names the way StructField.sql does with quoteIfNeeded. All four reach array_position's error text:

Query This PR Spark 4.2.0
array_position(array(make_dt_interval(1)), 'x') "ARRAY<Duration(µs)>" "ARRAY<INTERVAL DAY TO SECOND>"
array_position(array(make_ym_interval(1)), 'x') "ARRAY<Interval(YearMonth)>" "ARRAY<INTERVAL YEAR TO MONTH>"
array_position(array(make_time(1, 2, 3)), 'x') "ARRAY<Time64(ns)>" "ARRAY<TIME(6)>" (with the TIME type enabled)
array_position(array(named_struct('a b', 1)), 'x') "ARRAY<STRUCT<a b: BIGINT>>" "ARRAY<STRUCT<`a b`: INT NOT NULL>>"

The last row also differs in nullability and the literal's type, which this PR doesn't control. The first column for make_ym_interval used arrow_cast(1, 'Interval(YearMonth)'), since the crate has no make_ym_interval. Could we add these arms, and a helper for the quoting?

/// Spark's `QuotingUtils.quoteIfNeeded`.
fn quote_if_needed(name: &str) -> String {
    let mut chars = name.chars();
    let plain = chars
        .next()
        .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
        && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
    if plain {
        name.to_string()
    } else {
        format!("`{}`", name.replace('`', "``"))
    }
}
Suggested change
Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
List(field) | LargeList(field) | FixedSizeList(field, _) => {
format!("ARRAY<{}>", spark_sql_type(field.data_type()))
}
Struct(fields) => format!(
"STRUCT<{}>",
fields
.iter()
.map(|f| format!(
"{}: {}{}",
f.name(),
spark_sql_type(f.data_type()),
if f.is_nullable() { "" } else { " NOT NULL" }
Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
Interval(IntervalUnit::YearMonth) => "INTERVAL YEAR TO MONTH".to_string(),
Duration(_) => "INTERVAL DAY TO SECOND".to_string(),
// Arrow has no TIME precision. Spark's `make_time` returns TIME(6).
Time64(_) => "TIME(6)".to_string(),
List(field) | LargeList(field) | FixedSizeList(field, _) => {
format!("ARRAY<{}>", spark_sql_type(field.data_type()))
}
Struct(fields) => format!(
"STRUCT<{}>",
fields
.iter()
.map(|f| format!(
"{}: {}{}",
quote_if_needed(f.name()),
spark_sql_type(f.data_type()),
if f.is_nullable() { "" } else { " NOT NULL" }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 73b600c with your arms and quote_if_needed. I added .slt records for all four rows. For the struct I used a CAST(NULL AS INT) field, so the full message matches Spark 4.2.0 exactly.

…the common-type rules

- Unwrap dictionaries before the equality check, so two dictionary arguments
  reach the function as their value type and floats are normalized.
- Reject a map key type that the cast can make NULL, as Spark's
  findTypeForComplex does with Cast.forceNullable (DATE to TIMESTAMP_NTZ).
- Name YEAR TO MONTH and DAY TO SECOND intervals and TIME(6) the way Spark
  does, and quote struct field names with Spark's quoteIfNeeded.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove!

replies with changes here:

…map keys

The left == right shortcut returned two equal nested types unchanged, so a
dictionary inside them reached the function and -0.0 did not match 0.0.
Nested types now go through complex_type, which unwraps them.

force_nullable now looks through a dictionary key, so a dictionary-encoded
DATE key no longer widens to TIMESTAMP_NTZ.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

# Conflicts:
#	crates/spark/src/function/mod.rs
@andygrove
andygrove merged commit a3d8983 into apache:main Nov 18, 2026
8 checks passed
@andygrove
andygrove deleted the spark-common-type branch November 18, 2026 15:08
andygrove added a commit to andygrove/datafusion-spark that referenced this pull request Nov 18, 2026
…errors

Name interval, duration and time types as Spark does, quote struct field
names that need it, and use Arrow names for types Spark lacks.

Co-Authored-By: Claude <noreply@anthropic.com>
andygrove added a commit that referenced this pull request Nov 18, 2026
…350)

## Which issue does this PR close?

- Part of #346.

## Rationale for this change

Spark casts a function's arguments to the input types the function
declares, in one place. Here each function picks its accepted types in a
hand-written `Signature`, so some arguments that Spark casts are
rejected, and a rejected type gives DataFusion's error instead of
Spark's.

`sha2` shows both problems. Spark declares `(BINARY, INT)`:

| Query | Before | Spark 4.2.0 and this PR |
|---|---|---|
| `sha2('Spark', '256')` | `Function 'sha2' requires Int32, but received
String` | the SHA-256 hash |
| `sha2('Spark', 256.7::DOUBLE)` | `requires Int32, but received
Float64` | the SHA-256 hash |
| `sha2('Spark', 256.0::DECIMAL(10, 2))` | `requires Int32, but received
Decimal(10, 2)` | the SHA-256 hash |
| `sha2(1::INT, 256)` | `requires Binary, but received Int32` | ``
[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha2`
requires the "BINARY" type, however it has the type "INT". SQLSTATE:
42K09 `` |
| `sha2('Spark')` | `Function 'sha2' expects 2 arguments but received 1`
| `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha2` requires 2
parameters but the actual number is 1. ... `` |

Spark's message also names the argument's SQL text (`"1"`).
`coerce_types` sees only the types, so the message here says "it has the
type" instead.

## What changes are included in this PR?

- New `function/implicit_cast.rs`:
- `SparkInputType` mirrors the `AbstractDataType`s Spark functions
declare: the concrete scalar types, `Numeric`, `Integral`, `Decimal`,
`Array` and `OneOf` (Spark's `TypeCollection`).
- `implicit_cast` ports the private `implicitCast(DataType,
AbstractDataType)` from both `TypeCoercion` (ANSI mode off) and
`AnsiTypeCoercion` (ANSI mode on), with `Cast.canANSIStoreAssign` and
`DecimalType.forType`. The two differ, for example a string becomes a
BOOLEAN only with ANSI mode on.
- `SparkInputTypes` holds the declared types, the ANSI flag and the
function's `user_defined` `Signature`. A function returns its
`signature()`, calls `coerce` from `coerce_types`, and calls
`with_config` from `with_updated_config`. `coerce` returns Spark's
`UNEXPECTED_INPUT_TYPE` and `WRONG_NUM_ARGS` errors with the SQLSTATE.
- A string cast to BINARY keeps its layout (`Utf8View` becomes
`BinaryView`). Spark has no dictionary or fixed-size binary types, so a
dictionary argument has the type of its values, as in #345, and a
`FixedSizeBinary` argument is cast to BINARY. Before this PR `sha2`
rejected `FixedSizeBinary`. #349 covers encodings more broadly.
- `sha2` uses `Signature::user_defined` with Spark's `(BINARY, INT)`
declaration.

Left for later PRs:

- DATE and TIMESTAMP targets need the session time zone. They can come
with the first function that declares them.
- Struct and map types are left out for now. Spark functions often
declare the bare `StructType` or `MapType` (any struct or map, for
example `to_json` and `GetArrayStructFields`), and some declare
`AbstractMapType(key, value)`. They can be added when a function moving
onto this module needs them. Until then a struct or map argument to a
scalar declared type is rejected with `UNEXPECTED_INPUT_TYPE`, as in
Spark.
- The module is in its own file so it does not conflict with #345, which
adds `function/type_coercion.rs`. `spark_sql_type` is copied from #345,
and whichever PR lands second should use the other's copy. The two files
could merge once both land.
- The cast that runs is DataFusion's. With ANSI mode off Spark returns
NULL for `sha2('Spark', '256x')` and wraps `sha2('Spark', 4294967552)`
to 256, and both raise a cast error here. That is part of the CAST work
in #88.
- Most `SparkInputType` variants are unused until more functions move
over, so the enum has a scoped `allow(dead_code)`.

## Are these changes tested?

Yes. Unit tests in `implicit_cast.rs` cover each rule in both modes,
`OneOf`, arrays, dictionaries and the error text. New records in
`hash/sha2.slt` cover the casts, the errors and ANSI mode. I checked
`sha2.slt` against Spark 4.2.0 with the PySpark runner, and all 45
records match. Most of the new records fail without this change.

## Are there any user-facing changes?

`sha2` accepts STRING, FLOAT, DOUBLE and DECIMAL bit lengths, and its
errors for other types and for the wrong number of arguments now match
Spark's. There are no public API changes, since the new module is
`pub(crate)`.

This PR was prepared with AI assistance (Claude Code) and reviewed by
me.

---------

Co-authored-by: Claude <noreply@anthropic.com>
andygrove added a commit that referenced this pull request Nov 18, 2026
Closes #146.

**Stacked on #345**, which moves the common-type rules and Spark type
names into `function/type_coercion.rs`. It is merged into this branch,
so review the commits after the merge.

## Summary
- Adds Spark's `array_insert(array, pos, item)`, ported from DataFusion
Comet's `array_funcs/array_insert.rs` as a scalar UDF with type
coercion.
- 1-based index; negative counts from the end (`-1` appends). A position
past the end pads with NULLs. A negative position before the start puts
the item first, then NULL padding, then the original elements.
- NULL array or NULL position gives NULL; a NULL item is inserted as a
NULL element; index `0` fails with `INVALID_INDEX_OF_ZERO`.
- `SparkArrayInsert::new().with_legacy_negative_index(true)` gives
`spark.sql.legacy.negativeIndexInArrayInsert` behavior: `-1` inserts
before the last element, and a position before the start pads one more
NULL. Default is Spark 4's behavior. Once `SparkOptions` lands (#92) a
session could read the Spark config instead.

## Differences from Comet's version
- The size limit is checked before any NULL padding is allocated, so a
huge index fails with `COLLECTION_SIZE_LIMIT_EXCEEDED` instead of
allocating billions of NULLs.
- Index arithmetic is 64-bit, so `i32::MIN` reports a 2147483648-element
result (2147483649 in legacy mode). This matches Spark 4.2.0
(SPARK-56567); Spark 4.1.x and older overflow.

## Type rules
- The item and elements unify the way Spark's `findTightestCommonType`
does. Numeric types widen, and same-family types unify (Utf8/Utf8View,
timestamp units, DATE with TIMESTAMP, nested types differing only in
nullability). Spark has no common type for some pairs that DataFusion
would coerce, and these are rejected: string or binary with another
kind, decimal with float, two different decimal types, and an integer
with a decimal too narrow for its range. Nested lists, structs and maps
are checked the same way. The rules are the shared
`tightest_common_type` from #345, called with ANSI mode off as Spark's
`ArrayInsert` does, plus its non-ANSI struct check, so struct fields
pair by position and their types are not widened (Spark widens them only
with ANSI on). Map keys and values widen. `array_position` and
`array_remove` use the same rules.
- Type errors use Spark's error classes and type names, e.g. ``
[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] Input to `array_insert`
should have been "ARRAY" followed by a value with same element type, but
it's ["ARRAY<INT>", "STRING"]. SQLSTATE: 42K09 `` The message leaves out
Spark's `Cannot resolve "<expr>"` prefix, since the expression text
isn't available here.
- Spark requires an INT index, but DataFusion types bare integer
literals as BIGINT. Any integer type is accepted and cast to INT, so a
BIGINT outside the INT range fails the cast and doesn't wrap. Spark
rejects every BIGINT index. Non-integer indexes fail with
`UNEXPECTED_INPUT_TYPE`.
- The result's element field is always nullable, since padding can add
NULLs even when neither input has any.

## Testing
- Unit tests plus `array/array_insert.slt`. Expected values were
verified with pyspark==4.2.0 on 2026-11-15, including legacy mode. The
repo's PySpark runner skips most of the file because Spark has no
`make_array`/`arrow_typeof`, so the queries were checked as equivalent
Spark SQL using `array()`.
- `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D
warnings`, `cargo test -p datafusion-spark` and `cargo test -p
datafusion-spark-sqllogictest` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
andygrove added a commit that referenced this pull request Nov 18, 2026
Closes #142.

**Stacked on #345**, which moves the common-type rules, the orderability
check and Spark type names into `function/type_coercion.rs`. It is
merged into this branch, so review the commits after the merge.

## Summary

Adds a Spark-compatible `array_remove(array, element)`
(`SparkArrayRemove`), ported from DataFusion Comet's
`spark_array_remove`, and registers it as `array_remove`. It replaces
the DataFusion built-in of the same name when the Spark functions are
registered.

- It removes every element equal to `element`. NULL elements in the
array are always kept.
- It removes every matching element, where the DataFusion built-in
removes only the first one. It returns NULL when the array or the
element is NULL, as the built-in does.
- Equality follows Spark's ordering-based `genEqual`/`ordering.equiv` at
any depth of arrays and structs: `NaN == NaN`, `-0.0 == 0.0`, and NULLs
nested inside the compared values are equal. A constant element that is
not nested is compared with every value at once (in place for floats)
and the kept elements are found with bitmap operations. Otherwise the
comparison runs on float-normalized copies
(`float_normalize::normalize_array`, as in `array_position`). The kept
elements come from the original values, so `array_remove(array(-0.0,
1.0), 1.0)` returns `[-0.0]`, as in Spark.
- It supports `List` and `LargeList`, sliced inputs, and both scalar and
column arguments. A constant non-nested element is compared with arrow's
vectorized `eq` kernel. Nested elements and column elements use
`make_comparator`. Elements of NULL rows are not carried into the
result's values.
- `invoke_with_args` does the work itself and does not rely on
`simplify`.

## Differences from Comet's version

- Comet's `spark_array_remove` is only routed for float-containing
element types: other types use DataFusion's `array_remove_all`. It is
registered under a different name and uses `Signature::any(2)` with no
coercion, relying on Spark to have cast the arguments already. This
version handles every element type and does Spark's type coercion itself
(see below).
- Comet supports only 32-bit `List`. This version also supports
`LargeList`, and accepts `FixedSizeList` by coercing it to `List`.
- Comet has hand-written float kernels (a one-pass constant path that
reads the mask a word at a time, plus a per-row float path) and depends
on its `float_semantics`/`nested_comparison` helpers. This version has a
bitmap path for a constant element that is not nested, like Comet's
constant path, and otherwise reuses this crate's `normalize_array` plus
arrow's `make_comparator`. `benches/array_remove.rs` shows the bitmap
path is about 3.5x faster than the comparator path at 8192 rows. The
per-row float path is not ported.
- Comet keeps the elements of NULL rows in the child values (hidden by
the list's null buffer). This version drops them.
- Comet's unit tests are adapted rather than copied verbatim, because
they depend on Comet-only test helpers: float edge values as a column
and as a constant, nulls, structs with float fields, and scalar float32.
Tests for sliced lists, `LargeList`, and the coercion rules are added.

## Type rules

These are Spark's `ArrayRemove.inputTypes` / `checkInputDataTypes`. The
tightest-common-type logic, the non-ANSI struct field check and the
orderability check are shared with `array_position` through
`function/type_coercion.rs` (#345).

- The array element and the element are cast to
`findTightestCommonType`, and the result is an array of that type. For
example `array_remove(array(1, 2), 1.5D)` is `array<double>` and
`array_remove(array(1, 2), CAST(1 AS DECIMAL(12,2)))` is
`array<decimal(12,2)>`.
  - Numbers widen.
- An integral type widens to a decimal only if the decimal can hold all
its values.
  - DATE widens to TIMESTAMP.
- Pairs with no common type are an error: string and number, two
different decimals, decimal and float, and structs whose field names or
types differ.
  - All of these raise `[DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES]`.
- A first argument that is not an array raises
`ARRAY_FUNCTION_DIFF_TYPES`, as in Spark. This includes an untyped
`NULL`.
- A NULL-typed common element type is accepted, unlike `array_position`:
`array_remove(array(), NULL)` returns NULL with type `array<void>`.
- Map and INTERVAL elements, at any depth, raise
`[DATATYPE_MISMATCH.INVALID_ORDERING_TYPE]`.
- The error messages name Spark's types and end with the SQLSTATE, as
Spark's do.
- Struct field types are not widened, which matches Spark with ANSI mode
off. With ANSI on (Spark 4's default), Spark widens them:
`array_remove(array(named_struct('a', 1)), named_struct('a', 1L))`
returns `[]` in Spark with ANSI on, and is an error here. This is the
same choice `array_position` makes.

## Testing

- Unit tests in `array_remove.rs` cover:
- float edge values (both zeros, two NaN bit patterns, ±Inf, NULL) as a
column and as a constant, checking that the kept elements keep their
bits
  - NULL rows, NULL elements and a NULL element
  - sliced lists
  - `LargeList` scalars
  - struct elements with float fields
  - the coercion rules
- New `crates/sqllogictest/test_files/array/array_remove.slt` covers:
  - basics, NULL semantics, and error cases
  - float equality, including nested and in structs
  - nested NULL equality
  - type coercion
  - column inputs, including an `OFFSET`-sliced batch
  - `LargeList`
- Verified with pyspark 4.2.0:
- `pyspark_slt_runner.py` on `array_remove.slt`: 71 records pass with no
differences. The 6 records that use `arrow_typeof` are skipped, and
their result types (bigint, bigint, int, double, decimal(12,2)) were
checked by hand with `typeof`/schema in a local SparkSession.
- About 50 ad-hoc queries using Spark's `array()` were also checked by
hand, including the error classes for an untyped NULL array, decimal and
string mismatches, and maps.
- Commands, all passing:
  - `cargo fmt --all`
  - `cargo clippy --all-targets --all-features -- -D warnings`
  - `cargo test -p datafusion-spark` (341 unit tests + doc tests)
  - `cargo test -p datafusion-spark-sqllogictest` (250 of 250 files)
andygrove added a commit to andygrove/datafusion-spark that referenced this pull request Nov 18, 2026
Use spark_sql_type (apache#345) for the "not supported" and "not implemented
yet" errors, so they read "cast from BIGINT to TIMESTAMP_NTZ" rather
than "cast from Int64 to Timestamp(µs)", and check that parse_spark_type
round-trips with spark_sql_type.
andygrove added a commit that referenced this pull request Nov 18, 2026
…ernel (#317)

## Which issue does this PR close?

Part of #88: the "Rework the `spark_cast` UDF" item.

## Rationale for this change

`spark_cast(expr, 'timestamp')` only supported a `timestamp` target and
integer or float input, with its own conversion code. To test the
Spark-compatible cast kernel from SQL and SLT as each family is ported,
the function needs to accept any Spark type name and call the kernel.

## What changes are included in this PR?

- **Target type parser** (`cast/data_type.rs`): parses the second
argument with Spark's `dataType` grammar from `SqlBaseParser.g4`,
case-insensitively.
- **Primitive types:** `boolean`, `tinyint`/`byte`, `smallint`/`short`,
`int`/`integer`, `bigint`/`long`, `float`/`real`, `double`, `date`,
`binary` and `string`.
- **Decimals:** `decimal`/`dec`/`numeric`, with an optional precision
and scale. A bare `decimal` is `decimal(10, 0)`, and precision and scale
are validated.
- **Character types:** `char(n)` and `varchar(n)` become `STRING`, which
is what Spark casts them to by default. A missing length raises
`DATATYPE_MISSING_SIZE`, as in Spark.
  - **Timestamps:**
- `timestamp`, `timestamp_ltz` and `timestamp with local time zone` give
a `TIMESTAMP` tagged with the session time zone. That matches Spark's
default `spark.sql.timestampType`.
- `timestamp_ntz` and `timestamp without time zone` give a
`TIMESTAMP_NTZ`.
- **Complex types:** `array<…>`, `map<…, …>` (with Spark's `ArrowUtils`
field names `entries`, `key` and `value`), and `struct<…>`. Struct
fields can be backquoted, `NOT NULL` or have a `COMMENT`, and duplicate
field names are allowed, as in Spark.
- **Unsupported types:** intervals, `TIME`, `VARIANT`, `VOID`,
geospatial types and collated strings get an "unsupported" error.
Malformed type names get a parse error that names the problem.
- **`spark_cast` UDF:**
  - accepts any input type
  - requires the target type to be a string literal
  - computes its return field from the parsed type
- calls the kernel with `EvalMode::from_config` and the session time
zone as a `SparkTimeZone`

  The old conversion code in the UDF is removed.
- **Error messages:** the kernel's "not supported" and "not implemented
yet" errors name types the Spark way, using `spark_sql_type` from #345.
For example, they now say `cast from BIGINT to TIMESTAMP_NTZ` instead of
`cast from Int64 to Timestamp(µs)`.
- **Integer and float to `TIMESTAMP`:** ported into the kernel
(`cast/numeric.rs`). Spark has no cast from a number to `TIMESTAMP_NTZ`,
so that cast is rejected.

Float input follows Spark 4.2.0's `doubleToTimestamp` and
`doubleToTimestampAnsi`:

  | Mode | NaN or infinite | Out of range |
  |---|---|---|
  | Legacy | NULL | saturates, like Scala's `toLong` |
| ANSI | `CAST_INVALID_INPUT` | `CAST_OVERFLOW`, such as `1.0E25D` from
`DOUBLE` to `BIGINT` |
  | TRY | NULL | NULL |

The previous UDF already saturated in Legacy mode. Comet returns NULL
there, which differs from Spark.

## Are these changes tested?

Yes.

- **Unit tests:**
- the type parser: every primitive and complex type, invalid names and
unsupported types
- integer and float to timestamp in each eval mode, including
saturation, a NaN under a null, `FLOAT` widening and the error messages
- a round trip between `parse_spark_type` and `spark_sql_type` for each
canonical Spark type name
- the UDF: return fields, a non-literal target type, array and scalar
input, and reading ANSI mode from the config
- **`#[user_doc]`** for `spark_cast`, which documents the target-type
syntax and the two kinds of error.
- **New SLT file** `conversion/cast_target_type.slt`: return types from
`arrow_typeof`, the session time zone on `timestamp`, casts that already
work through Arrow, invalid and unsupported targets, and casts that are
not implemented yet.
- **Existing SLT files:** `cast_int_to_timestamp.slt` and
`cast_float_to_timestamp.slt` still pass, with these changes:
- The old "unsupported target" test used `spark_cast(100, 'string')`,
which now works, so it now uses `interval` instead.
  - It also now checks that a number can't be cast to `timestamp_ntz`.
  - The ANSI errors are checked against Spark's messages.
- It has a Legacy saturation test that casts the result back to
`bigint`. The old file said this couldn't be shown in SLT.

## Are there any user-facing changes?

Yes. `spark_cast` accepts any Spark type name as its target and any
input type.

- **Newly working:** casts that are already Spark-compatible through
Arrow now succeed. One example is `spark_cast(42, 'string')`.
- **Errors for casts not ported yet:** those return an explicit "not
supported" or "not implemented yet" error rather than a result that
differs from Spark's. A cast that Spark supports, such as TIMESTAMP or
DATE to a number, says "not implemented yet"; "not supported" is only
for casts Spark does not have. Integer narrowing is one example. The
unsuffixed literal `1` is a `BIGINT`, so even `spark_cast(1, 'int')`
needs that port.
- **Error message format:** in ANSI mode, errors for NaN and
out-of-range input now use Spark's `[CAST_INVALID_INPUT]` and
`[CAST_OVERFLOW]` messages.

_This PR was written by an LLM (Claude) from the Comet and Spark
sources, and reviewed before opening._

🤖 Generated with [Claude Code](https://claude.com/claude-code)
andygrove added a commit that referenced this pull request Nov 18, 2026
Closes #140.

**Stacked on #345**, which moves the orderability check and Spark type
names into `function/type_coercion.rs`. It is merged into this branch,
so review the commits after the merge.

## Summary

Adds Spark-compatible `array_max` and `array_min`, ported from Comet's
`native/spark-expr/src/array_funcs/array_extrema.rs`.

- NULL elements are skipped. A NULL array, an empty array, or an array
of only NULL
  elements returns NULL.
- Elements are compared with Spark's ordering:
- NaN is greater than every other value, including infinity, and all
NaNs are equal.
- `-0.0` equals `0.0`. On a tie the first element wins and is returned
unchanged, so
`array_max(array(-0.0, 0.0))` is `-0.0` and `array_max(array(0.0,
-0.0))` is `0.0`.
    The NaN payload bits are kept as well.
  - These float rules also apply inside nested arrays and structs.
  - Strings and binary compare bytewise.
- Arrays compare element by element and then by length. Structs compare
field by field.
    A NULL nested inside an array or struct sorts first.
- Supported element types: integers, floats, decimals, booleans,
strings, binary, dates,
timestamps, and nested arrays and structs of these. `List` and
`LargeList` are
  supported, and `FixedSizeList` is coerced to `List`.
- An untyped `NULL` argument and `array()` are accepted and return a
NULL of the NULL type,
  as in Spark (`void`).
- Planning errors use Spark's error classes:
  - `INVALID_ORDERING_TYPE` for map and INTERVAL elements at any depth.
  - `UNEXPECTED_INPUT_TYPE` for an argument that is not an array.
- The messages name Spark's types and end with the SQLSTATE, as Spark's
do.
- How it works:
- Primitive elements that Spark produces (integers, floats, decimals,
dates, timestamps)
    take a typed path: one
pass over the raw value buffer with no copy, no type-erased comparator
and no `take`.
Floats compare with `partial_cmp`, with NaN ranked above everything
else.
- All other types (strings, binary, booleans, nested arrays and structs,
intervals) take a
generic path. The values are canonicalized with the existing
`normalize_array` helper (NaN
to the canonical NaN, `-0.0` to `0.0`, at any depth) and compared with
arrow's
`make_comparator`, nulls first. The winner is then taken from the
original, un-normalized
    values.
- `invoke_with_args` does the work itself and does not depend on
`simplify`.
- The orderability check and the Spark type names come from
`function/type_coercion.rs` (#345).
- Both functions are documented with `#[user_doc]`.

## Differences from DataFusion's built-in `array_max` and `array_min`

These replace the built-ins when the Spark functions are registered. The
built-ins:

- order `-0.0` before `0.0`, so `array_max(array(-0.0, 0.0))` is `0.0`
and `array_min(array(0.0, -0.0))` is `-0.0`, where Spark returns the
first of the tied elements,
- sort a NULL nested inside an array or struct after other values, so
`array_max(array(array(1, NULL), array(1, 2)))` is `[1, NULL]`, where
Spark returns `[1, 2]`,
- reject a NULL argument at planning, where Spark returns NULL,
- fail at execution on map elements, where Spark rejects them at
planning.

Each of these has a test in `array/array_extrema.slt`, with the
built-in's result in a comment.

## Differences from Comet's version

- Comet wraps DataFusion's built-in `array_max`/`array_min`: it reuses
their signature and
return type, and hands non-float, non-nested element types to the
built-in. This version
has its own signature and `coerce_types`, and one code path for all
element types.
- Comet has a dedicated float scan. This version has a typed scan for
every primitive type,
  floats included.
- Comet also has a hand-written `take` for nested values, which keeps
child buffer capacity
bounded when a large list loses. This version uses arrow's `take` on the
generic path, so
`array_max` over arrays of large arrays can briefly over-reserve child
capacity. Both produce
  the same values.
- Comet supports only 32-bit `List`. This version also supports
`LargeList`, and coerces
  `FixedSizeList` to `List`.
- Comet does not check types. This version rejects map elements and
non-array arguments with
  Spark's error classes.

## Testing

- New unit tests in `array_extrema.rs`, adapted from Comet's tests,
plus:
- a randomized check that the typed float path picks bit-identical
results to the generic
path, over NaN payloads (including negative NaN), signed zeros,
infinities, NULLs and a
    sliced list. Swapping in IEEE `total_cmp` makes it fail.
- decimal precision/scale and timestamp time zone are kept by the typed
path
- The original tests:
- first-winner float semantics for f32 and f64: NaN payloads, signed
zeros, infinities,
    long lists
  - scalar and sliced input
  - strings and binary
  - `LargeList`
  - nested lists and structs
  - NULL-typed input
  - type rejection
- New `array/array_extrema.slt` (34 records). All 34 match Spark when
run with the PySpark
SLT runner on `pyspark==4.2.0`. I also checked the following directly in
PySpark 4.2.0:
- error messages for maps, non-arrays and the wrong number of arguments
  - the `void` result type for `NULL` and `array()`
  - that the zero sign is kept
- The sign of `-0.0` is checked with `CAST(... AS STRING)`, because the
test runner prints both
zeros as `0`. `atan2` cannot be used for this: Spark adds `0.0` to its
arguments, which turns
  `-0.0` into `0.0`.
- Not covered in the `.slt` file, because of literal typing (#97):
- `array_max(array(1.0, 2))`: Spark types the array as `DECIMAL(11,1)`
and returns `2.0`,
    while DataFusion returns a `DOUBLE`.
- Not tested:
- Interval elements. Spark orders `DAY TO SECOND` intervals, and PySpark
cannot collect
    `YEAR TO MONTH` intervals.
- The `ArrayMax`/`ArrayMin` definitions in the Spark 3.5.8 and 4.1.1
sources have the same
  logic. I did not do a full multi-version audit.
- New benchmark `benches/array_extrema.rs`: 8192 rows, lists of 0 to 31
elements, 10% NULL
  elements. Typed path vs generic path on this machine:

  | Benchmark | Generic | Typed | Change |
  |---|---|---|---|
  | `array_max` int64 | 443 µs | 184 µs | −57% |
  | `array_max` float64 | 491 µs | 298 µs | −39% |
  | `array_min` float64 | 485 µs | 297 µs | −38% |
| `array_max` utf8 (generic path, control) | 897 µs | 903 µs | no change
|

Canonicalizing floats and comparing with `total_cmp` instead was slower
(393 µs), so the
  `partial_cmp` version was kept.
- Commands, all passing:
  - `cargo fmt --all`
  - `cargo clippy --all-targets --all-features -- -D warnings`
  - `cargo test -p datafusion-spark`: 347 passed
- `cargo test -p datafusion-spark-sqllogictest`: 250 of 250 files passed
andygrove added a commit that referenced this pull request Nov 18, 2026
…de (#360)

## Which issue does this PR close?

- Relates to #88 and #346.

## Rationale for this change

#89, #304, #345 and #350 added a shared way to declare a function's
input types, find a common type, cast a value and read ANSI mode.
`CONTRIBUTING.md` does not mention any of them, so a new function is
likely to hand-write a `Signature`, call Arrow's `cast`, or take a
`bool` for ANSI mode instead.

## What changes are included in this PR?

- A new "Input types, casts and ANSI mode" section in `CONTRIBUTING.md`.
It says when to use `SparkInputTypes`, the rules in `type_coercion.rs`,
`spark_cast()` and `EvalMode`, and points to the module docs for
details.
- The audit skill's "Signature and coercion" and "ANSI handling" checks
now point at the guide.

The section says that `SparkInputTypes` only chooses target types and
that `spark_cast()` returns "not implemented" for casts that are not
ported yet (#88). I kept that wording on purpose, so it can be relaxed
as the cast port progresses.

## Are these changes tested?

Documentation only. I checked each name, path and example against
`apache/main` (`sha2.rs`, `parse_url.rs`, `array_position.rs`,
`implicit_cast.rs`, `type_coercion.rs`, `eval_mode.rs`, `cast/mod.rs`).

## Are there any user-facing changes?

No.

This PR was written with the help of an LLM (Claude Code).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants