Skip to content

feat(spark): add Spark-compatible array_insert function - #70

Merged
andygrove merged 15 commits into
apache:mainfrom
andygrove:feat-spark-array-insert
Nov 18, 2026
Merged

andygrove merged 15 commits into
apache:mainfrom
andygrove:feat-spark-array-insert

Conversation

@andygrove

@andygrove andygrove commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Closes #146.

Stacked on #345, which moves the common-type rules and Spark type names into function/type_coercion.rs. It is merged into this branch, so review the commits after the merge.

Summary

  • Adds Spark's array_insert(array, pos, item), ported from DataFusion Comet's array_funcs/array_insert.rs as a scalar UDF with type coercion.
  • 1-based index; negative counts from the end (-1 appends). A position past the end pads with NULLs. A negative position before the start puts the item first, then NULL padding, then the original elements.
  • NULL array or NULL position gives NULL; a NULL item is inserted as a NULL element; index 0 fails with INVALID_INDEX_OF_ZERO.
  • SparkArrayInsert::new().with_legacy_negative_index(true) gives spark.sql.legacy.negativeIndexInArrayInsert behavior: -1 inserts before the last element, and a position before the start pads one more NULL. Default is Spark 4's behavior. Once SparkOptions lands (Move SparkOptions into datafusion-spark as a config extension #92) a session could read the Spark config instead.

Differences from Comet's version

  • The size limit is checked before any NULL padding is allocated, so a huge index fails with COLLECTION_SIZE_LIMIT_EXCEEDED instead of allocating billions of NULLs.
  • Index arithmetic is 64-bit, so i32::MIN reports a 2147483648-element result (2147483649 in legacy mode). This matches Spark 4.2.0 (SPARK-56567); Spark 4.1.x and older overflow.

Type rules

  • The item and elements unify the way Spark's findTightestCommonType does. Numeric types widen, and same-family types unify (Utf8/Utf8View, timestamp units, DATE with TIMESTAMP, nested types differing only in nullability). Spark has no common type for some pairs that DataFusion would coerce, and these are rejected: string or binary with another kind, decimal with float, two different decimal types, and an integer with a decimal too narrow for its range. Nested lists, structs and maps are checked the same way. The rules are the shared tightest_common_type from refactor(spark): share Spark's common-type rules between functions #345, called with ANSI mode off as Spark's ArrayInsert does, plus its non-ANSI struct check, so struct fields pair by position and their types are not widened (Spark widens them only with ANSI on). Map keys and values widen. array_position and array_remove use the same rules.
  • Type errors use Spark's error classes and type names, e.g. [DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES] Input to `array_insert` should have been "ARRAY" followed by a value with same element type, but it's ["ARRAY<INT>", "STRING"]. SQLSTATE: 42K09 The message leaves out Spark's Cannot resolve "<expr>" prefix, since the expression text isn't available here.
  • Spark requires an INT index, but DataFusion types bare integer literals as BIGINT. Any integer type is accepted and cast to INT, so a BIGINT outside the INT range fails the cast and doesn't wrap. Spark rejects every BIGINT index. Non-integer indexes fail with UNEXPECTED_INPUT_TYPE.
  • The result's element field is always nullable, since padding can add NULLs even when neither input has any.

Testing

  • Unit tests plus array/array_insert.slt. Expected values were verified with pyspark==4.2.0 on 2026-11-15, including legacy mode. The repo's PySpark runner skips most of the file because Spark has no make_array/arrow_typeof, so the queries were checked as equivalent Spark SQL using array().
  • cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest pass.

🤖 Generated with Claude Code

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The place() index math matches Spark in both modes, the size limit equals ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH, and the row building through one MutableArrayData with bulk NULL padding looks good. This conflicts with #63 in array/mod.rs.

} else if let Some(common) = binary_numeric_coercion(elem_type, item) {
common
} else {
return exec_err!(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any element and item types that differ and are not both numeric end here. binary_numeric_coercion returns None for non-numeric pairs, so array_insert(col, 1, 'x') fails when col holds Utf8View elements (the parquet default) and the literal is Utf8. The same goes for Utf8 and LargeUtf8, different timestamp units, and nested types that differ only in nullability. Spark's findTightestCommonType accepts these. Could we fall back to type_union_resolution(&[elem_type.clone(), item.clone()]) after the numeric case? A Utf8View case in the .slt would cover it. I have not run it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Applied in 6122a04, with one change: type_union_resolution on its own coerces INT plus STRING to STRING, so array_insert(array(1), 1, 'x') would succeed where Spark 4.2.0 raises DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES. The fallback now runs after the numeric case, with a check that rejects any pair mixing string or binary with a different kind, at any nesting depth. Utf8View/LargeUtf8, timestamp units and nullability-only differences are now accepted. Added .slt cases for a Utf8View literal and column, LargeUtf8, and the INT/STRING error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback now runs after the numeric case, with a check that rejects any pair mixing string or binary with a different kind, at any nesting depth.

The fallback also accepts structs that Spark rejects, because type_union_resolution matches struct fields by name. With this PR, array_insert(make_array(named_struct('a', 1::INT, 'b', 2::INT)), 1, named_struct('b', 3::INT, 'a', 4::INT)) returns [{b: 3, a: 4}, {b: 2, a: 1}]. Spark 4.2.0 raises ARRAY_FUNCTION_DIFF_TYPES for the same query in both ANSI modes, because findTypeForComplex pairs struct fields by position. Struct field types also widen here: named_struct('a', 1::INT) with named_struct('a', 2::BIGINT) gives List(Struct("a": Int64)). Spark widens them only with ANSI on, and array_position on main rejects them to follow ANSI off.

Could we drop spark_rejects_pair and the DataFusion fallback and call tightest_common_type from array_position.rs, as #330 does? It already implements findTightestCommonType for these types and pairs struct fields by position, so the two functions would accept the same types. My review of #332 proposes moving it, with the Spark type names that spark_type_name here renders, into one shared module. It needs one fix first. Its Map arm recurses into the entries struct, whose arm requires identical field types, so map<int, int> and map<bigint, bigint> have no common type. Spark widens the key and the value in both ANSI modes, and the existing "Maps whose key and value types widen" case in array_insert.slt checks it. A .slt case with reordered struct fields would cover the struct point.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 918dc42. spark_rejects_pair, the DataFusion fallback and the local type-name helpers are gone. coerce_types calls tightest_common_type(elem, item, false) from #345 (which moved it, with the map key and value widening fix, into function/type_coercion.rs), together with its non-ANSI struct_field_types_match, as array_position and array_remove do. New .slt cases cover reordered struct fields and an INT vs BIGINT struct field, both ARRAY_FUNCTION_DIFF_TYPES. The existing map widening case passes. I confirmed all three on PySpark 4.2.0. This PR is now stacked on #345.


// Expected values are taken from Comet's Spark-verified array_insert tests.
#[test]
fn positive_positions() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

positive_positions, negative_positions, the NULL item test, zero_index_is_an_error and the i32::MAX half of the size limit test repeat what array_insert.slt asserts. CI runs only cargo test -p datafusion-spark-sqllogictest, not cargo test -p datafusion-spark, so the Rust unit tests here do not run in CI. Could they move to .slt, keeping Rust tests only for legacy mode and i32::MIN? Cases that neither file covers are legacy mode with a position before the start (-5 on [1] should give 6 elements: the item, four NULLs, then 1), the exact boundary -(len + 1), NULL padding for string or nested elements, and a sliced list.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6122a04. The duplicated Rust tests are removed and the missing cases moved to .slt. Added the -(len + 1) boundary, -5 on [1], and NULL padding for string and nested elements. The Rust tests that remain cover legacy mode (including -5 on [1] giving 6 elements), i32::MIN in both modes, and a new sliced-list test, since SQL can't produce a sliced list.

andygrove and others added 4 commits November 13, 2026 10:47
Port Comet's array_insert as a scalar UDF.
Fall back to type_union_resolution when the element and item types are not
both numeric, so Utf8View/LargeUtf8 elements accept a Utf8 item, timestamps of
different units unify, and nested types differing only in nullability are
accepted. Pairs that mix a string or binary type with another kind are still
rejected, since type_union_resolution would coerce INT and STRING to STRING
while Spark raises ARRAY_FUNCTION_DIFF_TYPES.

Move the Rust tests that duplicated array_insert.slt into the .slt, keeping
Rust tests only for legacy mode, i32::MIN and a sliced list. Add slt coverage
for the -(len + 1) boundary, NULL padding with string and nested elements,
Utf8View and LargeUtf8 elements, and a mismatched item type. All new expected
values were checked against Spark 4.2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove force-pushed the feat-spark-array-insert branch from 76bc4e8 to 6122a04 Compare November 13, 2026 17:53
andygrove and others added 2 commits November 15, 2026 11:30
- Reject element/item pairs Spark's findTightestCommonType has no type for:
  decimal with float, decimals of different precision or scale, and an
  integer with a decimal that cannot hold its range. Recurse into maps too.
- Prefix type errors with Spark's DATATYPE_MISMATCH.ARRAY_FUNCTION_DIFF_TYPES
  and UNEXPECTED_INPUT_TYPE classes and Spark type names.
- Make with_legacy_negative_index a chainable builder.
- Return a scalar for all-scalar input, size the child buffer exactly, and
  avoid a panic on an unexpected index type.
- Add slt coverage for decimals, maps, DATE with TIMESTAMP, SMALLINT/TINYINT
  and out-of-range BIGINT positions, and a column position over the size
  limit; assert full error text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andygrove pushed a commit that referenced this pull request Nov 16, 2026
)

## Summary

Addresses #46 by migrating the Spark 4 `encode` charset compatibility
fix to the dedicated `datafusion-spark` repository, following maintainer
instructions on apache/datafusion#27260 and #87.

In Spark 4.0, `encode` tightened charset validation and error handling
in two key ways:
1. **Canonical Charset Whitelist:** Rejects Java charset aliases
(`UTF8`, `LATIN1`, `ASCII`, `UTF-32BE`, `UTF-32LE`, etc.) with
`[INVALID_PARAMETER_VALUE.CHARSET]`, accepting only the 7 canonical
Spark charsets: `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16`, `UTF-16BE`,
`UTF-16LE`, and `UTF-32`.
2. **Unmappable Character Handling:** Raises
`[MALFORMED_CHARACTER_CODING] ... SQLSTATE: 22000` instead of silently
replacing unmappable characters with `?`.

This PR implements Spark 4.0 strict semantics by default while providing
programmatic builder methods to restore pre-Spark 4.0 legacy behavior:
- `SparkEncode::with_legacy_java_charsets(true)` restores legacy Java
charset alias resolution (Spark 3.5 behavior).
- `SparkEncode::with_legacy_coding_error_action(true)` restores legacy
replacement of unmappable characters with `?` (Spark 3.5 behavior).
- `SparkEncode::with_options(legacy_java_charsets,
legacy_coding_error_action)` configures both switches.

## Architectural & Configuration Compatibility

Per #92 and #93, `datafusion-spark` is moving toward local configuration
extensions and version-derived flags. Following the precedent
established in PR #70 (`SparkArrayInsert::with_legacy_negative_index`),
`SparkEncode` defines explicit builder methods on the struct. This
allows downstream consumers (such as Comet) to configure legacy behavior
directly, and provides the exact hooks needed once `SparkOptions` (#92)
is merged into the crate.

## Changes

- **`crates/spark/src/function/string/encode.rs`**:
- Enforce canonical 7-charset whitelist in strict mode; return
`[INVALID_PARAMETER_VALUE.CHARSET]` for non-canonical names.
  - Case-insensitive matching for canonical names.
- Raise `[MALFORMED_CHARACTER_CODING]` (SQLSTATE 22000) for unmappable
characters in `US-ASCII` and `ISO-8859-1`.
- Provide `with_legacy_java_charsets`,
`with_legacy_coding_error_action`, and `with_options` builder methods.
- Validate scalar charsets upfront so invalid charsets fail immediately
even if the input is empty.
  - Update `#[user_doc]` documentation and examples.
- Add 10 comprehensive unit tests covering canonical sets, case
insensitivity, alias rejection, legacy mode acceptance, unmappable
characters, independent flags, nulls, and array inputs.
- **`crates/sqllogictest/test_files/string/encode.slt`**:
- Update test cases to expect Spark 4 strict errors
(`INVALID_PARAMETER_VALUE.CHARSET` for `utf-32be`, `utf-32le`, `EBCDIC`;
`MALFORMED_CHARACTER_CODING` for `é` in `us-ascii` and `Ā` in
`iso-8859-1`).

## Validation

- `cargo fmt --all -- --check` (clean)
- `cargo clippy --all-targets --all-features -- -D warnings` (clean, 0
warnings)
- `cargo test -p datafusion-spark -- encode` (10/10 focused unit tests
passed)
- `cargo test -p datafusion-spark` (341/341 crate tests passed)
- `cargo test -p datafusion-spark-sqllogictest -- string/encode`
(passed)
- `cargo test -p datafusion-spark-sqllogictest` (245/245 test files
passed, 100%)
- `git diff --check` (clean)

Closes #46.

---

*AI assistance disclosure: This implementation was prepared and verified
with the assistance of Claude Code / Antigravity.*

---------

Signed-off-by: Subodh Bansode <subodh171011@gmail.com>
# Conflicts:
#	crates/spark/src/function/array/mod.rs

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The index math and the size-limit handling match Spark 4.2.0, including legacy mode.

Comment on lines +417 to +442
if pos == 0 {
return exec_err!(
"[INVALID_INDEX_OF_ZERO] The index 0 is invalid. An index shall be either < 0 or > 0 (the first element has index 1)."
);
}
let n = len as i64;
let pos = i64::from(pos);
// (original elements before the item, leading NULLs, trailing NULLs, new length)
let (before, leading, trailing, new_len) = if pos > 0 {
let idx = pos - 1;
if idx <= n {
(idx, 0, 0, n + 1)
} else {
(n, 0, idx - n, pos)
}
} else if -pos > n {
let new_len = -pos + i64::from(legacy);
(0, new_len - n - 1, 0, new_len)
} else {
(n + pos + i64::from(!legacy), 0, 0, n + 1)
};
if new_len > MAX_ROUNDED_ARRAY_LENGTH {
return exec_err!(
"[COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] Can't create array with {new_len} elements which exceeding the array size limit {MAX_ROUNDED_ARRAY_LENGTH}, unsuccessful try to create arrays in the function `array_insert`."
);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark 4.2.0 ends both of these messages with the SQLSTATE: ... (the first element has index 1). SQLSTATE: 22003 and ... in the function `array_insert`. SQLSTATE: 54000. Other hand-built runtime errors in the crate include it, for example encode. Could we append SQLSTATE: 22003 and SQLSTATE: 54000? The .slt patterns for the size limit don't anchor the end, so they keep passing, and the INVALID_INDEX_OF_ZERO cases could assert the full text.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 918dc42, with 22003 and 54000, and the INVALID_INDEX_OF_ZERO and size-limit cases now assert the full text. The two planning errors also end with SQLSTATE: 42K09 now, as in Spark.

Comment on lines +484 to +490
let values = list.values().to_data();
let item_data = item.to_data();
let mut out = MutableArrayData::with_capacities(
vec![&values, &item_data],
true,
Capacities::Array(total),
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MutableArrayData asserts that all of its inputs have the same type, so this panics when the item's type differs from the element type. coerce_types prevents that in SQL, but Comet calls invoke_with_args() without coercion. Calling invoke_with_args with a List(Int32) array and an Int64 item panics with Arrays with inconsistent types passed to MutableArrayData. Comet's version checks the types and returns an internal error. Could we keep that check, with a unit test for it?

Suggested change
let values = list.values().to_data();
let item_data = item.to_data();
let mut out = MutableArrayData::with_capacities(
vec![&values, &item_data],
true,
Capacities::Array(total),
);
let values = list.values().to_data();
let item_data = item.to_data();
if values.data_type() != item_data.data_type() {
return internal_err!(
"array_insert expects the item type {} to match the element type {}",
item_data.data_type(),
values.data_type()
);
}
let mut out = MutableArrayData::with_capacities(
vec![&values, &item_data],
true,
Capacities::Array(total),
);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 918dc42, with mismatched_item_type_is_an_error calling invoke_with_args with a List(Int32) array and an Int64 item.

Move tightest_common_type from array_position into function/type_coercion,
with an ANSI flag, a findTypeForComplex that widens map keys and values,
Spark's isOrderable (which also rejects INTERVAL), and Spark type names
for error messages. Move float_normalize up to function/ so functions
outside array can use it.

array_position now reports a map with different key or value types as
INVALID_ORDERING_TYPE, rejects INTERVAL elements, and names Spark types
and the SQLSTATE in its errors, as Spark does.
Spark's MapType has no field names, so map entries named differently
still have a common type. Keep contains_float private.
Use DataType::is_string and TimeUnit's ordering, match map entries
without collecting, keep complex_type private, and move the non-ANSI
struct field check and same_type into type_coercion beside the rules
they qualify.
Replace the DataFusion coercion fallback, which paired struct fields by
name and widened their types, with the shared tightest_common_type and
the non-ANSI struct field check, as array_position and array_remove do.
End both runtime errors with Spark's SQLSTATE, and return an error
rather than panic when the item type differs from the element type.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. This approves what #70 adds on top of #345. I have nothing to add beyond the existing threads.

andygrove added a commit that referenced this pull request Nov 18, 2026
)

## Which issue does this PR close?

- Relates to #332, #331, #330 and #70, which each carry their own copy
of these rules. Requested in [this review
comment](#332 (comment)).

## Rationale for this change

Functions that need one type for several arguments (`array_position`
now, `greatest`, `least`, `array_remove`, `array_max`, `array_min` and
`array_insert` in open PRs) each carry a copy of Spark's common-type
rules. The copies have already drifted apart. This PR gives them one
shared copy, so fixes land once.

It also fixes two `array_position` differences from Spark 4.2.0:

| Query | Before | Spark 4.2.0 |
|---|---|---|
| `array_position(array(map(1, 1)), map(1L, 1L))` |
`ARRAY_FUNCTION_DIFF_TYPES` | `INVALID_ORDERING_TYPE` on `"MAP<BIGINT,
BIGINT>"` |
| `array_position(array(make_interval(1)), make_interval(2))` | `1` |
`INVALID_ORDERING_TYPE` on `"INTERVAL"` |

## What changes are included in this PR?

- New `function/type_coercion.rs` with:
- `tightest_common_type`: Spark's `findTightestCommonType`. It takes an
ANSI flag, because `AnsiTypeCoercion` widens an integer and a FLOAT to
DOUBLE.
- `complex_type`: Spark's `findTypeForComplex`. It now widens map keys
and values, matched by position because Spark's `MapType` has no field
names. As in Spark, a map key does not widen from DATE to TIMESTAMP_NTZ,
because that cast can return NULL. The old copy required identical entry
types.
- `is_orderable`: Spark's `OrderUtils.isOrderable`. It now also rejects
`Interval(MonthDayNano)`, which is Spark's `CalendarIntervalType`.
- `spark_sql_type`: the names Spark's `toSQLType` gives, for error
messages. This includes `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO
SECOND` and `TIME(6)`, and struct field names quoted the way Spark
quotes them.
- A dictionary-encoded argument, such as a Parquet string column, has
the type of its values, since Spark has no dictionary types. Broader
handling of dictionary and run-end-encoded arguments is #349.
- `struct_field_types_match`: with ANSI mode off, Spark's implicit cast
cannot change a struct's field types, so a function that casts its
arguments to their tightest common type rejects structs whose field
types differ.
- `float_normalize` moves from `function/array/` to `function/`, so
functions outside `array` can use it.
- `array_position` uses the shared module and passes `ansi_mode =
false`, because Spark's `ArrayPosition` calls the non-ANSI
`TypeCoercion`. Its errors now name Spark types and end with the
SQLSTATE, as Spark's do. It also calls `struct_field_types_match`, which
keeps its current behavior for structs.
- For two string or binary layouts, `array_position` now picks the
layout that holds both (for example `LargeUtf8` over `Utf8`) instead of
the left one. Spark has one string type, so this has no effect a user
can see.

The wider-type functions (`findWiderTypeWithoutStringPromotion`,
`findWiderTypeForDecimal`) are left out of this PR because nothing on
`main` calls them yet. #332 adds them when it moves to this module.

## Are these changes tested?

Yes. There are unit tests in `type_coercion.rs`, and new `.slt` cases in
`array/array_position.slt` cover map widening, flat and nested
intervals, and the full error text. I checked them against Spark 4.2.0
with the PySpark runner. The runner flags two records, and both
mismatches were already there on `main`: line 129 (Spark's 0-based `[]`)
and line 308 (struct names that differ only in case, already documented
in the file).

## Are there any user-facing changes?

`array_position` error messages change as described above, and it
rejects INTERVAL elements. There are no API changes.

This PR was prepared with AI assistance (Claude Code) and reviewed by
me.
# Conflicts:
#	crates/spark/src/function/mod.rs
#	crates/spark/src/function/type_coercion.rs
#	crates/sqllogictest/test_files/array/array_position.slt

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

@andygrove
andygrove merged commit c796833 into apache:main Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port array_insert from Comet

3 participants