Skip to content

feat(spark): cast function arguments to Spark's declared input types - #350

Merged
andygrove merged 5 commits into
apache:mainfrom
andygrove:spark-input-types
Nov 18, 2026
Merged

andygrove merged 5 commits into
apache:mainfrom
andygrove:spark-input-types

Conversation

@andygrove

@andygrove andygrove commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Spark casts a function's arguments to the input types the function declares, in one place. Here each function picks its accepted types in a hand-written Signature, so some arguments that Spark casts are rejected, and a rejected type gives DataFusion's error instead of Spark's.

sha2 shows both problems. Spark declares (BINARY, INT):

Query Before Spark 4.2.0 and this PR
sha2('Spark', '256') Function 'sha2' requires Int32, but received String the SHA-256 hash
sha2('Spark', 256.7::DOUBLE) requires Int32, but received Float64 the SHA-256 hash
sha2('Spark', 256.0::DECIMAL(10, 2)) requires Int32, but received Decimal(10, 2) the SHA-256 hash
sha2(1::INT, 256) requires Binary, but received Int32 [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha2` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09
sha2('Spark') Function 'sha2' expects 2 arguments but received 1 [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha2` requires 2 parameters but the actual number is 1. ...

Spark's message also names the argument's SQL text ("1"). coerce_types sees only the types, so the message here says "it has the type" instead.

What changes are included in this PR?

  • New function/implicit_cast.rs:
    • SparkInputType mirrors the AbstractDataTypes Spark functions declare: the concrete scalar types, Numeric, Integral, Decimal, Array and OneOf (Spark's TypeCollection).
    • implicit_cast ports the private implicitCast(DataType, AbstractDataType) from both TypeCoercion (ANSI mode off) and AnsiTypeCoercion (ANSI mode on), with Cast.canANSIStoreAssign and DecimalType.forType. The two differ, for example a string becomes a BOOLEAN only with ANSI mode on.
    • SparkInputTypes holds the declared types, the ANSI flag and the function's user_defined Signature. A function returns its signature(), calls coerce from coerce_types, and calls with_config from with_updated_config. coerce returns Spark's UNEXPECTED_INPUT_TYPE and WRONG_NUM_ARGS errors with the SQLSTATE.
    • A string cast to BINARY keeps its layout (Utf8View becomes BinaryView). Spark has no dictionary or fixed-size binary types, so a dictionary argument has the type of its values, as in refactor(spark): share Spark's common-type rules between functions #345, and a FixedSizeBinary argument is cast to BINARY. Before this PR sha2 rejected FixedSizeBinary. Decide how functions handle dictionary and run-end-encoded arguments #349 covers encodings more broadly.
  • sha2 uses Signature::user_defined with Spark's (BINARY, INT) declaration.

Left for later PRs:

  • DATE and TIMESTAMP targets need the session time zone. They can come with the first function that declares them.
  • Struct and map types are left out for now. Spark functions often declare the bare StructType or MapType (any struct or map, for example to_json and GetArrayStructFields), and some declare AbstractMapType(key, value). They can be added when a function moving onto this module needs them. Until then a struct or map argument to a scalar declared type is rejected with UNEXPECTED_INPUT_TYPE, as in Spark.
  • The module is in its own file so it does not conflict with refactor(spark): share Spark's common-type rules between functions #345, which adds function/type_coercion.rs. spark_sql_type is copied from refactor(spark): share Spark's common-type rules between functions #345, and whichever PR lands second should use the other's copy. The two files could merge once both land.
  • The cast that runs is DataFusion's. With ANSI mode off Spark returns NULL for sha2('Spark', '256x') and wraps sha2('Spark', 4294967552) to 256, and both raise a cast error here. That is part of the CAST work in [EPIC] Port Comet's Spark-compatible CAST #88.
  • Most SparkInputType variants are unused until more functions move over, so the enum has a scoped allow(dead_code).

Are these changes tested?

Yes. Unit tests in implicit_cast.rs cover each rule in both modes, OneOf, arrays, dictionaries and the error text. New records in hash/sha2.slt cover the casts, the errors and ANSI mode. I checked sha2.slt against Spark 4.2.0 with the PySpark runner, and all 45 records match. Most of the new records fail without this change.

Are there any user-facing changes?

sha2 accepts STRING, FLOAT, DOUBLE and DECIMAL bit lengths, and its errors for other types and for the wrong number of arguments now match Spark's. There are no public API changes, since the new module is pub(crate).

This PR was prepared with AI assistance (Claude Code) and reviewed by me.

Add `function/implicit_cast.rs`, a port of Spark's `implicitCast` from
`TypeCoercion` and `AnsiTypeCoercion`. A function declares its input
types with `SparkInputType`, as Spark's `ImplicitCastInputTypes` does,
and coerces its arguments through `SparkInputTypes`, which returns
Spark's `UNEXPECTED_INPUT_TYPE` and `WRONG_NUM_ARGS` errors.

`sha2` is the first function on it. It now accepts STRING, DOUBLE,
FLOAT and DECIMAL bit lengths as Spark does.

Part of apache#346.
Spark has no fixed-size binary, and sha2 has no kernel for it, so
FixedSizeBinary arguments failed with an internal error. The implicit
cast now gives them Spark's BINARY layout, as it does for dictionaries.

Also test that sha2 picks up ANSI mode through with_updated_config, and
say in its docs that the bit length is cast to INT.
…he signature

SparkInputTypes now builds the user_defined Signature, so a function
cannot pair declared input types with a different signature. Use
arrow's DataType predicates instead of local copies, and flatten a
few match arms.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. I compared implicit_cast with Spark 4.2.0's own TypeCoercion.implicitCast and AnsiTypeCoercion.implicitCast, called through PySpark's JVM, for 24 argument types (every scalar type, arrays, a map, a struct, and the interval types) against 19 declared types (every SparkInputType variant, arrays of three of them, and three OneOf collections). All 912 results match in both modes.

Comment on lines +249 to +255
OneOf(types) => {
return types
.iter()
.map(|t| t.simple_string())
.collect::<Vec<_>>()
.join(" or ");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark writes a type collection with parentheses, and toSQLType upper-cases the whole simpleString of an abstract type, " or " included (TypeCollection.simpleString, toSQLType). For AbstractArrayType(TypeCollection(StringType, BinaryType)), Spark 4.2.0's toSQLType returns "ARRAY<(STRING OR BINARY)>", and this returns "ARRAY<STRING or BINARY>". A top-level OneOf goes through sql() and is already right, so this only shows up inside Array.

Suggested change
OneOf(types) => {
return types
.iter()
.map(|t| t.simple_string())
.collect::<Vec<_>>()
.join(" or ");
}
OneOf(types) => {
return format!(
"({})",
types
.iter()
.map(|t| t.simple_string())
.collect::<Vec<_>>()
.join(" OR ")
);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 57e8256. A collection inside an array now reads ARRAY<(STRING OR BINARY)>. I added a test for that and for the top-level ("STRING" or "BINARY") form.

Comment on lines +444 to +453
fn ordinal_number(idx: usize) -> String {
const ORDINALS: [&str; 10] = [
"first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth",
"ninth", "tenth",
];
ORDINALS
.get(idx)
.map(|s| s.to_string())
.unwrap_or_else(|| format!("{}th", idx + 1))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark spells out only the first three ordinals (ordinalNumber). In Spark 4.2.0, SELECT overlay('Spark SQL', '_', 6, array(1)) reports The 4th parameter requires the "INT" type, where this would say "fourth". The ordinals test at line 755 would then check ordinal_number(3) as "4th".

Suggested change
fn ordinal_number(idx: usize) -> String {
const ORDINALS: [&str; 10] = [
"first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth",
"ninth", "tenth",
];
ORDINALS
.get(idx)
.map(|s| s.to_string())
.unwrap_or_else(|| format!("{}th", idx + 1))
}
fn ordinal_number(idx: usize) -> String {
match idx {
0 => "first".to_string(),
1 => "second".to_string(),
2 => "third".to_string(),
_ => format!("{}th", idx + 1),
}
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 57e8256. ordinal_number now matches ordinalNumber, and the test checks 4th and 11th.

…n errors

Spark spells out only the first three ordinals ("4th parameter"), and
writes a type collection inside an array as ARRAY<(STRING OR BINARY)>.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove.

Comment on lines +482 to +492
Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(),
List(field) | LargeList(field) | FixedSizeList(field, _) => {
format!("ARRAY<{}>", spark_sql_type(field.data_type()))
}
Struct(fields) => format!(
"STRUCT<{}>",
fields
.iter()
.map(|f| format!(
"{}: {}{}",
f.name(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This copy of spark_sql_type doesn't have the names that #345 added in 73b600c for my comment there: INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND, TIME(6), and quoted struct field names (#345's version). So the two copies print different names for the same type, and sha2's errors use the Arrow names:

Query This PR Spark 4.2.0
sha2(make_dt_interval(1), 256) Arrow's Duration name "INTERVAL DAY TO SECOND"
sha2(arrow_cast(1, 'Interval(YearMonth)'), 256) "Interval(YearMonth)" "INTERVAL YEAR TO MONTH"
sha2(make_time(1, 2, 3), 256) "Time64(ns)" "TIME(6)" (with the TIME type enabled)
sha2(named_struct('a b', CAST(NULL AS INT)), 256) "STRUCT<a b: INT>" "STRUCT<`a b`: INT>"

The Spark column for the year-month row is from make_ym_interval(1, 0), since the crate has no make_ym_interval. Could this copy match #345's, including quote_if_needed? The copies also differ the other way: this one names Decimal256, FixedSizeBinary, and Date64, and #345's prints their Arrow names. An .slt error record for sha2(make_dt_interval(1), 256) would cover it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in d3f29ae. spark_sql_type now matches #345's copy: INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND, TIME(6), and struct field names go through quote_if_needed. Decimal256, FixedSizeBinary and Date64 print their Arrow names, as in #345. I added a unit test for the names and .slt error records for sha2(make_dt_interval(1), 256), Interval(YearMonth) and a struct with a quoted field name.

…errors

Name interval, duration and time types as Spark does, quote struct field
names that need it, and use Arrow names for types Spark lacks.

Co-Authored-By: Claude <noreply@anthropic.com>

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

@andygrove
andygrove merged commit 314b22c into apache:main Nov 18, 2026
8 checks passed
@andygrove
andygrove deleted the spark-input-types branch November 18, 2026 16:27
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @mbutrovich!

andygrove added a commit that referenced this pull request Nov 18, 2026
…de (#360)

## Which issue does this PR close?

- Relates to #88 and #346.

## Rationale for this change

#89, #304, #345 and #350 added a shared way to declare a function's
input types, find a common type, cast a value and read ANSI mode.
`CONTRIBUTING.md` does not mention any of them, so a new function is
likely to hand-write a `Signature`, call Arrow's `cast`, or take a
`bool` for ANSI mode instead.

## What changes are included in this PR?

- A new "Input types, casts and ANSI mode" section in `CONTRIBUTING.md`.
It says when to use `SparkInputTypes`, the rules in `type_coercion.rs`,
`spark_cast()` and `EvalMode`, and points to the module docs for
details.
- The audit skill's "Signature and coercion" and "ANSI handling" checks
now point at the guide.

The section says that `SparkInputTypes` only chooses target types and
that `spark_cast()` returns "not implemented" for casts that are not
ported yet (#88). I kept that wording on purpose, so it can be relaxed
as the cast port progresses.

## Are these changes tested?

Documentation only. I checked each name, path and example against
`apache/main` (`sha2.rs`, `parse_url.rs`, `array_position.rs`,
`implicit_cast.rs`, `type_coercion.rs`, `eval_mode.rs`, `cast/mod.rs`).

## Are there any user-facing changes?

No.

This PR was written with the help of an LLM (Claude Code).
andygrove added a commit that referenced this pull request Nov 18, 2026
## Which issue does this PR close?

- Part of #353.

## Rationale for this change

Spark declares `crc32`'s input type as BINARY, and casts a STRING
argument to it. `crc32` already accepts the same types, but it rejects
the others with DataFusion's errors instead of Spark's:

| Query | Before | This PR |
|---|---|---|
| `crc32(1::INT)` | `Function 'crc32' requires Binary, but received
Int32` ... | `` [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first
parameter of `crc32` requires the "BINARY" type, however it has the type
"INT". SQLSTATE: 42K09 `` |
| `crc32('a', 'b')` | `Function 'crc32' expects 1 arguments but received
2` ... | `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `crc32` requires 1
parameters but the actual number is 2. ... `` |

## What changes are included in this PR?

`crc32` declares Spark's `Crc32.inputTypes` with `SparkInputTypes` from
#350, the same way `sha2` does.

## Are these changes tested?

Yes. New records in `hash/crc32.slt` cover a dictionary-encoded string,
a column with NULLs, the errors for other types and for the wrong number
of arguments, and ANSI mode. The error records fail without this change.
I checked `crc32.slt` against Spark 4.2.0 with the PySpark runner, and
every record matches. The runner does not compare error messages, so I
checked them against Spark by hand. Spark's message also names the
argument's SQL text, as noted in #350.

## Are there any user-facing changes?

Only the error messages above. The results do not change.

This PR was prepared with AI assistance (Claude Code) and reviewed by
me.
andygrove added a commit that referenced this pull request Nov 18, 2026
## Which issue does this PR close?

- Part of #353.

## Rationale for this change

Spark declares `sha1`'s input type as BINARY, and casts a STRING
argument to it. `sha1` already accepts the same types, but it rejects
the others with DataFusion's errors instead of Spark's:

| Query | Before | This PR |
|---|---|---|
| `sha1(1::INT)` | `Function 'sha1' requires Binary, but received Int32`
... | `` [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter
of `sha1` requires the "BINARY" type, however it has the type "INT".
SQLSTATE: 42K09 `` |
| `sha1('a', 'b')` | `Function 'sha1' expects 1 arguments but received
2` ... | `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha1` requires 1
parameters but the actual number is 2. ... `` |

## What changes are included in this PR?

`sha1` declares Spark's `Sha1.inputTypes` with `SparkInputTypes` from
#350, the same way `sha2` does.

## Are these changes tested?

Yes. New records in `hash/sha1.slt` cover a dictionary-encoded string, a
column with NULLs, the errors for other types and for the wrong number
of arguments, and ANSI mode. The error records fail without this change.
I checked `sha1.slt` against Spark 4.2.0 with the PySpark runner, and
every record matches. The runner does not compare error messages, so I
checked them against Spark by hand. Spark's message also names the
argument's SQL text, as noted in #350.

## Are there any user-facing changes?

Only the error messages above. The results do not change.

This PR was prepared with AI assistance (Claude Code) and reviewed by
me.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants