Repository navigation
feat(spark): cast function arguments to Spark's declared input types - #350
Conversation
Add `function/implicit_cast.rs`, a port of Spark's `implicitCast` from `TypeCoercion` and `AnsiTypeCoercion`. A function declares its input types with `SparkInputType`, as Spark's `ImplicitCastInputTypes` does, and coerces its arguments through `SparkInputTypes`, which returns Spark's `UNEXPECTED_INPUT_TYPE` and `WRONG_NUM_ARGS` errors. `sha2` is the first function on it. It now accepts STRING, DOUBLE, FLOAT and DECIMAL bit lengths as Spark does. Part of apache#346.
Spark has no fixed-size binary, and sha2 has no kernel for it, so FixedSizeBinary arguments failed with an internal error. The implicit cast now gives them Spark's BINARY layout, as it does for dictionaries. Also test that sha2 picks up ANSI mode through with_updated_config, and say in its docs that the bit length is cast to INT.
…he signature SparkInputTypes now builds the user_defined Signature, so a function cannot pair declared input types with a different signature. Use arrow's DataType predicates instead of local copies, and flatten a few match arms.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. I compared implicit_cast with Spark 4.2.0's own TypeCoercion.implicitCast and AnsiTypeCoercion.implicitCast, called through PySpark's JVM, for 24 argument types (every scalar type, arrays, a map, a struct, and the interval types) against 19 declared types (every SparkInputType variant, arrays of three of them, and three OneOf collections). All 912 results match in both modes.
| OneOf(types) => { | ||
| return types | ||
| .iter() | ||
| .map(|t| t.simple_string()) | ||
| .collect::<Vec<_>>() | ||
| .join(" or "); | ||
| } |
There was a problem hiding this comment.
Spark writes a type collection with parentheses, and toSQLType upper-cases the whole simpleString of an abstract type, " or " included (TypeCollection.simpleString, toSQLType). For AbstractArrayType(TypeCollection(StringType, BinaryType)), Spark 4.2.0's toSQLType returns "ARRAY<(STRING OR BINARY)>", and this returns "ARRAY<STRING or BINARY>". A top-level OneOf goes through sql() and is already right, so this only shows up inside Array.
| OneOf(types) => { | |
| return types | |
| .iter() | |
| .map(|t| t.simple_string()) | |
| .collect::<Vec<_>>() | |
| .join(" or "); | |
| } | |
| OneOf(types) => { | |
| return format!( | |
| "({})", | |
| types | |
| .iter() | |
| .map(|t| t.simple_string()) | |
| .collect::<Vec<_>>() | |
| .join(" OR ") | |
| ); | |
| } |
There was a problem hiding this comment.
Thanks, fixed in 57e8256. A collection inside an array now reads ARRAY<(STRING OR BINARY)>. I added a test for that and for the top-level ("STRING" or "BINARY") form.
| fn ordinal_number(idx: usize) -> String { | ||
| const ORDINALS: [&str; 10] = [ | ||
| "first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth", | ||
| "ninth", "tenth", | ||
| ]; | ||
| ORDINALS | ||
| .get(idx) | ||
| .map(|s| s.to_string()) | ||
| .unwrap_or_else(|| format!("{}th", idx + 1)) | ||
| } |
There was a problem hiding this comment.
Spark spells out only the first three ordinals (ordinalNumber). In Spark 4.2.0, SELECT overlay('Spark SQL', '_', 6, array(1)) reports The 4th parameter requires the "INT" type, where this would say "fourth". The ordinals test at line 755 would then check ordinal_number(3) as "4th".
| fn ordinal_number(idx: usize) -> String { | |
| const ORDINALS: [&str; 10] = [ | |
| "first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth", | |
| "ninth", "tenth", | |
| ]; | |
| ORDINALS | |
| .get(idx) | |
| .map(|s| s.to_string()) | |
| .unwrap_or_else(|| format!("{}th", idx + 1)) | |
| } | |
| fn ordinal_number(idx: usize) -> String { | |
| match idx { | |
| 0 => "first".to_string(), | |
| 1 => "second".to_string(), | |
| 2 => "third".to_string(), | |
| _ => format!("{}th", idx + 1), | |
| } | |
| } |
There was a problem hiding this comment.
Thanks, fixed in 57e8256. ordinal_number now matches ordinalNumber, and the test checks 4th and 11th.
…n errors
Spark spells out only the first three ordinals ("4th parameter"), and
writes a type collection inside an array as ARRAY<(STRING OR BINARY)>.
| Interval(IntervalUnit::MonthDayNano) => "INTERVAL".to_string(), | ||
| List(field) | LargeList(field) | FixedSizeList(field, _) => { | ||
| format!("ARRAY<{}>", spark_sql_type(field.data_type())) | ||
| } | ||
| Struct(fields) => format!( | ||
| "STRUCT<{}>", | ||
| fields | ||
| .iter() | ||
| .map(|f| format!( | ||
| "{}: {}{}", | ||
| f.name(), |
There was a problem hiding this comment.
This copy of spark_sql_type doesn't have the names that #345 added in 73b600c for my comment there: INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND, TIME(6), and quoted struct field names (#345's version). So the two copies print different names for the same type, and sha2's errors use the Arrow names:
| Query | This PR | Spark 4.2.0 |
|---|---|---|
sha2(make_dt_interval(1), 256) |
Arrow's Duration name |
"INTERVAL DAY TO SECOND" |
sha2(arrow_cast(1, 'Interval(YearMonth)'), 256) |
"Interval(YearMonth)" |
"INTERVAL YEAR TO MONTH" |
sha2(make_time(1, 2, 3), 256) |
"Time64(ns)" |
"TIME(6)" (with the TIME type enabled) |
sha2(named_struct('a b', CAST(NULL AS INT)), 256) |
"STRUCT<a b: INT>" |
"STRUCT<`a b`: INT>" |
The Spark column for the year-month row is from make_ym_interval(1, 0), since the crate has no make_ym_interval. Could this copy match #345's, including quote_if_needed? The copies also differ the other way: this one names Decimal256, FixedSizeBinary, and Date64, and #345's prints their Arrow names. An .slt error record for sha2(make_dt_interval(1), 256) would cover it.
There was a problem hiding this comment.
Thanks, fixed in d3f29ae. spark_sql_type now matches #345's copy: INTERVAL YEAR TO MONTH, INTERVAL DAY TO SECOND, TIME(6), and struct field names go through quote_if_needed. Decimal256, FixedSizeBinary and Date64 print their Arrow names, as in #345. I added a unit test for the names and .slt error records for sha2(make_dt_interval(1), 256), Interval(YearMonth) and a struct with a quoted field name.
…errors Name interval, duration and time types as Spark does, quote struct field names that need it, and use Arrow names for types Spark lacks. Co-Authored-By: Claude <noreply@anthropic.com>
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
|
Thanks @mbutrovich! |
…de (#360) ## Which issue does this PR close? - Relates to #88 and #346. ## Rationale for this change #89, #304, #345 and #350 added a shared way to declare a function's input types, find a common type, cast a value and read ANSI mode. `CONTRIBUTING.md` does not mention any of them, so a new function is likely to hand-write a `Signature`, call Arrow's `cast`, or take a `bool` for ANSI mode instead. ## What changes are included in this PR? - A new "Input types, casts and ANSI mode" section in `CONTRIBUTING.md`. It says when to use `SparkInputTypes`, the rules in `type_coercion.rs`, `spark_cast()` and `EvalMode`, and points to the module docs for details. - The audit skill's "Signature and coercion" and "ANSI handling" checks now point at the guide. The section says that `SparkInputTypes` only chooses target types and that `spark_cast()` returns "not implemented" for casts that are not ported yet (#88). I kept that wording on purpose, so it can be relaxed as the cast port progresses. ## Are these changes tested? Documentation only. I checked each name, path and example against `apache/main` (`sha2.rs`, `parse_url.rs`, `array_position.rs`, `implicit_cast.rs`, `type_coercion.rs`, `eval_mode.rs`, `cast/mod.rs`). ## Are there any user-facing changes? No. This PR was written with the help of an LLM (Claude Code).
## Which issue does this PR close? - Part of #353. ## Rationale for this change Spark declares `crc32`'s input type as BINARY, and casts a STRING argument to it. `crc32` already accepts the same types, but it rejects the others with DataFusion's errors instead of Spark's: | Query | Before | This PR | |---|---|---| | `crc32(1::INT)` | `Function 'crc32' requires Binary, but received Int32` ... | `` [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `crc32` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09 `` | | `crc32('a', 'b')` | `Function 'crc32' expects 1 arguments but received 2` ... | `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `crc32` requires 1 parameters but the actual number is 2. ... `` | ## What changes are included in this PR? `crc32` declares Spark's `Crc32.inputTypes` with `SparkInputTypes` from #350, the same way `sha2` does. ## Are these changes tested? Yes. New records in `hash/crc32.slt` cover a dictionary-encoded string, a column with NULLs, the errors for other types and for the wrong number of arguments, and ANSI mode. The error records fail without this change. I checked `crc32.slt` against Spark 4.2.0 with the PySpark runner, and every record matches. The runner does not compare error messages, so I checked them against Spark by hand. Spark's message also names the argument's SQL text, as noted in #350. ## Are there any user-facing changes? Only the error messages above. The results do not change. This PR was prepared with AI assistance (Claude Code) and reviewed by me.
## Which issue does this PR close? - Part of #353. ## Rationale for this change Spark declares `sha1`'s input type as BINARY, and casts a STRING argument to it. `sha1` already accepts the same types, but it rejects the others with DataFusion's errors instead of Spark's: | Query | Before | This PR | |---|---|---| | `sha1(1::INT)` | `Function 'sha1' requires Binary, but received Int32` ... | `` [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha1` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09 `` | | `sha1('a', 'b')` | `Function 'sha1' expects 1 arguments but received 2` ... | `` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha1` requires 1 parameters but the actual number is 2. ... `` | ## What changes are included in this PR? `sha1` declares Spark's `Sha1.inputTypes` with `SparkInputTypes` from #350, the same way `sha2` does. ## Are these changes tested? Yes. New records in `hash/sha1.slt` cover a dictionary-encoded string, a column with NULLs, the errors for other types and for the wrong number of arguments, and ANSI mode. The error records fail without this change. I checked `sha1.slt` against Spark 4.2.0 with the PySpark runner, and every record matches. The runner does not compare error messages, so I checked them against Spark by hand. Spark's message also names the argument's SQL text, as noted in #350. ## Are there any user-facing changes? Only the error messages above. The results do not change. This PR was prepared with AI assistance (Claude Code) and reviewed by me.
Which issue does this PR close?
Rationale for this change
Spark casts a function's arguments to the input types the function declares, in one place. Here each function picks its accepted types in a hand-written
Signature, so some arguments that Spark casts are rejected, and a rejected type gives DataFusion's error instead of Spark's.sha2shows both problems. Spark declares(BINARY, INT):sha2('Spark', '256')Function 'sha2' requires Int32, but received Stringsha2('Spark', 256.7::DOUBLE)requires Int32, but received Float64sha2('Spark', 256.0::DECIMAL(10, 2))requires Int32, but received Decimal(10, 2)sha2(1::INT, 256)requires Binary, but received Int32[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha2` requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09sha2('Spark')Function 'sha2' expects 2 arguments but received 1[WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha2` requires 2 parameters but the actual number is 1. ...Spark's message also names the argument's SQL text (
"1").coerce_typessees only the types, so the message here says "it has the type" instead.What changes are included in this PR?
function/implicit_cast.rs:SparkInputTypemirrors theAbstractDataTypes Spark functions declare: the concrete scalar types,Numeric,Integral,Decimal,ArrayandOneOf(Spark'sTypeCollection).implicit_castports the privateimplicitCast(DataType, AbstractDataType)from bothTypeCoercion(ANSI mode off) andAnsiTypeCoercion(ANSI mode on), withCast.canANSIStoreAssignandDecimalType.forType. The two differ, for example a string becomes a BOOLEAN only with ANSI mode on.SparkInputTypesholds the declared types, the ANSI flag and the function'suser_definedSignature. A function returns itssignature(), callscoercefromcoerce_types, and callswith_configfromwith_updated_config.coercereturns Spark'sUNEXPECTED_INPUT_TYPEandWRONG_NUM_ARGSerrors with the SQLSTATE.Utf8ViewbecomesBinaryView). Spark has no dictionary or fixed-size binary types, so a dictionary argument has the type of its values, as in refactor(spark): share Spark's common-type rules between functions #345, and aFixedSizeBinaryargument is cast to BINARY. Before this PRsha2rejectedFixedSizeBinary. Decide how functions handle dictionary and run-end-encoded arguments #349 covers encodings more broadly.sha2usesSignature::user_definedwith Spark's(BINARY, INT)declaration.Left for later PRs:
StructTypeorMapType(any struct or map, for exampleto_jsonandGetArrayStructFields), and some declareAbstractMapType(key, value). They can be added when a function moving onto this module needs them. Until then a struct or map argument to a scalar declared type is rejected withUNEXPECTED_INPUT_TYPE, as in Spark.function/type_coercion.rs.spark_sql_typeis copied from refactor(spark): share Spark's common-type rules between functions #345, and whichever PR lands second should use the other's copy. The two files could merge once both land.sha2('Spark', '256x')and wrapssha2('Spark', 4294967552)to 256, and both raise a cast error here. That is part of the CAST work in [EPIC] Port Comet's Spark-compatible CAST #88.SparkInputTypevariants are unused until more functions move over, so the enum has a scopedallow(dead_code).Are these changes tested?
Yes. Unit tests in
implicit_cast.rscover each rule in both modes,OneOf, arrays, dictionaries and the error text. New records inhash/sha2.sltcover the casts, the errors and ANSI mode. I checkedsha2.sltagainst Spark 4.2.0 with the PySpark runner, and all 45 records match. Most of the new records fail without this change.Are there any user-facing changes?
sha2accepts STRING, FLOAT, DOUBLE and DECIMAL bit lengths, and its errors for other types and for the wrong number of arguments now match Spark's. There are no public API changes, since the new module ispub(crate).This PR was prepared with AI assistance (Claude Code) and reviewed by me.