Repository navigation
Description
Is your feature request related to a problem or challenge?
Spark casts a function's arguments to its declared input types in one place. Most expressions mix in ImplicitCastInputTypes and declare inputTypes: Seq[AbstractDataType], for example DoubleType, NumericType or TypeCollection(StringType, BinaryType). One rule, implicitCast(from, expected) in TypeCoercion and AnsiTypeCoercion, then inserts the casts, and checkInputDataTypes raises DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE for the rest.
Here each function writes a DataFusion Signature by hand (Exact, Coercible, or user_defined with coerce_types), and picks which source types to accept. That leads to the same bugs again and again:
- Arguments that Spark casts are rejected: [Bug] pmod rejects string arguments that Spark implicitly casts to a numeric type #26 (
pmod), [Bug] spark factorial rejects STRING, DECIMAL and floating-point inputs that Spark implicitly casts to INT #29 (factorial), [Bug] last_day, monthname, weekday, round and unbase64 reject arguments that Spark implicitly casts #41 (last_day,monthname,weekday,round,unbase64), and string arguments tolnandlogin feat(spark): add Spark-compatible ln, log, log10, log2 and log1p #334. - A rejected type gives DataFusion's
coercion from Utf8 to the signature Exact(Float64) failedinstead of Spark'sUNEXPECTED_INPUT_TYPE. - ANSI mode reaches a function's coercion only when that function opts in through
with_updated_config.
For example,array_position(andarray_removein feat(spark): add Spark-compatible array_remove #330) always coerce as Spark does with ANSI mode off. With ANSI mode on, Spark widens struct field types, soarray_position(array(named_struct('a', 1)), named_struct('a', 1L))returns1in Spark and is an error here. - Nothing checks a function's accepted types against Spark's
implicitCasttable, so reviewers check each one by hand.
Finding a common type for several arguments (greatest, array_position and the other array functions) already has one shared copy of Spark's rules in function/type_coercion.rs (#345). This issue is about the other half: casting to declared input types.
Describe the solution you'd like
Let a function declare its input types the way Spark does, and coerce them in one place:
- A
SparkInputTypeenum infunction/type_coercion.rs, mirroring theAbstractDataTypes that Spark functions use:Double,Integer,Long,Numeric,Integral,Decimal,String,Binary,Date,Timestamp,Boolean,Any,OneOf(Vec<SparkInputType>), and array and map forms. spark_coerce_inputs(name, arg_types, &[SparkInputType], ansi_mode) -> Result<Vec<DataType>>, a port ofimplicitCastfor both modes, that returns Spark'sUNEXPECTED_INPUT_TYPEerror, with the SQLSTATE, when an argument cannot be cast.- A small trait or macro that implements
ScalarUDFImpl::coerce_typesfrom the declared types, withSignature::user_defined, and passes ANSI mode throughwith_updated_config. - For functions that cast all their arguments to one common type (
greatest,least,coalesce, the array functions), a declared form that uses the existing common-type rules.
Then each function states what Spark declares, and the casts, the ANSI difference and the error text come out the same everywhere.
Points to settle:
user_definedsignatures lose DataFusion's signature-based documentation andinformation_schemaoutput. The declared types could generate that text instead.- The cast that runs is still DataFusion's cast, so differences such as a malformed string returning NULL with ANSI mode off ([EPIC] Port Comet's Spark-compatible CAST #88) stay with the cast work in [EPIC] Port Comet's Spark-compatible CAST #88.
- Comet calls the physical expressions directly and does its coercion in Spark's plan, so this mainly affects DataFusion SQL users.
Describe alternatives you've considered
Keep per-function Signatures and fix each gap as it is found, as in #26, #29 and #41. That works, but every new function repeats the work and the review.
Additional context
I suggest starting with a few functions (ln and log from #334, pmod for #26, factorial for #29) before migrating the rest.
Activity
feat(spark): cast function arguments to Spark's declared input types (#…
docs: explain input types, casts and ANSI mode in the contributor gui…