Skip to content

Declare Spark input types per function and coerce them in one place #346

Open

Description

@andygrove

Is your feature request related to a problem or challenge?

Spark casts a function's arguments to its declared input types in one place. Most expressions mix in ImplicitCastInputTypes and declare inputTypes: Seq[AbstractDataType], for example DoubleType, NumericType or TypeCollection(StringType, BinaryType). One rule, implicitCast(from, expected) in TypeCoercion and AnsiTypeCoercion, then inserts the casts, and checkInputDataTypes raises DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE for the rest.

Here each function writes a DataFusion Signature by hand (Exact, Coercible, or user_defined with coerce_types), and picks which source types to accept. That leads to the same bugs again and again:

Finding a common type for several arguments (greatest, array_position and the other array functions) already has one shared copy of Spark's rules in function/type_coercion.rs (#345). This issue is about the other half: casting to declared input types.

Describe the solution you'd like

Let a function declare its input types the way Spark does, and coerce them in one place:

  1. A SparkInputType enum in function/type_coercion.rs, mirroring the AbstractDataTypes that Spark functions use: Double, Integer, Long, Numeric, Integral, Decimal, String, Binary, Date, Timestamp, Boolean, Any, OneOf(Vec<SparkInputType>), and array and map forms.
  2. spark_coerce_inputs(name, arg_types, &[SparkInputType], ansi_mode) -> Result<Vec<DataType>>, a port of implicitCast for both modes, that returns Spark's UNEXPECTED_INPUT_TYPE error, with the SQLSTATE, when an argument cannot be cast.
  3. A small trait or macro that implements ScalarUDFImpl::coerce_types from the declared types, with Signature::user_defined, and passes ANSI mode through with_updated_config.
  4. For functions that cast all their arguments to one common type (greatest, least, coalesce, the array functions), a declared form that uses the existing common-type rules.

Then each function states what Spark declares, and the casts, the ANSI difference and the error text come out the same everywhere.

Points to settle:

  • user_defined signatures lose DataFusion's signature-based documentation and information_schema output. The declared types could generate that text instead.
  • The cast that runs is still DataFusion's cast, so differences such as a malformed string returning NULL with ANSI mode off ([EPIC] Port Comet's Spark-compatible CAST #88) stay with the cast work in [EPIC] Port Comet's Spark-compatible CAST #88.
  • Comet calls the physical expressions directly and does its coercion in Spark's plan, so this mainly affects DataFusion SQL users.

Describe alternatives you've considered

Keep per-function Signatures and fix each gap as it is found, as in #26, #29 and #41. That works, but every new function repeats the work and the review.

Additional context

I suggest starting with a few functions (ln and log from #334, pmod for #26, factorial for #29) before migrating the rest.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions