Skip to content

Move existing functions onto the shared cast and type-coercion helpers #353

Open

Description

@andygrove

Is your feature request related to a problem or challenge?

There are now shared helpers for casts and type coercion: the cast kernel spark_cast (#304), the common-type rules in function/type_coercion.rs (#345), and SparkInputTypes in function/implicit_cast.rs (#350). Many functions were written before these existed, so they still have their own copies of the same logic or use DataFusion's rules where Spark's apply. This issue tracks moving them onto the shared code. It follows on from #346.

Describe the solution you'd like

Duplicated code

  • Array and element coercion. array_position and array_remove each have a coerce_array_and_element, and array_insert does the same steps inline: unwrap the list, call tightest_common_type(.., false), check struct_field_types_match and is_orderable, then rebuild the List/LargeList. They differ only in how they report a non-array first argument and whether they check ordering. Move one version into type_coercion.rs, with the ARRAY_FUNCTION_DIFF_TYPES and INVALID_ORDERING_TYPE errors built there. array_contains, which still uses Signature::array_and_element, can then use it too.
  • String and binary layout widening in contains. unify in string/contains.rs repeats wider_physical_type in type_coercion.rs, and classify repeats the dictionary and FixedSizeBinary unwrapping in implicit_cast.rs.

Time zone conversion

  • to_utc_timestamp (to_utc_timestamp uses the wrong offset on daylight saving days and rejects GMT+1 #354). adjust_to_utc_time parses the zone with chrono's Tz and computes the offset itself, while from_utc_timestamp uses TimeZoneRules. It looks up the offset with offset_from_utc_datetime on a value that is a local time, so the result can be wrong on the day daylight saving time starts. Tz also does not accept Spark zone IDs such as GMT+1. Use TimeZoneRules::local_to_utc, the inverse of what from_utc_timestamp uses.

Functions to move onto SparkInputTypes

These need no new SparkInputType variants:

These need new variants first:

  • A concrete Decimal(p, s): make_time ([Integer, Integer, Decimal(16, 6)]), make_interval, make_dt_interval
  • Date and Timestamp: weekday, dayname/monthname (date_part_name.rs), datediff, next_day, to_utc_timestamp, unix_date/unix_*
  • Day-time and year-month intervals: positive
  • Variadic arguments: elt (Spark casts the index to INT, while it is cast to BIGINT here, and it is checked with Arrow's can_cast_types), concat_ws

These need to be checked against Spark first: try_sum, encode.

Casts at execution time

  • bround and positive cast a string argument to DOUBLE in invoke_with_args, so that a malformed string is NULL with ANSI mode off. Moving them onto SparkInputTypes now would leave the cast to the planner's DataFusion cast, which raises an error instead. Once string to number casts are ported in [EPIC] Port Comet's Spark-compatible CAST #88, use spark_cast there, or move them onto SparkInputTypes ([Numeric, Integer] for bround).

Common type rules

  • if uses DataFusion's try_type_union_resolution, array uses make_array's coerce_types_inner, and equal_null uses comparison_coercion. Spark uses findWiderCommonType for if and array, and its rules for binary comparisons for <=>. type_coercion.rs only has findTightestCommonType, so this needs findWiderCommonType ported first.

Describe alternatives you've considered

Leave the existing functions as they are and use the shared code only in new ones. The copies would then drift apart, and the gaps that #346 describes would remain in the older functions.

Additional context

Checked and left alone: Spark type names in errors all come from spark_sql_type. The other Arrow casts at execution time change the layout or width of values (pmod, shuffle, encode's charset), where Arrow's cast is correct. to_json formats timestamps in Spark's JSON format, not the CAST format.

I suggest the to_utc_timestamp fix first, since it is a bug, then the array and element helper, then sha1, crc32 and repeat.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions