Repository navigation
Description
Is your feature request related to a problem or challenge?
There are now shared helpers for casts and type coercion: the cast kernel spark_cast (#304), the common-type rules in function/type_coercion.rs (#345), and SparkInputTypes in function/implicit_cast.rs (#350). Many functions were written before these existed, so they still have their own copies of the same logic or use DataFusion's rules where Spark's apply. This issue tracks moving them onto the shared code. It follows on from #346.
Describe the solution you'd like
Duplicated code
- Array and element coercion.
array_positionandarray_removeeach have acoerce_array_and_element, andarray_insertdoes the same steps inline: unwrap the list, calltightest_common_type(.., false), checkstruct_field_types_matchandis_orderable, then rebuild theList/LargeList. They differ only in how they report a non-array first argument and whether they check ordering. Move one version intotype_coercion.rs, with theARRAY_FUNCTION_DIFF_TYPESandINVALID_ORDERING_TYPEerrors built there.array_contains, which still usesSignature::array_and_element, can then use it too. - String and binary layout widening in
contains.unifyinstring/contains.rsrepeatswider_physical_typeintype_coercion.rs, andclassifyrepeats the dictionary andFixedSizeBinaryunwrapping inimplicit_cast.rs.
Time zone conversion
-
to_utc_timestamp(to_utc_timestampuses the wrong offset on daylight saving days and rejectsGMT+1#354).adjust_to_utc_timeparses the zone with chrono'sTzand computes the offset itself, whilefrom_utc_timestampusesTimeZoneRules. It looks up the offset withoffset_from_utc_datetimeon a value that is a local time, so the result can be wrong on the day daylight saving time starts.Tzalso does not accept Spark zone IDs such asGMT+1. UseTimeZoneRules::local_to_utc, the inverse of whatfrom_utc_timestampuses.
Functions to move onto SparkInputTypes
These need no new SparkInputType variants:
-
sha1,crc32:[Binary], likesha2(feat(spark): cast sha1's argument to Spark's declared input type #355, feat(spark): cast crc32's argument to Spark's declared input type #356) -
repeat(array_repeat) (fix(spark): cast array_repeat's count to INT as Spark does #357): Spark takes an INT count and does not cast to it, soSparkInputTypesdoes not fit. The count is cast to BIGINT now. Cast any integer count to INT instead, asarray_insertandstackdo, because DataFusion types2as BIGINT.
These need new variants first:
- A concrete
Decimal(p, s):make_time([Integer, Integer, Decimal(16, 6)]),make_interval,make_dt_interval -
DateandTimestamp:weekday,dayname/monthname(date_part_name.rs),datediff,next_day,to_utc_timestamp,unix_date/unix_* - Day-time and year-month intervals:
positive - Variadic arguments:
elt(Spark casts the index to INT, while it is cast to BIGINT here, and it is checked with Arrow'scan_cast_types),concat_ws
These need to be checked against Spark first: try_sum, encode.
Casts at execution time
-
broundandpositivecast a string argument to DOUBLE ininvoke_with_args, so that a malformed string is NULL with ANSI mode off. Moving them ontoSparkInputTypesnow would leave the cast to the planner's DataFusion cast, which raises an error instead. Once string to number casts are ported in [EPIC] Port Comet's Spark-compatible CAST #88, usespark_castthere, or move them ontoSparkInputTypes([Numeric, Integer]forbround).
Common type rules
-
ifuses DataFusion'stry_type_union_resolution,arrayusesmake_array'scoerce_types_inner, andequal_nullusescomparison_coercion. Spark usesfindWiderCommonTypeforifandarray, and its rules for binary comparisons for<=>.type_coercion.rsonly hasfindTightestCommonType, so this needsfindWiderCommonTypeported first.
Describe alternatives you've considered
Leave the existing functions as they are and use the shared code only in new ones. The copies would then drift apart, and the gaps that #346 describes would remain in the older functions.
Additional context
Checked and left alone: Spark type names in errors all come from spark_sql_type. The other Arrow casts at execution time change the layout or width of values (pmod, shuffle, encode's charset), where Arrow's cast is correct. to_json formats timestamps in Spark's JSON format, not the CAST format.
I suggest the to_utc_timestamp fix first, since it is a bug, then the array and element helper, then sha1, crc32 and repeat.
Activity
feat(spark): cast crc32's argument to Spark's declared input type (#356)
feat(spark): cast sha1's argument to Spark's declared input type (#355)
fix(spark): cast array_repeat's count to INT as Spark does (#357)