Repository navigation
Description
Tracks porting Comet's Spark-compatible CAST (native/spark-expr/src/conversion_funcs, about 11.6k lines including about 150 Rust unit tests) into datafusion-spark. Once it is here, Comet can call it instead of its own copy, as it already does for about 34 other expressions.
The crate already has a small spark_cast(value, 'timestamp') UDF that covers integer and float to timestamp. This epic replaces it with the full implementation.
Design decisions (settle before porting starts)
- Public API. Proposal: a kernel
spark_cast(ColumnarValue, &DataType, &SparkCastOptions) -> Result<ColumnarValue>for engines that build physical plans (Comet), and thespark_cast(expr, '<spark type>')ScalarUDF built on it for SQL and SLT. Comet'sCastPhysicalExpr, with itsexpr_idandQueryContext, stays in Comet as a thin wrapper. - Eval mode.
ConfigOptionsonly hasenable_ansi_mode. Proposal: a publicEvalMode { Legacy, Ansi, Try }inSparkCastOptions. The UDF readsenable_ansi_mode, and a separatetry_castUDF selectsTry.EvalModeis added in feat(spark): add EvalMode and use it for parse_url and try_parse_url #89. - Errors. Comet maps a structured
SparkError(CastInvalidValue,InvalidInputInCastToDatetime,CastOverFlow,NumericValueOutOfRange) to JVM exceptions. This crate uses formatted strings, as inabs.rs. Cast needs a typed error, wrapped inDataFusionError::External, that renders Spark's message ([CAST_INVALID_INPUT] ...) and that Comet can downcast. This decision will apply to other functions too. - Spark-version options. Comet has
is_spark4_plus,decimal_zero_any_exponentanddecimal_skip_integral_digit_limit. Proposal: keep them as options that default to Spark 4.2.0 behavior. SLT tests the defaults, and Rust unit tests cover the other values (Introduce version-specific behavior in Spark expressions #16, apache/datafusion#23887). Where Spark has aspark.sql.legacy.*setting for a change, mirror that setting rather than checking the version. These options belong inSparkOptionsonce it moves into this crate as a config extension (MoveSparkOptionsandenable_ansi_modefromdatafusion-commontodatafusion-sparkdatafusion#27342, moved SparkOptions to datafusion-spark datafusion#27371). - Options that stay in Comet.
is_adapting_schemaandallow_cast_unsigned_intsexist for Comet's Parquet schema adapter, not Spark'sCast. Proposal: leave them out. Comet handles those cases before it calls the kernel. - Target-type syntax for the UDF. Parse Spark's type grammar (
int,decimal(10,2),timestamp_ntz,array<int>,map<string,int>,struct<a:int,b:string>,binary, ...), and rejectDictionaryand other types Spark can't name.
Foundation
- Cast framework:
SparkCastOptions,EvalMode, thecast_arraydispatcher with an explicit error for an unsupported cast, and theis_datafusion_spark_compatibleallowlist that falls through to Arrow's cast. Portutils.rs(spark_cast_postprocess,timestamp_micros_with_tz,cast_overflow) andtrim.rs, plus the crate-level helpers cast needs from Comet'sutils.rs(array_with_timezone,resolve_local_datetime,timestamp_ntz_to_timestamp,cast_timestamp_to_ntz). - Typed cast error and Spark-formatted messages
- Rework the
spark_castUDF: parse the target type, compute the return field from it inreturn_field_from_args, and readenable_ansi_modeandtime_zonefrom the config. Keepcast_int_to_timestamp.sltandcast_float_to_timestamp.sltpassing, and reconcile the current int and float to timestamp code with Comet's. - SLT conventions for cast: a
conversion/cast/directory with one file per source type family, and Legacy, ANSI (set datafusion.execution.enable_ansi_mode = true) and session time zone sections in each - Port Comet's criterion benchmarks (
native/spark-expr/benches/cast_*.rs) alongside each family below
Porting by source type
Each item ports the code and its Rust unit tests, adds SLT cases checked against PySpark (harvesting Comet's spark/src/test/resources/sql-tests/expressions/cast/*.sql and the Spark CastSuite*), and runs the audit-datafusion-spark-expression checks for that slice.
Boolean and numeric
- Boolean to integral, float, decimal, string and timestamp (
boolean.rs) - Integral to integral, with Legacy wrapping and ANSI
CAST_OVERFLOW(spark_cast_int_to_int) - Integral to float, boolean and string (Arrow fallthrough, confirm compatibility)
- Integral to decimal (
cast_int_to_decimal128) - Integral to binary, Legacy only (
cast_whole_num_to_binary!) - Integral, float, decimal and boolean to timestamp
- Float and double to integral (
spark_cast_nonintegral_numeric_to_integral) - Float and double to decimal (shortest round-trip via
ryu, matchingDouble.toString) - Float and double to string (
JavaFloatString, Java'sFloat.toStringandDouble.toStringformatting) - Decimal to decimal, with ANSI
NUMERIC_VALUE_OUT_OF_RANGE - Decimal to integral, float (single rounding), boolean and string (Legacy scientific notation such as
0E-18)
String
- String to boolean
- String to integral (
cast_string_to_int), including Java whitespace trimming - String to float and double
- String to decimal (exponents, Unicode digits, the integral-digit limit)
- String to date
- String to timestamp. This is the largest part of
string.rs, so it may need to split into date-time formats, time-onlyTstrings, and zone IDs and offsets. - String to timestamp_ntz
-
LargeUtf8andUtf8Viewinput for every string cast above
Binary
- Binary to string, including
binaryOutputStyleand invalid UTF-8 (decode_utf8_spark_lossyfromdatafusion-comet-common) - String to binary
Temporal
- Date to timestamp and timestamp_ntz, and date to int
- Timestamp to timestamp_ntz and back (session time zone)
- Date, timestamp and timestamp_ntz to string, and timestamp to date, long and double
Complex and special inputs
- Struct to struct (
castStructsemantics, duplicate field names) - Struct, array and map to string, Legacy and non-Legacy formats, using the
null_stringoption - Array to array, recursively, including Spark's null elements for
array<date>toarray<int> - Map to map
- Dictionary-encoded string and binary input, and
Nullinput
TRY mode
-
try_castUDF usingEvalMode::Try, with SLT coverage for each family above
Adoption
- Document cast behavior and its known divergences from Spark (Epic: Document Spark-specific behavior for every expression #55)
- Comet: replace
conversion_funcswithdatafusion-spark's cast, keepCastas a wrapper, runCometCastSuiteand the cast SQL tests on every supported Spark version, then delete Comet's copy (tracked in apache/datafusion-comet)
Out of scope
- Planning SQL
CAST(x AS t)tospark_castautomatically. DataFusion has noExprPlannerhook for cast, and Spark's implicit coercions differ from DataFusion's. Open a follow-up if needed. - Casts that Comet's native cast does not implement. Comet falls back to Spark for these.
Activity
fix(spark): implicitly cast strings in dayname and monthname
fix(spark): widen date_from_unix_date coercion and match Spark nullab…
fix(spark): harden make_time kernel and extend its tests
feat(spark): add Spark-compatible dayname function (#75)
feat(spark): add Spark-compatible date_from_unix_date function (#74)
feat(spark): add EvalMode and use it for parse_url and try_parse_url (#…
fix(spark): document make_date's differences from the DataFusion buil…
fix(spark): accept Date64 in dayofweek and weekday
fix(spark): regexp_extract string types, pattern cache and error classes
feat(spark): add the Spark-compatible cast framework (#304)
feat(spark): cast function arguments to Spark's declared input types (#…
feat(spark): parse Spark type names in spark_cast and call the cast k…
docs: explain input types, casts and ANSI mode in the contributor gui…
test(spark): mark make_date's Spark differences with xfail comments
docs(spark): fix private fdlibm doc links and note Spark's d/f string…
test(spark): mark ln's string cast differences with xfail comments
feat(spark): add Spark-compatible make_date (#329)