Skip to content

[EPIC] Port Comet's Spark-compatible CAST #88

Open

Description

@andygrove

Tracks porting Comet's Spark-compatible CAST (native/spark-expr/src/conversion_funcs, about 11.6k lines including about 150 Rust unit tests) into datafusion-spark. Once it is here, Comet can call it instead of its own copy, as it already does for about 34 other expressions.

The crate already has a small spark_cast(value, 'timestamp') UDF that covers integer and float to timestamp. This epic replaces it with the full implementation.

Design decisions (settle before porting starts)

  • Public API. Proposal: a kernel spark_cast(ColumnarValue, &DataType, &SparkCastOptions) -> Result<ColumnarValue> for engines that build physical plans (Comet), and the spark_cast(expr, '<spark type>') ScalarUDF built on it for SQL and SLT. Comet's Cast PhysicalExpr, with its expr_id and QueryContext, stays in Comet as a thin wrapper.
  • Eval mode. ConfigOptions only has enable_ansi_mode. Proposal: a public EvalMode { Legacy, Ansi, Try } in SparkCastOptions. The UDF reads enable_ansi_mode, and a separate try_cast UDF selects Try. EvalMode is added in feat(spark): add EvalMode and use it for parse_url and try_parse_url #89.
  • Errors. Comet maps a structured SparkError (CastInvalidValue, InvalidInputInCastToDatetime, CastOverFlow, NumericValueOutOfRange) to JVM exceptions. This crate uses formatted strings, as in abs.rs. Cast needs a typed error, wrapped in DataFusionError::External, that renders Spark's message ([CAST_INVALID_INPUT] ...) and that Comet can downcast. This decision will apply to other functions too.
  • Spark-version options. Comet has is_spark4_plus, decimal_zero_any_exponent and decimal_skip_integral_digit_limit. Proposal: keep them as options that default to Spark 4.2.0 behavior. SLT tests the defaults, and Rust unit tests cover the other values (Introduce version-specific behavior in Spark expressions #16, apache/datafusion#23887). Where Spark has a spark.sql.legacy.* setting for a change, mirror that setting rather than checking the version. These options belong in SparkOptions once it moves into this crate as a config extension (Move SparkOptions and enable_ansi_mode from datafusion-common to datafusion-spark datafusion#27342, moved SparkOptions to datafusion-spark datafusion#27371).
  • Options that stay in Comet. is_adapting_schema and allow_cast_unsigned_ints exist for Comet's Parquet schema adapter, not Spark's Cast. Proposal: leave them out. Comet handles those cases before it calls the kernel.
  • Target-type syntax for the UDF. Parse Spark's type grammar (int, decimal(10,2), timestamp_ntz, array<int>, map<string,int>, struct<a:int,b:string>, binary, ...), and reject Dictionary and other types Spark can't name.

Foundation

  • Cast framework: SparkCastOptions, EvalMode, the cast_array dispatcher with an explicit error for an unsupported cast, and the is_datafusion_spark_compatible allowlist that falls through to Arrow's cast. Port utils.rs (spark_cast_postprocess, timestamp_micros_with_tz, cast_overflow) and trim.rs, plus the crate-level helpers cast needs from Comet's utils.rs (array_with_timezone, resolve_local_datetime, timestamp_ntz_to_timestamp, cast_timestamp_to_ntz).
  • Typed cast error and Spark-formatted messages
  • Rework the spark_cast UDF: parse the target type, compute the return field from it in return_field_from_args, and read enable_ansi_mode and time_zone from the config. Keep cast_int_to_timestamp.slt and cast_float_to_timestamp.slt passing, and reconcile the current int and float to timestamp code with Comet's.
  • SLT conventions for cast: a conversion/cast/ directory with one file per source type family, and Legacy, ANSI (set datafusion.execution.enable_ansi_mode = true) and session time zone sections in each
  • Port Comet's criterion benchmarks (native/spark-expr/benches/cast_*.rs) alongside each family below

Porting by source type

Each item ports the code and its Rust unit tests, adds SLT cases checked against PySpark (harvesting Comet's spark/src/test/resources/sql-tests/expressions/cast/*.sql and the Spark CastSuite*), and runs the audit-datafusion-spark-expression checks for that slice.

Boolean and numeric

  • Boolean to integral, float, decimal, string and timestamp (boolean.rs)
  • Integral to integral, with Legacy wrapping and ANSI CAST_OVERFLOW (spark_cast_int_to_int)
  • Integral to float, boolean and string (Arrow fallthrough, confirm compatibility)
  • Integral to decimal (cast_int_to_decimal128)
  • Integral to binary, Legacy only (cast_whole_num_to_binary!)
  • Integral, float, decimal and boolean to timestamp
  • Float and double to integral (spark_cast_nonintegral_numeric_to_integral)
  • Float and double to decimal (shortest round-trip via ryu, matching Double.toString)
  • Float and double to string (JavaFloatString, Java's Float.toString and Double.toString formatting)
  • Decimal to decimal, with ANSI NUMERIC_VALUE_OUT_OF_RANGE
  • Decimal to integral, float (single rounding), boolean and string (Legacy scientific notation such as 0E-18)

String

  • String to boolean
  • String to integral (cast_string_to_int), including Java whitespace trimming
  • String to float and double
  • String to decimal (exponents, Unicode digits, the integral-digit limit)
  • String to date
  • String to timestamp. This is the largest part of string.rs, so it may need to split into date-time formats, time-only T strings, and zone IDs and offsets.
  • String to timestamp_ntz
  • LargeUtf8 and Utf8View input for every string cast above

Binary

  • Binary to string, including binaryOutputStyle and invalid UTF-8 (decode_utf8_spark_lossy from datafusion-comet-common)
  • String to binary

Temporal

  • Date to timestamp and timestamp_ntz, and date to int
  • Timestamp to timestamp_ntz and back (session time zone)
  • Date, timestamp and timestamp_ntz to string, and timestamp to date, long and double

Complex and special inputs

  • Struct to struct (castStruct semantics, duplicate field names)
  • Struct, array and map to string, Legacy and non-Legacy formats, using the null_string option
  • Array to array, recursively, including Spark's null elements for array<date> to array<int>
  • Map to map
  • Dictionary-encoded string and binary input, and Null input

TRY mode

  • try_cast UDF using EvalMode::Try, with SLT coverage for each family above

Adoption

  • Document cast behavior and its known divergences from Spark (Epic: Document Spark-specific behavior for every expression #55)
  • Comet: replace conversion_funcs with datafusion-spark's cast, keep Cast as a wrapper, run CometCastSuite and the cast SQL tests on every supported Spark version, then delete Comet's copy (tracked in apache/datafusion-comet)

Out of scope

  • Planning SQL CAST(x AS t) to spark_cast automatically. DataFusion has no ExprPlanner hook for cast, and Spark's implicit coercions differ from DataFusion's. Open a follow-up if needed.
  • Casts that Comet's native cast does not implement. Comet falls back to Spark for these.

Activity

  1. self-assigned this
    on Nov 13, 2026
  2. added
    EPICA larger project, actively underway, with sub tasks
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

EPICA larger project, actively underway, with sub tasks

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions