Skip to content

feat(spark): parse Spark type names in spark_cast and call the cast kernel - #317

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:feat-cast-udf
Nov 18, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:feat-cast-udf

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #88: the "Rework the spark_cast UDF" item.

Rationale for this change

spark_cast(expr, 'timestamp') only supported a timestamp target and integer or float input, with its own conversion code. To test the Spark-compatible cast kernel from SQL and SLT as each family is ported, the function needs to accept any Spark type name and call the kernel.

What changes are included in this PR?

  • Target type parser (cast/data_type.rs): parses the second argument with Spark's dataType grammar from SqlBaseParser.g4, case-insensitively.

    • Primitive types: boolean, tinyint/byte, smallint/short, int/integer, bigint/long, float/real, double, date, binary and string.
    • Decimals: decimal/dec/numeric, with an optional precision and scale. A bare decimal is decimal(10, 0), and precision and scale are validated.
    • Character types: char(n) and varchar(n) become STRING, which is what Spark casts them to by default. A missing length raises DATATYPE_MISSING_SIZE, as in Spark.
    • Timestamps:
      • timestamp, timestamp_ltz and timestamp with local time zone give a TIMESTAMP tagged with the session time zone. That matches Spark's default spark.sql.timestampType.
      • timestamp_ntz and timestamp without time zone give a TIMESTAMP_NTZ.
    • Complex types: array<…>, map<…, …> (with Spark's ArrowUtils field names entries, key and value), and struct<…>. Struct fields can be backquoted, NOT NULL or have a COMMENT, and duplicate field names are allowed, as in Spark.
    • Unsupported types: intervals, TIME, VARIANT, VOID, geospatial types and collated strings get an "unsupported" error. Malformed type names get a parse error that names the problem.
  • spark_cast UDF:

    • accepts any input type
    • requires the target type to be a string literal
    • computes its return field from the parsed type
    • calls the kernel with EvalMode::from_config and the session time zone as a SparkTimeZone

    The old conversion code in the UDF is removed.

  • Error messages: the kernel's "not supported" and "not implemented yet" errors name types the Spark way, using spark_sql_type from refactor(spark): share Spark's common-type rules between functions #345. For example, they now say cast from BIGINT to TIMESTAMP_NTZ instead of cast from Int64 to Timestamp(µs).

  • Integer and float to TIMESTAMP: ported into the kernel (cast/numeric.rs). Spark has no cast from a number to TIMESTAMP_NTZ, so that cast is rejected.

    Float input follows Spark 4.2.0's doubleToTimestamp and doubleToTimestampAnsi:

    Mode NaN or infinite Out of range
    Legacy NULL saturates, like Scala's toLong
    ANSI CAST_INVALID_INPUT CAST_OVERFLOW, such as 1.0E25D from DOUBLE to BIGINT
    TRY NULL NULL

    The previous UDF already saturated in Legacy mode. Comet returns NULL there, which differs from Spark.

Are these changes tested?

Yes.

  • Unit tests:
    • the type parser: every primitive and complex type, invalid names and unsupported types
    • integer and float to timestamp in each eval mode, including saturation, a NaN under a null, FLOAT widening and the error messages
    • a round trip between parse_spark_type and spark_sql_type for each canonical Spark type name
    • the UDF: return fields, a non-literal target type, array and scalar input, and reading ANSI mode from the config
  • #[user_doc] for spark_cast, which documents the target-type syntax and the two kinds of error.
  • New SLT file conversion/cast_target_type.slt: return types from arrow_typeof, the session time zone on timestamp, casts that already work through Arrow, invalid and unsupported targets, and casts that are not implemented yet.
  • Existing SLT files: cast_int_to_timestamp.slt and cast_float_to_timestamp.slt still pass, with these changes:
    • The old "unsupported target" test used spark_cast(100, 'string'), which now works, so it now uses interval instead.
    • It also now checks that a number can't be cast to timestamp_ntz.
    • The ANSI errors are checked against Spark's messages.
    • It has a Legacy saturation test that casts the result back to bigint. The old file said this couldn't be shown in SLT.

Are there any user-facing changes?

Yes. spark_cast accepts any Spark type name as its target and any input type.

  • Newly working: casts that are already Spark-compatible through Arrow now succeed. One example is spark_cast(42, 'string').
  • Errors for casts not ported yet: those return an explicit "not supported" or "not implemented yet" error rather than a result that differs from Spark's. A cast that Spark supports, such as TIMESTAMP or DATE to a number, says "not implemented yet"; "not supported" is only for casts Spark does not have. Integer narrowing is one example. The unsuffixed literal 1 is a BIGINT, so even spark_cast(1, 'int') needs that port.
  • Error message format: in ANSI mode, errors for NaN and out-of-range input now use Spark's [CAST_INVALID_INPUT] and [CAST_OVERFLOW] messages.

This PR was written by an LLM (Claude) from the Comet and Spark sources, and reviewed before opening.

🤖 Generated with Claude Code

mbutrovich pushed a commit that referenced this pull request Nov 16, 2026
## Which issue does this PR close?

Part of #56. It replaces that issue's plan to make 57.0.0 the first
release from this repository.

## Rationale for this change

New Spark-compatible functions and fixes land here between DataFusion
major releases. Tying `datafusion-spark` releases to DataFusion's would
hold them back, so this PR proposes releasing 56.x from `main` as often
as needed, until DataFusion 57.0.0 is released. That only works if
`main` doesn't break the public API in the meantime, so the policy
spells out what counts as a breaking change.

## What changes are included in this PR?

- **`dev/release/README.md`**: expands the Versioning section.
- The major version matches DataFusion's. Minor releases (such as
56.1.0) add functions, features and fixes. Patch releases (such as
56.0.1) contain only fixes.
- `main` stays on DataFusion 56 and releases 56.x until DataFusion
57.0.0 is released. Then it moves to DataFusion 57, and its next release
is 57.0.0.
- DataFusion 56.0.0 publishes `datafusion-spark` 56.0.0 from `branch-56`
in apache/datafusion. Later 56.x releases come from this repository, so
a DataFusion 56.x patch release should not publish `datafusion-spark`
again.
- What a 56.x release must not break, what is allowed, and how to avoid
a break with `#[deprecated]`. A change to a function's results that
makes it match Spark counts as a bug fix.
- Checking a release against the previous one with
`cargo-semver-checks`.
- **`CONTRIBUTING.md`**: a short "API Compatibility" section for
contributors that links to the policy.

## Are these changes tested?

Documentation only. To check that the policy can be applied today, I ran
`cargo semver-checks -p datafusion-spark --all-features` against the
crate at apache/datafusion's `56.0.0-rc1` tag. Results:

| Compared | Result |
|---|---|
| Current `main` | No breaking changes (229 checks pass) |
| The open cast PRs (#89, #304, #317) | No breaking changes |
| A test rename of a public function | Reported as a major-version
change, so the check does detect breaks |

Since rc1, `main` has only added public items, such as `dayname` and
`date_from_unix_date`.

## Are there any user-facing changes?

No code changes.

The policy needs agreement with apache/datafusion: its `branch-56`
release process still runs `cargo publish` for `datafusion-spark`, so
DataFusion 56.x patch releases would need to stop publishing it after
56.0.0.

_This PR was written by an LLM (Claude) and reviewed before opening._

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove
andygrove requested a review from kosiew November 17, 2026 13:21

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. This review covers only the commits after #304: the type parser, the UDF rework, and the numeric-to-timestamp casts. My #304 review covers the framework that this PR merges in.

}
}
"string" => DataType::Utf8,
"char" | "character" | "varchar" => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark requires a length for CHAR and VARCHAR (DataTypeAstBuilder.scala). In Spark 4.2.0, CAST(1 AS CHAR) fails with [DATATYPE_MISSING_SIZE] DataType "CHAR" requires a length parameter, and VARCHAR fails the same way. Here spark_cast(NULL, 'char') returns Utf8, because parameters() returns an empty list and only more than one length is rejected. Could this require exactly one length?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 15a4a69. A missing length now raises DATATYPE_MISSING_SIZE with Spark 4.2.0's exact text, which names the type as written (CHAR, CHARACTER or VARCHAR). A unit test covers all three, and cast_target_type.slt covers spark_cast(NULL, 'char').

Comment on lines +250 to +273
// Casts Spark supports that Arrow does not do the same way.
(
Utf8 | LargeUtf8 | Utf8View,
Boolean
| Int8
| Int16
| Int32
| Int64
| Float32
| Float64
| Decimal128(_, _)
| Date32
| Timestamp(_, _),
)
| (Boolean | Decimal128(_, _), Timestamp(_, Some(_)))
| (Boolean | Int8 | Int16 | Int32 | Int64, Decimal128(_, _))
| (Int8 | Int16 | Int32 | Int64, Binary)
| (Float32 | Float64, Decimal128(_, _) | Utf8)
| (Decimal128(_, _), Boolean | Float32 | Float64)
| (Date32, Timestamp(_, _))
| (Struct(_) | List(_) | Map(_, _), Utf8)
| (Struct(_), Struct(_))
| (List(_), List(_))
| (Map(_, _), Map(_, _)) => not_yet_ported(from_type, to_type, eval_mode),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several casts that Spark supports aren't in this arm, so they fall through to not_supported. The error then says Spark has no such cast. Spark 4.2.0 evaluates CAST(TIMESTAMP'2020-01-01 00:00:00' AS INT), AS SMALLINT, AS DOUBLE, and AS DECIMAL(20,2) in both modes, and AS BOOLEAN with ANSI mode off. With ANSI mode off, every cast from DATE to a numeric type or BOOLEAN returns NULL. Here spark_cast(spark_cast(1, 'timestamp'), 'int') fails with Spark-compatible cast from Timestamp(µs, "UTC") to Int32 is not supported. Could we add these pairs to this arm?

| (Timestamp(_, Some(_)), Boolean | Int8 | Int16 | Int32 | Float32 | Float64 | Decimal128(_, _))
| (Date32, Boolean | Int8 | Int16 | Int64 | Float32 | Float64 | Decimal128(_, _))

(Date32, Int32) belongs here as well. My #304 review raises that pair in the allowlist.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added both pairs to the not-yet-ported arm in 15a4a69, with a unit test that they report "not implemented yet". (Date32, Int32) came in with the #304 fix.

Comment on lines +23 to +34
/// `spark_cast(expr, '<type>')`: Spark's `CAST(expr AS <type>)`.
///
/// The target type is a string literal in Spark's SQL type syntax, such as
/// `'int'`, `'decimal(10, 2)'` or `'array<timestamp>'`. The cast follows
/// `datafusion.execution.enable_ansi_mode`, as Spark's follows
/// `spark.sql.ansi.enabled`, and uses `datafusion.execution.time_zone` as the
/// session time zone.
///
/// # References
/// - <https://spark.apache.org/docs/latest/api/sql/index.html#cast>
#[derive(Debug, PartialEq, Eq, Hash)]
pub struct SparkCast {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark_cast has no #[user_doc], and CONTRIBUTING.md asks for one on every function. Should we add it here, since this PR defines the target-type syntax and the casts that return "not implemented yet"? Without it, the function is missing from the reference that #115 generates.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 15a4a69. It documents the target-type syntax, the unsupported types, ANSI mode and the session time zone, and the "not implemented yet" vs "not supported" errors.

Comment on lines +195 to +199
statement error \[CAST_INVALID_INPUT\] The value 'NaN' of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed
SELECT spark_cast(arrow_cast('NaN', 'Float64'), 'timestamp');

# Infinity should error in ANSI mode
statement error
statement error \[CAST_INVALID_INPUT\] The value 'Infinity' of the type "DOUBLE" cannot be cast to "TIMESTAMP"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark 4.2.0 doesn't quote a double in this message. With ANSI mode on, CAST(CAST('NaN' AS DOUBLE) AS TIMESTAMP) fails with:

[CAST_INVALID_INPUT] The value NaN of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead.

Infinity is unquoted the same way. The PySpark runner doesn't compare error text, so it passes on these lines. The quotes come from SparkError's format string in #304 (see my comment on crates/spark/src/error.rs there). Once that changes, could these two expectations drop the quotes?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. With the #304 change merged in 346abab, both expectations drop the quotes, and so does the unit test that checked -Infinity.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

…ernel

- Parse the target type of `spark_cast(expr, '<type>')` with Spark's SQL
  type grammar, including decimals, CHAR/VARCHAR, TIMESTAMP variants and
  ARRAY, MAP and STRUCT types, and reject the types this crate does not
  support.
- Accept any input type, and call the Spark-compatible cast kernel with
  the eval mode and session time zone from the config.
- Port Comet's integer and float to TIMESTAMP casts into the kernel. In
  LEGACY mode an out-of-range float saturates, as Spark's
  `doubleToTimestamp` does, where Comet returns NULL.
Use spark_sql_type (apache#345) for the "not supported" and "not implemented
yet" errors, so they read "cast from BIGINT to TIMESTAMP_NTZ" rather
than "cast from Int64 to Timestamp(µs)", and check that parse_spark_type
round-trips with spark_sql_type.
@andygrove
andygrove marked this pull request as ready for review November 18, 2026 17:04
@andygrove
andygrove merged commit 99b44b3 into apache:main Nov 18, 2026
8 checks passed
@andygrove
andygrove deleted the feat-cast-udf branch November 18, 2026 17:05
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @mbutrovich!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants