Repository navigation
feat(spark): parse Spark type names in spark_cast and call the cast kernel - #317
Conversation
## Which issue does this PR close? Part of #56. It replaces that issue's plan to make 57.0.0 the first release from this repository. ## Rationale for this change New Spark-compatible functions and fixes land here between DataFusion major releases. Tying `datafusion-spark` releases to DataFusion's would hold them back, so this PR proposes releasing 56.x from `main` as often as needed, until DataFusion 57.0.0 is released. That only works if `main` doesn't break the public API in the meantime, so the policy spells out what counts as a breaking change. ## What changes are included in this PR? - **`dev/release/README.md`**: expands the Versioning section. - The major version matches DataFusion's. Minor releases (such as 56.1.0) add functions, features and fixes. Patch releases (such as 56.0.1) contain only fixes. - `main` stays on DataFusion 56 and releases 56.x until DataFusion 57.0.0 is released. Then it moves to DataFusion 57, and its next release is 57.0.0. - DataFusion 56.0.0 publishes `datafusion-spark` 56.0.0 from `branch-56` in apache/datafusion. Later 56.x releases come from this repository, so a DataFusion 56.x patch release should not publish `datafusion-spark` again. - What a 56.x release must not break, what is allowed, and how to avoid a break with `#[deprecated]`. A change to a function's results that makes it match Spark counts as a bug fix. - Checking a release against the previous one with `cargo-semver-checks`. - **`CONTRIBUTING.md`**: a short "API Compatibility" section for contributors that links to the policy. ## Are these changes tested? Documentation only. To check that the policy can be applied today, I ran `cargo semver-checks -p datafusion-spark --all-features` against the crate at apache/datafusion's `56.0.0-rc1` tag. Results: | Compared | Result | |---|---| | Current `main` | No breaking changes (229 checks pass) | | The open cast PRs (#89, #304, #317) | No breaking changes | | A test rename of a public function | Reported as a major-version change, so the check does detect breaks | Since rc1, `main` has only added public items, such as `dayname` and `date_from_unix_date`. ## Are there any user-facing changes? No code changes. The policy needs agreement with apache/datafusion: its `branch-56` release process still runs `cargo publish` for `datafusion-spark`, so DataFusion 56.x patch releases would need to stop publishing it after 56.0.0. _This PR was written by an LLM (Claude) and reviewed before opening._ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. This review covers only the commits after #304: the type parser, the UDF rework, and the numeric-to-timestamp casts. My #304 review covers the framework that this PR merges in.
| } | ||
| } | ||
| "string" => DataType::Utf8, | ||
| "char" | "character" | "varchar" => { |
There was a problem hiding this comment.
Spark requires a length for CHAR and VARCHAR (DataTypeAstBuilder.scala). In Spark 4.2.0, CAST(1 AS CHAR) fails with [DATATYPE_MISSING_SIZE] DataType "CHAR" requires a length parameter, and VARCHAR fails the same way. Here spark_cast(NULL, 'char') returns Utf8, because parameters() returns an empty list and only more than one length is rejected. Could this require exactly one length?
There was a problem hiding this comment.
Done in 15a4a69. A missing length now raises DATATYPE_MISSING_SIZE with Spark 4.2.0's exact text, which names the type as written (CHAR, CHARACTER or VARCHAR). A unit test covers all three, and cast_target_type.slt covers spark_cast(NULL, 'char').
| // Casts Spark supports that Arrow does not do the same way. | ||
| ( | ||
| Utf8 | LargeUtf8 | Utf8View, | ||
| Boolean | ||
| | Int8 | ||
| | Int16 | ||
| | Int32 | ||
| | Int64 | ||
| | Float32 | ||
| | Float64 | ||
| | Decimal128(_, _) | ||
| | Date32 | ||
| | Timestamp(_, _), | ||
| ) | ||
| | (Boolean | Decimal128(_, _), Timestamp(_, Some(_))) | ||
| | (Boolean | Int8 | Int16 | Int32 | Int64, Decimal128(_, _)) | ||
| | (Int8 | Int16 | Int32 | Int64, Binary) | ||
| | (Float32 | Float64, Decimal128(_, _) | Utf8) | ||
| | (Decimal128(_, _), Boolean | Float32 | Float64) | ||
| | (Date32, Timestamp(_, _)) | ||
| | (Struct(_) | List(_) | Map(_, _), Utf8) | ||
| | (Struct(_), Struct(_)) | ||
| | (List(_), List(_)) | ||
| | (Map(_, _), Map(_, _)) => not_yet_ported(from_type, to_type, eval_mode), |
There was a problem hiding this comment.
Several casts that Spark supports aren't in this arm, so they fall through to not_supported. The error then says Spark has no such cast. Spark 4.2.0 evaluates CAST(TIMESTAMP'2020-01-01 00:00:00' AS INT), AS SMALLINT, AS DOUBLE, and AS DECIMAL(20,2) in both modes, and AS BOOLEAN with ANSI mode off. With ANSI mode off, every cast from DATE to a numeric type or BOOLEAN returns NULL. Here spark_cast(spark_cast(1, 'timestamp'), 'int') fails with Spark-compatible cast from Timestamp(µs, "UTC") to Int32 is not supported. Could we add these pairs to this arm?
| (Timestamp(_, Some(_)), Boolean | Int8 | Int16 | Int32 | Float32 | Float64 | Decimal128(_, _))
| (Date32, Boolean | Int8 | Int16 | Int64 | Float32 | Float64 | Decimal128(_, _))(Date32, Int32) belongs here as well. My #304 review raises that pair in the allowlist.
| /// `spark_cast(expr, '<type>')`: Spark's `CAST(expr AS <type>)`. | ||
| /// | ||
| /// The target type is a string literal in Spark's SQL type syntax, such as | ||
| /// `'int'`, `'decimal(10, 2)'` or `'array<timestamp>'`. The cast follows | ||
| /// `datafusion.execution.enable_ansi_mode`, as Spark's follows | ||
| /// `spark.sql.ansi.enabled`, and uses `datafusion.execution.time_zone` as the | ||
| /// session time zone. | ||
| /// | ||
| /// # References | ||
| /// - <https://spark.apache.org/docs/latest/api/sql/index.html#cast> | ||
| #[derive(Debug, PartialEq, Eq, Hash)] | ||
| pub struct SparkCast { |
There was a problem hiding this comment.
spark_cast has no #[user_doc], and CONTRIBUTING.md asks for one on every function. Should we add it here, since this PR defines the target-type syntax and the casts that return "not implemented yet"? Without it, the function is missing from the reference that #115 generates.
There was a problem hiding this comment.
Added in 15a4a69. It documents the target-type syntax, the unsupported types, ANSI mode and the session time zone, and the "not implemented yet" vs "not supported" errors.
| statement error \[CAST_INVALID_INPUT\] The value 'NaN' of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed | ||
| SELECT spark_cast(arrow_cast('NaN', 'Float64'), 'timestamp'); | ||
|
|
||
| # Infinity should error in ANSI mode | ||
| statement error | ||
| statement error \[CAST_INVALID_INPUT\] The value 'Infinity' of the type "DOUBLE" cannot be cast to "TIMESTAMP" |
There was a problem hiding this comment.
Spark 4.2.0 doesn't quote a double in this message. With ANSI mode on, CAST(CAST('NaN' AS DOUBLE) AS TIMESTAMP) fails with:
[CAST_INVALID_INPUT] The value NaN of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead.
Infinity is unquoted the same way. The PySpark runner doesn't compare error text, so it passes on these lines. The quotes come from SparkError's format string in #304 (see my comment on crates/spark/src/error.rs there). Once that changes, could these two expectations drop the quotes?
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
…ernel - Parse the target type of `spark_cast(expr, '<type>')` with Spark's SQL type grammar, including decimals, CHAR/VARCHAR, TIMESTAMP variants and ARRAY, MAP and STRUCT types, and reject the types this crate does not support. - Accept any input type, and call the Spark-compatible cast kernel with the eval mode and session time zone from the config. - Port Comet's integer and float to TIMESTAMP casts into the kernel. In LEGACY mode an out-of-range float saturates, as Spark's `doubleToTimestamp` does, where Comet returns NULL.
Use spark_sql_type (apache#345) for the "not supported" and "not implemented yet" errors, so they read "cast from BIGINT to TIMESTAMP_NTZ" rather than "cast from Int64 to Timestamp(µs)", and check that parse_spark_type round-trips with spark_sql_type.
2a17d2b to
30fb25f
Compare
|
Thanks @mbutrovich! |
Which issue does this PR close?
Part of #88: the "Rework the
spark_castUDF" item.Rationale for this change
spark_cast(expr, 'timestamp')only supported atimestamptarget and integer or float input, with its own conversion code. To test the Spark-compatible cast kernel from SQL and SLT as each family is ported, the function needs to accept any Spark type name and call the kernel.What changes are included in this PR?
Target type parser (
cast/data_type.rs): parses the second argument with Spark'sdataTypegrammar fromSqlBaseParser.g4, case-insensitively.boolean,tinyint/byte,smallint/short,int/integer,bigint/long,float/real,double,date,binaryandstring.decimal/dec/numeric, with an optional precision and scale. A baredecimalisdecimal(10, 0), and precision and scale are validated.char(n)andvarchar(n)becomeSTRING, which is what Spark casts them to by default. A missing length raisesDATATYPE_MISSING_SIZE, as in Spark.timestamp,timestamp_ltzandtimestamp with local time zonegive aTIMESTAMPtagged with the session time zone. That matches Spark's defaultspark.sql.timestampType.timestamp_ntzandtimestamp without time zonegive aTIMESTAMP_NTZ.array<…>,map<…, …>(with Spark'sArrowUtilsfield namesentries,keyandvalue), andstruct<…>. Struct fields can be backquoted,NOT NULLor have aCOMMENT, and duplicate field names are allowed, as in Spark.TIME,VARIANT,VOID, geospatial types and collated strings get an "unsupported" error. Malformed type names get a parse error that names the problem.spark_castUDF:EvalMode::from_configand the session time zone as aSparkTimeZoneThe old conversion code in the UDF is removed.
Error messages: the kernel's "not supported" and "not implemented yet" errors name types the Spark way, using
spark_sql_typefrom refactor(spark): share Spark's common-type rules between functions #345. For example, they now saycast from BIGINT to TIMESTAMP_NTZinstead ofcast from Int64 to Timestamp(µs).Integer and float to
TIMESTAMP: ported into the kernel (cast/numeric.rs). Spark has no cast from a number toTIMESTAMP_NTZ, so that cast is rejected.Float input follows Spark 4.2.0's
doubleToTimestampanddoubleToTimestampAnsi:toLongCAST_INVALID_INPUTCAST_OVERFLOW, such as1.0E25DfromDOUBLEtoBIGINTThe previous UDF already saturated in Legacy mode. Comet returns NULL there, which differs from Spark.
Are these changes tested?
Yes.
FLOATwidening and the error messagesparse_spark_typeandspark_sql_typefor each canonical Spark type name#[user_doc]forspark_cast, which documents the target-type syntax and the two kinds of error.conversion/cast_target_type.slt: return types fromarrow_typeof, the session time zone ontimestamp, casts that already work through Arrow, invalid and unsupported targets, and casts that are not implemented yet.cast_int_to_timestamp.sltandcast_float_to_timestamp.sltstill pass, with these changes:spark_cast(100, 'string'), which now works, so it now usesintervalinstead.timestamp_ntz.bigint. The old file said this couldn't be shown in SLT.Are there any user-facing changes?
Yes.
spark_castaccepts any Spark type name as its target and any input type.spark_cast(42, 'string').1is aBIGINT, so evenspark_cast(1, 'int')needs that port.[CAST_INVALID_INPUT]and[CAST_OVERFLOW]messages.This PR was written by an LLM (Claude) from the Comet and Spark sources, and reviewed before opening.
🤖 Generated with Claude Code