Repository navigation
feat(spark): add Spark-compatible make_date - #329
Conversation
Port make_date from DataFusion Comet. Invalid dates return NULL, or raise DATETIME_FIELD_OUT_OF_BOUNDS when ANSI mode is enabled, with the java.time.LocalDate.of messages Spark surfaces. A valid date outside the DATE range raises ARITHMETIC_OVERFLOW in both modes, as in Spark. Closes apache#149.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. The java.time messages and the ARITHMETIC_OVERFLOW case match Spark 4.2.0 exactly in both ANSI modes, and the PySpark runner passes on the new .slt file.
| description = "Creates a date from `year`, `month` and `day` fields. DataFusion has no built-in \ | ||
| equivalent. Years from -999999999 to 999999999 are accepted (year 0 is 1 BC), as long as \ |
There was a problem hiding this comment.
DataFusion does have a built-in make_date (MakeDateFunc on branch-56). Registering the Spark functions replaces it, so this is a behavior change for make_date users. CONTRIBUTING.md asks the description to say how a function differs from the built-in of the same name.
From its code, the built-in differs in three ways:
- It raises an error for an invalid date regardless of ANSI mode (
make_date_inner). - It accepts only integer and string arguments.
- It is limited to chrono's year range (-262143 to 262142), so
make_date(300000, 6, 15)fails there.
Could we replace this sentence with those differences? Could the PR description also say that it replaces the built-in?
There was a problem hiding this comment.
Thanks, I hadn't noticed the built-in. I checked all three by running DataFusion's make_date, and they hold. 777724a replaces the sentence with those differences, and the PR description now says it replaces the built-in and lists them.
| With ANSI mode disabled, Spark wraps an out-of-range BIGINT and returns NULL for a \ | ||
| malformed string, while this function still errors (see \ |
There was a problem hiding this comment.
The ANSI-off cast gaps also cover floating-point and decimal arguments, which this sentence and the commented-out queries in the .slt file don't mention. On Spark 4.2.0 with ANSI off:
make_date(CAST('NaN' AS DOUBLE), 1, 1)returns0000-01-01, because Spark casts NaN to 0.make_date(CAST('Infinity' AS DOUBLE), 1, 1)returns NULL.make_date(CAST(99999999999 AS DECIMAL(20,0)), 1, 1)returns NULL.
Here all three raise a cast error, for example Can't cast value NaN to type Int32. NaN is the surprising one, because Spark returns a date rather than NULL. Could we add these as commented-out queries citing #88, and extend this sentence to cover them?
There was a problem hiding this comment.
Added in 777724a as commented-out queries citing #88, and the sentence now covers NaN, infinite DOUBLEs and out-of-range DECIMALs.
I got the DECIMAL case wrong in my first comment, sorry. With ANSI mode off, Spark casts to INT with numeric.toInt (Cast.scala), which for a DECIMAL is toLong.toInt (Decimal.scala). So an out-of-range DECIMAL wraps like a BIGINT. My 99999999999 example returns NULL only because it wraps to 1215752191, which is past the year range. On Spark 4.2.0 with ANSI mode off, make_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1) returns 2024-01-01. Here it raises Cast error: value of 4294969320 is out of range Int32.
A DOUBLE out of the INT range saturates instead. CAST(CAST(1e10 AS DOUBLE) AS INT) and CAST(CAST('Infinity' AS DOUBLE) AS INT) are both 2147483647, which is never a valid field. So make_date returns NULL for any of them, not only for infinities.
Could the sentence in the description (make_date.rs L35-L37) read like this?
With ANSI mode disabled, Spark wraps an out-of-range BIGINT or DECIMAL, returns NULL for a
malformed string or a floating-point value out of the INT range, and casts NaN to 0, while
this function still errors
Could the comment above the commented-out queries in make_date.slt (L125-L127) say the same, with the wrapping case added?
# query D
# SELECT make_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1);
# ----
# 2024-01-01
The int32_coercion() comment (date_part_name.rs L39-L40) says Spark "returns NULL or wraps". For date_from_unix_date the saturation gives a date: date_from_unix_date(CAST(1e10 AS DOUBLE)) is +5881580-07-11 on Spark 4.2.0 with ANSI mode off. Could that comment say "returns NULL, wraps, or saturates"?
There was a problem hiding this comment.
Thanks for the correction. In 98e1f48 the doc sentence now uses your wording. The slt file has the DECIMAL wrapping case you gave, a 1e10 DOUBLE case, and comments on saturation and wrapping. The int32_coercion comment now says "returns NULL, wraps, or saturates". I checked each value on PySpark 4.2.0 with ANSI off.
There was a problem hiding this comment.
In 98e1f48 the doc sentence now uses your wording.
My wording missed one more case, sorry. With ANSI mode off, Spark's cast from STRING to INT drops a fractional part, so make_date('2024.7', '1', '1') and make_date('2024', '1.9', '1') both return 2024-01-01 on Spark 4.2.0. Here both raise Cannot cast string '2024.7' to value of Int32 type (or '1.9') in both modes. With ANSI mode on, Spark raises CAST_INVALID_INPUT, so that half already matches.
Could the sentence (make_date.rs L35-L37) read like this?
With ANSI mode disabled, Spark wraps an out-of-range BIGINT or DECIMAL, truncates a string
with a fractional part, returns NULL for a malformed string or a floating-point value out of
the INT range, and casts NaN to 0, while this function still errors
Could make_date.slt also record it next to the malformed-string query (L110-L115)?
# query D
# SELECT make_date('2024.7', '1', '1');
# ----
# 2024-01-01
The int32_coercion() comment (date_part_name.rs L38-L40) misses two cases for date_from_unix_date. On Spark 4.2.0 with ANSI mode off, date_from_unix_date('19000.9') is 2022-01-08 and date_from_unix_date(CAST('NaN' AS DOUBLE)) is 1970-01-01. Both raise a cast error here. That list has now been incomplete twice, and the #[user_doc] descriptions carry the details. Could the comment say "which errors in cases where Spark with ANSI mode off returns a value or NULL" instead of naming each case?
There was a problem hiding this comment.
Thanks, all three are in 04199a2. The #[user_doc] sentence now uses your wording, make_date.slt has the '2024.7' case, and the int32_coercion() comment now says it errors in cases where Spark with ANSI mode off returns a value or NULL. I checked the four values you gave on PySpark 4.2.0 in both modes.
Since #368 landed, I also turned the commented-out records into live ones in 1815b8b. Each asserts DataFusion's current cast error and has a # xfail spark: marker linking #88, so the PySpark runner tracks them and will flag when they start matching. The NaN record casts the date to STRING, because PySpark can't convert year 0 to a Python date. After merging main, the unix_date record is skipif evaluate as in unix.slt, until #306 is fixed. All datetime/ files pass in both modes, and the runner reports 32 passes and 8 expected failures for make_date.slt.
| let int32 = || { | ||
| Coercion::new_implicit( | ||
| TypeSignatureClass::Native(logical_int32()), | ||
| vec![ | ||
| TypeSignatureClass::Integer, | ||
| TypeSignatureClass::Float, | ||
| TypeSignatureClass::Decimal, | ||
| TypeSignatureClass::Native(logical_string()), | ||
| ], | ||
| NativeType::Int32, | ||
| ) | ||
| }; |
There was a problem hiding this comment.
This coercion is the same as the one in date_from_unix_date (date_from_unix_date.rs). Both encode Spark's implicit cast to INT, and they'll need to change together once Spark-compatible casts land (#88). Could we move it into a shared helper next to date_signature() in date_part_name.rs, and use it from both functions?
There was a problem hiding this comment.
Done in 777724a. int32_coercion() now sits next to date_signature() in date_part_name.rs, and both make_date and date_from_unix_date use it.
| query error | ||
| SELECT make_date(true, 1, 1); | ||
|
|
||
| # A constant invalid date is not evaluated when there are no rows |
There was a problem hiding this comment.
This query passes even with the number_rows == 0 branch in invoke_with_args disabled, so it doesn't check what the comment says. I changed the condition to if false && number_rows == 0. This file still passed, and only test_make_date_scalar_empty_batch_does_not_validate failed. Could we drop this query and rely on that unit test, or change the comment to describe what the query does check?
There was a problem hiding this comment.
Dropped it in 777724a, since test_make_date_scalar_empty_batch_does_not_validate covers that case.
| 2024-02-29 | ||
|
|
||
| query D | ||
| SELECT make_date(2019, 7, NULL); |
There was a problem hiding this comment.
Could we add the literal NULL short-circuit cases here, with ANSI on? SELECT make_date(2024, 13, NULL), make_date(NULL, 13, 40); returns NULL NULL on Spark 4.2.0 in ANSI mode, and it does here too. Nothing checks that order on the all-scalar path, which has its own match arm. The column case further down covers only the array path.
…t-in Say how it differs from DataFusion's make_date, which it replaces, and which ANSI-off casts still differ from Spark, with commented-out queries citing apache#88. Share the INT coercion with date_from_unix_date. Test the literal NULL short-circuit in ANSI mode, and drop an empty-batch query that the unit test already covers.
There was a problem hiding this comment.
Thanks @andygrove. The DECIMAL and DOUBLE cases in 98e1f48 match Spark 4.2.0, including a DECIMAL past the BIGINT range and infinities of either sign.
reply with changes here: #329 (comment)
… off Spark with ANSI mode off truncates a string with a fractional part when casting it to INT, while this function errors. Record it in the make_date docs and slt, and make the int32_coercion comment general instead of listing each case.
Turn the commented-out ANSI-off cast records into live records that assert DataFusion's current error, each with a # xfail spark: marker that links apache#88, so the PySpark runner reports them and flags when they start matching. Skip the unix_date record in evaluate mode, as unix.slt does, until apache#306 is fixed.
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
|
Thanks @mbutrovich! |
Closes #149.
Summary
Adds a Spark-compatible
make_date(year, month, day)todatafusion-spark, ported from DataFusion Comet'sdatetime_funcs/make_date.rs. It replaces DataFusion's built-inmake_datewhen the Spark functions are registered.int32_coercion()helper asdate_from_unix_date. BOOLEAN is rejected.datafusion.execution.enable_ansi_mode = truethey raise[DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION]with Spark's message and SQLSTATE 22023. The messages are the onesjava.time.LocalDate.ofproduces, and the checks run in the same order.java.timeyear range is supported, including year 0 and negative years. Day counts are computed ini64.[ARITHMETIC_OVERFLOW]in both modes. This matches Spark, whereMathUtils.toIntExactis outside theDateTimeExceptionhandler.SELECT make_date(2024, 13, 1) FROM t WHERE falseunder ANSI returns no rows, as it does in Spark.ScalarFunctionArgs::config_options, asabsandeltdo.apache/mainhas no crate-rootEvalModeyet.try_make_date, so none is added.Differences from DataFusion's built-in
make_datemake_date(300000, 6, 15)fails there. This version accepts the years Spark accepts.Differences from Comet's version
DATETIME_FIELD_OUT_OF_BOUNDS(ANSI on) for a valid date outside the DATE range, for examplemake_date(5881580, 7, 12). Spark 4.2.0 raisesARITHMETIC_OVERFLOWin both modes, and this version does the same.fail_on_errorin the constructor. This version readsenable_ansi_modefrom the session config.Signature::any(3)and casts every argument to Int32 at runtime. This version uses a coercible INT signature, so a BOOLEAN argument is rejected at planning time, as in Spark.MakeDateExprwrapper skips the whole expression on empty batches, including errors raised while evaluating the arguments (for example an ANSI cast of'bad'). This version only skips the date check for all-scalar arguments whennumber_rows == 0. Errors raised while evaluating the arguments of a row-less batch are a DataFusion planner concern and are out of scope here.SparkErrortype.Known divergences from Spark (tracked by #88)
These are the same cast-related gaps as in
date_from_unix_date. They are captured as commented-out.sltqueries:make_date('abc', '2', '3'). DataFusion raises a cast error.make_date(3000000000, 1, 1). DataFusion raises a cast error.make_date(CAST('NaN' AS DOUBLE), 1, 1)is0000-01-01. An infinite or out-of-range DOUBLE saturates to 2147483647, an invalid year, so the result is NULL. A DECIMAL out of the INT range wraps like a BIGINT, somake_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1)is2024-01-01. DataFusion raises a cast error for all of these.Testing
MakeDatein 3.5.8 and 4.1.1. Semantics are the same in both; only the name of the error helper differs.crates/sqllogictest/scripts/pyspark_slt_runner.pyondatetime/make_date.sltgave 34 pass and 1 skipped (arrow_typeofdoes not exist in Spark).cargo fmt --allcargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-spark(346 passed)cargo test -p datafusion-spark-sqllogictest(249 of 249 files passed)