Skip to content

feat(spark): add Spark-compatible make_date - #329

Merged
andygrove merged 8 commits into
apache:mainfrom
andygrove:port-make-date
Nov 19, 2026
Merged

andygrove merged 8 commits into
apache:mainfrom
andygrove:port-make-date

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #149.

Summary

Adds a Spark-compatible make_date(year, month, day) to datafusion-spark, ported from DataFusion Comet's datetime_funcs/make_date.rs. It replaces DataFusion's built-in make_date when the Spark functions are registered.

  • Arguments are coerced to INT (integer, floating-point, decimal and string inputs) with the same int32_coercion() helper as date_from_unix_date. BOOLEAN is rejected.
  • If any input is NULL, the result is NULL. Rows with a NULL input are skipped before any check, so under ANSI an invalid month next to a NULL year does not raise an error.
  • Invalid dates (month 13, day 0, February 29 in a non-leap year, a year outside ±999999999) return NULL when ANSI mode is off. With datafusion.execution.enable_ansi_mode = true they raise [DATETIME_FIELD_OUT_OF_BOUNDS.WITH_SUGGESTION] with Spark's message and SQLSTATE 22023. The messages are the ones java.time.LocalDate.of produces, and the checks run in the same order.
  • The full java.time year range is supported, including year 0 and negative years. Day counts are computed in i64.
  • A valid date whose day count does not fit in DATE raises [ARITHMETIC_OVERFLOW] in both modes. This matches Spark, where MathUtils.toIntExact is outside the DateTimeException handler.
  • With all-scalar arguments and zero rows, an empty array comes back without the arguments being checked. As a result, SELECT make_date(2024, 13, 1) FROM t WHERE false under ANSI returns no rows, as it does in Spark.
  • The ANSI mode is read from ScalarFunctionArgs::config_options, as abs and elt do. apache/main has no crate-root EvalMode yet.
  • The result field is always nullable, because ANSI mode is not known at planning time.
  • Spark has no try_make_date, so none is added.

Differences from DataFusion's built-in make_date

  • The built-in raises an error for an invalid date regardless of ANSI mode. This version returns NULL with ANSI off.
  • The built-in accepts only integer and string arguments. This version also accepts floating-point and decimal arguments, as Spark does.
  • The built-in is limited to chrono's year range (-262143 to 262142), so make_date(300000, 6, 15) fails there. This version accepts the years Spark accepts.

Differences from Comet's version

  • Out-of-range epoch day: Comet returns NULL (ANSI off) or DATETIME_FIELD_OUT_OF_BOUNDS (ANSI on) for a valid date outside the DATE range, for example make_date(5881580, 7, 12). Spark 4.2.0 raises ARITHMETIC_OVERFLOW in both modes, and this version does the same.
  • Mode configuration: Comet takes fail_on_error in the constructor. This version reads enable_ansi_mode from the session config.
  • Signature: Comet uses Signature::any(3) and casts every argument to Int32 at runtime. This version uses a coercible INT signature, so a BOOLEAN argument is rejected at planning time, as in Spark.
  • Empty batches: Comet's MakeDateExpr wrapper skips the whole expression on empty batches, including errors raised while evaluating the arguments (for example an ANSI cast of 'bad'). This version only skips the date check for all-scalar arguments when number_rows == 0. Errors raised while evaluating the arguments of a row-less batch are a DataFusion planner concern and are out of scope here.
  • Errors: Errors are DataFusion execution errors carrying Spark's message text rather than Comet's SparkError type.

Known divergences from Spark (tracked by #88)

These are the same cast-related gaps as in date_from_unix_date. They are captured as commented-out .slt queries:

  • With ANSI off, Spark returns NULL for make_date('abc', '2', '3'). DataFusion raises a cast error.
  • With ANSI off, Spark wraps an out-of-range BIGINT to INT, which gives an invalid year and so NULL, for example make_date(3000000000, 1, 1). DataFusion raises a cast error.
  • With ANSI off, Spark casts NaN to 0, so make_date(CAST('NaN' AS DOUBLE), 1, 1) is 0000-01-01. An infinite or out-of-range DOUBLE saturates to 2147483647, an invalid year, so the result is NULL. A DECIMAL out of the INT range wraps like a BIGINT, so make_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1) is 2024-01-01. DataFusion raises a cast error for all of these.

Testing

  • Spark source: Read Spark's MakeDate in 3.5.8 and 4.1.1. Semantics are the same in both; only the name of the error helper differs.
  • PySpark 4.2.0: Checked every expected value against PySpark 4.2.0 with ANSI both on and off. That includes valid and invalid dates, every error message, the year 0, negative and large year cases, the overflow boundaries, NULL handling, array inputs and type coercion.
  • SLT runner: crates/sqllogictest/scripts/pyspark_slt_runner.py on datetime/make_date.slt gave 34 pass and 1 skipped (arrow_typeof does not exist in Spark).
  • Unit tests: Ported Comet's unit tests and added tests for the ANSI messages, the overflow error, scalar and array invocation, and the empty-batch behavior.
  • Commands that passed:
    • cargo fmt --all
    • cargo clippy --all-targets --all-features -- -D warnings
    • cargo test -p datafusion-spark (346 passed)
    • cargo test -p datafusion-spark-sqllogictest (249 of 249 files passed)

Port make_date from DataFusion Comet. Invalid dates return NULL, or raise
DATETIME_FIELD_OUT_OF_BOUNDS when ANSI mode is enabled, with the
java.time.LocalDate.of messages Spark surfaces. A valid date outside the
DATE range raises ARITHMETIC_OVERFLOW in both modes, as in Spark.

Closes apache#149.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The java.time messages and the ARITHMETIC_OVERFLOW case match Spark 4.2.0 exactly in both ANSI modes, and the PySpark runner passes on the new .slt file.

Comment on lines +26 to +27
description = "Creates a date from `year`, `month` and `day` fields. DataFusion has no built-in \
equivalent. Years from -999999999 to 999999999 are accepted (year 0 is 1 BC), as long as \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataFusion does have a built-in make_date (MakeDateFunc on branch-56). Registering the Spark functions replaces it, so this is a behavior change for make_date users. CONTRIBUTING.md asks the description to say how a function differs from the built-in of the same name.

From its code, the built-in differs in three ways:

  • It raises an error for an invalid date regardless of ANSI mode (make_date_inner).
  • It accepts only integer and string arguments.
  • It is limited to chrono's year range (-262143 to 262142), so make_date(300000, 6, 15) fails there.

Could we replace this sentence with those differences? Could the PR description also say that it replaces the built-in?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I hadn't noticed the built-in. I checked all three by running DataFusion's make_date, and they hold. 777724a replaces the sentence with those differences, and the PR description now says it replaces the built-in and lists them.

Comment on lines +35 to +36
With ANSI mode disabled, Spark wraps an out-of-range BIGINT and returns NULL for a \
malformed string, while this function still errors (see \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ANSI-off cast gaps also cover floating-point and decimal arguments, which this sentence and the commented-out queries in the .slt file don't mention. On Spark 4.2.0 with ANSI off:

  • make_date(CAST('NaN' AS DOUBLE), 1, 1) returns 0000-01-01, because Spark casts NaN to 0.
  • make_date(CAST('Infinity' AS DOUBLE), 1, 1) returns NULL.
  • make_date(CAST(99999999999 AS DECIMAL(20,0)), 1, 1) returns NULL.

Here all three raise a cast error, for example Can't cast value NaN to type Int32. NaN is the surprising one, because Spark returns a date rather than NULL. Could we add these as commented-out queries citing #88, and extend this sentence to cover them?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 777724a as commented-out queries citing #88, and the sentence now covers NaN, infinite DOUBLEs and out-of-range DECIMALs. I confirmed on PySpark 4.2.0 that NaN gives 0000-01-01.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 777724a as commented-out queries citing #88, and the sentence now covers NaN, infinite DOUBLEs and out-of-range DECIMALs.

I got the DECIMAL case wrong in my first comment, sorry. With ANSI mode off, Spark casts to INT with numeric.toInt (Cast.scala), which for a DECIMAL is toLong.toInt (Decimal.scala). So an out-of-range DECIMAL wraps like a BIGINT. My 99999999999 example returns NULL only because it wraps to 1215752191, which is past the year range. On Spark 4.2.0 with ANSI mode off, make_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1) returns 2024-01-01. Here it raises Cast error: value of 4294969320 is out of range Int32.

A DOUBLE out of the INT range saturates instead. CAST(CAST(1e10 AS DOUBLE) AS INT) and CAST(CAST('Infinity' AS DOUBLE) AS INT) are both 2147483647, which is never a valid field. So make_date returns NULL for any of them, not only for infinities.

Could the sentence in the description (make_date.rs L35-L37) read like this?

With ANSI mode disabled, Spark wraps an out-of-range BIGINT or DECIMAL, returns NULL for a
malformed string or a floating-point value out of the INT range, and casts NaN to 0, while
this function still errors

Could the comment above the commented-out queries in make_date.slt (L125-L127) say the same, with the wrapping case added?

# query D
# SELECT make_date(CAST(4294969320 AS DECIMAL(20,0)), 1, 1);
# ----
# 2024-01-01

The int32_coercion() comment (date_part_name.rs L39-L40) says Spark "returns NULL or wraps". For date_from_unix_date the saturation gives a date: date_from_unix_date(CAST(1e10 AS DOUBLE)) is +5881580-07-11 on Spark 4.2.0 with ANSI mode off. Could that comment say "returns NULL, wraps, or saturates"?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the correction. In 98e1f48 the doc sentence now uses your wording. The slt file has the DECIMAL wrapping case you gave, a 1e10 DOUBLE case, and comments on saturation and wrapping. The int32_coercion comment now says "returns NULL, wraps, or saturates". I checked each value on PySpark 4.2.0 with ANSI off.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In 98e1f48 the doc sentence now uses your wording.

My wording missed one more case, sorry. With ANSI mode off, Spark's cast from STRING to INT drops a fractional part, so make_date('2024.7', '1', '1') and make_date('2024', '1.9', '1') both return 2024-01-01 on Spark 4.2.0. Here both raise Cannot cast string '2024.7' to value of Int32 type (or '1.9') in both modes. With ANSI mode on, Spark raises CAST_INVALID_INPUT, so that half already matches.

Could the sentence (make_date.rs L35-L37) read like this?

With ANSI mode disabled, Spark wraps an out-of-range BIGINT or DECIMAL, truncates a string
with a fractional part, returns NULL for a malformed string or a floating-point value out of
the INT range, and casts NaN to 0, while this function still errors

Could make_date.slt also record it next to the malformed-string query (L110-L115)?

# query D
# SELECT make_date('2024.7', '1', '1');
# ----
# 2024-01-01

The int32_coercion() comment (date_part_name.rs L38-L40) misses two cases for date_from_unix_date. On Spark 4.2.0 with ANSI mode off, date_from_unix_date('19000.9') is 2022-01-08 and date_from_unix_date(CAST('NaN' AS DOUBLE)) is 1970-01-01. Both raise a cast error here. That list has now been incomplete twice, and the #[user_doc] descriptions carry the details. Could the comment say "which errors in cases where Spark with ANSI mode off returns a value or NULL" instead of naming each case?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all three are in 04199a2. The #[user_doc] sentence now uses your wording, make_date.slt has the '2024.7' case, and the int32_coercion() comment now says it errors in cases where Spark with ANSI mode off returns a value or NULL. I checked the four values you gave on PySpark 4.2.0 in both modes.

Since #368 landed, I also turned the commented-out records into live ones in 1815b8b. Each asserts DataFusion's current cast error and has a # xfail spark: marker linking #88, so the PySpark runner tracks them and will flag when they start matching. The NaN record casts the date to STRING, because PySpark can't convert year 0 to a Python date. After merging main, the unix_date record is skipif evaluate as in unix.slt, until #306 is fixed. All datetime/ files pass in both modes, and the runner reports 32 passes and 8 expected failures for make_date.slt.

Comment on lines +75 to +86
let int32 = || {
Coercion::new_implicit(
TypeSignatureClass::Native(logical_int32()),
vec![
TypeSignatureClass::Integer,
TypeSignatureClass::Float,
TypeSignatureClass::Decimal,
TypeSignatureClass::Native(logical_string()),
],
NativeType::Int32,
)
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This coercion is the same as the one in date_from_unix_date (date_from_unix_date.rs). Both encode Spark's implicit cast to INT, and they'll need to change together once Spark-compatible casts land (#88). Could we move it into a shared helper next to date_signature() in date_part_name.rs, and use it from both functions?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 777724a. int32_coercion() now sits next to date_signature() in date_part_name.rs, and both make_date and date_from_unix_date use it.

query error
SELECT make_date(true, 1, 1);

# A constant invalid date is not evaluated when there are no rows

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This query passes even with the number_rows == 0 branch in invoke_with_args disabled, so it doesn't check what the comment says. I changed the condition to if false && number_rows == 0. This file still passed, and only test_make_date_scalar_empty_batch_does_not_validate failed. Could we drop this query and rely on that unit test, or change the comment to describe what the query does check?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped it in 777724a, since test_make_date_scalar_empty_batch_does_not_validate covers that case.

2024-02-29

query D
SELECT make_date(2019, 7, NULL);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add the literal NULL short-circuit cases here, with ANSI on? SELECT make_date(2024, 13, NULL), make_date(NULL, 13, 40); returns NULL NULL on Spark 4.2.0 in ANSI mode, and it does here too. Nothing checks that order on the all-scalar path, which has its own match arm. The column case further down covers only the array path.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 777724a, in the ANSI block.

…t-in

Say how it differs from DataFusion's make_date, which it replaces, and
which ANSI-off casts still differ from Spark, with commented-out queries
citing apache#88. Share the INT coercion with date_from_unix_date. Test the
literal NULL short-circuit in ANSI mode, and drop an empty-batch query
that the unit test already covers.

@mbutrovich mbutrovich left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. The DECIMAL and DOUBLE cases in 98e1f48 match Spark 4.2.0, including a DECIMAL past the BIGINT range and infinities of either sign.

reply with changes here: #329 (comment)

… off

Spark with ANSI mode off truncates a string with a fractional part when
casting it to INT, while this function errors. Record it in the make_date
docs and slt, and make the int32_coercion comment general instead of listing
each case.
Turn the commented-out ANSI-off cast records into live records that assert
DataFusion's current error, each with a # xfail spark: marker that links apache#88,
so the PySpark runner reports them and flags when they start matching. Skip
the unix_date record in evaluate mode, as unix.slt does, until apache#306 is fixed.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

@andygrove
andygrove merged commit d3e4675 into apache:main Nov 19, 2026
10 checks passed
@andygrove
andygrove deleted the port-make-date branch November 19, 2026 15:03
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @mbutrovich!

andygrove added a commit that referenced this pull request Nov 19, 2026
#378)

#115 added a CI check that `docs/functions` matches the `#[user_doc]`
attributes. It merged after #329 and #357, which changed those
attributes, so the Lint workflow fails on main. This reruns
`dev/update_function_docs.sh`. It only changes generated docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port make_date from Comet

2 participants