Skip to content

feat(spark): add Spark-compatible dayofweek - #327

Open
andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:port-dayofweek
Open

andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:port-dayofweek

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Closes #151.

Summary

Adds a Spark-compatible dayofweek(date), which returns an INT from 1 (Sunday) to 7 (Saturday). It is ported from Comet's datetime_funcs/day_of_week.rs.

  • New crates/spark/src/function/datetime/dayofweek.rs. The result comes straight from the Date32 epoch day: ((days + 4) mod 7) + 1, computed in i64. This matches Spark's LocalDate.ofEpochDay(days).getDayOfWeek.plus(1).getValue, which is defined for every Int.
  • invoke_with_args handles scalar and array inputs, NULL and untyped NULL, and dictionary-encoded Date32. For a dictionary, it maps the values, rewraps the keys, and unpacks the result to plain Int32.
  • The signature is the same date_signature() that dayname and monthname use. Strings and timestamps are implicitly cast to DATE, as in Spark.
  • Documented with #[user_doc] and registered in datetime/mod.rs.
  • weekday now uses the same epoch-day helper instead of arrow's date_part(DayOfWeekMonday0). arrow builds a chrono date for each row, so dates outside chrono's range (such as date_from_unix_date(±2^31)) came back NULL. Spark returns 1 and 4 for those dates, and so does weekday now. The weekday signature is unchanged. Both functions cast a Date64 input (also inside a dictionary) to epoch days, so they accept every type their signatures admit.

Differences from Comet's version

  • Comet uses Signature::any(1) and expects the planner to hand it a Date32. Here the function has a coercible date signature, so strings and timestamps work from SQL. Comet's name spark_dayofweek becomes dayofweek.
  • A scalar input returns a scalar here. Comet always expands it to an array.
  • return_field_from_args passes the input's nullability through to the result, the same as weekday.
  • The kernel, the null-density switch between unary and unary_opt, and the dictionary handling are the same as in Comet. I found no behavioral difference between Comet and Spark 4.2.0 for Date32 input.

Known differences from Spark, which also apply to the existing dayname and weekday and are left as commented-out cases in the .slt:

Testing

  • Every expected value was checked against PySpark 4.2.0. This covers all seven weekdays, the epoch, a leap day, 0001-01-01, 9999-12-31, the 1582 Gregorian cutover, date_from_unix_date(-2147483648) = 3 and (2147483647) = 6, NULL and untyped NULL (int), strings, TIMESTAMP_NTZ, TIMESTAMP in UTC and in America/New_York, malformed strings with ANSI on and off, and an INT argument (DATATYPE_MISMATCH).
  • crates/sqllogictest/scripts/pyspark_slt_runner.py on dayofweek.slt and weekday.slt: every dayofweek record passes on Spark 4.2.0. Skipped records are the arrow_typeof checks and the dictionary case, marked skipif spark. The one weekday mismatch, weekday('not-a-date'), was already there (strings are not cast for weekday, [Bug] last_day, monthname, weekday, round and unbase64 reject arguments that Spark implicitly casts #41).
  • Unit tests ported from Comet: the week around the epoch, nulls at both densities, the full i32 range, agreement with weekday over a 400-year cycle, dictionary input, non-date input rejected, and scalar input. A nullability test was added.
  • Passing: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest (249 of 249 files).

Port dayofweek from DataFusion Comet. The value is computed straight from
the Date32 epoch day, so it is defined for the whole Int32 range as in
Spark. weekday now shares the same kernel, which fixes it returning NULL
for dates outside chrono's calendar range.
@andygrove
andygrove requested a review from kosiew November 17, 2026 13:21

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove. Computing the day from the epoch day, instead of through a chrono date, is a nice way to cover the whole Date32 range.

let mapped = Arc::new(map_dates(values, kernel)) as ArrayRef;
return Ok(cast(&dict.with_values(mapped), &DataType::Int32)?);
}
let Some(dates) = array.as_primitive_opt::<Date32Type>() else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

date_signature() matches the logical date type, so it also admits Date64, but this only handles Date32. SELECT dayofweek(arrow_cast('2024-01-07', 'Date64')) plans and then fails with Execution error: `dayofweek` expects a date, got Date64.

weekday has the same problem after this PR. On main it goes through arrow's date_part, which handles Date64, and SELECT weekday(arrow_cast('2024-01-07', 'Date64')) returns 6. With this PR it fails with the same error.

Could we cast a Date64 input to Date32 at the top of epoch_day_array? Then both functions handle every type their signatures accept.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in be65c39: epoch_day_array now casts a Date64 input to Date32 first, including Date64 dictionary values, so dayofweek and weekday handle every type their signatures accept. Both .slt files cover a scalar, a column with NULL and a date before the epoch, and dayofweek.slt also covers a dictionary. Both files fail without the fix.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both .slt files cover a scalar, a column with NULL and a date before the epoch, and dayofweek.slt also covers a dictionary.

The dictionary query in dayofweek.slt doesn't reach the new DataType::Date64 arm in the dictionary branch (dayofweek.rs L97-L100). I replaced that arm with DataType::Date64 => Arc::clone(dict.values()), and dayofweek.slt still passed. Coercion casts the dictionary to its value type before the function runs. With a column input, EXPLAIN shows dayofweek(CAST(CAST(CAST(column1 AS Date64) AS Dictionary(Int32, Date64)) AS Date64)), and the all-literal query in the file is folded to Int32(1) at planning.

Comet calls invoke_with_args directly, so only a unit test reaches this arm, the way dictionary_input_unpacks does for Date32 (L286-L299). Could we add one next to it, and drop the dictionary query from dayofweek.slt? For example, with Date64Array added to the use arrow::array::DictionaryArray; import:

#[test]
fn date64_dictionary_input() {
    let values =
        Arc::new(Date64Array::from(vec![Some(0), Some(-86_400_000), None])) as ArrayRef;
    let keys = Int32Array::from(vec![Some(0), Some(1), None, Some(2)]);
    let dict = DictionaryArray::<Int32Type>::new(keys, values);
    let out = invoke_array(Arc::new(dict));
    assert_eq!(out.iter().collect::<Vec<_>>(), [Some(5), Some(4), None, None]);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added date64_dictionary_input next to the Date32 dictionary test and removed the Date64 dictionary query from the .slt in 3e80f6c. With the arm replaced by Arc::clone(dict.values()) the new test fails with "expects a date, got Dictionary(Int32, Date64)".

NULL
1

# Spark returns NULL for a malformed string when ANSI mode is off. DataFusion's

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark also accepts partial dates as strings, in both ANSI modes. On Spark 4.2.0, SELECT dayofweek('2009') returns 5 and SELECT dayofweek('2009-07') returns 4. DataFusion's cast rejects both with ANSI on or off (Cannot cast string '2009' to value of Date32 type).

This comes from date_signature() on main (date_part_name.rs), so dayname and monthname behave the same way. The "String to date" item in #88 covers the fix. Could we add these two as commented-out queries with Spark's results next to this one? Could the #[user_doc] description also say that strings Spark reads as partial dates raise an error, not only malformed ones?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in be65c39 as a commented-out query with Spark's results (5 and 4, which I confirmed on PySpark 4.2.0 in both modes), citing #88. The #[user_doc] now says strings Spark reads as partial dates are an error as well.

date_signature admits Date64, so cast it to epoch days before computing
the day of the week. Note that strings Spark reads as partial dates are
an error here, with commented-out queries citing apache#88.
… test

Coercion unpacks a dictionary to its value type before the function runs,
so the Date64 dictionary query in dayofweek.slt never reached this arm.
Replace it with a unit test that calls invoke_with_args directly.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for the implementation. The shared epoch-day approach is appropriate and removes the chrono range limitation.

Requesting changes because Date64 values outside the Date32 range can produce NULL despite a non-nullable return type. Please address this before merging.

// The date signature also admits Date64, which holds milliseconds since the
// epoch. Arrow's cast turns it into epoch days.
if array.data_type() == &DataType::Date64 {
return epoch_day_array(name, &cast(array, &DataType::Date32)?, kernel);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Date64 values beyond the Date32 range cast to NULL, violating the non-nullable output contract for both UDFs. Please compute epoch days in i64 and add non-nullable SQL projection and dictionary regression tests.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ce9eb14. Date64 milliseconds are now floored to days in i64, so values outside the Date32 range get a day of the week instead of NULL. This also fixes times before the epoch that are not a whole day, which the cast moved to the following day. Both .slt files now cover these values as a plain column and as a Dictionary(Int32, Date64) column, and a unit test covers i64::MIN and i64::MAX.

# The whole Int32 epoch-day range: -5877641-06-23 and +5881580-07-11, which
# are outside chrono's calendar. Verified against Spark 4.2.0.
query II
SELECT weekday(date_from_unix_date(-2147483648)), weekday(date_from_unix_date(2147483647));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we test both UDFs using a two-row VALUES column containing i32::MIN and i32::MAX? This would exercise array evaluation without constant folding.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in ce9eb14 for both functions. The two dates also pass on Spark 4.2.0 as a column.

/// `unary` evaluates every slot and vectorizes, `unary_opt` visits only valid
/// slots but costs more per element, so switch on null density.
#[inline]
fn map_dates<F: Fn(i32) -> i32 + Copy>(dates: &Date32Array, kernel: F) -> Int32Array {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use unary unconditionally, or benchmark the 50% NULL cutoff against all-valid, mixed, and mostly-NULL batches? The extra branch lacks performance evidence.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched to unary for every batch in ce9eb14. The kernels are total, so evaluating the slots under NULLs is safe.

Arrow's Date64 to Date32 cast returns NULL for days outside the i32 range,
so a non-null Date64 could give a NULL result from a non-nullable field. The
cast also divides toward zero, which put a time before the epoch on the
following day. Floor the milliseconds to days in i64 instead, and make the
kernels take i64 days.

Use `unary` for every batch. The kernels are total, so evaluating the slots
under NULLs is safe, and the null-density switch had no benchmark behind it.

Add tests for Date64 values outside the Date32 range, plain and dictionary
encoded, and for the Date32 range limits as a column.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @andygrove!

@andygrove

Copy link
Copy Markdown
Member Author

@kosiew I think ce9eb14 covers all three of your points (i64 epoch days for Date64, the i32::MIN/MAX column tests, and unary for every batch). Could you take another look when you have a moment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port dayofweek from Comet

3 participants