Repository navigation
Conversation
Port dayofweek from DataFusion Comet. The value is computed straight from the Date32 epoch day, so it is defined for the whole Int32 range as in Spark. weekday now shares the same kernel, which fixes it returning NULL for dates outside chrono's calendar range.
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @andygrove. Computing the day from the epoch day, instead of through a chrono date, is a nice way to cover the whole Date32 range.
| let mapped = Arc::new(map_dates(values, kernel)) as ArrayRef; | ||
| return Ok(cast(&dict.with_values(mapped), &DataType::Int32)?); | ||
| } | ||
| let Some(dates) = array.as_primitive_opt::<Date32Type>() else { |
There was a problem hiding this comment.
date_signature() matches the logical date type, so it also admits Date64, but this only handles Date32. SELECT dayofweek(arrow_cast('2024-01-07', 'Date64')) plans and then fails with Execution error: `dayofweek` expects a date, got Date64.
weekday has the same problem after this PR. On main it goes through arrow's date_part, which handles Date64, and SELECT weekday(arrow_cast('2024-01-07', 'Date64')) returns 6. With this PR it fails with the same error.
Could we cast a Date64 input to Date32 at the top of epoch_day_array? Then both functions handle every type their signatures accept.
There was a problem hiding this comment.
Good catch, thanks. Fixed in be65c39: epoch_day_array now casts a Date64 input to Date32 first, including Date64 dictionary values, so dayofweek and weekday handle every type their signatures accept. Both .slt files cover a scalar, a column with NULL and a date before the epoch, and dayofweek.slt also covers a dictionary. Both files fail without the fix.
There was a problem hiding this comment.
Both
.sltfiles cover a scalar, a column with NULL and a date before the epoch, anddayofweek.sltalso covers a dictionary.
The dictionary query in dayofweek.slt doesn't reach the new DataType::Date64 arm in the dictionary branch (dayofweek.rs L97-L100). I replaced that arm with DataType::Date64 => Arc::clone(dict.values()), and dayofweek.slt still passed. Coercion casts the dictionary to its value type before the function runs. With a column input, EXPLAIN shows dayofweek(CAST(CAST(CAST(column1 AS Date64) AS Dictionary(Int32, Date64)) AS Date64)), and the all-literal query in the file is folded to Int32(1) at planning.
Comet calls invoke_with_args directly, so only a unit test reaches this arm, the way dictionary_input_unpacks does for Date32 (L286-L299). Could we add one next to it, and drop the dictionary query from dayofweek.slt? For example, with Date64Array added to the use arrow::array::DictionaryArray; import:
#[test]
fn date64_dictionary_input() {
let values =
Arc::new(Date64Array::from(vec![Some(0), Some(-86_400_000), None])) as ArrayRef;
let keys = Int32Array::from(vec![Some(0), Some(1), None, Some(2)]);
let dict = DictionaryArray::<Int32Type>::new(keys, values);
let out = invoke_array(Arc::new(dict));
assert_eq!(out.iter().collect::<Vec<_>>(), [Some(5), Some(4), None, None]);
}There was a problem hiding this comment.
Added date64_dictionary_input next to the Date32 dictionary test and removed the Date64 dictionary query from the .slt in 3e80f6c. With the arm replaced by Arc::clone(dict.values()) the new test fails with "expects a date, got Dictionary(Int32, Date64)".
| NULL | ||
| 1 | ||
|
|
||
| # Spark returns NULL for a malformed string when ANSI mode is off. DataFusion's |
There was a problem hiding this comment.
Spark also accepts partial dates as strings, in both ANSI modes. On Spark 4.2.0, SELECT dayofweek('2009') returns 5 and SELECT dayofweek('2009-07') returns 4. DataFusion's cast rejects both with ANSI on or off (Cannot cast string '2009' to value of Date32 type).
This comes from date_signature() on main (date_part_name.rs), so dayname and monthname behave the same way. The "String to date" item in #88 covers the fix. Could we add these two as commented-out queries with Spark's results next to this one? Could the #[user_doc] description also say that strings Spark reads as partial dates raise an error, not only malformed ones?
date_signature admits Date64, so cast it to epoch days before computing the day of the week. Note that strings Spark reads as partial dates are an error here, with commented-out queries citing apache#88.
… test Coercion unpacks a dictionary to its value type before the function runs, so the Date64 dictionary query in dayofweek.slt never reached this arm. Replace it with a unit test that calls invoke_with_args directly.
kosiew
left a comment
There was a problem hiding this comment.
@andygrove, thanks for the implementation. The shared epoch-day approach is appropriate and removes the chrono range limitation.
Requesting changes because Date64 values outside the Date32 range can produce NULL despite a non-nullable return type. Please address this before merging.
| // The date signature also admits Date64, which holds milliseconds since the | ||
| // epoch. Arrow's cast turns it into epoch days. | ||
| if array.data_type() == &DataType::Date64 { | ||
| return epoch_day_array(name, &cast(array, &DataType::Date32)?, kernel); |
There was a problem hiding this comment.
Date64 values beyond the Date32 range cast to NULL, violating the non-nullable output contract for both UDFs. Please compute epoch days in i64 and add non-nullable SQL projection and dictionary regression tests.
There was a problem hiding this comment.
Fixed in ce9eb14. Date64 milliseconds are now floored to days in i64, so values outside the Date32 range get a day of the week instead of NULL. This also fixes times before the epoch that are not a whole day, which the cast moved to the following day. Both .slt files now cover these values as a plain column and as a Dictionary(Int32, Date64) column, and a unit test covers i64::MIN and i64::MAX.
| # The whole Int32 epoch-day range: -5877641-06-23 and +5881580-07-11, which | ||
| # are outside chrono's calendar. Verified against Spark 4.2.0. | ||
| query II | ||
| SELECT weekday(date_from_unix_date(-2147483648)), weekday(date_from_unix_date(2147483647)); |
There was a problem hiding this comment.
Could we test both UDFs using a two-row VALUES column containing i32::MIN and i32::MAX? This would exercise array evaluation without constant folding.
There was a problem hiding this comment.
Added in ce9eb14 for both functions. The two dates also pass on Spark 4.2.0 as a column.
| /// `unary` evaluates every slot and vectorizes, `unary_opt` visits only valid | ||
| /// slots but costs more per element, so switch on null density. | ||
| #[inline] | ||
| fn map_dates<F: Fn(i32) -> i32 + Copy>(dates: &Date32Array, kernel: F) -> Int32Array { |
There was a problem hiding this comment.
Could we use unary unconditionally, or benchmark the 50% NULL cutoff against all-valid, mixed, and mostly-NULL batches? The extra branch lacks performance evidence.
There was a problem hiding this comment.
Switched to unary for every batch in ce9eb14. The kernels are total, so evaluating the slots under NULLs is safe.
Arrow's Date64 to Date32 cast returns NULL for days outside the i32 range, so a non-null Date64 could give a NULL result from a non-nullable field. The cast also divides toward zero, which put a time before the epoch on the following day. Floor the milliseconds to days in i64 instead, and make the kernels take i64 days. Use `unary` for every batch. The kernels are total, so evaluating the slots under NULLs is safe, and the null-density switch had no benchmark behind it. Add tests for Date64 values outside the Date32 range, plain and dictionary encoded, and for the Date32 range limits as a column.
mbutrovich
left a comment
There was a problem hiding this comment.
LGTM, thanks @andygrove!
Closes #151.
Summary
Adds a Spark-compatible
dayofweek(date), which returns an INT from 1 (Sunday) to 7 (Saturday). It is ported from Comet'sdatetime_funcs/day_of_week.rs.crates/spark/src/function/datetime/dayofweek.rs. The result comes straight from theDate32epoch day:((days + 4) mod 7) + 1, computed ini64. This matches Spark'sLocalDate.ofEpochDay(days).getDayOfWeek.plus(1).getValue, which is defined for everyInt.invoke_with_argshandles scalar and array inputs, NULL and untyped NULL, and dictionary-encodedDate32. For a dictionary, it maps the values, rewraps the keys, and unpacks the result to plainInt32.date_signature()thatdaynameandmonthnameuse. Strings and timestamps are implicitly cast to DATE, as in Spark.#[user_doc]and registered indatetime/mod.rs.weekdaynow uses the same epoch-day helper instead of arrow'sdate_part(DayOfWeekMonday0). arrow builds a chrono date for each row, so dates outside chrono's range (such asdate_from_unix_date(±2^31)) came back NULL. Spark returns 1 and 4 for those dates, and so doesweekdaynow. Theweekdaysignature is unchanged. Both functions cast aDate64input (also inside a dictionary) to epoch days, so they accept every type their signatures admit.Differences from Comet's version
Signature::any(1)and expects the planner to hand it aDate32. Here the function has a coercible date signature, so strings and timestamps work from SQL. Comet's namespark_dayofweekbecomesdayofweek.return_field_from_argspasses the input's nullability through to the result, the same asweekday.unaryandunary_opt, and the dictionary handling are the same as in Comet. I found no behavioral difference between Comet and Spark 4.2.0 forDate32input.Known differences from Spark, which also apply to the existing
daynameandweekdayand are left as commented-out cases in the.slt:'2009'and'2009-07'as the first day of that year or month, in both ANSI modes. DataFusion's cast rejects them ([EPIC] Port Comet's Spark-compatible CAST #88).Testing
date_from_unix_date(-2147483648)= 3 and(2147483647)= 6, NULL and untyped NULL (int), strings, TIMESTAMP_NTZ, TIMESTAMP in UTC and in America/New_York, malformed strings with ANSI on and off, and an INT argument (DATATYPE_MISMATCH).crates/sqllogictest/scripts/pyspark_slt_runner.pyondayofweek.sltandweekday.slt: everydayofweekrecord passes on Spark 4.2.0. Skipped records are thearrow_typeofchecks and the dictionary case, markedskipif spark. The oneweekdaymismatch,weekday('not-a-date'), was already there (strings are not cast forweekday, [Bug] last_day, monthname, weekday, round and unbase64 reject arguments that Spark implicitly casts #41).i32range, agreement withweekdayover a 400-year cycle, dictionary input, non-date input rejected, and scalar input. A nullability test was added.cargo fmt --all,cargo clippy --all-targets --all-features -- -D warnings,cargo test -p datafusion-sparkandcargo test -p datafusion-spark-sqllogictest(249 of 249 files).