You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native timestamp casts fail or panic for years beyond chrono's range #7471
Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.
Query
Comet
CAST(ts AS DATE)
CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
CAST(ts AS STRING)
same error
try_cast(ts AS DATE)
same error
CAST(dt AS TIMESTAMP)
native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34
Steps to reproduce
As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:
statement
CREATETABLEtsr(ts timestamp, dt date) USING parquet
statement
INSERT INTO tsr VALUES
(timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
(timestamp_micros(0), date_from_unix_date(0))
query
SELECT CAST(ts ASDATE) FROM tsr
query
SELECT CAST(ts AS STRING) FROM tsr
query
SELECT try_cast(ts ASDATE) FROM tsr
query
SELECT CAST(dt ASTIMESTAMP) FROM tsr
timestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.
Expected behavior
The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.
Additional context
The error paths:
pre_timestamp_cast in native/spark-expr/src/utils.rs (as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.
timestamp_ntz_to_timestamp and cast_timestamp_to_ntz in the same file, same pattern.
Date to timestamp in native/spark-expr/src/conversion_funcs/temporal.rs: epoch + chrono::Duration::days(d) panics. The comment above it says chrono's range covers every i32 day count, but it does not.
test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs asserts the error for i64::MAX.
Found while auditing which expressions can safely run as TopK later keys in #7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.
A cast from DATE to STRING has the same problem, raised in review on #7457 . Comet sends it to arrow-cast's value_to_string, whose ValueFormatter::write ignores the safe option, so a date beyond chrono's year range raises in every mode:
CREATETABLEbig_date(dt date) USING parquet;
INSERT INTO big_date SELECT date_from_unix_date(100000000);
SELECT dt, CAST(dt AS STRING) FROM big_date;
Spark returns the date and its string. Comet fails with Cast error: Failed to convert 100000000 to temporal for Date32 (checked on Spark 4.1 with a CometSqlFileTestSuite fixture). Implicit casts such as concat(s, dt) go the same way. #7457 now treats this cast as able to fail for TopK sort keys.
Activity
tlfua commentedon Nov 6, 2026
A cast from
DATEtoSTRINGhas the same problem, raised in review on #7457 . Comet sends it to arrow-cast'svalue_to_string, whoseValueFormatter::writeignores thesafeoption, so a date beyond chrono's year range raises in every mode:Spark returns the date and its string. Comet fails with
Cast error: Failed to convert 100000000 to temporal for Date32(checked on Spark 4.1 with aCometSqlFileTestSuitefixture). Implicit casts such asconcat(s, dt)go the same way. #7457 now treats this cast as able to fail for TopK sort keys.kazuyukitanimura commentedon Nov 9, 2026
we are moving onto jiff FYI
fix: keep a native lambda body with a null map key or a date/time cas…
Narrow the ANSI-off TopK check per review: drop timestamp casts, add …
fix: keep date functions out of native lambdas and let AND and OR sta…
feat: run filter and exists lambdas natively with DataFusion's higher…