Skip to content

fix: match Spark's error for NaN and infinity cast to timestamp under ANSI - #7634

Merged
andygrove merged 1 commit into
apache:mainfrom
FreakingMind:fix-ansi-nan-cast-to-timestamp
Nov 16, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
FreakingMind:fix-ansi-nan-cast-to-timestamp

Conversation

@FreakingMind

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7001.

Rationale for this change

With ANSI on, cast(d AS timestamp) for a NaN or infinite double or float fails in both engines, but with different errors. Spark throws SparkDateTimeException saying The value Infinity of the type "DOUBLE" cannot be cast to "TIMESTAMP". Comet threw SparkNumberFormatException saying The value 'inf' of the type "STRING". The error class and SQLSTATE already matched, only the exception class and the text were off.

Spark does this the same way in 3.5 through 4.2. doubleToTimestampAnsi calls the Double overload of invalidInputInCastToDatetimeError. A float is widened first, so the message says DOUBLE for float input too.

What changes are included in this PR?

  • The native cast now returns InvalidInputInCastToDatetime for NaN and infinities, with the value spelled the way Spark spells it (NaN, Infinity, -Infinity).
  • Both error converter shims have a new case for a DOUBLE source that calls Spark's Double overload. The string case is unchanged.
  • I renamed float_timestamp_overflow_text to double_sql_text, since the overflow error and the new error both use it now.

How are these changes tested?

  • A new SQL test, cast_float_to_timestamp_ansi.sql, compares the exception class and the message with Spark for double and float input. It fails on main with SparkNumberFormatException did not equal SparkDateTimeException.
  • New cases in SparkErrorConverterSuite for a DOUBLE and a STRING source.
  • An updated Rust unit test.

I ran the SQL test and the converter suite on Spark 4.1 and 3.5.

The existing CometNativeCastSuite float and double tests didn't catch this, because their generated data hits CAST_OVERFLOW on MaxValue before it reaches NaN.

… ANSI

With ANSI on, Spark rejects a NaN or infinite double or float cast to
TIMESTAMP with SparkDateTimeException, and its message shows the value
as a double: The value Infinity of the type "DOUBLE". Comet threw
SparkNumberFormatException and printed 'inf' as a STRING.

Closes apache#7001

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @FreakingMind. I checked this against DateTimeUtils.doubleToTimestampAnsi and toSQLValue(Double) in Spark 3.5 and 4.1, and the exception class, message and float widening all line up. Nice catch on why the existing cast suites never got to the NaN rows.

@andygrove
andygrove enabled auto-merge November 16, 2026 14:38
@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 16, 2026
@andygrove
andygrove added this pull request to the merge queue Nov 16, 2026
Merged via the queue into apache:main with commit c7da50c Nov 16, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation backport-1.2 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ANSI NaN and infinity use the wrong exception for cast to timestamp

2 participants