Repository navigation
fix: match Spark's error for NaN and infinity cast to timestamp under ANSI - #7634
Merged
andygrove merged 1 commit intoNov 16, 2026
Merged
Conversation
… ANSI With ANSI on, Spark rejects a NaN or infinite double or float cast to TIMESTAMP with SparkDateTimeException, and its message shows the value as a double: The value Infinity of the type "DOUBLE". Comet threw SparkNumberFormatException and printed 'inf' as a STRING. Closes apache#7001
andygrove
approved these changes
Nov 16, 2026
andygrove
left a comment
Member
There was a problem hiding this comment.
Thanks @FreakingMind. I checked this against DateTimeUtils.doubleToTimestampAnsi and toSQLValue(Double) in Spark 3.5 and 4.1, and the exception class, message and float widening all line up. Nice catch on why the existing cast suites never got to the NaN rows.
andygrove
enabled auto-merge
November 16, 2026 14:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7001.
Rationale for this change
With ANSI on,
cast(d AS timestamp)for a NaN or infinite double or float fails in both engines, but with different errors. Spark throwsSparkDateTimeExceptionsayingThe value Infinity of the type "DOUBLE" cannot be cast to "TIMESTAMP". Comet threwSparkNumberFormatExceptionsayingThe value 'inf' of the type "STRING". The error class and SQLSTATE already matched, only the exception class and the text were off.Spark does this the same way in 3.5 through 4.2.
doubleToTimestampAnsicalls theDoubleoverload ofinvalidInputInCastToDatetimeError. A float is widened first, so the message says DOUBLE for float input too.What changes are included in this PR?
InvalidInputInCastToDatetimefor NaN and infinities, with the value spelled the way Spark spells it (NaN,Infinity,-Infinity).Doubleoverload. The string case is unchanged.float_timestamp_overflow_texttodouble_sql_text, since the overflow error and the new error both use it now.How are these changes tested?
cast_float_to_timestamp_ansi.sql, compares the exception class and the message with Spark for double and float input. It fails on main withSparkNumberFormatException did not equal SparkDateTimeException.SparkErrorConverterSuitefor a DOUBLE and a STRING source.I ran the SQL test and the converter suite on Spark 4.1 and 3.5.
The existing
CometNativeCastSuitefloat and double tests didn't catch this, because their generated data hitsCAST_OVERFLOWonMaxValuebefore it reaches NaN.