[SPARK-56965][SQL] Add SQL parser support for TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p)#56041
[SPARK-56965][SQL] Add SQL parser support for TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p)#56041stevomitric wants to merge 3 commits into
Conversation
|
@MaxGekk please take a look at this PR. |
| val TIMESTAMP_NANOS_TYPES_ENABLED = | ||
| buildConf("spark.sql.timestampNanosTypes.enabled") | ||
| .internal() | ||
| .doc("When true, the SQL parser accepts the parameterized nanosecond-precision " + |
There was a problem hiding this comment.
The description is too restrictive (especially taking into account its name). The config shall disable other entry points like creating df from Java types: java.time.LocalDateTime.
There was a problem hiding this comment.
Makes sense. I generalized the doc and had to gate the check on DataTypes as well.
There was a problem hiding this comment.
Was this patch authored or co-authored using generative AI tooling?
Yes
BTW, use the tag "Generated-By", see
"
When providing contributions authored using generative AI tooling, a recommended practice is for contributors to indicate the tooling used to create the contribution. This should be included as a token in the source control commit message, for example including the phrase “Generated-by: ”. This allows for future release tooling to be considered that pulls this content into a machine parsable Tooling-Provenance file.
"
at https://www.apache.org/legal/generative-tooling.html
| parameters = Map("error" -> "'WITH'", "hint" -> "")) | ||
| } | ||
|
|
||
| test("invalid precision of the nanos timestamp data type") { |
There was a problem hiding this comment.
This test covers the two zone-suffixed forms - TIMESTAMP(6) WITHOUT TIME ZONE -> TIMESTAMP_NTZ error and TIMESTAMP(10) WITH LOCAL TIME ZONE -> TIMESTAMP_LTZ error - but does not exercise the bare TIMESTAMP(6) / TIMESTAMP(10) form under each SQLConf.TIMESTAMP_TYPE setting, which is the path that routes through SqlApiConf.get.timestampType in DataTypeAstBuilder.scala.
Alternatively, the assertions could be added to the existing test("Set default timestamp type") block at line 159, since that test already wraps both values of SQLConf.TIMESTAMP_TYPE and was modified by this PR to add the success-case parse("timestamp(9)") assertions — the invalid-precision counterpart would sit naturally next to them.
There was a problem hiding this comment.
Added both 6 and 10 precision checks.
2 participants
What changes were proposed in this pull request?
Adds SQL parser support for parameterized nanosecond-precision timestamp types introduced in SPARK-56876. The parser now accepts:
with
pfrom[7, 9]. Out-of-range precision throwsINVALID_TIMESTAMP_PRECISION; negative precision is rejected by the grammar asPARSE_SYNTAX_ERROR.The new syntax is gated behind a new internal preview flag
spark.sql.timestampNanosTypes.enabled(default false).Unparameterized TIMESTAMP, TIMESTAMP_NTZ, TIMESTAMP_LTZ, and the WITH/WITHOUT TIME ZONE variants continue to return the existing microsecond types - no behavior change.
Part of SPIP SPARK-56822 (https://issues.apache.org/jira/browse/SPARK-56822).
Why are the changes needed?
SPARK-56876 added TimestampNTZNanosType / TimestampLTZNanosType to the type system but explicitly left out SQL/DDL integration - users cannot declare these types in CREATE TABLE, CAST, or Column.cast(String) today. This PR is the parser sub-task of the SPIP and wires those spellings through DataTypeAstBuilder, behind a preview flag so the surface is opt-in until the cast/runtime sub-tasks land.
Does this PR introduce any user-facing change?
Yes made the parser changes to allow timestamps with precision.
How was this patch tested?
Extended DataTypeParserSuite
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7