Repository navigation
fix: accept second-precision zone offsets in native timestamp casts - #7579
Conversation
comphead
left a comment
There was a problem hiding this comment.
Thanks for the fix. I checked the accepted and rejected layouts against getZoneId on Spark 3.4.3, 3.5.8, 4.0.1 and 4.1.1, and the residual sign for negative offsets (-01:30:15 gives the -01:30 zone and a -15 second residual). They match. The parse_sign_offset change in the first bullet of the description is already on main from #7215, so the description, which becomes the squash message, could say that this PR adds the residual and its tests.
| return match adjusted { | ||
| Some(micros) => Ok(Some(micros)), | ||
| None if eval_mode == EvalMode::Ansi => { | ||
| Err(SparkError::InvalidInputInCastToDatetime { | ||
| value: value.to_string(), | ||
| from_type: "STRING".to_string(), | ||
| to_type: "TIMESTAMP".to_string(), | ||
| }) | ||
| } | ||
| None => Ok(None), | ||
| }; |
There was a problem hiding this comment.
This is the third copy of the "ANSI error, otherwise NULL" block in the TIMESTAMP path, next to the empty-input check at the top of timestamp_parser and the end of parse_timestamp_pattern. invalid_ntz_timestamp already does this for TIMESTAMP_NTZ. Could a TIMESTAMP sibling, or invalid_ntz_timestamp taking the target type, replace all three? This arm would then be None => invalid_timestamp(value, eval_mode).
| "2020-06-15T12:30:00-08:00", | ||
| "2020-06-15T12:30:00 UTC", | ||
| "2020-06-15T12:30:00+08:00:30", | ||
| "2020-06-15T12:30:00-01:30:15", |
There was a problem hiding this comment.
Could this list also have the rejected 2020-06-15T12:30:00+8:0:3? Spark validates the zone for TIMESTAMP_NTZ too (parseTimestampString calls getZoneId before the zone is dropped), so the NTZ path returns NULL for it, and that is worth pinning here as well as in the TIMESTAMP list.
|
@comphead both points are addressed in
Verification: |
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
Thanks for closing the gap in #7224. I traced the residual arithmetic by hand for both signs. For -01:30:15, tz_from_offset_secs truncates to -01:30 and tz_and_residual_secs returns a residual of -15. Subtracting it moves the instant 15 seconds later, which gives local time plus 5415 seconds as Spark does. The residual always has the sign of the offset, so it pushes in the same direction as the minute zone and cannot cancel an overflow in the zone step. That means the checked_sub underflow matches Spark's out-of-range result. I also checked that TIMESTAMP_NTZ still discards the residual and that extract_offset_suffix has no other callers that need the new tuple element.
One thing to fix in the docs. The hand-written prose in docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md (around line 143) still says offsets with a non-zero seconds part such as +08:00:30 are not supported, that Comet returns NULL or raises CAST_INVALID_INPUT, and it links #7224. This PR makes that statement false. Could that paragraph be removed, or reworded to say these offsets are now accepted? The list of accepted offset forms just above it (+07:30, -0800, +080000, +08:00:00) could also gain +08:00:30 and +080030.
I found no other concerns. Behavior against the latest release: the new accepted layouts are a bug fix that makes Comet match Spark, so I treated them as intended.
|
Updated the compatibility source template: removed the stale unsupported-offset paragraph and added |
andygrove
left a comment
There was a problem hiding this comment.
Thanks @ErikBPF, the doc update reads right now, and invalid_timestamp tidies up the three copies nicely. The last fork CI run I can find is on 703fdea99, before the merge with #7215 and the refactor. Could you push (or rebase on main) with Actions enabled on your fork so the cast suites run on the current head? Since the merge touched the parser this sits in, I'd like to see it green before merging.
The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity.
|
@andygrove I pushed again and merged current main ( Fork Actions reports The requested current-head green cast-suite verdict is therefore still outstanding. I have stopped pushing further trigger commits rather than claiming the earlier run validates this head. |
…pache#7579) * fix: accept second-precision zone offsets in native timestamp casts * test: cover second-precision zone offsets end to end * fix: match Spark's second-precision offset layouts and guard the residual * style: apply rustfmt to the merged parser tests * refactor: share the timestamp cast invalid-input error * docs: correct second-precision timestamp offsets * ci: rerun current timestamp cast changes * ci: request fork cast validation The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity. (cherry picked from commit c7fe182) Adapted for branch-1.2: the five frozen `spark-*/cast.md` compatibility pages get the same edit as `_category_template/cast.md`, because branch-1.2 commits the generated pages.
Which issue does this PR close?
Closes #7224.
Rationale for this change
The native
CAST(string AS TIMESTAMP)returns NULL for a zone offset with a non-zero seconds part, and raisesCAST_INVALID_INPUTunder ANSI mode, while Spark accepts those offsets throughZoneId.of. Arrow'sTzhas minute precision, so the offset cannot be carried as a single zone.What changes are included in this PR?
extract_offset_suffixreturns the minute-precision zone plus a residual in seconds, and the parser subtracts the residual after applying the zone.checked_sub: an instant pushed below the representable range returns NULL (or the ANSI error) instead of wrapping.TIMESTAMP_NTZcontinues to discard the validated zone; tests also pin the rejected+8:0:3layout.invalid_timestamphelper handles invalid TIMESTAMP values consistently.The accepted offset parser layouts already landed on
mainin #7215. This PR adds the residual adjustment and its tests.How are these changes tested?
+08:00:30,+080030,-01:30:15, and the rejected+8:0:3and+08:0:30.i64::MINmicroseconds, in Legacy, Try and ANSI modes.CometNativeCastSuitecases for bothTIMESTAMPandTIMESTAMP_NTZthat compare against Spark, including one layout Spark rejects.