Skip to content

fix: accept second-precision zone offsets in native timestamp casts - #7579

Merged
andygrove merged 10 commits into
apache:mainfrom
ErikBPF:fix/7224-cast-offset-seconds
Nov 17, 2026
Merged

andygrove merged 10 commits into
apache:mainfrom
ErikBPF:fix/7224-cast-offset-seconds

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7224.

Rationale for this change

The native CAST(string AS TIMESTAMP) returns NULL for a zone offset with a non-zero seconds part, and raises CAST_INVALID_INPUT under ANSI mode, while Spark accepts those offsets through ZoneId.of. Arrow's Tz has minute precision, so the offset cannot be carried as a single zone.

What changes are included in this PR?

  • extract_offset_suffix returns the minute-precision zone plus a residual in seconds, and the parser subtracts the residual after applying the zone.
  • The residual adjustment uses checked_sub: an instant pushed below the representable range returns NULL (or the ANSI error) instead of wrapping.
  • TIMESTAMP_NTZ continues to discard the validated zone; tests also pin the rejected +8:0:3 layout.
  • A shared invalid_timestamp helper handles invalid TIMESTAMP values consistently.
  • Update the compatibility template to document accepted second-precision offsets.

The accepted offset parser layouts already landed on main in #7215. This PR adds the residual adjustment and its tests.

How are these changes tested?

  • Unit tests for the accepted and rejected layouts, including +08:00:30, +080030, -01:30:15, and the rejected +8:0:3 and +08:0:30.
  • A boundary test for the residual subtraction at i64::MIN microseconds, in Legacy, Try and ANSI modes.
  • CometNativeCastSuite cases for both TIMESTAMP and TIMESTAMP_NTZ that compare against Spark, including one layout Spark rejects.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I checked the accepted and rejected layouts against getZoneId on Spark 3.4.3, 3.5.8, 4.0.1 and 4.1.1, and the residual sign for negative offsets (-01:30:15 gives the -01:30 zone and a -15 second residual). They match. The parse_sign_offset change in the first bullet of the description is already on main from #7215, so the description, which becomes the squash message, could say that this PR adds the residual and its tests.

Comment on lines +1557 to +1567
return match adjusted {
Some(micros) => Ok(Some(micros)),
None if eval_mode == EvalMode::Ansi => {
Err(SparkError::InvalidInputInCastToDatetime {
value: value.to_string(),
from_type: "STRING".to_string(),
to_type: "TIMESTAMP".to_string(),
})
}
None => Ok(None),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the third copy of the "ANSI error, otherwise NULL" block in the TIMESTAMP path, next to the empty-input check at the top of timestamp_parser and the end of parse_timestamp_pattern. invalid_ntz_timestamp already does this for TIMESTAMP_NTZ. Could a TIMESTAMP sibling, or invalid_ntz_timestamp taking the target type, replace all three? This arm would then be None => invalid_timestamp(value, eval_mode).

"2020-06-15T12:30:00-08:00",
"2020-06-15T12:30:00 UTC",
"2020-06-15T12:30:00+08:00:30",
"2020-06-15T12:30:00-01:30:15",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this list also have the rejected 2020-06-15T12:30:00+8:0:3? Spark validates the zone for TIMESTAMP_NTZ too (parseTimestampString calls getZoneId before the zone is dropped), so the NTZ path returns NULL for it, and that is worth pinning here as well as in the TIMESTAMP list.

@ErikBPF

ErikBPF commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@comphead both points are addressed in b150863:

  • invalid_timestamp(value, eval_mode) at string.rs:2128 now backs the empty-input arm (:1521), the residual-offset underflow (:1551) and parse_timestamp_pattern (:2075). The TIMESTAMP path has one copy and mirrors invalid_ntz_timestamp.
  • Added the rejected 2020-06-15T12:30:00+8:0:3 layout to the TIMESTAMP_NTZ values list. Spark's parseTimestampString calls getZoneId before the zone is dropped, so NTZ also returns NULL for it; the parser already did, and the test now pins that.

Verification: cargo fmt --check and cargo test -p datafusion-comet-spark-expr --lib timestamp_parser on Apollo (8 passed). The Scala suite was not run locally for the one-line list addition; CI covers it.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

Thanks for closing the gap in #7224. I traced the residual arithmetic by hand for both signs. For -01:30:15, tz_from_offset_secs truncates to -01:30 and tz_and_residual_secs returns a residual of -15. Subtracting it moves the instant 15 seconds later, which gives local time plus 5415 seconds as Spark does. The residual always has the sign of the offset, so it pushes in the same direction as the minute zone and cannot cancel an overflow in the zone step. That means the checked_sub underflow matches Spark's out-of-range result. I also checked that TIMESTAMP_NTZ still discards the residual and that extract_offset_suffix has no other callers that need the new tuple element.

One thing to fix in the docs. The hand-written prose in docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md (around line 143) still says offsets with a non-zero seconds part such as +08:00:30 are not supported, that Comet returns NULL or raises CAST_INVALID_INPUT, and it links #7224. This PR makes that statement false. Could that paragraph be removed, or reworded to say these offsets are now accepted? The list of accepted offset forms just above it (+07:30, -0800, +080000, +08:00:00) could also gain +08:00:30 and +080030.

I found no other concerns. Behavior against the latest release: the new accepted layouts are a bug fix that makes Comet match Spark, so I treated them as intended.

@ErikBPF

ErikBPF commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Updated the compatibility source template: removed the stale unsupported-offset paragraph and added +08:00:30 and +080030 as examples. The PR description now credits #7215 for the parser layouts and describes this PR's residual adjustment and tests. Markdown formatting and whitespace checks passed; runtime code is unchanged by this follow-up.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ErikBPF, the doc update reads right now, and invalid_timestamp tidies up the three copies nicely. The last fork CI run I can find is on 703fdea99, before the merge with #7215 and the refactor. Could you push (or rebase on main) with Actions enabled on your fork so the cast suites run on the current head? Since the merge touched the parser this sits in, I'd like to see it green before merging.

The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity.
@ErikBPF

ErikBPF commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@andygrove I pushed again and merged current main (6bd87954c) without rewriting history. Current head: 191b92a91f5729574328940bd3c5094bace86d96. The merge was clean; the PR remains three files (+107/-41). CI configuration, Markdown formatting and whitespace checks pass, but these are not cast-suite execution results.

Fork Actions reports enabled: true, all actions allowed, and ci_fork.yml reports active. Despite three pushes, including a retry using the authenticated GitHub CLI identity and the current-main merge, GitHub still lists only the old successful run on 703fdea99. There is no Actions check suite/run for the new head yet. The first two trigger commits changed no source.

The requested current-head green cast-suite verdict is therefore still outstanding. I have stopped pushing further trigger commits rather than claiming the earlier run validates this head.

@github-actions github-actions Bot added bug Something isn't working area:expressions Expression evaluation labels Nov 16, 2026
@andygrove
andygrove added this pull request to the merge queue Nov 17, 2026
Merged via the queue into apache:main with commit c7fe182 Nov 17, 2026
24 checks passed
andygrove pushed a commit to andygrove/datafusion-comet that referenced this pull request Nov 17, 2026
…pache#7579)

* fix: accept second-precision zone offsets in native timestamp casts

* test: cover second-precision zone offsets end to end

* fix: match Spark's second-precision offset layouts and guard the residual

* style: apply rustfmt to the merged parser tests

* refactor: share the timestamp cast invalid-input error

* docs: correct second-precision timestamp offsets

* ci: rerun current timestamp cast changes

* ci: request fork cast validation

The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity.

(cherry picked from commit c7fe182)

Adapted for branch-1.2: the five frozen `spark-*/cast.md` compatibility pages get the same
edit as `_category_template/cast.md`, because branch-1.2 commits the generated pages.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native string to timestamp cast returns NULL for zone offsets with a non-zero seconds part

3 participants