Skip to content

fix: [branch-1.2] resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast (#7215) and accept second-precision zone offsets (#7579) - #7651

Open
andygrove wants to merge 3 commits into
apache:branch-1.2from
andygrove:backport-7215-branch-1.2
Open

andygrove wants to merge 3 commits into
apache:branch-1.2from
andygrove:backport-7215-branch-1.2

Conversation

@andygrove

@andygrove andygrove commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #7215 and #7579 to branch-1.2. Closes #7209 and #7224 on this branch.

Rationale for this change

The native string-to-timestamp cast returns NULL (or CAST_INVALID_INPUT under ANSI) for several zone suffixes that Spark accepts through ZoneId.of(id, ZoneId.SHORT_IDS): SHORT_IDS aliases such as PST, region ids without a / such as CET and Zulu, a region id with no space before it, and offsets with a seconds field. #7215 fixes the first group. #7579 fixes offsets with a non-zero seconds part such as +08:00:30, which Arrow's minute-precision Tz cannot carry as a single zone. branch-1.2 has the same parser, and #7579 edits the "String to Timestamp" section that #7215 adds, so the two go together. See #7215, #7209, #7579 and #7224 for the full analysis.

What changes are included in this PR?

Two commits, each made with git cherry-pick -x:

The Rust and Scala changes are identical to the source commits apart from line numbers and blob hashes.

One adaptation, in both commits: the source commits update the "String to Timestamp" section of docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md. On a release branch the per-version pages (spark-3.4, spark-3.5, spark-4.0, spark-4.1, spark-4.2) hold a frozen copy of that section, so the same replacements are applied to each cast.md. The section was identical in all five before each change.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile only:

ninglinLiu and others added 2 commits November 16, 2026 10:57
…imestamp cast (apache#7215)

* fix: resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast

Closes apache#7209

* fix: address review on zone suffixes in string-to-timestamp cast

- Match the UTC/GMT/UT prefix forms before the region lookup, anchored at the
  trailing offset, and leave a name after '/' (Etc/UTC, Etc/GMT+8) to the region
  lookup. This removes the slowdown on ' UTC' and 'GMT+8' suffixes.
- Accept a one-digit hour in the +H:MM:SS offset form, as Spark pads it.
- Link the non-zero seconds offset gap to apache#7224 in the code and docs.
- Add summer PST/EST cases, a test that every SHORT_IDS target resolves, and
  a table-backed SQL file test with TIMESTAMP_NTZ and ANSI queries.
- List the supported zone suffixes in the cast compatibility guide.

* fix: skip zone lookups that cannot match, accept +HHMMSS, and check every SHORT_IDS alias against Spark

---------

Co-authored-by: NingLin Liu <224592858+ninglinLiu@users.noreply.github.com>
(cherry picked from commit 567dea8)
@comphead

Copy link
Copy Markdown
Contributor

#7579 merged to main today and closes #7224, the second-precision offset gap that this backport documents as unsupported. It changes the parser and the "String to Timestamp" section that #7215 adds, so it builds on this PR. Without it, the next 1.2 patch release ships the #7224 bug together with docs that name it, while main has the fix.

Could it go in as a second commit here (git cherry-pick -x c7fe182210), or as a follow-up backport once this one merges? A simulated cherry-pick onto this PR's head (git merge-tree) has no conflicts. I have not built or tested it. Its doc change touches only _category_template/cast.md, so the five frozen spark-*/cast.md copies would need the same edit as in this PR.

…pache#7579)

* fix: accept second-precision zone offsets in native timestamp casts

* test: cover second-precision zone offsets end to end

* fix: match Spark's second-precision offset layouts and guard the residual

* style: apply rustfmt to the merged parser tests

* refactor: share the timestamp cast invalid-input error

* docs: correct second-precision timestamp offsets

* ci: rerun current timestamp cast changes

* ci: request fork cast validation

The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity.

(cherry picked from commit c7fe182)

Adapted for branch-1.2: the five frozen `spark-*/cast.md` compatibility pages get the same
edit as `_category_template/cast.md`, because branch-1.2 commits the generated pages.
@andygrove andygrove changed the title fix: [branch-1.2] resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast (#7215) fix: [branch-1.2] resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast (#7215) and accept second-precision zone offsets (#7579) Nov 17, 2026
@andygrove

Copy link
Copy Markdown
Member Author

Added #7579 as a second commit (git cherry-pick -x c7fe182210), and the five frozen spark-*/cast.md copies get the same edit as the template. I rewrote the title and description to cover both. On this branch the 50 conversion_funcs::string tests pass, clippy with -D warnings, cargo fmt --check and prettier are clean. I did not rerun the SQL fixture or CometNativeCastSuite locally after the pick, so those rely on CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants