Repository navigation
fix: [branch-1.2] resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast (#7215) and accept second-precision zone offsets (#7579) - #7651
Conversation
…imestamp cast (apache#7215) * fix: resolve SHORT_IDS and region zone suffixes in native string-to-timestamp cast Closes apache#7209 * fix: address review on zone suffixes in string-to-timestamp cast - Match the UTC/GMT/UT prefix forms before the region lookup, anchored at the trailing offset, and leave a name after '/' (Etc/UTC, Etc/GMT+8) to the region lookup. This removes the slowdown on ' UTC' and 'GMT+8' suffixes. - Accept a one-digit hour in the +H:MM:SS offset form, as Spark pads it. - Link the non-zero seconds offset gap to apache#7224 in the code and docs. - Add summer PST/EST cases, a test that every SHORT_IDS target resolves, and a table-backed SQL file test with TIMESTAMP_NTZ and ANSI queries. - List the supported zone suffixes in the cast compatibility guide. * fix: skip zone lookups that cannot match, accept +HHMMSS, and check every SHORT_IDS alias against Spark --------- Co-authored-by: NingLin Liu <224592858+ninglinLiu@users.noreply.github.com> (cherry picked from commit 567dea8)
|
#7579 merged to Could it go in as a second commit here ( |
…pache#7579) * fix: accept second-precision zone offsets in native timestamp casts * test: cover second-precision zone offsets end to end * fix: match Spark's second-precision offset layouts and guard the residual * style: apply rustfmt to the merged parser tests * refactor: share the timestamp cast invalid-input error * docs: correct second-precision timestamp offsets * ci: rerun current timestamp cast changes * ci: request fork cast validation The previous push produced no Actions run. Retry with the authenticated GitHub CLI identity. (cherry picked from commit c7fe182) Adapted for branch-1.2: the five frozen `spark-*/cast.md` compatibility pages get the same edit as `_category_template/cast.md`, because branch-1.2 commits the generated pages.
|
Added #7579 as a second commit ( |
Which issue does this PR close?
Backport of #7215 and #7579 to
branch-1.2. Closes #7209 and #7224 on this branch.Rationale for this change
The native string-to-timestamp cast returns NULL (or
CAST_INVALID_INPUTunder ANSI) for several zone suffixes that Spark accepts throughZoneId.of(id, ZoneId.SHORT_IDS):SHORT_IDSaliases such asPST, region ids without a/such asCETandZulu, a region id with no space before it, and offsets with a seconds field. #7215 fixes the first group. #7579 fixes offsets with a non-zero seconds part such as+08:00:30, which Arrow's minute-precisionTzcannot carry as a single zone.branch-1.2has the same parser, and #7579 edits the "String to Timestamp" section that #7215 adds, so the two go together. See #7215, #7209, #7579 and #7224 for the full analysis.What changes are included in this PR?
Two commits, each made with
git cherry-pick -x:native/spark-expr/src/conversion_funcs/string.rs, its unit tests and thecast_string_to_timestamp_zone_ids.sqlfile test.checked_subguard so a residual that pushes the instant below the representable range returns NULL (or the ANSI error) instead of wrapping, the sharedinvalid_timestamphelper, and theCometNativeCastSuitecases.The Rust and Scala changes are identical to the source commits apart from line numbers and blob hashes.
One adaptation, in both commits: the source commits update the "String to Timestamp" section of
docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md. On a release branch the per-version pages (spark-3.4,spark-3.5,spark-4.0,spark-4.1,spark-4.2) hold a frozen copy of that section, so the same replacements are applied to eachcast.md. The section was identical in all five before each change.Other release branches:
branch-1.2is the newest release branch, so there is no newer one to port to.How are these changes tested?
On this branch, with JDK 17 and the default Spark profile only:
conversion_funcs::stringtests indatafusion-comet-spark-exprpass, including the residual-offset and underflow tests from fix: accept second-precision zone offsets in native timestamp casts #7579.cargo clippy --all-targets -- -D warningsandcargo fmt --checkare clean for that crate.cast.mdfiles, and the five frozen sections now match_category_template/cast.md.CometSqlFileTestSuiteoncast_string_to_timestamp_zone_ids.sqlpassed in both session time zone configurations against a native library built from this branch, and failed with Spark/Comet answer mismatches withstring.rsrestored tobranch-1.2's version.CometNativeCastSuite, locally after adding fix: accept second-precision zone offsets in native timestamp casts #7579.