Repository navigation
TIMESTAMP WITH TIME ZONE values with a numeric UTC offset come back naive #854
Description
Problem
DefaultTypeConverter resolves the zone of a timestamp with time zone value with dateutil.tz.gettz. Athena renders a numeric offset as +05:30 / -08:00, for which gettz returns None, so the cursor returns a naive wall-clock datetime and the instant is lost. Zone names (America/New_York, UTC) work.
Expected: an aware datetime with a fixed +05:30 offset.
Reproduction
from pyathena.converter import DefaultTypeConverter
DefaultTypeConverter().convert('timestamp with time zone', '2024-02-29 23:59:58.123 +05:30')
# datetime.datetime(2024, 2, 29, 23, 59, 58, 123000) -> tzinfo is NoneLive: SELECT TIMESTAMP '2024-02-29 23:59:58.123456 +05:30' through the REST cursor returns a naive value.
Environment
PyAthena 3.35.4 and master (c01c56f), Python 3.11, SQLAlchemy 2.0.52, Athena engine version 3.
Proposed fix (optional)
Parse a trailing [+-]HH:MM token into datetime.timezone(timedelta(...)) and fall back to the zone-name lookup otherwise. Happy to send a PR with unit tests if the approach is agreed; I can validate against my own Athena workgroup.
Activity
laughingman7743 commentedon Sep 26, 2026
Confirmed on master (c01c56f):
_to_datetime_with_tzhands the trailing token todateutil.tz.gettz, which returnsNonefor a numeric offset such as+05:30, so the value comes back naive. Zone names work. Nothing in the docs or the tracker covers this, so it is a bug we want fixed.The proposed approach is fine: parse a trailing
[+-]HH:MMtoken intodatetime.timezone(timedelta(...))and keep the zone-name lookup otherwise. Please also cover the negative offset and the existing zone-name path in the unit tests.Per CONTRIBUTING.md, please run the regression on the version before the fix and after it in your own AWS environment and record both results in the PR's TEST section, including the exact commands and counts. External-fork PRs are not run in the project's AWS CI.
laughingman7743 commentedon Sep 26, 2026
Target
masterfor the fix; the maintainers will port it to the3.xmaintenance branch afterwards, so please do not open a PR against3.x.laughingman7743 commentedon Sep 26, 2026
If you plan to send a PR for this, please say so here. If the issue stays open for a long time without a PR, the maintainers may pick it up and implement the fix themselves.
Keep the UTC offset of timestamp with time zone values
Backport #1033 (#854 only): Keep numeric UTC offsets of TIMESTAMP WIT…
laughingman7743 commentedon Oct 6, 2026
The fix is released in v3.38.0, backported to the 3.x line in #1104.
A
TIMESTAMP WITH TIME ZONEvalue with a numeric UTC offset now converts to an aware datetime with a fixed offset:On 3.x, this applies to the cursors that use the default converter (
Cursor,DictCursor, their async and asyncio variants, andS3FSCursor). The pandas, Arrow, and Polars cursors also convert these values in the upcoming 4.0.0 (#1033). Thank you again for the report.