Document the temporal type contract and gate the release on it #1221

Activity

1. laughingman7743

   added

   documentation

   on Oct 16, 2026
2. laughingman7743

   added a parent issue on Oct 16, 2026

   • Define and enforce the contract for Athena temporal types across cursors #1212
3. laughingman7743 commented on Oct 17, 2026

   

   laughingman7743

   on Oct 17, 2026

   MemberAuthor

   More actions

   Release-note items from #1233 (#1214), merged as 4ac04db:

   Fixed (silent wrong values)

   • With unload=True, PandasCursor, ArrowCursor, and PolarsCursor read timestamps at microseconds. Values outside 1677-09-21 to 2262-04-11 were wrapped or saturated before: 1600-01-02 03:04:05.123 was read as 2184-07-22 02:38:38.832551616 by PandasCursor and ArrowCursor, and as 2262-04-11 23:47:16.854775807 by PolarsCursor. This includes timestamps inside arrays, rows, and maps.

   Behavior changes (migration notes)

   • The dtype of UNLOAD timestamps changes from nanoseconds to microseconds, as for CSV results: datetime64[us], timestamp[us], Datetime("us").
     • PandasCursor: coerce_int96_timestamp_unit given to execute() still replaces the unit.
     • PolarsCursor: a schema given to execute() is used as given.
   • PolarsCursor with Polars 1.x cannot read these timestamps at microseconds itself.
     • Without read options given to execute() and with pyarrow installed, a whole UNLOAD read of a result with timestamps goes through pyarrow and PyAthena's S3FileSystem, and logs a warning that recommends Polars 2.0. In a local measurement it was about 4x slower and used about 1.8x the memory.
     • Chunked reads without read options already use pyarrow.
     • Otherwise (no pyarrow, or other read options such as storage_options or columns), Polars 1.x reads natively as before. The timestamps are cast to microseconds, and a warning says that values outside 1677-09-21 to 2262-04-11 are wrong.
   • Polars 2.0 does not read INT96 timestamps inside a Map at microseconds (INT96 timestamps inside a Map saturate although the schema requests Datetime("us") pola-rs/polars#30031). PyAthena reads such maps as lists of key-value structs and casts them back to Map.

   Benchmark input for this issue

   • On a local file of 5M rows (INT96 timestamps and strings), median of 5 reads, the time and peak RSS of the Polars 2.0, ArrowCursor, and PandasCursor whole reads did not change beyond run-to-run variation.
   • The Polars 1.x pyarrow read went from 17-19 ms / 339-354 MiB to 60-78 ms / 593-671 MiB.
   • S3-backed timing was not measured.
4. laughingman7743 commented on Oct 18, 2026

   

   laughingman7743

   on Oct 18, 2026

   MemberAuthor

   More actions

   Draft of the 4.2.0 release notes for the temporal type contract (#1212). It consolidates the items of #1224, #1233, #1234, #1236, #1273, and #1277 (and the comment above), and takes the Polars 1.x item from #1282, which is not merged yet. The contract is documented in docs/temporal_types.md (#1284).

   Temporal types

   Every cursor now returns one type per Athena temporal type, whatever the values and the read mode: see Temporal Types.

   Fixed: silent wrong values

   • With unload=True, PandasCursor, ArrowCursor, and PolarsCursor read timestamps at microseconds, including timestamps inside arrays, rows, and maps (Read UNLOAD timestamps at microseconds #1233). Values outside 1677-09-21 to 2262-04-11 were wrapped or saturated before: 1600-01-02 03:04:05.123 was read as 2184-07-22 02:38:38.832551616 by PandasCursor and ArrowCursor, and as 2262-04-11 23:47:16.854775807 by PolarsCursor. For Polars 1.x, see the behavior changes.
   • ArrowCursor reads NULL time, time with time zone, and timestamp with time zone values of CSV results as null instead of '' (Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236).
   • Chunked PolarsCursor reads of CSV results no longer fail on timestamps with more than nine fractional digits (TIMESTAMP(10) to TIMESTAMP(12)); the digits beyond the time unit are truncated (Convert the timestamp text of chunked Polars CSV reads in the scan #1234). With ignore_errors=True, whole and chunked reads truncate such timestamps instead of returning null.

   Behavior changes (migration notes)

   • UNLOAD timestamps are microseconds instead of nanoseconds, as for CSV results: datetime64[us], timestamp[us], and Datetime("us") (Read UNLOAD timestamps at microseconds #1233).
     • PandasCursor: coerce_int96_timestamp_unit given to execute() still replaces the unit.
     • PolarsCursor: a schema given to execute() is used as given.
     • PolarsCursor with Polars 1.x reads UNLOAD timestamps at nanoseconds and casts them to microseconds, so values outside 1677-09-21 to 2262-04-11 are wrong, and logs a warning that recommends Polars 2.0 or later. Chunked reads with pyarrow installed and no Polars read options read them correctly (Remove the Polars 1.x pyarrow fallback for whole UNLOAD reads with timestamps (GH-1280) #1282).
   • PandasCursor's temporal dtypes depend only on the Athena type (Give PandasCursor's temporal columns one dtype per Athena type #1273):
     • timestamp is always datetime64[us], with digits beyond microseconds truncated. It was [ns] for 7 or more fractional digits, [s] when all NULL or with engine="pyarrow" and no fraction, [ns] with engine="pyarrow" and a fraction, and str for a column that also held values outside the nanosecond range.
     • timestamp with time zone is datetime64[us, UTC]. It had the data's zone, or was object for several zones or an all-NULL column.
     • date from UNLOAD results is datetime64[us], as from CSV results. It was object (date).
     • The fetch methods return datetime, date, and aware datetime in UTC, and None for NULL, instead of pd.Timestamp and NaT.
     • With parse_dates, dtype, or converters given to execute(), PyAthena leaves the temporal columns to pandas, and the fetch methods return pandas' values for them.
     • engine="pyarrow" is now also used for results with timestamp with time zone columns.
   • ArrowCursor reads date from CSV results as date32 instead of timestamp[ms], as from UNLOAD results; as_polars() gives pl.Date instead of pl.Datetime("ms") (Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236).
   • A timestamp with time zone value with an unknown zone name, or with a local time that its zone repeats or skips (daylight saving time), raises DataError instead of returning a naive datetime or the earlier instant (Raise DataError for time zone values without one instant #1224). Convert such values to UTC in SQL (AT TIME ZONE 'UTC') or read to_unixtime().
   • A value that fails to convert raises DataError naming the column (Column 'name': ...) in every cursor (Give PandasCursor's temporal columns one dtype per Athena type #1273, Name the column in DataError from the fetch methods of every cursor #1277). PandasCursor whole reads raise it instead of an OperationalError that wrapped it; a DataError from a function given in converters is still wrapped in OperationalError.
   • tzdata>=2024.2 is required (Raise DataError for time zone values without one instant #1224).
5. laughingman7743

   added 8 commits that reference this issue on Oct 18, 2026

   Say where the cursors raise DataError for time zone values (GH-1221)

   ...

   82961e9

   Say where the cursors raise DataError for time zone values (GH-1221)

   ...

   5f077b2

   Exclude PandasCursor from the time zone types of the fetch methods (G…

   ...

   4da93d3

   State the UTC year range of PandasCursor and when DataError is raised (…

   ...

   ed3553d

   Name chunked PandasCursor reads among the places DataError is raised (G…

   ...

   9551663

   Name chunked PandasCursor reads among the places DataError is raised (G…

   ...

   35b149d

   Give examples instead of a list of where DataError is raised (GH-1221)

   ...

   7053f96

   Merge pull request #1284 from pyathena-dev/docs/1221-temporal-types

   ...

   7b2a084
6. laughingman7743 commented on Oct 18, 2026

   

   laughingman7743

   on Oct 18, 2026

   MemberAuthor

   More actions

   Closed by #1284 (merged as 7b2a084), which documents the contract. The 4.2.0 release-note draft is in #1221 (comment), and the benchmarks moved to #1287.
7. laughingman7743

   closed this as completedon Oct 18, 2026