Repository navigation
UNLOAD timestamps outside 1677-2262 are read as wrong values #1214
Description
Part of #1212.
Summary
With unload=True, PandasCursor, ArrowCursor, and PolarsCursor return wrong values for timestamps outside 1677-09-21 to 2262-04-11, and their timestamp dtype is ns instead of us.
Facts
Athena UNLOAD writes TIMESTAMP(3) as Parquet INT96 (created_by: parquet-mr-trino). The readers decode INT96 at nanoseconds, and values outside the nanosecond range overflow. Measured on master b2d7aa2 (pandas 3.0.6, pyarrow 26.0.0, Polars 2.0.0):
| Athena value | PandasCursor / ArrowCursor | PolarsCursor |
|---|---|---|
0001-01-01 00:00:00.000 |
1754-08-30 22:43:41.128654848 |
2262-04-11 23:47:16.854775807 |
1600-01-02 03:04:05.123 |
2184-07-22 02:38:38.832551616 |
2262-04-11 23:47:16.854775807 |
2300-01-02 03:04:05.123 |
1715-06-14 03:29:31.413448384 |
2262-04-11 23:47:16.854775807 |
9999-12-31 23:59:59.999 |
1816-03-30 05:56:08.065277376 |
2262-04-11 23:47:16.854775807 |
The UNLOAD dtype is datetime64[ns], timestamp[ns], or Datetime("ns"), while CSV results are us. Reading the same file with pyarrow's coerce_int96_timestamp_unit="us" returns the correct values at microseconds.
Expected behavior
INT96 is decoded at microseconds in whole and chunked reads, for every reader path, including Polars without pyarrow. The dtype is then us, the same as for CSV results. Whether Polars 1.x can decode INT96 at microseconds without pyarrow needs to be checked.
Years outside 1 to 9999 are out of scope (#1212).
Activity