You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Remove the Polars 1.x whole-read pyarrow fallback for UNLOAD timestamps #1280
Follow-up to #1214 / #1233 and the revised decision of #1235.
Current behavior
For a whole UNLOAD read with PolarsCursor on Polars 1.x, AthenaPolarsResultSet._read_parquet() (pyathena/polars/result_set.py:783) switches to pyarrow when the files hold timestamps, pyarrow is installed, and execute() was given no Polars read options: _read_parquet_with_pyarrow() reads the data manifest's files with pyarrow.parquet.ParquetDataset(..., coerce_int96_timestamp_unit="us") through PyAthena's S3FileSystem, and a warning says that the result is read with pyarrow instead and recommends Polars 2.0. docs/polars.md ("Timestamps of UNLOAD results", first bullet) describes this.
The reason is a limitation of Polars 1.x, which reads INT96 at nanoseconds whatever the read schema; Polars 2.0 reads them at the schema's unit. In every other case (read options given, or no pyarrow), Polars 1.x already reads the files natively: the timestamps are read at nanoseconds, cast to microseconds, and a warning says that values outside 1677-09-21 to 2262-04-11 are wrong.
Proposal
Remove the whole-read pyarrow fallback. PyAthena does not switch readers on the user's behalf to work around a dependency's limitation; Polars 1.x reads the result with Polars in every case, with the existing cast and warning, and users who need the full timestamp range upgrade to Polars 2.0 (the warning already says so).
Remove _read_parquet_with_pyarrow() and the pyarrow branch of _read_parquet(); _reads_with_pyarrow() then only governs the chunked read.
Keep the chunked read through ParquetChunkReader (_read_parquet_chunks()): it exists for bounded memory use, not for INT96, and it reads INT96 at microseconds as a consequence.
Update docs/polars.md (the UNLOAD timestamps section and the S3 read options section, which list this read among the reads through PyAthena's filesystem), the cursor docstrings, and the tests of Read UNLOAD timestamps at microseconds #1233 that cover the fallback (tests/pyathena/polars/test_cursor.py::TestPolarsCursor::test_unload_nested_timestamps and the Polars 1.x cases around it).
This is unreleased (#1233 is in 4.2.0), so there is no compatibility consequence.
Activity
Give the nanosecond warning one remedy and keep the test IDs (GH-1280)
Test the Polars 1.x cast of nested UNLOAD timestamps with in-range va…
Merge pull request #1282 from pyathena-dev/fix/1280-remove-polars-pya…