Skip to content

Remove the Polars 1.x whole-read pyarrow fallback for UNLOAD timestamps #1280

Closed

Description

@laughingman7743

Follow-up to #1214 / #1233 and the revised decision of #1235.

Current behavior

For a whole UNLOAD read with PolarsCursor on Polars 1.x, AthenaPolarsResultSet._read_parquet() (pyathena/polars/result_set.py:783) switches to pyarrow when the files hold timestamps, pyarrow is installed, and execute() was given no Polars read options: _read_parquet_with_pyarrow() reads the data manifest's files with pyarrow.parquet.ParquetDataset(..., coerce_int96_timestamp_unit="us") through PyAthena's S3FileSystem, and a warning says that the result is read with pyarrow instead and recommends Polars 2.0. docs/polars.md ("Timestamps of UNLOAD results", first bullet) describes this.

The reason is a limitation of Polars 1.x, which reads INT96 at nanoseconds whatever the read schema; Polars 2.0 reads them at the schema's unit. In every other case (read options given, or no pyarrow), Polars 1.x already reads the files natively: the timestamps are read at nanoseconds, cast to microseconds, and a warning says that values outside 1677-09-21 to 2262-04-11 are wrong.

Proposal

Remove the whole-read pyarrow fallback. PyAthena does not switch readers on the user's behalf to work around a dependency's limitation; Polars 1.x reads the result with Polars in every case, with the existing cast and warning, and users who need the full timestamp range upgrade to Polars 2.0 (the warning already says so).

  • Remove _read_parquet_with_pyarrow() and the pyarrow branch of _read_parquet(); _reads_with_pyarrow() then only governs the chunked read.
  • Keep the chunked read through ParquetChunkReader (_read_parquet_chunks()): it exists for bounded memory use, not for INT96, and it reads INT96 at microseconds as a consequence.
  • Update docs/polars.md (the UNLOAD timestamps section and the S3 read options section, which list this read among the reads through PyAthena's filesystem), the cursor docstrings, and the tests of Read UNLOAD timestamps at microseconds #1233 that cover the fallback (tests/pyathena/polars/test_cursor.py::TestPolarsCursor::test_unload_nested_timestamps and the Polars 1.x cases around it).

This is unreleased (#1233 is in 4.2.0), so there is no compatibility consequence.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions