Skip to content

PolarsCursor: schema=None given to execute() disables the microsecond read of UNLOAD timestamps #1283

Closed

Description

@laughingman7743

Found in the independent review of #1282 (Codex), pre-existing on master.

Behavior

Giving schema=None to execute() of PolarsCursor with the unload option disables the microsecond read of INT96 timestamps and its warning. schema=None is the default of pl.read_parquet() / pl.scan_parquet(), so a caller passing it on expects the default read.

Measured on master d7f6710 with Polars 2.0.0:

cursor = connect(cursor_class=PolarsCursor).cursor(unload=True)
q = "SELECT TIMESTAMP '1600-01-01 00:00:00.000' AS ts"
cursor.execute(q).as_polars()
# {'ts': Datetime('us')}  [datetime(1600, 1, 1, 0, 0)]
cursor.execute(q, schema=None).as_polars()
# {'ts': Datetime('ns')}  [datetime(2262, 4, 11, 23, 47, 16, 854775)], no warning

The same happens with the chunksize option (the scan takes the same arguments, and the pyarrow chunk reader is not used once any read option is given) and with Polars 1.x.

Cause

AthenaPolarsResultSet._int96_read_options() returns {} when "schema" in self._kwargs, whatever the value, so _output_dtypes() returns no casts and no warning is logged; _read_kwargs() then passes the caller's schema=None to Polars, which reads INT96 at nanoseconds.

For CSV results, docs/polars.md documents that schema_overrides=None replaces PyAthena's column types on purpose, so an explicit None replacing PyAthena's value is an existing convention. For the Parquet schema, that convention gives wrong values silently.

Options

  1. Treat an explicit schema=None as no schema given, for _int96_read_options() and _read_kwargs(): the result is read as without the option.
  2. Keep the convention that None replaces PyAthena's value, but cast the nanosecond read to microseconds and log the warning, as for Polars 1.x; the values outside 1677-09-21 to 2262-04-11 stay wrong.

Either way, docs/polars.md ("Timestamps of UNLOAD results are read at microseconds") should say what an explicit schema or schema=None does.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions