You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PolarsCursor: schema=None given to execute() disables the microsecond read of UNLOAD timestamps #1283
Found in the independent review of #1282 (Codex), pre-existing on master.
Behavior
Giving schema=None to execute() of PolarsCursor with the unload option disables the microsecond read of INT96 timestamps and its warning. schema=None is the default of pl.read_parquet() / pl.scan_parquet(), so a caller passing it on expects the default read.
The same happens with the chunksize option (the scan takes the same arguments, and the pyarrow chunk reader is not used once any read option is given) and with Polars 1.x.
Cause
AthenaPolarsResultSet._int96_read_options() returns {} when "schema" in self._kwargs, whatever the value, so _output_dtypes() returns no casts and no warning is logged; _read_kwargs() then passes the caller's schema=None to Polars, which reads INT96 at nanoseconds.
For CSV results, docs/polars.md documents that schema_overrides=None replaces PyAthena's column types on purpose, so an explicit None replacing PyAthena's value is an existing convention. For the Parquet schema, that convention gives wrong values silently.
Options
Treat an explicit schema=None as no schema given, for _int96_read_options() and _read_kwargs(): the result is read as without the option.
Keep the convention that None replaces PyAthena's value, but cast the nanosecond read to microseconds and log the warning, as for Polars 1.x; the values outside 1677-09-21 to 2262-04-11 stay wrong.
Either way, docs/polars.md ("Timestamps of UNLOAD results are read at microseconds") should say what an explicit schema or schema=None does.
Activity
Test both CSV read paths with each default option and Polars 1.x sche…
Say why schema_overrides keeps the CSV timestamps from being read as …
Say that the result set skips its own timestamp text read in those ca…
Merge pull request #1285 from pyathena-dev/fix/1283-polars-schema-none