Repository navigation
Chunked PolarsCursor CSV reads fail on timestamps with more fractional digits than the time unit #1191
Description
Summary
With chunksize, PolarsCursor fails to read a CSV result whose timestamp column has more fractional digits than the Polars Datetime time unit holds.
Without chunksize, the same result is read, with the values truncated.
_read_csv() (pyathena/polars/result_set.py:768) catches the ComputeError from pl.read_csv() and reads the timestamp columns again as text, then truncates them with _to_datetimes(), because Athena writes up to 12 fractional digits.
The chunked reader _scan_csv_chunks() (pyathena/polars/result_set.py:993 on #1190, _iter_csv_chunks() on master) passes the same Datetime dtypes to pl.scan_csv() and has no such fallback, so the read raises OperationalError.
Line numbers refer to the head of #1190 (c78e418). The fallback was added in fd80f98.
Found by a review of #1190.
Reproduction
A CSV result file "t"\n"2020-01-02 03:04:05.123456789012"\n with a timestamp column, read through AthenaPolarsResultSet with the S3 calls mocked:
chunksize=None:[(datetime.datetime(2020, 1, 2, 3, 4, 5, 123456),)]chunksize=10:OperationalError: could not parse `"2020-01-02 03:04:05.123456789012"` as dtype `datetime[μs]` at column 't' (column number 1)
Expected behavior
A chunked CSV read returns the same values as a whole read: timestamps with more fractional digits than the time unit are truncated, not rejected.
Activity
Read the timestamp columns of chunked Polars CSV results as text
Keep the whole Polars CSV read and convert chunks in the query plan
laughingman7743 commentedon Oct 17, 2026
Now a direct sub-issue of #1212, as #1216 covers only ArrowCursor. Years outside 1 to 9999 are out of scope (#1212), so Draft PR #1194 no longer needs to enforce the year range, and the fixed-length cut in
_to_datetimes()is correct for every year in range.Convert the timestamp text of chunked Polars CSV reads in the scan
laughingman7743 commentedon Oct 17, 2026
PR #1194 is superseded by #1234, rebuilt on the current master. Besides the chunked read, #1234 makes a whole read with
ignore_errorsread the timestamp columns as text: withignore_errors, Polars reads a timestamp with more than nine fractional digits as null instead of raisingComputeError, so the whole read returned null where the chunked read truncates. Polars parses up to nine fractional digits natively, so the failure in this issue applies toTIMESTAMP(10)toTIMESTAMP(12)values.