You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Read CSV dates and NULL temporal values in ArrowCursor as the contract defines #1216
ArrowCursor reads date from CSV as timestamp[ms]. UNLOAD gives date32, which the contract requires for both.
ArrowCursor reads NULL time, time with time zone, and timestamp with time zone values as '', while the fetch methods return None (moved here from ArrowCursor reads NULL time and time zone values as empty strings #1218). These columns are text in ArrowCursor, so they inherit the reader's string handling. Athena never writes an empty string for these types, so '' can only be NULL.
Expected behavior
ArrowCursor date is date32 from CSV, as from UNLOAD.
NULL time and time zone values are null. Whether varchar changes too is outside this issue.
PolarsCursor's chunked timestamp reads are tracked in #1191 (Draft PR #1194).
Chunked PolarsCursor CSV reads take the timestamp columns as String and convert them with _to_datetimes() in the scan_csv() query plan (cs.by_name(..., require_all=False), so a LazyFrame's schema is not resolved). strict follows ignore_errors. With schema, schema_overrides, or with_column_names given to execute(), or for headerless results, the columns are not read as text.
Whole reads keep the native parse with the text fallback on ComputeError. Reading all timestamps as text in whole reads was 2-4.5x faster but raised peak RSS by about 35-40% (5M rows, Polars 1.39.0 and 2.0.0), and was rejected.
Chunked reads with the lazy conversion: Polars 1.39.0 1.15-1.19 s → 0.19-0.23 s, 395-411 → 554-611 MiB; Polars 2.0.0 0.43-0.51 s → 0.19-0.21 s, 412-461 → 523-548 MiB. Polars 1.39.0 parses the whole file before collect_batches() returns the first batch.
changed the title [-]Parse timestamp text in ArrowCursor and PolarsCursor by Athena's format[/-][+]Read CSV dates and NULL temporal values in ArrowCursor as the contract defines[/+]on Oct 17, 2026
Activity
laughingman7743 commentedon Oct 16, 2026
Carrying over Draft PR #1194 (branch
fix/1191-polars-chunked-timestamps, head eee0346), which fixes #1191 and is to be reworked here.State of #1194:
Stringand convert them with_to_datetimes()in thescan_csv()query plan (cs.by_name(..., require_all=False), so a LazyFrame's schema is not resolved).strictfollowsignore_errors. Withschema,schema_overrides, orwith_column_namesgiven toexecute(), or for headerless results, the columns are not read as text.ComputeError. Reading all timestamps as text in whole reads was 2-4.5x faster but raised peak RSS by about 35-40% (5M rows, Polars 1.39.0 and 2.0.0), and was rejected.collect_batches()returns the first batch.To rework against #1212 before it can merge:
_to_datetimes()and return a wrong fraction (PolarsCursor truncates timestamp fractions wrongly for years outside 0000-9999 #1201, closed in favor of this issue); master raises for them in chunked reads.str.to_datetime()with the explicit format gave equal results for four-digit years and was 25-40% faster (from closed PR Parse Polars timestamp text without cutting it at a fixed length #1204).[-]Parse timestamp text in ArrowCursor and PolarsCursor by Athena's format[/-][+]Read CSV dates and NULL temporal values in ArrowCursor as the contract defines[/+]