Skip to content

Read CSV dates and NULL temporal values in ArrowCursor as the contract defines #1216

Closed
1 of 1 issue completed

Description

@laughingman7743

Part of #1212.

Summary

ArrowCursor reads date and NULL temporal values from CSV results differently from the contract in #1212.

Facts

Measured on master b2d7aa2 (pyarrow 26.0.0):

  • ArrowCursor reads date from CSV as timestamp[ms]. UNLOAD gives date32, which the contract requires for both.
  • ArrowCursor reads NULL time, time with time zone, and timestamp with time zone values as '', while the fetch methods return None (moved here from ArrowCursor reads NULL time and time zone values as empty strings #1218). These columns are text in ArrowCursor, so they inherit the reader's string handling. Athena never writes an empty string for these types, so '' can only be NULL.

Expected behavior

  • ArrowCursor date is date32 from CSV, as from UNLOAD.
  • NULL time and time zone values are null. Whether varchar changes too is outside this issue.

PolarsCursor's chunked timestamp reads are tracked in #1191 (Draft PR #1194).

Activity

  1. laughingman7743 commented on Oct 16, 2026

    @laughingman7743
    MemberAuthor

    Carrying over Draft PR #1194 (branch fix/1191-polars-chunked-timestamps, head eee0346), which fixes #1191 and is to be reworked here.

    State of #1194:

    • Chunked PolarsCursor CSV reads take the timestamp columns as String and convert them with _to_datetimes() in the scan_csv() query plan (cs.by_name(..., require_all=False), so a LazyFrame's schema is not resolved). strict follows ignore_errors. With schema, schema_overrides, or with_column_names given to execute(), or for headerless results, the columns are not read as text.
    • Whole reads keep the native parse with the text fallback on ComputeError. Reading all timestamps as text in whole reads was 2-4.5x faster but raised peak RSS by about 35-40% (5M rows, Polars 1.39.0 and 2.0.0), and was rejected.
    • Chunked reads with the lazy conversion: Polars 1.39.0 1.15-1.19 s → 0.19-0.23 s, 395-411 → 554-611 MiB; Polars 2.0.0 0.43-0.51 s → 0.19-0.21 s, 412-461 → 523-548 MiB. Polars 1.39.0 parses the whole file before collect_batches() returns the first batch.

    To rework against #1212 before it can merge:

  2. changed the title [-]Parse timestamp text in ArrowCursor and PolarsCursor by Athena's format[/-] [+]Read CSV dates and NULL temporal values in ArrowCursor as the contract defines[/+] on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions