Repository navigation
Benchmark the temporal type conversion changes of 4.2.0 #1287
Description
Moved from #1221 (part of #1212), whose documentation part was done in #1284.
Summary
Measure the elapsed time and peak RSS of the readers whose temporal conversion changed in 4.2.0, on the commit planned for 4.2.0, in the combined benchmark #1288 with #1159 and #1162.
This run measures 4.2.0 only. The comparison with the code before the changes comes from the local measurements of each PR (below). The combined run adds absolute values against real S3 and at the benchmark's scales.
Scope
Readers and paths whose conversion changed
- PandasCursor CSV results, whole and chunked, with the C, python, and pyarrow engines:
timestamp,date, andtimestamp with time zoneare converted per column after reading (Give PandasCursor's temporal columns one dtype per Athena type #1273). The fetch methods also convert the temporal columns todatetimeanddate(Give PandasCursor's temporal columns one dtype per Athena type #1273). - ArrowCursor CSV results:
dateis read asdate32, and the NULLtime,time with time zone, andtimestamp with time zonetext is replaced with null (Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236). - PolarsCursor CSV results, chunked: the timestamp text is converted in the scan (Convert the timestamp text of chunked Polars CSV reads in the scan #1234).
- UNLOAD results of PandasCursor, ArrowCursor, and PolarsCursor: INT96 timestamps are read at microseconds (Read UNLOAD timestamps at microseconds #1233, Remove the Polars 1.x pyarrow fallback for whole UNLOAD reads with timestamps (GH-1280) #1282). This includes the Polars 2.0
Mapread as a list of structs and cast back. - The fetch methods of Cursor, DictCursor, S3FSCursor, ArrowCursor, and PolarsCursor: the column-naming
DataError(Name the column in DataError from the fetch methods of every cursor #1277) and the time zone checks (Raise DataError for time zone values without one instant #1224).
Harness change
The current shapes (flat, nested, json) keep "timestamp" as a bigint (benchmarks/pyathena_bench/config.py, benchmarks/DATASET.md), so none of them converts a temporal value. Add a temporal shape:
- Columns, in
config.py: derived from the same snapshot, so that no new input data is needed. For example:- a
timestamp(3)fromfrom_unixtime("timestamp" / 1e6); - a
date; timestamp with time zonecolumns in UTC, in a zone name, and in an offset;time, if useful;- the flat scalar columns, so the result keeps a realistic width.
- a
- Cases, in
cases.py: add the shape toSHAPESandmatrix(). UNLOAD cannot writetimeorwith time zonetypes, or timestamps other thanTIMESTAMP(3). Either give UNLOAD cases a projection ofTIMESTAMP(3)anddateonly, or mark them unsupported, as thejsonshape does. - Update the harness tests under
benchmarks/tests,benchmarks/README.md, andbenchmarks/DATASET.md. - Merge this before the commit planned for 4.2.0, because the run measures that commit.
Out of scope: running the harness against v4.1.0. The harness uses the workspace's PyAthena, and the PR measurements already cover the comparison.
Local measurements so far (before and after)
The PRs measured on local files, without S3:
- Read UNLOAD timestamps at microseconds #1233: 5M rows, INT96 timestamps and strings. Polars 2.0, ArrowCursor, and PandasCursor whole reads did not change beyond run-to-run variation. The Polars 1.x pyarrow read went from 17-19 ms / 339-354 MiB to 60-78 ms / 593-671 MiB; Remove the Polars 1.x pyarrow fallback for whole UNLOAD reads with timestamps (GH-1280) #1282 removed that read.
- Convert the timestamp text of chunked Polars CSV reads in the scan #1234: chunked Polars CSV reads became faster, and their peak RSS grew with the text (table in the PR).
- Read CSV dates as date32 and temporal text NULL as null in ArrowCursor #1236: ArrowCursor CSV, 5M rows: 0.38-0.54 s → 0.60-0.64 s; the peak RSS ranges overlap.
- Give PandasCursor's temporal columns one dtype per Athena type #1273 (1M rows): PandasCursor CSV reads became 15-79% faster, with peak RSS from -10% to +12%, except a result of only a
timestamp with time zonecolumn (+94% to +104%, the text held until conversion). UNLOAD reads went from 0.05-0.06 s to 0.10-0.12 s, about 0.05 s of which is the cast of thedatecolumn.fetchall()became 68% faster with 19% less peak RSS. - Name the column in DataError from the fetch methods of every cursor #1277: +7% for GetQueryResults rows, +8% for DictCursor rows, +2-3% for S3FS rows, ±0 for Arrow.
- Raise DataError for time zone values without one instant #1224: per-value cost of the time zone checks in
_to_datetime_with_tz(), +3% for offsets and +10-11% forUTCand zone names.
Activity
Look up the shape's columns before choosing the Parquet projection (G…
Use a zone without DST and reject unknown transports in select_sql() (G…
Simplify the transport check of the temporal projection (GH-1287)
Name UNLOAD and CTAS as the writers without time zone types (GH-1287)
Compare the temporal matrix with the shape kept (GH-1287)
Merge pull request #1290 from pyathena-dev/feat/1287-temporal-bench-s…
laughingman7743 commentedon Oct 19, 2026
Measured in the combined 4.2.0 run (#1288), recorded in #1310 (
benchmarks/history/2026-10-19-9e99caa8.md, section "The temporal workload" and finding 6), in both Polars environments.At 1 million rows, client result time, temporal against flat:
timecolumns, they took 2.0, 3.4, and 1.5 s.event_timeandevent_date, stayed within 0.3 s of the flat workload.At 10 million rows, PandasCursor's CSV read with 100,000-row chunks took 163.7 s of client result time, against 76.5 s without chunks.
The improvement candidates are filed as #1308. Closed by #1310.