From ce8212dbead1da7575677fdc6d3a089bf5b7b32f Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Mon, 19 Oct 2026 22:01:13 +0900 Subject: [PATCH 1/5] Record the PyAthena 4.2.0 benchmark runs (GH-1288) Add the history entries of the cursor run and the filesystem suite of the 4.2.0 release commit 9e99caa8, and update RESULTS.md: the index, and the recommendations from the new measurements, including chunked reads of 100 million rows, read_sql_query() through the pandas and Polars drivers, and AioS3FileSystem for asyncio applications. Co-Authored-By: Claude Opus 5.5 --- benchmarks/RESULTS.md | 19 +- .../history/2026-10-19-9e99caa8-filesystem.md | 145 +++ benchmarks/history/2026-10-19-9e99caa8.md | 1027 +++++++++++++++++ 3 files changed, 1183 insertions(+), 8 deletions(-) create mode 100644 benchmarks/history/2026-10-19-9e99caa8-filesystem.md create mode 100644 benchmarks/history/2026-10-19-9e99caa8.md diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index 5f0e4d0b8..c893ffae0 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -18,18 +18,21 @@ Compare entries only when they used the same instance type, architecture, and sc | 2026-10-11 | `8a69ce03` | 4.0.0 | `r7g.2xlarge` (arm64), fleet of 12 | Spot | medium, large, xlarge | [2026-10-11-8a69ce03](history/2026-10-11-8a69ce03.md) | | 2026-10-13 | `437276e1` | 4.1.0 release commit | `r7g.2xlarge` (arm64), fleet of 8 | Spot | medium, large, xlarge | [2026-10-13-437276e1](history/2026-10-13-437276e1.md) | | 2026-10-13 | `437276e1` | 4.1.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-13-437276e1-filesystem](history/2026-10-13-437276e1-filesystem.md) | +| 2026-10-19 | `9e99caa8` | 4.2.0 release commit | `r7g.2xlarge` (arm64), fleet of 10 | Spot | medium, large, xlarge, xxlarge | [2026-10-19-9e99caa8](history/2026-10-19-9e99caa8.md) | +| 2026-10-19 | `9e99caa8` | 4.2.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-19-9e99caa8-filesystem](history/2026-10-19-9e99caa8-filesystem.md) | ## Recommendations -These recommendations come from the runs of `437276e1` on `r7g.2xlarge` and `c8gn.2xlarge` hosts and the 4.0.0 run; the entries give the measured cases and their limits. +These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8gn.2xlarge` hosts, and, where a row says so, from the 4.0.0 run; the entries give the measured cases and their limits. | Need | Use | Measured | | --- | --- | --- | -| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.6 to 19.2 seconds through UNLOAD, against 36.0 to 74.4 seconds through CSV; nested rows in 36.4 to 37.9 seconds with ArrowCursor, PolarsCursor, or PandasCursor with pyarrow dtypes, against 229.5 to 306.0 seconds through CSV | -| A pandas DataFrame with nested columns | PandasCursor with `unload=True` and `dtype_backend="pyarrow"` | 10 million nested rows in 37.7 seconds and 5,897 MiB, against 86.8 seconds and 19,664 MiB with the default dtypes | -| Bounded memory | `chunksize` with PandasCursor or PolarsCursor, with CSV or UNLOAD results | 10 million rows at 363 to 904 MiB with 10,000-row chunks, against 3,687 to 19,664 MiB without chunks; larger chunks use more memory | -| Python rows of a large result | ArrowCursor with `unload=True`, or S3FSCursor for constant memory | GetQueryResults paging through Cursor or DictCursor took 226 to 230 seconds for 1 million rows; ArrowCursor with `unload=True` returned 10 million flat rows in 27 seconds in the 4.0.0 run, and S3FSCursor in 329 seconds at 321 MiB | -| `pandas.read_sql_query()` | `awsathena+arrow` with `unload=true`; for bounded memory, `awsathena+polars` or `awsathena+pandas` with `connect_args={"cursor_kwargs": {"unload": True, "chunksize": 100000}}`, which replaces the cursor options of the URL, and `chunksize=100000` in `read_sql_query()` | 1 million flat rows in 7.0 seconds through `awsathena+arrow`, against 230 seconds through the default `awsathena+rest` | -| A Wrangler-like pandas read | PandasCursor | 10 million flat rows in 19.0 seconds through UNLOAD and 74.4 seconds through CSV, against 21.4 and 95.3 seconds for Wrangler | +| A DataFrame or Table of a large result | `unload=True` with ArrowCursor, PolarsCursor, or PandasCursor | 10 million flat rows in 18.2 to 19.8 seconds through UNLOAD, against 55.2 to 84.2 seconds through CSV; nested rows in 33.8 to 36.4 seconds with ArrowCursor, PolarsCursor, or PandasCursor with pyarrow dtypes, against 211.3 to 280.3 seconds through CSV | +| A pandas DataFrame with nested columns | PandasCursor with `unload=True` and `dtype_backend="pyarrow"` | 10 million nested rows in 34.2 seconds and 5,820 MiB, against 88.0 seconds and 19,498 MiB with the default dtypes | +| Bounded memory, including results larger than memory | `chunksize` with PandasCursor or PolarsCursor, with CSV or UNLOAD results | 100 million flat rows, about 29 GB of CSV, at 462 to 1,051 MiB with 100,000-row chunks; 10 million rows at 362 to 917 MiB with 10,000-row chunks, against 3,765 to 19,498 MiB without chunks; larger chunks use more memory | +| A whole CSV result in PolarsCursor | Polars 2.0, or `chunksize`; with Polars 1.x, the read holds a copy of the result file (#1307) | 10 million flat rows peaked at 7,547 MiB with Polars 2.0.0 and 9,641 MiB with Polars 1.44.2 | +| Python rows of a large result | ArrowCursor with `unload=True`, or S3FSCursor for constant memory | GetQueryResults paging through Cursor or DictCursor took 217 to 225 seconds for 1 million rows; ArrowCursor with `unload=True` returned 10 million flat rows in 27 seconds in the 4.0.0 run, and S3FSCursor in 349 seconds at 323 MiB, and 100 million rows in 3,387 seconds at 327 MiB | +| `pandas.read_sql_query()` | `awsathena+arrow`, `awsathena+polars`, or `awsathena+pandas` with `unload=true`; for bounded memory, `awsathena+polars` or `awsathena+pandas` with `connect_args={"cursor_kwargs": {"unload": True, "chunksize": 100000}}`, which replaces the cursor options of the URL, and `chunksize=100000` in `read_sql_query()` | 1 million flat rows in 6.7 to 7.0 seconds, against 227 seconds through the default `awsathena+rest`; 10 million flat rows in 48.7 to 50.9 seconds, and at 1,032 to 1,054 MiB with chunks through UNLOAD with the pandas and Polars drivers | +| A Wrangler-like pandas read | PandasCursor | 10 million flat rows in 19.8 seconds through UNLOAD and 84.2 seconds through CSV, against 23.4 and 108.9 seconds for Wrangler | | Asynchronous code | The native asyncio cursors, or the synchronous and ThreadPool APIs offloaded with `asyncio.to_thread()` or an executor; synchronous calls block the event loop while they poll | Single-query timings of the three APIs differed by a median of 1.7% in the 4.0.0 run | -| Parallel S3 writes or copies of large objects with more workers than an event loop's default executor has threads, `min(32, os.cpu_count() + 4)` | `S3FileSystem` | With 40 workers on 8 vCPUs, `AioS3FileSystem` wrote 1 GiB and 6 GiB objects at 0.75 to 0.83 times and copied 6 GiB at 0.30 to 0.35 times its throughput, and read 256 MiB and 1 GiB objects at 0.92 to 1.06 times; 64 MiB and 256 MiB writes ran at 0.90 to 1.12 times (#1159) | +| S3 transfers in asyncio applications | `AioS3FileSystem`; PyAthena keeps `S3FileSystem` registered for fsspec's `s3` and `s3a` protocols, and making `AioS3FileSystem` the default is to be revisited when boto3 supports asyncio natively, because it still runs the blocking boto3 calls in threads | With 40 workers on 8 vCPUs, `AioS3FileSystem` wrote 1 GiB and 6 GiB objects at 0.97 to 1.00 times and copied 6 GiB at 1.01 to 1.05 times the throughput of `S3FileSystem`, and read 1 GiB at 1.00 times; with 8 workers, it copied 6 GiB at 1.36 to 1.49 times (#1159) | diff --git a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md new file mode 100644 index 000000000..795280777 --- /dev/null +++ b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md @@ -0,0 +1,145 @@ + + +# S3 filesystem benchmark of the PyAthena 4.2.0 release commit on 2026-10-19 + +This entry records the [filesystem suite](../README.md#filesystem-suite) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), which is tagged `v4.2.0`, for #1159. +It repeats the selection of the [4.1.0 run](2026-10-13-437276e1-filesystem.md), after #1184 gave the asyncio paths of `AioS3FileSystem` thread pools sized by `max_workers` instead of the event loop's default executor, and #1165 bounded the parts in flight of `S3File` writes. +The cursor run of the same commit is a [separate entry](2026-10-19-9e99caa8.md). + +## Environment + +| Item | Value | +| --- | --- | +| Commit | `9e99caa84857f8d29abe63435461fc412e040c43`, with a clean checkout | +| Lockfile SHA-256 | `0715fe352738dc098f28c35ed3ff597ac9949387c6aa93595f728ce9f929eac7`, the same as the commit's `uv.lock` | +| Python | 3.12.14 | +| Libraries | boto3 1.43.102, fsspec 2026.9.0 | +| Host | one `c8gn.2xlarge` (arm64, 8 vCPUs, 16 GiB, 25 Gbps baseline network bandwidth), Spot, `us-west-2b`, Amazon Linux 2023 (`ami-0957e38de3db270cc`), kernel 6.18.54 | +| Objects | zero bytes in the stack's scratch bucket in `us-west-2`: 64 MiB (small), 256 MiB (medium), 1 GiB (large), and 6 GiB (xlarge) | +| Block size | 8 MiB for range requests, parts, and copied parts | + +## Selection + +The suite ran as in the 4.1.0 run: 16 jobs, five repetitions, no warmups. + +- **small, medium, large**: reads and writes. +- **xlarge**: writes and copies. + +Each operation ran with the sync and aio APIs, `max_workers` 8 and 40, and connection pools (`max_pool_connections`) of 10 and 40. +The sync API calls the filesystem on the trial's thread; the aio API runs reads and writes in `asyncio.to_thread()`, as `AioS3FSCursor` does, so that a file's parallel requests go through `S3AioExecutor` to the file's pool of `max_workers` threads, and awaits the copy coroutine on the trial's event loop, whose part copies run in the filesystem's pool of `max_workers` threads. + +## Run record + +The queue `fs-1` was published at 07:40 UTC, and its 16 jobs finished on the one host by 08:03 UTC without an interruption; the host then retired itself. +Each of the 64 cases recorded five successful trials and no failure. +The suite does not query Athena; the stack of the [cursor run](2026-10-19-9e99caa8.md) was deployed at the same time and published its queue after this suite had finished. + +## Findings + +1. **With 40 workers, aio writes and copies ran at the sync throughput** (#1184). + For the 6 GiB object, sync wrote 1,050 to 1,074 MiB/s and aio 1,047 to 1,052 MiB/s (0.98 to 1.00 times sync), against 0.75 to 0.77 times in the 4.1.0 run; for the 1 GiB object, 0.97 to 0.99 times, against 0.81 to 0.83. + Sync copied the 6 GiB object at 1,835 to 1,879 MiB/s and aio at 1,889 to 1,925 MiB/s (1.01 to 1.05 times), against 521 to 540 MiB/s (0.30 to 0.35 times) in the 4.1.0 run. + With the 1 GiB and 6 GiB objects, the aio processes peaked at 52 to 53 threads, against 24 to 25 in the 4.1.0 run and 51 for sync, so their thread counts follow `max_workers`. +2. **Reads and the 8-worker cases did not regress.** + For the 1 GiB object, aio read at 0.96 to 1.06 times sync; with 8 workers, aio wrote at 0.95 to 1.02 times sync from 256 MiB up, and copied at 1.36 to 1.49 times, against 1.33 to 1.46 in the 4.1.0 run. + The 64 MiB transfers ran at 0.74 to 1.29 times sync, without a consistent order among the settings, as in the 4.1.0 run. +3. **Writes no longer held the whole object in memory** (#1165). + Writing the 6 GiB object peaked at 690 to 699 MiB with 40 workers and 302 to 305 MiB with 8 workers, with either API, against 6,005 to 6,313 MiB in the 4.1.0 run; the 1 GiB object peaked at 688 to 691 and 302 to 304 MiB, against 1,245 to 1,376 MiB. +4. **Sync transfers of large objects ran faster than in the 4.1.0 run.** + Sync wrote the 6 GiB object at 1,050 to 1,074 MiB/s with 40 workers and 553 to 560 MiB/s with 8 workers, against 861 to 864 and 470 to 487 MiB/s, and copied it with 40 workers at 1,835 to 1,879 MiB/s, against 1,495 to 1,781 MiB/s. + The cause of these differences was not investigated. + +For #1159 and #685, the aio paths now match the sync paths for writes and copies, and `S3FileSystem` stays the filesystem that PyAthena registers for fsspec's `s3` and `s3a` protocols. +`AioS3FileSystem` is recommended for asyncio applications, and making it the default is to be revisited when boto3 supports asyncio natively, because `AioS3FileSystem` still runs the blocking boto3 calls in threads. + +## Limits + +- One host and five trials per case; one `c8gn.2xlarge` host in one Availability Zone. +- The objects hold zero bytes; S3 does not compress them, but other content was not measured. +- Reads of the 6 GiB object did not run, and copies only ran at 6 GiB. +- Comparisons with the 4.1.0 run use the same instance type and selection, but different hosts and days. + +## Results + +Median throughput is the object size divided by the median transfer time, in MiB per second; threads are the peak thread count of the trial process, and RSS its peak sampled resident memory, including the harness's common imports. + +| Scale | Operation | Workers | Pool | sync (MiB/s) | aio (MiB/s) | aio / sync | sync threads | aio threads | sync peak RSS (MiB) | aio peak RSS (MiB) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| small (64 MiB) | read | 8 | 10 | 295 | 253 | 0.86 | 19 | 21 | 332 | 322 | +| small (64 MiB) | read | 8 | 40 | 321 | 239 | 0.74 | 19 | 21 | 328 | 328 | +| small (64 MiB) | read | 40 | 10 | 308 | 230 | 0.75 | 19 | 21 | 330 | 344 | +| small (64 MiB) | read | 40 | 40 | 283 | 266 | 0.94 | 19 | 21 | 321 | 347 | +| small (64 MiB) | write | 8 | 10 | 131 | 168 | 1.29 | 19 | 21 | 285 | 287 | +| small (64 MiB) | write | 8 | 40 | 171 | 145 | 0.85 | 19 | 21 | 285 | 287 | +| small (64 MiB) | write | 40 | 10 | 162 | 170 | 1.05 | 19 | 21 | 285 | 287 | +| small (64 MiB) | write | 40 | 40 | 170 | 148 | 0.87 | 19 | 21 | 285 | 287 | +| medium (256 MiB) | read | 8 | 10 | 423 | 425 | 1.01 | 19 | 21 | 658 | 650 | +| medium (256 MiB) | read | 8 | 40 | 413 | 414 | 1.00 | 19 | 21 | 655 | 638 | +| medium (256 MiB) | read | 40 | 10 | 512 | 511 | 1.00 | 43 | 45 | 689 | 847 | +| medium (256 MiB) | read | 40 | 40 | 514 | 507 | 0.99 | 43 | 45 | 813 | 764 | +| medium (256 MiB) | write | 8 | 10 | 348 | 329 | 0.95 | 19 | 21 | 301 | 304 | +| medium (256 MiB) | write | 8 | 40 | 343 | 327 | 0.95 | 19 | 21 | 302 | 303 | +| medium (256 MiB) | write | 40 | 10 | 408 | 452 | 1.11 | 43 | 45 | 575 | 577 | +| medium (256 MiB) | write | 40 | 40 | 473 | 452 | 0.96 | 43 | 45 | 575 | 577 | +| large (1 GiB) | read | 8 | 10 | 438 | 465 | 1.06 | 19 | 21 | 2,220 | 2,207 | +| large (1 GiB) | read | 8 | 40 | 468 | 452 | 0.96 | 19 | 21 | 2,182 | 2,184 | +| large (1 GiB) | read | 40 | 10 | 602 | 600 | 1.00 | 51 | 53 | 2,423 | 2,369 | +| large (1 GiB) | read | 40 | 40 | 592 | 592 | 1.00 | 51 | 53 | 2,240 | 2,222 | +| large (1 GiB) | write | 8 | 10 | 496 | 506 | 1.02 | 19 | 21 | 302 | 304 | +| large (1 GiB) | write | 8 | 40 | 512 | 494 | 0.97 | 19 | 21 | 302 | 304 | +| large (1 GiB) | write | 40 | 10 | 831 | 804 | 0.97 | 51 | 53 | 688 | 691 | +| large (1 GiB) | write | 40 | 40 | 823 | 815 | 0.99 | 51 | 53 | 688 | 691 | +| xlarge (6 GiB) | write | 8 | 10 | 553 | 561 | 1.01 | 19 | 21 | 302 | 305 | +| xlarge (6 GiB) | write | 8 | 40 | 560 | 566 | 1.01 | 19 | 21 | 303 | 305 | +| xlarge (6 GiB) | write | 40 | 10 | 1,050 | 1,052 | 1.00 | 51 | 53 | 690 | 699 | +| xlarge (6 GiB) | write | 40 | 40 | 1,074 | 1,047 | 0.98 | 51 | 53 | 690 | 692 | +| xlarge (6 GiB) | copy | 8 | 10 | 285 | 425 | 1.49 | 19 | 20 | 191 | 192 | +| xlarge (6 GiB) | copy | 8 | 40 | 299 | 407 | 1.36 | 19 | 20 | 191 | 192 | +| xlarge (6 GiB) | copy | 40 | 10 | 1,835 | 1,925 | 1.05 | 51 | 52 | 217 | 218 | +| xlarge (6 GiB) | copy | 40 | 40 | 1,879 | 1,889 | 1.01 | 51 | 52 | 217 | 218 | + +### Compared with 4.1.0 + +The same cases in the [4.1.0 run](2026-10-13-437276e1-filesystem.md). + +| Scale | Operation | Workers | Pool | sync 4.1.0 (MiB/s) | sync (MiB/s) | aio 4.1.0 (MiB/s) | aio (MiB/s) | aio / sync 4.1.0 | aio / sync | aio threads 4.1.0 | aio threads | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| small (64 MiB) | read | 8 | 10 | 215 | 295 | 304 | 253 | 1.41 | 0.86 | 21 | 21 | +| small (64 MiB) | read | 8 | 40 | 260 | 321 | 309 | 239 | 1.19 | 0.74 | 21 | 21 | +| small (64 MiB) | read | 40 | 10 | 270 | 308 | 312 | 230 | 1.16 | 0.75 | 21 | 21 | +| small (64 MiB) | read | 40 | 40 | 274 | 283 | 202 | 266 | 0.74 | 0.94 | 21 | 21 | +| small (64 MiB) | write | 8 | 10 | 162 | 131 | 175 | 168 | 1.08 | 1.29 | 21 | 21 | +| small (64 MiB) | write | 8 | 40 | 172 | 171 | 167 | 145 | 0.97 | 0.85 | 21 | 21 | +| small (64 MiB) | write | 40 | 10 | 146 | 162 | 163 | 170 | 1.12 | 1.05 | 21 | 21 | +| small (64 MiB) | write | 40 | 40 | 180 | 170 | 162 | 148 | 0.90 | 0.87 | 21 | 21 | +| medium (256 MiB) | read | 8 | 10 | 382 | 423 | 418 | 425 | 1.09 | 1.01 | 22 | 21 | +| medium (256 MiB) | read | 8 | 40 | 359 | 413 | 412 | 414 | 1.15 | 1.00 | 23 | 21 | +| medium (256 MiB) | read | 40 | 10 | 488 | 512 | 457 | 511 | 0.94 | 1.00 | 25 | 45 | +| medium (256 MiB) | read | 40 | 40 | 509 | 514 | 468 | 507 | 0.92 | 0.99 | 25 | 45 | +| medium (256 MiB) | write | 8 | 10 | 302 | 348 | 329 | 329 | 1.09 | 0.95 | 22 | 21 | +| medium (256 MiB) | write | 8 | 40 | 321 | 343 | 346 | 327 | 1.08 | 0.95 | 22 | 21 | +| medium (256 MiB) | write | 40 | 10 | 453 | 408 | 407 | 452 | 0.90 | 1.11 | 25 | 45 | +| medium (256 MiB) | write | 40 | 40 | 448 | 473 | 421 | 452 | 0.94 | 0.96 | 25 | 45 | +| large (1 GiB) | read | 8 | 10 | 432 | 438 | 448 | 465 | 1.04 | 1.06 | 23 | 21 | +| large (1 GiB) | read | 8 | 40 | 453 | 468 | 449 | 452 | 0.99 | 0.96 | 24 | 21 | +| large (1 GiB) | read | 40 | 10 | 550 | 602 | 581 | 600 | 1.06 | 1.00 | 25 | 53 | +| large (1 GiB) | read | 40 | 40 | 565 | 592 | 590 | 592 | 1.04 | 1.00 | 25 | 53 | +| large (1 GiB) | write | 8 | 10 | 448 | 496 | 439 | 506 | 0.98 | 1.02 | 25 | 21 | +| large (1 GiB) | write | 8 | 40 | 423 | 512 | 433 | 494 | 1.03 | 0.97 | 25 | 21 | +| large (1 GiB) | write | 40 | 10 | 718 | 831 | 593 | 804 | 0.83 | 0.97 | 25 | 53 | +| large (1 GiB) | write | 40 | 40 | 712 | 823 | 575 | 815 | 0.81 | 0.99 | 25 | 53 | +| xlarge (6 GiB) | write | 8 | 10 | 470 | 553 | 497 | 561 | 1.06 | 1.01 | 25 | 21 | +| xlarge (6 GiB) | write | 8 | 40 | 487 | 560 | 485 | 566 | 1.00 | 1.01 | 25 | 21 | +| xlarge (6 GiB) | write | 40 | 10 | 861 | 1,050 | 647 | 1,052 | 0.75 | 1.00 | 25 | 53 | +| xlarge (6 GiB) | write | 40 | 40 | 864 | 1,074 | 665 | 1,047 | 0.77 | 0.98 | 25 | 53 | +| xlarge (6 GiB) | copy | 8 | 10 | 267 | 285 | 390 | 425 | 1.46 | 1.49 | 24 | 20 | +| xlarge (6 GiB) | copy | 8 | 40 | 274 | 299 | 366 | 407 | 1.33 | 1.36 | 24 | 20 | +| xlarge (6 GiB) | copy | 40 | 10 | 1,495 | 1,835 | 521 | 1,925 | 0.35 | 1.05 | 24 | 52 | +| xlarge (6 GiB) | copy | 40 | 40 | 1,781 | 1,879 | 540 | 1,889 | 0.30 | 1.01 | 24 | 52 | diff --git a/benchmarks/history/2026-10-19-9e99caa8.md b/benchmarks/history/2026-10-19-9e99caa8.md new file mode 100644 index 000000000..8aec381c8 --- /dev/null +++ b/benchmarks/history/2026-10-19-9e99caa8.md @@ -0,0 +1,1027 @@ + + +# Cursor benchmark of the PyAthena 4.2.0 release commit on 2026-10-19 + +This entry records a bounded large-data run of the [benchmark harness](../README.md) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), which is tagged `v4.2.0`. +The installed distribution reports the version `4.1.1.dev452+g9e99caa84` because the run measured the commit before it was tagged. +It repeats the selection of the [4.1.0 run](2026-10-13-437276e1.md) and adds the `temporal` workload (#1287), a scale of 100 million rows (#1162), and every Polars case again with Polars 2.0 (#1292). +The S3 filesystem comparison of the same commit is a [separate entry](2026-10-19-9e99caa8-filesystem.md). + +## Environment + +| Item | Value | +| --- | --- | +| Commit | `9e99caa84857f8d29abe63435461fc412e040c43` on every host, with a clean checkout | +| Lockfile SHA-256 | `0715fe352738dc098f28c35ed3ff597ac9949387c6aa93595f728ce9f929eac7`, the same as the commit's `uv.lock` | +| Python | 3.12.14 | +| Libraries | pandas 3.0.6, pyarrow 25.0.1, polars 1.44.2, awswrangler 3.17.1, SQLAlchemy 2.0.46, boto3 1.43.102, fsspec 2026.9.0, numpy 2.5.3, tzdata 2024.2 | +| Polars 2.0 environment | the same versions, with polars 2.0.0 instead of 1.44.2 (`.venv-polars2`) | +| Hosts | `r7g.2xlarge` (arm64, 8 vCPUs, 64 GiB), Spot, Amazon Linux 2023 (`ami-0957e38de3db270cc`), kernel 6.18.54 | +| Region and workgroup | `us-west-2`, `pyathena` | +| Athena engine | Athena engine version 3 | +| Input | `pyathena_benchmark.pypi_file_downloads`, partition `download_date='2026-09-17'`, see [the dataset reference](../DATASET.md) | +| Snapshots | medium 100,000 rows, large 1,000,000 rows, xlarge 10,000,000 rows, xxlarge 100,000,000 rows, prepared once and shared by every host | + +Every host recorded the same commit, lockfile, dependency versions, kernel, CPU count, and memory, 66,199,420,928 bytes (63,133 MiB). +The queue's host records list the instance type `r7g.2xlarge`, the Availability Zone `us-west-2d`, and the lifecycle `spot` for all ten hosts that ran jobs. + +## Selection + +The run used 167 jobs without warmups: + +- **The 4.1.0 selection**, 96 jobs, as described in the [4.1.0 entry](2026-10-13-437276e1.md#selection) and in [Running on a fleet](../README.md#running-on-a-fleet). +- **The `temporal` workload**, 34 jobs: at large, the row output of Cursor, DictCursor, and S3FSCursor and of the fetch methods of PandasCursor, ArrowCursor, and PolarsCursor at arraysize 1000, every native read of the DataFrame libraries and Wrangler, and `read_sql_query()` through the pandas, Arrow, Polars, and S3FS drivers without chunks and with 100,000-row chunks; at xlarge, the native reads without chunks and with 100,000-row chunks; all with three repetitions. +- **xxlarge (100 million rows)**, 9 jobs, flat: PandasCursor, PolarsCursor, and Wrangler with `chunksize=100000` from every transport, including PandasCursor UNLOAD with `dtype_backend="pyarrow"`, S3FSCursor rows at arraysize 1000, all with three repetitions, and one unchunked PandasCursor CSV read with one repetition. +- **Polars 2.0**, 28 jobs: every `polars` and `sqla_polars` job of the selections above again, with `jobs --environment polars2`. + +The configuration matched the 4.1.0 run's, `config.toml` with `chunksizes = [10000, 100000, 1000000]` and `timeout_seconds = 5400.0`, and added the scale `xxlarge = 100000000`. +Only the synchronous API ran. + +## Run record + +The stack of ten Spot hosts was created at 07:31 UTC, and its hosts started at 07:35 UTC in `us-west-2b` and `us-west-2d`. +Before the queue was published, the group replaced seven of its ten hosts between 08:00 and 08:07 UTC; one of them, the host that was preparing the snapshots, was terminated with the reason `Server.SpotInstanceTermination` at 08:01 UTC, and its preparation and manifest were lost. +The preparation ran again from a local checkout of the same commit with the same configuration against the stack, and the queue `release-1` was published from there at 08:52 UTC. +The snapshot tables of the lost preparation were removed when the stack was deleted. + +Workers used `--api-slots 10 --terminate-host`. +The first job started at 08:52 UTC and the last finished at 12:05 UTC; every job finished in its first attempt, so no trial was interrupted or superseded. +The longest job was S3FSCursor rows at xxlarge, three trials of about 56 minutes each. + +The entry's 367 measured cases, 68 of them in the Polars 2.0 environment, have 1,323 successful trials and no failed trial; 8 cases, Wrangler CSV with the nested shape, are unsupported. +The median Athena queue time of every case was at most 100 ms. +No workflow of this repository ran between 08:52 and 12:05 UTC; the daily database sweep had finished at 08:39 UTC. + +## Findings + +1. **`pandas.read_sql_query()` through the pandas and Polars drivers was faster than in the 4.1.0 run** (#1167, #1177, #254). + For 1 million flat rows, `awsathena+pandas` with `unload=true` took 7.02 seconds instead of 18.31, and `awsathena+polars` with `unload=true` 6.66 seconds instead of 11.28; for 10 million flat rows, 50.71 seconds instead of 162.60, and 48.71 seconds instead of 92.05. + Their client result times fell by 30 to 77% in every case of the two drivers, while `awsathena+arrow` through UNLOAD stayed within 3% of its 4.1.0 time (6.79 and 50.93 seconds). + Wrangler took 12.37 and 3.89 seconds for 1 million flat rows through CSV and UNLOAD, and 108.88 and 23.44 seconds for 10 million. +2. **Chunked reads of 100 million rows peaked at about the same memory as of 10 million rows** (#1162, #61). + With `chunksize=100000`, PandasCursor peaked at 462 MiB from CSV results and at 986 MiB (978 MiB with `dtype_backend="pyarrow"`) from UNLOAD results, PolarsCursor at 1,001 to 1,051 MiB with either Polars version, and Wrangler at 390 to 1,173 MiB; the same reads of 10 million rows peaked at 391 to 999 MiB. + PandasCursor took 699.88 seconds through CSV, against 1,233.06 seconds for Wrangler, and 34.86 to 37.95 seconds through UNLOAD, as did PolarsCursor and Wrangler (35.66 to 41.19 seconds); UNLOAD and CTAS wrote 8 files of up to 184 MiB. + S3FSCursor returned the 100 million rows in 3,386.53 seconds at 327 MiB, against 323 MiB for 10 million rows. + The one unchunked PandasCursor CSV read completed in 704.09 seconds at 49,824 MiB, 98.6% of the harness's memory limit of 50,506 MiB (`memory_fraction` 0.8 of the host's memory). +3. **Unchunked PolarsCursor CSV reads with Polars 1.44.2 peaked 25 to 45% higher than in the 4.1.0 run** (#1307). + At 10 million rows, the peak rose from 6,854 to 9,641 MiB (flat) and from 13,006 to 18,923 MiB (nested); at 1 million rows, from 1,004 to 1,259 MiB and from 1,561 to 2,105 MiB. + The library versions were the same as in the 4.1.0 run, and the client result times did not change. + Since #1169, PolarsCursor reads the CSV result file into `bytes` and passes them to `polars.read_csv()`; with Polars 1.x, that holds a copy of the file while it parses. + With Polars 2.0.0, the same reads peaked at 7,547 and 14,422 MiB at 10 million rows. +4. **Athena took longer to execute the flat CSV query at 10 million rows, so the total times of those cases grew while their client result times did not.** + The median `TotalExecutionTimeInMillis` of that query was 48.85 seconds, against 28.80 seconds in the 4.1.0 run; the other queries at xlarge took within 9% of their 4.1.0 times. + The native flat CSV reads of the DataFrame cursors and Wrangler at xlarge took 10 to 61% longer in total, while their client result times changed by +1 to -33%: for example, PolarsCursor with 10,000-row chunks took 54.75 seconds in total instead of 34.10, and 3.67 seconds of client result time instead of 5.48. +5. **CSV reads of PandasCursor, PolarsCursor, and ArrowCursor took less client result time than in the 4.1.0 run.** + At 10 million rows, PandasCursor's client result time fell from 44.43 to 34.30 seconds without chunks (flat), PolarsCursor's from 5.22 to 5.48 seconds to 3.67 to 3.82 seconds with chunks (flat), and ArrowCursor's from 16.84 to 10.95 seconds (nested). + The cause of these differences was not investigated. +6. **With the `temporal` workload, CSV reads and row output took several times the client result time of the flat workload** (#1287, #1308). + For 1 million rows from CSV results, Cursor took 304.91 seconds of client result time against 217.62 seconds for the flat workload, S3FSCursor 88.53 against 29.28 seconds, and the fetch methods of PandasCursor, ArrowCursor, and PolarsCursor 26.79, 37.85, and 33.95 seconds, against 1.96, 3.36, and 1.53 seconds from UNLOAD results, which hold no time zone or `time` columns. + PandasCursor's native CSV read took 24.04 seconds against 3.80 seconds for the flat workload, and Wrangler's 29.30 against 6.02 seconds. + At 10 million rows, PandasCursor's CSV read with 100,000-row chunks took 163.71 seconds of client result time, against 76.50 seconds without chunks; Wrangler took 264.93 and 154.22 seconds. + UNLOAD and CTAS reads, which add only `event_time` and `event_date`, took within 0.3 seconds of the flat workload's client result time at 1 million rows. +7. **Polars 2.0.0 measured like Polars 1.44.2 except for unchunked CSV reads.** + Across the 68 cases of both environments, median total times agreed within 11%, and client result times and peaks within 10%, with these exceptions, each given for Polars 2.0.0 against 1.44.2: + - Unchunked CSV reads peaked lower at large nested and at xlarge (7,547 against 9,641 MiB for 10 million flat rows), and higher at medium (327 against 260 MiB). + - Chunked CSV reads at xlarge took 8.37 to 8.64 seconds of client result time against 7.04 to 7.26 seconds (nested, 10,000 and 100,000-row chunks), and 3.35 to 3.43 against 3.76 to 3.82 seconds (flat, 100,000 and 1,000,000-row chunks). + - At 100 million rows, the chunked CSV read took 33.82 seconds of client result time against 41.04 seconds. +8. **Row output through GetQueryResults and S3FSCursor stayed within 7% of the 4.1.0 run.** + Cursor took 224.82 seconds for 1 million flat rows at arraysize 1000 (226.46 in the 4.1.0 run) and 1,013.56 seconds at arraysize 100 (1,062.61), DictCursor 216.81 seconds (230.44), and S3FSCursor 35.46 seconds (36.69), with the same peaks. + +## Limits + +- Only the synchronous API ran. +- xxlarge covered the flat shape only, without ArrowCursor, and with one trial of the unchunked PandasCursor read. +- The `temporal` workload has no 4.1.0 measurement; its comparisons use the flat workload of the same run, whose CSV results are smaller: Athena took 101.59 seconds to execute the temporal CSV query at 10 million rows, against 48.85 seconds for the flat one. + The local before-and-after measurements of the temporal conversion are in the pull requests that #1287 lists. +- Polars 2.0 ran the Polars families only; the other families ran with Polars 1.44.2 installed. +- Three or five trials per case make p95 values descriptive. +- Times include Athena execution and one-second polling; compare client result times for the client's share, as Athena's execution time of the same query changed between the runs (finding 4). +- Comparisons with the 4.1.0 run use the same instance type, snapshots' row counts, and selection, but different hosts, days, and snapshots of the same source partition. +- Peak RSS is sampled every 50 ms and can miss short peaks. +- Results from other instance types or architectures are not directly comparable. + +## Results + +Times are medians in seconds; memory is the peak sampled RSS of the trial process in MiB (2^20 bytes), including the harness's common imports. +"Total" is the end-to-end time from execute through consumption; "client result" starts when the client observes query completion. +Unless a table names Polars 2.0.0, PolarsCursor and `awsathena+polars` ran with Polars 1.44.2. + +### Athena execution time + +Median `TotalExecutionTimeInMillis` in seconds, from the PandasCursor (CSV and UNLOAD) and Wrangler (CTAS) cases without chunking, and at xxlarge with 100,000-row chunks. + +| Transport | Shape | medium (s) | large (s) | xlarge (s) | xlarge, 4.1.0 (s) | xxlarge (s) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | +| CSV | flat | 1.27 | 5.54 | 48.85 | 28.80 | 335.38 | +| CSV | nested | | 11.32 | 208.19 | 219.95 | | +| CSV | temporal | | 7.99 | 101.59 | | | +| CTAS | flat | 1.23 | 3.02 | 15.98 | 17.51 | 21.35 | +| CTAS | nested | | 4.47 | 32.19 | 33.22 | | +| CTAS | temporal | | 3.41 | 18.79 | | | +| UNLOAD | flat | 1.54 | 2.44 | 16.35 | 16.91 | 20.52 | +| UNLOAD | nested | | 4.54 | 31.06 | 30.20 | | +| UNLOAD | temporal | | 2.73 | 20.90 | | | + +The median Athena queue time of every case was at most 100 ms. + +### Native DataFrame output by chunksize + +Synchronous API, arraysize 1000; ArrowCursor has no chunksize. +Wrangler CSV cannot read nested results; those cases are unsupported. + +Median total time (s): + +| Scale | Library | Transport | Shape | No chunks | 10,000 | 100,000 | 1,000,000 | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | +| medium | PandasCursor | CSV | flat | 2.86 | | | | +| medium | PandasCursor | UNLOAD | flat | 2.58 | | | | +| medium | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 2.56 | | | | +| medium | ArrowCursor | CSV | flat | 2.51 | | | | +| medium | ArrowCursor | UNLOAD | flat | 2.39 | | | | +| medium | PolarsCursor | CSV | flat | 2.43 | | | | +| medium | PolarsCursor | UNLOAD | flat | 1.42 | | | | +| medium | Wrangler | CSV | flat | 2.97 | | | | +| medium | Wrangler | CTAS | flat | 2.83 | | | | +| medium | Wrangler | UNLOAD | flat | 2.48 | | | | +| large | PandasCursor | CSV | flat | 9.99 | 9.94 | 9.99 | 10.01 | +| large | PandasCursor | CSV | nested | 19.06 | 18.29 | 17.98 | 19.02 | +| large | PandasCursor | UNLOAD | flat | 3.69 | 3.71 | 3.42 | 3.39 | +| large | PandasCursor | UNLOAD | nested | 10.75 | 10.73 | 11.78 | 10.70 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 3.67 | 3.49 | 3.38 | 3.41 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 5.98 | 5.76 | 5.70 | 5.68 | +| large | ArrowCursor | CSV | flat | 8.24 | | | | +| large | ArrowCursor | CSV | nested | 13.77 | | | | +| large | ArrowCursor | UNLOAD | flat | 3.54 | | | | +| large | ArrowCursor | UNLOAD | nested | 6.84 | | | | +| large | PolarsCursor | CSV | flat | 7.18 | 7.10 | 7.07 | 7.14 | +| large | PolarsCursor | CSV | nested | 14.02 | 13.58 | 13.52 | 13.63 | +| large | PolarsCursor | UNLOAD | flat | 3.58 | 3.56 | 3.51 | 3.51 | +| large | PolarsCursor | UNLOAD | nested | 5.98 | 5.97 | 5.92 | 5.94 | +| large | Wrangler | CSV | flat | 12.37 | 14.11 | 13.86 | 13.77 | +| large | Wrangler | CSV | nested | unsupported | unsupported | unsupported | unsupported | +| large | Wrangler | CTAS | flat | 4.21 | 4.16 | 4.00 | 3.91 | +| large | Wrangler | CTAS | nested | 11.31 | 11.29 | 11.38 | 11.25 | +| large | Wrangler | UNLOAD | flat | 3.89 | 3.84 | 3.68 | 3.65 | +| large | Wrangler | UNLOAD | nested | 10.94 | 11.09 | 11.13 | 11.07 | +| xlarge | PandasCursor | CSV | flat | 84.17 | 85.96 | 81.02 | 85.08 | +| xlarge | PandasCursor | CSV | nested | 280.28 | 269.80 | 278.67 | 273.57 | +| xlarge | PandasCursor | UNLOAD | flat | 19.79 | 20.86 | 17.89 | 17.86 | +| xlarge | PandasCursor | UNLOAD | nested | 87.99 | 92.20 | 89.66 | 90.80 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 17.83 | 18.71 | 17.73 | 18.48 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 34.22 | 37.05 | 38.10 | 35.04 | +| xlarge | ArrowCursor | CSV | flat | 55.22 | | | | +| xlarge | ArrowCursor | CSV | nested | 211.33 | | | | +| xlarge | ArrowCursor | UNLOAD | flat | 18.16 | | | | +| xlarge | ArrowCursor | UNLOAD | nested | 36.38 | | | | +| xlarge | PolarsCursor | CSV | flat | 58.16 | 54.75 | 54.72 | 53.71 | +| xlarge | PolarsCursor | CSV | nested | 214.34 | 206.20 | 211.27 | 210.75 | +| xlarge | PolarsCursor | UNLOAD | flat | 18.51 | 19.26 | 17.92 | 18.23 | +| xlarge | PolarsCursor | UNLOAD | nested | 33.77 | 36.99 | 36.51 | 37.50 | +| xlarge | Wrangler | CSV | flat | 108.88 | 131.72 | 128.92 | 125.96 | +| xlarge | Wrangler | CSV | nested | unsupported | unsupported | unsupported | unsupported | +| xlarge | Wrangler | CTAS | flat | 20.64 | 20.42 | 17.68 | 18.47 | +| xlarge | Wrangler | CTAS | nested | 93.19 | 90.55 | 89.15 | 92.13 | +| xlarge | Wrangler | UNLOAD | flat | 23.44 | 20.00 | 18.78 | 18.22 | +| xlarge | Wrangler | UNLOAD | nested | 88.33 | 90.95 | 89.79 | 90.70 | + +Median client result time (s): + +| Scale | Library | Transport | Shape | No chunks | 10,000 | 100,000 | 1,000,000 | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | +| medium | PandasCursor | CSV | flat | 0.74 | | | | +| medium | PandasCursor | UNLOAD | flat | 0.46 | | | | +| medium | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 0.43 | | | | +| medium | ArrowCursor | CSV | flat | 0.37 | | | | +| medium | ArrowCursor | UNLOAD | flat | 0.27 | | | | +| medium | PolarsCursor | CSV | flat | 0.31 | | | | +| medium | PolarsCursor | UNLOAD | flat | 0.30 | | | | +| medium | Wrangler | CSV | flat | 0.73 | | | | +| medium | Wrangler | CTAS | flat | 0.60 | | | | +| medium | Wrangler | UNLOAD | flat | 0.25 | | | | +| large | PandasCursor | CSV | flat | 3.80 | 3.75 | 3.78 | 3.81 | +| large | PandasCursor | CSV | nested | 6.96 | 6.93 | 6.71 | 6.82 | +| large | PandasCursor | UNLOAD | flat | 0.54 | 0.56 | 0.28 | 0.25 | +| large | PandasCursor | UNLOAD | nested | 5.57 | 5.50 | 5.58 | 5.51 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 0.53 | 0.33 | 0.25 | 0.24 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 0.79 | 0.59 | 0.52 | 0.50 | +| large | ArrowCursor | CSV | flat | 1.05 | | | | +| large | ArrowCursor | CSV | nested | 1.56 | | | | +| large | ArrowCursor | UNLOAD | flat | 0.38 | | | | +| large | ArrowCursor | UNLOAD | nested | 0.66 | | | | +| large | PolarsCursor | CSV | flat | 0.99 | 0.90 | 0.88 | 0.89 | +| large | PolarsCursor | CSV | nested | 1.74 | 1.25 | 1.25 | 1.32 | +| large | PolarsCursor | UNLOAD | flat | 0.43 | 0.40 | 0.35 | 0.35 | +| large | PolarsCursor | UNLOAD | nested | 0.79 | 0.78 | 0.75 | 0.75 | +| large | Wrangler | CSV | flat | 6.02 | 7.76 | 7.51 | 7.38 | +| large | Wrangler | CSV | nested | | | | | +| large | Wrangler | CTAS | flat | 0.94 | 0.83 | 0.72 | 0.64 | +| large | Wrangler | CTAS | nested | 6.01 | 5.94 | 6.05 | 6.11 | +| large | Wrangler | UNLOAD | flat | 0.62 | 0.57 | 0.40 | 0.38 | +| large | Wrangler | UNLOAD | nested | 5.60 | 5.76 | 5.80 | 5.77 | +| xlarge | PandasCursor | CSV | flat | 34.30 | 35.01 | 35.04 | 33.66 | +| xlarge | PandasCursor | CSV | nested | 74.40 | 68.33 | 77.01 | 76.13 | +| xlarge | PandasCursor | UNLOAD | flat | 2.40 | 4.49 | 1.75 | 1.48 | +| xlarge | PandasCursor | UNLOAD | nested | 55.57 | 60.26 | 57.01 | 58.89 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 1.47 | 2.11 | 1.28 | 1.13 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 4.65 | 5.53 | 4.53 | 4.47 | +| xlarge | ArrowCursor | CSV | flat | 5.26 | | | | +| xlarge | ArrowCursor | CSV | nested | 10.95 | | | | +| xlarge | ArrowCursor | UNLOAD | flat | 1.77 | | | | +| xlarge | ArrowCursor | UNLOAD | nested | 4.75 | | | | +| xlarge | PolarsCursor | CSV | flat | 7.23 | 3.67 | 3.76 | 3.82 | +| xlarge | PolarsCursor | CSV | nested | 14.58 | 7.26 | 7.04 | 7.39 | +| xlarge | PolarsCursor | UNLOAD | flat | 1.19 | 1.88 | 1.55 | 1.53 | +| xlarge | PolarsCursor | UNLOAD | nested | 3.30 | 6.41 | 5.87 | 5.90 | +| xlarge | Wrangler | CSV | flat | 58.25 | 81.18 | 78.76 | 75.88 | +| xlarge | Wrangler | CSV | nested | | | | | +| xlarge | Wrangler | CTAS | flat | 4.01 | 3.58 | 2.00 | 1.75 | +| xlarge | Wrangler | CTAS | nested | 59.99 | 59.09 | 57.79 | 60.06 | +| xlarge | Wrangler | UNLOAD | flat | 3.72 | 3.28 | 2.00 | 1.47 | +| xlarge | Wrangler | UNLOAD | nested | 57.47 | 58.80 | 57.28 | 59.38 | + +Peak RSS (MiB): + +| Scale | Library | Transport | Shape | No chunks | 10,000 | 100,000 | 1,000,000 | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | +| medium | PandasCursor | CSV | flat | 305 | | | | +| medium | PandasCursor | UNLOAD | flat | 315 | | | | +| medium | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 321 | | | | +| medium | ArrowCursor | CSV | flat | 330 | | | | +| medium | ArrowCursor | UNLOAD | flat | 320 | | | | +| medium | PolarsCursor | CSV | flat | 260 | | | | +| medium | PolarsCursor | UNLOAD | flat | 229 | | | | +| medium | Wrangler | CSV | flat | 307 | | | | +| medium | Wrangler | CTAS | flat | 280 | | | | +| medium | Wrangler | UNLOAD | flat | 268 | | | | +| large | PandasCursor | CSV | flat | 717 | 351 | 447 | 717 | +| large | PandasCursor | CSV | nested | 1,050 | 391 | 505 | 1,063 | +| large | PandasCursor | UNLOAD | flat | 808 | 352 | 444 | 824 | +| large | PandasCursor | UNLOAD | nested | 2,114 | 372 | 781 | 2,147 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 759 | 333 | 440 | 794 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 875 | 357 | 513 | 890 | +| large | ArrowCursor | CSV | flat | 1,190 | | | | +| large | ArrowCursor | CSV | nested | 1,725 | | | | +| large | ArrowCursor | UNLOAD | flat | 793 | | | | +| large | ArrowCursor | UNLOAD | nested | 855 | | | | +| large | PolarsCursor | CSV | flat | 1,259 | 597 | 681 | 938 | +| large | PolarsCursor | CSV | nested | 2,105 | 862 | 1,001 | 1,440 | +| large | PolarsCursor | UNLOAD | flat | 490 | 357 | 465 | 948 | +| large | PolarsCursor | UNLOAD | nested | 743 | 399 | 689 | 1,339 | +| large | Wrangler | CSV | flat | 958 | 285 | 386 | 717 | +| large | Wrangler | CSV | nested | | | | | +| large | Wrangler | CTAS | flat | 737 | 357 | 460 | 849 | +| large | Wrangler | CTAS | nested | 2,059 | 369 | 743 | 2,175 | +| large | Wrangler | UNLOAD | flat | 701 | 344 | 462 | 806 | +| large | Wrangler | UNLOAD | nested | 2,080 | 365 | 744 | 2,161 | +| xlarge | PandasCursor | CSV | flat | 5,129 | 362 | 461 | 830 | +| xlarge | PandasCursor | CSV | nested | 7,849 | 389 | 520 | 1,177 | +| xlarge | PandasCursor | UNLOAD | flat | 5,419 | 635 | 771 | 2,111 | +| xlarge | PandasCursor | UNLOAD | nested | 19,498 | 601 | 1,087 | 5,195 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | flat | 5,017 | 664 | 740 | 1,932 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | nested | 5,820 | 577 | 793 | 2,460 | +| xlarge | ArrowCursor | CSV | flat | 4,143 | | | | +| xlarge | ArrowCursor | CSV | nested | 7,405 | | | | +| xlarge | ArrowCursor | UNLOAD | flat | 4,892 | | | | +| xlarge | ArrowCursor | UNLOAD | nested | 5,890 | | | | +| xlarge | PolarsCursor | CSV | flat | 9,641 | 904 | 978 | 1,806 | +| xlarge | PolarsCursor | CSV | nested | 18,923 | 917 | 1,115 | 2,793 | +| xlarge | PolarsCursor | UNLOAD | flat | 3,765 | 665 | 817 | 2,190 | +| xlarge | PolarsCursor | UNLOAD | nested | 6,135 | 633 | 949 | 3,056 | +| xlarge | Wrangler | CSV | flat | 7,655 | 280 | 391 | 1,195 | +| xlarge | Wrangler | CSV | nested | | | | | +| xlarge | Wrangler | CTAS | flat | 5,051 | 686 | 845 | 2,287 | +| xlarge | Wrangler | CTAS | nested | 19,049 | 710 | 1,117 | 4,815 | +| xlarge | Wrangler | UNLOAD | flat | 5,013 | 664 | 820 | 2,119 | +| xlarge | Wrangler | UNLOAD | nested | 18,886 | 700 | 1,100 | 4,759 | + +### Native output compared with 4.1.0 + +Cases that both runs measured, at large and xlarge, where the median total time or the peak RSS changed by at least 15%. +Total time includes Athena's execution, which varied between the runs; compare the client result times for the client's share. + +| Scale | Case | Total 4.1.0 (s) | Total (s) | Client result 4.1.0 (s) | Client result (s) | Peak RSS 4.1.0 (MiB) | Peak RSS (MiB) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| large | `single-arrow-sync-csv-native-a1000-c0-flat-n1` | 7.16 | 8.24 | 1.02 | 1.05 | 1,191 | 1,190 | +| large | `single-arrow-sync-unload-native-a1000-c0-nested-n1` | 5.91 | 6.84 | 0.74 | 0.66 | 887 | 855 | +| large | `single-pandas-sync-csv-native-a1000-c100000-nested-n1` | 21.20 | 17.98 | 8.91 | 6.71 | 505 | 505 | +| large | `single-pandas-sync-unload-native-a1000-c0-pyarrow-flat-n1` | 4.75 | 3.67 | 0.61 | 0.53 | 776 | 759 | +| large | `single-polars-sync-csv-native-a1000-c0-flat-n1` | 6.12 | 7.18 | 0.94 | 0.99 | 1,004 | 1,259 | +| large | `single-polars-sync-csv-native-a1000-c0-nested-n1` | 13.97 | 14.02 | 1.66 | 1.74 | 1,561 | 2,105 | +| large | `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 4.63 | 3.51 | 0.45 | 0.35 | 958 | 948 | +| large | `single-wrangler-sync-ctas-native-a1000-c10000-flat-n1` | 5.26 | 4.16 | 0.94 | 0.83 | 362 | 357 | +| large | `single-wrangler-sync-ctas-native-a1000-c100000-flat-n1` | 5.05 | 4.00 | 0.75 | 0.72 | 486 | 460 | +| xlarge | `single-arrow-sync-csv-native-a1000-c0-flat-n1` | 36.00 | 55.22 | 6.47 | 5.26 | 4,125 | 4,143 | +| xlarge | `single-pandas-sync-csv-native-a1000-c10000-flat-n1` | 72.59 | 85.96 | 42.91 | 35.01 | 363 | 362 | +| xlarge | `single-pandas-sync-csv-native-a1000-c100000-flat-n1` | 68.79 | 81.02 | 39.14 | 35.04 | 449 | 461 | +| xlarge | `single-pandas-sync-csv-native-a1000-c1000000-flat-n1` | 66.07 | 85.08 | 36.46 | 33.66 | 822 | 830 | +| xlarge | `single-pandas-sync-csv-native-a1000-c1000000-nested-n1` | 232.74 | 273.57 | 93.16 | 76.13 | 1,159 | 1,177 | +| xlarge | `single-polars-sync-csv-native-a1000-c0-flat-n1` | 38.85 | 58.16 | 7.16 | 7.23 | 6,854 | 9,641 | +| xlarge | `single-polars-sync-csv-native-a1000-c0-nested-n1` | 232.95 | 214.34 | 13.49 | 14.58 | 13,006 | 18,923 | +| xlarge | `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 34.10 | 54.75 | 5.48 | 3.67 | 874 | 904 | +| xlarge | `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 145.44 | 206.20 | 7.64 | 7.26 | 904 | 917 | +| xlarge | `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 34.48 | 54.72 | 5.22 | 3.76 | 983 | 978 | +| xlarge | `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 34.63 | 53.71 | 5.27 | 3.82 | 1,764 | 1,806 | +| xlarge | `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 148.93 | 210.75 | 8.93 | 7.39 | 2,700 | 2,793 | + +### `pandas.read_sql_query()` through SQLAlchemy + +`pandas.read_sql_query()` with an engine of each driver, synchronous, arraysize 1000, without chunks and with `chunksize=100000`, next to Wrangler's native reads. +Drivers marked UNLOAD connect with `unload=true` in the cursor options. + +| Scale | Shape | Reader | Transport | Total (s) | Total 4.1.0 (s) | Peak RSS (MiB) | Total, 100,000-row chunks (s) | Total, chunks, 4.1.0 (s) | Peak RSS, chunks (MiB) | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| large | flat | `awsathena+rest` | CSV | 226.99 | 230.17 | 1,648 | | | | +| large | flat | `awsathena+s3fs` | CSV | 38.22 | 39.18 | 1,727 | 38.08 | 39.18 | 549 | +| large | flat | `awsathena+pandas` | CSV | 14.81 | 26.45 | 1,754 | 14.46 | 26.28 | 667 | +| large | flat | `awsathena+pandas` | UNLOAD | 7.02 | 18.31 | 1,853 | 6.57 | 18.06 | 691 | +| large | flat | `awsathena+polars` | CSV | 10.31 | 14.66 | 1,888 | 10.04 | 14.25 | 941 | +| large | flat | `awsathena+polars` | UNLOAD | 6.66 | 11.28 | 1,749 | 6.46 | 11.29 | 725 | +| large | flat | `awsathena+arrow` | CSV | 10.51 | 10.57 | 2,286 | 10.39 | 10.64 | 1,350 | +| large | flat | `awsathena+arrow` | UNLOAD | 6.79 | 6.97 | 1,902 | 6.60 | 6.79 | 982 | +| large | flat | Wrangler (native) | CSV | 12.37 | 12.46 | 958 | 13.86 | 14.88 | 386 | +| large | flat | Wrangler (native) | CTAS | 4.21 | 4.39 | 737 | 4.00 | 5.05 | 460 | +| large | flat | Wrangler (native) | UNLOAD | 3.89 | 3.85 | 701 | 3.68 | 3.75 | 462 | +| large | nested | `awsathena+rest` | CSV | 294.11 | 301.81 | 4,247 | | | | +| large | nested | `awsathena+s3fs` | CSV | 142.01 | 145.82 | 4,297 | 142.51 | 145.09 | 1,094 | +| large | nested | `awsathena+pandas` | CSV | 21.41 | 30.55 | 2,364 | 22.00 | 29.53 | 837 | +| large | nested | `awsathena+pandas` | UNLOAD | 13.11 | 19.28 | 2,760 | 12.85 | 19.71 | 933 | +| large | nested | `awsathena+polars` | CSV | 16.61 | 19.66 | 2,488 | 14.70 | 19.92 | 1,364 | +| large | nested | `awsathena+polars` | UNLOAD | 13.98 | 20.51 | 4,295 | 14.55 | 19.25 | 1,371 | +| large | nested | `awsathena+arrow` | CSV | 16.33 | 16.93 | 2,666 | 15.47 | 16.95 | 1,716 | +| large | nested | `awsathena+arrow` | UNLOAD | 12.17 | 13.74 | 3,457 | 12.87 | 13.51 | 1,351 | +| large | nested | Wrangler (native) | CTAS | 11.31 | 12.44 | 2,059 | 11.38 | 11.53 | 743 | +| large | nested | Wrangler (native) | UNLOAD | 10.94 | 11.05 | 2,080 | 11.13 | 11.22 | 744 | +| xlarge | flat | `awsathena+s3fs` | CSV | 373.65 | 355.04 | 14,825 | 370.76 | 350.58 | 584 | +| xlarge | flat | `awsathena+pandas` | CSV | 132.45 | 230.84 | 16,039 | 135.51 | 221.65 | 698 | +| xlarge | flat | `awsathena+pandas` | UNLOAD | 50.71 | 162.60 | 16,096 | 49.92 | 158.08 | 1,032 | +| xlarge | flat | `awsathena+polars` | CSV | 88.39 | 110.55 | 14,978 | 79.40 | 101.67 | 1,281 | +| xlarge | flat | `awsathena+polars` | UNLOAD | 48.71 | 92.05 | 14,849 | 47.77 | 89.79 | 1,054 | +| xlarge | flat | `awsathena+arrow` | CSV | 88.41 | 68.06 | 15,374 | 84.68 | 66.79 | 4,177 | +| xlarge | flat | `awsathena+arrow` | UNLOAD | 50.93 | 50.09 | 16,264 | 48.79 | 48.93 | 5,078 | +| xlarge | flat | Wrangler (native) | CSV | 108.88 | 95.29 | 7,655 | 128.92 | 116.24 | 391 | +| xlarge | flat | Wrangler (native) | CTAS | 20.64 | 21.65 | 5,051 | 17.68 | 18.79 | 845 | +| xlarge | flat | Wrangler (native) | UNLOAD | 23.44 | 21.40 | 5,013 | 18.78 | 18.53 | 820 | +| xlarge | nested | `awsathena+pandas` | CSV | | | | 291.25 | 350.61 | 873 | +| xlarge | nested | `awsathena+pandas` | UNLOAD | | | | 113.90 | 180.17 | 1,270 | +| xlarge | nested | `awsathena+polars` | CSV | | | | 229.89 | 198.03 | 1,577 | +| xlarge | nested | `awsathena+polars` | UNLOAD | | | | 133.80 | 177.80 | 1,668 | +| xlarge | nested | `awsathena+arrow` | CSV | | | | 239.56 | 190.02 | 7,420 | +| xlarge | nested | `awsathena+arrow` | UNLOAD | | | | 115.69 | 120.05 | 5,928 | +| xlarge | nested | Wrangler (native) | CTAS | 93.19 | 94.75 | 19,049 | 89.15 | 91.17 | 1,117 | +| xlarge | nested | Wrangler (native) | UNLOAD | 88.33 | 90.80 | 18,886 | 89.79 | 89.21 | 1,100 | + +### The json workload + +The flat projection with `CAST(details AS JSON)`, read from CSV results, synchronous. + +| Scale | Library | Chunks | Total (s) | Total 4.1.0 (s) | Client result (s) | Peak RSS (MiB) | +| --- | --- | --- | ---: | ---: | ---: | ---: | +| large | PandasCursor | none | 24.47 | 25.67 | 14.24 | 4,127 | +| large | PandasCursor | 100,000 | 23.99 | 24.84 | 13.74 | 747 | +| large | Wrangler | none | 23.10 | 24.35 | 12.61 | 1,386 | +| large | Wrangler | 100,000 | 26.09 | 28.59 | 15.62 | 390 | +| xlarge | PandasCursor | none | 317.18 | 337.18 | 166.60 | 40,836 | +| xlarge | PandasCursor | 100,000 | 296.04 | 322.04 | 147.12 | 769 | +| xlarge | Wrangler | none | 284.33 | 296.16 | 132.63 | 12,778 | +| xlarge | Wrangler | 100,000 | 341.57 | 370.57 | 193.21 | 380 | + +### Row output + +Synchronous API without chunking; Cursor and DictCursor page through GetQueryResults, and S3FSCursor reads the CSV result file. + +| Scale | Cursor | Shape | Arraysize | Trials | Total (s) | Total 4.1.0 (s) | Client result (s) | Peak RSS (MiB) | Peak RSS 4.1.0 (MiB) | +| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| medium | Cursor | flat | 1,000 | 3 | 23.62 | 23.93 | 21.50 | 189 | 189 | +| medium | S3FSCursor | flat | 1,000 | 3 | 5.28 | 5.53 | 3.16 | 213 | 215 | +| large | Cursor | flat | 100 | 3 | 1,013.56 | 1,062.61 | 1,007.31 | 181 | 181 | +| large | Cursor | flat | 1,000 | 5 | 224.82 | 226.46 | 217.62 | 190 | 190 | +| large | Cursor | nested | 100 | 3 | 1,089.32 | 1,164.03 | 1,077.00 | 182 | 181 | +| large | Cursor | nested | 1,000 | 5 | 280.79 | 291.23 | 268.52 | 194 | 194 | +| large | DictCursor | flat | 1,000 | 3 | 216.81 | 230.44 | 210.58 | 190 | 190 | +| large | S3FSCursor | flat | 1,000 | 5 | 35.46 | 36.69 | 29.28 | 320 | 324 | +| large | S3FSCursor | nested | 1,000 | 5 | 135.14 | 140.69 | 122.70 | 328 | 325 | +| xlarge | S3FSCursor | flat | 1,000 | 3 | 348.66 | 328.50 | 296.61 | 323 | 321 | +| xxlarge | S3FSCursor | flat | 1,000 | 3 | 3,386.53 | | 3,060.24 | 327 | | + +### The temporal workload + +The flat projection with the columns of the `temporal` workload, described in the [README](../README.md#input-data), synchronous, arraysize 1000. +Its UNLOAD and CTAS reads add only `event_time` and `event_date`. +The flat columns give the same case of the flat workload in this run. +"Rows" are the fetch methods of the DataFrame cursors. + +| Scale | Reader | Transport | Chunks | Total (s) | Client result (s) | Peak RSS (MiB) | Client result, flat (s) | Peak RSS, flat (MiB) | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | +| large | Cursor | CSV | none | 313.12 | 304.91 | 195 | 217.62 | 190 | +| large | DictCursor | CSV | none | 316.32 | 307.04 | 195 | 210.58 | 190 | +| large | S3FSCursor | CSV | none | 96.76 | 88.53 | 314 | 29.28 | 320 | +| large | PandasCursor rows | CSV | none | 35.03 | 26.79 | 1,052 | | | +| large | PandasCursor rows | UNLOAD | none | 6.10 | 1.96 | 795 | | | +| large | ArrowCursor rows | CSV | none | 46.12 | 37.85 | 1,450 | | | +| large | ArrowCursor rows | UNLOAD | none | 7.56 | 3.36 | 827 | | | +| large | PolarsCursor rows | CSV | none | 42.25 | 33.95 | 1,703 | | | +| large | PolarsCursor rows | UNLOAD | none | 5.74 | 1.53 | 521 | | | +| large | PandasCursor | CSV | none | 32.25 | 24.04 | 1,051 | 3.80 | 717 | +| large | PandasCursor | CSV | 10,000 | 27.71 | 19.49 | 375 | 3.75 | 351 | +| large | PandasCursor | CSV | 100,000 | 26.83 | 18.60 | 468 | 3.78 | 447 | +| large | PandasCursor | CSV | 1,000,000 | 31.53 | 23.30 | 1,074 | 3.81 | 717 | +| large | PandasCursor | UNLOAD | none | 3.81 | 0.65 | 835 | 0.54 | 808 | +| large | PandasCursor | UNLOAD | 10,000 | 3.98 | 0.82 | 341 | 0.56 | 352 | +| large | PandasCursor | UNLOAD | 100,000 | 3.53 | 0.39 | 468 | 0.28 | 444 | +| large | PandasCursor | UNLOAD | 1,000,000 | 3.48 | 0.35 | 876 | 0.25 | 824 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | none | 3.70 | 0.54 | 801 | 0.53 | 759 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | 10,000 | 3.51 | 0.36 | 341 | 0.33 | 333 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | 100,000 | 3.40 | 0.26 | 434 | 0.25 | 440 | +| large | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | 1,000,000 | 3.40 | 0.26 | 776 | 0.24 | 794 | +| large | ArrowCursor | CSV | none | 9.72 | 1.26 | 1,477 | 1.05 | 1,190 | +| large | ArrowCursor | UNLOAD | none | 4.58 | 0.41 | 768 | 0.38 | 793 | +| large | PolarsCursor | CSV | none | 9.71 | 1.50 | 1,717 | 0.99 | 1,259 | +| large | PolarsCursor | CSV | 10,000 | 9.45 | 1.21 | 762 | 0.90 | 597 | +| large | PolarsCursor | CSV | 100,000 | 10.41 | 1.17 | 889 | 0.88 | 681 | +| large | PolarsCursor | CSV | 1,000,000 | 9.45 | 1.23 | 1,270 | 0.89 | 938 | +| large | PolarsCursor | UNLOAD | none | 3.69 | 0.47 | 512 | 0.43 | 490 | +| large | PolarsCursor | UNLOAD | 10,000 | 3.57 | 0.42 | 362 | 0.40 | 357 | +| large | PolarsCursor | UNLOAD | 100,000 | 3.52 | 0.37 | 471 | 0.35 | 465 | +| large | PolarsCursor | UNLOAD | 1,000,000 | 3.52 | 0.37 | 967 | 0.35 | 948 | +| large | Wrangler | CSV | none | 37.71 | 29.30 | 1,695 | 6.02 | 958 | +| large | Wrangler | CSV | 10,000 | 37.06 | 28.65 | 278 | 7.76 | 285 | +| large | Wrangler | CSV | 100,000 | 36.04 | 27.62 | 412 | 7.51 | 386 | +| large | Wrangler | CSV | 1,000,000 | 40.34 | 31.68 | 1,298 | 7.38 | 717 | +| large | Wrangler | CTAS | none | 5.19 | 0.92 | 746 | 0.94 | 737 | +| large | Wrangler | CTAS | 10,000 | 4.26 | 0.98 | 346 | 0.83 | 357 | +| large | Wrangler | CTAS | 100,000 | 5.22 | 0.77 | 475 | 0.72 | 460 | +| large | Wrangler | CTAS | 1,000,000 | 4.96 | 0.69 | 891 | 0.64 | 849 | +| large | Wrangler | UNLOAD | none | 4.93 | 0.63 | 743 | 0.62 | 701 | +| large | Wrangler | UNLOAD | 10,000 | 4.97 | 0.67 | 353 | 0.57 | 344 | +| large | Wrangler | UNLOAD | 100,000 | 3.72 | 0.46 | 483 | 0.40 | 462 | +| large | Wrangler | UNLOAD | 1,000,000 | 3.70 | 0.43 | 870 | 0.38 | 806 | +| large | `awsathena+s3fs` | CSV | none | 106.04 | 97.73 | 2,126 | 31.88 | 1,727 | +| large | `awsathena+s3fs` | CSV | 100,000 | 106.05 | 97.13 | 659 | 31.84 | 549 | +| large | `awsathena+pandas` | CSV | none | 40.34 | 31.94 | 2,181 | 8.61 | 1,754 | +| large | `awsathena+pandas` | CSV | 100,000 | 33.29 | 25.06 | 719 | 8.24 | 667 | +| large | `awsathena+pandas` | UNLOAD | none | 8.92 | 4.76 | 2,004 | 3.79 | 1,853 | +| large | `awsathena+pandas` | UNLOAD | 100,000 | 9.45 | 4.32 | 736 | 3.43 | 691 | +| large | `awsathena+polars` | CSV | none | 49.95 | 41.71 | 2,301 | 4.09 | 1,888 | +| large | `awsathena+polars` | CSV | 100,000 | 49.93 | 41.69 | 1,041 | 3.75 | 941 | +| large | `awsathena+polars` | UNLOAD | none | 8.51 | 4.41 | 1,915 | 3.51 | 1,749 | +| large | `awsathena+polars` | UNLOAD | 100,000 | 8.33 | 4.20 | 746 | 3.32 | 725 | +| large | `awsathena+arrow` | CSV | none | 54.06 | 45.80 | 2,574 | 4.31 | 2,286 | +| large | `awsathena+arrow` | CSV | 100,000 | 54.25 | 45.99 | 1,534 | 4.19 | 1,350 | +| large | `awsathena+arrow` | UNLOAD | none | 9.39 | 6.24 | 1,977 | 3.60 | 1,902 | +| large | `awsathena+arrow` | UNLOAD | 100,000 | 9.35 | 6.18 | 987 | 3.44 | 982 | +| xlarge | PandasCursor | CSV | none | 178.25 | 76.50 | 8,840 | 34.30 | 5,129 | +| xlarge | PandasCursor | CSV | 100,000 | 262.39 | 163.71 | 476 | 35.04 | 461 | +| xlarge | PandasCursor | UNLOAD | none | 23.90 | 2.78 | 5,840 | 2.40 | 5,419 | +| xlarge | PandasCursor | UNLOAD | 100,000 | 22.30 | 2.84 | 784 | 1.75 | 771 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | none | 21.47 | 1.88 | 5,076 | 1.47 | 5,017 | +| xlarge | PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | 100,000 | 20.92 | 1.52 | 769 | 1.28 | 740 | +| xlarge | ArrowCursor | CSV | none | 106.78 | 8.04 | 5,637 | 5.26 | 4,143 | +| xlarge | ArrowCursor | UNLOAD | none | 22.29 | 1.85 | 5,035 | 1.77 | 4,892 | +| xlarge | PolarsCursor | CSV | none | 112.07 | 12.36 | 14,215 | 7.23 | 9,641 | +| xlarge | PolarsCursor | CSV | 100,000 | 102.98 | 6.03 | 1,079 | 3.76 | 978 | +| xlarge | PolarsCursor | UNLOAD | none | 20.62 | 1.21 | 3,889 | 1.19 | 3,765 | +| xlarge | PolarsCursor | UNLOAD | 100,000 | 21.20 | 1.81 | 796 | 1.55 | 817 | +| xlarge | Wrangler | CSV | none | 259.53 | 154.22 | 15,225 | 58.25 | 7,655 | +| xlarge | Wrangler | CSV | 100,000 | 367.28 | 264.93 | 418 | 78.76 | 391 | +| xlarge | Wrangler | CTAS | none | 23.81 | 4.16 | 5,474 | 4.01 | 5,051 | +| xlarge | Wrangler | CTAS | 100,000 | 22.03 | 2.35 | 861 | 2.00 | 845 | +| xlarge | Wrangler | UNLOAD | none | 24.56 | 3.84 | 5,377 | 3.72 | 5,013 | +| xlarge | Wrangler | UNLOAD | 100,000 | 25.15 | 2.18 | 856 | 2.00 | 820 | + +### 100 million rows + +The flat workload at xxlarge, synchronous, arraysize 1000. +The 10 million rows column gives the same case at xlarge in this run. + +| Reader | Transport | Chunks | Environment | Trials | Total (s) | Client result (s) | Peak RSS (MiB) | Peak RSS, 10 million rows (MiB) | Output files | Largest file (MiB) | +| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| PandasCursor | CSV | none | | 1 | 704.09 | 368.35 | 49,824 | 5,129 | | | +| PandasCursor | CSV | 100,000 | | 3 | 699.88 | 383.42 | 462 | 461 | | | +| PandasCursor | UNLOAD | 100,000 | | 3 | 37.95 | 16.53 | 986 | 771 | 8 | 184 | +| PandasCursor (`dtype_backend="pyarrow"`) | UNLOAD | 100,000 | | 3 | 34.86 | 13.38 | 978 | 740 | 8 | 183 | +| PolarsCursor | CSV | 100,000 | Polars 1.44.2 | 3 | 373.57 | 41.04 | 1,001 | 978 | | | +| PolarsCursor | CSV | 100,000 | Polars 2.0.0 | 3 | 364.23 | 33.82 | 1,016 | 999 | | | +| PolarsCursor | UNLOAD | 100,000 | Polars 1.44.2 | 3 | 35.66 | 14.43 | 1,051 | 817 | 8 | 184 | +| PolarsCursor | UNLOAD | 100,000 | Polars 2.0.0 | 3 | 35.89 | 14.45 | 1,045 | 790 | 8 | 179 | +| Wrangler | CSV | 100,000 | | 3 | 1,233.06 | 913.82 | 390 | 391 | | | +| Wrangler | CTAS | 100,000 | | 3 | 41.19 | 19.27 | 1,173 | 845 | 8 | 175 | +| Wrangler | UNLOAD | 100,000 | | 3 | 39.66 | 17.89 | 1,134 | 820 | 8 | 183 | +| S3FSCursor rows, arraysize 1,000 | CSV | none | | 3 | 3,386.53 | 3,060.24 | 327 | 323 | | | + +### Polars 1.44.2 and 2.0.0 + +Every case of the `polars` and `sqla_polars` families, in the default environment with Polars 1.44.2 and in the second environment with Polars 2.0.0. + +| Scale | Case | Total, 1.44.2 (s) | Total, 2.0.0 (s) | Client result, 1.44.2 (s) | Client result, 2.0.0 (s) | Peak RSS, 1.44.2 (MiB) | Peak RSS, 2.0.0 (MiB) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| medium | `single-polars-sync-csv-native-a1000-c0-flat-n1` | 2.43 | 2.44 | 0.31 | 0.32 | 260 | 327 | +| medium | `single-polars-sync-unload-native-a1000-c0-flat-n1` | 1.42 | 1.41 | 0.30 | 0.29 | 229 | 239 | +| large | `single-polars-sync-csv-native-a1000-c0-flat-n1` | 7.18 | 7.16 | 0.99 | 0.93 | 1,259 | 1,181 | +| large | `single-polars-sync-csv-native-a1000-c0-nested-n1` | 14.02 | 14.10 | 1.74 | 1.77 | 2,105 | 1,739 | +| large | `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 9.71 | 9.80 | 1.50 | 1.53 | 1,717 | 1,550 | +| large | `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 7.10 | 7.11 | 0.90 | 0.85 | 597 | 612 | +| large | `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 13.58 | 13.49 | 1.25 | 1.27 | 862 | 876 | +| large | `single-polars-sync-csv-native-a1000-c10000-temporal-n1` | 9.45 | 9.43 | 1.21 | 1.20 | 762 | 824 | +| large | `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 7.07 | 7.12 | 0.88 | 0.86 | 681 | 687 | +| large | `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 13.52 | 13.54 | 1.25 | 1.27 | 1,001 | 1,066 | +| large | `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 10.41 | 9.47 | 1.17 | 1.21 | 889 | 903 | +| large | `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 7.14 | 7.09 | 0.89 | 0.90 | 938 | 948 | +| large | `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 13.63 | 13.63 | 1.32 | 1.32 | 1,440 | 1,470 | +| large | `single-polars-sync-csv-native-a1000-c1000000-temporal-n1` | 9.45 | 9.47 | 1.23 | 1.24 | 1,270 | 1,281 | +| large | `single-polars-sync-csv-rows-a1000-c0-temporal-n1` | 42.25 | 42.30 | 33.95 | 34.07 | 1,703 | 1,561 | +| large | `single-polars-sync-unload-native-a1000-c0-flat-n1` | 3.58 | 3.56 | 0.43 | 0.42 | 490 | 499 | +| large | `single-polars-sync-unload-native-a1000-c0-nested-n1` | 5.98 | 5.97 | 0.79 | 0.82 | 743 | 755 | +| large | `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 3.69 | 3.58 | 0.47 | 0.44 | 512 | 511 | +| large | `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 3.56 | 3.56 | 0.40 | 0.40 | 357 | 372 | +| large | `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 5.97 | 5.97 | 0.78 | 0.81 | 399 | 390 | +| large | `single-polars-sync-unload-native-a1000-c10000-temporal-n1` | 3.57 | 3.55 | 0.42 | 0.41 | 362 | 360 | +| large | `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 3.51 | 3.51 | 0.35 | 0.37 | 465 | 483 | +| large | `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 5.92 | 5.93 | 0.75 | 0.73 | 689 | 708 | +| large | `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 3.52 | 3.50 | 0.37 | 0.37 | 471 | 515 | +| large | `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 3.51 | 3.54 | 0.35 | 0.38 | 948 | 995 | +| large | `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 5.94 | 5.95 | 0.75 | 0.77 | 1,339 | 1,336 | +| large | `single-polars-sync-unload-native-a1000-c1000000-temporal-n1` | 3.52 | 3.53 | 0.37 | 0.38 | 967 | 1,040 | +| large | `single-polars-sync-unload-rows-a1000-c0-temporal-n1` | 5.74 | 5.61 | 1.53 | 1.46 | 521 | 525 | +| large | `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 10.31 | 10.30 | 4.09 | 4.10 | 1,888 | 1,877 | +| large | `single-sqla_polars-sync-csv-native-a1000-c0-nested-n1` | 16.61 | 16.76 | 4.35 | 4.37 | 2,488 | 2,537 | +| large | `single-sqla_polars-sync-csv-native-a1000-c0-temporal-n1` | 49.95 | 50.20 | 41.71 | 41.96 | 2,301 | 2,314 | +| large | `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 10.04 | 10.03 | 3.75 | 3.82 | 941 | 938 | +| large | `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 14.70 | 15.21 | 3.39 | 3.50 | 1,364 | 1,414 | +| large | `single-sqla_polars-sync-csv-native-a1000-c100000-temporal-n1` | 49.93 | 49.66 | 41.69 | 41.41 | 1,041 | 1,039 | +| large | `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 6.66 | 6.86 | 3.51 | 3.63 | 1,749 | 1,700 | +| large | `single-sqla_polars-sync-unload-native-a1000-c0-nested-n1` | 13.98 | 14.09 | 8.80 | 8.76 | 4,295 | 4,305 | +| large | `single-sqla_polars-sync-unload-native-a1000-c0-temporal-n1` | 8.51 | 9.48 | 4.41 | 4.31 | 1,915 | 1,878 | +| large | `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 6.46 | 6.63 | 3.32 | 3.45 | 725 | 715 | +| large | `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 14.55 | 14.67 | 9.37 | 9.45 | 1,371 | 1,361 | +| large | `single-sqla_polars-sync-unload-native-a1000-c100000-temporal-n1` | 8.33 | 8.28 | 4.20 | 4.13 | 746 | 757 | +| xlarge | `single-polars-sync-csv-native-a1000-c0-flat-n1` | 58.16 | 56.70 | 7.23 | 6.99 | 9,641 | 7,547 | +| xlarge | `single-polars-sync-csv-native-a1000-c0-nested-n1` | 214.34 | 213.54 | 14.58 | 14.92 | 18,923 | 14,422 | +| xlarge | `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 112.07 | 113.07 | 12.36 | 12.35 | 14,215 | 11,284 | +| xlarge | `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 54.75 | 57.75 | 3.67 | 3.41 | 904 | 917 | +| xlarge | `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 206.20 | 207.20 | 7.26 | 8.37 | 917 | 908 | +| xlarge | `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 54.72 | 53.28 | 3.76 | 3.35 | 978 | 999 | +| xlarge | `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 211.27 | 208.10 | 7.04 | 8.64 | 1,115 | 1,073 | +| xlarge | `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 102.98 | 106.63 | 6.03 | 5.66 | 1,079 | 1,088 | +| xlarge | `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 53.71 | 54.28 | 3.82 | 3.43 | 1,806 | 1,744 | +| xlarge | `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 210.75 | 207.70 | 7.39 | 7.98 | 2,793 | 2,666 | +| xlarge | `single-polars-sync-unload-native-a1000-c0-flat-n1` | 18.51 | 17.48 | 1.19 | 1.11 | 3,765 | 3,824 | +| xlarge | `single-polars-sync-unload-native-a1000-c0-nested-n1` | 33.77 | 34.98 | 3.30 | 3.22 | 6,135 | 6,172 | +| xlarge | `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 20.62 | 21.57 | 1.21 | 1.14 | 3,889 | 3,946 | +| xlarge | `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 19.26 | 19.31 | 1.88 | 1.93 | 665 | 673 | +| xlarge | `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 36.99 | 38.10 | 6.41 | 6.43 | 633 | 646 | +| xlarge | `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 17.92 | 18.00 | 1.55 | 1.50 | 817 | 790 | +| xlarge | `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 36.51 | 36.38 | 5.87 | 5.81 | 949 | 961 | +| xlarge | `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 21.20 | 20.37 | 1.81 | 1.89 | 796 | 791 | +| xlarge | `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 18.23 | 18.77 | 1.53 | 1.40 | 2,190 | 2,180 | +| xlarge | `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 37.50 | 36.45 | 5.90 | 5.55 | 3,056 | 3,085 | +| xlarge | `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 88.39 | 86.86 | 38.58 | 38.97 | 14,978 | 15,083 | +| xlarge | `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 79.40 | 81.40 | 30.30 | 31.30 | 1,281 | 1,283 | +| xlarge | `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 229.89 | 223.33 | 27.66 | 26.06 | 1,577 | 1,593 | +| xlarge | `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 48.71 | 48.20 | 32.25 | 31.82 | 14,849 | 14,847 | +| xlarge | `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 47.77 | 48.34 | 31.09 | 30.94 | 1,054 | 1,051 | +| xlarge | `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 133.80 | 133.47 | 101.10 | 100.77 | 1,668 | 1,670 | +| xxlarge | `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 373.57 | 364.23 | 41.04 | 33.82 | 1,001 | 1,016 | +| xxlarge | `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 35.66 | 35.89 | 14.43 | 14.45 | 1,051 | 1,045 | + +### Complete case tables + +Case IDs read `-----a-c[-]--n`; `c0` means no chunking. +The Cursor jobs at arraysize 100 and the REST driver jobs ran one repetition per job; their cases merge the three jobs. + +
+medium (100,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-arrow-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 2.51 | 2.56 | 0.37 | 330 | 163 | 22 | | | | +| `single-arrow-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 2.39 | 2.43 | 0.27 | 320 | 153 | 21 | 1 | 1 | | +| `single-cursor-sync-csv-rows-a1000-c0-flat-n1` | 3 | 0 | | 23.62 | 23.72 | 21.50 | 189 | 21 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 2.86 | 2.88 | 0.74 | 305 | 128 | 12 | | | | +| `single-pandas-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 2.58 | 2.61 | 0.46 | 315 | 147 | 21 | 1 | 1 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-flat-n1` | 3 | 0 | | 2.56 | 2.61 | 0.43 | 321 | 141 | 21 | 1 | 1 | | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 2.43 | 2.44 | 0.31 | 260 | 89 | 21 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 1.42 | 2.33 | 0.30 | 229 | 61 | 38 | 1 | 1 | | +| `single-s3fs-sync-csv-rows-a1000-c0-flat-n1` | 3 | 0 | | 5.28 | 5.29 | 3.16 | 213 | 45 | 12 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 2.97 | 2.97 | 0.73 | 307 | 129 | 11 | | | | +| `single-wrangler-sync-ctas-native-a1000-c0-flat-n1` | 3 | 0 | | 2.83 | 2.97 | 0.60 | 280 | 107 | 13 | 1 | 1 | | +| `single-wrangler-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 2.48 | 2.50 | 0.25 | 268 | 101 | 13 | 1 | 1 | | + +
+ +
+large (1,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-arrow-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 8.24 | 9.32 | 1.05 | 1,190 | 997 | 40 | | | | +| `single-arrow-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 13.77 | 14.01 | 1.56 | 1,725 | 1,393 | 52 | | | | +| `single-arrow-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 9.72 | 11.32 | 1.26 | 1,477 | 1,268 | 46 | | | | +| `single-arrow-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 46.12 | 46.73 | 37.85 | 1,450 | 1,257 | 49 | | | | +| `single-arrow-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 3.54 | 3.55 | 0.38 | 793 | 588 | 22 | 1 | 10 | | +| `single-arrow-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 6.84 | 6.87 | 0.66 | 855 | 686 | 22 | 1 | 12 | | +| `single-arrow-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 4.58 | 4.58 | 0.41 | 768 | 594 | 22 | 1 | 11 | | +| `single-arrow-sync-unload-rows-a1000-c0-temporal-n1` | 3 | 0 | | 7.56 | 7.56 | 3.36 | 827 | 614 | 22 | 1 | 11 | | +| `single-cursor-sync-csv-rows-a100-c0-flat-n1` | 3 | 0 | | 1,013.56 | 1,014.42 | 1,007.31 | 181 | 14 | 12 | | | | +| `single-cursor-sync-csv-rows-a100-c0-nested-n1` | 3 | 0 | | 1,089.32 | 1,096.89 | 1,077.00 | 182 | 14 | 12 | | | | +| `single-cursor-sync-csv-rows-a1000-c0-flat-n1` | 5 | 0 | | 224.82 | 226.42 | 217.62 | 190 | 22 | 12 | | | | +| `single-cursor-sync-csv-rows-a1000-c0-nested-n1` | 5 | 0 | | 280.79 | 284.30 | 268.52 | 194 | 27 | 12 | | | | +| `single-cursor-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 313.12 | 317.63 | 304.91 | 195 | 27 | 12 | | | | +| `single-dict-sync-csv-rows-a1000-c0-flat-n1` | 3 | 0 | | 216.81 | 222.17 | 210.58 | 190 | 23 | 12 | | | | +| `single-dict-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 316.32 | 318.33 | 307.04 | 195 | 27 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 9.99 | 10.12 | 3.80 | 717 | 547 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-json-n1` | 5 | 0 | | 24.47 | 24.49 | 14.24 | 4,127 | 3,846 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 19.06 | 20.12 | 6.96 | 1,050 | 872 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 32.25 | 33.17 | 24.04 | 1,051 | 882 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c10000-flat-n1` | 5 | 0 | | 9.94 | 10.85 | 3.75 | 351 | 173 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c10000-nested-n1` | 5 | 0 | | 18.29 | 19.86 | 6.93 | 391 | 207 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c10000-temporal-n1` | 3 | 0 | | 27.71 | 27.72 | 19.49 | 375 | 187 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 9.99 | 10.17 | 3.78 | 447 | 254 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-json-n1` | 5 | 0 | | 23.99 | 24.17 | 13.74 | 747 | 567 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 17.98 | 19.13 | 6.71 | 505 | 316 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 26.83 | 26.95 | 18.60 | 468 | 292 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c1000000-flat-n1` | 5 | 0 | | 10.01 | 10.75 | 3.81 | 717 | 547 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c1000000-nested-n1` | 5 | 0 | | 19.02 | 20.25 | 6.82 | 1,063 | 874 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 31.53 | 32.47 | 23.30 | 1,074 | 904 | 12 | | | | +| `single-pandas-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 35.03 | 35.19 | 26.79 | 1,052 | 875 | 12 | | | | +| `single-pandas-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 3.69 | 3.71 | 0.54 | 808 | 580 | 23 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 10.75 | 10.80 | 5.57 | 2,114 | 1,923 | 24 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-flat-n1` | 5 | 0 | | 3.67 | 5.50 | 0.53 | 759 | 579 | 23 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-nested-n1` | 5 | 0 | | 5.98 | 6.80 | 0.79 | 875 | 690 | 24 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-temporal-n1` | 3 | 0 | | 3.70 | 3.71 | 0.54 | 801 | 600 | 24 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 3.81 | 4.72 | 0.65 | 835 | 613 | 24 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c10000-flat-n1` | 5 | 0 | | 3.71 | 4.52 | 0.56 | 352 | 178 | 20 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c10000-nested-n1` | 5 | 0 | | 10.73 | 11.87 | 5.50 | 372 | 191 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c10000-pyarrow-flat-n1` | 5 | 0 | | 3.49 | 4.48 | 0.33 | 333 | 151 | 20 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c10000-pyarrow-nested-n1` | 5 | 0 | | 5.76 | 5.82 | 0.59 | 357 | 173 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c10000-pyarrow-temporal-n1` | 3 | 0 | | 3.51 | 4.44 | 0.36 | 341 | 162 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c10000-temporal-n1` | 3 | 0 | | 3.98 | 4.95 | 0.82 | 341 | 172 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 3.42 | 3.47 | 0.28 | 444 | 272 | 20 | 1 | 9 | | +| `single-pandas-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 11.78 | 11.97 | 5.58 | 781 | 594 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-flat-n1` | 5 | 0 | | 3.38 | 3.39 | 0.25 | 440 | 251 | 20 | 1 | 9 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-nested-n1` | 5 | 0 | | 5.70 | 6.52 | 0.52 | 513 | 325 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-temporal-n1` | 3 | 0 | | 3.40 | 4.34 | 0.26 | 434 | 260 | 20 | 1 | 11 | | +| `single-pandas-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 3.53 | 3.53 | 0.39 | 468 | 291 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c1000000-flat-n1` | 5 | 0 | | 3.39 | 3.44 | 0.25 | 824 | 629 | 20 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c1000000-nested-n1` | 5 | 0 | | 10.70 | 11.50 | 5.51 | 2,147 | 1,963 | 20 | 1 | 13 | | +| `single-pandas-sync-unload-native-a1000-c1000000-pyarrow-flat-n1` | 5 | 0 | | 3.41 | 5.23 | 0.24 | 794 | 585 | 20 | 1 | 10 | | +| `single-pandas-sync-unload-native-a1000-c1000000-pyarrow-nested-n1` | 5 | 0 | | 5.68 | 5.69 | 0.50 | 890 | 704 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c1000000-pyarrow-temporal-n1` | 3 | 0 | | 3.40 | 4.35 | 0.26 | 776 | 602 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 3.48 | 4.40 | 0.35 | 876 | 706 | 20 | 1 | 12 | | +| `single-pandas-sync-unload-rows-a1000-c0-temporal-n1` | 3 | 0 | | 6.10 | 6.12 | 1.96 | 795 | 621 | 24 | 1 | 12 | | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 7.18 | 8.04 | 0.99 | 1,259 | 1,050 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 14.02 | 14.07 | 1.74 | 2,105 | 1,857 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 9.71 | 9.73 | 1.50 | 1,717 | 1,488 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 5 | 0 | | 7.10 | 7.17 | 0.90 | 597 | 428 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 5 | 0 | | 13.58 | 14.36 | 1.25 | 862 | 616 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c10000-temporal-n1` | 3 | 0 | | 9.45 | 9.47 | 1.21 | 762 | 590 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 7.07 | 7.94 | 0.88 | 681 | 508 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 13.52 | 13.61 | 1.25 | 1,001 | 794 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 10.41 | 10.41 | 1.17 | 889 | 705 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 5 | 0 | | 7.14 | 8.03 | 0.89 | 938 | 769 | 39 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 5 | 0 | | 13.63 | 14.40 | 1.32 | 1,440 | 1,243 | 39 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 9.45 | 11.28 | 1.23 | 1,270 | 1,039 | 39 | | | | +| `single-polars-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 42.25 | 42.27 | 33.95 | 1,703 | 1,529 | 43 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 3.58 | 3.62 | 0.43 | 490 | 311 | 38 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 5.98 | 6.77 | 0.79 | 743 | 569 | 38 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 3.69 | 5.45 | 0.47 | 512 | 344 | 38 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 5 | 0 | | 3.56 | 4.37 | 0.40 | 357 | 169 | 39 | 1 | 9 | | +| `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 5 | 0 | | 5.97 | 5.99 | 0.78 | 399 | 215 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c10000-temporal-n1` | 3 | 0 | | 3.57 | 4.49 | 0.42 | 362 | 190 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 3.51 | 4.32 | 0.35 | 465 | 297 | 39 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 5.92 | 6.77 | 0.75 | 689 | 504 | 39 | 1 | 13 | | +| `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 3.52 | 4.42 | 0.37 | 471 | 296 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 5 | 0 | | 3.51 | 3.54 | 0.35 | 948 | 771 | 39 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 5 | 0 | | 5.94 | 6.96 | 0.75 | 1,339 | 1,144 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 3.52 | 4.41 | 0.37 | 967 | 793 | 39 | 1 | 12 | | +| `single-polars-sync-unload-rows-a1000-c0-temporal-n1` | 3 | 0 | | 5.74 | 7.55 | 1.53 | 521 | 349 | 38 | 1 | 12 | | +| `single-s3fs-sync-csv-rows-a1000-c0-flat-n1` | 5 | 0 | | 35.46 | 35.74 | 29.28 | 320 | 128 | 12 | | | | +| `single-s3fs-sync-csv-rows-a1000-c0-nested-n1` | 5 | 0 | | 135.14 | 135.64 | 122.70 | 328 | 152 | 12 | | | | +| `single-s3fs-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 96.76 | 97.02 | 88.53 | 314 | 143 | 12 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 10.51 | 10.77 | 4.31 | 2,286 | 2,047 | 59 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 16.33 | 17.59 | 4.03 | 2,666 | 2,441 | 52 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 54.06 | 54.43 | 45.80 | 2,574 | 2,405 | 38 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 10.39 | 11.50 | 4.19 | 1,350 | 1,101 | 56 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 15.47 | 16.42 | 4.13 | 1,716 | 1,428 | 54 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 54.25 | 54.35 | 45.99 | 1,534 | 1,270 | 39 | | | | +| `single-sqla_arrow-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 6.79 | 7.71 | 3.60 | 1,902 | 1,694 | 22 | 1 | 10 | | +| `single-sqla_arrow-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 12.17 | 14.17 | 7.96 | 3,457 | 3,269 | 22 | 1 | 12 | | +| `single-sqla_arrow-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 9.39 | 10.33 | 6.24 | 1,977 | 1,805 | 22 | 1 | 11 | | +| `single-sqla_arrow-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 6.60 | 7.41 | 3.44 | 982 | 793 | 22 | 1 | 9 | | +| `single-sqla_arrow-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 12.87 | 14.49 | 7.66 | 1,351 | 1,179 | 22 | 1 | 12 | | +| `single-sqla_arrow-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 9.35 | 10.27 | 6.18 | 987 | 809 | 22 | 1 | 12 | | +| `single-sqla_pandas-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 14.81 | 15.60 | 8.61 | 1,754 | 1,580 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 21.41 | 22.65 | 9.91 | 2,364 | 2,139 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 40.34 | 41.93 | 31.94 | 2,181 | 2,001 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 14.46 | 15.21 | 8.24 | 667 | 492 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 22.00 | 22.86 | 9.60 | 837 | 640 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 33.29 | 33.99 | 25.06 | 719 | 542 | 12 | | | | +| `single-sqla_pandas-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 7.02 | 8.78 | 3.79 | 1,853 | 1,668 | 23 | 1 | 10 | | +| `single-sqla_pandas-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 13.11 | 13.96 | 7.97 | 2,760 | 2,582 | 24 | 1 | 12 | | +| `single-sqla_pandas-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 8.92 | 8.94 | 4.76 | 2,004 | 1,809 | 24 | 1 | 12 | | +| `single-sqla_pandas-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 6.57 | 7.39 | 3.43 | 691 | 516 | 20 | 1 | 10 | | +| `single-sqla_pandas-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 12.85 | 13.66 | 8.46 | 933 | 754 | 20 | 1 | 12 | | +| `single-sqla_pandas-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 9.45 | 9.50 | 4.32 | 736 | 565 | 20 | 1 | 12 | | +| `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 10.31 | 10.32 | 4.09 | 1,888 | 1,671 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 16.61 | 16.64 | 4.35 | 2,488 | 2,264 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 49.95 | 50.03 | 41.71 | 2,301 | 2,132 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 10.04 | 11.77 | 3.75 | 941 | 767 | 40 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 14.70 | 15.70 | 3.39 | 1,364 | 1,167 | 40 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 49.93 | 51.06 | 41.69 | 1,041 | 870 | 41 | | | | +| `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 6.66 | 7.49 | 3.51 | 1,749 | 1,531 | 38 | 1 | 10 | | +| `single-sqla_polars-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 13.98 | 15.73 | 8.80 | 4,295 | 4,120 | 38 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 8.51 | 8.60 | 4.41 | 1,915 | 1,697 | 38 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 6.46 | 7.28 | 3.32 | 725 | 549 | 39 | 1 | 10 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 14.55 | 16.31 | 9.37 | 1,371 | 1,194 | 39 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 8.33 | 9.29 | 4.20 | 746 | 576 | 39 | 1 | 12 | | +| `single-sqla_rest-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 226.99 | 230.84 | 220.73 | 1,648 | 1,480 | 12 | | | | +| `single-sqla_rest-sync-csv-native-a1000-c0-nested-n1` | 3 | 0 | | 294.11 | 296.33 | 281.76 | 4,247 | 4,076 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 38.22 | 39.77 | 31.88 | 1,727 | 1,539 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 142.01 | 143.88 | 130.01 | 4,297 | 4,130 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 106.04 | 106.07 | 97.73 | 2,126 | 1,956 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 38.08 | 38.92 | 31.84 | 549 | 368 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 142.51 | 143.06 | 130.20 | 1,094 | 911 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 106.05 | 106.47 | 97.13 | 659 | 472 | 12 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 12.37 | 12.46 | 6.02 | 958 | 789 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-json-n1` | 5 | 0 | | 23.10 | 23.38 | 12.61 | 1,386 | 1,210 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 37.71 | 37.99 | 29.30 | 1,695 | 1,518 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c10000-flat-n1` | 5 | 0 | | 14.11 | 14.24 | 7.76 | 285 | 105 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c10000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c10000-temporal-n1` | 3 | 0 | | 37.06 | 37.06 | 28.65 | 278 | 101 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 13.86 | 14.45 | 7.51 | 386 | 215 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-json-n1` | 5 | 0 | | 26.09 | 26.80 | 15.62 | 390 | 206 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 36.04 | 39.52 | 27.62 | 412 | 237 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c1000000-flat-n1` | 5 | 0 | | 13.77 | 13.85 | 7.38 | 717 | 548 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c1000000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 40.34 | 41.06 | 31.68 | 1,298 | 1,126 | 11 | | | | +| `single-wrangler-sync-ctas-native-a1000-c0-flat-n1` | 5 | 0 | | 4.21 | 6.11 | 0.94 | 737 | 538 | 13 | 1 | 10 | | +| `single-wrangler-sync-ctas-native-a1000-c0-nested-n1` | 5 | 0 | | 11.31 | 11.37 | 6.01 | 2,059 | 1,890 | 14 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c0-temporal-n1` | 3 | 0 | | 5.19 | 5.39 | 0.92 | 746 | 575 | 14 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c10000-flat-n1` | 5 | 0 | | 4.16 | 5.14 | 0.83 | 357 | 174 | 20 | 1 | 10 | | +| `single-wrangler-sync-ctas-native-a1000-c10000-nested-n1` | 5 | 0 | | 11.29 | 12.15 | 5.94 | 369 | 198 | 20 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c10000-temporal-n1` | 3 | 0 | | 4.26 | 6.09 | 0.98 | 346 | 177 | 20 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-flat-n1` | 5 | 0 | | 4.00 | 4.12 | 0.72 | 460 | 273 | 20 | 1 | 10 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-nested-n1` | 5 | 0 | | 11.38 | 11.43 | 6.05 | 743 | 572 | 20 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-temporal-n1` | 3 | 0 | | 5.22 | 5.99 | 0.77 | 475 | 305 | 20 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c1000000-flat-n1` | 5 | 0 | | 3.91 | 4.71 | 0.64 | 849 | 598 | 20 | 1 | 10 | | +| `single-wrangler-sync-ctas-native-a1000-c1000000-nested-n1` | 5 | 0 | | 11.25 | 12.27 | 6.11 | 2,175 | 2,003 | 20 | 1 | 12 | | +| `single-wrangler-sync-ctas-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 4.96 | 5.02 | 0.69 | 891 | 707 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 3.89 | 3.94 | 0.62 | 701 | 515 | 13 | 1 | 9 | | +| `single-wrangler-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 10.94 | 10.99 | 5.60 | 2,080 | 1,890 | 14 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 4.93 | 4.94 | 0.63 | 743 | 551 | 14 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c10000-flat-n1` | 5 | 0 | | 3.84 | 4.66 | 0.57 | 344 | 165 | 20 | 1 | 10 | | +| `single-wrangler-sync-unload-native-a1000-c10000-nested-n1` | 5 | 0 | | 11.09 | 11.94 | 5.76 | 365 | 194 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c10000-temporal-n1` | 3 | 0 | | 4.97 | 5.01 | 0.67 | 353 | 182 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 3.68 | 3.69 | 0.40 | 462 | 270 | 20 | 1 | 9 | | +| `single-wrangler-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 11.13 | 11.92 | 5.80 | 744 | 557 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 3.72 | 3.74 | 0.46 | 483 | 310 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c1000000-flat-n1` | 5 | 0 | | 3.65 | 3.71 | 0.38 | 806 | 593 | 20 | 1 | 10 | | +| `single-wrangler-sync-unload-native-a1000-c1000000-nested-n1` | 5 | 0 | | 11.07 | 11.14 | 5.77 | 2,161 | 1,984 | 20 | 1 | 12 | | +| `single-wrangler-sync-unload-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 3.70 | 3.72 | 0.43 | 870 | 695 | 20 | 1 | 12 | | + +
+ +
+xlarge (10,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-arrow-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 55.22 | 57.95 | 5.26 | 4,143 | 3,967 | 53 | | | | +| `single-arrow-sync-csv-native-a1000-c0-nested-n1` | 3 | 0 | | 211.33 | 213.71 | 10.95 | 7,405 | 7,063 | 38 | | | | +| `single-arrow-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 106.78 | 114.20 | 8.04 | 5,637 | 5,440 | 50 | | | | +| `single-arrow-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 18.16 | 19.13 | 1.77 | 4,892 | 4,606 | 24 | 1 | 140 | | +| `single-arrow-sync-unload-native-a1000-c0-nested-n1` | 3 | 0 | | 36.38 | 36.41 | 4.75 | 5,890 | 5,515 | 25 | 1 | 174 | | +| `single-arrow-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 22.29 | 22.31 | 1.85 | 5,035 | 4,824 | 25 | 1 | 160 | | +| `single-pandas-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 84.17 | 89.57 | 34.30 | 5,129 | 4,782 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-json-n1` | 3 | 0 | | 317.18 | 317.28 | 166.60 | 40,836 | 40,575 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-nested-n1` | 3 | 0 | | 280.28 | 284.57 | 74.40 | 7,849 | 7,680 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 178.25 | 193.11 | 76.50 | 8,840 | 8,670 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c10000-flat-n1` | 3 | 0 | | 85.96 | 92.46 | 35.01 | 362 | 193 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c10000-nested-n1` | 3 | 0 | | 269.80 | 270.74 | 68.33 | 389 | 218 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 81.02 | 83.35 | 35.04 | 461 | 285 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-json-n1` | 3 | 0 | | 296.04 | 296.48 | 147.12 | 769 | 600 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 278.67 | 282.28 | 77.01 | 520 | 344 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 262.39 | 268.81 | 163.71 | 476 | 307 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c1000000-flat-n1` | 3 | 0 | | 85.08 | 85.51 | 33.66 | 830 | 649 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c1000000-nested-n1` | 3 | 0 | | 273.57 | 292.56 | 76.13 | 1,177 | 994 | 12 | | | | +| `single-pandas-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 19.79 | 20.00 | 2.40 | 5,419 | 5,149 | 40 | 1 | 142 | | +| `single-pandas-sync-unload-native-a1000-c0-nested-n1` | 3 | 0 | | 87.99 | 88.66 | 55.57 | 19,498 | 19,330 | 37 | 1 | 175 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-flat-n1` | 3 | 0 | | 17.83 | 18.91 | 1.47 | 5,017 | 4,798 | 40 | 1 | 138 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-nested-n1` | 3 | 0 | | 34.22 | 35.11 | 4.65 | 5,820 | 5,645 | 37 | 1 | 174 | | +| `single-pandas-sync-unload-native-a1000-c0-pyarrow-temporal-n1` | 3 | 0 | | 21.47 | 22.21 | 1.88 | 5,076 | 4,817 | 39 | 1 | 162 | | +| `single-pandas-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 23.90 | 26.02 | 2.78 | 5,840 | 5,610 | 39 | 1 | 160 | | +| `single-pandas-sync-unload-native-a1000-c10000-flat-n1` | 3 | 0 | | 20.86 | 21.74 | 4.49 | 635 | 443 | 37 | 1 | 140 | | +| `single-pandas-sync-unload-native-a1000-c10000-nested-n1` | 3 | 0 | | 92.20 | 93.66 | 60.26 | 601 | 426 | 33 | 1 | 173 | | +| `single-pandas-sync-unload-native-a1000-c10000-pyarrow-flat-n1` | 3 | 0 | | 18.71 | 20.31 | 2.11 | 664 | 466 | 37 | 1 | 140 | | +| `single-pandas-sync-unload-native-a1000-c10000-pyarrow-nested-n1` | 3 | 0 | | 37.05 | 41.23 | 5.53 | 577 | 409 | 33 | 1 | 174 | | +| `single-pandas-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 17.89 | 19.02 | 1.75 | 771 | 603 | 37 | 1 | 139 | | +| `single-pandas-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 89.66 | 92.11 | 57.01 | 1,087 | 916 | 33 | 1 | 172 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-flat-n1` | 3 | 0 | | 17.73 | 18.55 | 1.28 | 740 | 572 | 37 | 1 | 141 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-nested-n1` | 3 | 0 | | 38.10 | 38.16 | 4.53 | 793 | 619 | 33 | 1 | 173 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-temporal-n1` | 3 | 0 | | 20.92 | 21.88 | 1.52 | 769 | 592 | 35 | 1 | 159 | | +| `single-pandas-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 22.30 | 24.00 | 2.84 | 784 | 614 | 35 | 1 | 159 | | +| `single-pandas-sync-unload-native-a1000-c1000000-flat-n1` | 3 | 0 | | 17.86 | 17.94 | 1.48 | 2,111 | 1,856 | 37 | 1 | 140 | | +| `single-pandas-sync-unload-native-a1000-c1000000-nested-n1` | 3 | 0 | | 90.80 | 91.44 | 58.89 | 5,195 | 4,895 | 33 | 1 | 174 | | +| `single-pandas-sync-unload-native-a1000-c1000000-pyarrow-flat-n1` | 3 | 0 | | 18.48 | 19.47 | 1.13 | 1,932 | 1,681 | 37 | 1 | 141 | | +| `single-pandas-sync-unload-native-a1000-c1000000-pyarrow-nested-n1` | 3 | 0 | | 35.04 | 36.24 | 4.47 | 2,460 | 2,275 | 33 | 1 | 173 | | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 58.16 | 58.51 | 7.23 | 9,641 | 9,430 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-nested-n1` | 3 | 0 | | 214.34 | 214.99 | 14.58 | 18,923 | 18,717 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 112.07 | 113.89 | 12.36 | 14,215 | 14,019 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 3 | 0 | | 54.75 | 54.78 | 3.67 | 904 | 734 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 3 | 0 | | 206.20 | 215.40 | 7.26 | 917 | 680 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 54.72 | 54.83 | 3.76 | 978 | 785 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 211.27 | 219.99 | 7.04 | 1,115 | 890 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 102.98 | 107.18 | 6.03 | 1,079 | 909 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 3 | 0 | | 53.71 | 54.65 | 3.82 | 1,806 | 1,576 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 3 | 0 | | 210.75 | 214.55 | 7.39 | 2,793 | 2,479 | 40 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 18.51 | 18.85 | 1.19 | 3,765 | 3,589 | 38 | 1 | 141 | | +| `single-polars-sync-unload-native-a1000-c0-nested-n1` | 3 | 0 | | 33.77 | 34.94 | 3.30 | 6,135 | 5,945 | 38 | 1 | 170 | | +| `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 20.62 | 20.74 | 1.21 | 3,889 | 3,711 | 38 | 1 | 159 | | +| `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 3 | 0 | | 19.26 | 19.48 | 1.88 | 665 | 481 | 56 | 1 | 140 | | +| `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 3 | 0 | | 36.99 | 39.78 | 6.41 | 633 | 457 | 52 | 1 | 173 | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 17.92 | 17.98 | 1.55 | 817 | 643 | 56 | 1 | 139 | | +| `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 36.51 | 38.10 | 5.87 | 949 | 780 | 52 | 1 | 175 | | +| `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 21.20 | 21.26 | 1.81 | 796 | 625 | 54 | 1 | 161 | | +| `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 3 | 0 | | 18.23 | 18.82 | 1.53 | 2,190 | 1,906 | 56 | 1 | 141 | | +| `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 3 | 0 | | 37.50 | 37.51 | 5.90 | 3,056 | 2,680 | 52 | 1 | 174 | | +| `single-s3fs-sync-csv-rows-a1000-c0-flat-n1` | 3 | 0 | | 348.66 | 351.26 | 296.61 | 323 | 154 | 12 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 88.41 | 90.25 | 37.43 | 15,374 | 15,047 | 38 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 84.68 | 86.32 | 35.49 | 4,177 | 3,984 | 38 | | | | +| `single-sqla_arrow-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 239.56 | 239.70 | 36.12 | 7,420 | 7,188 | 58 | | | | +| `single-sqla_arrow-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 50.93 | 52.03 | 33.71 | 16,264 | 15,933 | 24 | 1 | 139 | | +| `single-sqla_arrow-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 48.79 | 49.45 | 31.14 | 5,078 | 4,798 | 24 | 1 | 141 | | +| `single-sqla_arrow-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 115.69 | 115.96 | 84.25 | 5,928 | 5,759 | 25 | 1 | 171 | | +| `single-sqla_pandas-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 132.45 | 132.67 | 82.68 | 16,039 | 15,863 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 135.51 | 135.56 | 84.55 | 698 | 527 | 12 | | | | +| `single-sqla_pandas-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 291.25 | 300.15 | 91.79 | 873 | 695 | 12 | | | | +| `single-sqla_pandas-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 50.71 | 51.79 | 34.35 | 16,096 | 15,924 | 40 | 1 | 139 | | +| `single-sqla_pandas-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 49.92 | 50.55 | 33.55 | 1,032 | 834 | 37 | 1 | 140 | | +| `single-sqla_pandas-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 113.90 | 115.99 | 83.59 | 1,270 | 1,095 | 33 | 1 | 173 | | +| `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 88.39 | 92.21 | 38.58 | 14,978 | 14,810 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 79.40 | 80.04 | 30.30 | 1,281 | 1,106 | 41 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 229.89 | 230.14 | 27.66 | 1,577 | 1,406 | 41 | | | | +| `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 48.71 | 53.22 | 32.25 | 14,849 | 14,675 | 38 | 1 | 140 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 47.77 | 48.27 | 31.09 | 1,054 | 865 | 56 | 1 | 140 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 133.80 | 135.57 | 101.10 | 1,668 | 1,474 | 52 | 1 | 172 | | +| `single-sqla_s3fs-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 373.65 | 376.54 | 322.96 | 14,825 | 14,651 | 12 | | | | +| `single-sqla_s3fs-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 370.76 | 371.92 | 319.85 | 584 | 409 | 12 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 108.88 | 109.76 | 58.25 | 7,655 | 7,463 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-json-n1` | 3 | 0 | | 284.33 | 305.63 | 132.63 | 12,778 | 12,597 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c0-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 259.53 | 388.94 | 154.22 | 15,225 | 15,038 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c10000-flat-n1` | 3 | 0 | | 131.72 | 133.16 | 81.18 | 280 | 110 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c10000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 128.92 | 129.27 | 78.76 | 391 | 222 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-json-n1` | 3 | 0 | | 341.57 | 350.66 | 193.21 | 380 | 211 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 367.28 | 385.00 | 264.93 | 418 | 240 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c1000000-flat-n1` | 3 | 0 | | 125.96 | 126.45 | 75.88 | 1,195 | 1,022 | 11 | | | | +| `single-wrangler-sync-csv-native-a1000-c1000000-nested-n1` | 0 | 0 | yes | | | | | | | | | Wrangler CSV does not support nested types | +| `single-wrangler-sync-ctas-native-a1000-c0-flat-n1` | 3 | 0 | | 20.64 | 20.70 | 4.01 | 5,051 | 4,825 | 39 | 1 | 139 | | +| `single-wrangler-sync-ctas-native-a1000-c0-nested-n1` | 3 | 0 | | 93.19 | 93.99 | 59.99 | 19,049 | 18,861 | 44 | 1 | 171 | | +| `single-wrangler-sync-ctas-native-a1000-c0-temporal-n1` | 3 | 0 | | 23.81 | 24.10 | 4.16 | 5,474 | 5,214 | 44 | 1 | 164 | | +| `single-wrangler-sync-ctas-native-a1000-c10000-flat-n1` | 3 | 0 | | 20.42 | 21.21 | 3.58 | 686 | 517 | 27 | 1 | 141 | | +| `single-wrangler-sync-ctas-native-a1000-c10000-nested-n1` | 3 | 0 | | 90.55 | 90.85 | 59.09 | 710 | 532 | 25 | 1 | 173 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-flat-n1` | 3 | 0 | | 17.68 | 19.64 | 2.00 | 845 | 665 | 27 | 1 | 141 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-nested-n1` | 3 | 0 | | 89.15 | 89.73 | 57.79 | 1,117 | 938 | 26 | 1 | 173 | | +| `single-wrangler-sync-ctas-native-a1000-c100000-temporal-n1` | 3 | 0 | | 22.03 | 23.98 | 2.35 | 861 | 690 | 26 | 1 | 160 | | +| `single-wrangler-sync-ctas-native-a1000-c1000000-flat-n1` | 3 | 0 | | 18.47 | 22.08 | 1.75 | 2,287 | 1,997 | 27 | 1 | 140 | | +| `single-wrangler-sync-ctas-native-a1000-c1000000-nested-n1` | 3 | 0 | | 92.13 | 93.45 | 60.06 | 4,815 | 4,564 | 26 | 1 | 174 | | +| `single-wrangler-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 23.44 | 24.49 | 3.72 | 5,013 | 4,816 | 40 | 1 | 143 | | +| `single-wrangler-sync-unload-native-a1000-c0-nested-n1` | 3 | 0 | | 88.33 | 90.36 | 57.47 | 18,886 | 18,705 | 44 | 1 | 174 | | +| `single-wrangler-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 24.56 | 24.58 | 3.84 | 5,377 | 5,207 | 44 | 1 | 161 | | +| `single-wrangler-sync-unload-native-a1000-c10000-flat-n1` | 3 | 0 | | 20.00 | 21.80 | 3.28 | 664 | 470 | 27 | 1 | 139 | | +| `single-wrangler-sync-unload-native-a1000-c10000-nested-n1` | 3 | 0 | | 90.95 | 91.60 | 58.80 | 700 | 530 | 26 | 1 | 177 | | +| `single-wrangler-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 18.78 | 19.60 | 2.00 | 820 | 642 | 27 | 1 | 142 | | +| `single-wrangler-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 89.79 | 102.44 | 57.28 | 1,100 | 929 | 26 | 1 | 172 | | +| `single-wrangler-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 25.15 | 25.81 | 2.18 | 856 | 686 | 26 | 1 | 162 | | +| `single-wrangler-sync-unload-native-a1000-c1000000-flat-n1` | 3 | 0 | | 18.22 | 22.75 | 1.47 | 2,119 | 1,946 | 27 | 1 | 140 | | +| `single-wrangler-sync-unload-native-a1000-c1000000-nested-n1` | 3 | 0 | | 90.70 | 92.29 | 59.38 | 4,759 | 4,554 | 25 | 1 | 172 | | + +
+ +
+xxlarge (100,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-pandas-sync-csv-native-a1000-c0-flat-n1` | 1 | 0 | | 704.09 | 704.09 | 368.35 | 49,824 | 49,656 | 12 | | | | +| `single-pandas-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 699.88 | 714.99 | 383.42 | 462 | 294 | 12 | | | | +| `single-pandas-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 37.95 | 38.37 | 16.53 | 986 | 814 | 37 | 8 | 184 | | +| `single-pandas-sync-unload-native-a1000-c100000-pyarrow-flat-n1` | 3 | 0 | | 34.86 | 36.66 | 13.38 | 978 | 765 | 37 | 8 | 183 | | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 373.57 | 374.45 | 41.04 | 1,001 | 832 | 40 | | | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 35.66 | 36.98 | 14.43 | 1,051 | 856 | 56 | 8 | 184 | | +| `single-s3fs-sync-csv-rows-a1000-c0-flat-n1` | 3 | 0 | | 3,386.53 | 3,389.48 | 3,060.24 | 327 | 159 | 12 | | | | +| `single-wrangler-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 1,233.06 | 1,264.50 | 913.82 | 390 | 221 | 11 | | | | +| `single-wrangler-sync-ctas-native-a1000-c100000-flat-n1` | 3 | 0 | | 41.19 | 42.02 | 19.27 | 1,173 | 998 | 35 | 8 | 175 | | +| `single-wrangler-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 39.66 | 41.10 | 17.89 | 1,134 | 953 | 35 | 8 | 183 | | + +
+ +
+Polars 2.0.0, medium (100,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 2.44 | 2.49 | 0.32 | 327 | 135 | 38 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 1.41 | 2.30 | 0.29 | 239 | 70 | 38 | 1 | 1 | | + +
+ +
+Polars 2.0.0, large (1,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 7.16 | 8.00 | 0.93 | 1,181 | 908 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 14.10 | 14.14 | 1.77 | 1,739 | 1,562 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 9.80 | 10.70 | 1.53 | 1,550 | 1,348 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 5 | 0 | | 7.11 | 7.89 | 0.85 | 612 | 438 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 5 | 0 | | 13.49 | 13.59 | 1.27 | 876 | 651 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c10000-temporal-n1` | 3 | 0 | | 9.43 | 9.55 | 1.20 | 824 | 627 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 7.12 | 8.09 | 0.86 | 687 | 516 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 13.54 | 13.76 | 1.27 | 1,066 | 837 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 9.47 | 10.37 | 1.21 | 903 | 684 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 5 | 0 | | 7.09 | 8.76 | 0.90 | 948 | 778 | 39 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 5 | 0 | | 13.63 | 13.76 | 1.32 | 1,470 | 1,221 | 39 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 9.47 | 9.55 | 1.24 | 1,281 | 1,078 | 39 | | | | +| `single-polars-sync-csv-rows-a1000-c0-temporal-n1` | 3 | 0 | | 42.30 | 42.44 | 34.07 | 1,561 | 1,318 | 43 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 3.56 | 4.38 | 0.42 | 499 | 327 | 38 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 5.97 | 6.03 | 0.82 | 755 | 578 | 38 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 3.58 | 5.41 | 0.44 | 511 | 338 | 38 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 5 | 0 | | 3.56 | 5.55 | 0.40 | 372 | 187 | 39 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 5 | 0 | | 5.97 | 6.97 | 0.81 | 390 | 214 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c10000-temporal-n1` | 3 | 0 | | 3.55 | 3.56 | 0.41 | 360 | 183 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 3.51 | 4.36 | 0.37 | 483 | 304 | 39 | 1 | 9 | | +| `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 5.93 | 6.74 | 0.73 | 708 | 518 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 3.50 | 4.46 | 0.37 | 515 | 313 | 39 | 1 | 11 | | +| `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 5 | 0 | | 3.54 | 4.35 | 0.38 | 995 | 774 | 39 | 1 | 10 | | +| `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 5 | 0 | | 5.95 | 6.01 | 0.77 | 1,336 | 1,155 | 39 | 1 | 12 | | +| `single-polars-sync-unload-native-a1000-c1000000-temporal-n1` | 3 | 0 | | 3.53 | 3.65 | 0.38 | 1,040 | 816 | 39 | 1 | 12 | | +| `single-polars-sync-unload-rows-a1000-c0-temporal-n1` | 3 | 0 | | 5.61 | 5.65 | 1.46 | 525 | 349 | 38 | 1 | 12 | | +| `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 5 | 0 | | 10.30 | 10.31 | 4.10 | 1,877 | 1,706 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c0-nested-n1` | 5 | 0 | | 16.76 | 17.76 | 4.37 | 2,537 | 2,300 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 50.20 | 50.52 | 41.96 | 2,314 | 2,142 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 5 | 0 | | 10.03 | 11.21 | 3.82 | 938 | 768 | 40 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 5 | 0 | | 15.21 | 15.83 | 3.50 | 1,414 | 1,213 | 40 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 49.66 | 49.77 | 41.41 | 1,039 | 870 | 41 | | | | +| `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 5 | 0 | | 6.86 | 8.60 | 3.63 | 1,700 | 1,515 | 38 | 1 | 9 | | +| `single-sqla_polars-sync-unload-native-a1000-c0-nested-n1` | 5 | 0 | | 14.09 | 15.05 | 8.76 | 4,305 | 4,120 | 38 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 9.48 | 9.52 | 4.31 | 1,878 | 1,696 | 38 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 5 | 0 | | 6.63 | 8.45 | 3.45 | 715 | 541 | 39 | 1 | 10 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 5 | 0 | | 14.67 | 14.70 | 9.45 | 1,361 | 1,186 | 39 | 1 | 12 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 8.28 | 9.25 | 4.13 | 757 | 568 | 39 | 1 | 12 | | + +
+ +
+Polars 2.0.0, xlarge (10,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 56.70 | 57.82 | 6.99 | 7,547 | 7,343 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-nested-n1` | 3 | 0 | | 213.54 | 216.16 | 14.92 | 14,422 | 14,186 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c0-temporal-n1` | 3 | 0 | | 113.07 | 114.95 | 12.35 | 11,284 | 11,053 | 43 | | | | +| `single-polars-sync-csv-native-a1000-c10000-flat-n1` | 3 | 0 | | 57.75 | 60.06 | 3.41 | 917 | 745 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c10000-nested-n1` | 3 | 0 | | 207.20 | 209.21 | 8.37 | 908 | 732 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 53.28 | 55.06 | 3.35 | 999 | 830 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 208.10 | 212.72 | 8.64 | 1,073 | 896 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c100000-temporal-n1` | 3 | 0 | | 106.63 | 109.08 | 5.66 | 1,088 | 917 | 41 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-flat-n1` | 3 | 0 | | 54.28 | 54.37 | 3.43 | 1,744 | 1,557 | 40 | | | | +| `single-polars-sync-csv-native-a1000-c1000000-nested-n1` | 3 | 0 | | 207.70 | 208.44 | 7.98 | 2,666 | 2,490 | 40 | | | | +| `single-polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 17.48 | 17.67 | 1.11 | 3,824 | 3,626 | 38 | 1 | 139 | | +| `single-polars-sync-unload-native-a1000-c0-nested-n1` | 3 | 0 | | 34.98 | 54.08 | 3.22 | 6,172 | 6,002 | 39 | 1 | 174 | | +| `single-polars-sync-unload-native-a1000-c0-temporal-n1` | 3 | 0 | | 21.57 | 21.72 | 1.14 | 3,946 | 3,710 | 38 | 1 | 161 | | +| `single-polars-sync-unload-native-a1000-c10000-flat-n1` | 3 | 0 | | 19.31 | 19.39 | 1.93 | 673 | 501 | 56 | 1 | 138 | | +| `single-polars-sync-unload-native-a1000-c10000-nested-n1` | 3 | 0 | | 38.10 | 39.71 | 6.43 | 646 | 457 | 52 | 1 | 172 | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 18.00 | 18.73 | 1.50 | 790 | 619 | 56 | 1 | 139 | | +| `single-polars-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 36.38 | 37.54 | 5.81 | 961 | 790 | 52 | 1 | 173 | | +| `single-polars-sync-unload-native-a1000-c100000-temporal-n1` | 3 | 0 | | 20.37 | 21.99 | 1.89 | 791 | 618 | 54 | 1 | 159 | | +| `single-polars-sync-unload-native-a1000-c1000000-flat-n1` | 3 | 0 | | 18.77 | 18.88 | 1.40 | 2,180 | 1,951 | 56 | 1 | 139 | | +| `single-polars-sync-unload-native-a1000-c1000000-nested-n1` | 3 | 0 | | 36.45 | 37.11 | 5.55 | 3,085 | 2,799 | 52 | 1 | 174 | | +| `single-sqla_polars-sync-csv-native-a1000-c0-flat-n1` | 3 | 0 | | 86.86 | 88.69 | 38.97 | 15,083 | 14,913 | 43 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 81.40 | 90.45 | 31.30 | 1,283 | 1,111 | 41 | | | | +| `single-sqla_polars-sync-csv-native-a1000-c100000-nested-n1` | 3 | 0 | | 223.33 | 229.72 | 26.06 | 1,593 | 1,405 | 41 | | | | +| `single-sqla_polars-sync-unload-native-a1000-c0-flat-n1` | 3 | 0 | | 48.20 | 49.28 | 31.82 | 14,847 | 14,678 | 38 | 1 | 140 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 48.34 | 48.35 | 30.94 | 1,051 | 866 | 56 | 1 | 141 | | +| `single-sqla_polars-sync-unload-native-a1000-c100000-nested-n1` | 3 | 0 | | 133.47 | 137.87 | 100.77 | 1,670 | 1,488 | 52 | 1 | 173 | | + +
+ +
+Polars 2.0.0, xxlarge (100,000,000 rows) + +| Case | OK | Fail | Unsupported | Total (s) | p95 (s) | Client result (s) | Peak RSS (MiB) | RSS increase (MiB) | Peak threads | Output files | Largest file (MiB) | Notes | +| --- | ---: | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | +| `single-polars-sync-csv-native-a1000-c100000-flat-n1` | 3 | 0 | | 364.23 | 372.32 | 33.82 | 1,016 | 839 | 41 | | | | +| `single-polars-sync-unload-native-a1000-c100000-flat-n1` | 3 | 0 | | 35.89 | 43.90 | 14.45 | 1,045 | 873 | 56 | 8 | 179 | | + +
From 32a00e2129f39b8956e91606952dac3d9975ec0e Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Mon, 19 Oct 2026 22:26:12 +0900 Subject: [PATCH 2/5] Say that 4.2.0 is tagged on baee716c, not on the measured commit (GH-1288) The release run of 9e99caa8 failed a test on Python 3.13, fixed in #1311, so v4.2.0 was tagged on its merge commit, which differs from the measured commit only in that test file. Also say that the snapshots of the lost preparation were removed before the stack was deleted. Co-Authored-By: Claude Opus 5.5 --- benchmarks/RESULTS.md | 4 ++-- benchmarks/history/2026-10-19-9e99caa8-filesystem.md | 5 +++-- benchmarks/history/2026-10-19-9e99caa8.md | 9 +++++---- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index c893ffae0..753322376 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -18,8 +18,8 @@ Compare entries only when they used the same instance type, architecture, and sc | 2026-10-11 | `8a69ce03` | 4.0.0 | `r7g.2xlarge` (arm64), fleet of 12 | Spot | medium, large, xlarge | [2026-10-11-8a69ce03](history/2026-10-11-8a69ce03.md) | | 2026-10-13 | `437276e1` | 4.1.0 release commit | `r7g.2xlarge` (arm64), fleet of 8 | Spot | medium, large, xlarge | [2026-10-13-437276e1](history/2026-10-13-437276e1.md) | | 2026-10-13 | `437276e1` | 4.1.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-13-437276e1-filesystem](history/2026-10-13-437276e1-filesystem.md) | -| 2026-10-19 | `9e99caa8` | 4.2.0 release commit | `r7g.2xlarge` (arm64), fleet of 10 | Spot | medium, large, xlarge, xxlarge | [2026-10-19-9e99caa8](history/2026-10-19-9e99caa8.md) | -| 2026-10-19 | `9e99caa8` | 4.2.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-19-9e99caa8-filesystem](history/2026-10-19-9e99caa8-filesystem.md) | +| 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file) | `r7g.2xlarge` (arm64), fleet of 10 | Spot | medium, large, xlarge, xxlarge | [2026-10-19-9e99caa8](history/2026-10-19-9e99caa8.md) | +| 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file), filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-19-9e99caa8-filesystem](history/2026-10-19-9e99caa8-filesystem.md) | ## Recommendations diff --git a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md index 795280777..ffb2efcbd 100644 --- a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md +++ b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md @@ -7,9 +7,10 @@ See LICENSE or https://opensource.org/licenses/MIT. SPDX-License-Identifier: MIT --> -# S3 filesystem benchmark of the PyAthena 4.2.0 release commit on 2026-10-19 +# S3 filesystem benchmark of PyAthena 4.2.0 on 2026-10-19 -This entry records the [filesystem suite](../README.md#filesystem-suite) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), which is tagged `v4.2.0`, for #1159. +This entry records the [filesystem suite](../README.md#filesystem-suite) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), the commit planned for the PyAthena 4.2.0 release, for #1159. +PyAthena 4.2.0 was tagged on `baee716c`, which differs from `9e99caa8` only in one test file (#1311). It repeats the selection of the [4.1.0 run](2026-10-13-437276e1-filesystem.md), after #1184 gave the asyncio paths of `AioS3FileSystem` thread pools sized by `max_workers` instead of the event loop's default executor, and #1165 bounded the parts in flight of `S3File` writes. The cursor run of the same commit is a [separate entry](2026-10-19-9e99caa8.md). diff --git a/benchmarks/history/2026-10-19-9e99caa8.md b/benchmarks/history/2026-10-19-9e99caa8.md index 8aec381c8..d94e2a111 100644 --- a/benchmarks/history/2026-10-19-9e99caa8.md +++ b/benchmarks/history/2026-10-19-9e99caa8.md @@ -7,10 +7,11 @@ See LICENSE or https://opensource.org/licenses/MIT. SPDX-License-Identifier: MIT --> -# Cursor benchmark of the PyAthena 4.2.0 release commit on 2026-10-19 +# Cursor benchmark of PyAthena 4.2.0 on 2026-10-19 -This entry records a bounded large-data run of the [benchmark harness](../README.md) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), which is tagged `v4.2.0`. -The installed distribution reports the version `4.1.1.dev452+g9e99caa84` because the run measured the commit before it was tagged. +This entry records a bounded large-data run of the [benchmark harness](../README.md) at commit [`9e99caa8`](https://github.com/pyathena-dev/PyAthena/tree/9e99caa84857f8d29abe63435461fc412e040c43/benchmarks), the commit planned for the PyAthena 4.2.0 release. +PyAthena 4.2.0 was tagged on `baee716c`, which differs from `9e99caa8` only in one test file (#1311), so the measured package code is that of 4.2.0. +The installed distribution reports the version `4.1.1.dev452+g9e99caa84` because its version is derived from the distance to the latest tag. It repeats the selection of the [4.1.0 run](2026-10-13-437276e1.md) and adds the `temporal` workload (#1287), a scale of 100 million rows (#1162), and every Polars case again with Polars 2.0 (#1292). The S3 filesystem comparison of the same commit is a [separate entry](2026-10-19-9e99caa8-filesystem.md). @@ -49,7 +50,7 @@ Only the synchronous API ran. The stack of ten Spot hosts was created at 07:31 UTC, and its hosts started at 07:35 UTC in `us-west-2b` and `us-west-2d`. Before the queue was published, the group replaced seven of its ten hosts between 08:00 and 08:07 UTC; one of them, the host that was preparing the snapshots, was terminated with the reason `Server.SpotInstanceTermination` at 08:01 UTC, and its preparation and manifest were lost. The preparation ran again from a local checkout of the same commit with the same configuration against the stack, and the queue `release-1` was published from there at 08:52 UTC. -The snapshot tables of the lost preparation were removed when the stack was deleted. +The snapshot tables and files of the lost preparation were removed before the stack was deleted. Workers used `--api-slots 10 --terminate-host`. The first job started at 08:52 UTC and the last finished at 12:05 UTC; every job finished in its first attempt, so no trial was interrupted or superseded. From 5185fd69a435810f2430fc7c5e8bf30f987874c0 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Mon, 19 Oct 2026 22:36:22 +0900 Subject: [PATCH 3/5] Qualify the read, temporal UNLOAD, and xxlarge Athena time statements (GH-1288) The 64 MiB aio reads ran slower than in the 4.1.0 run in three of four settings, the temporal UNLOAD reads through read_sql_query() took longer than the native reads, and the xxlarge CSV execution time came from the unchunked trial instead of the chunked cases that the caption names. Co-Authored-By: Claude Opus 5.5 --- benchmarks/history/2026-10-19-9e99caa8-filesystem.md | 4 +++- benchmarks/history/2026-10-19-9e99caa8.md | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md index ffb2efcbd..9573e1ce1 100644 --- a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md +++ b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md @@ -48,9 +48,11 @@ The suite does not query Athena; the stack of the [cursor run](2026-10-19-9e99ca For the 6 GiB object, sync wrote 1,050 to 1,074 MiB/s and aio 1,047 to 1,052 MiB/s (0.98 to 1.00 times sync), against 0.75 to 0.77 times in the 4.1.0 run; for the 1 GiB object, 0.97 to 0.99 times, against 0.81 to 0.83. Sync copied the 6 GiB object at 1,835 to 1,879 MiB/s and aio at 1,889 to 1,925 MiB/s (1.01 to 1.05 times), against 521 to 540 MiB/s (0.30 to 0.35 times) in the 4.1.0 run. With the 1 GiB and 6 GiB objects, the aio processes peaked at 52 to 53 threads, against 24 to 25 in the 4.1.0 run and 51 for sync, so their thread counts follow `max_workers`. -2. **Reads and the 8-worker cases did not regress.** +2. **Reads of 256 MiB and larger and the 8-worker cases did not regress.** For the 1 GiB object, aio read at 0.96 to 1.06 times sync; with 8 workers, aio wrote at 0.95 to 1.02 times sync from 256 MiB up, and copied at 1.36 to 1.49 times, against 1.33 to 1.46 in the 4.1.0 run. The 64 MiB transfers ran at 0.74 to 1.29 times sync, without a consistent order among the settings, as in the 4.1.0 run. + The aio reads of the 64 MiB object ran at 230 to 266 MiB/s, against 202 to 312 MiB/s in the 4.1.0 run: slower in three of the four settings, by 17 to 26%, and faster in the fourth; the sync reads ran at 283 to 321 MiB/s, against 215 to 274 MiB/s. + These measurements do not establish a cause. 3. **Writes no longer held the whole object in memory** (#1165). Writing the 6 GiB object peaked at 690 to 699 MiB with 40 workers and 302 to 305 MiB with 8 workers, with either API, against 6,005 to 6,313 MiB in the 4.1.0 run; the 1 GiB object peaked at 688 to 691 and 302 to 304 MiB, against 1,245 to 1,376 MiB. 4. **Sync transfers of large objects ran faster than in the 4.1.0 run.** diff --git a/benchmarks/history/2026-10-19-9e99caa8.md b/benchmarks/history/2026-10-19-9e99caa8.md index d94e2a111..654d0bb19 100644 --- a/benchmarks/history/2026-10-19-9e99caa8.md +++ b/benchmarks/history/2026-10-19-9e99caa8.md @@ -86,7 +86,8 @@ No workflow of this repository ran between 08:52 and 12:05 UTC; the daily databa For 1 million rows from CSV results, Cursor took 304.91 seconds of client result time against 217.62 seconds for the flat workload, S3FSCursor 88.53 against 29.28 seconds, and the fetch methods of PandasCursor, ArrowCursor, and PolarsCursor 26.79, 37.85, and 33.95 seconds, against 1.96, 3.36, and 1.53 seconds from UNLOAD results, which hold no time zone or `time` columns. PandasCursor's native CSV read took 24.04 seconds against 3.80 seconds for the flat workload, and Wrangler's 29.30 against 6.02 seconds. At 10 million rows, PandasCursor's CSV read with 100,000-row chunks took 163.71 seconds of client result time, against 76.50 seconds without chunks; Wrangler took 264.93 and 154.22 seconds. - UNLOAD and CTAS reads, which add only `event_time` and `event_date`, took within 0.3 seconds of the flat workload's client result time at 1 million rows. + The native UNLOAD and CTAS reads, which add only `event_time` and `event_date`, took within 0.3 seconds of the flat workload's client result time at 1 million rows. + Through `read_sql_query()`, whose row output converts those columns, the UNLOAD reads took 0.67 to 0.97 seconds longer with the pandas and Polars drivers and 2.65 to 2.75 seconds longer with the Arrow driver. 7. **Polars 2.0.0 measured like Polars 1.44.2 except for unchunked CSV reads.** Across the 68 cases of both environments, median total times agreed within 11%, and client result times and peaks within 10%, with these exceptions, each given for Polars 2.0.0 against 1.44.2: - Unchunked CSV reads peaked lower at large nested and at xlarge (7,547 against 9,641 MiB for 10 million flat rows), and higher at medium (327 against 260 MiB). @@ -120,7 +121,7 @@ Median `TotalExecutionTimeInMillis` in seconds, from the PandasCursor (CSV and U | Transport | Shape | medium (s) | large (s) | xlarge (s) | xlarge, 4.1.0 (s) | xxlarge (s) | | --- | --- | ---: | ---: | ---: | ---: | ---: | -| CSV | flat | 1.27 | 5.54 | 48.85 | 28.80 | 335.38 | +| CSV | flat | 1.27 | 5.54 | 48.85 | 28.80 | 318.65 | | CSV | nested | | 11.32 | 208.19 | 219.95 | | | CSV | temporal | | 7.99 | 101.59 | | | | CTAS | flat | 1.23 | 3.02 | 15.98 | 17.51 | 21.35 | From 081751792cdc57a4741ec11b4fdd0fa938ff3d35 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Mon, 19 Oct 2026 23:06:33 +0900 Subject: [PATCH 4/5] Scope the 8-worker non-regression heading to 256 MiB and larger (GH-1288) Co-Authored-By: Claude Opus 5.5 --- benchmarks/history/2026-10-19-9e99caa8-filesystem.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md index 9573e1ce1..4fc265af9 100644 --- a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md +++ b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md @@ -48,7 +48,7 @@ The suite does not query Athena; the stack of the [cursor run](2026-10-19-9e99ca For the 6 GiB object, sync wrote 1,050 to 1,074 MiB/s and aio 1,047 to 1,052 MiB/s (0.98 to 1.00 times sync), against 0.75 to 0.77 times in the 4.1.0 run; for the 1 GiB object, 0.97 to 0.99 times, against 0.81 to 0.83. Sync copied the 6 GiB object at 1,835 to 1,879 MiB/s and aio at 1,889 to 1,925 MiB/s (1.01 to 1.05 times), against 521 to 540 MiB/s (0.30 to 0.35 times) in the 4.1.0 run. With the 1 GiB and 6 GiB objects, the aio processes peaked at 52 to 53 threads, against 24 to 25 in the 4.1.0 run and 51 for sync, so their thread counts follow `max_workers`. -2. **Reads of 256 MiB and larger and the 8-worker cases did not regress.** +2. **Reads and 8-worker transfers of 256 MiB and larger did not regress.** For the 1 GiB object, aio read at 0.96 to 1.06 times sync; with 8 workers, aio wrote at 0.95 to 1.02 times sync from 256 MiB up, and copied at 1.36 to 1.49 times, against 1.33 to 1.46 in the 4.1.0 run. The 64 MiB transfers ran at 0.74 to 1.29 times sync, without a consistent order among the settings, as in the 4.1.0 run. The aio reads of the 64 MiB object ran at 230 to 266 MiB/s, against 202 to 312 MiB/s in the 4.1.0 run: slower in three of the four settings, by 17 to 26%, and faster in the fourth; the sync reads ran at 283 to 321 MiB/s, against 215 to 274 MiB/s. From dd357365308a6c75f868de8c3af2ffd4919b8d10 Mon Sep 17 00:00:00 2001 From: laughingman7743 Date: Mon, 19 Oct 2026 23:10:05 +0900 Subject: [PATCH 5/5] State the 256 MiB and larger filesystem results against sync, with the 4.1.0 exception (GH-1288) Co-Authored-By: Claude Opus 5.5 --- benchmarks/history/2026-10-19-9e99caa8-filesystem.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md index 4fc265af9..a4e8798e2 100644 --- a/benchmarks/history/2026-10-19-9e99caa8-filesystem.md +++ b/benchmarks/history/2026-10-19-9e99caa8-filesystem.md @@ -48,8 +48,9 @@ The suite does not query Athena; the stack of the [cursor run](2026-10-19-9e99ca For the 6 GiB object, sync wrote 1,050 to 1,074 MiB/s and aio 1,047 to 1,052 MiB/s (0.98 to 1.00 times sync), against 0.75 to 0.77 times in the 4.1.0 run; for the 1 GiB object, 0.97 to 0.99 times, against 0.81 to 0.83. Sync copied the 6 GiB object at 1,835 to 1,879 MiB/s and aio at 1,889 to 1,925 MiB/s (1.01 to 1.05 times), against 521 to 540 MiB/s (0.30 to 0.35 times) in the 4.1.0 run. With the 1 GiB and 6 GiB objects, the aio processes peaked at 52 to 53 threads, against 24 to 25 in the 4.1.0 run and 51 for sync, so their thread counts follow `max_workers`. -2. **Reads and 8-worker transfers of 256 MiB and larger did not regress.** +2. **For objects of 256 MiB and larger, aio reads and 8-worker writes ran at 0.95 to 1.06 times sync, and 8-worker copies at 1.36 to 1.49 times sync.** For the 1 GiB object, aio read at 0.96 to 1.06 times sync; with 8 workers, aio wrote at 0.95 to 1.02 times sync from 256 MiB up, and copied at 1.36 to 1.49 times, against 1.33 to 1.46 in the 4.1.0 run. + Against the 4.1.0 run, the aio throughput of these cases rose, except the 8-worker writes of 256 MiB, at 329 and 327 MiB/s against 329 and 346 MiB/s, which ran at 0.95 times sync instead of 1.08 to 1.09. The 64 MiB transfers ran at 0.74 to 1.29 times sync, without a consistent order among the settings, as in the 4.1.0 run. The aio reads of the 64 MiB object ran at 230 to 266 MiB/s, against 202 to 312 MiB/s in the 4.1.0 run: slower in three of the four settings, by 17 to 26%, and faster in the fourth; the sync reads ran at 283 to 321 MiB/s, against 215 to 274 MiB/s. These measurements do not establish a cause.