Repository navigation
Size the asyncio filesystem's parallel requests by max_workers instead of the default executor #1159
Description
Summary
S3AioExecutor and AioS3FileSystem._copy_object_with_multipart_upload() run blocking requests with asyncio.to_thread(). As a result, their parallelism is capped by the event loop's default executor, min(32, os.cpu_count() + 4) threads, whatever max_workers is.
Evidence
The filesystem suite of the 4.1.0 benchmark (benchmarks/history/2026-10-13-437276e1-filesystem.md, #1157, #685) ran on a c8gn.2xlarge host (8 vCPUs, so 12 threads in the default executor). With max_workers=40:
| Operation | sync (MiB/s) | aio (MiB/s) | aio / sync |
|---|---|---|---|
| write, 1 GiB | 712–718 | 575–593 | 0.81–0.83 |
| write, 6 GiB | 861–864 | 647–665 | 0.75–0.77 |
| multipart copy, 6 GiB | 1,495–1,781 | 521–540 | 0.30–0.35 |
The sync processes peaked at 51 threads and the aio processes at 24–25. With max_workers=8, aio matched or exceeded sync. Reads were alike.
AioS3Filedispatches parts throughS3AioExecutorto fsspec's IO loop.- The copy coroutine uses
asyncio.to_thread()on the caller's loop. - Neither overrides the default executor, and setting one is up to the application.
Proposal
Give the asyncio paths a thread pool sized by max_workers. For example, S3AioExecutor and the copy could use loop.run_in_executor() with a ThreadPoolExecutor(max_workers) owned by the file or the filesystem call, instead of the loop's default executor. Document the behavior if the cap is kept.
Re-measurement and the async default
#685 left one decision to this issue: whether writes and copies should default to AioS3FileSystem. After the change, re-run the filesystem suite and decide.
- Setup: the same selection on
c8gn.2xlarge, as in the README's commands for that host: reads and writes up to large, writes and copies at xlarge,max_workers8 and 40, pools 10 and 40, five repetitions. - Comparison: compare with
benchmarks/history/2026-10-13-437276e1-filesystem.md, and record the run as a new history entry. - Done when:
- with 40 workers, aio writes and copies of 1 GiB and 6 GiB reach the sync throughput, within run-to-run variation, and their thread counts follow
max_workers; - reads, and the 8-worker cases, do not regress;
- the decision on making
AioS3FileSystemthe default for writes and copies is recorded here and in Benchmark S3ThreadPoolExecutor vs S3AioExecutor before defaulting to async #685.
- with 40 workers, aio writes and copies of 1 GiB and 6 GiB reach the sync throughput, within run-to-run variation, and their thread counts follow
Activity
laughingman7743 commentedon Oct 14, 2026
Plan: the fix for this issue does not run the benchmark on its own. The re-measurement described here happens in a single combined benchmark run on the commit of the release that contains #1158, #1159, #1160, and #1161 (planned as 4.2.0), together with #1162. The run uses one
r7g.2xlargestack for the cursor andread_sql_query()cases and onec8gn.2xlargestack for the filesystem suite, on the same day.laughingman7743 commentedon Oct 14, 2026
A note from #1173 (PR #1176) for the redesign of the executors here.
The future of a part that fails through
S3Executor.submit_to()is in a reference cycle: therun()closure holds the future, the future holds the exception, and the exception's traceback holds the frame ofrun(). The frames belowrun()also hold the part'sbody. So the body of a failed part is freed by the garbage collector, not when the error is raised or the future is dropped. Measured on 3a50182: a future failed throughS3ThreadPoolExecutor.submit_to()stays alive after all references to it are dropped, untilgc.collect(). With the reproduction from #1173 (max_workers=4, 8 MiB parts, every part failing), about 32 MiB stay allocated beforegc.collect()and 0.2 MiB after.#1176 bounds this to at most
max_workersfailed parts by stopping the write at the wait. Releasing them on the raise is option 2 of #1173: insubmit_to(), drop the references ofrun()to the callable and its arguments, and clear the traceback frames belowrun(), beforeset_exception(). If the executors are reworked here, that could be included, or left out deliberately.Size the asyncio filesystem's threads by max_workers and add open_asy…
laughingman7743 commentedon Oct 18, 2026
Plan update: the combined benchmark is now tracked in #1288, with #1287 and #1162. It measures the 4.2.0 release candidate on master before the release, and 4.2.0 is tagged on the measured commit after the results are reviewed. Before, the plan was to measure the tagged release.
laughingman7743 commentedon Oct 19, 2026
Re-measured in the combined 4.2.0 run (#1288), with the same selection on one
c8gn.2xlargehost, recorded in #1310 (benchmarks/history/2026-10-19-9e99caa8-filesystem.md).max_workers.Decision on the default (#685):
S3FileSystemstays the filesystem that PyAthena registers for fsspec'ss3ands3aprotocols.AioS3FileSystemis now equivalent for writes and copies and is recommended for asyncio applications.AioS3FileSystemstill runs the blocking boto3 calls in threads.Closed by #1310.