Repository navigation
Run the batch operations of S3FileSystem concurrently #1322
Description
Part of #1315.
Summary
The batch operations of the synchronous S3FileSystem, cat, get, put, pipe, and copy with a list of paths, transfer one object at a time. In the s3fs comparison (#1318, recorded by #1321), they were 7 to 59 times slower than s3fs's over 1,000 objects of 1 MiB.
Measurements
1,000 objects of 1 MiB on one c8gn.2xlarge, objects per second (median of five trials):
| Operation | S3FileSystem |
AioS3FileSystem (batch size 40) |
s3fs (batch size 40) |
|---|---|---|---|
cat |
9.1 to 9.5 | 457.8 | 184.7 |
get |
20.3 | 286.0 | 205.2 |
put |
15.0 to 16.4 | 416.4 | 222.0 |
pipe |
15.5 to 15.7 | 484.1 | 235.4 |
copy |
6.5 to 6.9 | 345.1 | 412.7 |
max_workers and the connection pool (8 or 40) did not change the synchronous rates, because the operations do not use them.
Cause
S3FileSystem inherits cat, put, and pipe from fsspec's AbstractFileSystem, and its copy and get call super(); AbstractFileSystem loops over the paths and calls cat_file, get_file, put_file, pipe_file, or cp_file for each one in turn.
rm already deletes in DeleteObjects requests of 1,000 keys and ran at the s3fs rate.
Questions
- How
S3FileSystemruns the per-object calls concurrently: on the filesystem'sS3Executorsized bymax_workers, or on a separate pool, and how the per-object calls' own parallel requests (multipart parts, ranges) share it without deadlock. - Whether to accept fsspec's
batch_sizeargument andon_errorsemantics asAsyncFileSystemdoes, so that the two filesystems take the same arguments. - Error handling:
on_error="raise"stops after the first failure in fsspec's loop; with concurrent calls, what happens to the transfers already running, and to partially written local files and objects. - The per-object time of the sequential loop, about 106 to 110 ms for a 1 MiB
catwith one GetObject request each, is about three times the per-request time of anAioS3FileSystembatch lane at batch size 8 (about 37 ms). The cause was not investigated; it should be understood before the design, because it also affects single-object calls. - Connection pool:
max_workersdefaults tocpu_count() * 5(40 on 8 vCPUs), but the S3 client'smax_pool_connectionsstays at botocore's default of 10 unlesss3_configsets it. With more concurrent requests than the pool holds, urllib3 opens extra connections and discards them on return, with a warning each, so those requests can pay a new TCP and TLS handshake. Decide whether the batch concurrency is capped by the pool, asPandasCursorcaps its Parquet footer reads (pyathena/pandas/result_set.py,min(max_pool_connections, len(paths))), or the pool is sized from the concurrency. The single-object cases of Measure the filesystem suite against s3fs on c8gn.2xlarge #1318 showed no difference between pools of 10 and 40 with 40 workers for 8 MiB ranges, but the objects suite set the pool to the concurrency, so the cost for many 1 MiB requests with an undersized pool was not measured.
Validation
Re-run the objects suite's sync cases at the medium scale and compare them with the aio and s3fs cases of the entry above.
Activity
laughingman7743 commentedon Oct 24, 2026
Input from the filesystem audit of #1327 (report), for the design questions above. #1332 proposes the executor consolidation that this issue's concurrency would build on; decide the two together.
max_workers=50):rm(DeleteObjects batches), recursivechmodandclear_multipart_uploadsreached 40 requests in flight; recursivecopy,mv,getandcatof many paths reached 1. Also sequential:_find_levels(one ListObjectsV2 per directory),expand_path'sexists()per version path,_move_pairs' HeadObject per candidate. Each concurrent operation creates its own executor per call.S3AioThreadPoolthread that waits for work submitted to the same pool never finishes once every thread is such a waiter (max_workers=1: 0 of 2 done after 2 s). A synchronous batch method must not receive the asyncio filesystem's pool as its executor, andAioS3FileSystemcannot simply run a parallel synchronous batch method in its own pool.AioS3FileSystemalready runsput_file,pipe_fileandget_filein a filesystem thread that opens a file with its own pool, so parts in flight reachmax_workers ** 2(16 UploadPart and 22 threads atmax_workers=4). If the synchronousput_file,get_fileor_findbecome internally parallel, each asyncio delegation nests another pool.cat(paths, batch_size=N)is accepted byAioS3FileSystemand passed to GetObject byS3FileSystem(ParamValidationError);rm(..., max_workers=N)is accepted by sync and sent to DeleteObjects by aio (Align AioS3FileSystem with S3FileSystem for cancellation, recursive put, expand_path, and arguments #1337). Agree on the names for both filesystems._put,_get,_pipeand_sizesfail fast by cancelling coroutines, whose threads keep running and commit (Align AioS3FileSystem with S3FileSystem for cancellation, recursive put, expand_path, and arguments #1337);_cat,_cat_rangesand_copyrun everything, then raise;_mvand_delete_objectsgather withoutbatch_size.put/getwill call the child callbacks from worker threads, as the asyncio path already does; fsspec's synchronous callers get them on their own thread today.