diff --git a/benchmarks/RESULTS.md b/benchmarks/RESULTS.md index 753322376..597051bf8 100644 --- a/benchmarks/RESULTS.md +++ b/benchmarks/RESULTS.md @@ -20,10 +20,11 @@ Compare entries only when they used the same instance type, architecture, and sc | 2026-10-13 | `437276e1` | 4.1.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-13-437276e1-filesystem](history/2026-10-13-437276e1-filesystem.md) | | 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file) | `r7g.2xlarge` (arm64), fleet of 10 | Spot | medium, large, xlarge, xxlarge | [2026-10-19-9e99caa8](history/2026-10-19-9e99caa8.md) | | 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file), filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-19-9e99caa8-filesystem](history/2026-10-19-9e99caa8-filesystem.md) | +| 2026-10-23 | `7b20b499` | unreleased (after 4.2.1), filesystem and objects suites against s3fs | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (single objects); medium (object set) | [2026-10-23-7b20b499-filesystem](history/2026-10-23-7b20b499-filesystem.md) | ## Recommendations -These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8gn.2xlarge` hosts, and, where a row says so, from the 4.0.0 run; the entries give the measured cases and their limits. +These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8gn.2xlarge` hosts, and, where a row says so, from the 4.0.0 run and the s3fs comparison of `7b20b499`; the entries give the measured cases and their limits. | Need | Use | Measured | | --- | --- | --- | @@ -36,3 +37,4 @@ These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8g | A Wrangler-like pandas read | PandasCursor | 10 million flat rows in 19.8 seconds through UNLOAD and 84.2 seconds through CSV, against 23.4 and 108.9 seconds for Wrangler | | Asynchronous code | The native asyncio cursors, or the synchronous and ThreadPool APIs offloaded with `asyncio.to_thread()` or an executor; synchronous calls block the event loop while they poll | Single-query timings of the three APIs differed by a median of 1.7% in the 4.0.0 run | | S3 transfers in asyncio applications | `AioS3FileSystem`; PyAthena keeps `S3FileSystem` registered for fsspec's `s3` and `s3a` protocols, and making `AioS3FileSystem` the default is to be revisited when boto3 supports asyncio natively, because it still runs the blocking boto3 calls in threads | With 40 workers on 8 vCPUs, `AioS3FileSystem` wrote 1 GiB and 6 GiB objects at 0.97 to 1.00 times and copied 6 GiB at 1.01 to 1.05 times the throughput of `S3FileSystem`, and read 1 GiB at 1.00 times; with 8 workers, it copied 6 GiB at 1.36 to 1.49 times (#1159) | +| Transfers of many objects with one call (`cat`, `get`, `put`, or `copy` with a list of paths, or `pipe` with a dictionary of paths and bytes) | `AioS3FileSystem` with `batch_size`; the synchronous `S3FileSystem` transfers one object at a time | In the s3fs comparison of `7b20b499`, 1,000 objects of 1 MiB at 286 to 484 objects per second with a batch size of 40, against 9.5 to 20.3 for `S3FileSystem` and 184.7 to 235.4 for s3fs, except copies, at 345.1 against 412.7 for s3fs | diff --git a/benchmarks/history/2026-10-23-7b20b499-filesystem.md b/benchmarks/history/2026-10-23-7b20b499-filesystem.md new file mode 100644 index 000000000..a41a45803 --- /dev/null +++ b/benchmarks/history/2026-10-23-7b20b499-filesystem.md @@ -0,0 +1,338 @@ + + +# S3 filesystem benchmark against s3fs on 2026-10-23 + +This entry records the [filesystem suite](../README.md#filesystem-suite) and the [objects suite](../README.md#objects-suite) at commit [`7b20b499`](https://github.com/pyathena-dev/PyAthena/tree/7b20b499bb3c5e28d06e15df38746ebceb9e4b56/benchmarks), for #1318. +It compares PyAthena's `S3FileSystem` (API `sync`) and `AioS3FileSystem` (API `aio`) with [s3fs](https://github.com/fsspec/s3fs)'s `S3FileSystem` (API `s3fs`), which #1316 added to the filesystem suite, and adds the many-object operations of #1317. +The decision that the results inform is #1315's: whether PyAthena's filesystem is worth offering as an s3fs alternative. + +## Environment + +| Item | Value | +| --- | --- | +| Commit | `7b20b499bb3c5e28d06e15df38746ebceb9e4b56`, with a clean checkout | +| Lockfile SHA-256 | `1694ce397d88c087b9e1018abb8219d0c1debcf71d4770a29d02ca9de4aa907c`, the same as the commit's `uv.lock` | +| Python | 3.12.14 | +| PyAthena cases | the benchmark environment: boto3 1.43.102, botocore 1.43.102, fsspec 2026.9.0 | +| s3fs cases | the `s3fs` environment: s3fs 2026.9.0, aiobotocore 3.9.2, aiohttp 3.14.5, botocore 1.43.102, fsspec 2026.9.0 | +| Host | one `c8gn.2xlarge` (arm64, 8 vCPUs, 16 GiB, 25 Gbps baseline network bandwidth), Spot, Amazon Linux 2023 (`ami-025d1d5c1b9b090aa`), kernel 6.18.54: in `us-west-2b` for the main run, and in `us-west-2a` for the supplementary run | +| Single objects | zero bytes in the stack's scratch bucket in `us-west-2`: 64 MiB (small), 256 MiB (medium), 1 GiB (large), and 6 GiB (xlarge) | +| Many objects | the medium set: 1,000 objects of 1 MiB of zero bytes, and a listing of 1,000 directories of 100 empty objects, 100,000 keys | +| Block size | single objects: 8 MiB for range requests, parts, copied parts, and s3fs's `chunksize`, and 50 MiB for s3fs in the supplementary run; many objects: each filesystem's defaults, under which an object of 1 MiB fits in one range or part | + +## Selection + +Both suites ran with five repetitions and no warmups: 66 jobs in the queue `fs-s3fs-1`. + +- **Single objects**: `read`, `get`, `write`, and `put` at small, medium, and large, and `write` and `copy` at xlarge, with the three APIs, `max_workers` 8 and 40, and connection pools (`max_pool_connections`) of 10 and 40. + For s3fs, the workers are `max_concurrency`, which only `get_file()` and `put_file()` use, and the pool is set through `config_kwargs`. +- **Many objects**: `cat`, `get`, `put`, `pipe`, `copy`, `rm`, `find`, and `info` over the medium set with the three APIs, at concurrency 8 and 40, which sets the filesystem's workers and pool and the `batch_size` of the `aio` and s3fs operations. + +The PyAthena cases ran in the benchmark environment and the s3fs cases in the `s3fs` environment of the same checkout. + +The block size of 8 MiB is the filesystem suite's setting for every API, as in the [4.2.0 run](2026-10-19-9e99caa8-filesystem.md); PyAthena's defaults are 5 MiB for files and 5 GiB parts for multipart copies, and s3fs's is 50 MiB. +Because s3fs's files upload their parts one at a time and its managed copy copies them one at a time, the size of a part sets the number of sequential requests, so a supplementary queue, `fs-s3fs-50m-1`, repeated the 14 single-object jobs of s3fs with a block size of 50 MiB, s3fs's `default_block_size`, the part size of its managed copy, and the `chunksize` of its `get_file()` and `put_file()` when none is given. + +## Run record + +The queue `fs-s3fs-1` was published at 15:46 UTC on 2026-10-23, and its 66 jobs finished on the one host by 20:03 UTC without an interruption; the host then retired itself. +Each of the 216 cases recorded five successful trials and no failure. +The run took about 4 hours 17 minutes against the 20 to 30 minutes that #1318 estimated from the 4.2.0 run, which had neither s3fs nor the sequential batch operations of `S3FileSystem`. +The longest jobs were s3fs's 6 GiB copy (62 minutes), s3fs's 6 GiB write (31 minutes), and the synchronous filesystem's `copy` (25 minutes) and `cat` (18 minutes) of 1,000 objects. +After recovering the results, `cleanup --execute` removed the source objects and sets, and the stack was deleted. + +The stack was deployed again from the same commit for the supplementary queue `fs-s3fs-50m-1`, published at 02:00 UTC on 2026-10-24; its 14 jobs finished by 03:24 UTC without an interruption, and each of the 56 cases recorded five successful trials and no failure. +Its longest jobs were s3fs's 6 GiB copy (42 minutes) and write (22 minutes). +Its source objects were uploaded again, removed with `cleanup --execute` after recovery, and the stack was deleted. + +## Findings + +1. **With parts of the same size, PyAthena read, wrote, and copied one object faster than s3fs.** + s3fs read each object at 85 to 93 MiB/s with any setting, as one stream through fsspec's read-ahead cache, against 203 to 600 MiB/s for PyAthena's parallel range requests: 2.4 to 6.5 times s3fs, 4.8 to 6.5 times for the 1 GiB object. + s3fs wrote at 53 to 71 MiB/s, one part at a time, against 123 to 1,066 MiB/s: 2.1 to 2.9 times s3fs for the 64 MiB object and 7.7 to 15.9 times for the 6 GiB object. + It copied the 6 GiB object at 32 to 35 MiB/s with 768 sequential UploadPartCopy requests of 8 MiB, against 264 to 1,753 MiB/s, 7.5 to 52.5 times. + Its `put_file()`, which uploads `max_concurrency` parts at a time, ran at 120 to 250 MiB/s, against 146 to 793 MiB/s, 1.1 to 3.2 times. + With 50 MiB, s3fs's default, s3fs wrote at 75 to 97 MiB/s, 1.2 to 1.5 times its rate with 8 MiB, and copied the 6 GiB object at 47 to 53 MiB/s, 1.35 to 1.58 times; its reads stayed at 75 to 91 MiB/s. + PyAthena, with 8 MiB, still read at 2.3 to 8.0 times, wrote at 1.6 to 11.1 times, put at 1.5 to 3.0 times, and copied at 5.3 to 33.4 times these rates. + With 50 MiB chunks, s3fs's `put_file()` of the 64 MiB object is one PutObject request instead of a multipart upload, since the object is smaller than two chunks. +2. **s3fs's `get_file()` ran at 2.2 to 3.7 times PyAthena's with 8 MiB chunks, and at 5 to 9 times with its default 50 MiB for objects of 256 MiB and larger.** + s3fs downloaded at 162 to 243 MiB/s with `max_concurrency` range requests of 8 MiB at a time, against 54 to 73 MiB/s for both PyAthena APIs with every setting; PyAthena's `get_file()` reads through a file whose first fetch requests two blocks in parallel and each later fetch one block, so 40 workers add no parallel requests over 8. + With 50 MiB chunks, s3fs downloaded the 256 MiB object at 349 to 356 MiB/s and the 1 GiB object at 451 to 632 MiB/s; the 64 MiB object, two chunks, ran at 103 to 104 MiB/s, 1.4 to 1.5 times PyAthena. +3. **The synchronous `S3FileSystem` ran the batch operations over many objects one object at a time.** + It catted, downloaded, uploaded, piped, and copied 6.5 to 20.3 objects per second at either concurrency, against 86.6 to 412.7 for s3fs: s3fs was 7.0 to 59.5 times faster. + These operations fall through to fsspec's sequential loops in `AbstractFileSystem`. + Deletes, which every API sends as DeleteObjects requests of 1,000 keys, and listings, which every API pages through ListObjectsV2, ran within 0.94 to 1.14 times of each other. +4. **`AioS3FileSystem` matched s3fs at concurrency 8 and transferred faster at concurrency 40, except for copies and metadata.** + At concurrency 8, s3fs ran at 0.94 to 1.14 times the aio throughput of every operation but `cat`, where `AioS3FileSystem` ran at 218 objects per second against 143. + At concurrency 40, `AioS3FileSystem` catted, downloaded, uploaded, and piped at 286 to 484 objects per second, 1.4 to 2.5 times s3fs, while s3fs copied at 1.20 times and ran `info()` at 1.45 times the aio throughput. + The aio processes of the transfers and `info()` peaked at 52 threads at concurrency 40, against 13 for s3fs; the event loop of the trial lagged at most 83 ms, and its 95th percentile stayed at or below 5.25 ms except for `info()` at concurrency 40, at 23 ms. +5. **s3fs used fewer threads and less CPU time for reads, and more memory for downloads.** + s3fs's processes peaked at 13 threads in every case but `rm`, where every API's peaked at 43 or 44 with the 32 threads that the harness uses to prepare the copies to delete; in the single-object cases, PyAthena's peaked at 19 to 22 with 8 workers and up to 54 with 40, and at 13 to 16 for downloads, which keep at most two range requests in flight. + s3fs's reads took about a third of the CPU time of PyAthena's: 1.70 to 1.76 seconds against 4.81 to 5.94 seconds for the 1 GiB object, which PyAthena reads in parallel parts and then joins. + For objects of 256 MiB and larger, PyAthena's writes, puts, downloads, and copies took 0.94 to 1.45 times the CPU time of s3fs's, for example 22.6 to 25.9 seconds for the 6 GiB write against 23.8 to 24.2 for s3fs. + s3fs's 1 GiB download with `max_concurrency` 40 peaked at 927 to 1,137 MiB with 8 MiB chunks and 1,743 to 2,137 MiB with 50 MiB chunks, against 278 to 299 MiB for PyAthena; its writes of 256 MiB and larger objects peaked at 225 to 227 MiB, against 302 to 305 MiB for PyAthena with 8 workers, and 575 to 698 MiB with 40, which keep up to `max_workers` parts in flight. + +## Recommendation for #1315 + +PyAthena's filesystem has a measured throughput advantage in common scenarios: single-object reads, writes, and copies with either API, and batch transfers of `AioS3FileSystem` at high concurrency. +This meets #1315's first decision criterion, with the dependency advantage that the issue describes. +It also has two gaps that a user migrating from s3fs would meet, and that should be closed before a standalone package is designed: + +- the batch operations of the synchronous `S3FileSystem` (`cat`, `get`, `put`, `pipe`, `copy`), which run one object at a time; +- `get_file()`, which reads one block at a time instead of in parallel ranges, at a fifth to a ninth of s3fs's default throughput for objects of 256 MiB and larger. + +The copies and `info()` calls of `AioS3FileSystem` at concurrency 40, at 0.84 and 0.69 times s3fs, are smaller differences that this run does not explain. + +## Limits + +- One host and five trials per case; the supplementary run used another host in another Availability Zone, and its table compares it with the main run's cases. +- The objects hold zero bytes; S3 does not compress them, but other content was not measured. +- Only the medium set of 1,000 objects of 1 MiB ran; small objects and larger sets were not measured. +- s3fs ran with the settings of the suite, not tuned beyond its block size; for example, its file reads use fsspec's default read-ahead cache. +- The `info` operation of the synchronous filesystem runs through the harness's thread pool, not through a batch operation of the filesystem. +- Every download writes the object to the tmpfs `/tmp`, which the RSS does not include. +- PyAthena's single-object cases ran only with 8 MiB blocks; its defaults, 5 MiB for files and 5 GiB parts for multipart copies, were not measured. + +## Results + +Median throughput is the transferred bytes or objects divided by the median total time; threads are the peak thread count of the trial process, RSS its peak sampled resident memory, including the harness's common imports, and CPU the median process CPU time, user and system, of the operation. + +### Single objects + +Throughput with 8 MiB blocks: + +| Scale | Operation | Workers | Pool | sync (MiB/s) | aio (MiB/s) | s3fs (MiB/s) | s3fs / sync | s3fs / aio | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| small (64 MiB) | read | 8 | 10 | 240 | 203 | 86 | 0.36 | 0.42 | +| small (64 MiB) | read | 8 | 40 | 317 | 281 | 85 | 0.27 | 0.30 | +| small (64 MiB) | read | 40 | 10 | 257 | 285 | 85 | 0.33 | 0.30 | +| small (64 MiB) | read | 40 | 40 | 264 | 246 | 87 | 0.33 | 0.35 | +| small (64 MiB) | get | 8 | 10 | 72 | 70 | 165 | 2.28 | 2.37 | +| small (64 MiB) | get | 8 | 40 | 73 | 68 | 162 | 2.21 | 2.39 | +| small (64 MiB) | get | 40 | 10 | 72 | 73 | 171 | 2.37 | 2.35 | +| small (64 MiB) | get | 40 | 40 | 72 | 71 | 181 | 2.50 | 2.54 | +| small (64 MiB) | write | 8 | 10 | 148 | 155 | 53 | 0.36 | 0.34 | +| small (64 MiB) | write | 8 | 40 | 167 | 123 | 60 | 0.36 | 0.48 | +| small (64 MiB) | write | 40 | 10 | 129 | 151 | 61 | 0.47 | 0.40 | +| small (64 MiB) | write | 40 | 40 | 158 | 144 | 55 | 0.35 | 0.38 | +| small (64 MiB) | put | 8 | 10 | 146 | 162 | 124 | 0.85 | 0.77 | +| small (64 MiB) | put | 8 | 40 | 165 | 164 | 129 | 0.78 | 0.79 | +| small (64 MiB) | put | 40 | 10 | 159 | 168 | 120 | 0.76 | 0.71 | +| small (64 MiB) | put | 40 | 40 | 148 | 171 | 134 | 0.91 | 0.78 | +| medium (256 MiB) | read | 8 | 10 | 353 | 370 | 91 | 0.26 | 0.25 | +| medium (256 MiB) | read | 8 | 40 | 371 | 354 | 91 | 0.25 | 0.26 | +| medium (256 MiB) | read | 40 | 10 | 519 | 499 | 90 | 0.17 | 0.18 | +| medium (256 MiB) | read | 40 | 40 | 516 | 464 | 91 | 0.18 | 0.20 | +| medium (256 MiB) | get | 8 | 10 | 54 | 70 | 203 | 3.75 | 2.91 | +| medium (256 MiB) | get | 8 | 40 | 69 | 67 | 199 | 2.87 | 2.96 | +| medium (256 MiB) | get | 40 | 10 | 67 | 68 | 225 | 3.36 | 3.32 | +| medium (256 MiB) | get | 40 | 40 | 66 | 70 | 224 | 3.42 | 3.22 | +| medium (256 MiB) | write | 8 | 10 | 314 | 325 | 61 | 0.19 | 0.19 | +| medium (256 MiB) | write | 8 | 40 | 303 | 338 | 62 | 0.21 | 0.18 | +| medium (256 MiB) | write | 40 | 10 | 441 | 447 | 66 | 0.15 | 0.15 | +| medium (256 MiB) | write | 40 | 40 | 427 | 420 | 60 | 0.14 | 0.14 | +| medium (256 MiB) | put | 8 | 10 | 327 | 312 | 173 | 0.53 | 0.56 | +| medium (256 MiB) | put | 8 | 40 | 330 | 339 | 179 | 0.54 | 0.53 | +| medium (256 MiB) | put | 40 | 10 | 448 | 392 | 211 | 0.47 | 0.54 | +| medium (256 MiB) | put | 40 | 40 | 450 | 428 | 210 | 0.47 | 0.49 | +| large (1 GiB) | read | 8 | 10 | 440 | 446 | 92 | 0.21 | 0.21 | +| large (1 GiB) | read | 8 | 40 | 452 | 448 | 92 | 0.20 | 0.21 | +| large (1 GiB) | read | 40 | 10 | 591 | 595 | 93 | 0.16 | 0.16 | +| large (1 GiB) | read | 40 | 40 | 600 | 575 | 92 | 0.15 | 0.16 | +| large (1 GiB) | get | 8 | 10 | 69 | 69 | 222 | 3.22 | 3.22 | +| large (1 GiB) | get | 8 | 40 | 70 | 69 | 221 | 3.18 | 3.20 | +| large (1 GiB) | get | 40 | 10 | 68 | 69 | 242 | 3.58 | 3.49 | +| large (1 GiB) | get | 40 | 40 | 69 | 70 | 243 | 3.53 | 3.49 | +| large (1 GiB) | write | 8 | 10 | 495 | 484 | 67 | 0.14 | 0.14 | +| large (1 GiB) | write | 8 | 40 | 446 | 490 | 70 | 0.16 | 0.14 | +| large (1 GiB) | write | 40 | 10 | 802 | 778 | 71 | 0.09 | 0.09 | +| large (1 GiB) | write | 40 | 40 | 809 | 772 | 68 | 0.08 | 0.09 | +| large (1 GiB) | put | 8 | 10 | 463 | 481 | 180 | 0.39 | 0.37 | +| large (1 GiB) | put | 8 | 40 | 472 | 473 | 185 | 0.39 | 0.39 | +| large (1 GiB) | put | 40 | 10 | 744 | 793 | 248 | 0.33 | 0.31 | +| large (1 GiB) | put | 40 | 40 | 767 | 768 | 250 | 0.33 | 0.33 | +| xlarge (6 GiB) | write | 8 | 10 | 534 | 551 | 69 | 0.13 | 0.13 | +| xlarge (6 GiB) | write | 8 | 40 | 531 | 538 | 65 | 0.12 | 0.12 | +| xlarge (6 GiB) | write | 40 | 10 | 1,066 | 1,037 | 67 | 0.06 | 0.06 | +| xlarge (6 GiB) | write | 40 | 40 | 1,062 | 1,034 | 68 | 0.06 | 0.07 | +| xlarge (6 GiB) | copy | 8 | 10 | 266 | 369 | 35 | 0.13 | 0.10 | +| xlarge (6 GiB) | copy | 8 | 40 | 264 | 326 | 35 | 0.13 | 0.11 | +| xlarge (6 GiB) | copy | 40 | 10 | 1,438 | 1,478 | 32 | 0.02 | 0.02 | +| xlarge (6 GiB) | copy | 40 | 40 | 1,753 | 1,643 | 33 | 0.02 | 0.02 | + +Process cost with 8 MiB blocks: + +| Scale | Operation | Workers | Pool | sync threads | aio threads | s3fs threads | sync peak RSS (MiB) | aio peak RSS (MiB) | s3fs peak RSS (MiB) | sync CPU (s) | aio CPU (s) | s3fs CPU (s) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| small (64 MiB) | read | 8 | 10 | 19 | 21 | 13 | 316 | 310 | 288 | 0.36 | 0.36 | 0.12 | +| small (64 MiB) | read | 8 | 40 | 19 | 21 | 13 | 289 | 330 | 304 | 0.36 | 0.37 | 0.12 | +| small (64 MiB) | read | 40 | 10 | 19 | 21 | 13 | 291 | 341 | 304 | 0.37 | 0.38 | 0.13 | +| small (64 MiB) | read | 40 | 40 | 19 | 21 | 13 | 314 | 323 | 300 | 0.38 | 0.38 | 0.11 | +| small (64 MiB) | get | 8 | 10 | 13 | 16 | 13 | 276 | 254 | 329 | 0.32 | 0.32 | 0.28 | +| small (64 MiB) | get | 8 | 40 | 13 | 16 | 13 | 276 | 272 | 329 | 0.31 | 0.31 | 0.30 | +| small (64 MiB) | get | 40 | 10 | 13 | 16 | 13 | 272 | 273 | 328 | 0.32 | 0.32 | 0.29 | +| small (64 MiB) | get | 40 | 40 | 13 | 16 | 13 | 272 | 269 | 334 | 0.32 | 0.32 | 0.28 | +| small (64 MiB) | write | 8 | 10 | 19 | 21 | 13 | 285 | 287 | 225 | 0.32 | 0.32 | 0.26 | +| small (64 MiB) | write | 8 | 40 | 19 | 21 | 13 | 285 | 287 | 225 | 0.32 | 0.32 | 0.26 | +| small (64 MiB) | write | 40 | 10 | 19 | 21 | 13 | 285 | 287 | 225 | 0.31 | 0.32 | 0.26 | +| small (64 MiB) | write | 40 | 40 | 19 | 21 | 13 | 285 | 287 | 226 | 0.32 | 0.32 | 0.26 | +| small (64 MiB) | put | 8 | 10 | 19 | 22 | 13 | 293 | 295 | 303 | 0.32 | 0.33 | 0.25 | +| small (64 MiB) | put | 8 | 40 | 19 | 22 | 13 | 293 | 295 | 301 | 0.33 | 0.33 | 0.25 | +| small (64 MiB) | put | 40 | 10 | 19 | 22 | 13 | 293 | 295 | 304 | 0.32 | 0.33 | 0.25 | +| small (64 MiB) | put | 40 | 40 | 19 | 22 | 13 | 293 | 295 | 303 | 0.33 | 0.32 | 0.25 | +| medium (256 MiB) | read | 8 | 10 | 19 | 21 | 13 | 601 | 671 | 675 | 1.29 | 1.30 | 0.38 | +| medium (256 MiB) | read | 8 | 40 | 19 | 21 | 13 | 649 | 710 | 707 | 1.30 | 1.28 | 0.39 | +| medium (256 MiB) | read | 40 | 10 | 43 | 45 | 13 | 830 | 759 | 693 | 1.56 | 1.59 | 0.46 | +| medium (256 MiB) | read | 40 | 40 | 43 | 45 | 13 | 832 | 744 | 667 | 1.56 | 1.61 | 0.38 | +| medium (256 MiB) | get | 8 | 10 | 13 | 16 | 13 | 282 | 288 | 400 | 1.17 | 1.15 | 1.08 | +| medium (256 MiB) | get | 8 | 40 | 13 | 16 | 13 | 282 | 296 | 399 | 1.16 | 1.14 | 1.10 | +| medium (256 MiB) | get | 40 | 10 | 13 | 16 | 13 | 274 | 272 | 525 | 1.15 | 1.16 | 1.05 | +| medium (256 MiB) | get | 40 | 40 | 13 | 16 | 13 | 274 | 273 | 629 | 1.16 | 1.14 | 1.08 | +| medium (256 MiB) | write | 8 | 10 | 19 | 21 | 13 | 302 | 303 | 225 | 1.03 | 1.04 | 0.99 | +| medium (256 MiB) | write | 8 | 40 | 19 | 21 | 13 | 302 | 304 | 226 | 1.02 | 1.05 | 1.02 | +| medium (256 MiB) | write | 40 | 10 | 43 | 45 | 13 | 575 | 578 | 225 | 1.34 | 1.31 | 1.02 | +| medium (256 MiB) | write | 40 | 40 | 43 | 45 | 13 | 575 | 577 | 226 | 1.34 | 1.33 | 1.01 | +| medium (256 MiB) | put | 8 | 10 | 19 | 22 | 13 | 310 | 311 | 303 | 1.05 | 1.05 | 0.93 | +| medium (256 MiB) | put | 8 | 40 | 19 | 22 | 13 | 310 | 312 | 301 | 1.04 | 1.05 | 0.92 | +| medium (256 MiB) | put | 40 | 10 | 43 | 46 | 13 | 579 | 585 | 502 | 1.36 | 1.36 | 0.94 | +| medium (256 MiB) | put | 40 | 40 | 43 | 46 | 13 | 583 | 580 | 594 | 1.35 | 1.35 | 0.98 | +| large (1 GiB) | read | 8 | 10 | 19 | 21 | 13 | 2,222 | 2,184 | 2,300 | 4.81 | 5.15 | 1.70 | +| large (1 GiB) | read | 8 | 40 | 19 | 21 | 13 | 2,219 | 2,159 | 2,280 | 4.98 | 4.98 | 1.70 | +| large (1 GiB) | read | 40 | 10 | 51 | 53 | 13 | 2,333 | 2,420 | 2,286 | 5.78 | 5.84 | 1.73 | +| large (1 GiB) | read | 40 | 40 | 51 | 53 | 13 | 2,270 | 2,269 | 2,277 | 5.87 | 5.94 | 1.76 | +| large (1 GiB) | get | 8 | 10 | 13 | 16 | 13 | 278 | 299 | 408 | 4.50 | 4.44 | 4.08 | +| large (1 GiB) | get | 8 | 40 | 13 | 16 | 13 | 278 | 297 | 410 | 4.38 | 4.33 | 4.08 | +| large (1 GiB) | get | 40 | 10 | 13 | 16 | 13 | 282 | 292 | 927 | 4.36 | 4.44 | 4.08 | +| large (1 GiB) | get | 40 | 40 | 13 | 16 | 13 | 282 | 299 | 1,137 | 4.47 | 4.44 | 4.11 | +| large (1 GiB) | write | 8 | 10 | 19 | 21 | 13 | 302 | 304 | 226 | 3.91 | 3.91 | 4.01 | +| large (1 GiB) | write | 8 | 40 | 19 | 21 | 13 | 302 | 304 | 225 | 3.87 | 3.88 | 4.00 | +| large (1 GiB) | write | 40 | 10 | 51 | 53 | 13 | 688 | 691 | 225 | 4.54 | 4.64 | 4.01 | +| large (1 GiB) | write | 40 | 40 | 51 | 53 | 13 | 688 | 698 | 225 | 4.59 | 4.64 | 4.03 | +| large (1 GiB) | put | 8 | 10 | 19 | 22 | 13 | 310 | 312 | 301 | 3.90 | 3.88 | 3.56 | +| large (1 GiB) | put | 8 | 40 | 19 | 22 | 13 | 310 | 312 | 302 | 3.90 | 3.89 | 3.60 | +| large (1 GiB) | put | 40 | 10 | 51 | 54 | 13 | 696 | 698 | 566 | 4.64 | 4.64 | 3.56 | +| large (1 GiB) | put | 40 | 40 | 51 | 54 | 13 | 696 | 698 | 693 | 4.57 | 4.67 | 3.66 | +| xlarge (6 GiB) | write | 8 | 10 | 19 | 21 | 13 | 303 | 305 | 227 | 22.56 | 22.86 | 23.77 | +| xlarge (6 GiB) | write | 8 | 40 | 19 | 21 | 13 | 303 | 305 | 226 | 22.83 | 22.86 | 24.21 | +| xlarge (6 GiB) | write | 40 | 10 | 51 | 53 | 13 | 690 | 692 | 226 | 25.70 | 25.90 | 24.06 | +| xlarge (6 GiB) | write | 40 | 40 | 51 | 53 | 13 | 690 | 692 | 226 | 25.71 | 25.91 | 24.05 | +| xlarge (6 GiB) | copy | 8 | 10 | 19 | 20 | 13 | 190 | 192 | 212 | 1.07 | 1.09 | 0.98 | +| xlarge (6 GiB) | copy | 8 | 40 | 19 | 20 | 13 | 191 | 192 | 212 | 1.10 | 1.17 | 1.01 | +| xlarge (6 GiB) | copy | 40 | 10 | 51 | 52 | 13 | 217 | 218 | 212 | 1.23 | 1.33 | 0.99 | +| xlarge (6 GiB) | copy | 40 | 40 | 51 | 52 | 13 | 217 | 218 | 212 | 1.23 | 1.35 | 0.98 | + +### s3fs with 50 MiB blocks + +The supplementary run of s3fs, beside the main run's s3fs, sync, and aio throughput with 8 MiB blocks. +Every s3fs process peaked at 13 threads. +The 64 MiB `put` with 50 MiB chunks is one PutObject request. + +| Scale | Operation | Workers | Pool | s3fs 8 MiB (MiB/s) | s3fs 50 MiB (MiB/s) | 50 MiB / 8 MiB | sync (MiB/s) | aio (MiB/s) | s3fs 50 MiB / sync | s3fs 50 MiB / aio | s3fs 50 MiB peak RSS (MiB) | s3fs 50 MiB CPU (s) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| small (64 MiB) | read | 8 | 10 | 86 | 87 | 1.00 | 240 | 203 | 0.36 | 0.43 | 299 | 0.12 | +| small (64 MiB) | read | 8 | 40 | 85 | 87 | 1.02 | 317 | 281 | 0.27 | 0.31 | 275 | 0.11 | +| small (64 MiB) | read | 40 | 10 | 85 | 87 | 1.01 | 257 | 285 | 0.34 | 0.30 | 284 | 0.13 | +| small (64 MiB) | read | 40 | 40 | 87 | 87 | 1.00 | 264 | 246 | 0.33 | 0.35 | 288 | 0.12 | +| small (64 MiB) | get | 8 | 10 | 165 | 103 | 0.63 | 72 | 70 | 1.43 | 1.48 | 320 | 0.14 | +| small (64 MiB) | get | 8 | 40 | 162 | 103 | 0.64 | 73 | 68 | 1.41 | 1.53 | 309 | 0.14 | +| small (64 MiB) | get | 40 | 10 | 171 | 104 | 0.61 | 72 | 73 | 1.45 | 1.44 | 318 | 0.13 | +| small (64 MiB) | get | 40 | 40 | 181 | 104 | 0.57 | 72 | 71 | 1.43 | 1.46 | 320 | 0.13 | +| small (64 MiB) | write | 8 | 10 | 53 | 75 | 1.41 | 148 | 155 | 0.51 | 0.49 | 310 | 0.26 | +| small (64 MiB) | write | 8 | 40 | 60 | 76 | 1.28 | 167 | 123 | 0.46 | 0.62 | 310 | 0.26 | +| small (64 MiB) | write | 40 | 10 | 61 | 75 | 1.23 | 129 | 151 | 0.58 | 0.49 | 310 | 0.26 | +| small (64 MiB) | write | 40 | 40 | 55 | 75 | 1.38 | 158 | 144 | 0.48 | 0.52 | 310 | 0.26 | +| small (64 MiB) | put | 8 | 10 | 124 | 99 | 0.79 | 146 | 162 | 0.68 | 0.61 | 274 | 0.23 | +| small (64 MiB) | put | 8 | 40 | 129 | 99 | 0.77 | 165 | 164 | 0.60 | 0.61 | 274 | 0.23 | +| small (64 MiB) | put | 40 | 10 | 120 | 99 | 0.83 | 159 | 168 | 0.63 | 0.59 | 274 | 0.24 | +| small (64 MiB) | put | 40 | 40 | 134 | 99 | 0.74 | 148 | 171 | 0.67 | 0.58 | 274 | 0.23 | +| medium (256 MiB) | read | 8 | 10 | 91 | 91 | 1.00 | 353 | 370 | 0.26 | 0.25 | 673 | 0.45 | +| medium (256 MiB) | read | 8 | 40 | 91 | 91 | 1.00 | 371 | 354 | 0.25 | 0.26 | 692 | 0.43 | +| medium (256 MiB) | read | 40 | 10 | 90 | 91 | 1.01 | 519 | 499 | 0.18 | 0.18 | 682 | 0.47 | +| medium (256 MiB) | read | 40 | 40 | 91 | 91 | 1.00 | 516 | 464 | 0.18 | 0.20 | 691 | 0.39 | +| medium (256 MiB) | get | 8 | 10 | 203 | 356 | 1.75 | 54 | 70 | 6.58 | 5.11 | 720 | 0.46 | +| medium (256 MiB) | get | 8 | 40 | 199 | 354 | 1.78 | 69 | 67 | 5.10 | 5.26 | 719 | 0.47 | +| medium (256 MiB) | get | 40 | 10 | 225 | 355 | 1.58 | 67 | 68 | 5.29 | 5.24 | 712 | 0.48 | +| medium (256 MiB) | get | 40 | 40 | 224 | 349 | 1.56 | 66 | 70 | 5.32 | 5.01 | 720 | 0.49 | +| medium (256 MiB) | write | 8 | 10 | 61 | 90 | 1.47 | 314 | 325 | 0.29 | 0.28 | 310 | 0.98 | +| medium (256 MiB) | write | 8 | 40 | 62 | 90 | 1.45 | 303 | 338 | 0.30 | 0.27 | 310 | 1.00 | +| medium (256 MiB) | write | 40 | 10 | 66 | 90 | 1.37 | 441 | 447 | 0.20 | 0.20 | 310 | 0.98 | +| medium (256 MiB) | write | 40 | 40 | 60 | 90 | 1.50 | 427 | 420 | 0.21 | 0.21 | 310 | 1.00 | +| medium (256 MiB) | put | 8 | 10 | 173 | 208 | 1.20 | 327 | 312 | 0.63 | 0.67 | 489 | 0.91 | +| medium (256 MiB) | put | 8 | 40 | 179 | 216 | 1.21 | 330 | 339 | 0.66 | 0.64 | 489 | 0.91 | +| medium (256 MiB) | put | 40 | 10 | 211 | 206 | 0.98 | 448 | 392 | 0.46 | 0.52 | 490 | 0.92 | +| medium (256 MiB) | put | 40 | 40 | 210 | 216 | 1.03 | 450 | 428 | 0.48 | 0.50 | 488 | 0.92 | +| large (1 GiB) | read | 8 | 10 | 92 | 78 | 0.85 | 440 | 446 | 0.18 | 0.18 | 2,281 | 1.52 | +| large (1 GiB) | read | 8 | 40 | 92 | 83 | 0.89 | 452 | 448 | 0.18 | 0.18 | 2,269 | 1.77 | +| large (1 GiB) | read | 40 | 10 | 93 | 75 | 0.81 | 591 | 595 | 0.13 | 0.13 | 2,283 | 1.88 | +| large (1 GiB) | read | 40 | 40 | 92 | 91 | 0.98 | 600 | 575 | 0.15 | 0.16 | 2,296 | 1.54 | +| large (1 GiB) | get | 8 | 10 | 222 | 451 | 2.04 | 69 | 69 | 6.56 | 6.54 | 1,361 | 1.61 | +| large (1 GiB) | get | 8 | 40 | 221 | 474 | 2.14 | 70 | 69 | 6.80 | 6.84 | 1,316 | 1.58 | +| large (1 GiB) | get | 40 | 10 | 242 | 620 | 2.56 | 68 | 69 | 9.16 | 8.94 | 1,743 | 1.44 | +| large (1 GiB) | get | 40 | 40 | 243 | 632 | 2.60 | 69 | 70 | 9.20 | 9.08 | 2,137 | 1.48 | +| large (1 GiB) | write | 8 | 10 | 67 | 95 | 1.41 | 495 | 484 | 0.19 | 0.20 | 310 | 3.87 | +| large (1 GiB) | write | 8 | 40 | 70 | 94 | 1.36 | 446 | 490 | 0.21 | 0.19 | 310 | 3.85 | +| large (1 GiB) | write | 40 | 10 | 71 | 95 | 1.34 | 802 | 778 | 0.12 | 0.12 | 310 | 3.89 | +| large (1 GiB) | write | 40 | 40 | 68 | 96 | 1.40 | 809 | 772 | 0.12 | 0.12 | 310 | 3.85 | +| large (1 GiB) | put | 8 | 10 | 180 | 255 | 1.42 | 463 | 481 | 0.55 | 0.53 | 644 | 3.56 | +| large (1 GiB) | put | 8 | 40 | 185 | 250 | 1.36 | 472 | 473 | 0.53 | 0.53 | 643 | 3.58 | +| large (1 GiB) | put | 40 | 10 | 248 | 268 | 1.08 | 744 | 793 | 0.36 | 0.34 | 1,275 | 3.58 | +| large (1 GiB) | put | 40 | 40 | 250 | 261 | 1.04 | 767 | 768 | 0.34 | 0.34 | 1,322 | 3.60 | +| xlarge (6 GiB) | write | 8 | 10 | 69 | 97 | 1.40 | 534 | 551 | 0.18 | 0.18 | 312 | 22.88 | +| xlarge (6 GiB) | write | 8 | 40 | 65 | 97 | 1.50 | 531 | 538 | 0.18 | 0.18 | 312 | 22.97 | +| xlarge (6 GiB) | write | 40 | 10 | 67 | 96 | 1.44 | 1,066 | 1,037 | 0.09 | 0.09 | 312 | 22.66 | +| xlarge (6 GiB) | write | 40 | 40 | 68 | 96 | 1.43 | 1,062 | 1,034 | 0.09 | 0.09 | 312 | 22.69 | +| xlarge (6 GiB) | copy | 8 | 10 | 35 | 50 | 1.42 | 266 | 369 | 0.19 | 0.14 | 211 | 0.19 | +| xlarge (6 GiB) | copy | 8 | 40 | 35 | 47 | 1.35 | 264 | 326 | 0.18 | 0.15 | 211 | 0.19 | +| xlarge (6 GiB) | copy | 40 | 10 | 32 | 51 | 1.58 | 1,438 | 1,478 | 0.04 | 0.03 | 212 | 0.19 | +| xlarge (6 GiB) | copy | 40 | 40 | 33 | 53 | 1.58 | 1,753 | 1,643 | 0.03 | 0.03 | 211 | 0.19 | + +### Many objects + +Throughput and median total time: + +| Operation | Concurrency | sync (objects/s) | aio (objects/s) | s3fs (objects/s) | s3fs / sync | s3fs / aio | sync (s) | aio (s) | s3fs (s) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| cat | 8 | 9.1 | 218.4 | 143.3 | 15.73 | 0.66 | 109.79 | 4.58 | 6.98 | +| cat | 40 | 9.5 | 457.8 | 184.7 | 19.53 | 0.40 | 105.72 | 2.18 | 5.41 | +| get | 8 | 20.3 | 159.7 | 180.3 | 8.87 | 1.13 | 49.17 | 6.26 | 5.55 | +| get | 40 | 20.3 | 286.0 | 205.2 | 10.13 | 0.72 | 49.35 | 3.50 | 4.87 | +| put | 8 | 16.4 | 117.8 | 115.4 | 7.03 | 0.98 | 60.95 | 8.49 | 8.67 | +| put | 40 | 15.0 | 416.4 | 222.0 | 14.76 | 0.53 | 66.51 | 2.40 | 4.50 | +| pipe | 8 | 15.7 | 118.6 | 125.4 | 7.99 | 1.06 | 63.73 | 8.43 | 7.97 | +| pipe | 40 | 15.5 | 484.1 | 235.4 | 15.22 | 0.49 | 64.67 | 2.07 | 4.25 | +| copy | 8 | 6.5 | 90.4 | 86.6 | 13.24 | 0.96 | 152.78 | 11.06 | 11.54 | +| copy | 40 | 6.9 | 345.1 | 412.7 | 59.49 | 1.20 | 144.17 | 2.90 | 2.42 | +| rm | 8 | 1,250.7 | 1,290.0 | 1,212.5 | 0.97 | 0.94 | 0.80 | 0.78 | 0.82 | +| rm | 40 | 1,265.6 | 1,284.4 | 1,239.0 | 0.98 | 0.96 | 0.79 | 0.78 | 0.81 | +| find | 8 | 7,753.8 | 7,013.2 | 7,992.6 | 1.03 | 1.14 | 12.90 | 14.26 | 12.51 | +| find | 40 | 6,909.8 | 6,963.2 | 7,025.1 | 1.02 | 1.01 | 14.47 | 14.36 | 14.23 | +| info | 8 | 554.7 | 555.5 | 604.9 | 1.09 | 1.09 | 1.80 | 1.80 | 1.65 | +| info | 40 | 888.6 | 828.9 | 1,199.4 | 1.35 | 1.45 | 1.13 | 1.21 | 0.83 | + +Process cost: + +| Operation | Concurrency | sync threads | aio threads | s3fs threads | sync peak RSS (MiB) | aio peak RSS (MiB) | s3fs peak RSS (MiB) | sync CPU (s) | aio CPU (s) | s3fs CPU (s) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| cat | 8 | 11 | 20 | 13 | 1,182 | 1,190 | 1,222 | 5.30 | 5.28 | 5.65 | +| cat | 40 | 11 | 52 | 13 | 1,183 | 1,217 | 1,263 | 5.26 | 6.46 | 5.41 | +| get | 8 | 11 | 20 | 13 | 185 | 201 | 220 | 6.68 | 6.68 | 4.79 | +| get | 40 | 11 | 52 | 13 | 185 | 258 | 253 | 6.62 | 7.90 | 4.80 | +| put | 8 | 11 | 20 | 13 | 189 | 225 | 247 | 5.17 | 5.17 | 4.40 | +| put | 40 | 11 | 52 | 13 | 189 | 379 | 369 | 5.20 | 5.93 | 4.38 | +| pipe | 8 | 11 | 20 | 13 | 188 | 209 | 240 | 4.83 | 4.88 | 4.30 | +| pipe | 40 | 11 | 52 | 13 | 188 | 299 | 345 | 4.81 | 5.59 | 4.17 | +| copy | 8 | 11 | 20 | 13 | 187 | 195 | 215 | 2.80 | 2.72 | 1.96 | +| copy | 40 | 11 | 52 | 13 | 187 | 221 | 223 | 2.81 | 2.89 | 1.81 | +| rm | 8 | 43 | 44 | 44 | 211 | 211 | 237 | 0.04 | 0.04 | 0.05 | +| rm | 40 | 43 | 44 | 44 | 210 | 211 | 238 | 0.04 | 0.04 | 0.05 | +| find | 8 | 11 | 13 | 13 | 304 | 305 | 342 | 6.17 | 6.20 | 5.87 | +| find | 40 | 11 | 13 | 13 | 304 | 305 | 342 | 6.18 | 6.20 | 5.87 | +| info | 8 | 19 | 20 | 13 | 192 | 193 | 213 | 1.17 | 1.28 | 0.85 | +| info | 40 | 51 | 52 | 13 | 218 | 218 | 221 | 1.41 | 1.49 | 0.82 |