Skip to content

Record the filesystem comparison with s3fs (GH-1318) #1321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
laughingman7743 merged 3 commits into from
Oct 24, 2026
Merged

Record the filesystem comparison with s3fs (GH-1318) #1321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3bcddf6
Record the filesystem comparison with s3fs (GH-1318)
laughingman7743 Oct 24, 2026
047c8e4
Name the scales and compare matching rates in the s3fs results row (G…
laughingman7743 Oct 24, 2026
83a49ef
Scope block sizes, download parallelism, and thread counts in the s3f…
laughingman7743 Oct 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion benchmarks/RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@ Compare entries only when they used the same instance type, architecture, and sc
| 2026-10-13 | `437276e1` | 4.1.0 release commit, filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-13-437276e1-filesystem](history/2026-10-13-437276e1-filesystem.md) |
| 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file) | `r7g.2xlarge` (arm64), fleet of 10 | Spot | medium, large, xlarge, xxlarge | [2026-10-19-9e99caa8](history/2026-10-19-9e99caa8.md) |
| 2026-10-19 | `9e99caa8` | 4.2.0 (tagged on `baee716c`, which differs only in one test file), filesystem suite | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (objects) | [2026-10-19-9e99caa8-filesystem](history/2026-10-19-9e99caa8-filesystem.md) |
| 2026-10-23 | `7b20b499` | unreleased (after 4.2.1), filesystem and objects suites against s3fs | `c8gn.2xlarge` (arm64), one host | Spot | small, medium, large, xlarge (single objects); medium (object set) | [2026-10-23-7b20b499-filesystem](history/2026-10-23-7b20b499-filesystem.md) |

## Recommendations

These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8gn.2xlarge` hosts, and, where a row says so, from the 4.0.0 run; the entries give the measured cases and their limits.
These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8gn.2xlarge` hosts, and, where a row says so, from the 4.0.0 run and the s3fs comparison of `7b20b499`; the entries give the measured cases and their limits.

| Need | Use | Measured |
| --- | --- | --- |
Expand All @@ -36,3 +37,4 @@ These recommendations come from the runs of `9e99caa8` on `r7g.2xlarge` and `c8g
| A Wrangler-like pandas read | PandasCursor | 10 million flat rows in 19.8 seconds through UNLOAD and 84.2 seconds through CSV, against 23.4 and 108.9 seconds for Wrangler |
| Asynchronous code | The native asyncio cursors, or the synchronous and ThreadPool APIs offloaded with `asyncio.to_thread()` or an executor; synchronous calls block the event loop while they poll | Single-query timings of the three APIs differed by a median of 1.7% in the 4.0.0 run |
| S3 transfers in asyncio applications | `AioS3FileSystem`; PyAthena keeps `S3FileSystem` registered for fsspec's `s3` and `s3a` protocols, and making `AioS3FileSystem` the default is to be revisited when boto3 supports asyncio natively, because it still runs the blocking boto3 calls in threads | With 40 workers on 8 vCPUs, `AioS3FileSystem` wrote 1 GiB and 6 GiB objects at 0.97 to 1.00 times and copied 6 GiB at 1.01 to 1.05 times the throughput of `S3FileSystem`, and read 1 GiB at 1.00 times; with 8 workers, it copied 6 GiB at 1.36 to 1.49 times (#1159) |
| Transfers of many objects with one call (`cat`, `get`, `put`, or `copy` with a list of paths, or `pipe` with a dictionary of paths and bytes) | `AioS3FileSystem` with `batch_size`; the synchronous `S3FileSystem` transfers one object at a time | In the s3fs comparison of `7b20b499`, 1,000 objects of 1 MiB at 286 to 484 objects per second with a batch size of 40, against 9.5 to 20.3 for `S3FileSystem` and 184.7 to 235.4 for s3fs, except copies, at 345.1 against 412.7 for s3fs |
Loading
Loading