Repository navigation
chore(bench): add directory-fsync benchmark harness - #13
Open
reidkaufmann wants to merge 1 commit into
Open
reidkaufmann wants to merge 1 commit into
reidkaufmann wants to merge 1 commit into
Conversation
Package the LocalFileSystem directory-fsync micro-benchmark as a reusable tool under bench/dirsync (not a workspace member): - src/main.rs: driver; ops put/create/delete/copy/rename/multipart over steady/shared/cold/spread layouts, one CSV row per run. Cargo features `ext-trait` (ObjectStoreExt, >= 0.13) and `with-fsync` gate API differences; copy rotates over --copy-sources sources per writer to stay under ext4's 65000-link limit with fsync off. - build-variants.sh NAME=REV ...: worktree + generated manifest + own CARGO_TARGET_DIR per rev; features chosen from the rev's source. - run-matrix.sh: the matrix (default 24 cells, per-round order rotation, --only/--cell subsets, --sync-between-runs, --rotation cycle) and the strace syscall-count pass; file systems from --fs or BENCH_ROOT_<FS>. - aggregate.py: paired within-round ratios (median, min-max) and full tables; variants and baseline taken from the CSV. - README.md: method, requirements, quick start, reading results, pitfalls. Motivating issue: apache#888
There was a problem hiding this comment.
🟡 Changes recommended
Unsafe deletion-path validation can erase unrelated data, while zero derived operation budgets can turn short tracing runs into unbounded workloads that fill the benchmark filesystem.
7 open issues
Clamp the derived multipart operation budget above zero · New Resolve and validate the deletion target before recursive removal · New Add run IDs to prevent cross-invocation result pairing · New Do not associate stale CSV rows with failed traced runs · New Validate minimum depths or keep writer IDs in generated names · New Distribute the operation cap exactly across concurrent writers · New Include cleanup and max-ops in CSV rows and aggregation keys · New
What changed in this PR
Adds a standalone benchmark harness for comparing LocalFileSystem directory-fsync designs across object_store revisions, without changing the root build, tests, or CI.
Changes:
- Builds revision-specific drivers and measures throughput and latency through the public
ObjectStoreAPI. - Runs configurable workload matrices with variant rotation, a no-fsync baseline, and optional syscall tracing.
- Produces paired-round comparisons and documents setup, interpretation, and measurement pitfalls.
| File | Description |
|---|---|
bench/dirsync/src/main.rs |
Implements workloads, preparation, timing, and CSV output. |
bench/dirsync/run-matrix.sh |
Runs workload matrices and optional strace passes. |
bench/dirsync/README.md |
Documents usage, requirements, and benchmark limitations. |
bench/dirsync/Cargo.toml |
Defines the standalone driver and compatibility features. |
bench/dirsync/build-variants.sh |
Builds drivers against separate revision worktrees. |
bench/dirsync/aggregate.py |
Generates paired ratios and summary tables from CSV results. |
bench/dirsync/.gitignore |
Excludes generated builds, results, and lockfiles. |
Review effort: Balanced
| for cell in "${strace_cells[@]}"; do | ||
| read -r op depth state size conc cleanup <<<"$cell" | ||
| ops=$STRACE_OPS | ||
| if [ "$op" = multipart ]; then ops=$(( STRACE_OPS / 40 )); fi |
| let a = Arc::new(parse()); | ||
| let root = std::path::PathBuf::from(&a.root); | ||
| assert!( | ||
| root.components().count() >= 3, |
| for r in csv.DictReader(f): | ||
| cells, variants = per_fs.setdefault(r["fs"], (defaultdict(lambda: defaultdict(dict)), [])) | ||
| key = (r["op"], int(r["depth"]), r["state"], int(r["size"]), int(r["conc"])) | ||
| cells[key][r["variant"]][int(r["round"])] = r |
Comment on lines
+226
to
+233
| run strace -c -f -e "trace=$TRACE" -o "$tmp" "${BIN[$label]}" --root "$root/strace" --fs "$fs" \ | ||
| --variant "$label" --fsync "${FSYNC[$label]}" --op "$op" --depth "$depth" --state "$state" \ | ||
| --size "$size" --conc "$conc" --cleanup "$cleanup" \ | ||
| --secs 1000 --warmup 0 --max-ops "$ops" --round 0 --csv "$csv" $EXTRA \ | ||
| || echo "FAILED strace $fs $label '$cell'" | ||
| if [ $DRY -eq 0 ]; then | ||
| { tail -n 1 "$csv"; cat "$tmp"; echo; } >> "$sum" | ||
| fi |
Comment on lines
+151
to
+152
| assert!(args.conc >= 1, "--conc must be >= 1"); | ||
| assert!(args.copy_sources >= 1, "--copy-sources must be >= 1"); |
| if max_ops == 0 { | ||
| u64::MAX | ||
| } else { | ||
| (max_ops / conc as u64).max(1) |
| .open(path) | ||
| .unwrap(); | ||
| if new && a.csv != "/dev/stdout" { | ||
| writeln!(f, "fs,variant,op,depth,state,size,conc,round,ops,wall_s,ops_per_s,p50_us,p90_us,p99_us,p999_us,max_us,mean_us,load1_start,load1_end,fsync").unwrap(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Benchmark harness for the durable-write designs of
LocalFileSystem(with_fsync(true)), motivated by apache#888. It compares object_store git revs on the publicObjectStoreAPI: throughput, latency and (optionally, under strace) syscalls and fsyncs per operation.It produced the comparisons in #7 (registry #5 / #6 vs the directory-handle cache #7) and can drive other revs, e.g. the 0.12.5 mirrors #11 and #12.
Contents
bench/dirsync/build-variants.sh NAME=REV ...: one driver binary per rev, each in its own worktree and target dir.bench/dirsync/run-matrix.sh: the 24 default cells (put/create/delete/copy/rename/multipart; steady, shared, cold and spread layouts; 1/8/64 writers), variant order rotated per round,--fsync offfloor,--sync-between-runs,--strace.bench/dirsync/aggregate.py: paired per-round ratios (median, min-max) and full tables.Not a workspace member; the root build, tests and CI are unchanged.
Checks: fmt, clippy, shellcheck, a smoke run with two revs plus the floor (including strace). Not run here: RAT and typos (tools not installed), a full multi-round matrix with these scripts.