Skip to content

chore(bench): add directory-fsync benchmark harness - #13

Open
reidkaufmann wants to merge 1 commit into
base-object-store-0132from
bench/dirsync
Open

reidkaufmann wants to merge 1 commit into
base-object-store-0132from
bench/dirsync

Conversation

@reidkaufmann

Copy link
Copy Markdown
Collaborator

Benchmark harness for the durable-write designs of LocalFileSystem (with_fsync(true)), motivated by apache#888. It compares object_store git revs on the public ObjectStore API: throughput, latency and (optionally, under strace) syscalls and fsyncs per operation.

It produced the comparisons in #7 (registry #5 / #6 vs the directory-handle cache #7) and can drive other revs, e.g. the 0.12.5 mirrors #11 and #12.

Contents
  • bench/dirsync/build-variants.sh NAME=REV ...: one driver binary per rev, each in its own worktree and target dir.
  • bench/dirsync/run-matrix.sh: the 24 default cells (put/create/delete/copy/rename/multipart; steady, shared, cold and spread layouts; 1/8/64 writers), variant order rotated per round, --fsync off floor, --sync-between-runs, --strace.
  • bench/dirsync/aggregate.py: paired per-round ratios (median, min-max) and full tables.
  • README: requirements, quick start, how to read results, pitfalls (order effects, host noise, ext4 link limit).

Not a workspace member; the root build, tests and CI are unchanged.

Checks: fmt, clippy, shellcheck, a smoke run with two revs plus the floor (including strace). Not run here: RAT and typos (tools not installed), a full multi-round matrix with these scripts.

Package the LocalFileSystem directory-fsync micro-benchmark as a reusable
tool under bench/dirsync (not a workspace member):

- src/main.rs: driver; ops put/create/delete/copy/rename/multipart over
  steady/shared/cold/spread layouts, one CSV row per run. Cargo features
  `ext-trait` (ObjectStoreExt, >= 0.13) and `with-fsync` gate API
  differences; copy rotates over --copy-sources sources per writer to
  stay under ext4's 65000-link limit with fsync off.
- build-variants.sh NAME=REV ...: worktree + generated manifest + own
  CARGO_TARGET_DIR per rev; features chosen from the rev's source.
- run-matrix.sh: the matrix (default 24 cells, per-round order rotation,
  --only/--cell subsets, --sync-between-runs, --rotation cycle) and the
  strace syscall-count pass; file systems from --fs or BENCH_ROOT_<FS>.
- aggregate.py: paired within-round ratios (median, min-max) and full
  tables; variants and baseline taken from the CSV.
- README.md: method, requirements, quick start, reading results, pitfalls.

Motivating issue: apache#888
Copilot AI balanced review requested due to automatic review settings November 19, 2026 20:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unsafe deletion-path validation can erase unrelated data, while zero derived operation budgets can turn short tracing runs into unbounded workloads that fill the benchmark filesystem.

7 open issues
What changed in this PR

Adds a standalone benchmark harness for comparing LocalFileSystem directory-fsync designs across object_store revisions, without changing the root build, tests, or CI.

Changes:

  • Builds revision-specific drivers and measures throughput and latency through the public ObjectStore API.
  • Runs configurable workload matrices with variant rotation, a no-fsync baseline, and optional syscall tracing.
  • Produces paired-round comparisons and documents setup, interpretation, and measurement pitfalls.
File Description
bench/​dirsync/​src/​main.rs Implements workloads, preparation, timing, and CSV output.
bench/​dirsync/​run-matrix.sh Runs workload matrices and optional strace passes.
bench/​dirsync/​README.md Documents usage, requirements, and benchmark limitations.
bench/​dirsync/​Cargo.toml Defines the standalone driver and compatibility features.
bench/​dirsync/​build-variants.sh Builds drivers against separate revision worktrees.
bench/​dirsync/​aggregate.py Generates paired ratios and summary tables from CSV results.
bench/​dirsync/​.gitignore Excludes generated builds, results, and lockfiles.

Review effort: Balanced

for cell in "${strace_cells[@]}"; do
read -r op depth state size conc cleanup <<<"$cell"
ops=$STRACE_OPS
if [ "$op" = multipart ]; then ops=$(( STRACE_OPS / 40 )); fi
Comment thread bench/dirsync/src/main.rs
let a = Arc::new(parse());
let root = std::path::PathBuf::from(&a.root);
assert!(
root.components().count() >= 3,
for r in csv.DictReader(f):
cells, variants = per_fs.setdefault(r["fs"], (defaultdict(lambda: defaultdict(dict)), []))
key = (r["op"], int(r["depth"]), r["state"], int(r["size"]), int(r["conc"]))
cells[key][r["variant"]][int(r["round"])] = r
Comment on lines +226 to +233
run strace -c -f -e "trace=$TRACE" -o "$tmp" "${BIN[$label]}" --root "$root/strace" --fs "$fs" \
--variant "$label" --fsync "${FSYNC[$label]}" --op "$op" --depth "$depth" --state "$state" \
--size "$size" --conc "$conc" --cleanup "$cleanup" \
--secs 1000 --warmup 0 --max-ops "$ops" --round 0 --csv "$csv" $EXTRA \
|| echo "FAILED strace $fs $label '$cell'"
if [ $DRY -eq 0 ]; then
{ tail -n 1 "$csv"; cat "$tmp"; echo; } >> "$sum"
fi
Comment thread bench/dirsync/src/main.rs
Comment on lines +151 to +152
assert!(args.conc >= 1, "--conc must be >= 1");
assert!(args.copy_sources >= 1, "--copy-sources must be >= 1");
Comment thread bench/dirsync/src/main.rs
if max_ops == 0 {
u64::MAX
} else {
(max_ops / conc as u64).max(1)
Comment thread bench/dirsync/src/main.rs
.open(path)
.unwrap();
if new && a.csv != "/dev/stdout" {
writeln!(f, "fs,variant,op,depth,state,size,conc,round,ops,wall_s,ops_per_s,p50_us,p90_us,p99_us,p999_us,max_us,mean_us,load1_start,load1_end,fsync").unwrap();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants