Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions bench/dirsync/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# build-variants.sh / run-matrix.sh defaults
work/
out/
# cargo, when built in place
target/
Cargo.lock
47 changes: 47 additions & 0 deletions bench/dirsync/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Standalone crate (not a workspace member). `build-variants.sh` copies this manifest per
# variant with the `object_store` path rewritten to that variant's worktree; this copy
# builds against the enclosing checkout so that `cargo fmt`/`cargo clippy` work here.

[package]
name = "dirsync-bench"
version = "0.1.0"
edition = "2021"
publish = false

[workspace]

[[bin]]
name = "dirsync-bench"
path = "src/main.rs"

[features]
default = ["ext-trait", "with-fsync"]
# object_store >= 0.13: `put`/`put_multipart` live on `ObjectStoreExt`.
ext-trait = []
# The rev has `LocalFileSystem::with_fsync` (fork branches, upstream >= 0.14).
with-fsync = []

[dependencies]
object_store = { path = "../..", default-features = false, features = ["fs"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
hdrhistogram = { version = "7", default-features = false }

[profile.release]
debug = false
159 changes: 159 additions & 0 deletions bench/dirsync/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Directory-fsync micro-benchmark for `LocalFileSystem`

Compares object_store git revs on the write paths of `LocalFileSystem` with `with_fsync(true)`:
how many directory fsyncs and other syscalls each design issues per operation, and what that costs
in throughput and latency on a given file system. Motivating issue:
[apache/arrow-rs-object-store#888](https://github.com/apache/arrow-rs-object-store/issues/888)
(durable directory creation / fsync of parent directories).

This crate is not a workspace member; the repository's own `cargo build/test/clippy` ignore it.

## What it measures

One **run** = one binary (`src/main.rs`) built against one object_store rev, one operation, one
layout, `conc` tokio tasks each looping sequentially through the public `ObjectStore` API. Latency
is measured around each `await`. A run prepares a fresh store root, does a discarded warm-up
(0.7 s), then a timed phase (3 s, or until `--max-ops` in total), appends one CSV row, and removes
the root (untimed).

Operations (`--op`): `put` (overwrite), `create` (`PutMode::Create`, unique names), `delete` of
pre-populated objects each alone in its leaf directory (with `--cleanup on` every delete also
removes that directory), `copy` (hard link into a sibling directory, rotating over
`--copy-sources` sources per writer), `rename` of pre-populated sources into a sibling directory,
`multipart` (`put_multipart`, `--parts` x `put_part(--size)`, `complete`).

Layouts (`--state`): `steady` (one pre-created, fsynced chain per writer), `shared` (one chain for
all writers), `cold` (every op creates a fresh chain of `--depth` directories), `spread`
(`{prefix}/n1/pt_wal/{file}`, prefix one of 4096 hashed values per op; 12288 directories
pre-created and `sync`ed, a working set larger than a 4096-entry cache).

A **cell** is `op depth state size conc max-ops cleanup`; the default list in `run-matrix.sh` is
24 cells (put steady/shared/cold at 1/8/64 writers, depth-6 cold, spread, create, delete, copy,
rename, multipart). A **variant** is a rev built by `build-variants.sh`; a variant may also be run
with `--fsync off` as the no-durability floor without a separate build.

The optional **strace pass** runs each variant over a fixed op count under
`strace -c -f -e trace=fsync,fdatasync,openat,...` and records the syscall table next to the CSV
row, so counts per op can be compared between designs (counts, not timings).

## Requirements

- Linux (the driver reads `/proc/loadavg` and calls `sync`; the directory-handle designs are
Linux-only).
- Rust toolchain of this repository (`rust-toolchain.toml`), git worktrees, bash >= 4, python3.
- For the strace pass: `strace` >= 6.6 (`-c -f` over a multi-threaded tokio runtime; the
reference runs used 6.19). `run-matrix.sh` warns below 6.6.
- Two file systems to compare file systems (for example an xfs and an ext4 mount); a single
directory on `$TMPDIR` is enough for a smoke test. The bench root is **removed** before and
after every run; never point it at a directory with data.
- Disk: the multipart cell writes 800 x 10 MiB per run; the delete cells pre-populate 125k files.

## Quick start

```sh
cd bench/dirsync

# 1. Build one driver binary per rev (any rev resolvable in this repository; fetch first).
# Worktrees, generated manifests and targets go under $WORK (default ./work).
./build-variants.sh base=influxdata/base-object-store-0132 \
dirfd=influxdata/local-dirfd-cache-0132

# 2. Run a subset: one round, two cells, the second variant also as the no-fsync floor.
./run-matrix.sh --fs tmp=/tmp/dirsync-bench --out ./out --rounds 1 \
--only '^put 3 steady 4096 (8|64) ' base dirfd floor=dirfd:nofsync

# 3. Aggregate (paired ratios against the first variant; --baseline / --pairs to change).
./aggregate.py both out/ops-tmp.csv
```

Full matrix on two file systems with the strace pass, 5 rounds:

```sh
export BENCH_ROOT_XFS=/mnt/xfs/dirsync-bench BENCH_ROOT_EXT4=/mnt/ext4/dirsync-bench
./run-matrix.sh --out ./out --rounds 5 --strace base dirfd floor=dirfd:nofsync | tee out/run.log
./aggregate.py paired out/ops-xfs.csv out/ops-ext4.csv
```

Options worth knowing (`run-matrix.sh --help` lists all): `--rotation cycle` (every variant runs
first in turn, needed for order-sensitive cells), `--sync-between-runs[=S]` (`sync` + pause before
every run), `--cell "..."` (own cell list), `--strace-only`, `--extra "--copy-sources 64"`,
`--dry-run`.

`build-variants.sh` picks the driver's cargo features from the rev's source: `ext-trait` when
`ObjectStoreExt` exists (object_store >= 0.13), `with-fsync` when `LocalFileSystem::with_fsync`
exists (this fork's 0.12.5 and 0.13.2 branches, upstream >= 0.14). A rev without `with_fsync`
builds but accepts only `--fsync off`. Checked against 0.12.5 and 0.13.2 revs of this fork.

## Reading the results

- `ops-<fs>.csv`: one row per run with `ops`, `wall_s`, `ops_per_s`, latency percentiles in
microseconds, the 1-minute load average at start and end, and `fsync`.
- `aggregate.py paired`: per cell, for each pair A/B, the ratio of ops/s **within the same round**,
then the median and min-max over rounds, and the per-variant median p50. This is the statistic
to read: it cancels the drift between rounds that a shared host produces.
- `aggregate.py full`: medians over rounds with min-max of ops/s, p50/p99/p999, each variant's
ratio to the baseline's median, plus two lines per file system: the load average range per
round and the per-round normalized throughput (every run divided by its cell+variant median
over rounds). A round whose normalized median is far from 1.0 was disturbed as a whole; a wide
p10-p90 means individual cells were.
- `strace-<fs>.txt`: per traced run the CSV row and the `strace -c` table; divide the counts by
`max-ops` for per-op syscalls. The driver's own preparation and teardown are included
(`unlinkat` from `remove_dir_all`, `openat`/`mkdir` from pre-population, `sync`).

## Pitfalls

- **Only paired within-round ratios are meaningful.** Absolute numbers depend on other tenants of
the disk; in past runs the same cells moved 2-3x between days while paired ratios held. Read the
medians over rounds with the min-max as the warning; a single round's ratio can be an outlier
when interference changed in the middle of a cell.
- **Position effect in the cell.** Variants run back to back on the same directory; the first
variant of a cell writes into a file system with no dirty data from the previous variant's
teardown. Multipart (two 5 MiB parts, 2 fsyncs per op) at 1 writer gave the first variant about
2x, and a no-fsync variant leaves ~8 GB of dirty data behind for whoever follows it. The
default rotation (forward in odd rounds, reversed in even rounds) does not give middle
variants a first position; use `--rotation cycle` and `--sync-between-runs` for such cells,
or treat them as not interpretable.
- **Host noise sampling.** The CSV carries the load average at start and end of each run; the
`full` aggregate prints the per-round load range and normalized throughput. Processes outside
the container are not visible from inside; a `sync`-heavy neighbour shows up only as longer
fsyncs. For a long matrix, log `top` or `/proc/loadavg` every minute alongside (`run.log`
timestamps let you align them).
- **Copy with fsync off and ext4's link limit.** `copy` is a hard link, and ext4 allows 65000
links per inode (`EMLINK`). A no-fsync variant made ~270k copies/s; with one source per
writer that exceeds the limit within one 3 s run at 8 writers. The driver therefore rotates
over `--copy-sources` sources per writer (default 16); raise it (`--extra "--copy-sources 64"`)
for no-fsync copy at 1 writer, or skip that cell. xfs has no comparable limit.
- **`delete` and `rename` need `--max-ops`** (pre-populated sources); delete at 100000 ops
finishes in well under 3 s on fast disks, so its timed phase is shorter than the others.
- **bash `set -u`** in scripts of this kind: do not reference a variable in the same `local`
statement that declares it (`local a=$1 b=$a`); declare one per line. The scripts here follow
that and pass shellcheck.
- `ext4` vs `xfs` is only like-for-like when both sit on comparable devices and fill levels; read
each file system's ratios on their own.

## Files

- `src/main.rs`, `Cargo.toml`: the driver (path dependency on `../..` so `cargo fmt`/`clippy`
work here; `build-variants.sh` rewrites that path per variant).
- `build-variants.sh`: worktree + build per `NAME=REV` into `$WORK/{src,build,target}/NAME`.
- `run-matrix.sh`: matrix driver, strace pass; writes `$OUT/ops-<fs>.csv`, `strace-<fs>.{txt,csv}`.
- `aggregate.py`: `paired` / `full` / `both` markdown tables from the CSVs.
184 changes: 184 additions & 0 deletions bench/dirsync/aggregate.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
#!/usr/bin/env python3
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Aggregate run-matrix.sh CSV files (ops-<fs>.csv) into markdown tables.

`paired`: per cell, the ratio of ops/s between two variants measured in the same round, then the
median and min-max of those ratios over rounds. `full`: per cell and variant, median over rounds
of ops/s (with min-max) and of the per-round p50/p99/p999 latencies, plus per-round load and
normalized-throughput lines that show whether a round was disturbed.

usage: aggregate.py [--baseline V] [--pairs A/B,...] paired|full|both CSV...
--baseline V variant every other variant is paired against (default: the first variant in
the file)
--pairs ... explicit A/B pairs instead of the default; variants are the CSV `variant` labels
"""
import argparse
import csv
import statistics as st
from collections import defaultdict

OPS = ["put", "create", "delete", "copy", "rename", "multipart"]
STATES = {"steady": 0, "shared": 1, "cold": 2, "spread": 3}


def load(path):
"""fs -> (cells: key -> variant -> round -> row, variants in order of first appearance)."""
per_fs = {}
with open(path, newline="") as f:
for r in csv.DictReader(f):
cells, variants = per_fs.setdefault(r["fs"], (defaultdict(lambda: defaultdict(dict)), []))
key = (r["op"], int(r["depth"]), r["state"], int(r["size"]), int(r["conc"]))
cells[key][r["variant"]][int(r["round"])] = r
if r["variant"] not in variants:
variants.append(r["variant"])
return per_fs


def med(rows, col):
return st.median(float(r[col]) for r in rows)


def fmt_us(v):
return f"{v / 1000:.2f}" if v >= 1000 else f"{v / 1000:.3f}"


def cell_order(k):
return (OPS.index(k[0]) if k[0] in OPS else len(OPS), k[3], STATES.get(k[2], 9), k[1], k[4])


def size_s(size):
return {4096: "4K", 1048576: "1M", 5242880: "5M"}.get(size, str(size))


def paired(by, a, b, col="ops_per_s"):
rounds = sorted(set(by.get(a, {})) & set(by.get(b, {})))
if not rounds:
return None
ratios = [float(by[a][r][col]) / float(by[b][r][col]) for r in rounds]
return st.median(ratios), min(ratios), max(ratios), len(ratios)


def paired_table(fs, cells, variants, pairs):
out = [f"### {fs}: paired ratios of ops/s (per round, same cell; median over rounds, min-max)\n"]
head = ["op", "depth", "state", "size", "conc"] + [f"{a}/{b}" for a, b in pairs]
head += [f"{a} p50 / {b} p50" for a, b in pairs[:1]] + [f"{v} p50 ms" for v in variants]
out.append("| " + " | ".join(head) + " |")
out.append("|" + "---|" * len(head))
for key in sorted(cells, key=cell_order):
op, depth, state, size, conc = key
by = cells[key]
cols = []
for a, b in pairs:
p = paired(by, a, b)
cols.append(f"{p[0]:.3f} ({p[1]:.2f}-{p[2]:.2f}) n={p[3]}" if p else "")
for a, b in pairs[:1]:
p50 = paired(by, a, b, "p50_us")
cols.append(f"{p50[0]:.3f}" if p50 else "")
for v in variants:
rows = list(by.get(v, {}).values())
cols.append(fmt_us(med(rows, "p50_us")) if rows else "")
out.append(f"| {op} | {depth} | {state} | {size_s(size)} | {conc} | " + " | ".join(cols) + " |")
return "\n".join(out)


def table(fs, cells, variants, baseline):
out = [f"### {fs}: ops/s (median over rounds, min-max) and latency ms "
"(median over rounds of per-round p50/p99/p999)\n"]
out.append(f"| op | depth | state | size | conc | var | rounds | ops/s med (min-max) | p50 ms | p99 ms | "
f"p999 ms | vs {baseline} ops/s |")
out.append("|---|---|---|---|---|---|---|---|---|---|---|---|")
for key in sorted(cells, key=cell_order):
op, depth, state, size, conc = key
by = cells[key]
base = med(by[baseline].values(), "ops_per_s") if by.get(baseline) else None
for v in variants:
rows = list(by.get(v, {}).values())
if not rows:
continue
tp = [float(r["ops_per_s"]) for r in rows]
m = st.median(tp)
rb = "1" if v == baseline else (f"{m / base:.3f}" if base else "")
out.append(
f"| {op} | {depth} | {state} | {size_s(size)} | {conc} | {v} | {len(rows)} | "
f"{m:.0f} ({min(tp):.0f}-{max(tp):.0f}) | {fmt_us(med(rows, 'p50_us'))} | "
f"{fmt_us(med(rows, 'p99_us'))} | {fmt_us(med(rows, 'p999_us'))} | {rb} |"
)
return "\n".join(out)


def rows_of(cells):
for by in cells.values():
for rounds in by.values():
yield from rounds.values()


def loads(cells):
"""round -> (min, max) of the 1-minute load average at run start."""
by_round = defaultdict(list)
for r in rows_of(cells):
try:
by_round[int(r["round"])].append(float(r["load1_start"]))
except ValueError:
pass
return {k: (min(v), max(v)) for k, v in sorted(by_round.items()) if v}


def normalized(cells):
"""Each run divided by its cell+variant median over rounds; median (p10-p90) per round."""
norm = defaultdict(list)
for by in cells.values():
for rounds in by.values():
m = st.median(float(r["ops_per_s"]) for r in rounds.values())
for rd, r in rounds.items():
norm[rd].append(float(r["ops_per_s"]) / m)
return ", ".join(
f"r{rd}: {st.median(v):.3f} ({sorted(v)[len(v) // 10]:.2f}-{sorted(v)[len(v) * 9 // 10]:.2f})"
for rd, v in sorted(norm.items())
)


def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--baseline")
ap.add_argument("--pairs", help="comma-separated A/B pairs")
ap.add_argument("mode", choices=["paired", "full", "both"])
ap.add_argument("csv", nargs="+")
args = ap.parse_args()
for path in args.csv:
for fs, (cells, variants) in load(path).items():
baseline = args.baseline or variants[0]
if baseline not in variants:
raise SystemExit(f"{path}: baseline {baseline} not among variants {variants}")
if args.pairs:
pairs = [tuple(p.split("/")) for p in args.pairs.split(",")]
else:
pairs = [(v, baseline) for v in variants if v != baseline]
if args.mode in ("paired", "both"):
print(paired_table(fs, cells, variants, pairs))
print()
if args.mode in ("full", "both"):
print(table(fs, cells, variants, baseline))
print()
print(f"{fs}: 1-min load average at run start, per round (min-max over the round's runs):",
", ".join(f"r{k}: {lo:.1f}-{hi:.1f}" for k, (lo, hi) in loads(cells).items()))
print(f"{fs}: per-round normalized throughput median (p10-p90):", normalized(cells))
print()


if __name__ == "__main__":
main()
Loading