From 78e4b604319121a414c62400af7cb2f3ada775ef Mon Sep 17 00:00:00 2001 From: Mile Druzijanic Date: Tue, 28 Jul 2026 11:27:53 +0200 Subject: [PATCH] readme simplification --- benchmarks/GET/dfly_bench/GET_reproduce.md | 77 +----------- benchmarks/GET/memtier/GET_reproduce.md | 131 +-------------------- benchmarks/SET/dfly_bench/SET_reproduce.md | 63 +--------- benchmarks/SET/memtier/SET_reproduce.md | 24 +--- 4 files changed, 16 insertions(+), 279 deletions(-) diff --git a/benchmarks/GET/dfly_bench/GET_reproduce.md b/benchmarks/GET/dfly_bench/GET_reproduce.md index 69cea79..8d9a5f6 100644 --- a/benchmarks/GET/dfly_bench/GET_reproduce.md +++ b/benchmarks/GET/dfly_bench/GET_reproduce.md @@ -1,26 +1,4 @@ -# GET benchmark (Dragonfly) with dfly_bench — reproduce & analysis - -Reproduction of the 128-byte GET workload on `m7g.8xlarge` (Graviton3, 32 vCPU) -using Dragonfly's own load generator, `dfly_bench`, instead of memtier. At the -same 160 connections, GET throughput and tail latency trade off directly against -the pipeline depth: - -- **Unthrottled, deep pipeline (120):** ~13.5M ops/s but p99.9 blows up to ~33 ms. -- **Recommended (pipeline 30, unthrottled):** **10.31M ops/s at p99.9 = 0.996 ms** — - best tail, still above 10M ops/s. -- **Pipeline 30 throttled to ~11M:** 11.82M ops/s at p99.9 = 7.67 ms. -- **Engine comparison (pipeline 30, unthrottled):** Dragonfly **9.88M ops/s** vs - valkey **1.61M** and redis **1.48M** — Dragonfly is ~6.1–6.7x the throughput at a - fraction of the latency (p99.9 1.12 ms vs 8.79 ms / 6.96 ms). - -All configs are CPU-bound on the server (Dragonfly saturates its shards); the tail -is governed by how many requests are kept in flight (`conns x pipeline`). - -## Setup - -Provisioned with `dfbench setup` (installs both memtier and dfly_bench on the -client; `dfly_bench` is built from `dragonflydb/dragonfly` at `--dfly-bench-ref`, -default `main`): +### Stateful setup: ``` dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \ @@ -34,19 +12,7 @@ Note: pick an availability zone that has `c6gn.8xlarge` capacity (AWS suggested client share a cluster placement group, so both instance types must have capacity in the chosen AZ. -| Component | Value | -| --------- | ----- | -| Server | m7g.8xlarge (arm64), Ubuntu 24.04, `--tune-network` | -| Client | c6gn.8xlarge (arm64) | -| Dragonfly build | `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz` (`i1.39.4-32-gcc0f424a`) | -| Dragonfly flags | `--num_shards 16`, `--conn_use_incoming_cpu` | -| Load generator | `dfly_bench` (ref `main`) | -| Value size | 128 B | -| Keyspace | 100,000,000 keys (`--key_maximum`) | -| Connections | 160 (`--dfly-bench-threads 32` x `--dfly-bench-conns 5`) | -| Key distribution | `U` (uniform, ≈ memtier random pattern `R`) | - -## Test command (recommended config: pipeline 30, unthrottled) +### Test run: ``` dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command GET \ @@ -57,17 +23,8 @@ dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --comman --dfly-bench-key-dist U --dfly-bench-qps 0 ``` -`--dfly-bench-qps 0` is closed-loop max load (coordinated omission). To pin throughput near a -target rate instead, set `--dfly-bench-qps` per connection = target_total / 160 (e.g. -`--dfly-bench-qps 68750` ≈ 11M ops/s total). Redis/Valkey are the preinstalled builds started -with `io_threads=32` (auto); only Dragonfly is reinstalled from `--dragonfly_version`. -## Engine comparison (dragonfly vs redis vs valkey) - -Latest 3-engine run (benchmark `20260721-114528`, pipeline 30, `--qps 0`, 160 -connections, 60s trial, value size 128B). Redis 8.8.0 and Valkey 9.1.0 with -`io_threads=32`; each engine is preloaded to 100M keys by dfly_bench, then measured -back-to-back with a 30s cooldown between engines. +### Expected results: | Engine | Throughput (median) | p50 | p99 | p99.9 | Avg latency | | ------ | ------------------- | --- | --- | ----- | ----------- | @@ -75,15 +32,9 @@ back-to-back with a 30s cooldown between engines. | valkey | 1.61M ops/s | 2.544 ms | 7.885 ms | 8.791 ms | 2.959 ms | | redis | 1.48M ops/s | 3.250 ms | 6.076 ms | 6.956 ms | 3.218 ms | -**Takeaway:** at the same client config Dragonfly delivers ~6.1x Valkey and ~6.7x -Redis throughput while keeping p99.9 near 1 ms (vs ~7–9 ms). Redis/Valkey are -single-shard event loops that cannot spread a uniform GET load across all 32 cores -the way Dragonfly's 16 shards do, so they saturate a small number of threads well -below the box's capacity. (Dragonfly here reads 9.88M ops/s vs the 10.31M in the -pipeline sweep below — the two runs are on different EC2 instances from separate -`dfbench setup`s, so expect ~5% run-to-run variance.) -## Results (Dragonfly, pipeline sweep) + +### Results (Dragonfly, pipeline sweep) Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, 160 connections, 60s trial @@ -94,22 +45,4 @@ Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, 16 | 30 | `--dfly-bench-qps 68750` (~11M) | 11.82M ops/s | 1.064 ms | 3.669 ms | 7.666 ms | 1.173 ms | | **30** | **`--dfly-bench-qps 0`** | **10.31M ops/s** | **0.431 ms** | **0.765 ms** | **0.996 ms** | **0.442 ms** | -**Reading the sweep:** with 160 connections at max load, tail latency scales with -in-flight depth (`conns x pipeline`). Dropping the pipeline from 120 to 30 cuts -p99.9 by ~33x (33.3 ms -> 1.0 ms) while only trimming throughput ~24% (13.5M -> -10.3M), because Dragonfly is CPU-bound and 160x30 still keeps every shard busy. -Throttling pipeline-30 up toward 11M (`--dfly-bench-qps 68750`) recovers throughput but -re-inflates the tail to ~7.7 ms — still under 10 ms, if you want the extra ops/s. - -## Comparison with memtier - -The memtier reproduction (see [../memtier/GET_reproduce.md](../memtier/GET_reproduce.md)) -measured Dragonfly at **11.63M ops/s, p99.9 = 7.97 ms** with pipeline 120. dfly_bench -at pipeline 120 pushes harder (13.5M ops/s) but with a much longer tail, so the two -tools are not directly comparable at identical pipeline settings — dfly_bench's -closed-loop driver keeps deeper queues. For a memtier-like tail, use a lower -pipeline (30) or throttle with `--dfly-bench-qps`. - -Both tools' results share the same schema (this run is tagged `tool="dfly_bench"`), -so a memtier GET file and a dfly_bench GET file diff with `dfbench compare`. diff --git a/benchmarks/GET/memtier/GET_reproduce.md b/benchmarks/GET/memtier/GET_reproduce.md index 9c5e1de..e93b085 100644 --- a/benchmarks/GET/memtier/GET_reproduce.md +++ b/benchmarks/GET/memtier/GET_reproduce.md @@ -1,9 +1,4 @@ -# GET benchmark (Dragonfly) — reproduce & analysis - -## TL;DR -On `m7g.8xlarge` (Graviton3, 32 vCPU, 8 ENA queues), 128-byte **GET peaks at ~11.4–11.5M ops/s and is CPU-bound, not network-bound.** All 32 cores sit at ~0% idle while the ENA network-allowance counters never trip. This is the opposite of SET on the same box, which is ingress-bandwidth-bound at ~15 Gbps. - -## Stateful setup +### Stateful setup: ``` dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \ @@ -12,7 +7,7 @@ dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \ --engines dragonfly,redis,valkey --availability-zone us-east-1 ``` -## Test run (best config: clients=5) +### Test run: ``` dfbench run-commands --id maxqps2 --engines dragonfly,redis,valkey --command GET \ @@ -25,7 +20,7 @@ dfbench run-commands --id maxqps2 --engines dragonfly,redis,valkey --command GET --test-time 60 --warmup-time 10 --trials 1 --distinct-client-seed ``` -## Results +### Expected results: Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, pipeline 120, clients=5 (160 connections) @@ -37,8 +32,6 @@ Engine comparison (Dragonfly build `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz | valkey | 1.98M ops/s | 9.727 ms | 9.983 ms | 10.111 ms | 9.677 ms | | redis | 1.52M ops/s | 10.431 ms | 21.119 ms | 38.143 ms | 12.653 ms | -Dragonfly is **~5.9× Valkey and ~7.7× Redis** on 128B GET. Redis/Valkey are single-threaded for -command execution, so they cannot use the 32 cores the way Dragonfly's shards do. ### Dragonfly connection sweep (why clients=5) @@ -49,120 +42,4 @@ command execution, so they cannot use the 32 cores the way Dragonfly's shards do **clients=5 is the sweet spot.** Doubling connections to clients=10 is a regression (congestion collapse): throughput drops ~18% and latency ~2.5×, while server CPU utilization also drops — a -sign of too much in-flight work, not a capacity gain. Use `--memtier-clients 5`. - -### Same run with dfly_bench - -`dfbench run-commands-dfly` reproduces this GET workload with Dragonfly's own -load generator instead of memtier. dfly_bench is installed on the client at -`dfbench setup` time (alongside memtier); override the build with -`--dfly-bench-ref`. - -``` -dfbench run-commands-dfly --id maxqps2 --engines dragonfly --command GET \ - --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ - --dragonfly_num_shards 16 \ - --key-maximum 100000000 --test-time 60 --warmup-time 10 --trials 1 \ - --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 120 --dfly-bench-data-size 128 \ - --dfly-bench-key-dist U --dfly-bench-qps 0 -``` - -Notes: -- **dfly_bench self-preloads.** dfly_bench uses its own key prefix - (`--key_prefix`, default `key:`), which differs from memtier's keys, so GET's - SET preload is run by dfly_bench (sequential distribution, one connection per - thread covering the whole `--key_maximum` range) rather than reused from a - memtier preload. -- **Flag mapping:** `--dfly-bench-threads` -> `--proactor_threads`, `--dfly-bench-conns` -> - `-c` (32x5 = 160 connections), `--dfly-bench-key-dist U` approximates memtier's random - key pattern `R`, `--dfly-bench-qps 0` = closed-loop max load. -- Results use the same schema (tagged `tool="dfly_bench"`), so - `dfbench compare ` diffs the two tools. -- dfly_bench's `--json_out_file` writes a memtier-compatible latency report, so - the same parser produces the throughput / p50 / p99 / p99.9 / avg columns. - -## Bottleneck: CPU, not network (verified) - -Two independent measurements taken during the same 60s measured trial. - -### mpstat (server, aggregate over the trial) - -``` -usr=75.2 sys=8.9 irq=9.5 soft=6.3 iowait=0.0 idle=0.12 (busy 99.9%) -per-core idle across all 32 CPUs: min=0.00 avg=0.12 max=0.31 -``` - -Every core is saturated. The per-core split also confirms the tuned layout -(`--dragonfly_num_shards 16` + `--dragonfly_conn_use_incoming_cpu` + `--tune-network`): - -| Cores | Role | %usr | %soft | -| ----- | ---- | ---- | ----- | -| cpu0–15 | 16 Dragonfly shards | ~90–92% | 0% | -| cpu16–23 | connection I/O | ~57–68% | ~6–7% | -| cpu24–31 | network softirq (8 ENA RX queues) | ~48–57% | ~17–19% | - -The low 16 CPUs run shards (pure userspace), the top 8 CPUs absorb the NIC softirq — exactly the -intended shard/network separation — and nothing is left idle. - -### ethtool ENA allowance counters - -`ethtool -S ens5` before/after plus a 1 Hz time-series, split by phase: - -| Phase | egress (tx) | ingress (rx) | `bw_in_exceeded` | `bw_out_exceeded` | `pps_exceeded` | -| ----- | ----------- | ------------ | ---------------- | ----------------- | -------------- | -| Preload (SET 100M) | ~0.5 Gbps | **~15.0 Gbps** | **climbs (throttled)** | 0 | 0 | -| GET trial | ~12.2 Gbps avg (12.9 peak) | ~3.5 Gbps | **0** | **0** | **0** | - -Whole-run before→after: `bw_in_allowance_exceeded` Δ ≈ +1.8M (100% during the SET preload), -`bw_out_allowance_exceeded` Δ = 0, `pps_allowance_exceeded` Δ = 0. - -**Interpretation:** during GET the server sends ~12.2 Gbps and AWS never throttles it — there is -~2–3 Gbps of headroom versus the ~15 Gbps the NIC demonstrably sustains on ingress during the SET -preload. So GET is limited by Dragonfly CPU (idle ≈ 0), not by the network allowance. SET/writes -are the workload that hits the network wall on this instance. - -## Consequences -- **To push GET higher:** more/faster server cores (e.g. `m8g.16xlarge`, Graviton4/64 vCPU). GET - scales with CPU here, so a network-optimized box is *not* what GET needs. -- **SET/writes** are the ones that want a network-optimized instance (`m8gn.16xlarge`, up to 32 ENA - queues), since their ingress genuinely saturates the ~15 Gbps allowance. - -## Underlying memtier commands -dfbench prints each memtier invocation to stderr. For this run: - -``` -# preload (populate 100M keys with 128B values) -memtier_benchmark --server= --port=6379 --threads=32 --clients=5 \ - --requests=625000 --command="SET __key__ __data__" --command-key-pattern=P \ - --data-size=128 --pipeline=120 --key-maximum=100000000 - -# warmup (10s, discarded) -memtier_benchmark --server= --port=6379 --threads=32 --clients=5 \ - --test-time=10 --command="GET __key__" --command-key-pattern=R \ - --data-size=128 --pipeline=120 --key-maximum=100000000 --distinct-client-seed - -# measured trial (60s) -memtier_benchmark --server= --port=6379 --threads=32 --clients=5 \ - --test-time=60 --command="GET __key__" --command-key-pattern=R \ - --data-size=128 --pipeline=120 --key-maximum=100000000 --distinct-client-seed -``` - -## How to re-verify (raw captures kept) -Raw data from the analyzed run lives alongside this doc: -- `benchmarks/GET/ena/before.txt`, `after.txt`, `timeseries.txt` — ENA counters (1 Hz series). -- `benchmarks/GET/mpstat/get-mpstat.log` — per-core mpstat (1s) across the whole run. - -To reproduce the capture, in one server session (`dfbench ssh --id maxqps2 --target server`): - -``` -# ENA allowance counters, 1 Hz -IFACE=$(ip -o -4 route show to default | awk '{print $5}') # ens5 -for i in $(seq 1 150); do echo "=== ts=$(date +%s) ==="; ethtool -S $IFACE; sleep 1; done - -# per-core CPU, 1s -mpstat -P ALL 1 150 -``` - -Then launch the `run-commands` GET test above from the client machine and look at the counters -during the final 60s (the measured trial). If `%idle`≈0 and `bw_out/pps_allowance_exceeded` stay -flat, GET is CPU-bound; if `bw_out_allowance_exceeded` climbs, it has become network-bound. +sign of too much in-flight work, not a capacity gain. Use `--memtier-clients 5`. \ No newline at end of file diff --git a/benchmarks/SET/dfly_bench/SET_reproduce.md b/benchmarks/SET/dfly_bench/SET_reproduce.md index b58cc6b..9bfd5e4 100644 --- a/benchmarks/SET/dfly_bench/SET_reproduce.md +++ b/benchmarks/SET/dfly_bench/SET_reproduce.md @@ -1,21 +1,4 @@ -# SET benchmark (Dragonfly vs Redis vs Valkey) with dfly_bench — reproduce & results - -Reproduction of the 128-byte SET workload on `m7g.8xlarge` (Graviton3, 32 vCPU) -using Dragonfly's own load generator, `dfly_bench`, instead of memtier, across all -three engines at 160 connections / pipeline 30 / max load: - -- **dragonfly 7.78M ops/s** (p99.9 11.5 ms) — ~7.4x Redis and ~7.4x Valkey. -- **redis 1.06M ops/s** (p99.9 13.4 ms), **valkey 1.05M ops/s** (p99.9 15.8 ms). -- **Near-perfect memtier parity:** dfly_bench SET matches the memtier SET numbers - within ~2% (Dragonfly 7.78M vs 7.94M), because SET is ingress-bandwidth-bound at - ~15 Gbps, so both load generators hit the same network ceiling regardless of the - client-side driver. - -## Setup - -Provisioned with `dfbench setup` (installs both memtier and dfly_bench on the -client; `dfly_bench` is built from `dragonflydb/dragonfly` at `--dfly-bench-ref`, -default `main`): +### Stateful setup: ``` dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \ @@ -28,20 +11,8 @@ Note: pick an availability zone that has `c6gn.8xlarge` capacity (AWS suggested `us-east-1b/c/d/f`). Server and client share a cluster placement group, so both instance types must have capacity in the chosen AZ. -| Component | Value | -| --------- | ----- | -| Server | m7g.8xlarge (arm64), Ubuntu 24.04, `--tune-network` | -| Client | c6gn.8xlarge (arm64) | -| Dragonfly build | `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz` (`i1.39.4-32-gcc0f424a`) | -| Dragonfly flags | `--num_shards 16`, `--conn_use_incoming_cpu` | -| Redis / Valkey | Redis 8.8.0 / Valkey 9.1.0, `io_threads=8` | -| Load generator | `dfly_bench` (ref `main`) | -| Value size | 128 B | -| Keyspace | 100,000,000 keys (`--key_maximum`) | -| Connections | 160 (`--dfly-bench-threads 32` x `--dfly-bench-conns 5`) | -| Key distribution | `U` (uniform, ≈ memtier random pattern `R`) | -## Test command +### Test run: ``` dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command SET \ @@ -53,38 +24,10 @@ dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --comman --dfly-bench-key-dist U --dfly-bench-qps 0 ``` -## Results - -Latest 3-engine run (benchmark `20260721-115805`, pipeline 30, `--dfly-bench-qps 0`, 160 -connections, 1 x 300s trial, value size 128B). +### Expected results: | Engine | Throughput (median) | p50 | p99 | p99.9 | Avg latency | | ------ | ------------------- | --- | --- | ----- | ----------- | | **dragonfly** | **7.78M ops/s** | 0.483 ms | 3.271 ms | 11.505 ms | 0.600 ms | | redis | 1.06M ops/s | 3.896 ms | 7.571 ms | 13.354 ms | 4.534 ms | | valkey | 1.05M ops/s | 4.461 ms | 11.501 ms | 15.771 ms | 4.541 ms | - -**Takeaway:** Dragonfly delivers ~7.4x the write throughput of Redis/Valkey. Redis -and Valkey are effectively single-threaded on the command path, so more `io_threads` -does not lift SET much; Dragonfly spreads writes across its 16 shards. All three -engines' RPS is capped by the server's ~15 Gbps ingress budget at this value size -(the throttling shows up as periodic dips in the redis/valkey RPS trace). - -## Comparison with memtier - -memtier SET reproduction (see [../memtier/SET_reproduce.md](../memtier/SET_reproduce.md)): - -| Engine | memtier throughput | dfly_bench throughput | Δ | -| ------ | ------------------ | --------------------- | - | -| dragonfly | 7.94M ops/s | 7.78M ops/s | -2.0% | -| redis | 1.06M ops/s | 1.06M ops/s | ~0% | -| valkey | 1.07M ops/s | 1.05M ops/s | -1.9% | - -Unlike GET (where dfly_bench's closed-loop driver pushes noticeably harder than -memtier), **SET reproduces almost exactly** across the two tools. SET is -bandwidth-bound rather than CPU/queue-bound, so both drivers converge on the same -ceiling and the tail latencies line up too (Dragonfly p99.9 11.5 ms vs memtier -10.5 ms). - -Both tools' results share the same schema (this run is tagged `tool="dfly_bench"`), -so `dfbench compare ` diffs them directly. diff --git a/benchmarks/SET/memtier/SET_reproduce.md b/benchmarks/SET/memtier/SET_reproduce.md index 0340d7b..b618889 100644 --- a/benchmarks/SET/memtier/SET_reproduce.md +++ b/benchmarks/SET/memtier/SET_reproduce.md @@ -1,12 +1,15 @@ ### Stateful setup: +``` dfbench setup --id test --tune-network --ubuntu-version 24.04 \ --server-instance m7g.8xlarge --server-arch arm64 \ --client-instance c6gn.8xlarge --client-arch arm64 \ --engines dragonfly,redis,valkey --availability-zone us-east-1c +``` ### Test run: +``` dfbench run-commands --id test --engines dragonfly,redis,valkey --command SET \ --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ --dragonfly_num_shards 16 \ @@ -15,6 +18,7 @@ dfbench run-commands --id test --engines dragonfly,redis,valkey --command SET \ --test-time 300 --trials 1 --distinct-client-seed \ --redis_io_threads 8 --valkey_io_threads 8 \ --dragonfly_conn_use_incoming_cpu=true +``` ### Expected results: @@ -26,23 +30,3 @@ Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | redis | 1.06M ops/s | 3.935 ms | 7.583 ms | 12.735 ms | 4.511 ms | | valkey | 1.07M ops/s | 4.479 ms | 10.815 ms | 15.039 ms | 4.493 ms | -### Same run with dfly_bench - -`dfbench run-commands-dfly` reproduces the same workload with Dragonfly's own -load generator instead of memtier. dfly_bench is installed on the client at -`dfbench setup` time (alongside memtier); override the build with -`--dfly-bench-ref`. - -``` -dfbench run-commands-dfly --id test --engines dragonfly --command SET \ - --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ - --dragonfly_num_shards 16 \ - --key-maximum 100000000 --test-time 300 --trials 1 \ - --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 30 --dfly-bench-data-size 128 \ - --dfly-bench-qps 0 -``` - -Flag mapping vs memtier: `--dfly-bench-threads` -> `--proactor_threads`, `--dfly-bench-conns` --> `-c` (so 32x5 = 160 connections, same as memtier threadsxclients), `--dfly-bench-qps 0` -= closed-loop max load. Results use the same schema (tagged `tool="dfly_bench"`), -so `dfbench compare ` diffs the two tools.