Skip to content

readme simplification #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
miledxz merged 1 commit into from
Jul 28, 2026
Merged

readme simplification #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 5 additions & 72 deletions benchmarks/GET/dfly_bench/GET_reproduce.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,4 @@
# GET benchmark (Dragonfly) with dfly_bench — reproduce & analysis

Reproduction of the 128-byte GET workload on `m7g.8xlarge` (Graviton3, 32 vCPU)
using Dragonfly's own load generator, `dfly_bench`, instead of memtier. At the
same 160 connections, GET throughput and tail latency trade off directly against
the pipeline depth:

- **Unthrottled, deep pipeline (120):** ~13.5M ops/s but p99.9 blows up to ~33 ms.
- **Recommended (pipeline 30, unthrottled):** **10.31M ops/s at p99.9 = 0.996 ms** —
best tail, still above 10M ops/s.
- **Pipeline 30 throttled to ~11M:** 11.82M ops/s at p99.9 = 7.67 ms.
- **Engine comparison (pipeline 30, unthrottled):** Dragonfly **9.88M ops/s** vs
valkey **1.61M** and redis **1.48M** — Dragonfly is ~6.1–6.7x the throughput at a
fraction of the latency (p99.9 1.12 ms vs 8.79 ms / 6.96 ms).

All configs are CPU-bound on the server (Dragonfly saturates its shards); the tail
is governed by how many requests are kept in flight (`conns x pipeline`).

## Setup

Provisioned with `dfbench setup` (installs both memtier and dfly_bench on the
client; `dfly_bench` is built from `dragonflydb/dragonfly` at `--dfly-bench-ref`,
default `main`):
### Stateful setup:

```
dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \
Expand All @@ -34,19 +12,7 @@ Note: pick an availability zone that has `c6gn.8xlarge` capacity (AWS suggested
client share a cluster placement group, so both instance types must have capacity
in the chosen AZ.

| Component | Value |
| --------- | ----- |
| Server | m7g.8xlarge (arm64), Ubuntu 24.04, `--tune-network` |
| Client | c6gn.8xlarge (arm64) |
| Dragonfly build | `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz` (`i1.39.4-32-gcc0f424a`) |
| Dragonfly flags | `--num_shards 16`, `--conn_use_incoming_cpu` |
| Load generator | `dfly_bench` (ref `main`) |
| Value size | 128 B |
| Keyspace | 100,000,000 keys (`--key_maximum`) |
| Connections | 160 (`--dfly-bench-threads 32` x `--dfly-bench-conns 5`) |
| Key distribution | `U` (uniform, ≈ memtier random pattern `R`) |

## Test command (recommended config: pipeline 30, unthrottled)
### Test run:

```
dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command GET \
Expand All @@ -57,33 +23,18 @@ dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --comman
--dfly-bench-key-dist U --dfly-bench-qps 0
```

`--dfly-bench-qps 0` is closed-loop max load (coordinated omission). To pin throughput near a
target rate instead, set `--dfly-bench-qps` per connection = target_total / 160 (e.g.
`--dfly-bench-qps 68750` ≈ 11M ops/s total). Redis/Valkey are the preinstalled builds started
with `io_threads=32` (auto); only Dragonfly is reinstalled from `--dragonfly_version`.

## Engine comparison (dragonfly vs redis vs valkey)

Latest 3-engine run (benchmark `20260721-114528`, pipeline 30, `--qps 0`, 160
connections, 60s trial, value size 128B). Redis 8.8.0 and Valkey 9.1.0 with
`io_threads=32`; each engine is preloaded to 100M keys by dfly_bench, then measured
back-to-back with a 30s cooldown between engines.
### Expected results:

| Engine | Throughput (median) | p50 | p99 | p99.9 | Avg latency |
| ------ | ------------------- | --- | --- | ----- | ----------- |
| **dragonfly** | **9.88M ops/s** | 0.457 ms | 0.785 ms | 1.123 ms | 0.463 ms |
| valkey | 1.61M ops/s | 2.544 ms | 7.885 ms | 8.791 ms | 2.959 ms |
| redis | 1.48M ops/s | 3.250 ms | 6.076 ms | 6.956 ms | 3.218 ms |

**Takeaway:** at the same client config Dragonfly delivers ~6.1x Valkey and ~6.7x
Redis throughput while keeping p99.9 near 1 ms (vs ~7–9 ms). Redis/Valkey are
single-shard event loops that cannot spread a uniform GET load across all 32 cores
the way Dragonfly's 16 shards do, so they saturate a small number of threads well
below the box's capacity. (Dragonfly here reads 9.88M ops/s vs the 10.31M in the
pipeline sweep below — the two runs are on different EC2 instances from separate
`dfbench setup`s, so expect ~5% run-to-run variance.)

## Results (Dragonfly, pipeline sweep)

### Results (Dragonfly, pipeline sweep)

Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, 160 connections, 60s trial

Expand All @@ -94,22 +45,4 @@ Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, 16
| 30 | `--dfly-bench-qps 68750` (~11M) | 11.82M ops/s | 1.064 ms | 3.669 ms | 7.666 ms | 1.173 ms |
| **30** | **`--dfly-bench-qps 0`** | **10.31M ops/s** | **0.431 ms** | **0.765 ms** | **0.996 ms** | **0.442 ms** |

**Reading the sweep:** with 160 connections at max load, tail latency scales with
in-flight depth (`conns x pipeline`). Dropping the pipeline from 120 to 30 cuts
p99.9 by ~33x (33.3 ms -> 1.0 ms) while only trimming throughput ~24% (13.5M ->
10.3M), because Dragonfly is CPU-bound and 160x30 still keeps every shard busy.
Throttling pipeline-30 up toward 11M (`--dfly-bench-qps 68750`) recovers throughput but
re-inflates the tail to ~7.7 ms — still under 10 ms, if you want the extra ops/s.

## Comparison with memtier

The memtier reproduction (see [../memtier/GET_reproduce.md](../memtier/GET_reproduce.md))
measured Dragonfly at **11.63M ops/s, p99.9 = 7.97 ms** with pipeline 120. dfly_bench
at pipeline 120 pushes harder (13.5M ops/s) but with a much longer tail, so the two
tools are not directly comparable at identical pipeline settings — dfly_bench's
closed-loop driver keeps deeper queues. For a memtier-like tail, use a lower
pipeline (30) or throttle with `--dfly-bench-qps`.

Both tools' results share the same schema (this run is tagged `tool="dfly_bench"`),
so a memtier GET file and a dfly_bench GET file diff with `dfbench compare`.

131 changes: 4 additions & 127 deletions benchmarks/GET/memtier/GET_reproduce.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,4 @@
# GET benchmark (Dragonfly) — reproduce & analysis

## TL;DR
On `m7g.8xlarge` (Graviton3, 32 vCPU, 8 ENA queues), 128-byte **GET peaks at ~11.4–11.5M ops/s and is CPU-bound, not network-bound.** All 32 cores sit at ~0% idle while the ENA network-allowance counters never trip. This is the opposite of SET on the same box, which is ingress-bandwidth-bound at ~15 Gbps.

## Stateful setup
### Stateful setup:

```
dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \
Expand All @@ -12,7 +7,7 @@ dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \
--engines dragonfly,redis,valkey --availability-zone us-east-1
```

## Test run (best config: clients=5)
### Test run:

```
dfbench run-commands --id maxqps2 --engines dragonfly,redis,valkey --command GET \
Expand All @@ -25,7 +20,7 @@ dfbench run-commands --id maxqps2 --engines dragonfly,redis,valkey --command GET
--test-time 60 --warmup-time 10 --trials 1 --distinct-client-seed
```

## Results
### Expected results:

Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, pipeline 120, clients=5 (160 connections)

Expand All @@ -37,8 +32,6 @@ Engine comparison (Dragonfly build `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz
| valkey | 1.98M ops/s | 9.727 ms | 9.983 ms | 10.111 ms | 9.677 ms |
| redis | 1.52M ops/s | 10.431 ms | 21.119 ms | 38.143 ms | 12.653 ms |

Dragonfly is **~5.9× Valkey and ~7.7× Redis** on 128B GET. Redis/Valkey are single-threaded for
command execution, so they cannot use the 32 cores the way Dragonfly's shards do.

### Dragonfly connection sweep (why clients=5)

Expand All @@ -49,120 +42,4 @@ command execution, so they cannot use the 32 cores the way Dragonfly's shards do

**clients=5 is the sweet spot.** Doubling connections to clients=10 is a regression (congestion
collapse): throughput drops ~18% and latency ~2.5×, while server CPU utilization also drops — a
sign of too much in-flight work, not a capacity gain. Use `--memtier-clients 5`.

### Same run with dfly_bench

`dfbench run-commands-dfly` reproduces this GET workload with Dragonfly's own
load generator instead of memtier. dfly_bench is installed on the client at
`dfbench setup` time (alongside memtier); override the build with
`--dfly-bench-ref`.

```
dfbench run-commands-dfly --id maxqps2 --engines dragonfly --command GET \
--dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \
--dragonfly_num_shards 16 \
--key-maximum 100000000 --test-time 60 --warmup-time 10 --trials 1 \
--dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 120 --dfly-bench-data-size 128 \
--dfly-bench-key-dist U --dfly-bench-qps 0
```

Notes:
- **dfly_bench self-preloads.** dfly_bench uses its own key prefix
(`--key_prefix`, default `key:`), which differs from memtier's keys, so GET's
SET preload is run by dfly_bench (sequential distribution, one connection per
thread covering the whole `--key_maximum` range) rather than reused from a
memtier preload.
- **Flag mapping:** `--dfly-bench-threads` -> `--proactor_threads`, `--dfly-bench-conns` ->
`-c` (32x5 = 160 connections), `--dfly-bench-key-dist U` approximates memtier's random
key pattern `R`, `--dfly-bench-qps 0` = closed-loop max load.
- Results use the same schema (tagged `tool="dfly_bench"`), so
`dfbench compare <memtier-get.json> <dflybench-get.json>` diffs the two tools.
- dfly_bench's `--json_out_file` writes a memtier-compatible latency report, so
the same parser produces the throughput / p50 / p99 / p99.9 / avg columns.

## Bottleneck: CPU, not network (verified)

Two independent measurements taken during the same 60s measured trial.

### mpstat (server, aggregate over the trial)

```
usr=75.2 sys=8.9 irq=9.5 soft=6.3 iowait=0.0 idle=0.12 (busy 99.9%)
per-core idle across all 32 CPUs: min=0.00 avg=0.12 max=0.31
```

Every core is saturated. The per-core split also confirms the tuned layout
(`--dragonfly_num_shards 16` + `--dragonfly_conn_use_incoming_cpu` + `--tune-network`):

| Cores | Role | %usr | %soft |
| ----- | ---- | ---- | ----- |
| cpu0–15 | 16 Dragonfly shards | ~90–92% | 0% |
| cpu16–23 | connection I/O | ~57–68% | ~6–7% |
| cpu24–31 | network softirq (8 ENA RX queues) | ~48–57% | ~17–19% |

The low 16 CPUs run shards (pure userspace), the top 8 CPUs absorb the NIC softirq — exactly the
intended shard/network separation — and nothing is left idle.

### ethtool ENA allowance counters

`ethtool -S ens5` before/after plus a 1 Hz time-series, split by phase:

| Phase | egress (tx) | ingress (rx) | `bw_in_exceeded` | `bw_out_exceeded` | `pps_exceeded` |
| ----- | ----------- | ------------ | ---------------- | ----------------- | -------------- |
| Preload (SET 100M) | ~0.5 Gbps | **~15.0 Gbps** | **climbs (throttled)** | 0 | 0 |
| GET trial | ~12.2 Gbps avg (12.9 peak) | ~3.5 Gbps | **0** | **0** | **0** |

Whole-run before→after: `bw_in_allowance_exceeded` Δ ≈ +1.8M (100% during the SET preload),
`bw_out_allowance_exceeded` Δ = 0, `pps_allowance_exceeded` Δ = 0.

**Interpretation:** during GET the server sends ~12.2 Gbps and AWS never throttles it — there is
~2–3 Gbps of headroom versus the ~15 Gbps the NIC demonstrably sustains on ingress during the SET
preload. So GET is limited by Dragonfly CPU (idle ≈ 0), not by the network allowance. SET/writes
are the workload that hits the network wall on this instance.

## Consequences
- **To push GET higher:** more/faster server cores (e.g. `m8g.16xlarge`, Graviton4/64 vCPU). GET
scales with CPU here, so a network-optimized box is *not* what GET needs.
- **SET/writes** are the ones that want a network-optimized instance (`m8gn.16xlarge`, up to 32 ENA
queues), since their ingress genuinely saturates the ~15 Gbps allowance.

## Underlying memtier commands
dfbench prints each memtier invocation to stderr. For this run:

```
# preload (populate 100M keys with 128B values)
memtier_benchmark --server=<server-ip> --port=6379 --threads=32 --clients=5 \
--requests=625000 --command="SET __key__ __data__" --command-key-pattern=P \
--data-size=128 --pipeline=120 --key-maximum=100000000

# warmup (10s, discarded)
memtier_benchmark --server=<server-ip> --port=6379 --threads=32 --clients=5 \
--test-time=10 --command="GET __key__" --command-key-pattern=R \
--data-size=128 --pipeline=120 --key-maximum=100000000 --distinct-client-seed

# measured trial (60s)
memtier_benchmark --server=<server-ip> --port=6379 --threads=32 --clients=5 \
--test-time=60 --command="GET __key__" --command-key-pattern=R \
--data-size=128 --pipeline=120 --key-maximum=100000000 --distinct-client-seed
```

## How to re-verify (raw captures kept)
Raw data from the analyzed run lives alongside this doc:
- `benchmarks/GET/ena/before.txt`, `after.txt`, `timeseries.txt` — ENA counters (1 Hz series).
- `benchmarks/GET/mpstat/get-mpstat.log` — per-core mpstat (1s) across the whole run.

To reproduce the capture, in one server session (`dfbench ssh --id maxqps2 --target server`):

```
# ENA allowance counters, 1 Hz
IFACE=$(ip -o -4 route show to default | awk '{print $5}') # ens5
for i in $(seq 1 150); do echo "=== ts=$(date +%s) ==="; ethtool -S $IFACE; sleep 1; done

# per-core CPU, 1s
mpstat -P ALL 1 150
```

Then launch the `run-commands` GET test above from the client machine and look at the counters
during the final 60s (the measured trial). If `%idle`≈0 and `bw_out/pps_allowance_exceeded` stay
flat, GET is CPU-bound; if `bw_out_allowance_exceeded` climbs, it has become network-bound.
sign of too much in-flight work, not a capacity gain. Use `--memtier-clients 5`.
63 changes: 3 additions & 60 deletions benchmarks/SET/dfly_bench/SET_reproduce.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,4 @@
# SET benchmark (Dragonfly vs Redis vs Valkey) with dfly_bench — reproduce & results

Reproduction of the 128-byte SET workload on `m7g.8xlarge` (Graviton3, 32 vCPU)
using Dragonfly's own load generator, `dfly_bench`, instead of memtier, across all
three engines at 160 connections / pipeline 30 / max load:

- **dragonfly 7.78M ops/s** (p99.9 11.5 ms) — ~7.4x Redis and ~7.4x Valkey.
- **redis 1.06M ops/s** (p99.9 13.4 ms), **valkey 1.05M ops/s** (p99.9 15.8 ms).
- **Near-perfect memtier parity:** dfly_bench SET matches the memtier SET numbers
within ~2% (Dragonfly 7.78M vs 7.94M), because SET is ingress-bandwidth-bound at
~15 Gbps, so both load generators hit the same network ceiling regardless of the
client-side driver.

## Setup

Provisioned with `dfbench setup` (installs both memtier and dfly_bench on the
client; `dfly_bench` is built from `dragonflydb/dragonfly` at `--dfly-bench-ref`,
default `main`):
### Stateful setup:

```
dfbench setup --id maxqps2 --tune-network --ubuntu-version 24.04 \
Expand All @@ -28,20 +11,8 @@ Note: pick an availability zone that has `c6gn.8xlarge` capacity (AWS suggested
`us-east-1b/c/d/f`). Server and client share a cluster placement group, so both
instance types must have capacity in the chosen AZ.

| Component | Value |
| --------- | ----- |
| Server | m7g.8xlarge (arm64), Ubuntu 24.04, `--tune-network` |
| Client | c6gn.8xlarge (arm64) |
| Dragonfly build | `s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz` (`i1.39.4-32-gcc0f424a`) |
| Dragonfly flags | `--num_shards 16`, `--conn_use_incoming_cpu` |
| Redis / Valkey | Redis 8.8.0 / Valkey 9.1.0, `io_threads=8` |
| Load generator | `dfly_bench` (ref `main`) |
| Value size | 128 B |
| Keyspace | 100,000,000 keys (`--key_maximum`) |
| Connections | 160 (`--dfly-bench-threads 32` x `--dfly-bench-conns 5`) |
| Key distribution | `U` (uniform, ≈ memtier random pattern `R`) |

## Test command
### Test run:

```
dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command SET \
Expand All @@ -53,38 +24,10 @@ dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --comman
--dfly-bench-key-dist U --dfly-bench-qps 0
```

## Results

Latest 3-engine run (benchmark `20260721-115805`, pipeline 30, `--dfly-bench-qps 0`, 160
connections, 1 x 300s trial, value size 128B).
### Expected results:

| Engine | Throughput (median) | p50 | p99 | p99.9 | Avg latency |
| ------ | ------------------- | --- | --- | ----- | ----------- |
| **dragonfly** | **7.78M ops/s** | 0.483 ms | 3.271 ms | 11.505 ms | 0.600 ms |
| redis | 1.06M ops/s | 3.896 ms | 7.571 ms | 13.354 ms | 4.534 ms |
| valkey | 1.05M ops/s | 4.461 ms | 11.501 ms | 15.771 ms | 4.541 ms |

**Takeaway:** Dragonfly delivers ~7.4x the write throughput of Redis/Valkey. Redis
and Valkey are effectively single-threaded on the command path, so more `io_threads`
does not lift SET much; Dragonfly spreads writes across its 16 shards. All three
engines' RPS is capped by the server's ~15 Gbps ingress budget at this value size
(the throttling shows up as periodic dips in the redis/valkey RPS trace).

## Comparison with memtier

memtier SET reproduction (see [../memtier/SET_reproduce.md](../memtier/SET_reproduce.md)):

| Engine | memtier throughput | dfly_bench throughput | Δ |
| ------ | ------------------ | --------------------- | - |
| dragonfly | 7.94M ops/s | 7.78M ops/s | -2.0% |
| redis | 1.06M ops/s | 1.06M ops/s | ~0% |
| valkey | 1.07M ops/s | 1.05M ops/s | -1.9% |

Unlike GET (where dfly_bench's closed-loop driver pushes noticeably harder than
memtier), **SET reproduces almost exactly** across the two tools. SET is
bandwidth-bound rather than CPU/queue-bound, so both drivers converge on the same
ceiling and the tail latencies line up too (Dragonfly p99.9 11.5 ms vs memtier
10.5 ms).

Both tools' results share the same schema (this run is tagged `tool="dfly_bench"`),
so `dfbench compare <memtier-set.json> <dflybench-set.json>` diffs them directly.
Loading
Loading