dfbench

Benchmark Dragonfly, Redis, and Valkey on real AWS infrastructure with one command, or drive a long-lived benchmark environment across many runs.

dfbench provisions an isolated VPC plus two (or three) EC2 instances via a single CloudFormation stack, runs memtier_benchmark (or dfly_bench for the master-replica scenario) over SSH, produces JSON/CSV results you can diff between runs, and tears the whole thing down with one call.

Two modes

Mode	Commands	Use when...
Stateful	`setup` -> `run` (repeat) -> `destroy`	You want to iterate on memtier parameters, SSH into the instances, or run multiple benchmarks against the same environment.
One-shot	`quickstart`	CI, a demo, or a single number where you do not need the environment afterwards.
Replication	`setup --replicas N` -> `run-replication` -> `destroy`	Dragonfly master/replica scenarios: sync duration, sustained load under compression flags, regression detection.

All three modes share the same CloudFormation template, the same SSH key material, and the same results schemas, so files produced by any mode can be fed into dfbench compare.

Commands at a glance

One-liner per subcommand; jump into the full section for flags and examples.

Command	Purpose	State?
`setup`	Provision an isolated VPC + EC2 fleet and pre-install every engine.	mutates
`run`	Execute a `memtier_benchmark` pass (N trials, all engines) against an existing setup.	mutates
`run-commands`	Benchmark one Redis command (SET, GET, LPUSH, ...) per invocation against an existing setup.	mutates
`run-commands-dfly`	Same as `run-commands`, but drives load with `dfly_bench` instead of memtier.	mutates
`run-replication`	Execute the Dragonfly master/replica sweep against a setup with `--replicas >= 1`.	mutates
`quickstart`	One-shot: `setup` + `run` + `destroy` in a single invocation.	mutates
`compare`	Diff two result files (auto-detects memtier vs replication) and exit non-zero on regression.	read-only
`info`	Print a setup's status, instance IPs, SSH cheat-sheet, engine inventory, and benchmark history.	read-only
`list`	Enumerate every known setup with age and status. `--all` includes destroyed rows.	read-only
`ssh`	Drop into an interactive shell on the server, client, or a replica. `--print` prints the command instead.	read-only
`destroy`	Delete the CloudFormation stack and local SSH keys. Idempotent.	mutates
`template`	Render the CloudFormation YAML without calling AWS - ideal for security review / `validate-template`.	read-only
`version`	Print the embedded build version.	read-only

The "mutates" commands take an advisory POSIX flock on ~/.dfbench/runs/<id>/.lock; the "read-only" commands never block, so info / list / ssh --print / template work mid-benchmark.

Prerequisites

Go 1.26 (repository is github.com/dragonflydb/benchmarking, module requires Go 1.26).
AWS credentials on the current shell (profile, env vars, or SSO).
Permission to create the AWS resources listed below.
A clock: a typical quickstart run takes 10-15 minutes end to end.

Cost. Two c7g.4xlarge instances in us-east-1 cost around $1.16/hr; a quickstart session is roughly 15-30 minutes. The approval screen prints a live hourly estimate from pricing/pricing.go so you can sanity-check before you hit y.

IAM actions required. Any policy granting the actions below is sufficient; AdministratorAccess is the simplest choice for a demo.

cloudformation:CreateStack, DeleteStack, DescribeStacks, DescribeStackEvents, UpdateStack
ec2:* on the resources the template manages (VPC, subnet, IGW, route table, SG, key pair, placement group, instance)
ssm:GetParameters on /aws/service/canonical/ubuntu/server/22.04/*
sts:GetCallerIdentity (used only for the pre-flight auth check)

No dfbench-specific IAM role is created; the credentials in your shell are the credentials CloudFormation will see.

Install

From the repo root:

make dfbench
# -> build/dfbench

This builds the single binary build/dfbench. The make all target includes dfbench alongside dfcloud, dfadmin, dfctl, dfk, and dftest so a fresh checkout picks it up automatically.

30-second quickstart

./build/dfbench quickstart \
  --engines dragonfly,redis,valkey \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64

What happens:

A 4096-bit SSH key pair is generated locally at ~/.dfbench/runs/<id>/ssh/.
A CloudFormation template is rendered, saved to disk, and printed as a plain-English plan plus $/hr estimate.
You approve with y. The stack is created with OnFailure=DELETE (automatic rollback on any failure).
memtier_benchmark is built on the client EC2; Dragonfly, Redis, and Valkey are installed on the server EC2.
Each engine is started, warmed up, benchmarked for --trials trials (default 3), stopped, and 30-second cooldown before the next.
A comparison table is printed to stdout and results.json is saved under ~/.dfbench/runs/<id>/.
The stack is deleted. Pass --skip-destroy to keep it running.

Skip the approval prompt with --yes or by exporting DFBENCH_NO_CONFIRM=1.

Stateful lifecycle

Use the stateful flow when you want to iterate. Each step below maps to one subcommand.

flowchart LR
  U[User] --> S["dfbench setup --id NAME"]
  S --> Plan[Render CFN + print plan + cost]
  Plan --> Approve["Prompt y/N"]
  Approve -->|yes| Create[CreateStack]
  Create --> Wait[Wait for CREATE_COMPLETE]
  Wait --> InstMem[Install memtier on client]
  InstMem --> InstEng[Pre-install engines on server and replicas]
  InstEng --> Ready["status=ready"]
  Ready --> R["dfbench run --id NAME"]
  R --> Trials[Start engine, warmup, trials, stop]
  Trials --> Save[Append BenchmarkRun to state.json]
  Save --> Ready
  Ready --> RR["dfbench run-replication --id NAME"]
  RR --> RepFlow[Master + replica + sustained load]
  RepFlow --> Ready
  Ready --> I["dfbench info / ssh / list"]
  Ready --> D["dfbench destroy --id NAME"] --> Gone[DeleteStack]

1. Provision

dfbench setup \
  --id perf-may \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64

Prints the plan, asks for confirmation, provisions the stack, and pre-installs every engine you select via --engines (default: dragonfly,redis,valkey). Use --dry-run to stop after the plan, or --yes to skip the prompt.

Everything from this point lives under ~/.dfbench/runs/perf-may/.

2. Benchmark (as many times as you want)

dfbench run --id perf-may \
  --engines dragonfly,redis,valkey \
  --memtier-threads 8 --memtier-clients 50 \
  --memtier-requests 1000000 --memtier-pipeline 30 \
  --trials 3

Each invocation:

Writes artifacts under ~/.dfbench/runs/perf-may/benchmarks/<timestamp>/.
Appends a BenchmarkRun record to the setup's state.json.
Restores status to ready on exit (including on Ctrl+C) so the setup stays usable.

3. Inspect

dfbench list - all known setups with status and age.
dfbench info --id perf-may - detailed status, instance IPs, SSH cheat-sheet, installed engines, past benchmarks, running cost.
dfbench ssh --id perf-may --target server - interactive shell on the server instance. Use --print to print the exact ssh -i ... ubuntu@<ip> command instead of running it.

4. Tear down

dfbench destroy --id perf-may

Deletes the CloudFormation stack and removes the local SSH key files. Idempotent; if the stack is already gone (manual cleanup, prior rollback) the command still succeeds.

Per-command benchmark

Use this scenario when you want a per-command performance profile instead of a single mixed SET:GET number: each invocation runs one dedicated memtier_benchmark test for one Redis command, per engine, using memtier's arbitrary-command mode (--command="SET __key__ __data__"). Benchmark commands one at a time; each run produces its own results file.

1. Provision (32-vCPU reference spec)

dfbench setup \
  --id percmd \
  --server-instance m7g.8xlarge --server-arch arm64 \
  --client-instance c6gn.8xlarge --client-arch arm64

2. Benchmark one command per invocation

dfbench run-commands --id percmd \
  --engines dragonfly,redis,valkey \
  --command SET --dragonfly_num_shards -1 \
  --memtier-threads 32 --memtier-clients 5 --memtier-pipeline 20

Repeat with the next command whenever you're ready - the setup stays provisioned between runs (--dragonfly_num_shards is required whenever dragonfly is among the engines; -1 auto-derives the tuned value):

dfbench run-commands --id percmd --command GET --dragonfly_num_shards -1
dfbench run-commands --id percmd --command LPUSH --dragonfly_num_shards -1
dfbench run-commands --id percmd --command LRANGE --dragonfly_num_shards -1

For each engine the scenario:

FLUSHALL, then preloads the keyspace if the command reads existing data. Preloads match the data type: GET/EXISTS/DEL get plain strings, LRANGE/LPOP get 100-element lists, SMEMBERS gets 10-member sets, and so on. Preloads use memtier's parallel key pattern so the whole key range is covered exactly once per item.
Runs a discarded warmup pass (--warmup-time, default 10s).
Runs --trials measured trials of --test-time seconds each. Mutating commands (SET, INCR, LPUSH, ...) get a fresh flush + preload before every trial; read-only commands (GET, LRANGE, ...) keep their preloaded keyspace across trials.

Supported commands: strings (SET, GET, MSET, MGET, INCR, APPEND), keyspace ops (EXISTS, EXPIRE, TTL, DEL), lists (LPUSH, RPUSH, LPOP, RPOP, LRANGE), hashes (HSET, HGET, HGETALL), sets (SADD, SISMEMBER, SMEMBERS), sorted sets (ZADD, ZSCORE, ZRANGE), and PING.

Draining commands (DEL, LPOP, RPOP) can empty their preloaded keyspace mid-trial; their numbers include miss-path operations and are marked approximate in the results.

3. Compare across runs

Compare two runs of the same command (e.g. before/after an engine upgrade):

dfbench compare \
  ~/.dfbench/runs/percmd/commands/20260706-120000/results.json \
  ~/.dfbench/runs/percmd/commands/20260707-093000/results.json

Auto-detects the per-command shape (via the top-level scenario: "commands" tag) and diffs median throughput, p50, p99, p99.9, and avg latency per engine, with the same exit-code contract as the other comparators.

Maximum-throughput configuration

To drive Dragonfly at its optimal single-node configuration (the setup that reaches ~10M QPS for SET on a c7g.8xlarge, data-bound rather than network-bound), combine server network tuning with Dragonfly's shard/connection-affinity flags.

Provision with --tune-network so net_tune.sh pins the NIC IRQs and RPS/XPS steering to the top CPUs (e.g. 16-31 on a 32-vCPU box), leaving the low CPUs (0-15) free for shards:

dfbench setup --id maxqps --tune-network \
  --server-instance c7g.8xlarge --server-arch arm64 \
  --client-instance c7gn.8xlarge --client-arch arm64 \
  --engines dragonfly

If you plan to run a Dragonfly build compiled against a newer glibc (e.g. one that needs GLIBC_2.38), add --ubuntu-version 24.04 so the server launches with a matching OS. The default is Ubuntu 22.04 (glibc 2.35), which runs the published releases but not builds linked against 2.38:

dfbench setup --id maxqps --tune-network --ubuntu-version 24.04 \
  --server-instance m7g.8xlarge --server-arch arm64 \
  --client-instance c6gn.8xlarge --client-arch arm64 \
  --engines dragonfly

Then benchmark. --dragonfly_num_shards must be stated explicitly when benchmarking dragonfly (pass -1 for the auto-derived tuned value); --dragonfly_conn_use_incoming_cpu and --redis_pin_network / --valkey_pin_network default to the tuned configuration:

dfbench run-commands --id maxqps --engines dragonfly \
  --command SET --memtier-threads 32 --memtier-clients 5 \
  --memtier-pipeline 35 --memtier-data-size 128 \
  --dragonfly_num_shards -1 \
  --key-maximum 100000000 --test-time 300 \
  --trials 1 --warmup-time 0 --distinct-client-seed

This launches Dragonfly with --conn_use_incoming_cpu and --num_shards=$(VCPUS - 2*NUM_RX_QUEUES) (16 on a 32-vCPU/8-queue box). Validate on the server with htop: the low CPUs should be ~100% green (shards, userspace) and the high CPUs mixed green/red (network I/O). Ensure the client is not the bottleneck - the result should be data-bound (well under the NIC line rate).

Notes and caveats:

--num_shards is coupled to the NIC RX-queue count; the auto value only equals 16 on 32-vCPU/8-queue instances, which is why it is derived on the server rather than hardcoded.
Redis/Valkey pinning (--redis_pin_network / --valkey_pin_network) is best-effort: taskset constrains the process cpuset (CPU 0 plus the I/O CPUs), but the engines offer no per-thread affinity guarantee. Pair with --redis_io_threads 8 / --valkey_io_threads 8 (or 16).
Comparing binaries (e.g. latest vs an older tag): keep the tuning, shard, and memtier settings byte-identical between runs so any delta is attributable to the binary. setup installs the latest release; use run-commands --dragonfly_version to reinstall a specific build on the same tuned setup right before a run, then diff the result files:

# Baseline: pin an older release.
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version v1.31.0 --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

# Candidate: latest release (or a nightly/custom tarball URL).
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version latest --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

--dragonfly_version accepts a release tag (v1.31.0), latest, a full https:// URL to a dragonfly-<arch>.tar.gz tarball, or a path to a local file. Drop an unreleased binary someone hands you into the repo and point at it directly; dfbench uploads it to the server and installs it (raw binary or .tar.gz, extracted automatically):

# Local binary dropped into the repo (path is relative to your CWD).
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version ./bin/dragonfly --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

The uploaded binary must match the server's CPU architecture (arm64 for the c7g/c7gn instances above). The installed version is recorded per-engine in results.json.

Replication scenario

Use this scenario to measure Dragonfly replication: sync duration under a given dataset size, sustained RPS and p99 latency during sync, and the effect of replica-side flags like --replication_stream_compression=zstd.

1. Provision with at least one replica

dfbench setup \
  --id rep-demo \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 1

--replica-instance and --replica-arch default to the server's values, so the simplest form is what is shown above. Setup pre-installs Dragonfly on the replica(s) concurrently (up to four at a time), so subsequent run-replication invocations pay zero install cost.

2. Run the sweep

dfbench run-replication --id rep-demo \
  --cases qps=100000,qps=50000,maxload \
  --test-time 30 \
  --preload-keys 500000 \
  --value-size 4096 \
  --ratio 1:0 \
  --master-flags "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb --replication_stream_compression=zstd"

For each case (one row of the QPS sweep) the scenario:

Starts Dragonfly on the server as master.
FLUSHALL and preloads the keyspace from the client using dfly_bench for 30s.
Sets a READY=true sentinel on the master and starts the replica with --replicaof=<master-private-ip>:6379.
Waits for the Transitioned into stable sync log marker and records sync_duration_ms.
Drives sustained load for --test-time seconds at the target QPS while a background goroutine polls INFO replication every 2s; a drop in connected_slaves from 1 to 0 ends the case early with termination_reason=replica_disconnect.
FLUSHALL ASYNC and kills both processes so the next case starts clean.

3. Compare across runs

dfbench compare \
  ~/.dfbench/runs/rep-demo/replications/20260408-143217/results.json \
  ~/.dfbench/runs/rep-demo/replications/20260408-152103/results.json

Auto-detects that the files are replication results (via the top-level scenario: "replication" tag) and prints a per-case table:

median_rps (higher is better)
median_p99_us (lower is better)
sync_duration_ms (lower is better)

Exit codes: 0 (within tolerance), 1 (any metric regressed beyond --regression-pct, default 5%), 2 (input error: unreadable file or mismatched shapes).

What `dfly_bench` is measuring

The flag choices in bench/replication/dflybench.go mirror playground/replication-bench/benchmark.py exactly so dfbench numbers are comparable with the upstream Python prototype:

Preload: --proactor_threads=4 -c=15 --pipeline=10 --qps=0 --test_time=30. 60 concurrent writers, random keys over [0, --preload-keys). --preload-keys names the keyspace range (the prototype's --key_maximum), not the number of keys actually written.
Load: --proactor_threads=1 -c=2 --pipeline=5 --qps=<case QPS>. Two connections is the crucial bit: dfly_bench's --qps is a per-connection cap, so a qps=100000 case schedules 200k ops/s total. maxload emits --qps=0 (explicitly "no cap"); omitting the flag would silently pick up dfly_bench's default of 20 req/sec.

Command reference

Every subcommand runs under an advisory POSIX flock on ~/.dfbench/runs/<id>/.lock to prevent concurrent mutations on the same setup.

`setup`

Provision a stateful benchmark environment. Persists state at ~/.dfbench/runs/<id>/state.json and refuses to overwrite an existing setup unless it was previously destroyed.

Key flags (all optional except --id is auto-generated from a timestamp if omitted):

Flag	Default	Notes
`--id`	timestamp	Directory name under `~/.dfbench/runs/`, and the suffix on the CloudFormation stack name `dfbench-<id>`.
`--server-instance`	`c6i.2xlarge`	Must be cluster-placement-compatible (no `t2/t3/t3a/t4g/a1/mac*`).
`--client-instance`	`c5.2xlarge`	Same constraint.
`--server-arch`, `--client-arch`	`x86_64`	`x86_64` or `arm64`. A non-fatal warning is printed when the arch disagrees with the instance family.
`--ubuntu-version`	`22.04`	Ubuntu release for all instances: `20.04` / `22.04` / `24.04`. Selects Canonical's SSM AMI (gp3 volume for 24.04, gp2 for older). Use `24.04` for Dragonfly builds compiled against a newer glibc (e.g. `GLIBC_2.38`); the default 22.04 ships glibc 2.35. Recorded in `state.json` as `ubuntu_version`.
`--engines`	`dragonfly,redis,valkey`	Comma-separated subset. Pre-installed up front so `run` is fast.
`--region`	`us-east-1`	AWS region.
`--ssh-user`	`ubuntu`	SSH user on the Ubuntu 22.04 AMIs.
`--replicas`	`0`	Non-zero provisions `--replicas` replica EC2(s). Required for `dfbench run-replication`.
`--replica-instance`, `--replica-arch`	`--server-instance`, `--server-arch`	Defaults mirror the server.
`--tune-network`	`false`	After install, run `net_tune.sh` on the server to pin NIC IRQs and RPS/XPS to the top CPUs, freeing the low CPUs for Dragonfly shards. Recommended for high-throughput runs on many-vCPU instances (e.g. `c7g.8xlarge`); needs a physical NIC (ENA) and root. Non-fatal: a tuning failure does not sink the setup. Recorded in `state.json` as `tune_network`.
`--dry-run`	`false`	Render plan, save `stack.yaml`, exit before any AWS call.
`--yes`	`false`	Skip the interactive prompt.

Example - provision a two-replica arm64 fleet in eu-west-1, but stop after rendering the plan so you can review it before committing any AWS spend:

dfbench setup \
  --id perf-eu \
  --region eu-west-1 \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 2 \
  --dry-run

The rendered CloudFormation template is saved at ~/.dfbench/runs/perf-eu/stack.yaml. Drop the --dry-run and rerun once you're happy.

`run`

Execute a memtier_benchmark benchmark against an existing setup. Requires --id; every engine listed via --engines must already be installed (see dfbench info). The per-run memtier flags below are not persisted in state, so pass them on every invocation.

Flag	Default	Notes
`--id`	(required)	Setup previously created by `dfbench setup`.
`--engines`	`dragonfly,redis,valkey`	Subset of the setup's installed engines.
`--memtier-threads`	`4`	Total concurrent connections = threads * clients.
`--memtier-clients`	`50`	Clients per thread.
`--memtier-requests`	`1000000`	Requests per trial, per engine.
`--memtier-ratio`	`1:10`	SET:GET; `1:10` means 91% reads.
`--memtier-data-size`	`256`	Value size in bytes.
`--memtier-pipeline`	`30`	Requests pipelined per response wait.
`--warmup-requests`	`100000`	Discarded warmup pass. `0` disables.
`--trials`	`3`	Measured trials per engine. Results aggregate as median with [min..max] range.
`--distinct-client-seed`	`false`	Give each memtier client a different RNG seed. Widens the key space and lowers cache-hit rate by ~5-15%. Off for backward-comparable numbers.
`--output`	`table`	`table` / `json` / `csv`. `results.json` is always written under `benchmarks/<ts>/`.

Example - dragonfly-only, no warmup, distinct-client-seed ON, stdout as JSON redirected to a file for scripting:

dfbench run --id perf-may \
  --engines dragonfly \
  --memtier-threads 8 --memtier-clients 100 \
  --memtier-requests 5000000 --memtier-pipeline 50 \
  --memtier-ratio 1:4 \
  --warmup-requests 0 \
  --distinct-client-seed \
  --trials 5 \
  --output json > /tmp/perf-may-2026-04-08.json

Progress banners go to stderr when --output is json or csv, so the redirected file is a single well-formed payload.

`run-commands`

Benchmark one Redis command against an existing setup. Requires --id and either --command (a built-in command) or --command-template (a custom one); writes to ~/.dfbench/runs/<id>/commands/<timestamp>/. Run it once per command you want to profile.

Flag	Default	Notes
`--id`	(required)	Setup previously created by `dfbench setup`.
`--command`	(one of --command / --command-template)	Single built-in command name, e.g. `SET`, `GET`, `LRANGE`. See the supported list above. Mutually exclusive with `--command-template`.
`--engines`	`dragonfly,redis,valkey`	Subset of the setup's installed engines.
`--memtier-threads`	`32`	Total concurrent connections = threads * clients.
`--memtier-clients`	`5`	Clients per thread.
`--memtier-pipeline`	`20`	Requests pipelined per response wait.
`--memtier-data-size`	`256`	Bytes filled into `__data__` placeholders.
`--key-maximum`	`100000000`	Key range for keyed commands. Size it to 30-90% of server memory so the working set dwarfs CPU caches: 100M keys x 256B values is ~26GB of data plus per-key overhead, ~30% of an m7g.8xlarge (128GB). Commands with multi-item structured preloads (LRANGE, ZRANGE, ...) use a built-in 1M-key range (100 items/key = 100M preload ops) instead.
`--test-time`	`300`	Seconds of measured load per trial (time-boxed, not request-counted, so run time is predictable).
`--warmup-time`	`10`	Seconds of discarded warmup load per engine. `0` disables.
`--trials`	`3`	Measured trials per engine. Aggregated as median with [min..max].
`--distinct-client-seed`	`true`	Each memtier client gets its own RNG seed so the 160 clients don't replay the same key sequence. Disable with `=false`.
`--redis_io_threads`, `--valkey_io_threads`	`0` (one per vCPU)	Override `--io-threads` for Redis / Valkey respectively; their docs suggest ~8 can outperform higher counts on many-core boxes. Dragonfly is unaffected (proactor threads always match vCPUs). Recorded in `results.json` as `io_threads` (per engine). Deprecated shared alias: `--io-threads` (sets both).
`--dragonfly_num_shards`	none (required)	Dragonfly `--num_shards`. Required when benchmarking dragonfly so the config always states the shard count explicitly (no silent default). `-1` auto-derives `VCPUS - 2*NUM_RX_QUEUES` on the server so shards stay off the network-I/O CPUs that `setup --tune-network` reserves; `0` uses Dragonfly's default (one shard per proactor); `>0` sets it explicitly. Ignored by Redis/Valkey. Recorded as `num_shards`. Deprecated alias: `--num-shards`.
`--dragonfly_conn_use_incoming_cpu`	`true`	Run Dragonfly with `--conn_use_incoming_cpu` so each connection's I/O runs on the CPU that received its NIC interrupt. Pair with `dfbench setup --tune-network`. Disable with `=false`. Ignored by Redis/Valkey. Recorded as `conn_use_incoming_cpu`. Deprecated alias: `--conn-use-incoming-cpu`.
`--redis_pin_network`, `--valkey_pin_network`	`true`	Best-effort `taskset` pinning for Redis / Valkey respectively onto CPU 0 plus the network-I/O CPUs (falls back to no pinning if NIC-queue detection fails). Disable with `=false`. Ignored by Dragonfly. Recorded as `pin_network` (per engine). Deprecated shared alias: `--pin-network` (sets both).
`--dragonfly_version`	`""` (keep installed)	Reinstall Dragonfly at this build before benchmarking so version deltas are measured on identical tuning. Accepts a release tag (`v1.31.0`), `latest`, a full `https://` URL to a `dragonfly-<arch>.tar.gz` tarball, or a path to a local file (a raw binary or `.tar.gz` in the repo) that is uploaded to the server. Empty keeps whatever `setup` installed. Ignored by Redis/Valkey. The installed version is recorded per-engine in `results.json`. Deprecated alias: `--dfly-version`.
`--output`	`table`	`table` / `json` / `csv`. `results.json` + `results.csv` are always written.

Example - quick single-engine, single-trial smoke run on GET with a small keyspace (not representative numbers; use the defaults for real measurements):

dfbench run-commands --id percmd \
  --engines dragonfly \
  --command GET --dragonfly_num_shards -1 \
  --key-maximum 1000000 --test-time 15 --trials 1

Custom commands

To benchmark a command that isn't in the built-in registry, replace --command <name> with --command-template (the two are mutually exclusive). The template uses memtier's arbitrary-command form with the same __key__ / __data__ placeholders as the built-in specs (__data__ is filled from --memtier-data-size).

Flag	Default	Notes
`--command-template`	`""`	Custom command, e.g. `"SETEX __key__ 3600 __data__"`. Mutually exclusive with `--command`.
`--command-name`	first word of the template	Label used in output files and `results.json`.
`--memtier-command-key-pattern`	`R`	memtier `--command-key-pattern` for the measured pass: `R` (random), `S` (sequential), `P` (parallel), `G` (gaussian). memtier-specific, so it carries the `--memtier-` prefix; the dfly variant uses `--dfly-bench-key-dist` instead.
`--command-mutates`	`false`	Mark the command as mutating: flush + re-preload before every trial.
`--command-drains`	`false`	Mark the command as draining preloaded data (like `DEL`, `LPOP`); implies `--command-mutates` and flags results approximate.
`--preload-template`	`""`	Optional preload command, e.g. `"SET __key__ __data__"`; must contain `__key__`.
`--preload-items`	`1`	Items written per key by the preload (list/set/zset cardinality).
`--command-key-maximum`	`0` (use `--key-maximum`)	Per-command key range override. Required when `--preload-items > 1` so a multi-item preload doesn't run against the full 100M range.

Example - a mutating custom command (SETEX) and a read command (GETRANGE) preloaded with plain strings:

dfbench run-commands --id percmd --engines dragonfly --dragonfly_num_shards -1 \
  --command-template "SETEX __key__ 3600 __data__" --command-name SETEX --command-mutates

dfbench run-commands --id percmd --engines dragonfly --dragonfly_num_shards -1 \
  --command-template "GETRANGE __key__ 0 50" --command-name GETRANGE \
  --preload-template "SET __key__ __data__"

Budget note: one run is roughly (engines) x (trials x test-time + warmup + preload + pauses); the default 3 engines x 3 x 300s lands around 50-60 minutes per command. Preloading a 100M keyspace for read commands adds roughly 20-60s per engine depending on its write throughput.

Memory note: size --key-maximum to the smallest engine memory you benchmark. 100M x 256B assumes a ~128GB server; on smaller instances lower it proportionally or the engines will evict or OOM during preload.

`run-commands-dfly`

The dfly_bench counterpart to run-commands: it drives the same per-command flush/preload/warmup/trials cycle, but with Dragonfly's own load generator instead of memtier. dfly_bench is built from the dragonflydb/dragonfly source and installed on the client at dfbench setup time (alongside memtier); override the build with --dfly-bench-ref. Results use the same schema as run-commands (tagged tool="dfly_bench") and land in ~/.dfbench/runs/<id>/commands/<timestamp>/, so a memtier file and a dfly_bench file for the same command diff with dfbench compare.

Because dfly_bench uses its own key prefix (--key_prefix, default key:), read commands (GET, ...) are preloaded by dfly_bench itself (sequential distribution, one connection per thread covering the whole --key-maximum range) rather than reusing a memtier preload.

Flag naming: client load generator vs. server engine. The prefix tells you what a flag tunes:

--memtier-* (in run-commands) and --dfly-bench-* (in run-commands-dfly) configure the client-side load generator (memtier_benchmark and dfly_bench respectively). The two commands are deliberately kept separate so it's always explicit which tool is driving load.
--dragonfly_*, --redis_*, --valkey_* configure the server engine under test and behave identically in both commands. In particular, --dragonfly_num_shards is Dragonfly server tuning, not a client flag.

So --dfly-bench-threads sets dfly_bench's --proactor_threads on the client, while --dragonfly_num_shards sets Dragonfly's shard count on the server; they are unrelated.

The --dfly-bench-* flags were previously named --dfly-* (--dfly-threads, --dfly-conns, --dfly-pipeline, --dfly-data-size). Those spellings still work as hidden, deprecated aliases (they print a deprecation notice); prefer the --dfly-bench-* names.

Flag	Default	Notes
`--id`	(required)	Setup previously created by `dfbench setup`.
`--command`	(one of --command / --command-template)	Single built-in command name, e.g. `SET`, `GET`, `LRANGE`. Mutually exclusive with `--command-template`.
`--command-template` (+ `--command-name`, `--command-mutates`, `--command-drains`, `--preload-template`, `--preload-items`, `--command-key-maximum`)	(see Custom commands)	Define a command not in the registry. Same workload flags as `run-commands`. There is no key-pattern flag here (that is memtier-specific); the measured distribution comes from `--dfly-bench-key-dist`.
`--engines`	`dragonfly`	Subset of the setup's installed engines (`dfly_bench` speaks RESP, so Redis/Valkey work too).
`--dfly-bench-threads`	`32`	`dfly_bench --proactor_threads` (analogous to memtier threads). Total connections = threads * conns.
`--dfly-bench-conns`	`5`	`dfly_bench -c`, connections per thread (analogous to memtier clients).
`--dfly-bench-pipeline`	`20`	`dfly_bench --pipeline`: max pending requests per connection.
`--dfly-bench-data-size`	`256`	`dfly_bench -d`: value size in bytes (`__data__` placeholders).
`--key-maximum`	`100000000`	Key range (`--key_maximum`). Structured preloads use a built-in 1M-key range.
`--dfly-bench-key-dist`	`U`	`dfly_bench --key_dist` for the measured pass: `U` (uniform, the memtier random `R` equivalent), `N`, `Z`, `S`.
`--dfly-bench-qps`	`0`	`dfly_bench --qps` per connection; `0` = closed-loop max load (coordinated omission).
`--test-time`	`300`	Seconds of measured load per trial.
`--warmup-time`	`10`	Seconds of discarded warmup load per engine. `0` disables.
`--trials`	`3`	Measured trials per engine (median with [min..max]).
`--dfly-bench-ref`	`main`	Git ref of `dragonflydb/dragonfly` to build `dfly_bench` from if it isn't already installed.
`--dragonfly_num_shards`, `--dragonfly_conn_use_incoming_cpu`, `--dragonfly_version`	(as in `run-commands`)	Server-side Dragonfly tuning; identical semantics to `run-commands`.
`--redis_io_threads`, `--valkey_io_threads`, `--redis_pin_network`, `--valkey_pin_network`	(as in `run-commands`)	Redis/Valkey tuning.
`--output`	`table`	`table` / `json` / `csv`.

dfly_bench's --json_out_file writes a memtier-compatible latency report, so the same parser produces the throughput / p50 / p99 / p99.9 / avg columns for both tools.

dfbench run-commands-dfly --id percmd \
  --engines dragonfly \
  --command GET --dragonfly_num_shards -1 \
  --key-maximum 1000000 --test-time 15 --trials 1

`run-replication`

Execute the Dragonfly master/replica benchmark scenario against an existing setup that has at least one replica. Writes to ~/.dfbench/runs/<id>/replications/<timestamp>/.

Flag	Default	Notes
`--id`	(required)	Setup created by `dfbench setup --replicas >= 1`.
`--cases`	upstream playground's 5 levels	Comma-separated `qps=<int>[k]` or `maxload` tokens. Empty = `qps=100000,qps=80000,qps=50000,qps=30000,maxload`.
`--test-time`	`60`	Seconds of sustained load per case.
`--preload-keys`	`1_000_000`	Keyspace range for preload (not a hard count).
`--value-size`	`256`	Value size in bytes; used for both preload and load.
`--ratio`	`1:1`	SET:GET for the load phase. Use `1:0` to reproduce the upstream playground write-heavy workload. Preload is always 1:0.
`--master-flags`	`""`	Space-separated extra flags for the master Dragonfly process.
`--replica-flags`	`""`	Same, for the replica. Examples: `--replication_stream_compression=zstd`, `--maxmemory=25gb`.
`--dfly-ref`	`main`	Git ref (branch, tag, or SHA) of `dragonflydb/dragonfly` used when building `dfly_bench` on the client. Pinning matters: `dfly_bench`'s stdout format has shifted before, and the scrape regex in dfbench is coupled to a known schema.
`--output-dir`	`~/.dfbench/runs/<id>/replications/<ts>/`	Where `results.json` and the playground CSV land.

Example - compression A/B test. Run the same single-case sweep twice against the same setup, once with zstd replication-stream compression enabled and once without, then diff the two JSON files:

dfbench run-replication --id rep-demo \
  --cases qps=80000 --test-time 60 \
  --preload-keys 500000 --value-size 4096 --ratio 1:0 \
  --master-flags  "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb" \
  --output-dir ~/.dfbench/runs/rep-demo/replications/nocomp/

dfbench run-replication --id rep-demo \
  --cases qps=80000 --test-time 60 \
  --preload-keys 500000 --value-size 4096 --ratio 1:0 \
  --master-flags  "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb --replication_stream_compression=zstd" \
  --output-dir ~/.dfbench/runs/rep-demo/replications/zstd/

dfbench compare \
  ~/.dfbench/runs/rep-demo/replications/nocomp/results.json \
  ~/.dfbench/runs/rep-demo/replications/zstd/results.json

`quickstart`

One-shot: provision, benchmark, tear down. Accepts the union of setup and run flags, plus --skip-destroy to leave the stack running. Good for CI and demos; use the stateful flow for iteration.

Example - Dragonfly-only, five trials, keep the stack up after the run so you can SSH in and poke around (useful when numbers look surprising):

./build/dfbench quickstart \
  --engines dragonfly \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --trials 5 \
  --skip-destroy \
  --yes

Remember to follow up with dfbench list and dfbench destroy --id <ts> once you're done - quickstart auto-generates the ID from the current timestamp.

`compare`

dfbench compare <baseline.json> <candidate.json>

Auto-detects the shape of each file (memtier when the top-level has engines; replication when it has scenario: "replication" or cases; per-command when it has scenario: "commands" or commands) and emits a per-metric diff table or JSON payload.

Flag	Default	Notes
`--regression-pct`	`5.0`	Percent regression threshold. Exits 1 when exceeded on any metric.
`--format`	`table`	`table` (default, tabwriter) or `json`.

Exit codes: 0 (within tolerance), 1 (regression), 2 (input error).

Memtier example - diff two dfbench run results with a tighter 2% tolerance:

dfbench compare \
  ~/.dfbench/runs/perf-may/benchmarks/20260408-143217/results.json \
  ~/.dfbench/runs/perf-may/benchmarks/20260409-091802/results.json \
  --regression-pct 2.0

Replication example with JSON output, suitable for piping into jq or a downstream dashboard:

dfbench compare baseline.json candidate.json --format json \
  | jq '.cases[] | select(.regressions != [])'

CI wiring - branch on the exit code so a flaky run doesn't silently pass:

dfbench compare baseline.json candidate.json
case $? in
  0) echo "ok: candidate within tolerance" ;;
  1) echo "REGRESSION; see output above"; exit 1 ;;
  2) echo "input error (missing / mismatched files)"; exit 2 ;;
esac

`info`, `list`, `ssh`, `destroy`, `template`, `version`

All six are thin wrappers around the local state file or the CloudFormation API; none of them re-run any benchmark work. Examples are copy-pasteable against a real setup called perf-may.

`info`

Read-only dump from state.json. No AWS calls, no side effects. Shows status, instance metadata (public + private IPs), SSH cheat-sheet, installed-engine inventory, past benchmark history, running cost estimate, and next-step hints.

dfbench info --id perf-may

Sample output (truncated):

Setup: perf-may
  Status:       ready
  Created:      2026-04-08T14:30:11Z (17m ago)
  Region:       us-east-1
  Stack:        dfbench-perf-may
  AWS Account:  123456789012

Instances:
  server:  c7g.4xlarge arm64   public=3.88.12.4   private=10.0.1.42
  client:  c7g.4xlarge arm64   public=3.88.12.17  private=10.0.1.88
  Est. cost:    $1.156/hr  (running ~17m -> ~$0.33 so far)

SSH:
  Key:    /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa
  Server: ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa ubuntu@3.88.12.4
  Client: ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa ubuntu@3.88.12.17

Engines (pre-installed on server):
  NAME       VERSION              STATUS
  dragonfly  dragonfly v1.27.0    installed
  redis      Redis server v=7.4   installed
  valkey     Valkey server v=7.2  installed

Next steps:
  dfbench run     --id perf-may --engines dragonfly,redis,valkey
  dfbench ssh     --id perf-may --target server
  dfbench destroy --id perf-may

`list`

Every setup with local state, newest first. By default destroyed rows are hidden so you see only actionable setups.

dfbench list

ID         STATUS  REGION     SERVER               CLIENT               AGE
perf-may   ready   us-east-1  c7g.4xlarge arm64    c7g.4xlarge arm64    17m
rep-demo   ready   us-east-1  c7g.4xlarge arm64    c7g.4xlarge arm64    2h14m

Add --all to include destroyed setups (handy for audit / recovering an ID from a week-old state file):

dfbench list --all

`ssh`

Interactive shell via syscall.Exec - dfbench replaces itself with /usr/bin/ssh, so your terminal becomes the session. --target defaults to server; valid values are server, client, and replica-N (0-based).

dfbench ssh --id perf-may                    # defaults to --target server
dfbench ssh --id rep-demo --target replica-0

Use --print when you want the command as a string (for scripts, Wiki pages, or a second terminal tab):

dfbench ssh --id perf-may --target client --print
# -> ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa -o StrictHostKeyChecking=no \
#        -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR ubuntu@3.88.12.17

`destroy`

Delete the CloudFormation stack and the local SSH key material. Asks for confirmation unless --yes. Idempotent: a second invocation against an already-deleted stack succeeds with a short note.

dfbench destroy --id perf-may --yes
# [1/2] Deleting CloudFormation stack dfbench-perf-may...
# [2/2] Removing local SSH key files...
# Done.

dfbench destroy --id perf-may --yes
# setup "perf-may" not found (run 'dfbench list' to see available setups)

`template`

Render the CloudFormation YAML dfbench would apply for a given flag set, without calling AWS. Convenient for security review and for piping into aws cloudformation validate-template. Uses --run-id (not --id) because no state file is involved.

dfbench template \
  --run-id preview \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 1 \
  --out /tmp/dfbench-preview.yaml

aws cloudformation validate-template \
  --template-body file:///tmp/dfbench-preview.yaml

The embedded public key is a harmless placeholder; pass --public-key ~/.ssh/mykey.pub if you want the preview to reflect real key material.

`version`

dfbench version
# dfbench v0.3.0 (commit abc1234, built 2026-04-08)

Layout on disk

Every setup gets its own self-describing directory under ~/.dfbench/runs/:

~/.dfbench/runs/<id>/
  state.json                        lifecycle + AWS resources + engine + benchmark history
  stack.yaml                        CloudFormation template dfbench applied (saved before any AWS call)
  .lock                             POSIX flock; setup/run/destroy acquire it exclusively
  ssh/
    id_rsa                          4096-bit RSA private key, mode 0600, never leaves the laptop
    id_rsa.pub                      public half, embedded in the CFN template as the EC2 key pair
  benchmarks/
    20260408-143217/
      config.json                   memtier flags for this particular run
      results.json                  aggregated + per-trial stats
      memtier/
        dragonfly-t1.json           raw memtier_benchmark JSON per (engine, trial)
        dragonfly-t2.json
        ...
  commands/
    20260706-120000/
      results.json                  CommandRunResults: per-command x per-engine aggregates
      results.csv                   one row per (command, engine, trial) + median rows
      memtier/
        dragonfly-SET-t1.json       raw memtier_benchmark JSON per (engine, command, trial)
        ...
  replications/
    20260408-143217/
      results.json                  ReplicationRunResults (see below)
      benchmark_results.csv         upstream playground schema: Run_Name, Time_Seconds, RPS_Now, p99_Latency_us

Who writes what:

File	Written by	When
`state.json`	every mutating command	Before and after each phase transition (provisioning, installing, benchmarking, destroying).
`stack.yaml`	`setup`, `quickstart`, `template`	Before any AWS call, so a crashed run still leaves an audit trail.
`.lock`	`setup`, `run`, `run-commands`, `run-replication`, `destroy`	Acquired on entry, released on exit (including panics). Non-blocking: concurrent invocations fail fast.
`ssh/id_rsa`, `id_rsa.pub`	`setup`, `quickstart`	During `[1/6] Generating SSH key pair...`. Deleted by `destroy`.
`benchmarks/<ts>/`	`run`	One directory per invocation.
`commands/<ts>/`	`run-commands`	One directory per invocation.
`replications/<ts>/`	`run-replication`	One directory per invocation.

Result file schemas

Both commands produce self-contained JSON: every input flag, every instance type, every per-trial number is captured so compare can reconstruct what the run looked like months later.

Memtier (`dfbench run`, `dfbench quickstart`)

{
  "run_id": "20260408-143217",
  "created_at": "2026-04-08T14:32:17Z",
  "completed_at": "2026-04-08T14:41:05Z",
  "status": "completed",
  "region": "us-east-1",
  "aws_account": "123456789012",
  "server_instance": "c7g.4xlarge",
  "client_instance": "c7g.4xlarge",
  "memtier": {
    "threads": 8, "clients": 50, "requests": 1000000,
    "ratio": "1:10", "data_size": 256, "pipeline": 30,
    "distinct_client_seed": false
  },
  "warmup_requests": 100000,
  "trials": 3,
  "engines": [
    {
      "engine": "dragonfly",
      "version": "dragonfly v1.27.0",
      "trials": [
        {"trial": 1, "throughput": 2646502, "p50": 4.6, "p99": 7.1, "p999": 9.8, "avg_latency": 4.72}
      ],
      "throughput": {"median": 2646502, "min": 2601000, "max": 2672000, "stddev": 35400},
      "p50": {...}, "p99": {...}, "p999": {...}, "avg_latency": {...}
    }
  ]
}

Schema defined in bench/output.go (RunResults) and bench/result.go (AggregateResult, TrialResult, Stats). The aggregation is median-centred because a single outlier trial (cold cache, noisy neighbour) should not drown out the signal from the other two.

Replication (`dfbench run-replication`)

{
  "scenario": "replication",
  "run_id": "20260408-143217",
  "setup_id": "rep-demo",
  "created_at": "...",
  "completed_at": "...",
  "region": "us-east-1",
  "master_instance": "c7g.4xlarge",
  "replica_instance": "c7g.4xlarge",
  "client_instance": "c7g.4xlarge",
  "dfly_bench_ref": "main",
  "cases": [
    {
      "run_name": "qps=100000",
      "preload_args": "dfly_bench --h=... --ratio=1:0 ...",
      "load_args":    "dfly_bench --h=... --ratio=1:0 ...",
      "master_flags":  ["--maxmemory=25gb"],
      "replica_flags": ["--maxmemory=25gb", "--replication_stream_compression=zstd"],
      "series": [
        {"t_sec": 1, "rps": 198500, "p99_us": 2100},
        {"t_sec": 2, "rps": 199800, "p99_us": 2050}
      ],
      "sync_duration_ms": 3427,
      "termination_reason": "timeout"
    }
  ]
}

Schema defined in bench/replication/results.go (ReplicationRunResults, ReplicationCase, RepSample). termination_reason is one of timeout, replica_disconnect, error, ctx_cancelled.

Alongside the JSON, a benchmark_results.csv with Run_Name,Time_Seconds,RPS_Now,p99_Latency_us is written so the upstream playground compare.py drops in unchanged.

Per-command (`dfbench run-commands`)

{
  "scenario": "commands",
  "run_id": "20260706-120000",
  "setup_id": "percmd",
  "created_at": "...",
  "completed_at": "...",
  "status": "completed",
  "region": "us-east-1",
  "server_instance": "m7g.8xlarge",
  "client_instance": "c6gn.8xlarge",
  "memtier": {
    "threads": 32, "clients": 5, "pipeline": 20, "data_size": 256,
    "key_maximum": 100000000, "test_time_sec": 300, "warmup_time_sec": 10,
    "distinct_client_seed": true
  },
  "trials": 3,
  "commands": [
    {
      "command": "SET",
      "template": "SET __key__ __data__",
      "engines": [
        {
          "engine": "dragonfly",
          "version": "dragonfly v1.27.0",
          "trials": [
            {"trial": 1, "throughput": 2646502, "p50": 4.6, "p99": 7.1, "p999": 9.8, "avg_latency": 4.72}
          ],
          "throughput": {"median": 2646502, "min": 2601000, "max": 2672000, "stddev": 35400},
          "p50": {}, "p99": {}, "p999": {}, "avg_latency": {}
        }
      ]
    }
  ]
}

Schema defined in bench/commands/results.go (CommandRunResults, CommandResult); the per-engine aggregates reuse AggregateResult/Stats from bench/result.go. The commands array holds the single command the run measured (the array shape keeps the compare tooling uniform). Draining commands (DEL, LPOP, RPOP) additionally carry "approximate": true because their keyspace can empty mid-trial. A results.csv with one row per (engine, trial) plus median rows is written alongside.

Security notes

The SSH private key is 4096-bit RSA, generated locally inside ~/.dfbench/runs/<id>/ssh/id_rsa. The file is opened with O_CREATE|O_WRONLY|O_TRUNC, 0o600 so there is no window where the unencrypted bytes sit at the umask default of 0o644. See remote/sshkey.go.
Only the public half (id_rsa.pub) ships to AWS, as AWS::EC2::KeyPair.PublicKeyMaterial inside the CloudFormation template.
SSH port 22 is open to 0.0.0.0/0 on the VPC's security group. Acceptable for a short-lived, ephemeral benchmark fleet; not acceptable for long-lived hosts. If you keep setups around, narrow the rule by hand or tear them down with dfbench destroy.
The Redis port (6379) is reachable only from inside the security group (intra-SG rule with IpProtocol: -1). The operator's laptop cannot hit it directly; replication RESP is tunneled through SSH (see remote/ssh.go DialTunnel and bench/replication/orchestrator.go DialVia).
StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null are deliberate: every EC2 instance dfbench talks to was minted seconds earlier by the same process, so TOFU on an ephemeral host key would be pure ceremony. See the comment in remote/ssh.go near InsecureIgnoreHostKey.
destroy removes both key files on successful teardown. A failed teardown leaves them in place so you can retry.

Troubleshooting

Symptom	Cause	Fix
`setup` fails with `expired token` / `SSO session has expired`	AWS SSO lease expired between `aws sso login` and now.	Re-authenticate (`aws sso login`) and retry. `dfbench setup` runs `sts:GetCallerIdentity` up front for exactly this reason.
`setup` rejects `t3.small` / `t3.medium` / `a1.*`	Cluster placement groups require enhanced-networking-capable, dedicated-placement-friendly families. Burstable (`t*`) and first-gen Graviton (`a1`) don't qualify.	Use a `c` or `m` family: `c5.large`, `c6i.large`, `c7g.large` (arm64). See `ValidateClusterPlacementCompatible` in `cmd/config.go`.
`destroy` reports success but the stack is still visible in the AWS console	Historical bug (now fixed): AWS API errors from `DeleteStack` / the waiter used to be silently swallowed.	Pull latest; the refactored `Destroy` / `classifyStackDeleteResult` in `infra/cfn.go` now surface auth / quota / throttling errors as hard failures.
`run-replication` ends with `FLUSHALL on master...` and hangs for 30s+	Synchronous `FLUSHALL` blocks after a heavily-loaded run.	Already fixed: `RESPClient.FlushAll` in `bench/replication/orchestrator.go` issues `FLUSHALL ASYNC`.
`run-replication` errors: `ERR Out of memory` during preload	Default `--preload-keys` and `--value-size` exceed the instance's memory budget.	Set `--master-flags "--maxmemory=25gb"` (or larger), lower `--preload-keys`, or shrink `--value-size`. Dragonfly will evict rather than OOM.
Maxload case appears to run at ~20 req/sec	`dfly_bench --qps` defaults to 20 when omitted. dfbench explicitly emits `--qps=0` for `maxload`; if you see this, verify `--dfly-ref` resolves to a revision whose `dfly_bench` treats 0 as unbounded.	See `BuildDflyBenchArgs` in `bench/replication/dflybench.go`.
`run` refuses to start with `setup "foo" is not ready (status="benchmarking")`	Previous run crashed or was killed before the deferred status-restore ran.	Rare now; `run` and `run-replication` wrap their context with `signal.NotifyContext(Interrupt, SIGTERM)` so `Ctrl+C` unwinds cleanly. If it happens: `dfbench destroy --id ...` and start over.
`dfly_bench` build fails on the client: missing `helio/blaze.sh`	Submodule population failed.	`InstallDflyBench` in `bench/dflybench.go` falls back to a full clone + `git submodule update --init --recursive`. If it still fails, check the error output for missing apt packages (`bison`, `libboost-fiber-dev`, `libunwind-dev`).
`destroy --id foo` returns `setup "foo" not found`	Typo, or state file moved / deleted.	`dfbench list` to see what IDs exist. Old versions leaked the internal path; the current message is deliberately terse.

FAQ

Why CloudFormation and not raw EC2 API calls? The template is a single artifact you can diff, review, validate with aws cloudformation validate-template, and tear down atomically with DeleteStack. OnFailure=DELETE replaces dfbench's old hand-rolled cleanup registry: any resource that got created before a failure is rolled back server-side, no local LIFO bookkeeping needed.

Why a cluster placement group? It colocates the server, client, and replica on the same physical rack, giving sub-millisecond inter-node RTT. That's what makes benchmark numbers comparable across engines rather than swamped by AZ-level network jitter. The trade-off is the burstable/a1/mac instance-family restriction (see Troubleshooting).

Why does the client talk to the server over the private IP? Lower latency, no cross-AZ or NAT gateway charges, and the DB port never appears on the public internet.

Why does the operator laptop talk to everything over the public IP? No bastion overhead for a short-lived fleet, and the SG rule is already SSH-from-0.0.0.0/0.

Why do replication RESP calls tunnel through SSH? Master and replica listen on VPC-private addresses that the laptop cannot reach directly. Instead of adding an ingress rule for the operator's IP, dfbench uses DialTunnel (which sets up a TCP forwarding over the existing SSH session) to make the EC2 instance act as a bastion for RESP frames.

Why build dfly_bench from source on the client? Dragonfly does not publish dfly_bench as a release artifact. We pin to --dfly-ref because its stdout format has changed historically and the scrape regexes in bench/replication/dflybench.go are coupled to a known schema.

Why --preload-keys is not the number of keys written. It is the keyspace range (dfly_bench --key_maximum). Preload is time-boxed at 30 seconds matching the upstream playground, so at 4KB values and a small instance the DB ends up partially populated. That is intentional: it is the shape replication has to cope with under sustained load.

Why two connections (-c=2) during load? dfly_bench's --qps is a per-connection cap. qps=100000 -c=2 schedules 200k ops/s in total, which matches the upstream baseline line-for-line.

Why FLUSHALL ASYNC? Synchronous FLUSHALL after a write-heavy run can block for tens of seconds waiting for shard deletion. Replica teardown happens immediately afterwards, so we do not need the sync guarantee; ASYNC schedules and acks right away.

Name		Name	Last commit message	Last commit date
Latest commit History 26 Commits
.github/workflows		.github/workflows
bench		bench
benchmarks		benchmarks
cmd		cmd
engine		engine
infra		infra
internal/build		internal/build
pricing		pricing
remote		remote
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

dfbench

Table of contents

Two modes

Commands at a glance

Prerequisites

Install

30-second quickstart

Stateful lifecycle

1. Provision

2. Benchmark (as many times as you want)

3. Inspect

4. Tear down

Per-command benchmark

1. Provision (32-vCPU reference spec)

2. Benchmark one command per invocation

3. Compare across runs

Maximum-throughput configuration

Replication scenario

1. Provision with at least one replica

2. Run the sweep

3. Compare across runs

What dfly_bench is measuring

Command reference

setup

run

run-commands

Custom commands

run-commands-dfly

run-replication

quickstart

compare

info, list, ssh, destroy, template, version

info

list

ssh

destroy

template

version

Layout on disk

Result file schemas

Memtier (dfbench run, dfbench quickstart)

Replication (dfbench run-replication)

Per-command (dfbench run-commands)

Security notes

Troubleshooting

FAQ

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

What `dfly_bench` is measuring

`setup`

`run`

`run-commands`

`run-commands-dfly`

`run-replication`

`quickstart`

`compare`

`info`, `list`, `ssh`, `destroy`, `template`, `version`

`info`

`list`

`ssh`

`destroy`

`template`

`version`

Memtier (`dfbench run`, `dfbench quickstart`)

Replication (`dfbench run-replication`)

Per-command (`dfbench run-commands`)