diff --git a/README.md b/README.md index 2459e98..757e564 100644 --- a/README.md +++ b/README.md @@ -248,17 +248,18 @@ dfbench setup \ ```bash dfbench run-commands --id percmd \ --engines dragonfly,redis,valkey \ - --command SET \ + --command SET --dragonfly_num_shards -1 \ --memtier-threads 32 --memtier-clients 5 --memtier-pipeline 20 ``` Repeat with the next command whenever you're ready - the setup stays -provisioned between runs: +provisioned between runs (`--dragonfly_num_shards` is required whenever +dragonfly is among the engines; `-1` auto-derives the tuned value): ```bash -dfbench run-commands --id percmd --command GET -dfbench run-commands --id percmd --command LPUSH -dfbench run-commands --id percmd --command LRANGE +dfbench run-commands --id percmd --command GET --dragonfly_num_shards -1 +dfbench run-commands --id percmd --command LPUSH --dragonfly_num_shards -1 +dfbench run-commands --id percmd --command LRANGE --dragonfly_num_shards -1 ``` For each engine the scenario: @@ -329,15 +330,16 @@ dfbench setup --id maxqps --tune-network --ubuntu-version 24.04 \ --engines dragonfly ``` -Then benchmark. `--dragonfly_num_shards` (auto), -`--dragonfly_conn_use_incoming_cpu`, and `--redis_pin_network` / -`--valkey_pin_network` default to the tuned configuration, so a plain -`run-commands` already reproduces it: +Then benchmark. `--dragonfly_num_shards` must be stated explicitly when +benchmarking dragonfly (pass `-1` for the auto-derived tuned value); +`--dragonfly_conn_use_incoming_cpu` and `--redis_pin_network` / +`--valkey_pin_network` default to the tuned configuration: ```bash dfbench run-commands --id maxqps --engines dragonfly \ --command SET --memtier-threads 32 --memtier-clients 5 \ --memtier-pipeline 35 --memtier-data-size 128 \ + --dragonfly_num_shards -1 \ --key-maximum 100000000 --test-time 300 \ --trials 1 --warmup-time 0 --distinct-client-seed ``` @@ -367,12 +369,12 @@ Notes and caveats: ```bash # Baseline: pin an older release. dfbench run-commands --id maxqps --engines dragonfly --command SET \ - --dragonfly_version v1.31.0 \ + --dragonfly_version v1.31.0 --dragonfly_num_shards -1 \ --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1 # Candidate: latest release (or a nightly/custom tarball URL). dfbench run-commands --id maxqps --engines dragonfly --command SET \ - --dragonfly_version latest \ + --dragonfly_version latest --dragonfly_num_shards -1 \ --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1 ``` @@ -385,7 +387,7 @@ dfbench run-commands --id maxqps --engines dragonfly --command SET \ ```bash # Local binary dropped into the repo (path is relative to your CWD). dfbench run-commands --id maxqps --engines dragonfly --command SET \ - --dragonfly_version ./bin/dragonfly \ + --dragonfly_version ./bin/dragonfly --dragonfly_num_shards -1 \ --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1 ``` @@ -596,7 +598,7 @@ Run it once per command you want to profile. | `--trials` | `3` | Measured trials per engine. Aggregated as median with [min..max]. | | `--distinct-client-seed` | `true` | Each memtier client gets its own RNG seed so the 160 clients don't replay the same key sequence. Disable with `=false`. | | `--redis_io_threads`, `--valkey_io_threads` | `0` (one per vCPU) | Override `--io-threads` for Redis / Valkey respectively; their docs suggest ~8 can outperform higher counts on many-core boxes. Dragonfly is unaffected (proactor threads always match vCPUs). Recorded in `results.json` as `io_threads` (per engine). Deprecated shared alias: `--io-threads` (sets both). | -| `--dragonfly_num_shards` | `-1` (auto) | Dragonfly `--num_shards`. `-1` auto-derives `VCPUS - 2*NUM_RX_QUEUES` on the server so shards stay off the network-I/O CPUs that `setup --tune-network` reserves; `0` uses Dragonfly's default (one shard per proactor); `>0` sets it explicitly. Ignored by Redis/Valkey. Recorded as `num_shards`. Deprecated alias: `--num-shards`. | +| `--dragonfly_num_shards` | none (required) | Dragonfly `--num_shards`. Required when benchmarking dragonfly so the config always states the shard count explicitly (no silent default). `-1` auto-derives `VCPUS - 2*NUM_RX_QUEUES` on the server so shards stay off the network-I/O CPUs that `setup --tune-network` reserves; `0` uses Dragonfly's default (one shard per proactor); `>0` sets it explicitly. Ignored by Redis/Valkey. Recorded as `num_shards`. Deprecated alias: `--num-shards`. | | `--dragonfly_conn_use_incoming_cpu` | `true` | Run Dragonfly with `--conn_use_incoming_cpu` so each connection's I/O runs on the CPU that received its NIC interrupt. Pair with `dfbench setup --tune-network`. Disable with `=false`. Ignored by Redis/Valkey. Recorded as `conn_use_incoming_cpu`. Deprecated alias: `--conn-use-incoming-cpu`. | | `--redis_pin_network`, `--valkey_pin_network` | `true` | Best-effort `taskset` pinning for Redis / Valkey respectively onto CPU 0 plus the network-I/O CPUs (falls back to no pinning if NIC-queue detection fails). Disable with `=false`. Ignored by Dragonfly. Recorded as `pin_network` (per engine). Deprecated shared alias: `--pin-network` (sets both). | | `--dragonfly_version` | `""` (keep installed) | Reinstall Dragonfly at this build before benchmarking so version deltas are measured on identical tuning. Accepts a release tag (`v1.31.0`), `latest`, a full `https://` URL to a `dragonfly-.tar.gz` tarball, or a path to a local file (a raw binary or `.tar.gz` in the repo) that is uploaded to the server. Empty keeps whatever `setup` installed. Ignored by Redis/Valkey. The installed version is recorded per-engine in `results.json`. Deprecated alias: `--dfly-version`. | @@ -609,7 +611,7 @@ measurements): ```bash dfbench run-commands --id percmd \ --engines dragonfly \ - --command GET \ + --command GET --dragonfly_num_shards -1 \ --key-maximum 1000000 --test-time 15 --trials 1 ``` @@ -672,8 +674,8 @@ server; they are unrelated. | `--dfly-bench-pipeline` | `20` | `dfly_bench --pipeline`: max pending requests per connection. | | `--dfly-bench-data-size` | `256` | `dfly_bench -d`: value size in bytes (`__data__` placeholders). | | `--key-maximum` | `100000000` | Key range (`--key_maximum`). Structured preloads use a built-in 1M-key range. | -| `--key-dist` | `U` | `dfly_bench --key_dist` for the measured pass: `U` (uniform, the memtier random `R` equivalent), `N`, `Z`, `S`. | -| `--qps` | `0` | `dfly_bench --qps` per connection; `0` = closed-loop max load (coordinated omission). | +| `--dfly-bench-key-dist` | `U` | `dfly_bench --key_dist` for the measured pass: `U` (uniform, the memtier random `R` equivalent), `N`, `Z`, `S`. | +| `--dfly-bench-qps` | `0` | `dfly_bench --qps` per connection; `0` = closed-loop max load (coordinated omission). | | `--test-time` | `300` | Seconds of measured load per trial. | | `--warmup-time` | `10` | Seconds of discarded warmup load per engine. `0` disables. | | `--trials` | `3` | Measured trials per engine (median with [min..max]). | @@ -689,7 +691,7 @@ avg columns for both tools. ```bash dfbench run-commands-dfly --id percmd \ --engines dragonfly \ - --command GET \ + --command GET --dragonfly_num_shards -1 \ --key-maximum 1000000 --test-time 15 --trials 1 ``` diff --git a/benchmarks/GET/dfly_bench/GET_reproduce.md b/benchmarks/GET/dfly_bench/GET_reproduce.md index 2341288..69cea79 100644 --- a/benchmarks/GET/dfly_bench/GET_reproduce.md +++ b/benchmarks/GET/dfly_bench/GET_reproduce.md @@ -52,14 +52,14 @@ in the chosen AZ. dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command GET \ --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ --dragonfly_num_shards 16 \ - --key-maximum 100000000 --key-dist U \ + --key-maximum 100000000 --test-time 60 --warmup-time 10 --trials 1 \ --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 30 --dfly-bench-data-size 128 \ - --qps 0 --test-time 60 --warmup-time 10 --trials 1 + --dfly-bench-key-dist U --dfly-bench-qps 0 ``` -`--qps 0` is closed-loop max load (coordinated omission). To pin throughput near a -target rate instead, set `--qps` per connection = target_total / 160 (e.g. -`--qps 68750` ≈ 11M ops/s total). Redis/Valkey are the preinstalled builds started +`--dfly-bench-qps 0` is closed-loop max load (coordinated omission). To pin throughput near a +target rate instead, set `--dfly-bench-qps` per connection = target_total / 160 (e.g. +`--dfly-bench-qps 68750` ≈ 11M ops/s total). Redis/Valkey are the preinstalled builds started with `io_threads=32` (auto); only Dragonfly is reinstalled from `--dragonfly_version`. ## Engine comparison (dragonfly vs redis vs valkey) @@ -89,16 +89,16 @@ Server: m7g.8xlarge (arm64) | Client: c6gn.8xlarge (arm64) | value size 128B, 16 | Pipeline | Rate | Throughput | p50 | p99 | p99.9 | Avg latency | | -------- | ---------- | ---------- | -------- | -------- | ---------- | ----------- | -| 120 | `--qps 0` | 13.47M ops/s | 1.064 ms | 7.435 ms | 33.307 ms | 1.344 ms | -| 80 | `--qps 0` | 12.98M ops/s | 0.780 ms | 3.971 ms | 15.049 ms | 0.928 ms | -| 30 | `--qps 68750` (~11M) | 11.82M ops/s | 1.064 ms | 3.669 ms | 7.666 ms | 1.173 ms | -| **30** | **`--qps 0`** | **10.31M ops/s** | **0.431 ms** | **0.765 ms** | **0.996 ms** | **0.442 ms** | +| 120 | `--dfly-bench-qps 0` | 13.47M ops/s | 1.064 ms | 7.435 ms | 33.307 ms | 1.344 ms | +| 80 | `--dfly-bench-qps 0` | 12.98M ops/s | 0.780 ms | 3.971 ms | 15.049 ms | 0.928 ms | +| 30 | `--dfly-bench-qps 68750` (~11M) | 11.82M ops/s | 1.064 ms | 3.669 ms | 7.666 ms | 1.173 ms | +| **30** | **`--dfly-bench-qps 0`** | **10.31M ops/s** | **0.431 ms** | **0.765 ms** | **0.996 ms** | **0.442 ms** | **Reading the sweep:** with 160 connections at max load, tail latency scales with in-flight depth (`conns x pipeline`). Dropping the pipeline from 120 to 30 cuts p99.9 by ~33x (33.3 ms -> 1.0 ms) while only trimming throughput ~24% (13.5M -> 10.3M), because Dragonfly is CPU-bound and 160x30 still keeps every shard busy. -Throttling pipeline-30 up toward 11M (`--qps 68750`) recovers throughput but +Throttling pipeline-30 up toward 11M (`--dfly-bench-qps 68750`) recovers throughput but re-inflates the tail to ~7.7 ms — still under 10 ms, if you want the extra ops/s. ## Comparison with memtier @@ -108,7 +108,7 @@ measured Dragonfly at **11.63M ops/s, p99.9 = 7.97 ms** with pipeline 120. dfly_ at pipeline 120 pushes harder (13.5M ops/s) but with a much longer tail, so the two tools are not directly comparable at identical pipeline settings — dfly_bench's closed-loop driver keeps deeper queues. For a memtier-like tail, use a lower -pipeline (30) or throttle with `--qps`. +pipeline (30) or throttle with `--dfly-bench-qps`. Both tools' results share the same schema (this run is tagged `tool="dfly_bench"`), so a memtier GET file and a dfly_bench GET file diff with `dfbench compare`. diff --git a/benchmarks/GET/memtier/GET_reproduce.md b/benchmarks/GET/memtier/GET_reproduce.md index 6fd1b76..9c5e1de 100644 --- a/benchmarks/GET/memtier/GET_reproduce.md +++ b/benchmarks/GET/memtier/GET_reproduce.md @@ -62,9 +62,9 @@ load generator instead of memtier. dfly_bench is installed on the client at dfbench run-commands-dfly --id maxqps2 --engines dragonfly --command GET \ --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ --dragonfly_num_shards 16 \ + --key-maximum 100000000 --test-time 60 --warmup-time 10 --trials 1 \ --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 120 --dfly-bench-data-size 128 \ - --key-maximum 100000000 --key-dist U --qps 0 \ - --test-time 60 --warmup-time 10 --trials 1 + --dfly-bench-key-dist U --dfly-bench-qps 0 ``` Notes: @@ -74,8 +74,8 @@ Notes: thread covering the whole `--key_maximum` range) rather than reused from a memtier preload. - **Flag mapping:** `--dfly-bench-threads` -> `--proactor_threads`, `--dfly-bench-conns` -> - `-c` (32x5 = 160 connections), `--key-dist U` approximates memtier's random - key pattern `R`, `--qps 0` = closed-loop max load. + `-c` (32x5 = 160 connections), `--dfly-bench-key-dist U` approximates memtier's random + key pattern `R`, `--dfly-bench-qps 0` = closed-loop max load. - Results use the same schema (tagged `tool="dfly_bench"`), so `dfbench compare ` diffs the two tools. - dfly_bench's `--json_out_file` writes a memtier-compatible latency report, so diff --git a/benchmarks/SET/dfly_bench/SET_reproduce.md b/benchmarks/SET/dfly_bench/SET_reproduce.md index c99bdd9..b58cc6b 100644 --- a/benchmarks/SET/dfly_bench/SET_reproduce.md +++ b/benchmarks/SET/dfly_bench/SET_reproduce.md @@ -47,15 +47,15 @@ instance types must have capacity in the chosen AZ. dfbench run-commands-dfly --id maxqps2 --engines dragonfly,redis,valkey --command SET \ --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ --dragonfly_num_shards 16 --dragonfly_conn_use_incoming_cpu=true \ - --key-maximum 100000000 --key-dist U \ + --redis_io_threads 8 --valkey_io_threads 8 \ + --key-maximum 100000000 --test-time 300 --warmup-time 10 --trials 1 \ --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 30 --dfly-bench-data-size 128 \ - --qps 0 --test-time 300 --warmup-time 10 --trials 1 \ - --redis_io_threads 8 --valkey_io_threads 8 + --dfly-bench-key-dist U --dfly-bench-qps 0 ``` ## Results -Latest 3-engine run (benchmark `20260721-115805`, pipeline 30, `--qps 0`, 160 +Latest 3-engine run (benchmark `20260721-115805`, pipeline 30, `--dfly-bench-qps 0`, 160 connections, 1 x 300s trial, value size 128B). | Engine | Throughput (median) | p50 | p99 | p99.9 | Avg latency | diff --git a/benchmarks/SET/memtier/SET_reproduce.md b/benchmarks/SET/memtier/SET_reproduce.md index cc78f31..0340d7b 100644 --- a/benchmarks/SET/memtier/SET_reproduce.md +++ b/benchmarks/SET/memtier/SET_reproduce.md @@ -35,13 +35,14 @@ load generator instead of memtier. dfly_bench is installed on the client at ``` dfbench run-commands-dfly --id test --engines dragonfly --command SET \ + --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz \ --dragonfly_num_shards 16 \ + --key-maximum 100000000 --test-time 300 --trials 1 \ --dfly-bench-threads 32 --dfly-bench-conns 5 --dfly-bench-pipeline 30 --dfly-bench-data-size 128 \ - --key-maximum 100000000 --qps 0 \ - --test-time 300 --trials 1 --dragonfly_version s3://df-use1-pub/v2/dragonfly-aarch64.tar.gz + --dfly-bench-qps 0 ``` Flag mapping vs memtier: `--dfly-bench-threads` -> `--proactor_threads`, `--dfly-bench-conns` --> `-c` (so 32x5 = 160 connections, same as memtier threadsxclients), `--qps 0` +-> `-c` (so 32x5 = 160 connections, same as memtier threadsxclients), `--dfly-bench-qps 0` = closed-loop max load. Results use the same schema (tagged `tool="dfly_bench"`), so `dfbench compare ` diffs the two tools. diff --git a/cmd/run_commands.go b/cmd/run_commands.go index 2d4f3d7..a640d29 100644 --- a/cmd/run_commands.go +++ b/cmd/run_commands.go @@ -6,6 +6,7 @@ import ( "os" "os/signal" "path/filepath" + "slices" "strconv" "strings" "syscall" @@ -102,6 +103,14 @@ marked approximate in the output.`, if keyMaximum <= 0 { return fmt.Errorf("--key-maximum must be > 0, got %d", keyMaximum) } + // num_shards must be stated explicitly (via the flag or its + // deprecated alias) so a config never hides the shard count + // behind a silent auto default. + if slices.Contains(engines, "dragonfly") && + !cmd.Flags().Changed("dragonfly_num_shards") && !cmd.Flags().Changed("num-shards") { + return fmt.Errorf("--dragonfly_num_shards is required when benchmarking dragonfly; " + + "pass an explicit value (-1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server, 0 uses Dragonfly's default, >0 sets it explicitly)") + } // Resolve engine-specific tuning: the engine-prefixed flag wins, // else the deprecated shared alias, else the default. tuning, err := resolveEngineTuning(cmd.Flags()) @@ -157,7 +166,7 @@ marked approximate in the output.`, // Dragonfly-specific tuning (ignored by Redis/Valkey). f.IntVar(&dflyNumShards, "dragonfly_num_shards", -1, - "Dragonfly --num_shards: -1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server (keeps shards off the network-I/O CPUs), 0 uses Dragonfly's default (one shard per proactor), >0 sets it explicitly.") + "Dragonfly --num_shards (required when benchmarking dragonfly): -1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server (keeps shards off the network-I/O CPUs), 0 uses Dragonfly's default (one shard per proactor), >0 sets it explicitly.") f.BoolVar(&dflyConnIncoming, "dragonfly_conn_use_incoming_cpu", true, "Run Dragonfly with --conn_use_incoming_cpu so each connection's I/O runs on the CPU that received its NIC interrupt. Pair with 'dfbench setup --tune-network'. Disable with =false.") f.StringVar(&dflyVersion, "dragonfly_version", "", diff --git a/cmd/run_commands_dfly.go b/cmd/run_commands_dfly.go index 8264ff8..f55c40c 100644 --- a/cmd/run_commands_dfly.go +++ b/cmd/run_commands_dfly.go @@ -6,6 +6,7 @@ import ( "os" "os/signal" "path/filepath" + "slices" "strings" "syscall" "time" @@ -93,6 +94,10 @@ compare'.`, if keyMaximum <= 0 { return fmt.Errorf("--key-maximum must be > 0, got %d", keyMaximum) } + if slices.Contains(engines, "dragonfly") && !cmd.Flags().Changed("dragonfly_num_shards") { + return fmt.Errorf("--dragonfly_num_shards is required when benchmarking dragonfly; " + + "pass an explicit value (-1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server, 0 uses Dragonfly's default, >0 sets it explicitly)") + } tuning, err := resolveEngineTuning(cmd.Flags()) if err != nil { return err @@ -152,9 +157,16 @@ compare'.`, _ = f.MarkDeprecated("dfly-data-size", "use --dfly-bench-data-size") f.IntVar(&keyMaximum, "key-maximum", 100_000_000, "Key range (--key_maximum). Size it to 30-90% of server memory. Commands with multi-item structured preloads use a built-in 1M-key range instead.") - f.StringVar(&keyDist, "key-dist", "U", + f.StringVar(&keyDist, "dfly-bench-key-dist", "U", "dfly_bench --key_dist for the measured pass: U (uniform, the memtier random equivalent), N (normal), Z (zipfian), S (sequential)") - f.IntVar(&qps, "qps", 0, "dfly_bench --qps per connection; 0 means closed-loop max load (coordinated omission)") + f.IntVar(&qps, "dfly-bench-qps", 0, "dfly_bench --qps per connection; 0 means closed-loop max load (coordinated omission)") + + // Deprecated pre-prefix aliases: keep old commands working. Hidden from + // help and print a deprecation notice when used. + f.StringVar(&keyDist, "key-dist", "U", "Deprecated alias for --dfly-bench-key-dist") + f.IntVar(&qps, "qps", 0, "Deprecated alias for --dfly-bench-qps") + _ = f.MarkDeprecated("key-dist", "use --dfly-bench-key-dist") + _ = f.MarkDeprecated("qps", "use --dfly-bench-qps") f.IntVar(&testTime, "test-time", 300, "Seconds of measured load per trial") f.IntVar(&warmupTime, "warmup-time", 10, "Seconds of discarded warmup load per (engine, command); 0 disables") f.IntVar(&trials, "trials", 3, "Number of measured trials per (engine, command) (>=1)") @@ -163,7 +175,7 @@ compare'.`, // Dragonfly-specific server tuning (ignored by Redis/Valkey). f.IntVar(&dflyNumShards, "dragonfly_num_shards", -1, - "Dragonfly --num_shards: -1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server, 0 uses Dragonfly's default, >0 sets it explicitly.") + "Dragonfly --num_shards (required when benchmarking dragonfly): -1 auto-derives VCPUS - 2*NUM_RX_QUEUES on the server, 0 uses Dragonfly's default, >0 sets it explicitly.") f.BoolVar(&dflyConnIncoming, "dragonfly_conn_use_incoming_cpu", true, "Run Dragonfly with --conn_use_incoming_cpu. Pair with 'dfbench setup --tune-network'. Disable with =false.") f.StringVar(&dflyVersion, "dragonfly_version", "",