Skip to content

Latest commit

 

History

History
259 lines (214 loc) · 13.5 KB

benchmarks.md

File metadata and controls

259 lines (214 loc) · 13.5 KB

Benchmarks

How to run the saturation benchmark, the performance matrix and other workloads.

Saturation benchmark

The saturation benchmark measures Weir's aggregation of independent single-record business requests. Client OS processes issue one record per database call or SDK RPC. Concurrency levels count total workers across all processes; with four processes, levels 8 and 32 mean 2 and 8 workers per process. Every process owns its driver or SDK connection pool and a disjoint key partition. There is no client batching.

Run

Run on Linux with Docker so database CPU quotas can be enforced and inspected:

make benchmark

# Equivalent command; choose a fresh output directory for each run.
go run ./cmd/weir-lab -mode saturation \
  -weir .tools/weir -backend all -database-cpus 1 \
  -client-processes 4 \
  -concurrency-levels 4,8,32,128,256 -batch-sizes 1 \
  -records 8192 -payload-bytes 1024 -write-percent 0 \
  -rounds 3 -warmup-duration 10s -duration 20s \
  -output results/local/read

weir-lab requires the binary's revision to match versions.json and refuses to overwrite an existing output directory.

Workload

Use -write-percent 0, 10 or 100 for reads, mixed traffic or writes. MongoDB uses one FindOne or ReplaceOne per ordinary native call. Search uses one-ID POST _mget or single-document PUT _doc. Both paths use identical bodies, read/write policies and a complete 10s request budget, without business retries. Warmup, initialization and independent persisted-data verification are excluded from timing. Pools and revision state continue from warmup into measurement. AB/BA order alternates across rounds. Failed calls or uncertain acknowledgements invalidate the comparison.

Lua read-modify-write

For Lua read-modify-write, add -lua-mutations -write-percent 100. Each native MongoDB request uses one atomic UpdateOne aggregation pipeline, with $set: {revision: {$subtract: [int32(1), "$revision"]}} and majority acknowledgement. Each native Search request uses one POST _update Painless script with ctx._source.revision = 1 - ctx._source.revision and retry_on_conflict=0. Neither baseline performs a client read or transaction. These are native controls for this fixed transformation, rather than a claim about arbitrary Lua programs. The SDK sends one Lua AtomicTransform per request with the same computation. The function updates current.revision = 1 - current.revision and returns the current document; independent postflight checks require the BSON revision to retain its int32 type and verify every field and final revision on both paths. Fixture documents are identical on both paths; Weir adds no document metadata. The logical request/result document-byte counters exclude scripts and commands, so they are zero for transform-only calls. Backend reads and replacements still contribute to the measured network and CPU costs.

Disable aggregation

Repeat with -backend-batch-limit 1 to disable server aggregation. Keep other settings identical and use a fresh output directory. Adapter batch-size distributions and the RPC/adapter ratio show whether aggregation occurred.

Reports

Reports retain per-round throughput, pooled latency histograms, client process identities, start lag, CPU/RSS, database CPU/memory/I/O/network samples and raw Weir metrics. Metrics report stream completions, terminal records, queue wait, adapter time, rejections and operations per adapter invocation. MongoDB also records selected business-command deltas at the timed boundaries for both single-record and bulk calls. These counters exclude handshakes and monitoring commands. Adapter invocations can split by namespace, action and byte limits; their counts are distinct from database wire commands. Missing samples and reset counters are reported as unavailable. business_operation_timeouts counts failed logical operations whose request context expired before the call returned; an expired bulk context counts every failed operation in that batch. This diagnostic does not override confirmed mutation outcomes.

Saturation criteria

A database-saturated comparison requires both paths to independently reach saturation: at least five CPU intervals cover 80% of measurement time, mean CPU reaches 90% of the inspected quota, that threshold holds during 80% of measurement time in every round, and the next concurrency adds at most 10% throughput. The highest concurrency alone cannot prove a plateau. A plateau with low database CPU leaves maximum-capacity comparison unavailable. Reports still show observed throughput and latency at each matched concurrency. No performance ratio is a CI threshold.

Batching settings

Without -backend-batch-limit the lab leaves Weir's batching at its defaults. An explicit limit (batching.max_operations) writes all four batching controls: -backend-batch-bytes 0 writes 8 MiB, -backend-batch-wait 0 executes immediately and -backend-max-in-flight 0 selects Weir's default of 16 concurrent executions per Store. Weir's execution limit bounds matched-concurrency comparisons, so raise it to the highest client level when the native path should not out-run it. Batches that are not full also follow Weir's adaptive limit, which starts at two batches, can grow to that maximum from a little over one probe interval after its first execution, about 12.5 seconds when executions are fast, and holds them back while executions take more than four times their baseline. A measurement that starts within that time includes it. The report keeps the generated Weir configuration, and the raw metrics preserve the actual settings and weir_store_adaptive_executions_limit. The SDK and native callers both use the complete 10s request budget.

Limits of these measurements

These are closed-loop matched-concurrency measurements, not fixed offered-rate experiments. Local macOS measurements describe the observed workload on that host and do not show Linux database saturation or a production capacity limit.

Saturation workflow

saturation.yml runs only on manual dispatch on a Linux amd64 runner. It installs the pinned server and (unless the backend is search) the native MongoDB fixture, then runs one weir-lab saturation matrix with the selected write percentage, Search CPU quota, concurrency levels, backend batch limit and execution limit, Lua mutations, backend, mode and records per call. The default execution limit of 256 matches the highest default client level. The default curve is 4/8/32/128/256 total workers across four client processes, three alternating rounds and 10s warmup plus 20s measurement per path. Native Mongo uses all host CPUs; database-cpus applies only to Docker Search. Reports, the run log and fixture logs are uploaded as a workflow artifact.

gh workflow run saturation.yml --ref main \
  -f backend=all -f write-percent=10 -f client-levels=8,32 \
  -f backend-batch-limit=32 -f backend-max-in-flight=256 -f mode=saturation -f batch-sizes=1

Performance matrix

scripts/run_performance_matrix.py builds weir-lab once and runs read, mixed (10%), write and Lua cases in sequence, at total concurrency 8/32 with three alternating rounds. Stages use 10s warmup for MongoDB or 20s for Elasticsearch, and 20s measurement. --warmup-duration, --duration, --database-cpus, --concurrency-levels, --client-processes, --records, --payload-bytes, --rounds and --batch-limit override these. Docker databases default to an enforced 1 CPU quota and an 8192-document working set. --bulk appends read and write cases using 32 records per call from one client process; --bulk-concurrency-levels controls their concurrency, at most 64. Each case writes its reports and a log into a new output directory, and the matrix stops at the first failed case.

scripts/summarize_performance.py compares two matrix outputs, for example before and after a versions.json update or with different batch limits:

python3 scripts/run_performance_matrix.py --backend mongo \
  --mongod "$(scripts/mongod.sh)" --output results/local/mongo-before --bulk
python3 scripts/run_performance_matrix.py --backend mongo \
  --mongod "$(scripts/mongod.sh)" --batch-limit 128 --output results/local/mongo-after --bulk
python3 scripts/summarize_performance.py --backend mongo \
  --before results/local/mongo-before --after results/local/mongo-after \
  --output results/local/mongo-comparison

Summary

The summary recomputes throughput from all successful records and measured time, keeps each round's throughput and p95, and requires every compared case to have the expected report kind (single for the core cases, bulk for the bulk cases), matching workload parameters and paired round order. Missing core cases mark the summary partial; a one-sided bulk case is listed as missing. Server CPU time and Go allocation costs use Prometheus counter differences around each timed stage, including small metrics collection overhead. RSS is a sampled maximum. Physical database commands are reported only when counters were captured around the full stage. Histograms without separate 31 and 32 boundaries cannot determine the exact 32-record full-batch fraction.

Other workload modes

make diagnostic runs a finite, deterministic single-operation comparison. Preparation, warmup, reset and postflight checks remain outside timing:

go run ./cmd/weir-lab -mode fixed -weir .tools/weir -backend all \
  -operations 10000 -warmup 1000 -records 1024 -payload-bytes 1024 \
  -concurrency 8 -rounds 3 -write-percent 10 -output results/local/mixed

For bulk-versus-bulk overhead measurements, use -mode bulk-saturation -concurrency-levels 8,32,64 -batch-sizes 32. This mode measures client batching overhead separately from the saturation benchmark.

For independently provisioned endpoints, see cmd/weir-bench -help. It requires explicit endpoints and creates an exclusive namespace. Remote addresses require an explicit flag; credential-bearing URLs are rejected.

Saturation against deployed services

weir-bench -mode saturation runs the saturation benchmark above against a database and Weir that you deploy, for example on separate Kubernetes nodes so that clients, Weir and the database never share CPU. The command starts the same independent client processes, each with its own pool, and drives the same alternating direct/Weir ladder. Reports have the same format as weir-lab.

weir-bench -mode saturation -allow-remote -backend mongo \
  -mongo-uri 'mongodb://10.0.1.20:27017/?directConnection=true' \
  -weir-address 10.0.2.30:7447 -weir-diagnostics 10.0.2.30:7449 -store mongo \
  -database-cpus 2 -client-processes 16 -concurrency-levels 64,256,1024,2048 \
  -records 32768 -padding-bytes 1024 -write-percent 100 \
  -rounds 3 -warmup-duration 10s -duration 20s -output results/local/eks-mongo

Weir and Search endpoints are explicit IP:port values; resolve a Kubernetes headless Service to its Pod IP first. The Weir Store must point at the same database. Database CPU comes from the database itself: serverStatus extra_info on Linux mongod and _nodes/stats/process on Elasticsearch, divided by -database-cpus, which must equal the database's CPU limit. Each sample uses the database's own read time, and Weir CPU and RSS come from its diagnostics /metrics. The MongoDB URI must reach the mongod that takes the writes, usually a single node or the primary; a mongos router is rejected. Elasticsearch must be a single node. The run stops before the ladder when the database reports no process CPU, such as mongod on macOS. Client CPU uses /proc on Linux, so a minimal image without ps works. The whole run is bounded by -timeout, derived from the ladder when unset.

Run the client on its own node with enough CPU for the highest level; give the database fixed CPU requests and limits so its utilization has a stable denominator. Results describe that database size and topology, not a production capacity limit.

Mongo endpoint profiles

Mongo endpoints use a bounded (4096-byte), credential-free mongodb:// profile with an empty or root path and explicit numeric ports (1–65535). A direct profile has one seed, no replica set, and omitted or true directConnection. Replica discovery uses 1–7 unique seeds, an explicit replicaSet (1–64 ASCII letters, digits, _ or -), and omitted or false directConnection, for example mongodb://127.0.0.1:27017,[::1]:27018/?replicaSet=bench. Only those two query keys are accepted; authentication, TLS options, database paths, SRV, load balancing and read-preference overrides are not supported here. Every seed is validated, including canonical duplicate detection. Loopback IPs and literal localhost are permitted by default; every other IP or syntactically valid DNS name requires --allow-remote. Validation never resolves DNS. This is seed authorization, not network confinement: discovery may dial members not listed in the supplied seeds. Programmatic workload callers must set Config.AllowRemote for the same opt-in.

The direct path

The comparison path named direct means the native database client, not directConnection=true. Its Mongo options preserve replica discovery, primary reads, majority writes, disabled retries/compression and worker-sized pooling. Discovery uses poll monitoring with a 1-second heartbeat, matching the server; direct profiles retain the driver's default heartbeat. Reports record canonical credential-free seeds, profile, replica-set name and polling policy for the native connection. Parser/options tests are offline checks; live discovery and primary movement need a separate replica-set fixture.