How to run the saturation benchmark, the performance matrix and other workloads.
The saturation benchmark measures Weir's aggregation of independent single-record business requests. Client OS processes issue one record per database call or SDK RPC. Concurrency levels count total workers across all processes; with four processes, levels 8 and 32 mean 2 and 8 workers per process. Every process owns its driver or SDK connection pool and a disjoint key partition. There is no client batching.
Run on Linux with Docker so database CPU quotas can be enforced and inspected:
make benchmark
# Equivalent command; choose a fresh output directory for each run.
go run ./cmd/weir-lab -mode saturation \
-weir .tools/weir -backend all -database-cpus 1 \
-client-processes 4 \
-concurrency-levels 4,8,32,128,256 -batch-sizes 1 \
-records 8192 -payload-bytes 1024 -write-percent 0 \
-rounds 3 -warmup-duration 10s -duration 20s \
-output results/local/readweir-lab requires the binary's revision to match versions.json and refuses
to overwrite an existing output directory.
Use -write-percent 0, 10 or 100 for reads, mixed traffic or writes.
MongoDB uses one FindOne or ReplaceOne per ordinary native call. Search uses
one-ID POST _mget or single-document PUT _doc. Both paths use identical bodies,
read/write policies and a complete 10s request budget, without business retries.
Warmup, initialization and independent persisted-data verification are excluded
from timing. Pools and revision state continue from warmup into measurement.
AB/BA order alternates across rounds. Failed calls or uncertain acknowledgements
invalidate the comparison.
For Lua read-modify-write, add -lua-mutations -write-percent 100. Each native
MongoDB request uses one atomic UpdateOne aggregation pipeline, with
$set: {revision: {$subtract: [int32(1), "$revision"]}} and majority acknowledgement.
Each native Search request uses one POST _update Painless script with
ctx._source.revision = 1 - ctx._source.revision and retry_on_conflict=0.
Neither baseline performs a client read or transaction. These are native controls
for this fixed transformation, rather than a claim about arbitrary Lua programs.
The SDK sends one Lua AtomicTransform per request with the same computation.
The function updates current.revision = 1 - current.revision and returns the
current document; independent postflight checks require the BSON revision to
retain its int32 type and verify every field and final revision on both paths.
Fixture documents are identical on both paths; Weir adds no document metadata.
The logical request/result document-byte counters exclude scripts and commands,
so they are zero for transform-only calls. Backend reads and replacements still
contribute to the measured network and CPU costs.
Repeat with -backend-batch-limit 1 to disable server aggregation. Keep other
settings identical and use a fresh output directory. Adapter batch-size
distributions and the RPC/adapter ratio show whether aggregation occurred.
Reports retain per-round throughput, pooled latency histograms, client process
identities, start lag, CPU/RSS, database CPU/memory/I/O/network samples and raw
Weir metrics. Metrics report stream completions, terminal records, queue wait,
adapter time, rejections and operations per adapter invocation. MongoDB also
records selected business-command deltas at the timed boundaries for both
single-record and bulk calls. These counters exclude handshakes and monitoring
commands. Adapter invocations can split by namespace, action and byte limits;
their counts are distinct from database wire commands. Missing samples and reset
counters are reported as unavailable. business_operation_timeouts counts failed
logical operations whose request context expired before the call returned; an
expired bulk context counts every failed operation in that batch. This diagnostic
does not override confirmed mutation outcomes.
A database-saturated comparison requires both paths to independently reach saturation: at least five CPU intervals cover 80% of measurement time, mean CPU reaches 90% of the inspected quota, that threshold holds during 80% of measurement time in every round, and the next concurrency adds at most 10% throughput. The highest concurrency alone cannot prove a plateau. A plateau with low database CPU leaves maximum-capacity comparison unavailable. Reports still show observed throughput and latency at each matched concurrency. No performance ratio is a CI threshold.
Without -backend-batch-limit the lab leaves Weir's batching at its defaults.
An explicit limit (batching.max_operations) writes all four batching controls:
-backend-batch-bytes 0 writes 8 MiB, -backend-batch-wait 0 executes
immediately and -backend-max-in-flight 0 selects Weir's default of 16
concurrent executions per Store. Weir's execution limit bounds matched-concurrency
comparisons, so raise it to the highest client level when the native path should
not out-run it. Batches that are not full also follow Weir's adaptive limit,
which starts at two batches, can grow to that maximum from a little over one
probe interval after its first execution, about 12.5 seconds when executions are
fast, and holds them back while executions take more than four times their
baseline. A measurement that
starts within that time includes it. The report keeps
the generated Weir configuration, and the raw metrics preserve the actual
settings and weir_store_adaptive_executions_limit. The SDK and native callers both use the
complete 10s request budget.
These are closed-loop matched-concurrency measurements, not fixed offered-rate experiments. Local macOS measurements describe the observed workload on that host and do not show Linux database saturation or a production capacity limit.
saturation.yml runs only on manual dispatch on a Linux amd64 runner. It installs
the pinned server and (unless the backend is search) the native MongoDB
fixture, then runs one weir-lab saturation matrix with the selected write
percentage, Search CPU quota, concurrency levels, backend batch limit and
execution limit, Lua mutations, backend, mode and records per call. The default
execution limit of 256 matches the highest default client level. The default curve is
4/8/32/128/256 total workers across four client processes, three alternating
rounds and 10s warmup plus 20s measurement per path. Native Mongo uses all host
CPUs; database-cpus applies only to Docker Search. Reports, the run log and
fixture logs are uploaded as a workflow artifact.
gh workflow run saturation.yml --ref main \
-f backend=all -f write-percent=10 -f client-levels=8,32 \
-f backend-batch-limit=32 -f backend-max-in-flight=256 -f mode=saturation -f batch-sizes=1scripts/run_performance_matrix.py builds weir-lab once and runs read, mixed
(10%), write and Lua cases in sequence, at total concurrency 8/32 with three
alternating rounds. Stages use 10s warmup for MongoDB or 20s for Elasticsearch,
and 20s measurement. --warmup-duration, --duration, --database-cpus,
--concurrency-levels, --client-processes, --records, --payload-bytes,
--rounds and --batch-limit override these. Docker databases default to an
enforced 1 CPU quota and an 8192-document working set. --bulk appends read and
write cases using 32 records per call from one client process;
--bulk-concurrency-levels controls their concurrency, at most 64. Each case
writes its reports and a log into a new output directory, and the matrix stops
at the first failed case.
scripts/summarize_performance.py compares two matrix outputs, for example
before and after a versions.json update or with different batch limits:
python3 scripts/run_performance_matrix.py --backend mongo \
--mongod "$(scripts/mongod.sh)" --output results/local/mongo-before --bulk
python3 scripts/run_performance_matrix.py --backend mongo \
--mongod "$(scripts/mongod.sh)" --batch-limit 128 --output results/local/mongo-after --bulk
python3 scripts/summarize_performance.py --backend mongo \
--before results/local/mongo-before --after results/local/mongo-after \
--output results/local/mongo-comparisonThe summary recomputes throughput from all successful records and measured time,
keeps each round's throughput and p95, and requires every compared case to have
the expected report kind (single for the core cases, bulk for the bulk cases),
matching workload parameters and paired round order. Missing core cases mark the
summary partial; a one-sided bulk case is listed as missing. Server CPU time and
Go allocation costs use Prometheus counter differences around each timed stage,
including small metrics collection overhead. RSS is a sampled maximum. Physical
database commands are reported only when counters were captured around the full
stage. Histograms without separate 31 and 32 boundaries cannot determine the
exact 32-record full-batch fraction.
make diagnostic runs a finite, deterministic single-operation comparison.
Preparation, warmup, reset and postflight checks remain outside timing:
go run ./cmd/weir-lab -mode fixed -weir .tools/weir -backend all \
-operations 10000 -warmup 1000 -records 1024 -payload-bytes 1024 \
-concurrency 8 -rounds 3 -write-percent 10 -output results/local/mixedFor bulk-versus-bulk overhead measurements, use
-mode bulk-saturation -concurrency-levels 8,32,64 -batch-sizes 32.
This mode measures client batching overhead separately from the saturation
benchmark.
For independently provisioned endpoints, see cmd/weir-bench -help. It requires
explicit endpoints and creates an exclusive namespace. Remote addresses require
an explicit flag; credential-bearing URLs are rejected.
weir-bench -mode saturation runs the saturation benchmark above against a
database and Weir that you deploy, for example on separate Kubernetes nodes so
that clients, Weir and the database never share CPU. The command starts the
same independent client processes, each with its own pool, and drives the same
alternating direct/Weir ladder. Reports have the same format as weir-lab.
weir-bench -mode saturation -allow-remote -backend mongo \
-mongo-uri 'mongodb://10.0.1.20:27017/?directConnection=true' \
-weir-address 10.0.2.30:7447 -weir-diagnostics 10.0.2.30:7449 -store mongo \
-database-cpus 2 -client-processes 16 -concurrency-levels 64,256,1024,2048 \
-records 32768 -padding-bytes 1024 -write-percent 100 \
-rounds 3 -warmup-duration 10s -duration 20s -output results/local/eks-mongoWeir and Search endpoints are explicit IP:port values; resolve a Kubernetes
headless Service to its Pod IP first. The Weir Store must point at the same
database. Database CPU comes from the database itself: serverStatus
extra_info on Linux mongod and _nodes/stats/process on Elasticsearch,
divided by -database-cpus, which must equal the database's CPU limit.
Each sample uses the database's own read time, and Weir CPU and RSS come from
its diagnostics /metrics. The MongoDB URI must reach the mongod that takes
the writes, usually a single node or the primary; a mongos router is
rejected. Elasticsearch must be a single node. The run stops before the ladder
when the database reports no process CPU, such as mongod on macOS. Client CPU
uses /proc on Linux, so a minimal image without ps works.
The whole run is bounded by -timeout, derived from the ladder when unset.
Run the client on its own node with enough CPU for the highest level; give the database fixed CPU requests and limits so its utilization has a stable denominator. Results describe that database size and topology, not a production capacity limit.
Mongo endpoints use a bounded (4096-byte), credential-free mongodb:// profile
with an empty or root path and explicit numeric ports (1–65535). A direct profile
has one seed, no replica set, and omitted or true directConnection. Replica
discovery uses 1–7 unique seeds, an explicit replicaSet (1–64 ASCII letters,
digits, _ or -), and omitted or false directConnection, for example
mongodb://127.0.0.1:27017,[::1]:27018/?replicaSet=bench.
Only those two query keys are accepted; authentication, TLS options, database
paths, SRV, load balancing and read-preference overrides are not supported here.
Every seed is validated, including canonical duplicate detection. Loopback IPs
and literal localhost are permitted by default; every other IP or syntactically
valid DNS name requires --allow-remote. Validation never resolves DNS. This is
seed authorization, not network confinement: discovery may dial members not
listed in the supplied seeds. Programmatic workload callers must set
Config.AllowRemote for the same opt-in.
The comparison path named direct means the native database client, not
directConnection=true. Its Mongo options preserve replica discovery, primary
reads, majority writes, disabled retries/compression and worker-sized pooling.
Discovery uses poll monitoring with a 1-second heartbeat, matching the server;
direct profiles retain the driver's default heartbeat. Reports record canonical
credential-free seeds, profile, replica-set name and polling policy for the
native connection. Parser/options tests are offline checks; live discovery and
primary movement need a separate replica-set fixture.