ezmsg-perf: much more comprehensive perf test with analysis

[griffinmilsap]

ezmsg-perf

More and better perf tests with an analysis suite! Addresses aspects of #7. Generates reports like this one: report_20250920_174928.html

Description

The current ezmsg performance test utils/perf_test.py is insufficient to understand how backend changes affect system performance comprehensively. This PR adds a whole test suite for assessing backend changes systematically. It does so by implementing new test configurations and evaluates different communication strategies.

Configurations (config):

• fanin: many publishers to one subscriber
• fanout: one publisher to many subscribers
• relay: one publisher to one subscriber through many relays

Communication strategies (comms):

• local: all subs, relays, and pubs are in the SAME process
• shm / tcp: some clients move to a second process; comms via shared memory / TCP
  • fanin: all publishers moved
  • fanout: all subscribers moved
  • relay: the publisher and all relay nodes moved
• shm_spread / tcp_spread: each client in its own process; comms via SHM / TCP respectively

Variables:

• n_clients: pubs (fanin), subs (fanout), or relays (relay)
• msg_size: nominal message size (bytes)

Metrics:

• sample_rate: messages/sec at the sink (higher = better)
• data_rate: bytes/sec at the sink (higher = better)
• latency_mean: average send -> receive latency in seconds (lower = better)

Usage

❯ ezmsg-perf       
usage: ezmsg-perf [-h] {run,summary} ...
ezmsg-perf: error: the following arguments are required: command

❯ ezmsg-perf run -h
usage: ezmsg-perf run [-h] [--duration DURATION] [--num-buffers NUM_BUFFERS]

options:
  -h, --help            show this help message and exit
  --duration DURATION   individual test duration in seconds (default = 2.0)
  --num-buffers NUM_BUFFERS
                        shared memory buffers (default = 32)

❯ ezmsg-perf summary -h 
usage: ezmsg-perf summary [-h] [--baseline BASELINE] [--html] perf

positional arguments:
  perf                  perf test

options:
  -h, --help            show this help message and exit
  --baseline BASELINE, -b BASELINE
                        baseline perf test for comparison
  --html                generate an html output file and render results in browser

Instructions

1. Start on dev branch and run a baseline perf test with ezmsg-perf run. This will write a perf file (ex. perf_20250920_100047.txt)
2. Generate a report from performance test results with ezmsg-perf summary perf_20250920_100047.txt --html
3. Make a new branch off of dev and implement your new backend change.
4. Run a new perf test with ezmsg-perf run. This will generate a new perf file (ex. perf_20250920_1392911.txt
5. Run a comparison report of performance to the dev branch with ezmsg-perf summary perf_20250920_1392911.txt -b perf_20250920_100047.txt --html
6. Check out perf test results and look for enhanced performance and regressions

[griffinmilsap]

This PR has been consumed into #198 and will be closed.