[FR] Report cross-thread statistics for multithreaded runs

[shuralnik-sc]

Is your feature request related to a problem? Please describe.
When running multithreaded benchmarks - in certain cases there is a need to understand a "fairness" of the measured code, i.e. if work is equally distributed among all threads and/or if all threads were able to acquire some resource in an equal (or intentionally unequal) manner.
Currently counters for all threads are summed and normalized which in fact leads to a loss of information about how work of each thread compares to its peers.

Describe the solution you'd like
Is it possible to output the same set of statistics for multithreaded runs on a cross-thread basis as it exists today for repeated runs? I.e. to see median and stddev between N threads of the same multithreaded run?

Describe alternatives you've considered
Tried to leverage existing API, like custom counters but it appears that it also undergoes the same averaging for all threads so basically info about individual thread results is lost already. So there is no alternative as I understand it.

[dmah42]

you're right that this isn't offered. the context here is that multi-threading something to benchmark it is anticipated to be to see what the impact of the threading (and thread counts) has on the overall performance (and throughput, etc). separating out the thread reports would run counter to that expectation.

that being said, i understand the desire and i don't think the library currently supports it, so if someone wants to come up with a design for how we can offer this without complicating the reporting or main loop too much, i'm open to hearing about it.

[jlecavelier]

Just for the record I raised that issue in a previous ticket :
#1849
Wouldn't you agree that it is reasonable for someone who registers a benchmark to run in a multithreaded fashion, explicitly overriding the memory manager, to expect gathering of memory statistics happening as they set it up, in a multi threaded fashion, and not in serial?

I will do some digging and check out how I got around that issue.
I do think that this is a reasonable issue to tackle

[Frank20021]

Design: cross-thread statistics for multithreaded runs

I am interested in implementing this. Below is a concrete design that fits the current runner/reporter architecture. I used automated assistance to locate the relevant code paths; I have read those paths myself and this design is mine to defend.

Problem

In a multithreaded run, each State has its own counters and ThreadTimer. At the end of RunInThread (src/benchmark_runner.cc), those values are summed into a single ThreadManager::Result via Increment, then the per-thread State and timer are destroyed. CreateRunReport then calls Finish on the already summed counters.

After that point the distribution across threads is gone. Existing ComputeStats only sees one aggregated Run per repetition, so it cannot recover median/stddev between threads.

Goal

Opt-in statistics across the N threads of one reported multithreaded run, analogous to today’s mean/median/stddev/cv across repetitions. Default output and the summed counter row stay unchanged.

Non-goals (v1)

• No per-thread rows (threads:8/thread:3). That would change the main report shape a lot.
• No change to how the primary iteration row is computed (still sum, then Finish).
• No cross-thread stats during warmup, iteration search, memory-manager runs, or profiler-manager runs.
• No new reporter type and no required JSON schema change beyond extra run_type: "aggregate" rows (same as repetitions).

Capture point

The only place samples still exist is the mutex block at the end of RunInThread. When the feature is enabled and threads > 1, copy that thread’s result before or beside the existing Increment:

• iterations
• cpu/real/manual time
• UserCounters (raw values, before Finish)

Store them on ThreadManager as std::vector<Result> (size N), guarded by the same mutex. Default path allocates nothing extra.

Do not put these samples into RunResults.non_aggregates before ComputeStats. ComputeStats assumes every successful run is one repetition of the same iteration Run. Mixing thread-stat rows in would break that (and benchmark_name() for aggregates appends _aggregate_name).

How to compute the statistics

For each reported repetition (the run that CreateRunReport already builds):

1. For each thread, Finish that thread’s counters with:
   • iterations = that thread’s iteration count
   • time = that thread’s seconds (cpu, real, or manual — same choice as the benchmark)
   • num_threads = 1 so kAvgThreads is not applied twice. Each sample is “this thread’s contribution.”
2. For each statistic in the benchmark’s existing statistics_ list (mean, median, stddev, cv, plus any ComputeStatistics the user added), build one BenchmarkReporter::Run with run_type = RT_Aggregate and aggregate_name = "thread_" + Stat.name_.
3. Fill that row’s counters by applying Stat.compute_ to the per-thread finished values for each counter name.
4. Also fill real_accumulated_time / cpu_accumulated_time from the per-thread times (same helpers). That answers “were the threads equally busy?” for free once samples are kept.
5. iterations on these rows = thread count (the sample size), matching how repetition aggregates use the repetition count.
6. Skip emitting these rows if threads < 2 or the run was skipped.

Reuse StatisticsMean, StatisticsMedian, StatisticsStdDev, StatisticsCV as-is. Do not call ComputeStats() on the thread samples; extract a small helper that shares the “vector in, StatisticsFunc out” pattern if it keeps the runner readable.

Worked example (raw counters work = {1, 2, 3}, no rate flags):

• existing iteration row: work = 6 (sum; Finish unchanged)
• _thread_mean: 2
• _thread_median: 2
• _thread_stddev: 1
• _thread_cv: 50%

When they appear in the report

Thread stats belong to one run, not to the whole repetition group.

After ComputeStats(non_aggregates) in GetResults(), splice each repetition’s thread-stat rows immediately after that repetition’s iteration row in the vector that reporters already receive. Then the existing Report() path does not need a third list or reporter special cases.

Console/JSON/CSV already print RT_Aggregate via name + "_" + aggregate_name (Run::benchmark_name()), so names look like:

BM_Fairness/threads:8
BM_Fairness/threads:8_thread_mean
BM_Fairness/threads:8_thread_median
BM_Fairness/threads:8_thread_stddev
BM_Fairness/threads:8_thread_cv

The thread_ prefix avoids colliding with repetition _mean / _median / _stddev / _cv. JSON already has aggregate_name and aggregate_unit; cv stays percentage.

If --benchmark_repetitions > 1, each repetition is followed by its own thread-stat rows, then the usual repetition aggregates. That matches the request (“one multithreaded run”) and stays honest if imbalance differs across reps.

ReportAggregatesOnly / DisplayAggregatesOnly continue to mean repetition aggregates. Thread-stat rows are tied to a specific iteration run, so they follow that run (hidden when that run is hidden). We can revisit that later if people want thread stats kept when iteration rows are suppressed.

Public API (opt-in, default off)

Per benchmark, same style as ReportAggregatesOnly:

BENCHMARK(BM_Fairness)
    ->Threads(8)
    ->ReportThreadStatistics();

Global flag, same style as other report flags:

--benchmark_report_thread_statistics={true|false}   (default false)

The per-benchmark setter overrides the flag for that benchmark.

No new Counter flags. Fairness is a reporting concern, not a counter-scaling concern. Existing kAvgThreads / kIsRate behavior on the primary row does not change.

Memory / performance

Off: no extra allocations, same hot path plus a boolean check.

On: one extra Result copy per thread per DoNIterations call that uses the thread runner (including iteration search). Search results are discarded with ThreadManager; only the reported run is turned into output. Cost is O(threads × counters) and is paid only when requested.

Tests

• Extend test/user_counters_threads_test.cc: unequal per-thread values via state.thread_index(); assert sum row unchanged and thread median/stddev/cv.
• Cover kIsRate / kAvgThreads: thread-stat samples are finished with num_threads = 1.
• threads:1 + feature on → no extra rows.
• Feature off → current golden output unchanged.
• Custom ComputeStatistics("max", ...) → _thread_max.
• Repetitions + thread stats: no ComputeStats CHECK failures; names do not collide.
• test/statistics_gtest.cc only if a helper is extracted.

Files (expected)

• include/benchmark/benchmark_api.h — ReportThreadStatistics
• src/benchmark_register.cc — store the bool
• src/benchmark_api_internal.* — plumb it onto BenchmarkInstance
• src/commandlineflags.* / src/benchmark.cc — flag
• src/thread_manager.h — optional vector<Result> samples
• src/benchmark_runner.cc — capture, compute, splice
• src/statistics.cc / .h — optional small helper
• docs/user_guide.md — short subsection
• tests as above
• Python bindings only if the C++ API is accepted

Why this and not the alternatives

• Per-thread iteration rows: too much reporter/output change; maintainers asked not to complicate the main report.
• New counter names (work_median on the same row): pollutes the counter map and breaks tabular/JSON consumers.
• Calling ComputeStats on synthetic per-thread Runs: reuses more code but conflates repetition aggregates with thread aggregates and is easy to wire into the wrong vector.

This is intended as a design for #1902. #2182 already sketches the same opt-in extra-row idea (_thread_mean, etc.). The points above are the capture/ComputeStats isolation and Finish-per-thread rules I’d want that implementation (or a follow-up) to match.

Happy to adjust naming, whether times are included in v1, or whether this is flag-only vs API-only.

[dmah42]

the design looks reasonable to me. i'll look over the code to see if i can spot any issues. thanks so much.