feat: interactive compare session

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## Unreleased
+
+### Interactive comparison session
+
+`betterbench compare` with no arguments scans every run under `$BETTERBENCH_HOME/runs/` and opens an interactive comparison session — a gallery of all saved runs, each with its timestamp, note chips, and phase; pick two and their pair page opens with a comparison band (paired decode CIs, latency/prefill/concurrency median deltas, combined decode). The session is a loopback-only temporary server — 127.0.0.1, a kernel-picked free port, **Ctrl-C to stop** — and it writes no files of its own; it only reads run directories and serves them.
+
+### The compare band's stat honesty
+
+The band's *decode by category* rows are **paired CIs at 95%** — the per-pass `decode_tps` series paired by pass index, truncated to the shorter side. Latency, prefill, and concurrency deltas are **medians-only**: pass-level series from two uninterleaved runs have no shared trial identity, so BetterBench won't manufacture an interval. A banner on every pair page is always on: cross-file compare is unpaired in time, drift is indistinguishable from the change under test, and the verdict path is `betterbench ab`. Mismatch chips (corpus version, sampling, host, GPU, differing `--note` values) flag *which* deltas not to trust; a phase missing on one side renders "not measured", never zero.
+
 ## 0.6.0
 
 **Upgrading:** prefill throughput may read *lower* than it did on 0.5.0, and

README.md

@@ -100,6 +100,41 @@ betterbench report ~/.betterbench/runs/20260211-091234-qwen3-8/results.json --ht
 base directory. (`betterbench report`/`compare` read whatever path you point
 them at and write nothing unless told to.)
 
+## Compare your runs
+
+Two forms of `betterbench compare`, one keyword:
+
+```bash
+# two saved results side by side — a per-category decode table in the terminal
+betterbench compare results.json another.json
+
+# no args — an interactive session: it scans $BETTERBENCH_HOME/runs,
+# opens a browser at a gallery of every saved run, and stops on Ctrl-C
+betterbench compare
+```
+
+The no-args form serves every run directory under `$BETTERBENCH_HOME/runs/`
+(`~/.betterbench/runs/` when the variable is unset) from a loopback-only
+server that stops on Ctrl-C; the URL it prints is the entry point and the
+server writes no files. Pick a run to open its report; pick two to open the
+pair page.
+
+The pair page's comparison band shows the paired decode statistics — the
+per-pass `decode_tps` series paired by pass index, truncated to the shorter
+side, with a paired-t 95% CI and a SIG/noise verdict per category — plus the combined (weighted, each side's own
+weights) decode, and median deltas for TTFT/ITL
+(stream updates when batched), prefill per depth, and the concurrency
+medians per level.
+
+The orange banner is always on, for a reason: cross-file comparisons are
+**unpaired in time**, so drift between the two runs — thermal, cache-
+warmth, minutes or days apart — is indistinguishable from the change under
+test. For a verdict: run `betterbench ab`. The mismatch chips next to the
+banner — corpus version, sampling, host, GPU — flag which of the deltas
+not to trust (a corpus-version mismatch means the prompts differed, so
+every Δ in that page is apples-to-oranges); a phase missing on one side
+shows as "not measured", never as zero.
+
 ## Authentication
 
 A request goes out unauthorised by default — right for a local vLLM or

betterbench/cli.py

@@ -1,4 +1,4 @@
-"""BetterBench command line: run · report · compare · ab."""
+"""BetterBench command line: run · report · compare (files | no args = interactive session) · ab."""
 from __future__ import annotations
 
 import argparse
@@ -17,7 +17,7 @@
 from .html_report import render_html
 from .report import render_ab_markdown, render_markdown, sample_gate
 from .runner import concurrency_sweep, paired_ab, prefill_sweep, single_stream
-from .runs import allocate_run_dir
+from .runs import allocate_run_dir, betterbench_home, RUNS_SUBDIR
 from . import update
 
 
@@ -286,8 +286,14 @@ def cmd_compare(args):
     """Offline paired compare of two results.json (per-category decode-tps).
     Note: only valid if both were collected on the same warm box / interleaved —
     for a rigorous comparison use `ab`."""
-    A = json.loads(Path(args.a).read_text())
-    B = json.loads(Path(args.b).read_text())
+    if len(args.results) == 0:
+        from . import session
+        session.start(betterbench_home() / RUNS_SUBDIR)
+        return
+    if len(args.results) != 2:
+        sys.exit(f"expected 2 results files — or none, for the interactive session — got {len(args.results)}")
+    A = json.loads(Path(args.results[0]).read_text())
+    B = json.loads(Path(args.results[1]).read_text())
     print("# BetterBench compare (offline, per-category decode t/s)\n")
     print("| category | A med | B med | Δ% | 95% CI | verdict |")
     print("|---|--:|--:|--:|---|---|")
@@ -404,7 +410,9 @@ def main(argv=None):
     ab.set_defaults(func=cmd_ab)
 
     cmp = sub.add_parser("compare", help="offline compare two results.json", parents=[common])
-    cmp.add_argument("a"); cmp.add_argument("b")
+    cmp.add_argument("results", nargs="*", metavar="RESULT_JSON",
+                    help="two results.json to compare in the terminal; with "
+                         "none, opens the interactive compare session in a browser")
     cmp.set_defaults(func=cmd_compare)
 
     args = p.parse_args(argv)