feat: interactive compare session - #2
Open
danielcherubini wants to merge 12 commits into
Open
danielcherubini wants to merge 12 commits into
danielcherubini wants to merge 12 commits into
Conversation
…output identical)
…rds, no horizontal scroll
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
betterbench comparewith no arguments now opens a temporary, loopback-only (127.0.0.1, kernel-picked port) GET-only browser session over$BETTERBENCH_HOME/runs: gallery of all runs (skipped/corrupt/ab-only dirs stay visible as muted rows),/run/<slug>report pages (the on-diskrender_htmloutput + an injected session bar), and a pair page with an A/B toolbar, an always-on "unpaired in time" banner, mismatch chips (corpus/sampling/host/GPU/notes), and a comparison band with paired 95% decode CIs (pass-index pairing, truncated) + median-only latency/prefill/concurrency deltas + combined decode (each run's own weights). No files written, per-request re-scan (a run landing mid-session appears on the next click), three distinct plain-text 404 forms, slug allowlist = per-request directory listing (no path probing/traversal).betterbench compare a bterminal form keeps its existing behaviour verbatim (byte-pinned).betterbench/html_report.pysplit intorender_sections+render_documentso the session can reuse the exact report machinery — on-diskresults.htmloutput byte-identical (golden-pinned).## Compare your runs, METHODOLOGY## Cross-run compare(CIs vs medians-only and why), CHANGELOG## Unreleased.docs/decisions/0001-compare-session-ephemeral-server.md; plan:docs/roadmap/compare-session.md.%-named regression coverage,/run404 fallback instead of connection reset, test protocol, doc nits). 158 tests green; ruff at the 17-item branch baseline (baseline 18 minus the E702 on the subparser semicolon line this PR's spec removes).Test plan
python -m pytest tests -q→ 158 passed~/.local/bin/ruff check betterbench/ tests/→ the 17-item branch baseline, no driftBETTERBENCH_HOME=<dir-with-a-run> betterbench compare→ three lines printed, gallery in browser, pick two runs → pair page with banner/chips/band/iframes; Ctrl-C stopsbetterbench compare a.json b.json→ table byte-identical to beforeresults.json-corrupt run dir renders as a muted gallery row and 404s with the error text on/run/<slug>; a run dir created while the session is open appears on the next gallery reload