Skip to content

feat: interactive compare session - #2

Open
danielcherubini wants to merge 12 commits into
GGZ14:mainfrom
danielcherubini:feature/compare-session
Open

danielcherubini wants to merge 12 commits into
GGZ14:mainfrom
danielcherubini:feature/compare-session

Conversation

@danielcherubini

Copy link
Copy Markdown
Contributor

Summary

  • betterbench compare with no arguments now opens a temporary, loopback-only (127.0.0.1, kernel-picked port) GET-only browser session over $BETTERBENCH_HOME/runs: gallery of all runs (skipped/corrupt/ab-only dirs stay visible as muted rows), /run/<slug> report pages (the on-disk render_html output + an injected session bar), and a pair page with an A/B toolbar, an always-on "unpaired in time" banner, mismatch chips (corpus/sampling/host/GPU/notes), and a comparison band with paired 95% decode CIs (pass-index pairing, truncated) + median-only latency/prefill/concurrency deltas + combined decode (each run's own weights). No files written, per-request re-scan (a run landing mid-session appears on the next click), three distinct plain-text 404 forms, slug allowlist = per-request directory listing (no path probing/traversal).
  • The two-arg betterbench compare a b terminal form keeps its existing behaviour verbatim (byte-pinned).
  • betterbench/html_report.py split into render_sections + render_document so the session can reuse the exact report machinery — on-disk results.html output byte-identical (golden-pinned).
  • Docs: README ## Compare your runs, METHODOLOGY ## Cross-run compare (CIs vs medians-only and why), CHANGELOG ## Unreleased.
  • Rationale: docs/decisions/0001-compare-session-ephemeral-server.md; plan: docs/roadmap/compare-session.md.
  • Review: 3-pass review loop — 13 actionable findings resolved (gallery degradation for malformed runs, encode-once/decode-once slug URL convention with %-named regression coverage, /run 404 fallback instead of connection reset, test protocol, doc nits). 158 tests green; ruff at the 17-item branch baseline (baseline 18 minus the E702 on the subparser semicolon line this PR's spec removes).

Test plan

  • python -m pytest tests -q → 158 passed
  • ~/.local/bin/ruff check betterbench/ tests/ → the 17-item branch baseline, no drift
  • Manual: BETTERBENCH_HOME=<dir-with-a-run> betterbench compare → three lines printed, gallery in browser, pick two runs → pair page with banner/chips/band/iframes; Ctrl-C stops
  • Manual: betterbench compare a.json b.json → table byte-identical to before
  • Manual point-checks: a results.json-corrupt run dir renders as a muted gallery row and 404s with the error text on /run/<slug>; a run dir created while the session is open appears on the next gallery reload

@danielcherubini

Copy link
Copy Markdown
Contributor Author
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant