Feat/bench core abstraction - #7

Open

Abhinavexist wants to merge 38 commits into

mainfrom

feat/bench-core-abstraction

Commits on Aug 16, 2026

feat: multi-provider benchmark runners, harness fixes, and tooling
Abhinavexist
committed
b4cb41f
feat(bench_core): unified provider/model abstraction with tests
Abhinavexist
committed
b0a9009
refactor(bench_core): consolidate per-model target YAMLs into one targets.yaml
Abhinavexist
committed
e3fdbcb
chore(bench_core): add anthropic dependency; remove dead line in openai_compat
Abhinavexist
committed
2c12cc7
docs(bench_core): trim module docstrings to one line
Abhinavexist
committed
8f563e0
chore: remove one-off debug probe scripts
Abhinavexist
committed
4b710e3
feat(bench_core): migrate GPQA onto the abstraction; retire 4 per-provider forks
Abhinavexist
committed
4ba6054
feat(bench_core): migrate ASR (VoxPopuli) onto the abstraction; retire 2 forks
Abhinavexist
committed
70dcdcd
feat(bench_core): migrate MMMLU onto the abstraction; retire 2 forks
Abhinavexist
committed
2742539
feat(bench_core): migrate MMMU-Pro onto the abstraction; retire fork + modal app
Abhinavexist
committed
7471182
feat(bench_core): migrate RefCOCO onto the abstraction; retire 5 forks + oracle re-scorer
Abhinavexist
committed
9d03e15
feat(bench_core): migrate OCRBench v2 onto the abstraction; retire 10 forks
Abhinavexist
committed
f0c4be9
feat(bench_core): migrate olmOCR onto the abstraction; retire 6 drivers + 6 runners
Abhinavexist
committed
650ee04
feat(bench_core): migrate Spider2-Lite onto the abstraction; retire the fork
Abhinavexist
committed
b4657d3
feat(bench_core): backfill legacy results + point report at the new contract (Steps 2+5)
Abhinavexist
committed
8ee79a5
ci(bench_core): GitHub Actions — offline test gate + benchmark runner (Stage 6)
Abhinavexist
committed
b32683f
ci: interactive add-target workflow + run benchmarks on merge
Abhinavexist
committed
e5af384
ci: no default model — nothing runs unless explicitly selected/flagged
Abhinavexist
committed
743372c
ci: audit — eliminate accidental and unnecessary runs
Abhinavexist
committed
b5f2a12
fix(benchmarks): restore full MMMLU + RefCOCO+/g variants (don't lite-ify)
Abhinavexist
committed
7c5046c
chore: remove JigsawStack object-detection benchmark (unused)
Abhinavexist
committed
f6ccd76
docs: reflow README setup section
Abhinavexist
committed
2edfd5c
chore: stop tracking results/ — run artifacts, never in git
Abhinavexist
committed
4c15a35
docs: trim README to a scannable overview
Abhinavexist
committed
e416ef1
chore: remove BFCL from the repo
Abhinavexist
committed
c2e22e3
fix(cli): load .env so provider keys are picked up on live runs
Abhinavexist
committed
821a99e
docs: reflow README
Abhinavexist
committed
ed2c47a

Commits on Aug 17, 2026