Benchmark Runner — UI prototype (draft, for feedback) - #70
Draft
kwiscion wants to merge 30 commits into
Draft
Conversation
…der) Run-as-object data layer for the open-suite report. Seeds three runs (Qwen3.5-9B base + balanced_v1 DPO candidate with real image.png numbers, Bielik pending) against Open PL Suite v1. lib/runs.js derives Δ vs base and guard status at build time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Board (/runner): flat ranked list of runs anchored to Qwen3.5-9B, two PL aggregates + Δ, guard light, dummy submit placeholder. Run page (/runner/[id]): provenance header, headline aggregates + guard strip, per-task table with Δ vs base, broken-task callout — reproduces the image.png report. Static-prerendered from public/results/runs/*.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Seed Qwen3.5-9B base and Bielik-11B as demo runs so the board exercises the full comparison UI (ranking, Δ vs base, EN guard light — Bielik trips red on English regression). Marked demo:true with a visible placeholder caveat on board + report; candidate numbers stay real (from image.png). Adds BENCHMARK_RUNNER.md (idea, user stories, flow + architecture diagrams). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@kwiscion is attempting to deploy a commit to the kwikiel's projects Team on Vercel. A member of the Team first needs to authorize it. |
…sion) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses all 16 findings: allowOverwrite, deterministic-URL reads (no list-per-read), dynamicParams + generateMetadata, suite publishing, submission/v1 in a queue prefix with cooldown/size-cap/dedup, env-restore in tests, pure path helpers, real-seed validation test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…body cap, id collisions) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rwrite) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vendor Open PL Leaderboard tasks (B1); per-task protocol map, null!=broken (B2); flock single-worker + re-read claim (B3); submit-route base/suite validation + immutable approved prefix (B4); drop eval-time decon, optional decon_ref provenance only (M1/M2); realistic 14B cap + vLLM params + OOM-as- failed (M3); subprocess env/exit handling (M4); real captured lm-eval fixture before the transform (M5); version-gated re-baseline vs in-place overwrite (M6). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…coupled tasks) Second-pass review fixes: prefix-based submission lifecycle + queue-approve CLI + delBlob/moveSubmission close the approval TOCTOU (B1/B2); static base/suite allowlist on the public route, no live Blob list (B3); limited-sample guard tripwires (B4); GPU-free tasks reordered first (B5); documented submission/v1 lifecycle (B6); atomic batched re-baseline (B7). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…runbook Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…le, queue CLIs Add fs-fallback write path to putJson (mkdir -p + writeFile under FS_ROOT when !usingBlob()), delBlob helper, and prefix-based submission lifecycle helpers (listSubmissions, getSubmission, moveSubmission with put-before-delete safety). Add stagePath to blob-paths.js for the 5 lifecycle stages. Create 4 queue CLI scripts (queue-list, queue-approve, queue-claim, queue-resolve) that call the store helpers directly and work in both blob and local fs modes. Add --local flag to publish-run.mjs to skip the blob token guard. Tests: 29/29 pass (17 Phase 2 + 12 new covering fs writes, delBlob, full stage flow, publish-run --local, stagePath). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ovisional YAML Add bench/runner/suite_cfg.py with load_suite_cfg() and validate_suite_cfg(), plus the provisional suite_open_pl_v1.yaml (task names pending GPU dry-run). 24 pytest cases cover happy paths, all validation rules, and the shipped YAML. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add bench/runner/lm_eval_to_run.py with to_run_v1() that maps lm-eval results + suite config + metadata into a run/v1 dict. Implements the 4-state per-task logic (ok/broken with partial-unresolved handling), robust metric resolution for both simple and multi-segment filter keys, aggregate computation (excludes broken/null), and limited-sample guards. Tests: 21 cases covering all 5 mandated scenarios (ok+both, ok+gen-only, ok+mcq-only, fully-broken, partial), aggregates, guards with flexible-extract metric key form, mmlu group aggregate, demo flag, broken guard edge case, and validateRun schema compliance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add bench/runner/run_one.py (run_one, _invoke_lm_eval, CLI) with RunnerOOM/RunnerBadRun/RunnerConfigError exceptions and publish exit-code mapping. 14 unit tests monkeypatch the GPU eval and Node publish subprocess; all 59 runner tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… allowlists Validate base/suite against ALLOWED_BASE and ALLOWED_SUITE; reject 400 on mismatch. No live Blob list on the public path (B3). Defaults preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tick() polls approved queue, claims one item, runs eval via run_one, resolves done/failed. RunnerConfigError intentionally skips resolve to preserve the queue item. All subprocess/GPU seams monkeypatched in tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. Worker orphan bug: catch base RunnerError (not just OOM/BadRun) so a generic lm-eval failure resolves the item as "failed" instead of leaving it stuck in running/. RunnerConfigError still skips resolve. 2. vLLM tuning: _invoke_lm_eval now folds suite_cfg["vllm"] dict into --model_args when backend=="vllm" (e.g. gpu_memory_utilization, max_model_len). Non-vllm backends unchanged. 3. Injection seam: replace f-string JS interpolation in _get_submission with argv-based scripts/queue-get.mjs CLI (mirrors queue-list/claim/ resolve pattern). Eliminates shell-injection vector. 4. Wire validator: run_one() calls validate_suite_cfg() after load; raises RunnerConfigError on structural errors instead of letting a raw KeyError propagate. 6 new tests (74 total runner tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark Runner — UI prototype (draft, for feedback)
A new section that turns "wrzuć model → dostań wyniki" into a real page: a bake-off / candidate report over one pinned, versioned open benchmark suite. Reproduces the
image.pngreport format (provenance → headline aggregates + EN guard strip → per-task table → broken-tasks callout) and adds a ranked board.This is a draft for UI/UX feedback — not the final data or backend.
What to look at (Vercel preview)
/runner— the board: models ranked by generative aggregate, Δ vs base (Qwen3.5-9B), EN regression-guard light, link into each report./runner/<id>— the per-run report, e.g.qwen3.5-9b-balanced-v1-dpo(real numbers from our Polish-eval artifact).Design decisions baked in
open-pl-v1) — numbers comparable only within a suite version.exact_match+ MCQaccshown side by side.Data status
qwen3.5-9b-balanced-v1-dpo= real measured numbers.qwen3.5-9b(base) andbielik-11b-v3= placeholder (demo: true, marked in the UI) until a real GPU run. Bielik intentionally trips the EN guard red to exercise that state.Architecture
Static
run/v1JSON inpublic/results/runs/→lib/runs.jsderives Δ + guard status at build → server-component pages, prerendered. No backend yet.Next phases (Vercel Blob results store, submission backend, GPU runner) are planned separately.
Docs:
docs/benchmark_runner/(overview + design spec).🤖 Generated with Claude Code