Benchmark runner

[kwiscion]

Note

Medium Risk
New public POST endpoints (runner submit, comments) accept user input and write to blob/local storage—abuse and data-integrity depend on rate limits and validation; runner cooldown is per-instance only.

Overview
Next.js site and benchmark runner. The static lab site moves into app/ with shared layout, live fetches from public/results/, and new surfaces: leaderboard (incl. PolNative), bench catalog + PR-based benchmark submissions, engineering log, experiments log, and community pages. POST /api/runner/submit queues HF model evals (open-pl-v1 on qwen3.5-9b) with validation, rate limits, and honeypot; lib/store.js persists runs/suites/submissions to Vercel Blob in prod or public/results locally (.env.example documents both blob vars).

Community APIs. /api/sota-comments and /api/v4-comments add moderated comment threads with the same blob-or-local .data/ fallback pattern.

Data / training policy (docs). Bielik is removed as a distillation teacher after open-judge fact errors; v3 mix lineage and plans are updated. V4 adds DATASET_MANIFEST.md, extended audits (V3_DATA_AUDIT.md), and README notes (INCLUDE-44 loader, pip install -e .). Competitor notes in BIELIK_RECIPE_NOTES.md and deploy hygiene (.gitignore, .vercelignore, Python 3.13 pin).

^{Reviewed by Cursor Bugbot for commit 3c13f7a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Run page ignores guard math

High Severity

The run report’s EN regression strip uses each guard’s stored status, while the benchmark runner board derives pass/fail via guardStatus from scores vs the base run. When JSON status is stale or wrong, the drill-down can show “EN regresja: OK” while the board shows a failed guard for the same run.

 

^{Reviewed by Cursor Bugbot for commit 7084bcc. Configure here.}

[cursor]

Missing base shows guard OK

Medium Severity

When a run lists EN guards and a base id but that base run file is missing, guardStatus returns "ok" instead of "na" or "fail". The board can show a green guard light even though no regression check against the anchor was possible.

 

^{Reviewed by Cursor Bugbot for commit 7084bcc. Configure here.}

[cursor]

BenchCount defaults to ten

Low Severity

BenchCount renders 10 while leaderboard JSON is still loading. If the live file has a different benchmark count, the homepage briefly shows a wrong number instead of a loading state like MeasureNote in the same file.

 

^{Reviewed by Cursor Bugbot for commit 7084bcc. Configure here.}

[cursor]

Blob read skips local fallback

Medium Severity

With BLOB_READ_WRITE_TOKEN set, readBlob returns an empty array when no blob exists or fetch fails. readComments treats that as success and never falls back to the local .data file, so GET can return zero comments even when local storage has data.

Additional Locations (1)

• app/api/v4-comments/route.js#L61-L65

 

^{Reviewed by Cursor Bugbot for commit 7084bcc. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9cedb17. Configure here.}

[cursor]

Concurrent POSTs lose comments

Medium Severity

Comment creation reads the full list, prepends one entry, and writes the JSON back with no locking or merge. Two overlapping POST handlers can both read the same snapshot; whichever put/writeFile runs last wins, so the other new comment never appears in the stored file.

Additional Locations (1)

• app/api/v4-comments/route.js#L98-L101

 

^{Reviewed by Cursor Bugbot for commit 9cedb17. Configure here.}