Repository navigation
Add an eval that scores real reviews against known problems - #51
Conversation
Model and prompt changes had nothing to be measured against. apps/eval replays pull requests with known problems through runReview with the service's own models and strengths, matches the posted findings to the expected ones (by file and line, or with --judge by a model), and reports recall, precision against the labels, false alarms on clean pull requests, verdict accuracy, real problems the verifier dropped, and cost and time per review, optionally next to an earlier run. Cases are a pull request's base and head plus what a good review reports. An inject.patch beside a case plants a known bug on top of a real head without pushing anything. Six cases from this repository seed it: two real bugs that later commits fixed, the two fixes as clean cases, and two planted variants of them. The service's model builders now take only the model variables, so the eval builds models exactly as the service does, and the engine exports openSandbox for reviews that run commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>
🟢 Tier S · Ready to merge
Adds Latest changes: The latest commits restrict GitHub-token use to HTTPS URLs on
📂 Walkthrough · 29
✅ Fixed since the last review · 1
Reviewed the commits since |
…e provider - GITHUB_TOKEN only goes to github.com clone URLs: git sends it as a header to whatever host a case names. - Line matching is a maximum matching, so a finding between two problems no longer costs the other one its match, and results do not depend on label order. - Like the service, every model must use the review model's provider, since they share one HANSI_AI_SECRET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>
Requested by Torsten · project thread
Until now we changed models and prompts (Haiku/Sonnet, strengths) with nothing to measure them against. This adds
apps/eval: it runs pull requests with known problems throughrunReview, using the service's own models and strengths, and scores what the review posts. Nothing goes to GitHub or Appwrite.What it reports
Results are shown per case, as Markdown on stdout, optionally next to an earlier run. The full JSON report goes to
apps/eval/results/and is gitignored.By default a finding counts when it is on the same file, at the expected severity or above, and within 5 lines of the problem. With
--judge, a model instead picks which finding on the file reports the problem. That stops a different problem on the same line from counting, and lets a finding on the caller count.Cases
Each case is a
cases/<id>/case.jsonwith the repo, base, head, PR text, optional.hansi.jsonsettings, and the expected findings and verdict. An optionalinject.patchis applied on top of the head as one more commit, which plants a known bug in a real PR without pushing anything. It is seeded with six cases from this repo:sandbox-upload-drops-tracked-archivebackground-sandbox-close-waitssandbox-unpack-clean,background-sandbox-close-cleanbackground-sandbox-close-unhandled-rejection.catchremovedsandbox-execute-timeout-and-url-leakSix cases is a seed, not a benchmark. The README explains how to add real-bug, clean and planted cases.
Other changes
createStrengthsandcreateReviewModelin the service now take only the model variables (ModelEnv), and the service exports./modelsand./pricing. This lets the eval build models exactly the way the service does.openSandbox, so an eval run withHANSI_ORCHESTRATOR_ENDPOINTset runs commands as production does.apps/evalis added to.dockerignore. I checked that the image's frozen install still passes without it.Testing
bun run test(467 pass),bun run checkandbun run lintpass.Generated by Claude Code