Skip to content

Add an eval that scores real reviews against known problems - #51

Merged
TorstenDittmann merged 3 commits into
mainfrom
claude/project-thread-c9ll05
Nov 18, 2026
Merged

TorstenDittmann merged 3 commits into
mainfrom
claude/project-thread-c9ll05

Forward the eval's token only over https

ddc7d4a
Select commit
Loading
Failed to load commit list.
Appwrite / Hansi succeeded Nov 18, 2026 in 25s

Tier S: Ready to merge

The previously reported credential-forwarding issue is fixed by requiring HTTPS, and the new regression test covers the HTTP case.

Adds apps/eval, a CLI that checks out labeled real, clean, and planted pull-request cases, runs the service review pipeline, and reports finding, verdict, cost, and usage metrics. It includes seed cases, optional model judging, scoring and report generation, and tests for case loading and review runs. The service and engine export model, pricing, and sandbox helpers so evaluations use the same components as production.