Skip to content

Add an eval that scores real reviews against known problems - #51

Merged
TorstenDittmann merged 3 commits into
mainfrom
claude/project-thread-c9ll05
Nov 18, 2026
Merged

TorstenDittmann merged 3 commits into
mainfrom
claude/project-thread-c9ll05

Conversation

@TorstenDittmann

Copy link
Copy Markdown
Contributor

Requested by Torsten · project thread

Until now we changed models and prompts (Haiku/Sonnet, strengths) with nothing to measure them against. This adds apps/eval: it runs pull requests with known problems through runReview, using the service's own models and strengths, and scores what the review posts. Nothing goes to GitHub or Appwrite.

HANSI_REVIEW_MODEL=bedrock:… HANSI_AI_SECRET=… HANSI_AI_REGION=eu-central-1 \
  bun run eval --strength normal --runs 3 [--judge] [--compare results/<earlier>.json]

What it reports

  • Recall: expected problems that a posted finding reported.
  • Precision against the labels, and false alarms on clean PRs.
  • Verdict accuracy.
  • Real problems dropped: problems the reviewer did report, but the verifier or a filter then threw away.
  • Cost, time and sandbox commands per review. Cost uses models.dev prices, as the service's metrics do.

Results are shown per case, as Markdown on stdout, optionally next to an earlier run. The full JSON report goes to apps/eval/results/ and is gitignored.

By default a finding counts when it is on the same file, at the expected severity or above, and within 5 lines of the problem. With --judge, a model instead picks which finding on the file reports the problem. That stops a different problem on the same line from counting, and lets a finding on the caller count.

Cases

Each case is a cases/<id>/case.json with the repo, base, head, PR text, optional .hansi.json settings, and the expected findings and verdict. An optional inject.patch is applied on top of the head as one more commit, which plants a known bug in a real PR without pushing anything. It is seeded with six cases from this repo:

Case Kind
sandbox-upload-drops-tracked-archive Real bug from fb22b90, fixed in 6b4a72d
background-sandbox-close-waits Real bug from ce18b20, fixed in 40a6a63
sandbox-unpack-clean, background-sandbox-close-clean The two fixes, expected to approve
background-sandbox-close-unhandled-rejection 40a6a63 with its .catch removed
sandbox-execute-timeout-and-url-leak 6b4a72d with a seconds-for-milliseconds timeout and the capability URL in an error

Six cases is a seed, not a benchmark. The README explains how to add real-bug, clean and planted cases.

Other changes

  • createStrengths and createReviewModel in the service now take only the model variables (ModelEnv), and the service exports ./models and ./pricing. This lets the eval build models exactly the way the service does.
  • The engine now exports openSandbox, so an eval run with HANSI_ORCHESTRATOR_ENDPOINT set runs commands as production does.
  • apps/eval is added to .dockerignore. I checked that the image's frozen install still passes without it.

Testing

  • bun run test (467 pass), bun run check and bun run lint pass.
  • The new unit tests cover matching, scoring, the report and case loading. An end-to-end test does a local partial clone, applies a patch, runs a scripted review and scores it.
  • All six seed cases were checked out from GitHub, their patches applied, and every expected line confirmed to be in the diff.
  • I did not run it against a real model: that needs a provider key, which this environment does not have.

Generated by Claude Code

Model and prompt changes had nothing to be measured against. apps/eval
replays pull requests with known problems through runReview with the
service's own models and strengths, matches the posted findings to the
expected ones (by file and line, or with --judge by a model), and reports
recall, precision against the labels, false alarms on clean pull requests,
verdict accuracy, real problems the verifier dropped, and cost and time per
review, optionally next to an earlier run.

Cases are a pull request's base and head plus what a good review reports.
An inject.patch beside a case plants a known bug on top of a real head
without pushing anything. Six cases from this repository seed it: two real
bugs that later commits fixed, the two fixes as clean cases, and two
planted variants of them.

The service's model builders now take only the model variables, so the
eval builds models exactly as the service does, and the engine exports
openSandbox for reviews that run commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG
Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>
@TorstenDittmann TorstenDittmann self-assigned this Nov 18, 2026
@appwrite

appwrite Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

The previously reported credential-forwarding issue is fixed by requiring HTTPS, and the new regression test covers the HTTP case.

Adds apps/eval, a CLI that checks out labeled real, clean, and planted pull-request cases, runs the service review pipeline, and reports finding, verdict, cost, and usage metrics. It includes seed cases, optional model judging, scoring and report generation, and tests for case loading and review runs. The service and engine export model, pricing, and sandbox helpers so evaluations use the same components as production.

Latest changes: The latest commits restrict GitHub-token use to HTTPS URLs on github.com and add a regression test for plain HTTP URLs.

Verdict New comments Fixed Still open
✅ Approved 0 1 0
📂 Walkthrough · 29
File Change
.dockerignore Excludes the eval workspace from the service image context.
AGENTS.md Documents the eval workspace and command.
README.md Adds the eval command and links to its usage guide.
apps/eval/.gitignore Ignores generated evaluation reports.
apps/eval/README.md Documents running evaluations and adding labeled cases.
apps/eval/cases/** Adds seed real-bug, clean, and planted evaluation cases with expected findings and verdicts.
apps/eval/package.json, apps/eval/tsconfig.json Adds workspace scripts, dependencies, and TypeScript configuration.
apps/eval/src/case.ts Validates and loads cases, derives clone URLs, and restricts GitHub credential handling to HTTPS GitHub URLs.
apps/eval/src/case.test.ts Tests case loading, validation, clone URLs, and HTTPS-only GitHub detection.
apps/eval/src/env.ts, apps/eval/src/env.test.ts Loads and tests evaluation model and sandbox settings.
apps/eval/src/index.ts Adds the evaluation CLI, case runner, concurrency controls, and report output.
apps/eval/src/judge.ts Adds optional model-based matching of findings to expected problems.
apps/eval/src/report.ts, apps/eval/src/report.test.ts Aggregates evaluation metrics and renders Markdown reports and comparisons.
apps/eval/src/run.ts, apps/eval/src/run.test.ts Checks out cases, applies injected patches, runs reviews, and scores results.
apps/eval/src/score.ts, apps/eval/src/score.test.ts Matches findings to labels and computes tested finding and verdict scores.
apps/service/package.json Exports the service model and pricing helpers for the eval workspace.
apps/service/src/models.ts Types model factories against model-only environment inputs.
bun.lock Locks the eval workspace dependencies.
package.json Registers the eval workspace and root command.
packages/engine/src/index.ts Exports sandbox creation for production-like evaluation runs.
apps/eval/package.json Defines the eval workspace scripts and dependencies.
apps/eval/tsconfig.json Adds TypeScript configuration for the eval workspace.
apps/eval/src/env.ts Loads and validates model and sandbox settings for eval runs.
apps/eval/src/report.ts Aggregates eval metrics and renders Markdown reports.
apps/eval/src/report.test.ts Tests report aggregation and Markdown rendering.
apps/eval/src/run.ts Checks out cases, runs reviews, and scores their results.
apps/eval/src/run.test.ts Tests local checkout, patch injection, and scripted review scoring.
apps/eval/src/score.ts Matches reported findings to expected problems and computes scores.
apps/eval/src/score.test.ts Tests line matching, judging, dropped findings, and verdict scoring.
✅ Fixed since the last review · 1
  • Reject plain-HTTP GitHub URLs before forwarding the token · apps/eval/src/case.ts:80

Reviewed the commits since 3cb80bc · Details · Comment @appwrite review to re-run, or mention @appwrite with a question.

@appwrite appwrite Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Tier D · 1 blocking finding to address. Summary

Comment thread apps/eval/src/run.ts Resolved
Comment thread apps/eval/src/score.ts Resolved
Comment thread apps/eval/src/env.ts Resolved
…e provider

- GITHUB_TOKEN only goes to github.com clone URLs: git sends it as a header
  to whatever host a case names.
- Line matching is a maximum matching, so a finding between two problems no
  longer costs the other one its match, and results do not depend on label
  order.
- Like the service, every model must use the review model's provider, since
  they share one HANSI_AI_SECRET.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG
Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>

@appwrite appwrite Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Tier B · 1 blocking finding to address. Summary

Comment thread apps/eval/src/case.ts Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3M7trTDakSwaifPsQZBEG
Co-authored-by: Torsten Dittmann <1759475+TorstenDittmann@users.noreply.github.com>

@appwrite appwrite Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

@TorstenDittmann
TorstenDittmann merged commit cdd40e8 into main Nov 18, 2026
4 checks passed
@TorstenDittmann
TorstenDittmann deleted the claude/project-thread-c9ll05 branch November 18, 2026 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants