Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@
*.md
**/vendor
apps/consumer/tests
apps/eval
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ The engine, the review and chat jobs and the GitHub helpers started as a port of
| `bun run check` | Type-check every workspace (`tsc`) |
| `bun run lint` | Prettier check + ESLint |
| `bun run format` | Apply Prettier |
| `bun run eval` | Score reviews against `apps/eval` |

Run `bun run format`, `bun run lint`, `bun run check` and `bun run test` before pushing; CI runs the last three.

Expand All @@ -26,6 +27,7 @@ The queue consumer is PHP (`apps/consumer`), on PHP 8.5 with Swoole as in `appwr

- **apps/service** -- the process: env, logger, health server, the jobs port the consumer hands deliveries to, the worker that runs them and decides retries, job handlers. It starts the consumer as its child, so the container runs and stops as one
- **apps/consumer** -- the queue consumer, in PHP on Appwrite's own utopia-php/queue: `Broker\Nats` on Cloud (adopting the queue Appwrite provisioned, with its attempts and backoff read from the work consumer), `Broker\Redis` on a self-hosted Appwrite (where it runs the attempts itself). Each delivery is one `POST /v1/jobs` to the service on loopback, settled by the outcome it answers with
- **apps/eval** -- replays pull requests with known problems (`cases/`) through a real review with the service's models and scores recall, false alarms, verdicts and cost. `bun run eval`; see its README. Run it before and after a model or prompt change
- **packages/engine** -- the engine and the contracts it works against (`GitHubClient`, `SandboxClient`, `ResultsSink`, job types). Also the review itself: the agent loop on the Vercel AI SDK, its read-only tools over a local checkout (`git`, `rg`), the prompts, and the code that places, verifies and grades findings. In-memory fakes and a scripted model in `@hansi/engine/testing`. `runReviewJob` and `runChatJob` take a job from the pull request to what is posted on GitHub: the summary comment, the review, the check run, thread resolution and replies. Their only I/O of its own is git, for the checkout; everything else goes through the contracts
- **packages/appwrite** -- Appwrite adapters: results sink and installation tokens (internal API), job payload parsing
- **packages/github** -- GitHub client on octokit: the helpers ported from hansi.codes (`pulls.ts`), authenticated with installation tokens from Appwrite, renewed before they expire
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,9 +250,10 @@ bun run check # type-check every workspace
bun run lint # Prettier and ESLint
bun run format # apply Prettier
bun run dev # run the service with reload
bun run eval # score real reviews against known problems (apps/eval)
```

Tests use in-memory fakes and a scripted model from `@hansi/engine/testing`, and never call a real model, GitHub or Appwrite.
Tests use in-memory fakes and a scripted model from `@hansi/engine/testing`, and never call a real model, GitHub or Appwrite. The eval is the opposite: it reviews pull requests with known problems using real models, and reports recall, false alarms, verdicts and cost. Run it before and after changing a model or a prompt; [apps/eval](apps/eval/README.md) says how, and how to add a case.

The consumer runs on PHP 8.5 with Swoole. Without a local PHP 8.5, run its tests in the image it ships on:

Expand Down
1 change: 1 addition & 0 deletions apps/eval/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
results/
71 changes: 71 additions & 0 deletions apps/eval/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Eval

Replays pull requests with known problems through a real review and scores what Hansi finds, so a model, prompt or strength change comes with numbers instead of impressions.

Each case is a pull request and what a good review of it reports. A run reviews every case with the service's own models and engine (no GitHub, no Appwrite: the review is never posted), matches the posted findings to the expected ones, and writes a report.

```sh
export HANSI_REVIEW_MODEL=bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0
export HANSI_AI_SECRET=… HANSI_AI_REGION=eu-central-1
bun run eval --runs 3
bun run eval --strength basic --runs 3 --compare results/<earlier>.json
```

Models come from the same variables as the service (`HANSI_REVIEW_MODEL`, `HANSI_VERIFY_MODEL`, the `HANSI_BASIC_*` and `HANSI_EXPERT_*` models, the reasoning efforts and `HANSI_AI_*`), so a run reviews with what production would at that strength. With `HANSI_ORCHESTRATOR_ENDPOINT` and `HANSI_SANDBOX_IMAGE`, normal and expert reviews run commands in a sandbox as they do in production; without them they only read the code, and the report says so. `GITHUB_TOKEN` is used to clone private repositories. `bun run eval --help` lists the options.

## What the report says

| Metric | Meaning |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Recall | Expected problems a posted finding reported, out of all expected problems. |
| Precision (vs labels) | Posted findings that reported an expected problem, out of all posted on cases with problems. Labels can be incomplete. |
| False alarms on clean PRs | Reviews of clean cases that posted any finding. |
| Verdict accuracy | Reviews that reached the case's expected verdict. |
| Real problems dropped | Expected problems a finding reported, but the verifier or a filter then dropped. |
| Cost, time, commands | Per review. Cost uses models.dev's list prices, as the service's metrics do. |

Models are not deterministic, so use `--runs 3` or more before reading anything into a difference. The JSON report in `results/` (not committed) keeps every run's findings, so an extra finding can be read to see whether it is a real problem the case does not list.

A finding counts for an expected problem when it is on the same file, at the problem's severity or above, and within `--tolerance` lines of it (5 by default). With `--judge`, a model decides instead, among every finding on the file: slower and costs a little, but it does not count a different problem on the same line, and it does count the right problem reported on a caller. `HANSI_JUDGE_MODEL` picks the judge's model, the verify model by default.

## Adding a case

A case is a directory under `cases/` with a `case.json`:

```json
{
"description": "Real bug: … Fixed later in <sha>.",
"source": "https://github.com/appwrite/hansi/commit/<sha>",
"repo": "appwrite/hansi",
"pull": 27,
"base": "<40-character sha>",
"head": "<40-character sha>",
"pullRequest": { "title": "…", "body": "…" },
"expected": {
"verdict": "request_changes",
"findings": [
{
"id": "short-name",
"path": "src/file.ts",
"lines": [10, 12],
"issue": "What is wrong and what it breaks, in a sentence or two.",
"severity": "major"
}
]
}
}
```

- `repo` is `owner/name` on GitHub or a clone URL. With `pull`, the head is fetched from the pull request's ref, so heads on forks work; without it, the head commit is fetched by its SHA.
- `lines` are lines of the file at the head, after any patch. `severity` is the least a finding needs to count; leave it out to count any.
- `config` takes the repository settings as in `.hansi.json`. Leave it out to read `.hansi.json` from the base commit, as a review does.
- `linkedIssues` takes `{ number, title, body }` entries, as a review gets them from the pull request.
- An empty `findings` list makes a clean case: a good review posts nothing and approves.

There are three kinds of cases, and the set needs all three:

- **Real bugs**: a pull request that shipped a bug a later commit fixed. Base and head are that pull request's; the expected finding is the bug the fix names.
- **Clean**: a pull request that needed nothing, such as the fix itself. These measure false alarms.
- **Planted**: a real pull request with an `inject.patch` beside `case.json`, applied on top of the head as one more commit. It plants a known bug without pushing anything anywhere. Make it with `git diff` in a checkout of the head.

Run a new case on its own first (`bun run eval --case <id>`) and read the review: a case whose expected problem no reasonable reviewer would see from the pull request measures nothing.
15 changes: 15 additions & 0 deletions apps/eval/cases/background-sandbox-close-clean/case.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"description": "Clean: the fix for the background sandbox close. A good review approves it.",
"source": "https://github.com/appwrite/hansi/commit/40a6a63bc4c320e8f83d257cdd83d2a33f6281cb",
"repo": "appwrite/hansi",
"base": "ce18b201051950955d680acd3e7dc3b767c47051",
"head": "40a6a63bc4c320e8f83d257cdd83d2a33f6281cb",
"pullRequest": {
"title": "fix(engine): don't wait for a starting sandbox when the review closes it",
"body": "A review that runs no command closed its background sandbox by awaiting\nthe create, so it still sat through the whole start. Close now returns\nat once and the sandbox is closed once it is up."
},
"expected": {
"verdict": "approve",
"findings": []
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"description": "Planted: the same fix with the catch on the deferred close removed, so a sandbox that fails to start after the review closed it rejects with no handler.",
"source": "https://github.com/appwrite/hansi/commit/40a6a63bc4c320e8f83d257cdd83d2a33f6281cb plus inject.patch",
"repo": "appwrite/hansi",
"base": "ce18b201051950955d680acd3e7dc3b767c47051",
"head": "40a6a63bc4c320e8f83d257cdd83d2a33f6281cb",
"pullRequest": {
"title": "fix(engine): don't wait for a starting sandbox when the review closes it",
"body": "A review that runs no command closed its background sandbox by awaiting\nthe create, so it still sat through the whole start. Close now returns\nat once and the sandbox is closed once it is up."
},
"expected": {
"verdict": "request_changes",
"findings": [
{
"id": "unhandled-rejection",
"path": "packages/engine/src/review-sandbox.ts",
"lines": [115, 115],
"severity": "minor",
"issue": "`void opening.then((sandbox) => sandbox.close())` has no catch. When the sandbox fails to start, or its close fails, after the review already closed it, the promise rejects with no handler: an unhandled rejection that can take down the service process with every other job in it."
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
diff --git a/packages/engine/src/review-sandbox.ts b/packages/engine/src/review-sandbox.ts
index edcc4fc..f1bf854 100644
--- a/packages/engine/src/review-sandbox.ts
+++ b/packages/engine/src/review-sandbox.ts
@@ -112,7 +112,7 @@ export function backgroundSandbox(open: () => Promise<ReviewSandbox>): ReviewSan
// once it is up.
close: async () => {
if (opened) return opened.close();
- void opening.then((sandbox) => sandbox.close()).catch(() => undefined);
+ void opening.then((sandbox) => sandbox.close());
}
};
}
21 changes: 21 additions & 0 deletions apps/eval/cases/background-sandbox-close-waits/case.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"description": "Real bug: closing a background sandbox awaits its start, so a review that ran no command still waits the whole cold start. Fixed later in 40a6a63.",
"source": "https://github.com/appwrite/hansi/commit/ce18b201051950955d680acd3e7dc3b767c47051 (PR #25), fixed by 40a6a63",
"repo": "appwrite/hansi",
"base": "45e7a1aabb28dc194fab49e94d0bbf5e1ea1a470",
"head": "ce18b201051950955d680acd3e7dc3b767c47051",
"pullRequest": {
"title": "feat(engine): start the review sandbox in the background",
"body": "A cold sandbox takes 30 to 60 seconds on production: about 7s to start\nthe pod, then the pull request archive is downloaded and unpacked before\nthe orchestrator answers. The review waited for all of it before the\nmodel saw the diff.\n\nThe sandbox now starts while the review runs, and only the first command\nwaits for it. A sandbox that cannot start is still logged and recorded,\nand every command then fails with the reason. The 'Starting the sandbox'\nprogress phase goes away, since the review no longer stops there."
},
"expected": {
"findings": [
{
"id": "close-awaits-start",
"path": "packages/engine/src/review-sandbox.ts",
"lines": [100, 127],
"issue": "backgroundSandbox passes `() => opening` as `started`, so close() awaits the sandbox's creation before closing it. A review that ran no command still blocks at the end for the whole 30 to 60 second cold start the change meant to take off the critical path."
}
]
}
}
30 changes: 30 additions & 0 deletions apps/eval/cases/sandbox-execute-timeout-and-url-leak/case.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"description": "Planted: two bugs in the sandbox client next to the unpack fix. A request timeout in seconds where milliseconds are due, and the capability URL written into an error message.",
"source": "https://github.com/appwrite/hansi/commit/6b4a72d477fcb61261d77592a9c251bd97d25d3f plus inject.patch",
"repo": "appwrite/hansi",
"base": "fb22b909fdb21a3231084652ead26d7c013fbe33",
"head": "6b4a72d477fcb61261d77592a9c251bd97d25d3f",
"pullRequest": {
"title": "fix(sandbox): keep a tracked file at the workspace archive's path",
"body": "The unpack removed the archive after extracting it, so a pull request that\ntracks a root .hansi-workspace.tar.gz lost that file. The archive is now\nunlinked once open on tar's stdin, and the tracked file is extracted in its\nplace."
},
"expected": {
"verdict": "request_changes",
"findings": [
{
"id": "timeout-units",
"path": "packages/sandbox/src/orchestrator.ts",
"lines": [135, 135],
"severity": "major",
"issue": "AbortSignal.timeout takes milliseconds, but `timeoutSeconds + REQUEST_GRACE_MS` adds seconds to the 10,000 ms grace. Every command request is aborted after about 10 seconds whatever its timeout, so long commands are reported as timed out."
},
{
"id": "url-in-error",
"path": "packages/sandbox/src/orchestrator.ts",
"lines": [144, 144],
"severity": "major",
"issue": "The sandbox URL is a capability (anyone holding it can run commands, and the class says never to log it), but the error message now includes it. SandboxError messages reach logs and the review trace, leaking the capability."
}
]
}
}
22 changes: 22 additions & 0 deletions apps/eval/cases/sandbox-execute-timeout-and-url-leak/inject.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
diff --git a/packages/sandbox/src/orchestrator.ts b/packages/sandbox/src/orchestrator.ts
index 301ea88..5cba46c 100644
--- a/packages/sandbox/src/orchestrator.ts
+++ b/packages/sandbox/src/orchestrator.ts
@@ -132,7 +132,7 @@ class OrchestratorSandbox implements Sandbox {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ command, timeoutSeconds }),
- signal: AbortSignal.timeout(timeoutSeconds * 1000 + REQUEST_GRACE_MS)
+ signal: AbortSignal.timeout(timeoutSeconds + REQUEST_GRACE_MS)
});
} catch (error) {
if (error instanceof DOMException && error.name === 'TimeoutError') {
@@ -141,7 +141,7 @@ class OrchestratorSandbox implements Sandbox {
throw error;
}
if (!response.ok) {
- throw new SandboxError(`sandbox ${this.id} execute: ${response.status}`);
+ throw new SandboxError(`sandbox ${this.id} execute at ${this.url}: ${response.status}`);
}

const result = (await response.json()) as ExecuteResponse;
15 changes: 15 additions & 0 deletions apps/eval/cases/sandbox-unpack-clean/case.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"description": "Clean: the fix that keeps a tracked file at the workspace archive's path. A good review approves it.",
"source": "https://github.com/appwrite/hansi/commit/6b4a72d477fcb61261d77592a9c251bd97d25d3f",
"repo": "appwrite/hansi",
"base": "fb22b909fdb21a3231084652ead26d7c013fbe33",
"head": "6b4a72d477fcb61261d77592a9c251bd97d25d3f",
"pullRequest": {
"title": "fix(sandbox): keep a tracked file at the workspace archive's path",
"body": "The unpack removed the archive after extracting it, so a pull request that\ntracks a root .hansi-workspace.tar.gz lost that file. The archive is now\nunlinked once open on tar's stdin, and the tracked file is extracted in its\nplace."
},
"expected": {
"verdict": "approve",
"findings": []
}
}
21 changes: 21 additions & 0 deletions apps/eval/cases/sandbox-upload-drops-tracked-archive/case.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"description": "Real bug: unpacking the uploaded workspace deletes a tracked file at the archive's path. Fixed later in 6b4a72d.",
"source": "https://github.com/appwrite/hansi/commit/fb22b909fdb21a3231084652ead26d7c013fbe33 (PR #27), fixed by 6b4a72d",
"repo": "appwrite/hansi",
"base": "4014f6a0569e07775833730753d895fd37fa0dce",
"head": "fb22b909fdb21a3231084652ead26d7c013fbe33",
"pullRequest": {
"title": "perf(sandbox): upload the checkout instead of downloading a tarball",
"body": "The sandbox used to download GitHub's tarball of the pull request head and\nunpack it before it reported ready: for appwrite/appwrite about 420 MB,\n10 to 18 s to download and 7 to 10 s to unpack, on every review.\n\nHansi already has the checkout. openSandbox now writes it with git\narchive, without binary files (images, videos, archives) and files over\n10 MB, uploads it to the sandbox agent's /files and unpacks it there, as\nhansi.codes does with sbx. For appwrite/appwrite that is 26 MB written in\nabout 2.4 s, while the sandbox starts. The run_command description says\nwhat is left out.\n\ngetArchiveUrl has no caller left and goes."
},
"expected": {
"findings": [
{
"id": "rm-tracked-archive",
"path": "packages/sandbox/src/orchestrator.ts",
"lines": [171, 171],
"issue": "The unpack runs `tar -xzf .hansi-workspace.tar.gz && rm -f .hansi-workspace.tar.gz` in the workspace. A repository that tracks a file at that same root path gets it extracted by tar and then deleted by the rm, so the sandbox's checkout is missing a tracked file."
}
]
}
}
21 changes: 21 additions & 0 deletions apps/eval/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "@hansi/eval",
"license": "BSD-3-Clause",
"private": true,
"type": "module",
"scripts": {
"eval": "bun src/index.ts",
"check": "tsc -p ."
},
"dependencies": {
"@hansi/engine": "workspace:*",
"@hansi/sandbox": "workspace:*",
"@hansi/service": "workspace:*",
"ai": "catalog:",
"zod": "catalog:"
},
"devDependencies": {
"@types/bun": "catalog:",
"typescript": "catalog:"
}
}
Loading