diff --git a/.dockerignore b/.dockerignore index 8a3e703..da96269 100644 --- a/.dockerignore +++ b/.dockerignore @@ -7,3 +7,4 @@ *.md **/vendor apps/consumer/tests +apps/eval diff --git a/AGENTS.md b/AGENTS.md index 5a16eaa..3c3f727 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,7 @@ The engine, the review and chat jobs and the GitHub helpers started as a port of | `bun run check` | Type-check every workspace (`tsc`) | | `bun run lint` | Prettier check + ESLint | | `bun run format` | Apply Prettier | +| `bun run eval` | Score reviews against `apps/eval` | Run `bun run format`, `bun run lint`, `bun run check` and `bun run test` before pushing; CI runs the last three. @@ -26,6 +27,7 @@ The queue consumer is PHP (`apps/consumer`), on PHP 8.5 with Swoole as in `appwr - **apps/service** -- the process: env, logger, health server, the jobs port the consumer hands deliveries to, the worker that runs them and decides retries, job handlers. It starts the consumer as its child, so the container runs and stops as one - **apps/consumer** -- the queue consumer, in PHP on Appwrite's own utopia-php/queue: `Broker\Nats` on Cloud (adopting the queue Appwrite provisioned, with its attempts and backoff read from the work consumer), `Broker\Redis` on a self-hosted Appwrite (where it runs the attempts itself). Each delivery is one `POST /v1/jobs` to the service on loopback, settled by the outcome it answers with +- **apps/eval** -- replays pull requests with known problems (`cases/`) through a real review with the service's models and scores recall, false alarms, verdicts and cost. `bun run eval`; see its README. Run it before and after a model or prompt change - **packages/engine** -- the engine and the contracts it works against (`GitHubClient`, `SandboxClient`, `ResultsSink`, job types). Also the review itself: the agent loop on the Vercel AI SDK, its read-only tools over a local checkout (`git`, `rg`), the prompts, and the code that places, verifies and grades findings. In-memory fakes and a scripted model in `@hansi/engine/testing`. `runReviewJob` and `runChatJob` take a job from the pull request to what is posted on GitHub: the summary comment, the review, the check run, thread resolution and replies. Their only I/O of its own is git, for the checkout; everything else goes through the contracts - **packages/appwrite** -- Appwrite adapters: results sink and installation tokens (internal API), job payload parsing - **packages/github** -- GitHub client on octokit: the helpers ported from hansi.codes (`pulls.ts`), authenticated with installation tokens from Appwrite, renewed before they expire diff --git a/README.md b/README.md index fdef33f..079a03c 100644 --- a/README.md +++ b/README.md @@ -250,9 +250,10 @@ bun run check # type-check every workspace bun run lint # Prettier and ESLint bun run format # apply Prettier bun run dev # run the service with reload +bun run eval # score real reviews against known problems (apps/eval) ``` -Tests use in-memory fakes and a scripted model from `@hansi/engine/testing`, and never call a real model, GitHub or Appwrite. +Tests use in-memory fakes and a scripted model from `@hansi/engine/testing`, and never call a real model, GitHub or Appwrite. The eval is the opposite: it reviews pull requests with known problems using real models, and reports recall, false alarms, verdicts and cost. Run it before and after changing a model or a prompt; [apps/eval](apps/eval/README.md) says how, and how to add a case. The consumer runs on PHP 8.5 with Swoole. Without a local PHP 8.5, run its tests in the image it ships on: diff --git a/apps/eval/.gitignore b/apps/eval/.gitignore new file mode 100644 index 0000000..fbca225 --- /dev/null +++ b/apps/eval/.gitignore @@ -0,0 +1 @@ +results/ diff --git a/apps/eval/README.md b/apps/eval/README.md new file mode 100644 index 0000000..2cf5777 --- /dev/null +++ b/apps/eval/README.md @@ -0,0 +1,71 @@ +# Eval + +Replays pull requests with known problems through a real review and scores what Hansi finds, so a model, prompt or strength change comes with numbers instead of impressions. + +Each case is a pull request and what a good review of it reports. A run reviews every case with the service's own models and engine (no GitHub, no Appwrite: the review is never posted), matches the posted findings to the expected ones, and writes a report. + +```sh +export HANSI_REVIEW_MODEL=bedrock:global.anthropic.claude-sonnet-4-5-20250929-v1:0 +export HANSI_AI_SECRET=… HANSI_AI_REGION=eu-central-1 +bun run eval --runs 3 +bun run eval --strength basic --runs 3 --compare results/.json +``` + +Models come from the same variables as the service (`HANSI_REVIEW_MODEL`, `HANSI_VERIFY_MODEL`, the `HANSI_BASIC_*` and `HANSI_EXPERT_*` models, the reasoning efforts and `HANSI_AI_*`), so a run reviews with what production would at that strength. With `HANSI_ORCHESTRATOR_ENDPOINT` and `HANSI_SANDBOX_IMAGE`, normal and expert reviews run commands in a sandbox as they do in production; without them they only read the code, and the report says so. `GITHUB_TOKEN` is used to clone private repositories. `bun run eval --help` lists the options. + +## What the report says + +| Metric | Meaning | +| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| Recall | Expected problems a posted finding reported, out of all expected problems. | +| Precision (vs labels) | Posted findings that reported an expected problem, out of all posted on cases with problems. Labels can be incomplete. | +| False alarms on clean PRs | Reviews of clean cases that posted any finding. | +| Verdict accuracy | Reviews that reached the case's expected verdict. | +| Real problems dropped | Expected problems a finding reported, but the verifier or a filter then dropped. | +| Cost, time, commands | Per review. Cost uses models.dev's list prices, as the service's metrics do. | + +Models are not deterministic, so use `--runs 3` or more before reading anything into a difference. The JSON report in `results/` (not committed) keeps every run's findings, so an extra finding can be read to see whether it is a real problem the case does not list. + +A finding counts for an expected problem when it is on the same file, at the problem's severity or above, and within `--tolerance` lines of it (5 by default). With `--judge`, a model decides instead, among every finding on the file: slower and costs a little, but it does not count a different problem on the same line, and it does count the right problem reported on a caller. `HANSI_JUDGE_MODEL` picks the judge's model, the verify model by default. + +## Adding a case + +A case is a directory under `cases/` with a `case.json`: + +```json +{ + "description": "Real bug: … Fixed later in .", + "source": "https://github.com/appwrite/hansi/commit/", + "repo": "appwrite/hansi", + "pull": 27, + "base": "<40-character sha>", + "head": "<40-character sha>", + "pullRequest": { "title": "…", "body": "…" }, + "expected": { + "verdict": "request_changes", + "findings": [ + { + "id": "short-name", + "path": "src/file.ts", + "lines": [10, 12], + "issue": "What is wrong and what it breaks, in a sentence or two.", + "severity": "major" + } + ] + } +} +``` + +- `repo` is `owner/name` on GitHub or a clone URL. With `pull`, the head is fetched from the pull request's ref, so heads on forks work; without it, the head commit is fetched by its SHA. +- `lines` are lines of the file at the head, after any patch. `severity` is the least a finding needs to count; leave it out to count any. +- `config` takes the repository settings as in `.hansi.json`. Leave it out to read `.hansi.json` from the base commit, as a review does. +- `linkedIssues` takes `{ number, title, body }` entries, as a review gets them from the pull request. +- An empty `findings` list makes a clean case: a good review posts nothing and approves. + +There are three kinds of cases, and the set needs all three: + +- **Real bugs**: a pull request that shipped a bug a later commit fixed. Base and head are that pull request's; the expected finding is the bug the fix names. +- **Clean**: a pull request that needed nothing, such as the fix itself. These measure false alarms. +- **Planted**: a real pull request with an `inject.patch` beside `case.json`, applied on top of the head as one more commit. It plants a known bug without pushing anything anywhere. Make it with `git diff` in a checkout of the head. + +Run a new case on its own first (`bun run eval --case `) and read the review: a case whose expected problem no reasonable reviewer would see from the pull request measures nothing. diff --git a/apps/eval/cases/background-sandbox-close-clean/case.json b/apps/eval/cases/background-sandbox-close-clean/case.json new file mode 100644 index 0000000..0303dad --- /dev/null +++ b/apps/eval/cases/background-sandbox-close-clean/case.json @@ -0,0 +1,15 @@ +{ + "description": "Clean: the fix for the background sandbox close. A good review approves it.", + "source": "https://github.com/appwrite/hansi/commit/40a6a63bc4c320e8f83d257cdd83d2a33f6281cb", + "repo": "appwrite/hansi", + "base": "ce18b201051950955d680acd3e7dc3b767c47051", + "head": "40a6a63bc4c320e8f83d257cdd83d2a33f6281cb", + "pullRequest": { + "title": "fix(engine): don't wait for a starting sandbox when the review closes it", + "body": "A review that runs no command closed its background sandbox by awaiting\nthe create, so it still sat through the whole start. Close now returns\nat once and the sandbox is closed once it is up." + }, + "expected": { + "verdict": "approve", + "findings": [] + } +} diff --git a/apps/eval/cases/background-sandbox-close-unhandled-rejection/case.json b/apps/eval/cases/background-sandbox-close-unhandled-rejection/case.json new file mode 100644 index 0000000..16c0e8f --- /dev/null +++ b/apps/eval/cases/background-sandbox-close-unhandled-rejection/case.json @@ -0,0 +1,23 @@ +{ + "description": "Planted: the same fix with the catch on the deferred close removed, so a sandbox that fails to start after the review closed it rejects with no handler.", + "source": "https://github.com/appwrite/hansi/commit/40a6a63bc4c320e8f83d257cdd83d2a33f6281cb plus inject.patch", + "repo": "appwrite/hansi", + "base": "ce18b201051950955d680acd3e7dc3b767c47051", + "head": "40a6a63bc4c320e8f83d257cdd83d2a33f6281cb", + "pullRequest": { + "title": "fix(engine): don't wait for a starting sandbox when the review closes it", + "body": "A review that runs no command closed its background sandbox by awaiting\nthe create, so it still sat through the whole start. Close now returns\nat once and the sandbox is closed once it is up." + }, + "expected": { + "verdict": "request_changes", + "findings": [ + { + "id": "unhandled-rejection", + "path": "packages/engine/src/review-sandbox.ts", + "lines": [115, 115], + "severity": "minor", + "issue": "`void opening.then((sandbox) => sandbox.close())` has no catch. When the sandbox fails to start, or its close fails, after the review already closed it, the promise rejects with no handler: an unhandled rejection that can take down the service process with every other job in it." + } + ] + } +} diff --git a/apps/eval/cases/background-sandbox-close-unhandled-rejection/inject.patch b/apps/eval/cases/background-sandbox-close-unhandled-rejection/inject.patch new file mode 100644 index 0000000..90b7fb4 --- /dev/null +++ b/apps/eval/cases/background-sandbox-close-unhandled-rejection/inject.patch @@ -0,0 +1,13 @@ +diff --git a/packages/engine/src/review-sandbox.ts b/packages/engine/src/review-sandbox.ts +index edcc4fc..f1bf854 100644 +--- a/packages/engine/src/review-sandbox.ts ++++ b/packages/engine/src/review-sandbox.ts +@@ -112,7 +112,7 @@ export function backgroundSandbox(open: () => Promise): ReviewSan + // once it is up. + close: async () => { + if (opened) return opened.close(); +- void opening.then((sandbox) => sandbox.close()).catch(() => undefined); ++ void opening.then((sandbox) => sandbox.close()); + } + }; + } diff --git a/apps/eval/cases/background-sandbox-close-waits/case.json b/apps/eval/cases/background-sandbox-close-waits/case.json new file mode 100644 index 0000000..49bf169 --- /dev/null +++ b/apps/eval/cases/background-sandbox-close-waits/case.json @@ -0,0 +1,21 @@ +{ + "description": "Real bug: closing a background sandbox awaits its start, so a review that ran no command still waits the whole cold start. Fixed later in 40a6a63.", + "source": "https://github.com/appwrite/hansi/commit/ce18b201051950955d680acd3e7dc3b767c47051 (PR #25), fixed by 40a6a63", + "repo": "appwrite/hansi", + "base": "45e7a1aabb28dc194fab49e94d0bbf5e1ea1a470", + "head": "ce18b201051950955d680acd3e7dc3b767c47051", + "pullRequest": { + "title": "feat(engine): start the review sandbox in the background", + "body": "A cold sandbox takes 30 to 60 seconds on production: about 7s to start\nthe pod, then the pull request archive is downloaded and unpacked before\nthe orchestrator answers. The review waited for all of it before the\nmodel saw the diff.\n\nThe sandbox now starts while the review runs, and only the first command\nwaits for it. A sandbox that cannot start is still logged and recorded,\nand every command then fails with the reason. The 'Starting the sandbox'\nprogress phase goes away, since the review no longer stops there." + }, + "expected": { + "findings": [ + { + "id": "close-awaits-start", + "path": "packages/engine/src/review-sandbox.ts", + "lines": [100, 127], + "issue": "backgroundSandbox passes `() => opening` as `started`, so close() awaits the sandbox's creation before closing it. A review that ran no command still blocks at the end for the whole 30 to 60 second cold start the change meant to take off the critical path." + } + ] + } +} diff --git a/apps/eval/cases/sandbox-execute-timeout-and-url-leak/case.json b/apps/eval/cases/sandbox-execute-timeout-and-url-leak/case.json new file mode 100644 index 0000000..84390ad --- /dev/null +++ b/apps/eval/cases/sandbox-execute-timeout-and-url-leak/case.json @@ -0,0 +1,30 @@ +{ + "description": "Planted: two bugs in the sandbox client next to the unpack fix. A request timeout in seconds where milliseconds are due, and the capability URL written into an error message.", + "source": "https://github.com/appwrite/hansi/commit/6b4a72d477fcb61261d77592a9c251bd97d25d3f plus inject.patch", + "repo": "appwrite/hansi", + "base": "fb22b909fdb21a3231084652ead26d7c013fbe33", + "head": "6b4a72d477fcb61261d77592a9c251bd97d25d3f", + "pullRequest": { + "title": "fix(sandbox): keep a tracked file at the workspace archive's path", + "body": "The unpack removed the archive after extracting it, so a pull request that\ntracks a root .hansi-workspace.tar.gz lost that file. The archive is now\nunlinked once open on tar's stdin, and the tracked file is extracted in its\nplace." + }, + "expected": { + "verdict": "request_changes", + "findings": [ + { + "id": "timeout-units", + "path": "packages/sandbox/src/orchestrator.ts", + "lines": [135, 135], + "severity": "major", + "issue": "AbortSignal.timeout takes milliseconds, but `timeoutSeconds + REQUEST_GRACE_MS` adds seconds to the 10,000 ms grace. Every command request is aborted after about 10 seconds whatever its timeout, so long commands are reported as timed out." + }, + { + "id": "url-in-error", + "path": "packages/sandbox/src/orchestrator.ts", + "lines": [144, 144], + "severity": "major", + "issue": "The sandbox URL is a capability (anyone holding it can run commands, and the class says never to log it), but the error message now includes it. SandboxError messages reach logs and the review trace, leaking the capability." + } + ] + } +} diff --git a/apps/eval/cases/sandbox-execute-timeout-and-url-leak/inject.patch b/apps/eval/cases/sandbox-execute-timeout-and-url-leak/inject.patch new file mode 100644 index 0000000..24a61f1 --- /dev/null +++ b/apps/eval/cases/sandbox-execute-timeout-and-url-leak/inject.patch @@ -0,0 +1,22 @@ +diff --git a/packages/sandbox/src/orchestrator.ts b/packages/sandbox/src/orchestrator.ts +index 301ea88..5cba46c 100644 +--- a/packages/sandbox/src/orchestrator.ts ++++ b/packages/sandbox/src/orchestrator.ts +@@ -132,7 +132,7 @@ class OrchestratorSandbox implements Sandbox { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ command, timeoutSeconds }), +- signal: AbortSignal.timeout(timeoutSeconds * 1000 + REQUEST_GRACE_MS) ++ signal: AbortSignal.timeout(timeoutSeconds + REQUEST_GRACE_MS) + }); + } catch (error) { + if (error instanceof DOMException && error.name === 'TimeoutError') { +@@ -141,7 +141,7 @@ class OrchestratorSandbox implements Sandbox { + throw error; + } + if (!response.ok) { +- throw new SandboxError(`sandbox ${this.id} execute: ${response.status}`); ++ throw new SandboxError(`sandbox ${this.id} execute at ${this.url}: ${response.status}`); + } + + const result = (await response.json()) as ExecuteResponse; diff --git a/apps/eval/cases/sandbox-unpack-clean/case.json b/apps/eval/cases/sandbox-unpack-clean/case.json new file mode 100644 index 0000000..07568df --- /dev/null +++ b/apps/eval/cases/sandbox-unpack-clean/case.json @@ -0,0 +1,15 @@ +{ + "description": "Clean: the fix that keeps a tracked file at the workspace archive's path. A good review approves it.", + "source": "https://github.com/appwrite/hansi/commit/6b4a72d477fcb61261d77592a9c251bd97d25d3f", + "repo": "appwrite/hansi", + "base": "fb22b909fdb21a3231084652ead26d7c013fbe33", + "head": "6b4a72d477fcb61261d77592a9c251bd97d25d3f", + "pullRequest": { + "title": "fix(sandbox): keep a tracked file at the workspace archive's path", + "body": "The unpack removed the archive after extracting it, so a pull request that\ntracks a root .hansi-workspace.tar.gz lost that file. The archive is now\nunlinked once open on tar's stdin, and the tracked file is extracted in its\nplace." + }, + "expected": { + "verdict": "approve", + "findings": [] + } +} diff --git a/apps/eval/cases/sandbox-upload-drops-tracked-archive/case.json b/apps/eval/cases/sandbox-upload-drops-tracked-archive/case.json new file mode 100644 index 0000000..7da5d32 --- /dev/null +++ b/apps/eval/cases/sandbox-upload-drops-tracked-archive/case.json @@ -0,0 +1,21 @@ +{ + "description": "Real bug: unpacking the uploaded workspace deletes a tracked file at the archive's path. Fixed later in 6b4a72d.", + "source": "https://github.com/appwrite/hansi/commit/fb22b909fdb21a3231084652ead26d7c013fbe33 (PR #27), fixed by 6b4a72d", + "repo": "appwrite/hansi", + "base": "4014f6a0569e07775833730753d895fd37fa0dce", + "head": "fb22b909fdb21a3231084652ead26d7c013fbe33", + "pullRequest": { + "title": "perf(sandbox): upload the checkout instead of downloading a tarball", + "body": "The sandbox used to download GitHub's tarball of the pull request head and\nunpack it before it reported ready: for appwrite/appwrite about 420 MB,\n10 to 18 s to download and 7 to 10 s to unpack, on every review.\n\nHansi already has the checkout. openSandbox now writes it with git\narchive, without binary files (images, videos, archives) and files over\n10 MB, uploads it to the sandbox agent's /files and unpacks it there, as\nhansi.codes does with sbx. For appwrite/appwrite that is 26 MB written in\nabout 2.4 s, while the sandbox starts. The run_command description says\nwhat is left out.\n\ngetArchiveUrl has no caller left and goes." + }, + "expected": { + "findings": [ + { + "id": "rm-tracked-archive", + "path": "packages/sandbox/src/orchestrator.ts", + "lines": [171, 171], + "issue": "The unpack runs `tar -xzf .hansi-workspace.tar.gz && rm -f .hansi-workspace.tar.gz` in the workspace. A repository that tracks a file at that same root path gets it extracted by tar and then deleted by the rm, so the sandbox's checkout is missing a tracked file." + } + ] + } +} diff --git a/apps/eval/package.json b/apps/eval/package.json new file mode 100644 index 0000000..72063d8 --- /dev/null +++ b/apps/eval/package.json @@ -0,0 +1,21 @@ +{ + "name": "@hansi/eval", + "license": "BSD-3-Clause", + "private": true, + "type": "module", + "scripts": { + "eval": "bun src/index.ts", + "check": "tsc -p ." + }, + "dependencies": { + "@hansi/engine": "workspace:*", + "@hansi/sandbox": "workspace:*", + "@hansi/service": "workspace:*", + "ai": "catalog:", + "zod": "catalog:" + }, + "devDependencies": { + "@types/bun": "catalog:", + "typescript": "catalog:" + } +} diff --git a/apps/eval/src/case.test.ts b/apps/eval/src/case.test.ts new file mode 100644 index 0000000..4945a03 --- /dev/null +++ b/apps/eval/src/case.test.ts @@ -0,0 +1,87 @@ +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { cloneUrl, isGitHubUrl, loadCases } from './case'; + +const valid = { + description: 'Missing await', + repo: 'appwrite/hansi', + pull: 12, + base: 'a'.repeat(40), + head: 'b'.repeat(40), + pullRequest: { title: 'Fix things' }, + expected: { + verdict: 'request_changes', + findings: [{ id: 'await', path: 'src/x.ts', lines: [3, 4], issue: 'Not awaited.' }] + } +}; + +describe('loadCases', () => { + let root: string; + const write = async (id: string, body: unknown, patch?: string) => { + await mkdir(join(root, id)); + await writeFile(join(root, id, 'case.json'), JSON.stringify(body)); + if (patch) await writeFile(join(root, id, 'inject.patch'), patch); + }; + + beforeAll(async () => { + root = await mkdtemp(join(tmpdir(), 'hansi-eval-cases-')); + }); + afterAll(() => rm(root, { recursive: true, force: true })); + + test('reads cases with defaults and their patch, filtered by id', async () => { + await write('a-plain', valid); + await write('b-planted', valid, 'diff --git a/x b/x\n'); + const cases = await loadCases(root); + expect(cases.map((c) => c.id)).toEqual(['a-plain', 'b-planted']); + expect(cases[0]).toMatchObject({ + pullRequest: { title: 'Fix things', body: '', author: 'octocat' }, + linkedIssues: [] + }); + expect(cases[0]?.patch).toBeUndefined(); + expect(cases[1]?.patch).toBe(join(root, 'b-planted', 'inject.patch')); + expect((await loadCases(root, ['planted'])).map((c) => c.id)).toEqual(['b-planted']); + }); + + test('names the case and every problem in it', async () => { + const broken = await mkdtemp(join(tmpdir(), 'hansi-eval-cases-')); + await mkdir(join(broken, 'bad')); + await writeFile( + join(broken, 'bad', 'case.json'), + JSON.stringify({ ...valid, base: 'main', expected: { findings: [{ id: 'x' }] } }) + ); + const error = await loadCases(broken).catch((e: Error) => e); + expect(String(error)).toContain('case bad is invalid'); + expect(String(error)).toContain('base: a full 40-character commit SHA'); + expect(String(error)).toContain('expected.findings.0.path'); + await rm(broken, { recursive: true, force: true }); + }); + + test('rejects an expected finding listed twice', async () => { + const twice = await mkdtemp(join(tmpdir(), 'hansi-eval-cases-')); + const finding = valid.expected.findings[0]; + await mkdir(join(twice, 'dup')); + await writeFile( + join(twice, 'dup', 'case.json'), + JSON.stringify({ ...valid, expected: { findings: [finding, finding] } }) + ); + expect(String(await loadCases(twice).catch((e: Error) => e))).toContain( + 'expected finding await is listed twice' + ); + await rm(twice, { recursive: true, force: true }); + }); +}); + +test('cloneUrl reads owner/name as GitHub and keeps URLs', () => { + expect(cloneUrl('appwrite/hansi')).toBe('https://github.com/appwrite/hansi.git'); + expect(cloneUrl('file:///tmp/repo')).toBe('file:///tmp/repo'); +}); + +test('isGitHubUrl only holds for https on github.com', () => { + expect(isGitHubUrl('https://github.com/appwrite/hansi.git')).toBe(true); + expect(isGitHubUrl('https://github.com.evil.example/x.git')).toBe(false); + expect(isGitHubUrl('https://evil.example/github.com/x.git')).toBe(false); + expect(isGitHubUrl('http://github.com/appwrite/hansi.git')).toBe(false); + expect(isGitHubUrl('file:///tmp/repo')).toBe(false); +}); diff --git a/apps/eval/src/case.ts b/apps/eval/src/case.ts new file mode 100644 index 0000000..7afcf5d --- /dev/null +++ b/apps/eval/src/case.ts @@ -0,0 +1,119 @@ +import { readdir, readFile, stat } from 'node:fs/promises'; +import { join } from 'node:path'; +import { severities, verdicts } from '@hansi/engine'; +import { z } from 'zod'; + +// An eval case is one pull request with what a good review of it finds. Cases live in +// `cases//case.json`; a case may carry `inject.patch`, applied on top of the head as one more +// commit, to plant a known bug in a real pull request without pushing anything anywhere. + +const sha = z.string().regex(/^[0-9a-f]{40}$/, 'a full 40-character commit SHA'); + +export const expectedFindingSchema = z.object({ + /** Short, stable name for the problem, used in reports, e.g. `missing-await`. */ + id: z.string().min(1), + path: z.string().min(1), + /** New-file lines the problem is on, first and last. A finding counts near them. */ + lines: z + .tuple([z.number().int().positive(), z.number().int().positive()]) + .refine(([start, end]) => end >= start, 'the last line comes after the first'), + /** What is wrong, in a sentence or two: what the judge compares a finding against. */ + issue: z.string().min(1), + /** The least severity a finding needs to count. Unset counts any severity. */ + severity: z.enum(severities).optional() +}); + +export const caseSchema = z.object({ + /** What the case checks, for whoever reads a report. */ + description: z.string().min(1), + /** Where the case comes from, e.g. the pull request and the commit that fixed the bug. */ + source: z.string().optional(), + /** `owner/name` on GitHub, or a clone URL. */ + repo: z.string().min(1), + /** Pull request number; its head is fetched from `refs/pull//head`, so forks work. */ + pull: z.number().int().positive().optional(), + base: sha, + head: sha, + pullRequest: z.object({ + title: z.string(), + body: z.string().default(''), + author: z.string().default('octocat') + }), + /** + * The repository settings, as in `.hansi.json`. Unset reads `.hansi.json` from the base commit, + * as a review does. + */ + config: z.record(z.string(), z.unknown()).optional(), + linkedIssues: z + .array(z.object({ number: z.number().int(), title: z.string(), body: z.string() })) + .default([]), + expected: z.object({ + /** The verdict a good review reaches. Unset is not scored. */ + verdict: z.enum(verdicts).optional(), + /** Every problem a good review reports. Empty means a clean pull request. */ + findings: z.array(expectedFindingSchema) + }) +}); + +export type ExpectedFinding = z.infer; + +export interface EvalCase extends z.infer { + id: string; + /** Absolute path of `inject.patch`, when the case has one. */ + patch?: string; +} + +export const CASE_FILE = 'case.json'; +export const PATCH_FILE = 'inject.patch'; + +/** The clone URL of a case's repository. */ +export function cloneUrl(repo: string): string { + return repo.includes('://') || repo.startsWith('git@') ? repo : `https://github.com/${repo}.git`; +} + +/** + * Whether a clone URL is https on github.com. Only those get `GITHUB_TOKEN`: git sends a token as + * a header to whatever host it fetches from, in the clear over http, and a case may name any URL. + */ +export function isGitHubUrl(url: string): boolean { + try { + const parsed = new URL(url); + return parsed.protocol === 'https:' && parsed.hostname === 'github.com'; + } catch { + return false; + } +} + +/** Reads one case directory. Throws with the case id and every problem in its file. */ +export async function loadCase(dir: string, id: string): Promise { + const raw = JSON.parse(await readFile(join(dir, CASE_FILE), 'utf8')) as unknown; + const parsed = caseSchema.safeParse(raw); + if (!parsed.success) { + const problems = parsed.error.issues.map( + (i) => `${i.path.join('.') || '(root)'}: ${i.message}` + ); + throw new Error(`case ${id} is invalid:\n ${problems.join('\n ')}`); + } + const ids = parsed.data.expected.findings.map((f) => f.id); + const repeated = ids.find((findingId, index) => ids.indexOf(findingId) !== index); + if (repeated) + throw new Error(`case ${id} is invalid:\n expected finding ${repeated} is listed twice`); + + const patch = join(dir, PATCH_FILE); + const hasPatch = await stat(patch).then( + (s) => s.isFile(), + () => false + ); + return { ...parsed.data, id, ...(hasPatch && { patch }) }; +} + +/** Every case under `root`, by id, keeping those whose id contains one of `filters`. */ +export async function loadCases(root: string, filters: string[] = []): Promise { + const entries = await readdir(root, { withFileTypes: true }); + const ids = entries + .filter((e) => e.isDirectory()) + .map((e) => e.name) + .filter((id) => filters.length === 0 || filters.some((f) => id.includes(f))) + .sort(); + return Promise.all(ids.map((id) => loadCase(join(root, id), id))); +} diff --git a/apps/eval/src/env.test.ts b/apps/eval/src/env.test.ts new file mode 100644 index 0000000..d0d2540 --- /dev/null +++ b/apps/eval/src/env.test.ts @@ -0,0 +1,36 @@ +import { expect, test } from 'bun:test'; +import { loadEvalEnv } from './env'; + +const fail = (message: string): never => { + throw new Error(message); +}; + +test('reads the model variables', () => { + const env = loadEvalEnv( + { HANSI_REVIEW_MODEL: 'anthropic:claude-sonnet-4-5', HANSI_AI_SECRET: 'key' }, + fail + ); + expect(env).toMatchObject({ + HANSI_REVIEW_MODEL: 'anthropic:claude-sonnet-4-5', + HANSI_SANDBOX_RUNTIME_CLASS: 'gvisor' + }); +}); + +test('rejects models on another provider than the review model, as the service does', () => { + expect(() => + loadEvalEnv( + { + HANSI_REVIEW_MODEL: 'anthropic:claude-sonnet-4-5', + HANSI_JUDGE_MODEL: 'openai:gpt-5', + HANSI_AI_SECRET: 'key' + }, + fail + ) + ).toThrow("HANSI_JUDGE_MODEL: use the review model's provider, anthropic"); +}); + +test('names every missing variable at once', () => { + expect(() => loadEvalEnv({ HANSI_REVIEW_MODEL: 'bedrock:x' }, fail)).toThrow( + 'HANSI_AI_SECRET is required for bedrock\nHANSI_AI_REGION is required for bedrock' + ); +}); diff --git a/apps/eval/src/env.ts b/apps/eval/src/env.ts new file mode 100644 index 0000000..c38413a --- /dev/null +++ b/apps/eval/src/env.ts @@ -0,0 +1,102 @@ +import { reasoningEfforts, type ReasoningEffort } from '@hansi/engine'; +import { + isProviderId, + missingConfig, + parseModelId, + providerIds, + type ModelEnv, + type ProviderId +} from '@hansi/service/models'; + +// The eval reads the service's model variables, so a run reviews with what production would, +// and nothing else the service needs (Appwrite, the queue). + +export interface EvalEnv extends ModelEnv { + /** The judge's model; defaults to the verify model, then the review model. */ + HANSI_JUDGE_MODEL?: string; + HANSI_ORCHESTRATOR_ENDPOINT?: string; + HANSI_ORCHESTRATOR_KEY?: string; + HANSI_SANDBOX_IMAGE?: string; + HANSI_SANDBOX_RUNTIME_CLASS: 'runc' | 'gvisor' | 'kata'; +} + +const MODEL_VARIABLES = [ + 'HANSI_REVIEW_MODEL', + 'HANSI_VERIFY_MODEL', + 'HANSI_BASIC_REVIEW_MODEL', + 'HANSI_BASIC_VERIFY_MODEL', + 'HANSI_EXPERT_REVIEW_MODEL', + 'HANSI_EXPERT_VERIFY_MODEL', + 'HANSI_JUDGE_MODEL' +] as const; + +/** Reads and checks the variables; calls `fail` with every problem at once. */ +export function loadEvalEnv( + source: Record, + fail: (message: string) => never +): EvalEnv { + const read = (name: string) => source[name]?.trim() || undefined; + const problems: string[] = []; + + if (!read('HANSI_REVIEW_MODEL')) problems.push('HANSI_REVIEW_MODEL is required'); + const providers = new Set(); + for (const name of MODEL_VARIABLES) { + const id = read(name); + if (!id) continue; + const { provider, model } = parseModelId(id); + if (!isProviderId(provider) || !model) { + problems.push(`${name}: use provider:model, with a provider of ${providerIds.join(', ')}`); + } else { + providers.add(provider); + } + } + + const reasoning = (name: string): ReasoningEffort | undefined => { + const value = read(name); + if (value === undefined) return undefined; + if ((reasoningEfforts as readonly string[]).includes(value)) return value as ReasoningEffort; + problems.push(`${name}: one of ${reasoningEfforts.join(', ')}`); + return undefined; + }; + const runtimeClass = read('HANSI_SANDBOX_RUNTIME_CLASS') ?? 'gvisor'; + if (!['runc', 'gvisor', 'kata'].includes(runtimeClass)) { + problems.push('HANSI_SANDBOX_RUNTIME_CLASS: one of runc, gvisor, kata'); + } + + const env: EvalEnv = { + HANSI_REVIEW_MODEL: read('HANSI_REVIEW_MODEL') ?? '', + HANSI_VERIFY_MODEL: read('HANSI_VERIFY_MODEL'), + HANSI_BASIC_REVIEW_MODEL: read('HANSI_BASIC_REVIEW_MODEL'), + HANSI_BASIC_VERIFY_MODEL: read('HANSI_BASIC_VERIFY_MODEL'), + HANSI_EXPERT_REVIEW_MODEL: read('HANSI_EXPERT_REVIEW_MODEL'), + HANSI_EXPERT_VERIFY_MODEL: read('HANSI_EXPERT_VERIFY_MODEL'), + HANSI_JUDGE_MODEL: read('HANSI_JUDGE_MODEL'), + HANSI_REVIEW_REASONING: reasoning('HANSI_REVIEW_REASONING'), + HANSI_VERIFY_REASONING: reasoning('HANSI_VERIFY_REASONING'), + HANSI_AI_SECRET: read('HANSI_AI_SECRET'), + HANSI_AI_REGION: read('HANSI_AI_REGION'), + HANSI_AI_BASE_URL: read('HANSI_AI_BASE_URL'), + HANSI_ORCHESTRATOR_ENDPOINT: read('HANSI_ORCHESTRATOR_ENDPOINT'), + HANSI_ORCHESTRATOR_KEY: read('HANSI_ORCHESTRATOR_KEY'), + HANSI_SANDBOX_IMAGE: read('HANSI_SANDBOX_IMAGE'), + HANSI_SANDBOX_RUNTIME_CLASS: runtimeClass as EvalEnv['HANSI_SANDBOX_RUNTIME_CLASS'] + }; + // As in the service: every model shares one HANSI_AI_SECRET, so they share the review + // model's provider. + const reviewProvider = parseModelId(env.HANSI_REVIEW_MODEL).provider; + for (const name of MODEL_VARIABLES) { + const id = read(name); + if (id && name !== 'HANSI_REVIEW_MODEL' && parseModelId(id).provider !== reviewProvider) { + problems.push( + `${name}: use the review model's provider, ${reviewProvider}: there is one HANSI_AI_SECRET` + ); + } + } + for (const provider of providers) { + for (const missing of missingConfig(provider, env)) { + problems.push(`${missing} is required for ${provider}`); + } + } + if (problems.length) fail(problems.join('\n')); + return env; +} diff --git a/apps/eval/src/index.ts b/apps/eval/src/index.ts new file mode 100644 index 0000000..350a02b --- /dev/null +++ b/apps/eval/src/index.ts @@ -0,0 +1,207 @@ +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import { dirname, join, resolve } from 'node:path'; +import { parseArgs } from 'node:util'; +import { modelUsage, type Log, type ModelUsage, type Strength } from '@hansi/engine'; +import { OrchestratorSandboxClient } from '@hansi/sandbox'; +import { createReviewModel, createStrengths } from '@hansi/service/models'; +import { estimateCost, findPrice, loadPriceCatalog } from '@hansi/service/pricing'; +import { loadCases, type EvalCase } from './case'; +import { loadEvalEnv } from './env'; +import { modelJudge } from './judge'; +import { renderMarkdown, runCost, summarize, type Report } from './report'; +import { checkoutCase, reviewCase, type CaseRun, type RunSettings } from './run'; +import { DEFAULT_TOLERANCE } from './score'; + +const HELP = `Replays eval cases through a real review and scores what it finds. + +Usage: bun run eval [options] + + --strength basic, normal or expert (default normal) + --case Only cases whose id contains this; repeatable + --runs Reviews per case, since models are not deterministic (default 1) + --concurrency Reviews at once (default 2) + --judge Let a model decide which finding reports an expected problem, instead + of matching by file and line + --tolerance Lines a finding may sit from an expected problem (default ${DEFAULT_TOLERANCE}) + --label Name of this run in the report (default: the strength and review model) + --out Where to write the JSON report (default results/