diff --git a/README.md b/README.md index c08223d..332aa24 100644 --- a/README.md +++ b/README.md @@ -14,18 +14,23 @@ its own did as well as either context store, with nothing to build or serve. Thi holds the public voiceover cases and experiments; I also include the results from General Environments' own codebase, as it is a great example of rapid agentic engineering. -**The comparison.** This repository has several commercial reviewers turned on (Codex, Greptile, -CodeRabbit, Gravity, Devin and Sonar) next to its own review action. That seems excessive, but it -means every pull request here compares them side by side. +**The first cross-model result.** On the 12 public eval tasks, gpt-6.1-sol (high) found about 1.7 +times as many of the known bugs as gpt-6-luna (medium), at about 19 times the cost per task ($4.65 +against $0.24, API-equivalent). The numbers are under [Results so far](#results-so-far). + +**The comparison.** Several commercial reviewers were turned on here next to the repository's own +review action, but only two of them reviewed its pull requests: CodeRabbit (60 comments across 18 +pull requests) and GitHub Copilot (17). Codex, Greptile, Gravity, Devin and Sonar posted nothing, so +the side-by-side on these pull requests is CodeRabbit, Copilot and the review action. **The design.** I went through all my code and security review plugins and kept what I thought -was valuable from each; the sources are cited at the bottom of this page. +was valuable from each; every reviewer cites its sources in its own folder (see the last section). -The full write-up will be a Medium article *(link when published)*, written as a tutorial: import -this repository into General Environments, run its benchmarks across several models and harnesses, -and find which one suits this use case best for cost and speed. The article also shows the results -on a larger, more complex codebase (General Environments itself), and how benchmarking and analysis -turn the review into a self-improving workflow. This README is its summary. +The full write-up will be a Medium article *(link when published)* that compares and evaluates: +this repository's evals are imported into General Environments and run across models and harnesses, +for review quality and for cost. Running every task on every model proved expensive, so part of the +work is finding the most economically viable set of tasks and models. The article also covers a +larger, more complex codebase (General Environments itself). This README is its summary. ## What is in this repository @@ -101,7 +106,8 @@ scripts rather than committed. ## The CI action: add the review to your project Every pull request gets reviewed by the seven-reviewer team on a GitHub runner, and the findings are -posted as one review with inline comments plus a summary that updates on each push. +posted as one review with inline comments plus a summary that updates on each push. A review takes +2 to 6 minutes here (median 3.5 over the last ten runs). 1. **Install it into your repository** (from a checkout of this one): @@ -161,6 +167,47 @@ run time, so the same tasks compare any of them. - `evals/cases/cr-self-01`: this repository's own history, the first commit of the CI action and the 18 confirmed bugs it contained. Can the review tool find the bugs in itself? +### Results so far + +Two full runs on 2026-10-12, one trial per task: the review team on the Codex CLI with gpt-6-luna +(reasoning medium) and with gpt-6.1-sol (high), each graded by a Codex judge on gpt-6-luna. Cost is +Harbor's API-equivalent figure for the review team (the runs used a ChatGPT subscription); the +judge's cost is not included. Over the 12 public tasks: + +| Agent model | Recall | Findings per task | Input tokens per task | Cost per task | +|---|---|---|---|---| +| gpt-6-luna (medium) | 0.31 | 5.6 | 15.5M | $0.24 | +| gpt-6.1-sol (high) | 0.54 | 13.4 | 21.7M | $4.65 | + +- **Sol's recall is about 1.7 times Luna's, at about 19 times the cost.** It also writes 2.4 times + as many findings. It led on 8 tasks, Luna on 1 (vo-x20, 0.80 to 0), and 3 were tied. +- **Single tasks are noisy.** With one trial each, a key of one or two issues swings a whole point + (vo-x7, vo-x1); read the totals, not one task. +- **One known judge miss.** On vo-x4 under Sol the judge scored six findings 0 without opening + `findings.json`: the only such miss in all 44 trials of the two full runs (22 tasks each, the + 12 public ones plus 10 private). It stays: every run is graded the same way. +- The `-llm-judge` tasks have not been run yet. + +Per trial (recall, findings, tokens, cost): `research/results/team-evals-2026-10-12.csv`. + +### Run the evals yourself + +In General Environments, import this repository: every task folder under `evals/` becomes part of +one benchmark, and the run picks the agent, model and judge. With Harbor directly: + + export REWARDKIT_JUDGE=codex REWARDKIT_MODEL=gpt-6-luna \ + CODEX_AUTH_JSON="$(cat ~/.codex/auth.json)" + harbor run -y -p evals/team-tasks/vo-x7 -a codex -m openai/gpt-6-luna \ + --ak reasoning_effort=medium --ae CODEX_AUTH_JSON_PATH=$HOME/.codex/auth.json + +A team task runs eight agent sessions, so a full sweep adds up quickly. To start cheaply, take +the six tasks chosen for content and spread of size: vo-x7 and vo-x4 (one clear bug in a small +change), vo-t06 (frontend lifecycle bugs), vo-t01 (blocking calls and unchecked input in a +backend), vo-t04 (a broken path allowlist and a secrets migration) and vo-t02 (the security +showcase: path traversal, unsafe pickle loading, SSRF). Each also has an `-llm-judge` version. +A judge from another model family than the agent is the safer choice; the results above used the +same family for both. + The loop this repository is building toward: 1. A pull request changes the review logic: a skill, a prompt, the launcher, a runner row. @@ -181,34 +228,13 @@ Steps 1, 2 (the digest) and 4 work today; the rest is this version's work. ## Status -Built in the open. Current version: v0.1 in progress. Model observations so far: gpt-6-luna leans -to minor findings and padded summaries, and two runs on nearly the same change disagreed on the -most serious finding; a comparison across models on the evals is next. +Current version: v0.1 in progress. Model observations so far: gpt-6-luna leans to minor findings +and padded summaries, and two runs on nearly the same change disagreed on the most serious finding. +The first cross-model results are above; the `-llm-judge` tasks and more models are next. ## Sources and citations -
-The review plugins and skills the reviewers were built from - -Each reviewer skill records, passage by passage, which source it came from, the source file and -lines, and whether the text is verbatim or changed: see `CITATIONS.md` in each skill folder under -[`src/plugin/skills/`](src/plugin/skills/). - -- [agent-skills](https://github.com/addyosmani/agent-skills) (Addy Osmani) -- [brooks-lint](https://github.com/hyhmrright/brooks-lint) -- [compound-engineering](https://github.com/EveryInc/compound-engineering-plugin) (Every): `ce-code-review`, `ce-simplify-code` -- claude-code-workflows (Seth Hobson): comprehensive-review, developer-essentials, agent-teams, - backend-development, data-validation-suite -- claude-plugins-official (Anthropic): code-review, code-simplifier, feature-dev code-reviewer, - security-guidance -- pr-review-toolkit (Anthropic) -- mattpocock-skills: code-review -- superpowers: requesting and receiving code review -- codebase-audit-suite (levnikolaevich-skills-marketplace) -- [Claude-BugHunter](https://github.com/elementalsouls/Claude-BugHunter): security-arsenal -- [Cursor plugins](https://github.com/cursor/plugins): thermos -- [antigravity-awesome-skills](https://github.com/sickn33/antigravity-awesome-skills): 007 (renat), - secrets-management, security-audit, security-scanning-security-hardening -- security-lens-reviewer, security-reviewer, multi-agent-review - -
+Each reviewer skill cites its sources passage by passage (the source, its file and lines, and +whether the text is verbatim or changed) in `CITATIONS.md` in its folder under +[`src/plugin/skills/`](src/plugin/skills/); [`src/plugin/provenance/`](src/plugin/provenance/) +records what each source contributed and what was left out. diff --git a/research/results/team-evals-2026-10-12.csv b/research/results/team-evals-2026-10-12.csv new file mode 100644 index 0000000..0cbfaab --- /dev/null +++ b/research/results/team-evals-2026-10-12.csv @@ -0,0 +1,25 @@ +task,agent_model,recall,findings,input_tokens,cached_tokens,output_tokens,cost_usd +vo-t01,gpt-6-luna (medium),0.0,4,19776518,18983424,46739,0.29 +vo-t02,gpt-6-luna (medium),0.4118,10,17305931,16447232,44853,0.27 +vo-t03,gpt-6-luna (medium),0.2222,12,26443444,25360640,53714,0.39 +vo-t04,gpt-6-luna (medium),0.3571,6,23304785,22397696,58324,0.34 +vo-t05,gpt-6-luna (medium),0.0,5,17436927,16616960,41804,0.27 +vo-t06,gpt-6-luna (medium),0.4286,5,13617639,13018880,30582,0.21 +vo-x1,gpt-6-luna (medium),0.0,5,13428711,12776704,34999,0.21 +vo-x17,gpt-6-luna (medium),0.5,4,14637531,13916160,36371,0.23 +vo-x20,gpt-6-luna (medium),0.8,9,16612346,15913728,46519,0.25 +vo-x4,gpt-6-luna (medium),0.0,2,7575154,7139840,22601,0.13 +vo-x7,gpt-6-luna (medium),0.0,2,7674338,7175680,20349,0.13 +vo-x8,gpt-6-luna (medium),1.0,3,8762852,8249600,24380,0.15 +vo-t01,gpt-6.1-sol (high),0.2857,20,32386585,31105152,71286,6.41 +vo-t02,gpt-6.1-sol (high),0.5294,26,23064465,21884160,71708,5.27 +vo-t03,gpt-6.1-sol (high),0.4444,20,35342654,33705344,76033,7.41 +vo-t04,gpt-6.1-sol (high),0.7143,20,20162670,19036416,66012,4.82 +vo-t05,gpt-6.1-sol (high),0.375,22,24373656,23155456,65535,5.41 +vo-t06,gpt-6.1-sol (high),0.5714,10,14682201,13904768,47199,3.42 +vo-x1,gpt-6.1-sol (high),1.0,12,22122899,21266688,53604,4.38 +vo-x17,gpt-6.1-sol (high),0.5,7,25665586,24458112,52452,5.4 +vo-x20,gpt-6.1-sol (high),0.0,13,31439768,30230016,58725,6.03 +vo-x4,gpt-6.1-sol (high),0.0,6,10727396,10174336,31172,2.44 +vo-x7,gpt-6.1-sol (high),1.0,2,10425048,9807616,25558,2.47 +vo-x8,gpt-6.1-sol (high),1.0,3,10557123,10039936,28010,2.32