davidbzyk · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026
diff --git a/README.md b/README.md
@@ -14,18 +14,23 @@ its own did as well as either context store, with nothing to build or serve. Thi
 holds the public voiceover cases and experiments; I also include the results from General
 Environments' own codebase, as it is a great example of rapid agentic engineering.
 
-**The comparison.** This repository has several commercial reviewers turned on (Codex, Greptile,
-CodeRabbit, Gravity, Devin and Sonar) next to its own review action. That seems excessive, but it
-means every pull request here compares them side by side.
+**The first cross-model result.** On the 12 public eval tasks, gpt-6.1-sol (high) found about 1.7
+times as many of the known bugs as gpt-6-luna (medium), at about 19 times the cost per task ($4.65
+against $0.24, API-equivalent). The numbers are under [Results so far](#results-so-far).
+
+**The comparison.** Several commercial reviewers were turned on here next to the repository's own
+review action, but only two of them reviewed its pull requests: CodeRabbit (60 comments across 18
+pull requests) and GitHub Copilot (17). Codex, Greptile, Gravity, Devin and Sonar posted nothing, so
+the side-by-side on these pull requests is CodeRabbit, Copilot and the review action.
 
 **The design.** I went through all my code and security review plugins and kept what I thought
-was valuable from each; the sources are cited at the bottom of this page.
+was valuable from each; every reviewer cites its sources in its own folder (see the last section).
 
-The full write-up will be a Medium article *(link when published)*, written as a tutorial: import
-this repository into General Environments, run its benchmarks across several models and harnesses,
-and find which one suits this use case best for cost and speed. The article also shows the results
-on a larger, more complex codebase (General Environments itself), and how benchmarking and analysis
-turn the review into a self-improving workflow. This README is its summary.
+The full write-up will be a Medium article *(link when published)* that compares and evaluates:
+this repository's evals are imported into General Environments and run across models and harnesses,
+for review quality and for cost. Running every task on every model proved expensive, so part of the
+work is finding the most economically viable set of tasks and models. The article also covers a
+larger, more complex codebase (General Environments itself). This README is its summary.
 
 ## What is in this repository
 
@@ -101,7 +106,8 @@ scripts rather than committed.
 ## The CI action: add the review to your project
 
 Every pull request gets reviewed by the seven-reviewer team on a GitHub runner, and the findings are
-posted as one review with inline comments plus a summary that updates on each push.
+posted as one review with inline comments plus a summary that updates on each push. A review takes
+2 to 6 minutes here (median 3.5 over the last ten runs).
 
 1. **Install it into your repository** (from a checkout of this one):
 
@@ -161,6 +167,47 @@ run time, so the same tasks compare any of them.
 - `evals/cases/cr-self-01`: this repository's own history, the first commit of the CI action and
   the 18 confirmed bugs it contained. Can the review tool find the bugs in itself?
 
+### Results so far
+
+Two full runs on 2026-10-12, one trial per task: the review team on the Codex CLI with gpt-6-luna
+(reasoning medium) and with gpt-6.1-sol (high), each graded by a Codex judge on gpt-6-luna. Cost is
+Harbor's API-equivalent figure for the review team (the runs used a ChatGPT subscription); the
+judge's cost is not included. Over the 12 public tasks:
+
+| Agent model | Recall | Findings per task | Input tokens per task | Cost per task |
+|---|---|---|---|---|
+| gpt-6-luna (medium) | 0.31 | 5.6 | 15.5M | $0.24 |
+| gpt-6.1-sol (high) | 0.54 | 13.4 | 21.7M | $4.65 |
+
+- **Sol's recall is about 1.7 times Luna's, at about 19 times the cost.** It also writes 2.4 times
+  as many findings. It led on 8 tasks, Luna on 1 (vo-x20, 0.80 to 0), and 3 were tied.
+- **Single tasks are noisy.** With one trial each, a key of one or two issues swings a whole point
+  (vo-x7, vo-x1); read the totals, not one task.
+- **One known judge miss.** On vo-x4 under Sol the judge scored six findings 0 without opening
+  `findings.json`: the only such miss in all 44 trials of the two full runs (22 tasks each, the
+  12 public ones plus 10 private). It stays: every run is graded the same way.
+- The `-llm-judge` tasks have not been run yet.
+
+Per trial (recall, findings, tokens, cost): `research/results/team-evals-2026-10-12.csv`.
+
+### Run the evals yourself
+
+In General Environments, import this repository: every task folder under `evals/` becomes part of
+one benchmark, and the run picks the agent, model and judge. With Harbor directly:
+
+    export REWARDKIT_JUDGE=codex REWARDKIT_MODEL=gpt-6-luna \
+        CODEX_AUTH_JSON="$(cat ~/.codex/auth.json)"
+    harbor run -y -p evals/team-tasks/vo-x7 -a codex -m openai/gpt-6-luna \
+        --ak reasoning_effort=medium --ae CODEX_AUTH_JSON_PATH=$HOME/.codex/auth.json
+
+A team task runs eight agent sessions, so a full sweep adds up quickly. To start cheaply, take
+the six tasks chosen for content and spread of size: vo-x7 and vo-x4 (one clear bug in a small
+change), vo-t06 (frontend lifecycle bugs), vo-t01 (blocking calls and unchecked input in a
+backend), vo-t04 (a broken path allowlist and a secrets migration) and vo-t02 (the security
+showcase: path traversal, unsafe pickle loading, SSRF). Each also has an `-llm-judge` version.
+A judge from another model family than the agent is the safer choice; the results above used the
+same family for both.
+
 The loop this repository is building toward:
 
 1. A pull request changes the review logic: a skill, a prompt, the launcher, a runner row.
@@ -181,34 +228,13 @@ Steps 1, 2 (the digest) and 4 work today; the rest is this version's work.
 
 ## Status
 
-Built in the open. Current version: v0.1 in progress. Model observations so far: gpt-6-luna leans
-to minor findings and padded summaries, and two runs on nearly the same change disagreed on the
-most serious finding; a comparison across models on the evals is next.
+Current version: v0.1 in progress. Model observations so far: gpt-6-luna leans to minor findings
+and padded summaries, and two runs on nearly the same change disagreed on the most serious finding.
+The first cross-model results are above; the `-llm-judge` tasks and more models are next.
 
 ## Sources and citations
 
-<details>
-<summary>The review plugins and skills the reviewers were built from</summary>
-
-Each reviewer skill records, passage by passage, which source it came from, the source file and
-lines, and whether the text is verbatim or changed: see `CITATIONS.md` in each skill folder under
-[`src/plugin/skills/`](src/plugin/skills/).
-
-- [agent-skills](https://github.com/addyosmani/agent-skills) (Addy Osmani)
-- [brooks-lint](https://github.com/hyhmrright/brooks-lint)
-- [compound-engineering](https://github.com/EveryInc/compound-engineering-plugin) (Every): `ce-code-review`, `ce-simplify-code`
-- claude-code-workflows (Seth Hobson): comprehensive-review, developer-essentials, agent-teams,
-  backend-development, data-validation-suite
-- claude-plugins-official (Anthropic): code-review, code-simplifier, feature-dev code-reviewer,
-  security-guidance
-- pr-review-toolkit (Anthropic)
-- mattpocock-skills: code-review
-- superpowers: requesting and receiving code review
-- codebase-audit-suite (levnikolaevich-skills-marketplace)
-- [Claude-BugHunter](https://github.com/elementalsouls/Claude-BugHunter): security-arsenal
-- [Cursor plugins](https://github.com/cursor/plugins): thermos
-- [antigravity-awesome-skills](https://github.com/sickn33/antigravity-awesome-skills): 007 (renat),
-  secrets-management, security-audit, security-scanning-security-hardening
-- security-lens-reviewer, security-reviewer, multi-agent-review
-
-</details>
+Each reviewer skill cites its sources passage by passage (the source, its file and lines, and
+whether the text is verbatim or changed) in `CITATIONS.md` in its folder under
+[`src/plugin/skills/`](src/plugin/skills/); [`src/plugin/provenance/`](src/plugin/provenance/)
+records what each source contributed and what was left out.
diff --git a/research/results/team-evals-2026-10-12.csv b/research/results/team-evals-2026-10-12.csv
@@ -0,0 +1,25 @@
+task,agent_model,recall,findings,input_tokens,cached_tokens,output_tokens,cost_usd
+vo-t01,gpt-6-luna (medium),0.0,4,19776518,18983424,46739,0.29
+vo-t02,gpt-6-luna (medium),0.4118,10,17305931,16447232,44853,0.27
+vo-t03,gpt-6-luna (medium),0.2222,12,26443444,25360640,53714,0.39
+vo-t04,gpt-6-luna (medium),0.3571,6,23304785,22397696,58324,0.34
+vo-t05,gpt-6-luna (medium),0.0,5,17436927,16616960,41804,0.27
+vo-t06,gpt-6-luna (medium),0.4286,5,13617639,13018880,30582,0.21
+vo-x1,gpt-6-luna (medium),0.0,5,13428711,12776704,34999,0.21
+vo-x17,gpt-6-luna (medium),0.5,4,14637531,13916160,36371,0.23
+vo-x20,gpt-6-luna (medium),0.8,9,16612346,15913728,46519,0.25
+vo-x4,gpt-6-luna (medium),0.0,2,7575154,7139840,22601,0.13
+vo-x7,gpt-6-luna (medium),0.0,2,7674338,7175680,20349,0.13
+vo-x8,gpt-6-luna (medium),1.0,3,8762852,8249600,24380,0.15
+vo-t01,gpt-6.1-sol (high),0.2857,20,32386585,31105152,71286,6.41
+vo-t02,gpt-6.1-sol (high),0.5294,26,23064465,21884160,71708,5.27
+vo-t03,gpt-6.1-sol (high),0.4444,20,35342654,33705344,76033,7.41
+vo-t04,gpt-6.1-sol (high),0.7143,20,20162670,19036416,66012,4.82
+vo-t05,gpt-6.1-sol (high),0.375,22,24373656,23155456,65535,5.41
+vo-t06,gpt-6.1-sol (high),0.5714,10,14682201,13904768,47199,3.42
+vo-x1,gpt-6.1-sol (high),1.0,12,22122899,21266688,53604,4.38
+vo-x17,gpt-6.1-sol (high),0.5,7,25665586,24458112,52452,5.4
+vo-x20,gpt-6.1-sol (high),0.0,13,31439768,30230016,58725,6.03
+vo-x4,gpt-6.1-sol (high),0.0,6,10727396,10174336,31172,2.44
+vo-x7,gpt-6.1-sol (high),1.0,2,10425048,9807616,25558,2.47
+vo-x8,gpt-6.1-sol (high),1.0,3,10557123,10039936,28010,2.32