Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ turn the review into a self-improving workflow. This README is its summary.
|---|---|---|
| The review team | Seven reviewers (security, tests, correctness, architecture, code quality, compliance, language) and a validator, run by one orchestrator, merged into one de-duplicated set of findings | `src/review_core/`, `src/plugin/` |
| The CI action | Reviews every pull request on a GitHub runner with Codex and posts the findings like the commercial review bots do | `action/` |
| The evals | Rubric-graded Harbor tasks built from real, confirmed bugs, for any agent, harness, model and judge | `evals/` |
| The evals | Rubric-graded Harbor tasks built from real, confirmed bugs, run by one agent or by the whole review team, for any agent, harness, model and judge | `evals/` |
| The research | The voiceover cases, the frozen answer keys, the OKF notes and every scored review | `research/` |

## The research: does giving review agents more context help?
Expand Down Expand Up @@ -145,6 +145,11 @@ scores how many of them a review finds. The agent, harness, model and judge are
run time, so the same tasks compare any of them.

- `evals/tasks/vo-t01` to `vo-t06`: the six scored voiceover changes, one task per pull request.
- `evals/team-tasks/vo-t01` to `vo-t06`: the same six changes reviewed by the team as it ships. Each
is a multi-step Harbor task: seven reviewer steps, each a fresh agent session given the
orchestrator's own message, then the validator step, whose final findings the same judge grades.
One trajectory per step, so you can see each reviewer come in and out. The single-agent tasks
above are the baseline.
- `evals/cases/cr-self-01`: this repository's own history, the first commit of the CI action and
the 18 confirmed bugs it contained. Can the review tool find the bugs in itself?

Expand Down
3 changes: 2 additions & 1 deletion ci/tracked-paths.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,11 @@ deny_globs = [
".env.*",
# The general-environments eval tasks hold private code and accepted comments: never public.
"evals/tasks/ge-*",
"evals/team-tasks/ge-*",
]
# Paths whose case material is published on purpose: the voiceover eval tasks (public repository,
# key unsealed after the scored run, decision 92). Deny globs still win.
allow_globs = ["evals/tasks/vo-*"]
allow_globs = ["evals/tasks/vo-*", "evals/team-tasks/vo-*"]
# The denied names those paths may still hold: each task's change archive.
published_names = ["repo.tar.gz"]
# Templates of .env that hold no values.
Expand Down
4 changes: 3 additions & 1 deletion evals/tasks/vo-t01/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t01/environment/repo.tar.gz
Binary file not shown.
Empty file modified evals/tasks/vo-t01/tests/judge_prelude.sh
100644 → 100755
Empty file.
4 changes: 3 additions & 1 deletion evals/tasks/vo-t02/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t02/environment/repo.tar.gz
Binary file not shown.
Empty file modified evals/tasks/vo-t02/tests/judge_prelude.sh
100644 → 100755
Empty file.
4 changes: 3 additions & 1 deletion evals/tasks/vo-t03/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t03/environment/repo.tar.gz
Binary file not shown.
4 changes: 3 additions & 1 deletion evals/tasks/vo-t04/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t04/environment/repo.tar.gz
Binary file not shown.
4 changes: 3 additions & 1 deletion evals/tasks/vo-t05/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t05/environment/repo.tar.gz
Binary file not shown.
4 changes: 3 additions & 1 deletion evals/tasks/vo-t06/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified evals/tasks/vo-t06/environment/repo.tar.gz
Binary file not shown.
23 changes: 23 additions & 0 deletions evals/team-tasks/vo-t01/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions evals/team-tasks/vo-t01/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

76 changes: 76 additions & 0 deletions evals/team-tasks/vo-t01/environment/collect_findings.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file added evals/team-tasks/vo-t01/environment/repo.tar.gz
Binary file not shown.
Loading
Loading