Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Not reviewed by the code-review action, and collapsed in pull request diffs on GitHub:
# frozen snapshots of other projects and research data, and eval tasks the generator writes.
research/** linguist-vendored
evals/tasks/** linguist-generated
evals/team-tasks/** linguist-generated
15 changes: 7 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ turn the review into a self-improving workflow. This README is its summary.
|---|---|---|
| The review team | Seven reviewers (security, tests, correctness, architecture, code quality, compliance, language) and a validator, run by one orchestrator, merged into one de-duplicated set of findings | `src/review_core/`, `src/plugin/` |
| The CI action | Reviews every pull request on a GitHub runner with Codex and posts the findings like the commercial review bots do | `action/` |
| The evals | Rubric-graded Harbor tasks built from real, confirmed bugs, run by one agent or by the whole review team, for any agent, harness, model and judge | `evals/` |
| The evals | Rubric-graded Harbor tasks built from real, confirmed bugs, each run by the whole review team as a multi-step task, for any agent, harness, model and judge | `evals/` |
| The research | The voiceover cases, the frozen answer keys, the OKF notes and every scored review | `research/` |

## The research: does giving review agents more context help?
Expand Down Expand Up @@ -95,7 +95,7 @@ chance. The same answer as above: context stores do not yet earn their cost.
voiceover commits, and each case's manifest and intent), the answer keys and their lock files
(`key/`), the OKF notes (`okf-notes/`), and every scored review with its scores (`results/`). The
exact trees the reviewers read for the six test cases ship inside the eval tasks
(`evals/tasks/vo-t0N/environment/repo.tar.gz`). The vector stores (14 GB) are rebuilt from the
(`evals/team-tasks/vo-t0N/environment/repo.tar.gz`). The vector stores (14 GB) are rebuilt from the
scripts rather than committed.

## The CI action: add the review to your project
Expand Down Expand Up @@ -144,12 +144,11 @@ Each eval is a Harbor task: a real change, its confirmed bugs as a rubric, and a
scores how many of them a review finds. The agent, harness, model and judge are all chosen at
run time, so the same tasks compare any of them.

- `evals/tasks/vo-t01` to `vo-t06`: the six scored voiceover changes, one task per pull request.
- `evals/team-tasks/vo-t01` to `vo-t06`: the same six changes reviewed by the team as it ships. Each
is a multi-step Harbor task: seven reviewer steps, each a fresh agent session given the
orchestrator's own message, then the validator step, whose final findings the same judge grades.
One trajectory per step, so you can see each reviewer come in and out. The single-agent tasks
above are the baseline.
- `evals/team-tasks/vo-t01` to `vo-t06`: the six scored voiceover changes, one task per pull
request, reviewed by the team as it ships. Each is a multi-step Harbor task: seven reviewer
steps, each a fresh agent session given the orchestrator's own message, then the validator step,
whose final findings a rubric judge grades. One trajectory per step, so you can see each
reviewer come in and out.
- `evals/cases/cr-self-01`: this repository's own history, the first commit of the CI action and
the 18 confirmed bugs it contained. Can the review tool find the bugs in itself?

Expand Down
5 changes: 3 additions & 2 deletions ci/tracked-paths.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ deny_globs = [
".env",
".env.*",
# The general-environments eval tasks hold private code and accepted comments: never public.
"evals/tasks/ge-*",
"evals/team-tasks/ge-*",
# Single-agent tasks run one agent, not the review team: they are not evals here.
"evals/tasks/*",
]
# Paths whose case material is published on purpose: the voiceover eval tasks (public repository,
# key unsealed after the scored run, decision 92). Deny globs still win.
allow_globs = ["evals/tasks/vo-*", "evals/team-tasks/vo-*"]
allow_globs = ["evals/team-tasks/vo-*"]
# The denied names those paths may still hold: each task's change archive.
published_names = ["repo.tar.gz"]
# Templates of .env that hold no values.
Expand Down
33 changes: 0 additions & 33 deletions evals/tasks/vo-t01/README.md

This file was deleted.

24 changes: 0 additions & 24 deletions evals/tasks/vo-t01/environment/Dockerfile

This file was deleted.

Binary file removed evals/tasks/vo-t01/environment/repo.tar.gz
Binary file not shown.
23 changes: 0 additions & 23 deletions evals/tasks/vo-t01/instruction.md

This file was deleted.

42 changes: 0 additions & 42 deletions evals/tasks/vo-t01/solution/findings.json

This file was deleted.

6 changes: 0 additions & 6 deletions evals/tasks/vo-t01/solution/solve.sh

This file was deleted.

84 changes: 0 additions & 84 deletions evals/tasks/vo-t01/task.toml

This file was deleted.

26 changes: 0 additions & 26 deletions evals/tasks/vo-t01/tests/Dockerfile

This file was deleted.

Loading
Loading