Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,11 @@ run time, so the same tasks compare any of them.
changes, run the same way. Each key is the pull request's first bot review (Codex or CodeRabbit),
keeping only the findings the reviewed commit confirms on reading. Five of them (all but
vo-x20) were the study's tuning changes, so they are not blind like the six scored ones.
- `evals/team-tasks/<slug>-llm-judge` for vo-x7, vo-x4, vo-t06, vo-t01, vo-t04 and vo-t02: the same
six changes with another grader. A judge with file tools reads the review, the known issues and
the code at the reviewed commit, and rates four 1-5 scales whose weights sum to 1: known issues
(0.30), whether other findings hold up in the code (0.25), severity and fixes (0.15), and
economic value, the hard engineering work the review saves (0.30).
- `evals/cases/cr-self-01`: this repository's own history, the first commit of the CI action and
the 18 confirmed bugs it contained. Can the review tool find the bugs in itself?

Expand Down
32 changes: 32 additions & 0 deletions evals/team-tasks/vo-t01-llm-judge/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions evals/team-tasks/vo-t01-llm-judge/environment/Dockerfile

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

76 changes: 76 additions & 0 deletions evals/team-tasks/vo-t01-llm-judge/environment/collect_findings.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file not shown.
Loading
Loading