Repository navigation
Evals: the review team on the six voiceover cases, as multi-step tasks - #12
Merged
Merged
Conversation
evals/team-tasks/vo-t01..06 run the team as it ships: seven reviewer steps, each a fresh session with the orchestrator's message, then the validator, graded by the same rubric judge. The single-agent tasks get the byte-identical archive format. Both are generated in code_review_private and copied here. The tracked-path rules publish vo team-task archives and refuse general-environments ones, as for the single-agent tasks.
|
Important Review skippedToo many files! This PR contains 719 files, which is 569 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to Team to raise the limit. Usage-priced reviews support at most 300 files. ⚙️ Run configuration
⛔ Files ignored due to path filters (12)
📒 Files selected for processing (719)
You can disable this status message by setting the
Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Custom Code ReviewNo issues found on What the change does
codex-ci · gpt-6-luna · effort high · Codex 0.161.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The single-agent evals tested one agent told to review. These test the review team as it ships.
evals/team-tasks/vo-t01tovo-t06: each is a multi-step Harbor task. There are seven reviewer steps, each a fresh agent session given the orchestrator's own message (read from the shippedreview-orchestratorskill). Then the validator step:first-passruns before it,finalizeafter it, and the same rubric judge grades the final findings. One trajectory per step. Skills are baked in atskills_dir, so any Harbor agent works.evals/tasks/vo-*: the single-agent baseline in the byte-identical archive format (no gzip time, git index or host modes). The code is unchanged.ci/tracked-paths.toml: publishes vo team-task archives and refuses general-environments ones, as for the single-agent tasks (test_public_voiceover_tasks_may_ship_their_case_material_but_ge_tasks_never, red then green).Both sets are generated in the private repository and copied here; the commit passed the private-text check (
REVIEW_PRIVATE_ROOT). In private runs (Codex gpt-6-luna, Claude judge), the team found 1/5 on vo-t01 in each of three runs, where the single agent found 0/5. Every team task passes nop 0.0 and Oracle 1.0.