Evals: an API judge gets findings too long to paste as its judge files

[davidbzyk]

The 12 voiceover tasks' grader, copied byte for byte from code_review_private: past the 100 KB paste limit, finish.py lists findings.json under [judge] files so an API judge (which cannot open files) gets the review as user content. Agent judges are unchanged. Only steps/validator/tests/finish.py changes in each task.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

📝 Summary

Summary by CodeRabbit

• Improvements
  • Judges can now access findings as a file when the findings are too large to fit in the rubric prompt, supporting more reliable evaluation of lengthy results.

Walkthrough

Twelve validator task variants now pass the workspace findings-file path into rubric preparation. Rubric TOML files with a [judge] section receive a files entry for that path. Existing prompt fallback behavior for oversized findings remains.

Changes

Judge Findings Access

Layer / File(s)	Summary
Rubric findings-file attachment evals/team-t*/steps/validator/tests/finish.py, evals/team-tasks/vo-x*/steps/validator/tests/finish.py	Rubric builders accept the findings-file path and add it to TOML files containing a [judge] section. Oversized-findings prompt fallback behavior remains in place.
Finish-flow path wiring evals/team-t*/steps/validator/tests/finish.py, evals/team-tasks/vo-x*/steps/validator/tests/finish.py	The finish flows pass the workspace findings-file path to rubric preparation. The judge call continues to use the prepared rubric and its existing arguments.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix

Merge Risk: 🟡 Moderate · up to 33d6a

The change lets API judges read long findings from a file. Findings larger than 1 MiB are still dropped, so those judges would grade without the review content. Address this before merging, or explicitly accept the limitation.

Pre-merge checks · 5

Passed checks (5 passed)

Check name	Status	Explanation
Description check		The description clearly explains that the PR passes oversized findings files to API judges while leaving agent judges unchanged.
Title check		The title clearly summarizes the main change: API judges receive findings that are too large to paste into judge files.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 48 functions across 12 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

✨ Simplify code

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{I’m a rabbit with findings in tow,
Through judge TOML, their file paths now show.
If the prompt grows too wide,
The file stays beside,
And I hop where the validators go!}

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: d348a4ed-0b9f-4355-9427-002b7cbc14ea

📥 Commits

Reviewing files that changed from the base of the PR and between ea48210 and 33d6a79.

📒 Files selected for processing (12)

• evals/team-tasks/vo-t01/steps/validator/tests/finish.py
• evals/team-tasks/vo-t02/steps/validator/tests/finish.py
• evals/team-tasks/vo-t03/steps/validator/tests/finish.py
• evals/team-tasks/vo-t04/steps/validator/tests/finish.py
• evals/team-tasks/vo-t05/steps/validator/tests/finish.py
• evals/team-tasks/vo-t06/steps/validator/tests/finish.py
• evals/team-tasks/vo-x1/steps/validator/tests/finish.py
• evals/team-tasks/vo-x17/steps/validator/tests/finish.py
• evals/team-tasks/vo-x20/steps/validator/tests/finish.py
• evals/team-tasks/vo-x4/steps/validator/tests/finish.py
• evals/team-tasks/vo-x7/steps/validator/tests/finish.py
• evals/team-tasks/vo-x8/steps/validator/tests/finish.py

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

📜 Review details

🧰 Additional context used

🪛 ast-grep (0.45.3)

evals/team-tasks/vo-t01/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x8/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-t06/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-t02/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x20/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-t04/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-t05/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x7/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x4/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-t03/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x1/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)

evals/team-tasks/vo-x17/steps/validator/tests/finish.py

[info] 51-51: use jsonify instead of json.dumps for JSON output
Context: json.dumps(str(findings_file))
Note: [CWE-116] Improper Encoding or Escaping of Output.

(use-jsonify)