Skip to content

pr_runtime: run vitest with its verbose reporter and parse it - #143

Open
k21993 wants to merge 1 commit into
huggingface:mainfrom
k21993:fix/vitest-reporter-output
Open

k21993 wants to merge 1 commit into
huggingface:mainfrom
k21993:fix/vitest-reporter-output

Conversation

@k21993

@k21993 k21993 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • normalize_test_cmds_for_runtime now gives vitest --reporter=verbose instead of --verbose. Vitest 3-5 reject --verbose with CACError: Unknown option, so no test ran.
  • Both JS parsers strip ANSI escape codes, which vitest emits whenever TERM is unset (as in a non-TTY docker exec). In a vitest log, they read only the fully qualified verbose lines and drop the run suffixes ( 12ms, (retry xN), (repeat xN), heap usage, skip note). A project label stays in the name as |unit| file > suite > test, whether or not the output is colored.
  • Vitest's default reporter is ignored on purpose. It collapses a fully passing file to one line, so a patch that fixed more than the gold patch would lose its F2P tests (details in pr_runtime can't use vitest repos: --verbose crashes and output doesn't parse #142).
  • Jest's padded colored header ( FAIL file) is accepted, so colored jest output parses the same as plain jest output.

Results on a real vitest 5.0.1 bug fix (the div fix from #142, captured with an empty environment):

Command run F2P Agent patch graded
main npx vitest run --verboseCACError, no tests none task dropped
main parser, even with the right flag colored verbose output → {} none task dropped
this PR npx vitest run --reporter=verbose math.test.ts > math > rejects division by zero 1.0, resolved=True

Test plan

  • uv run --all-extras pytest -q: 1957 passed, 6 skipped (live-network tests)
  • uv run ruff check . and uv run ruff format --check .
  • New tests/test_vitest_output.py runs every case against both parsers. Its fixtures are verbatim runner output with local paths shortened to /repo: vitest 5.0.1 before and after the fix, the default reporter, two projects with and without color, vitest 3.2.7, and colored jest 30.5.0.
  • 21 of the 23 new tests fail on main (including the normalize_test_cmds_for_runtime case), and all pass with this change.
  • The standalone verifier, run with python -I, auto-detects the runner from npx vitest run --reporter=verbose and grades the agent patch at reward 1.0.
  • Pathological 200k-character lines (repeated (retry x1), [, digits, >, escape fragments) parse in under 100 ms.
  • Manually ran the normalized command on vitest 3.2.7, 4.1.11 and 5.0.1.

Out of scope

  • Wrappers such as npm test get no reporter flag, as before (flags would need to go after --), so a vitest suite behind npm test still uses the default reporter.
  • Mocha's spec reporter (, 1) title) still doesn't parse. I'll file that separately.
  • Vitest 2's tree-style verbose output, project names containing spaces with color, and logs that mix jest and vitest runs.
  • Tasks generated before this change keep their baked verifier.

Closes #142

normalize_test_cmds_for_runtime appended `--verbose` to vitest commands,
which vitest 3-5 reject with `CACError: Unknown option`, so no test ran.
Its per-test output comes from `--reporter=verbose` instead.

That output was not parseable either: vitest colors it whenever TERM is
unset, and its bare ` 12ms` duration was left in the test name. Strip
escape codes, then read vitest's fully qualified verbose lines, dropping
the run suffixes and keeping project labels as part of the identity.

Vitest's default reporter is ignored on purpose: it collapses a fully
passing file to one line, so a patch that fixed more than the gold patch
would lose its FAIL_TO_PASS tests from the log.

Stripping escape codes also lets colored jest output parse, once its
padded ` FAIL ` header is accepted.

Closes huggingface#142
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pr_runtime can't use vitest repos: --verbose crashes and output doesn't parse

1 participant