Description
Repos tested with vitest can't produce pr_runtime or cve_patches tasks. Three things go wrong in sequence.
1. The normalized command crashes. normalize_test_cmds_for_runtime (src/repo2rlenv/pipelines/pr_runtime.py L675-682) appends --verbose to any command containing jest|mocha|vitest. Vitest has no such flag:
$ npx vitest run --verbose
CACError: Unknown option `--verbose`
Reproduced on vitest 3.2.7, 4.1.11 and 5.0.1. No test runs, so pre and post both parse to {}.
2. Vitest output is colored in the sandbox. Vitest's bundled tinyrainbow enables color unless NO_COLOR, TERM=dumb or FORCE_TTY=false is set, even when output is piped. A non-TTY docker exec has no TERM, so every line starts with an escape code, and _JEST_TEST_RE (log_parsers/jest_parser.py L54-56, _pr_runtime_verifier.py L177-179) never matches. With an empty environment (env -i PATH=... HOME=...), npx vitest run --reporter=verbose parses to {} on main.
3. Uncolored lines keep the duration in the name. The verbose reporter prints ✓ math.test.ts > math > adds 1ms, a bare 1ms rather than jest's (1 ms). So the test is math.test.ts > math > adds 1ms in one run and ... adds 4ms in the next, and F2P/P2P names stop matching at grading time.
A trap in the obvious fix
Parsing vitest's default reporter isn't safe, even after stripping color. It expands a file only while that file has a failure, and collapses a fully passing file to one line. Real vitest 5.0.1 output for a fix to div, where the gold patch leaves an unrelated test in the same file failing:
# gold patch: file still has a failure, so tests are listed
❯ math (3)
✓ adds 1ms
✓ rejects division by zero 0ms
× rounds half up 3ms
# agent patch that also fixes `rounds half up`
✓ math.test.ts (3 tests) 2ms
F2P would be rejects division by zero, and the agent's more complete patch would lose it from the log and score 0.
Proposed fix
- For vitest commands, drop
--verbose and append --reporter=verbose (unless a --reporter is already set). Jest and mocha keep --verbose.
- In both parsers, strip ANSI escape codes. When the log has vitest's
RUN vX.Y.Z or Test Files marker, record only verbose lines (✓ file > suite > test), dropping vitest's run suffixes (duration, (retry xN), (repeat xN), heap usage, skip note). Keep a project label (|unit|) as part of the name, so the same file run in two projects stays distinct.
- Accept jest's padded colored header (
FAIL file), so colored jest output parses the same as plain output.
I have a PR ready and will link it here. Mocha has a separate parsing problem, and I'll file it on its own.
Description
Repos tested with vitest can't produce
pr_runtimeorcve_patchestasks. Three things go wrong in sequence.1. The normalized command crashes.
normalize_test_cmds_for_runtime(src/repo2rlenv/pipelines/pr_runtime.pyL675-682) appends--verboseto any command containingjest|mocha|vitest. Vitest has no such flag:Reproduced on vitest 3.2.7, 4.1.11 and 5.0.1. No test runs, so pre and post both parse to
{}.2. Vitest output is colored in the sandbox. Vitest's bundled tinyrainbow enables color unless
NO_COLOR,TERM=dumborFORCE_TTY=falseis set, even when output is piped. A non-TTYdocker exechas noTERM, so every line starts with an escape code, and_JEST_TEST_RE(log_parsers/jest_parser.pyL54-56,_pr_runtime_verifier.pyL177-179) never matches. With an empty environment (env -i PATH=... HOME=...),npx vitest run --reporter=verboseparses to{}on main.3. Uncolored lines keep the duration in the name. The verbose reporter prints
✓ math.test.ts > math > adds 1ms, a bare1msrather than jest's(1 ms). So the test ismath.test.ts > math > adds 1msin one run and... adds 4msin the next, and F2P/P2P names stop matching at grading time.A trap in the obvious fix
Parsing vitest's default reporter isn't safe, even after stripping color. It expands a file only while that file has a failure, and collapses a fully passing file to one line. Real vitest 5.0.1 output for a fix to
div, where the gold patch leaves an unrelated test in the same file failing:F2P would be
rejects division by zero, and the agent's more complete patch would lose it from the log and score 0.Proposed fix
--verboseand append--reporter=verbose(unless a--reporteris already set). Jest and mocha keep--verbose.RUN vX.Y.ZorTest Filesmarker, record only verbose lines (✓ file > suite > test), dropping vitest's run suffixes (duration,(retry xN),(repeat xN), heap usage, skip note). Keep a project label (|unit|) as part of the name, so the same file run in two projects stays distinct.FAIL file), so colored jest output parses the same as plain output.I have a PR ready and will link it here. Mocha has a separate parsing problem, and I'll file it on its own.