Conversation
normalize_test_cmds_for_runtime appended `--verbose` to vitest commands, which vitest 3-5 reject with `CACError: Unknown option`, so no test ran. Its per-test output comes from `--reporter=verbose` instead. That output was not parseable either: vitest colors it whenever TERM is unset, and its bare ` 12ms` duration was left in the test name. Strip escape codes, then read vitest's fully qualified verbose lines, dropping the run suffixes and keeping project labels as part of the identity. Vitest's default reporter is ignored on purpose: it collapses a fully passing file to one line, so a patch that fixed more than the gold patch would lose its FAIL_TO_PASS tests from the log. Stripping escape codes also lets colored jest output parse, once its padded ` FAIL ` header is accepted. Closes huggingface#142
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
normalize_test_cmds_for_runtimenow gives vitest--reporter=verboseinstead of--verbose. Vitest 3-5 reject--verbosewithCACError: Unknown option, so no test ran.TERMis unset (as in a non-TTYdocker exec). In a vitest log, they read only the fully qualified verbose lines and drop the run suffixes (12ms,(retry xN),(repeat xN), heap usage, skip note). A project label stays in the name as|unit| file > suite > test, whether or not the output is colored.FAIL file) is accepted, so colored jest output parses the same as plain jest output.Results on a real vitest 5.0.1 bug fix (the
divfix from #142, captured with an empty environment):npx vitest run --verbose→CACError, no tests{}npx vitest run --reporter=verbosemath.test.ts > math > rejects division by zeroresolved=TrueTest plan
uv run --all-extras pytest -q: 1957 passed, 6 skipped (live-network tests)uv run ruff check .anduv run ruff format --check .tests/test_vitest_output.pyruns every case against both parsers. Its fixtures are verbatim runner output with local paths shortened to/repo: vitest 5.0.1 before and after the fix, the default reporter, two projects with and without color, vitest 3.2.7, and colored jest 30.5.0.normalize_test_cmds_for_runtimecase), and all pass with this change.python -I, auto-detects the runner fromnpx vitest run --reporter=verboseand grades the agent patch at reward 1.0.(retry x1),[, digits,>, escape fragments) parse in under 100 ms.Out of scope
npm testget no reporter flag, as before (flags would need to go after--), so a vitest suite behindnpm teststill uses the default reporter.✔,1) title) still doesn't parse. I'll file that separately.Closes #142