Skip to content

log_parsers: keep cargo should_panic and doctest results - #139

Merged
adithya-s-k merged 1 commit into
huggingface:mainfrom
k21993:fix/cargo-test-mode-names
Sep 17, 2026
Merged

adithya-s-k merged 1 commit into
huggingface:mainfrom
k21993:fix/cargo-test-mode-names

Conversation

@k21993

@k21993 k21993 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Parse libtest result lines whose names contain a test mode or spaces (tests::x - should panic, src/lib.rs - add (line 30) - compile fail), in both log_parsers/cargo_parser.py and the standalone copy in _pr_runtime_verifier.py. On main, these lines don't match, so a fix whose new test is #[should_panic] has no FAIL_TO_PASS test and gets dropped.
  • Key doctests by file and item without (line N), and let the worst status among an item's doctests win. The line number moves whenever a patch edits code above a doctest, so a verbatim key would score a correct patch that shifts lines as a regression.
  • Match libtest's literal ... separator instead of \s+\.\.\.\s+, so names can contain spaces without backtracking on long lines.

The reproduction from #138 uses real cargo test output. The reference fix keeps the doctest at (line 3). The agent's patch is equally correct but adds a helper above div, which moves the doctest to (line 9).

Parser F2P P2P Reference fix graded Agent patch graded
main [] divides task dropped task dropped
spaces allowed, names verbatim rejects_zero divides, src/lib.rs - div (line 3) 1.0 0.5, doctest reported as a regression
this PR rejects_zero divides, src/lib.rs - div 1.0 1.0

Test plan

  • uv run --all-extras pytest -q: 1912 passed, 6 skipped (live-network tests)
  • uv run ruff check . and uv run ruff format --check .
  • New tests/test_cargo_test_names.py runs every case against both parsers, the same way tests/test_pytest_node_ids.py does. Its fixtures are verbatim cargo test --no-fail-fast output from rustc 1.98.1: one crate covering every test mode, plus the pre-fix, reference-fix and agent runs above.
  • 25 of the 40 new tests fail on main, and all 40 pass with this change.
  • The standalone verifier, run with python -I, grades the shifted-doctest agent run at reward 1.0 with resolved=True.
  • Four 200k-character pathological lines parse in a few milliseconds, with a subprocess timeout guard like the one in Fix pytest parsing of parametrized node IDs containing spaces #91.

Out of scope

  • Tests with the same name in different crates or test binaries still collide, and the last write wins, as before.
  • cargo nextest, --format terse or JSON output, and ANSI-colored logs.
  • Tasks generated before this change keep the verifier baked into them.

Closes #138

libtest writes `test {name} - {mode} ... ` for should_panic, compile_fail
and no_run tests, and doctest names contain spaces. The `\S+` name pattern
matched none of them, so a fix whose new test is #[should_panic] had no
FAIL_TO_PASS test and was dropped at validation.

Doctests are keyed by file and item without `(line N)`: the line moves
whenever a patch edits code above the doctest, so a verbatim key made a
correct agent patch that shifted lines score as a P2P regression. The
worst status among an item's doctests wins.

Both the canonical parser and the standalone verifier copy change, and
the new tests run against both using real rustc 1.98.1 output.

Closes huggingface#138
@adithya-s-k
adithya-s-k merged commit 4422c1c into huggingface:main Sep 17, 2026
11 checks passed
@adithya-s-k

Copy link
Copy Markdown
Collaborator

Thanks, tested and merged. The should_panic and shifted-doctest cases now work in both parsers. Existing Cargo tasks will need their verifier and test IDs updated together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cargo parser drops should_panic tests and doctests

2 participants