Skip to content

cargo parser drops should_panic tests and doctests #138

Description

@k21993

Description

src/repo2rlenv/log_parsers/cargo_parser.py (L40-42) and its standalone copy in src/repo2rlenv/pipelines/_pr_runtime_verifier.py (L142) read the test name as \S+. libtest writes test {name} - {test_mode} ... for #[should_panic], compile_fail and no_run tests, and doctest names contain spaces (write_test_name, TestDesc::test_mode):

test tests::panics_on_zero - should panic ... ok
test src/lib.rs - add (line 30) - compile fail ... ok
test src/lib.rs - add (line 34) - compile ... ok
test src/lib.rs - math::div (line 10) ... ok

None of these lines match, so the tests never reach the status map. On a real cargo test --no-fail-fast run (rustc 1.98.1) that covers every test mode, main records 6 of 18 results. Every should panic test and every doctest is missing.

Impact

  • A fix whose new test is #[should_panic] gets no FAIL_TO_PASS test. Validation ends with "no fail-to-pass tests after validation" (pr_runtime_validate.py L301), and the candidate is dropped. This affects pr_runtime, commit_runtime and cve_patches.
  • Doctests never contribute to F2P or P2P.

Reproduction

A small crate with tests/div.rs:

#[test]
fn divides() { assert_eq!(scenario::div(6, 3), 2); }

#[test]
#[should_panic(expected = "divide by zero")]
fn rejects_zero() { scenario::div(1, 0); }

At base, div returns 0 for a zero divisor. The gold patch makes it panic!("divide by zero"). Real logs:

# pre (base + tests)
test divides ... ok
test rejects_zero - should panic ... FAILED
test src/lib.rs - div (line 3) ... ok

# post (gold patch)
test divides ... ok
test rejects_zero - should panic ... ok
test src/lib.rs - div (line 3) ... ok

On main, F2P is [] and P2P is ["divides"], so the task is dropped.

A trap in the obvious fix

Allowing spaces in names isn't enough, because doctest names carry (line N). That number moves whenever a patch edits code above the doctest. With names kept verbatim, P2P becomes ["divides", "src/lib.rs - div (line 3)"].

A correct agent patch that adds a helper function above div produces test src/lib.rs - div (line 9) ... ok. Given that real run, the verifier returns reward=0.5, resolved=False, and lists the doctest as a regression.

Proposed fix

  • Strip the mode suffix ( - should panic, - compile fail, - compile). The identity is the test path, tests::panics_on_zero.
  • Key doctests by file and item without the line number (src/lib.rs - math::div, or src/lib.rs for crate-level docs). When an item has several doctests, the worst status wins, so a line shift can't change the identity.
  • Make the same change in both copies, with tests that run against both parsers (the same approach as Fix pytest parsing of parametrized node IDs containing spaces #91).

I have a PR ready and will link it here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions