You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
src/repo2rlenv/log_parsers/cargo_parser.py (L40-42) and its standalone copy in src/repo2rlenv/pipelines/_pr_runtime_verifier.py (L142) read the test name as \S+. libtest writes test {name} - {test_mode} ... for #[should_panic], compile_fail and no_run tests, and doctest names contain spaces (write_test_name, TestDesc::test_mode):
test tests::panics_on_zero - should panic ... ok
test src/lib.rs - add (line 30) - compile fail ... ok
test src/lib.rs - add (line 34) - compile ... ok
test src/lib.rs - math::div (line 10) ... ok
None of these lines match, so the tests never reach the status map. On a real cargo test --no-fail-fast run (rustc 1.98.1) that covers every test mode, main records 6 of 18 results. Every should panic test and every doctest is missing.
Impact
A fix whose new test is #[should_panic] gets no FAIL_TO_PASS test. Validation ends with "no fail-to-pass tests after validation" (pr_runtime_validate.py L301), and the candidate is dropped. This affects pr_runtime, commit_runtime and cve_patches.
Doctests never contribute to F2P or P2P.
Reproduction
A small crate with tests/div.rs:
#[test]fndivides(){assert_eq!(scenario::div(6,3),2);}#[test]#[should_panic(expected = "divide by zero")]fnrejects_zero(){ scenario::div(1,0);}
At base, div returns 0 for a zero divisor. The gold patch makes it panic!("divide by zero"). Real logs:
# pre (base + tests)
test divides ... ok
test rejects_zero - should panic ... FAILED
test src/lib.rs - div (line 3) ... ok
# post (gold patch)
test divides ... ok
test rejects_zero - should panic ... ok
test src/lib.rs - div (line 3) ... ok
On main, F2P is [] and P2P is ["divides"], so the task is dropped.
A trap in the obvious fix
Allowing spaces in names isn't enough, because doctest names carry (line N). That number moves whenever a patch edits code above the doctest. With names kept verbatim, P2P becomes ["divides", "src/lib.rs - div (line 3)"].
A correct agent patch that adds a helper function above div produces test src/lib.rs - div (line 9) ... ok. Given that real run, the verifier returns reward=0.5, resolved=False, and lists the doctest as a regression.
Proposed fix
Strip the mode suffix ( - should panic, - compile fail, - compile). The identity is the test path, tests::panics_on_zero.
Key doctests by file and item without the line number (src/lib.rs - math::div, or src/lib.rs for crate-level docs). When an item has several doctests, the worst status wins, so a line shift can't change the identity.
Description
src/repo2rlenv/log_parsers/cargo_parser.py(L40-42) and its standalone copy insrc/repo2rlenv/pipelines/_pr_runtime_verifier.py(L142) read the test name as\S+. libtest writestest {name} - {test_mode} ...for#[should_panic],compile_failandno_runtests, and doctest names contain spaces (write_test_name,TestDesc::test_mode):None of these lines match, so the tests never reach the status map. On a real
cargo test --no-fail-fastrun (rustc 1.98.1) that covers every test mode, main records 6 of 18 results. Everyshould panictest and every doctest is missing.Impact
#[should_panic]gets no FAIL_TO_PASS test. Validation ends with "no fail-to-pass tests after validation" (pr_runtime_validate.pyL301), and the candidate is dropped. This affectspr_runtime,commit_runtimeandcve_patches.Reproduction
A small crate with
tests/div.rs:At base,
divreturns 0 for a zero divisor. The gold patch makes itpanic!("divide by zero"). Real logs:On main, F2P is
[]and P2P is["divides"], so the task is dropped.A trap in the obvious fix
Allowing spaces in names isn't enough, because doctest names carry
(line N). That number moves whenever a patch edits code above the doctest. With names kept verbatim, P2P becomes["divides", "src/lib.rs - div (line 3)"].A correct agent patch that adds a helper function above
divproducestest src/lib.rs - div (line 9) ... ok. Given that real run, the verifier returnsreward=0.5,resolved=False, and lists the doctest as a regression.Proposed fix
- should panic,- compile fail,- compile). The identity is the test path,tests::panics_on_zero.src/lib.rs - math::div, orsrc/lib.rsfor crate-level docs). When an item has several doctests, the worst status wins, so a line shift can't change the identity.I have a PR ready and will link it here.