Skip to content

pytest runner: leave the agent's new test files out of the base run - #60

Open
ArjunS07 wants to merge 2 commits into
fix/regression-base-extra-argsfrom
fix/base-run-ignores-new-test-files
Open

ArjunS07 wants to merge 2 commits into
fix/regression-base-extra-argsfrom
fix/base-run-ignores-new-test-files

Conversation

@ArjunS07

Copy link
Copy Markdown
Contributor

Purpose

The regression check stashes the agent's edits and reruns the selected tests at base. git stash keeps untracked files, so a test file the agent added stays in place. At base it imports code that does not exist, collection fails, and the base run gives no results. The check then counts as not run, and with no snapshot reference the trial is graded unverified (-1.25).

Seen on uxarray#1072: both Codex trials added a test file (test/test_bounds.py, test/test_face_bounds.py), and both base runs returned no results.

Changes

  • template/tests/pytest_runner.py run_base_and_diff: untracked test_*.py / *_test.py files are passed to the base run as --ignore=<path>. Those tests do not exist at base, so they cannot be regressions.
    • Before: the base run collected them and failed.
  • tests/docker/test_pytest_runner_regression.py: test_new_test_file_from_the_agent_is_left_out_of_the_base_run; fails without the change.

Verification

  • tests/docker/test_pytest_runner_regression.py: 6 passed.

Notes

Stacked on #57 (the runner deployed in the Phase 2 task copies).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant