test.sh: discard agent edits to the measurement setup before timing and tests

[ArjunS07]

Purpose

An agent patch that edits the measurement setup (benchmark files, ASV config, conftest.py, .github/) was rejected as a whole, even when the rest of the patch was a real optimization. User decision (2026-10-09): the verifier discards those edits and measures with the task's own files.

How it works

1. test.sh captures patch.diff first. It keeps the full agent diff.
2. A new step then lists every path that differs from the starting-tree commit (/opt/fc_baseline_sha) and matches one of the protected patterns. The patterns are a copy of PROTECTED_PATCH_PATTERNS in fcrl/grading/gates.py:

   (^|/)benchmarks?/   (^|/)asv\.conf   (^|/)\.snapshots/   (^|/)conftest\.py$   (^|/)\.github/
   

3. Each path goes back to the starting tree:
   • modified, deleted or type-changed: git checkout <base> -- <path>;
   • added: removed from the index and deleted.
4. The step lists the protected paths again and writes /logs/artifacts/protected_restore.json:

   {"ran": true, "base": "<sha>", "discarded": ["..."], "remaining": []}

   If the step fails: {"ran": false, "error": "<message>"}.
5. The rebuild check, LSV timing, the confirmation pass, snapshots and pytest all run after this step, so they use the task's files. pytest also uses the task's conftest.py files.

The step does nothing without /opt/fc_baseline_sha. Against the upstream commit, an untracked benchmark suite (for example astropy-benchmarks/) would count as "added" and be deleted.

Changes

File	Effect	Before
template/tests/test.sh	Agent edits to protected paths are discarded before timing and tests; protected_restore.json records them	Protected edits stayed in the tree; the grader failed the patch

Usage

The grader in formulacode-verified-rl reads protected_restore.json:

File	Grader action
ran: true, remaining empty	No tamper failure; discarded is kept as information
ran: true, remaining not empty	Tamper failure
ran: false or no file (older tasks)	The current rule

Verification

In the astropy#14253 image, after the starting-tree commit from setup.sh, with the agent's patch plus test edits (new .github/new.yml, modified root conftest.py, new astropy/units/tests/conftest.py, deleted astropy-benchmarks/benchmarks/wcs.py, new benchmarks/sp ace.py):

• discarded lists all 6 protected paths, and remaining is empty.
• After the step, the diff holds only astropy/units/quantity.py, astropy/units/tests/test_quantity.py and the changelog entry.
• units.py and conftest.py have the same md5 as the starting tree, and astropy-benchmarks/benchmarks/ is complete (17 entries).
• With no agent changes: {"ran": true, "discarded": [], "remaining": []}.
• tests/docker/test_template_skip_flags.py and test_pytest_runner_regression.py: 22 passed.

Notes

• The pattern list exists twice, here and in fcrl/grading/gates.py. A change to one needs the same change to the other.