[ArjunS07]

Purpose

The stacked PRs #66, #67 and #68 were merged into their base branches after those bases were already merged, so their changes did not reach main. Only #33 and #65 are on main. This PR brings the rest to main in one merge. It also adds four small fixes and the template tag template-v1.5.

How it works

graph LR
  V["verify/core<br/>#66 + #67"] -->|merge 791fa89| R["release/template-v1<br/>#65 + #68 + #66"]
  R -->|this PR| M[main]

Loading

Changes

1. The changes of measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66, docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67 and task template: restore test files before tests, and run the changed package's tests folder #68 on main.
   • Effect: main gets the measurement/ package and fc-measure, fc-build-tasks, RenderShortProblemsRunner as the default render, the measure time limit, the docs overview, and the test file protection in the grader.
   • Before: these changes were only on release/template-v1 (measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66, task template: restore test files before tests, and run the changed package's tests folder #68) and verify/core (docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67).
2. One merge commit, 791fa89, joins verify/core into release/template-v1. It has no conflicts and no other change.
3. The test test_a_test_file_that_no_longer_imports_is_a_regression runs from the repo folder (9579842).
   • Effect: the test passes. It calls run_pytest_and_collect in the test process, so it now changes the working folder to the repo first, as test.sh does with cd /workspace/repo.
   • Before: the test failed. pytest 8 sets its root folder to the common parent of the working folder and the test paths when no config file is found. That parent was /tmp, so the node ids had the temporary folder in front of them.
   • Task containers are not affected, because test.sh runs the runner from /workspace/repo and the base and wider runs use cwd=repo_root.
4. pre-commit run --all-files formatting (fc6ac81).
   • Effect: the "quality" CI job passes. base_restore.json gets the standard JSON indent, ruff sorts one import in resolution/orchestrator.py, and ruff format changes 8 test files in tests/docker/.
   • Before: pre-commit changed these 10 files, so the job failed.
   • No file that goes into a task image changed.
5. scrub_git.sh removes nested .git folders (de89d89).
   • Effect: the image build deletes .git folders up to 4 levels under /workspace, /tmp and /opt, except the repo's own .git. These are benchmark repos cloned at their latest commit, for example /tmp/benchmarks for dask. The build fails if one remains.
   • Before: the deleted verify/scrub.py did this, but scrub_git.sh did not.
   • Extra arguments after the repo replace the 3 folders, so the test does not search the real /tmp.
6. The apply_oracle_patch.py docstring no longer names the deleted datasmith.verify.in_image.coverage_check (f153113).

Verification

• git merge-base --is-ancestor with the merge commits of measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 (5d7ebe7), docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67 (18aaa8b) and task template: restore test files before tests, and run the changed package's tests folder #68 (4d1aa7d): all three are in this branch.
• git merge-tree with main: no conflicts.
• PYTHONPATH=src pytest -q -m "not slow" tests on f153113 with pytest 8.4.2: 1833 passed, 0 failed, 10 skipped, 39 deselected.
  • Before 9579842 the same run gave 1832 passed and 1 failed (test_a_test_file_that_no_longer_imports_is_a_regression). It also failed on task template: restore test files before tests, and run the changed package's tests folder #68's own head (d3b18e6).
• tests/docker/test_scrub_git.py with a nested benchmark repo: fails without the scrub_git.sh change and passes with it.
• pre-commit run --all-files: no hook fails and no file changes.

Notes

• Merge this PR with a merge commit, not a squash, so the template tags (template-v1 to template-v1.5) stay on the history of main.
• The tag template-v1.5 is on f153113, the head of this PR. It has task template: restore test files before tests, and run the changed package's tests folder #68, the nested .git removal and the formatting. New task images need a new render with this tag.