[ArjunS07]

Purpose

The task template that built and measured the Phase 1 survey was on unmerged branches, so main does not build the same tasks. main also pins an lsv commit (cab3cc8) that is not on lsv main. This PR puts the whole template on one branch, so main can take it in one merge. New task builds use the tag template-v1.4 on this branch.

How it works

release/template-v1 starts at survey/phase1-template-20261008 (head of #56). Then main and the open template PRs were merged into it in stack order, as local merges. None of those PRs was merged on GitHub.

graph LR
  main --> R["release/template-v1"]
  S["survey/phase1-template-20261008"] --> R
  P62["#57, #60, #61, #62"] --> R
  P58["#58, #59, #64"] --> R
  P34["#30, #34"] --> R

Loading

Changes

1. PRs that this branch contains. Each PR keeps its own description. All of them are closed now, with a comment that points to this PR.

PR	Change
#32, #50, #52, #53, #54, #55, #56	Merged on GitHub into the template branches: git history removal, base file restore, missing benchmark dependencies, confirmation pass, skip flags, --only
#40, #41, #42, #43	In the survey branch: tests for changes to compiled sources only, sqlite3 before pandas, repo package before site-packages, check for a shadowed project
#57, #60, #61, #62	pytest runner: the base run uses the agent's arguments, leaves out new test files, uses a --basetemp in the container, and compares committed changes against the base commit
#58, #59	Prompt and verifier: edits to the measurement setup are discarded before timing
#64	LSV_REF → lsv 9702a48 (lsv#10: copy sys.modules before the extension scan), then lsv 909ec2f in change 5
#30, #34	parser.py marks reward.txt as not used for training, and the paired timing merge

2. Env image build uses the pinned lsv and snapshot-tester (tag template-v1).

   • docker/templates/docker_build_final.sh installs lsv and snapshot-tester at the same commits as LSV_REF and SNAPSHOT_TESTER_REF.
   • Effect: environment images and task images use the same lsv.
   • Before: the installs used the latest commit of each repo.

3. Gate lease id in the log (tag template-v1.1).

   • setup.sh and test.sh write each acquire to measure_gate.txt as <step> <lease id> 1|0.
   • Effect: a stopped job can release its gate leases.
   • Before: each acquire was written as <step> 1|0, so the leases of a stopped job stayed until their time limit.

4. Shorter comments (tag template-v1.2).

   • Comments in parser.py, lsv_measure.py and records.py are 1 or 2 lines.
   • Effect: no change in behavior.

5. LSV_REF → lsv 909ec2f (tag template-v1.3).

   • Effect: task images install lsv with lightspeed: a failing teardown does not stop the survey lsv#7, so a failing benchmark teardown after a failing benchmark body does not stop the coverage survey.
   • Before: template-v1.2 pinned lsv 9702a48, which stopped the survey in that case. Between the two lsv commits only survey.py (2 lines) and its test change.

6. Measure gate is a lock folder of slot files, not the HTTP gate (tag template-v1.4).

   • The host mounts a folder with one empty file per slot into the trial. The number of files is the number of slots.
   • tests/measure_lock.sh takes a free slot with flock -n and tries again every 2 seconds. setup.sh and test.sh load it with source.
   • The slot is held around the same steps as before: lsv_init (init), lsv_measure (measure), the confirmation pass (confirm), and snapshots plus pytest (tests).
   • mg_release calls flock -u before it closes the file descriptor, so a process that the step left running does not keep the slot. The kernel also frees the slot when the trial process stops.
   • Settings: FC_MEASURE_LOCK_DIR (default /fc-measure-lock) and FC_MEASURE_LOCK_WAIT in seconds (default 7200, the same limit as the old acquire).
   • Each step writes <step> <slot file name> 1 to measure_gate.txt. A folder without slot files, or a wait past the limit, writes <step> none 0 at once or after the wait, so the readers drop the trial as ungated as before.
   • Effect: trials need no network access to the host and no token. A stopped job leaves no lease behind.
   • Before: curl called MEASURE_GATE_URL to acquire and release a lease, and a stopped job kept its leases until their time limit.

7. FC_SKIP_LSV flag in test.sh (tag template-v1.4).

   • It is read from the incoming environment with FC_SKIP_PYTEST and FC_SKIP_SNAPSHOT, before the profile.d files are sourced.
   • When it is 1, test.sh does not run lsv_measure or the confirmation pass, and writes lsv_skipped.json.
   • parser.py then writes reward.json with lsv_skipped set, no speedups, and the pytest and snapshot results.
   • Effect: a correctness trial does not spend time on timing.
   • Before: each trial timed the patch.

8. Precheck fields in the image build check (tag template-v1.4).

   • apply_oracle_patch.py adds by_deps, always_affected, edited_before and edited_overlap to patch_info.json, and emit_measure.py copies them into manifest["verify"].
   • The definitions are the same as in verify/in_image/coverage_check.py on measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66. The hunk header parser (old_ranges, overlapping) moved in unchanged.
   • by_deps counts the benchmarks that LSV selects from the deps database of lsv_init, through a file that the patch changes, without the always affected benchmarks. A separate git index of the starting tree keeps the image's own edits out of that file list.
   • Effect: the image build check gives the same coverage facts as the precheck, without a second container run.
   • Before: only the precheck on measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 computed these fields.

9. LSV_REF → lsv 6858286 (tag template-v1.4).

   • lightspeed: a changed file that is not source is hashed, not parsed lsv#12 is merged. A changed file that is not source code (for example a *.prof dump that an agent leaves in the repo) is hashed, not parsed.
   • Effect: such a file does not stop the timing with "invalid or missing encoding declaration".
   • Before: lsv 909ec2f raised SyntaxError on such a file. Between the two lsv commits only process_code.py (7 lines) and a new test change.

Not included:

• task template: skip the timing of a patch that the host tamper gate rejects #39 (skip timing when the tamper check rejects a patch). It is closed as obsolete. test.sh: discard agent edits to the measurement setup before timing and tests #59 restores the protected measurement files before timing, so no protected edit remains for task template: skip the timing of a patch that the host tamper gate rejects #39 to find.
• Migration 00033: oracle result columns on formulacode_task_overrides #29, task template: rebuild compiled code after a patch, remove benchmark temp folders, log measure gate acquires #35, task template: register the asv machine at trial start so asv run works without --machine #36, task template: task env on PATH, no oracle diff in the image, prompt commands that work #37, task template: diff the agent's work against a commit of the image's starting tree #38: they are closed, because main already has them.

Usage

Render and build from the latest tag:

git -C datasmith checkout template-v1.4
fc-render-tasks render --records RECORDS.jsonl --out TASKS

Verification

Run again on the current head (0e244db, tag template-v1.2):

$ pytest -q tests --ignore=tests/integration
1809 passed, 11 skipped in 507.11s

• The same command gave 1809 passed, 11 skipped in 1400.81s on the commit tagged template-v1.
• environment/Dockerfile pins LSV_REF=909ec2f and SNAPSHOT_TESTER_REF=f08bb0c. The env image build script uses the same commits.

After the LSV_REF change (head 3f4bc00, tag template-v1.3): pytest -q tests/docker: 453 passed, 10 skipped.

On head 4bf55df (tag template-v1.4):

$ pytest -q tests/docker
475 passed, 10 skipped in 54.55s
$ pytest -q tests/test_overrides_wiring.py tests/agents/test_sandbox.py
87 passed
$ uv run --python 3.8 --with pytest python -m pytest -q --noconftest tests/docker/test_apply_oracle_patch.py tests/docker/test_emit_measure.py
42 passed

• New tests: test_measure_lock.py (slot held and freed, a second trial takes the other slot, all slots busy waits then runs ungated, a missing or empty folder logs ungated at once). The FC_SKIP_LSV tests are in test_template_skip_flags.py. The precheck field tests repeat the cases of tests/verify/test_precheck.py from measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66.
• lightspeed: a changed file that is not source is hashed, not parsed lsv#12: pytest test/test_lightspeed_survey.py test/test_lightspeed_binary_file.py gave 5 passed. The lsv CI fails on main too, in upstream asv tests.

Notes

• Merge order: this PR first, then task template: restore test files before tests, and run the changed package's tests folder #68 and measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66, then docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67.
  • task template: restore test files before tests, and run the changed package's tests folder #68 and measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 are based on this branch.
  • docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67 is based on measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66.
• The tags template-v1 to template-v1.4 point at commits on this branch. They stay valid after a merge.
• git merge-tree of template-v1.4 with measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 and task template: restore test files before tests, and run the changed package's tests folder #68 gives no conflicts.
• The measure job and RL training still pass MEASURE_GATE_URL. They must mount the lock folder at /fc-measure-lock instead. That change comes in a later PR.