measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default by ArjunS07 · Pull Request #66 · formula-code/datasmith

ArjunS07 · 2026-10-19T03:58:01Z

Purpose

The first version of this PR put the task approval steps in one package, verify/, with a command for each step. Review found that the package mixed unrelated concerns, kept utilities as main modules, edited rendered prompts after the fact and ran an HTTP service on the host. This version moves each part to the package of its domain, deletes the survey workflow (formulacode-verified-rl keeps it) and replaces the HTTP measure gate with a lock folder. The short prompt is now a runner class and the pipeline default.

How it works

New and changed files:

src/datasmith/
  measurement/                 new package: timing of tasks
    noise.py                   per-benchmark noise rule (moved, logic unchanged)
    run.py                     measure job: oracle and no-change runs per task, lock folder, no-change snapshot check
    precheck.py                decide() over the image build check fields
    agents.py                  NopTouch harbor agent (moved)
    cli.py                     fc-measure precheck | measure
  docker/
    images.py                  + task_image_tag(), ImageManager.build_task_image()
    manifest.py                + check_task_image()
  harbor_adapter/
    utils.py                   + render_task_toml(docker_image=), patch_files()
    stamp.py                   + env_digest()
    regen.py                   + export --from-db, render --images
  runners/
    build_task_images.py       new: BuildTaskImagesRunner(BaseRunner), fc-build-tasks
    render_problems.py         + RenderShortProblemsRunner(RenderProblemsRunner)
    harbor_healthcheck.py      + snapshots_dir (keeps the oracle snapshot reference)
  verify/                      deleted

graph LR
  export["fc-render-tasks export --from-db"] --> render["fc-render-tasks render"] --> build["fc-build-tasks"] --> pin["fc-render-tasks render --images"] --> precheck["fc-measure precheck"] --> measure["fc-measure measure"]

Lock folder in place of the measure gate.

The host has one folder with one empty file per timing slot. FC_MEASURE_LOCK_DIR sets the folder (default /tmp/fc-measure-lock).
fc-measure measure makes the folder with FC_MEASURE_LOCK_SLOTS files (default 9, the slot count of the old gate) when it is empty.
Each trial gets the folder as a read-only bind mount at /fc-measure-lock.
The template (tests/measure_lock.sh, in task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65) takes a slot with flock for each timing step and logs <step> <slot> 1 or <step> none 0 to measure_gate.txt.
A trial with a 0 line, or with no log, counts as an infrastructure failure, so the job runs again.

Where each former verify/ module went, and why.

Former module	Now	Why
`noise.py`	`measurement/noise.py`	The noise rule is part of timing. The one-line repo extraction replaces the import from `survey.py`.
`agents.py`	`measurement/agents.py`	NopTouch is the no-change run of the measure job.
`measure.py`	`measurement/run.py`	The HTTP gate code is gone (gate URL, token, timeouts, lease release, the `release-leases` command). `result.json` loads with harbor's `TrialResult` model. Functions take paths, not a `Survey` object.
`services/measure_gate.py`	deleted	Trials take a slot file of a host folder, so no service runs on the host. This works the same on an AWS host.
`capture.py`	`runners/harbor_healthcheck.py` (`snapshots_dir`) and `measurement/run.snapshot_mismatch`	The healthcheck already runs the oracle and stores its snapshots. The no-change check loads the snapshot summary of the measure job's no-change runs. The rewrite of the rendered `test.sh` and the task copy are gone.
`precheck.py`, `in_image/coverage_check.py`	`measurement/precheck.py`	Template-v1.4 writes `by_deps`, `always_affected`, `edited_before` and `edited_overlap` in the image build check. `precheck()` loads them from `candidate_containers.build_manifest` and `decide()` compares them.
`images.py`	`docker/images.py`, `docker/manifest.py`, `runners/build_task_images.py`	Image code goes in `docker/` and a pipeline step goes in `runners/`. Duplicate helpers are deleted (shell call, image check, disk checks, JSON fetch).
`pin.py`	`harbor_adapter/utils.render_task_toml(docker_image=)`, `regen render --images`	The tag goes into `task.toml` when the task renders. No tool edits `task.toml` after the render.
`scrub.py`	deleted	The template's `scrub_git.sh` runs at the end of every image build (#50).
`env_image.py`	deleted	`SynthesizeImagesRunner` already builds the environment image at `base_sha`.
`registry.py`	deleted	The ECR registry was a temporary store for the survey. formulacode-verified-rl keeps `aws/ecr_push.py`. Publishing is a step of `fc-data`.
`selection.py`	`regen export --from-db`	Records come from `records.to_record` and `utils.db.fetch_all`. The sampling policy stays in formulacode-verified-rl.
`prompts.py`, `prompt_check.py`, `short_prompt.py`	`runners/render_problems.RenderShortProblemsRunner`	The prompt is made short when it renders. No tool edits a rendered `instruction.md`. formulacode-verified-rl keeps `apply_gold_prompt.py`.
`review.py`, `survey.py`, `report.py`, `release.py`, `status.py`	deleted	This is survey workflow, not library code. formulacode-verified-rl keeps its copies.
`cli.py`	`measurement/cli.py` (`fc-measure`), `runners/build_task_images.main` (`fc-build-tasks`)	Each CLI parses arguments and calls its package.

Changes

Measurement package.
- Effect: fc-measure measure TASKS_ROOT OUT TASK ... writes OUT/oracle/<task>.json with 3 oracle runs and 3 no-change runs per task. Each run also records snapshot_mismatch, which is the count of snapshot entries that pass or skip in the oracle reference and fail in that run.
- Effect: the job config is JSON, so the module does not need yaml.
- Before: fc-verify measure SURVEY TASK loaded and wrote the survey state files and needed the HTTP gate.
Lock folder.
- Effect: the measure job mounts the host lock folder at /fc-measure-lock.
- Before: trials called the gate service at MEASURE_GATE_URL with a token from a secret file.
Precheck from the image build check.
- Effect: fc-measure precheck TASKS_ROOT TASK ... loads the measure block from the database and returns base_edited, patch_failed, uncovered or nothing.
- Effect: an image built before template-v1.4 has no by_deps, so its task is measured.
- Before: a docker run of coverage_check.py in the task image, with the result cached in coverage/<task>.json.
Oracle snapshot reference.
- Effect: run_harbor_healthcheck(..., snapshots_dir=DIR) also copies each oracle reference with at least one entry that is not a skip to DIR/<task>/.
- Before: capture ran 2 extra harbor jobs per task on a copy of the task with a rewritten test.sh.
Task image builds.
- Effect: fc-build-tasks TASKS_ROOT OUT builds each task with BuildTaskImagesRunner, so it gets the progress rows and the end summary of BaseRunner. A verified image stays on the Docker daemon, and an image that fails its check is removed.
- Effect: the disk limits use the free space of Docker's data folder (utils.docker_prune.docker_root, now public).
- Before: fc-verify build also saved each image as a zstd archive and checked the free space of /.
Image pin at render time.
- Effect: fc-render-tasks render --images OUT/manifest.jsonl writes docker_image in task.toml when the last verified row of the task has the same environment/ digest.
- Before: fc-verify pins and publish edited task.toml after the render.
Records from the database.
- Effect: fc-render-tasks export --from-db --ids-file IDS --out FILE writes render records for new tasks.
- Before: fc-verify records and select in selection.py.
Short prompts.
- Effect: RenderShortProblemsRunner passes no linked issues to the render. It does not store a problem text that names ASV benchmarks, a PR or issue link, code or a diff, or an internal function or class that the patch changes. It raises PromptLeak, which BaseRunner counts.
- Effect: fc-data uses RenderShortProblemsRunner for the render_problems stage and selects patch for it.
- Before: FC_PROMPT_VERSION=short|detailed in RenderProblemsRunner, and prompt-check and prompt commands that edited rendered tasks.
- The extractor instruction change in agents/extractors.py stays.
Removed from the first version.
- fc-verify and its entry point.
- The lazy dotenv import in datasmith/__init__.py, because no command needs to import with the standard library only.
- oracle_tests_rerun stays in noise.verify. The empty problem text fix goes with prompts.py.
- The harbor pin to e7d19622 and uv.lock stay.

Usage

fc-render-tasks export --from-db --ids-file ids.md --out records.jsonl
fc-render-tasks render --records records.jsonl --out TASKS
fc-build-tasks TASKS IMAGES --ids-file ids.md --workers 6
fc-render-tasks render --records records.jsonl --out TASKS --force --images IMAGES/manifest.jsonl
fc-measure precheck TASKS owner/repo#N ...
FC_MEASURE_LOCK_DIR=/tmp/fc-measure-lock fc-measure measure TASKS OUT owner/repo#N ... --workers 2

RL trials on the same host mount the same folder (harbor mounts_json):

{"type": "bind", "source": "${FC_MEASURE_LOCK_DIR}", "target": "/fc-measure-lock", "read_only": true}

Verification

On the head 13b98a0:

$ PYTHONPATH=src pytest -q -m "not slow" tests
1819 passed, 10 skipped, 39 deselected
$ PYTHONPATH=src pytest -q -m "not slow" tests/measurement tests/docker tests/runners tests/agents
1060 passed, 10 skipped, 6 deselected

Before this restructure, the same full run gave 1863 passed. The difference is the tests of the deleted modules.
ruff check and ruff format pass on the changed files. The 2 S310 findings in harbor_healthcheck.py are on the base too.
Size against release/template-v1:
- src: +3555 lines before, +1126 lines now.
- tests: +1003 lines before, +403 lines now.
Noise rule on the stored Phase 1 survey (413 records): classify_all and verify give the same output from measurement.noise as from the old verify.noise.
trial_run on 354 trials of 60 random Phase 1 records:
- All 354 result.json files load with TrialResult.
- Role, run, speedups, tests, patch failures and timing are the same as the old trial_run on all 354.
- The gate flag differs on 210 trials. All 210 have no measure_gate.txt (an older template), so they now count as trials without a lock slot.
flock on a file in a read-only bind mount takes the lock inside a container (docker run -v DIR:/l:ro).

Notes

This PR is based on task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65 (template-v1.4: lock folder, FC_SKIP_LSV, precheck fields). Merge task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65 first.
formula-code/formulacode-verified-rl#184 and formula-code/formulacode-verified-rl#186 import datasmith.verify. They must change next, to datasmith.measurement, datasmith.docker and datasmith.harbor_adapter, or keep their own copies.
skyrl RL trials must mount the lock folder at /fc-measure-lock. Without the mount, measure_lock.sh runs each timing step without a slot.
The render source digest changes, because harbor_adapter/utils.py and adapter.py change. fc-render-tasks check reports older renders as changed. The environment/ digest and the image tags do not change.
Not done:
- scrub_git.sh does not remove nested .git folders, which scrub.py did. This is a template change, so it needs the next template version on release/template-v1.
- The confirmation spec still goes to the task as tests/confirm.json in a copy of the task, because the template loads it only from that file.
- The measure job and the healthcheck do not share harbor job code. The healthcheck runs harbor in the process, and the measure job runs the harbor command with a log file and without secret variables. A shared helper would not be smaller.
- --max-heavy is gone, because it used the old coverage/<task>.json files.
docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67 is based on this PR and has conflicts with it: utils/docker_prune.py, runners/harbor_healthcheck.py, and 10 files under verify/ that this PR deletes. Its measure --time-limit and prompt changes are in verify/ and must move or go.

… not_approved rules

…) with the in-image check

…gs the gate lease id

…s, license, PR author, prompt audit); register measure in the CLI

…ges import with the standard library

… run it with system python

…stages

…/core

…mpt check, model drafts for review FC_PROMPT_VERSION=short leaves the linked issues out of the prompt and asks the extractor for symptoms only; fc-verify prompt-check, prompt-draft and prompt --drafts; readiness reports prompt findings.

…mulacode-verified-rl uses Unpinned, the dependency resolved to harbor's main, which is 1305 commits behind; the lock had f600c47 (February).

…/core

…rerun)

… text is empty (same as #67)

atharvas · 2026-10-19T20:53:01Z

It might be better to make a new class called RenderShortProblemsRunner and set that as the default.

atharvas · 2026-10-19T20:54:28Z

+HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? ")
+
+
+def old_ranges(diff):


Advice: use https://github.com/DietrichGebert/ponytail/ to ensure that the helper functions are not already maintained in an external package.

atharvas · 2026-10-19T20:54:36Z

+    return out
+
+
+def overlapping(image_diff, patch_text):


Advice: use https://github.com/DietrichGebert/ponytail/ to ensure that the helper functions are not already maintained in an external package.

atharvas · 2026-10-19T20:56:56Z

This file seems sloppy. I would have expected this to be part of a common set of utilities for verification. Unsure whether this will be an issue on separate EC2 instances.

atharvas · 2026-10-19T20:59:34Z

It seems like a separate src/datasmith/measurement/ module for these rules would be useful. Also, use https://github.com/DietrichGebert/ponytail/ to ensure that the helper functions are not already maintained in an external package.

atharvas · 2026-10-19T21:03:27Z

Is this still needed? Why do we need a meta-prompt editor when we can directly edit the prompt(s)? Try folding this into the upstream components of datasmith.

atharvas · 2026-10-19T21:03:50Z

Isn't the publishing a downstream step in fc-data? Why is this needed?

atharvas · 2026-10-19T21:10:32Z

I'm not sure why this belongs to the /verify module.

atharvas · 2026-10-19T21:10:52Z

Not sure why this is in /verify

Seems like many of our /verify/* sub-modules are utilities that run either on docker container startup or on tear-down.

atharvas · 2026-10-19T21:18:15Z

The internal organization of the /verify/ folder is a complete mess and extremely sloppy. Each sub-component should be redone with a https://github.com/DietrichGebert/ponytail/blob/main/skills/ponytail/SKILL.md pass. Three issues I'm generally seeing:

Utilities as main modules -- Maybe a datasmith.verify.utils.* would be better organization for these?

Many repeated functionalities that have no justification for being in the verification module -- such as meta-level prompt editing skills.

The many valid verification skills are currently grouped into: within image / perpetual services / and "others". This misc grouping (files under datasmith.verify.*) are not well motivated to be top-level modules.

…moved from verify

…elete verify/capture.py

… a slot of a host lock folder instead of the HTTP measure gate

…late-v1.4); delete the in-image coverage check

…sRunner, the image pin at render time; delete verify images, pin, registry, scrub and env_image

…giveaways, and is the pipeline default; drop FC_PROMPT_VERSION

…rt); drop the unused grade hook

…re) and fc-build-tasks expose what stays

… when it cannot be read

…mark the configured Supabase URL open as allowed

Bring #66, #67 and #68 to main (they were merged into their stacked bases)

ArjunS07 added 6 commits October 19, 2026 03:46

verify: fc-verify with status, classify, report, review and automatic…

ef70351

… not_approved rules

verify: precheck stage (patch applies, base edits, benchmark coverage…

7fcb5eb

…) with the in-image check

verify: build, pin, scrub and publish stages for task images

61f41a8

verify: measure stage, measure gate service and NopTouch; template lo…

37fb966

…gs the gate lease id

verify: test reads the job config with Path

964e046

verify: release readiness (registry, template commits, runs, CPU type…

c5f37e1

…s, license, PR author, prompt audit); register measure in the CLI

ArjunS07 changed the title ~~fc-verify: review, build, publish and precheck stages~~ fc-verify: review, build, publish, precheck, measure and readiness stages Oct 19, 2026

ArjunS07 added 13 commits October 19, 2026 05:11

verify: snapshot capture stage

3d3c26b

verify: port the build guard, disk stop and image check tests

2ed3f64

verify: PR number from the image name too

493750c

verify: an empty pin reads as no pin

6759111

datasmith: load dotenv only when tokens.env exists, so the verify sta…

08dca8a

…ges import with the standard library

verify: the command imports with the standard library, so build hosts…

f76f520

… run it with system python

verify: push verified manifest rows to the registry as they are

c8ace12

verify: select (records, database candidates, sampler) and env-image …

a2faa02

…stages

verify: port the sampler order and replacement tests

764896c

Merge remote-tracking branch 'origin/release/template-v1' into verify…

7887c21

…/core

verify: apply approved prompt text to rendered tasks

fe662ba

Revert an unrelated lint change to harbor_healthcheck.py

38c2048

ArjunS07 changed the title ~~fc-verify: review, build, publish, precheck, measure and readiness stages~~ fc-verify: the task pipeline in datasmith (select to readiness), short prompts by default Oct 19, 2026

ArjunS07 added 2 commits October 19, 2026 06:02

Pin harbor to formulacode/harbor-reliability e7d19622, the commit for…

9f97aa9

…mulacode-verified-rl uses Unpinned, the dependency resolved to harbor's main, which is 1305 commits behind; the lock had f600c47 (February).

Merge remote-tracking branch 'origin/release/template-v1' into verify…

d10e6a6

…/core

This was referenced Oct 19, 2026

docs: pipeline overview, stale docs deleted, shorter comments; fc-measure: measure time limit #67

Merged

task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65

Merged

ArjunS07 added 3 commits October 19, 2026 20:09

verify: count oracle test failures from the test rerun (oracle_tests_…

fa3ae71

…rerun)

prompts: rewrite handles an empty problem text

8f0db18

prompts: keep the blank line before the next heading when the problem…

5d56364

… text is empty (same as #67)

atharvas reviewed Oct 19, 2026

View reviewed changes

Merge origin/release/template-v1 (template-v1.4) into verify/core

498c2bd

ArjunS07 added 9 commits October 19, 2026 22:17

measurement: new package with the noise rule and the NopTouch agent, …

3964b62

…moved from verify

harbor healthcheck: keep the oracle snapshot reference in a folder; d…

1963ea7

…elete verify/capture.py

measurement: the measure job moves to measurement/run.py; trials take…

f80f61e

… a slot of a host lock folder instead of the HTTP measure gate

measurement: precheck decides from the image build check fields (temp…

c4dbcf6

…late-v1.4); delete the in-image coverage check

images: task image build in docker/, the build loop as BuildTaskImage…

2f98e4e

…sRunner, the image pin at render time; delete verify images, pin, registry, scrub and env_image

prompts: RenderShortProblemsRunner renders the symptom only, refuses …

6dce9d6

…giveaways, and is the pipeline default; drop FC_PROMPT_VERSION

measurement: write the harbor job config as JSON (drops the yaml impo…

6cf5990

…rt); drop the unused grade hook

verify: delete the package and fc-verify; fc-measure (precheck, measu…

7f5dda7

…re) and fc-build-tasks expose what stays

build_task_images: look up Docker's data dir once, and fall back to /…

13b98a0

… when it cannot be read

ArjunS07 changed the title ~~fc-verify: the task pipeline in datasmith (select to readiness), short prompts by default~~ measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default Oct 19, 2026

measurement, harbor_adapter: type annotations for mypy; ruff format; …

7854274

…mark the configured Supabase URL open as allowed

ArjunS07 merged commit 5d7ebe7 into release/template-v1 Oct 20, 2026
2 of 3 checks passed

ArjunS07 mentioned this pull request Oct 20, 2026

Bring #66, #67 and #68 to main (they were merged into their stacked bases) #69

Merged

ArjunS07 added a commit that referenced this pull request Oct 20, 2026

Merge pull request #69 from formula-code/release/template-v1

843dc26

Bring #66, #67 and #68 to main (they were merged into their stacked bases)

Navigation Menu

Repository navigation

measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default - #66

measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default#66
ArjunS07 merged 35 commits into
release/template-v1from
verify/core

ArjunS07 commented Oct 19, 2026 •

edited

Loading

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

atharvas Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? ")


		def old_ranges(diff):

Conversation

ArjunS07 commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

How it works

Changes

Usage

Verification

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ArjunS07 commented Oct 19, 2026 •

edited

Loading