[ArjunS07]

Purpose

The repo had dated plans, handoffs, module design notes and a copy of the rl-prep notes that no longer describe the code. Some docs said that tasks are built with terminal-bench and run in fc-eval, and many comments told history instead of what the code does. This PR deletes the stale files, corrects the wrong docs, shortens comments, and adds one page that describes the whole pipeline across repos. It also adds a time limit to fc-measure measure, because some pandas and scikit-learn tasks took 8 to 17 hours.

How it works

flowchart LR
    A["fc-data"] --> B["fc-render-tasks export --from-db"] --> C["fc-render-tasks render"] --> D["fc-build-tasks"] --> E["fc-render-tasks render --images"]
    E --> F["fc-measure precheck"] --> G["fc-measure measure"] --> H["classify, review, release<br/>(formulacode-verified-rl)"] --> I["agent trials and RL"]

Loading

docs/OVERVIEW.md explains each step above, what "approved" means, where the approved list is, how to build one task again, and which branch or commit each repo uses. It says to render from the tag template-v1.4.

docs/OVERVIEW.md                    # new: the pipeline across repos
.github/pull_request_template.md    # new: same file as #33
docs/superpowers/                   # deleted
docs/design/components/             # deleted
docs/design/archived/               # deleted
docs/stage6-failure-analysis.md     # deleted
HANDOFF.md                          # deleted
rl-prep/rl-prep/                    # deleted
src/datasmith/measurement/run.py    # + Settings.time_limit_s, TooSlow
src/datasmith/measurement/cli.py    # + fc-measure measure --time-limit
src/datasmith/runners/render_problems.py  # IPython display hooks are public names

Changes

1. Stale docs deleted.
   • Deleted: docs/superpowers/ (30 dated specs, plans, handoffs and reports), docs/stage6-failure-analysis.md, docs/design/archived/, HANDOFF.md and rl-prep/rl-prep/.
   • Deleted: docs/design/components/ (9 module notes). They describe modules that no longer exist, such as datasmith.execution and datasmith/core/api/. mkdocs already left them out of the site.
   • Deleted: the dataset/verify.py loop in README.md, CLAUDE.md and docs/guide/verification.md, because dataset/ is no longer in git.
   • Effect: no dated plan or log stays in the repo. Facts that are still true moved. Why repositories fail synthesis is in docs/guide/synthesis.md. The conditions to turn on DATASMITH_PV_ENABLED are in CLAUDE.md.
   • Before: CLAUDE.md, a design doc and two tests pointed at the spec files as the source of intent. CLAUDE.md told agents to read the module notes before a change.
2. Wrong docs corrected. README, docs/index.md, DATASET_CARD.md, CLAUDE.md and docs/guide/lsv-development.md.
   • Effect: the docs say that datasmith.harbor_adapter (fc-render-tasks) renders the tasks and that they run on Harbor.
   • Effect: the docs say that Harbor's adapter imports datasmith, and that images install LSV at the pinned LSV_REF.
   • Before: the docs named terminal-bench's adapter and fc-eval, called harbor_adapter a vendored copy of Harbor's adapter, and described an editable LSV install that no longer exists.
3. Docs for the commands of measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66. docs/OVERVIEW.md and CLAUDE.md.
   • Effect: the steps use fc-render-tasks export --from-db, fc-build-tasks, fc-render-tasks render --images, fc-measure precheck and fc-measure measure. The measure step describes the host lock folder (FC_MEASURE_LOCK_DIR, FC_MEASURE_LOCK_SLOTS).
   • Effect: CLAUDE.md lists measurement/ in place of verify/, and stage 5 names RenderShortProblemsRunner.
   • Before: the overview described the fc-verify commands and the HTTP measure gate, which measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 deletes.
4. Comments in src/. About 310 comment and docstring blocks in agents/, docker/, github/, resolution/, runners/, update/, utils/, publish/ and harbor_adapter/.
   • Effect: each says what the code does and why, in 1 or 2 lines, with no dates, history, host names or jargon. Four comments that did not match the code now match it.
   • Before: long narrative blocks with dates and past incidents.
5. Host paths removed. preflight.py, publish/huggingface.py, scripts/prepass_trial.py, scripts/load_benchmark_information.py.
   • Effect: the Hugging Face token path defaults to ~/.cache/huggingface/token, and HF_TOKEN_PATH overrides it.
   • Effect: the benchmark loader uses DATASMITH_TB_ROOT. The prepass script does not set TMPDIR.
   • Before: the defaults were paths on one machine under /mnt/sdd*.
6. fc-measure measure --time-limit SECONDS.
   • Effect: a measure job that runs past the limit gets SIGINT, so harbor removes its containers, then SIGKILL after 120 s. The task ends as too_slow and gets no record in OUT/oracle/.
   • Effect: harbor stops the trial containers on SIGINT, and the kernel then releases their lock slots.
   • Before: a slow job ran until it finished. The default 0 keeps that behavior.
7. IPython display hooks are public names.
   • Effect: the giveaway check of RenderShortProblemsRunner (_leaks) accepts a problem text that names _repr_html_ or another _repr_*_ or _ipython_*_ hook.
   • Before: the check refused such a text as internal code.
8. Lint. Sorted imports in resolution/orchestrator.py, noqa: S310 on the two URL opens in runners/harbor_healthcheck.py, and ruff format on publish/huggingface.py and runners/harbor_healthcheck.py.
   • Effect: ruff check and ruff format --check pass on the changed files.
9. PR template. .github/pull_request_template.md, the same file as in Add PR template and PR style rules #33.
   • Effect: new PRs get the sections Purpose, How it works, Changes, Usage, Verification and Notes.
   • Before: no template on this branch.

Usage

fc-measure measure tasks/ out/ owner/repo#N --workers 2 --time-limit 21600

Verification

Run on the head abe8074:

• PYTHONPATH=src python -m pytest -q -m "not slow" tests: 1821 passed, 10 skipped, 39 deselected. measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 alone gives 1819 passed. The 2 new tests are the time limit and the display hook.
• ruff check and ruff format --check on the changed Python files: all pass.
• git merge-tree --write-tree origin/verify/core origin/docs/overview-and-comments: no conflicts.

Notes

• Merge order: task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65, then measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66, then this PR. This PR is based on verify/core (measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66), and it merges verify/core at 13b98a0.
• measurement: timing, noise rule and precheck in a new package; task image builds as a runner; short prompts by default #66 deletes src/datasmith/verify/. This PR accepts that deletion. What happened to its earlier verify/ changes:
  • The measure time limit moved to measurement/run.py and measurement/cli.py (change 6). It has no lease release now, because there is no gate service.
  • The display hook rule moved to render_problems._leaks (change 7).
  • The precheck fallback to the tag in task.toml is gone, because fc-measure precheck loads the image build check from the database and does not run the image.
  • The base_edited note text is gone with review.py. The decide() docstring in measurement/precheck.py already says what the check finds.
  • The prompt apply rules are not in datasmith. They are: apply an entry only when approved is a review date, refuse inline relevant_issues, and fill an empty problem text. They belong to apply_gold_prompt.py in formula-code/formulacode-verified-rl.
• The PR template links to docs/PR_STYLE.md, which only Add PR template and PR style rules #33 adds. Merge Add PR template and PR style rules #33 too, or the link has no target. The two template files are identical, so the PRs do not conflict.
• This PR does not change the task template. The shorter comments in tests/parser.py and tests/lsv_measure.py are in task template v1: the Phase 1 template and its fixes in one branch, for one merge into main #65.
• Files that are copied into images were not changed, except one docstring in agents/templates/local_ci.py that lost a link to a deleted spec. A comment change in docker/templates/ or agents/templates/ invalidates the Docker build cache. For this reason, the docstring of docker/templates/apply_oracle_patch.py still names the deleted datasmith.verify.in_image.coverage_check.
• The branch is docs/overview-and-comments, because a remote branch named cleanup blocks the name cleanup/docs-and-comments.

[atharvas]

Looks good. I'm in favor of an even more aggressive archiving of legacy documentation and code.