Skip to content

Feature branch: study-first rewrite - #114

Merged
asmacdo merged 208 commits into
mainfrom
study-first-rewrite
Sep 4, 2026
Merged

asmacdo merged 208 commits into
mainfrom
study-first-rewrite

Conversation

@asmacdo

@asmacdo asmacdo commented Aug 21, 2026 •

Copy link
Copy Markdown
Member

tldr:

  • The study-first substrate rewrite. The study becomes the primary unit; a campaign is a config-epoch record inside a study (.mechababs/campaigns/<label>/ — configs, pinned env, statefile), not a standalone dataset. The design contract is docs/spec.md (landed via Study-first direction — docs RFC, implementation to follow (draft) #101); docs/output_structure.md carries the target shape.
  • New CLI, uvx-first: campaign init · add-dataset · iterate · status · jobs · update-env · retire-derivative · test-cluster. The bootstrap.sh / configure / wide-ledger path is fully replaced.
  • Derivative directories name the campaign that made them: <App>-<Ver>[+<sourcedata-id>]+<label>, so a new campaign on a study never collides with a previous campaign's derivative and never forces a retire (campaign-per-bump is the working posture).
  • Every mutating transition is a datalad run --explicit record in the study's own history — scaffold and merge are dispatched with declared outputs; submit is provably non-mutating and stays plain. A study carries the verbatim, re-executable commands that built its derivatives.
  • Superstudy: the same campaign at a study-of-studies. Membership catalog at the super, per-member statefile shards, iterate fans out in catalog order (--batch is the whole tick's budget), and every level commits its own facts — the member its cells, the super its gitlink + catalog row.
  • Environment as provenance: one venv per campaign; the campaign pyproject.toml declares intent (git-ref pins), the committed uv.lock records resolution; the guard is location + uv sync --check; campaign update-env is the supported bump, and a detached member supports rerun-reproduction.
  • e2e rewritten as staged scenarios (lone-study spine + superstudy fan-out), running on the host (no scheduler) and in podman against real SLURM; mechababs test-cluster ships the same suite to validate a site config.
  • Proven on real hardware: a full MRIQC cell end-to-end on Unity (2026-08-26), a 265-job superstudy array over ds000030 (2026-08-30), and the OpenNeuroStudies shakeout now running on this branch.

Closes

Closes #40 — babs merge now runs under datalad run (the dispatched merge verb, with a decide-recheck guard and the RIA pull-back).
Closes #50 — the rewrite lands 419 unit tests over the new CLI; state/construct from the issue's untested list no longer exist.
Closes #60 — merge_config re-homed as mechababs/compose.py; the root script is deleted in the dead-code sweep below.
Closes #62 — the container shim is retired: babs resolves images natively from a ReproNim/containers clone (babs#399), validated at runtime on Unity; require_shim is gone and the shim script goes in the sweep.
Closes #63 — superseded and delivered by campaign update-env (edit the rev, update-env, or --upgrade for a moved branch pin).
Closes #65 — the e2e now builds real isolated campaign venvs in the container (cluster-axis env_constraints cover the glibc-2.17 wheels); --system-site-packages existed only in the deleted bootstrap path.
Closes #81 — configs use babs's $BABS_TMPDIR placeholder; the relative -w + --pwd fixup is gone, verified on Unity with no nipype trip.
Closes #82 — the per-cell recursive save-walk is gone: dispatch declares its outputs (--explicit), and the once-per-tick clean check is shallow (--ignore-submodules=dirty).
Closes #90 — every campaign commits its uv.lock (and the pyproject that produced it) as the environment record.
Closes #91 — configure no longer exists; campaign init records its work as one labeled transition.
Closes #106 — repo-wide ruff format + lint sweep, enforced in CI at pinned ruff 0.16.4 with a pinned rule set (landed via #120).
Closes #126 — campaign update-env exists (resolved differently than filed: the env stamp was dropped entirely in favor of uv sync --check, see #129).

The model: study-first, and what a campaign is now

The old model made the campaign a datalad dataset that contained everything — code submodules, sourcedata, derivatives, a wide ledger. This branch inverts it: the study is the container, and mechababs' entire footprint in one is additive — new derivatives under derivatives/, and a campaign record under .mechababs/campaigns/<label>/ holding the app + cluster configs, the pinned environment (pyproject.toml + uv.lock), the tall statefile (sourcedata+derivatives.tsv, one row per source-dataset × app cell), and the pinned inclusions. A study accumulates campaigns over time and stays operable standalone: clone it, rebuild the venv from the lock, and everything needed to reproduce or extend it is inside.

State stays derived, not declared: a pure route() reads the shard's columns (babs, merged) and both iterate and status route through it, so there is one reading of what a cell's state is. Identity columns are inputs and never overwritten.

The verbs, and the provenance they leave

Outer commands (iterate, add-dataset, status, retire-derivative, update-env) enforce selection (MECHABABS_CAMPAIGN, set by the campaign's env.sh), the configured-level rule, and the env-match guard. The mutating transitions are delegated to a hidden inner CLI (mechababs-inner scaffold|submit|merge) dispatched via datalad run --explicit -d <study> with per-verb declared outputs — so the study's git history holds each transition as a verbatim command that datalad rerun can replay, and the inner verbs carry their own cell-state and study-local env checks so a replay validates itself wherever it runs.

Per-verb commit shapes are deliberate: scaffold leaves its outputs for the run record to commit (one clean RUNCMD); submit was verified non-mutating and runs plain (dispatch.plain refuses if the tree or HEAD moved); merge moves subdataset HEADs mid-command, so it lands as the record-wrapping merge-commit bubble. add-dataset, update-env and retire-derivative are labeled plain saves — each one's docstring argues why a run record there would be a false promise (resolution against the live world; content deletion; host paths that resolve nowhere else).

Chained cells: an app config's depends_on names its producer by stem; a dependent cell waits (visibly) until the producer's cell merges, then resolve_input_origins wires the producer's derivative in as a babs input.

Superstudy

campaign init --superstudy <path> creates-or-adopts a study-of-studies; its campaign dir carries a membership catalog (studies+sourcedata.tsv) instead of a statefile — per-cell state shards to the members. add-dataset --study <member|url> selects (or clones) a member, writes its campaign footprint (configs seeded once, lock copied down) and a marker recording the super's datalad-id — an identity, so a member cloned elsewhere is correctly refused rather than silently adopted. iterate fans out in catalog order; only installed members advance; one member's drift stops that member, not the fan-out; the single-writer flock lives at the operated level. A campaign is operated only from its configured level — a member refuses, with no override; what a detached member supports is reproduction, which needs no flag.

Environment story

campaign init generates a campaign pyproject.toml (mechababs self-pinned from its own PEP 610 record; babs defaulting to the latest release, --babs URL@REF to pin a branch; datalad ≥ 1.6 floor) and locks + syncs a private venv. The lock is committed: it is the environment provenance. The guard asks two questions — is this the campaign's venv (location), and does it match the study's committed lock (uv sync --check) — and campaign update-env is the one verb that runs without it, because its job is exactly to restore that match (fresh clone, wiped site, historical checkout during rerun, or a deliberate bump via a pyproject edit / --upgrade). At a superstudy, update-env --study <member> copies the resulting lock into that member's footprint — the acknowledgment that its remaining work moves onto the new environment.

Cluster configs can carry env_constraints (uv constraint-dependencies), which is how an old-glibc site (slurm-docker-ci, Sherlock) resolves to versions with usable wheels.

Deleted, and what replaced it

bootstrap.sh + configure + construct.py + guard.py → campaign init. The wide ledger (state.py) + old iterate.py/status.py/retire.py → the tall shard, route(), and the rewritten verbs. merge_config.py → compose.py. run_on_cluster.sh + the old --campaign-path test-cluster → test-cluster --scratch-path, runnable via uvx from anywhere. The container shim → native ReproNim/containers resolution (babs#399). Each deletion happened in the chunk that replaced it, with grep evidence in the chunk PR; the stranded roots (bootstrap.sh, merge_config.py, the shim script, preflight.py, validate.clone_ref) went in the dead-code sweep (#140), vulture + grep over package/tests/docs/examples.

How it was verified
Temporary pins and caps (each tracked, none load-bearing forever)
  • datalad is promoted to a direct dependency at the BF: nested datalad run - fix crash when command commits in a subdataset below a plain dir datalad/datalad#7904 fix sha (nested-subdataset save crash under datalad run); reverts to the transitive ≥ 1.6 floor when a release carries it.
  • The e2e pins babs to git main because no babs release carries babs#399 (native container resolution); flips back to the release default at the next babs release.
  • The generated campaign env is capped python <3.14: h5py (via babs → niworkflows → nitransforms) ships no cp314 wheel; lift when it does.

Before merge

The pre-rewrite main is not tagged; this PR's first parent is the last commit that runs the old substrate.

Follow-up issues are filed after the merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U3EMv3ByezXDHCHJrTcsqN

asmacdo and others added 30 commits July 23, 2026 15:31
Seven user stories that form the requirements spine the study-first
direction answers to: sweep many pre-made studies, work in a single
study with no campaign, author a study from assorted source datasets,
add derivatives later, clone-and-extend (own and foreign-tool studies),
and move a run to another cluster. A mix of what works today and what we
are building toward; the list is expected to grow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite output_structure.md around the study as the primary unit:
- study-<id>/ holds derivatives + a .mechababs/campaigns/<label>/ record
  (a campaign is a config-epoch run, explicitly NOT a dataset)
- superstudy/ is an optional study-of-studies coordinator; homing it is
  optional (orchestration prov lives in the member studies)
- derivative + prov + publishing carried over; prov Bundle now points at
  the study rather than a separate campaign dataset
- open questions flagged with 💬 (create-a-study gap + metadata files,
  derivative-attempts placement, studies.tsv name overload, prov shape)

use_cases.md: add single-study/no-campaign, produce-one-derivative-easily
(with the config-reuse + orchestration-provenance framing and a nipoppy
nod for the pedagogical case), foreign-tool study, add-dataset-mid-campaign,
source-dataset-changes (💬), and crippled-filesystem stories; add the 💬
needs-feedback legend.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Walks the intended UX: uvx-from-a-pinned-ref bootstrap (no global install,
no invented syntax — plain uv), a per-dataset .mechababs/ env + uv.lock as
just-in-time provenance, familiar source-the-venv daily use, cwd-scoped
iterate/status, and campaign-by-label. Flags the unresolved campaign-selection
and env-enforcement mechanics with a 💬 needs-discussion block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From the discussion with Yarik on mega-studies: studies live in their
remote and are installed only while there is work to do on them.

- act on one study within a superstudy (scoping, not exclusivity)
- release a finished study; mechababs neither pushes nor is disturbed by it
- see the state of the set without holding the data
- run under a finite disk/inode budget
- choose what gets worked next

Use cases only, no implementation. The chaining case is deliberately
absent -- it turned out not to be about eviction.
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
- QC gate as a prerequisite to the expensive stages, and lab-owned
  off-OpenNeuro datasets folded into study authoring (both from Logan's
  Sherlock experience)
- consolidating a multi-cluster campaign back into one superstudy
- normalize on 'superstudy'; distinct member paths in the superstudy tree
From Yarik's review on PR #101: dashboarding that works at every level,
summaries derived from the level beneath rather than maintained beside
it, and state files conventional enough for other tools to read.

His terminology point (leaf-to-root vs 'bottom-up') is a doc-wide
vocabulary question, not a use case -- left for output_structure.md.
Yarik asked for terminology alignment. Marked 💬 -- the definitions are
a proposal, not a decision, and the leaf-to-root direction is the part
most likely to be argued.
Two boundaries that keep coming up as design questions. Each is paired
with the tolerance requirement it implies, since "not our job" alone
doesn't tell the design anything.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
just a commit to allow a PR to start feature branch
…study, mid-sweep release bump, add-shapes, cross-cluster consolidation sharpened
…ents pass stated in place

use_cases: terminology settled (root/leaves, up=toward root); selector decided (explicit study
argument, working dir gives the level); dataset-change case stays open, stated plainly.
output_structure: generic sourcedata slots (raw/rawbids) + derivative +<id> naming; statefile
named sourcedata+derivatives.tsv, inclusions/ + env.sh in the campaign footprint; retire moves
outside the study (replaces in-study derivative-attempts); superstudy fan-out happens at
add-dataset, not campaign init; membership file is studies+sourcedata.tsv (no collision with
OpenNeuroStudies' root indexes) + coarse per-member lifecycle; prov/ marked out of scope for
the study-first implementation.
…no bootstrap verb), per-campaign venv + env.sh, explicit add-dataset selection, configured-level iterate
The decisions of record for the study-first implementation, distilled from the
requirements pass: installer/env (uvx campaign init, one venv per campaign,
mutable-via-history lock), layout (existing-study input, +<id> derivative
naming), campaign definition (env-var selection, user-provided configs,
depends_on input/gate kinds), statefile & reconciler (tall per-study shard,
datalad run --explicit transitions, canonical-super/mirror-member, coarse
committed lifecycle), CLI surface (mechababs + inner verbs, add-dataset shapes,
retire-outside-study), and non-goals. Open items stated in place.

quickstart: campaign selection is always the env var (no default-if-one).
Study-first direction — docs RFC, implementation to follow (draft)
… env-match guard

Study-first moves the unit of work from a campaign dataset to a study that
already exists, with each campaign a config epoch recorded inside it at
.mechababs/campaigns/<label>/ (docs/spec.md, docs/output_structure.md).

Two modules carry the shapes every later verb needs:

- study.require_study_root — one place answering "am I in a study", kept
  deliberately shallow (a dataset root, not a BIDS validation).
- campaign — the campaign dir layout, the tall statefile schema, selection via
  MECHABABS_CAMPAIGN (always explicit, no default-if-only-one), and the
  env-match guard.

The guard refuses both drift directions: running from some other python, and
running from the campaign venv after the committed lock moved. The lock is
mutable through git history — that is how a mid-sweep version bump works — so
the venv carries a stamp of the lock it was built from, written where it cannot
be committed or outlive its environment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…study

`mechababs campaign init <label> --apps … --cluster … [--limit N]`, run at the
root of an existing study, writes .mechababs/campaigns/<label>/: the user's app
and cluster configs copied in, campaign.yaml (bundle order, cluster, limit), the
header-only tall statefile, env.sh, pyproject.toml + uv.lock, and the venv the
lock builds.

This is the one command that runs before the campaign environment exists, so it
takes no env-match guard and may run ephemerally from a pinned ref
(`uvx --from git+…@<ref> mechababs campaign init …`). Everything after it runs
from the venv it builds.

The lock is the provenance: `uv lock` resolves mechababs and babs to exact
commits, and that committed file — not a vendored code clone — is what says
which tools ran. The mechababs pin is read from the running install's PEP 610
direct_url.json, so the campaign records the mechababs actually invoked rather
than a ref the user would have to retype; babs, which mechababs only shells out
to, is named by --babs URL@REF, carrying over bootstrap.sh's interface and
default so a PR branch (or a local checkout) stays a config change.

Configs are user-provided by path or URL and copied in — never a bare name
resolved against a directory mechababs knows about. A depends_on naming an app
outside the bundle is refused here, earlier than the per-cell check at
add-dataset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"campaign 'nprep''s venv" is what a possessive on a quoted label renders as.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pin was already there, unexplained. It is load-bearing for the study-first
`datalad run` capture: 1.4 loses the run record when the wrapped command commits
its own work, which every scaffold does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hares

`require_selected_campaign` = in a study, with a campaign selected, running that
campaign's environment. `campaign init` is the one command that does not take it:
it runs before the environment exists, being what creates it.

Also clears ruff on the new modules (the repo-wide pass is #106;
this only avoids adding to it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g uv to PyPI

An unsourced 'mechababs' requirement resolves against PyPI, where the package
does not exist (a release is #113) — the resolver error would land
far from the cause. Name --mechababs URL@REF as the fix at the point of failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cessors

add-dataset needs the same path-scoped `datalad save --to-git` campaign init
writes with, so it moves to utils beside the other datalad primitives (the flock,
the save scope) rather than being imported out of campaign_init.

The tall statefile gains read/write accessors and a per-campaign flock filename.
Unlike the wide ledger it replaces, its columns do not vary with the app bundle,
so the schema is the module's rather than the file's header. The flock is
gitignored from inside the campaign dir: a lock left in the tree would dirty the
study, which the transition verbs read as unattributable work.
add-dataset fills a cell's identity columns by sniffing the same per-subject TSV
selection reads later — so the reader lives beside it rather than in a second
module that would drift.

Two things the existing reader did not have to know. A study's TSV covers every
source dataset in it, keyed by `source_id`, so a study holding several needs the
rows narrowed; a study holding one must NOT match on that key, or a generic
sourcedata slot (`sourcedata/raw`, whose directory name is not a dataset id)
would find nothing. And `processing_level` comes from the data — a dataset is
session-level when its rows name real sessions — not from which of the two
filenames the metadata arrived in.
`mechababs add-dataset --sourcedata <path>`: the explicit step that says which
data a campaign acts on. It walks UP from the path to the enclosing study — so
the same command serves a lone study and a member of a superstudy — sniffs that
source dataset in the study's per-subject metadata to fill the cell's identity
columns, and writes one cell per app in the campaign's bundle into that study's
statefile shard, `depends_on` carried from each app config.

It never brings data in: a path that is not there, or not inside a study, is
refused rather than installed. Derived columns start empty, which is what routes
each cell to scaffold on the next tick; the subject inclusion is scaffold's, not
this verb's.

A dangling `depends_on` fails here, where the cell is written, rather than
parking the cell forever in a reconciler that can never resolve the edge. Adding
the whole bundle at once satisfies its own edges. Re-adding leaves existing cells
exactly as they are — state and all — and writes only what is missing, so a
campaign whose bundle grew needs no second verb; when nothing is missing it
refuses rather than committing nothing.

Three things the spec did not pin down, now recorded in it: `source_dataset` is
the study-relative path (what scaffold needs, and no assumption that sourcedata
sits at `sourcedata/<id>`); `processing_level` is read from the metadata rows,
not from which filename the study uses; and the `source_id` match applies only
when a study describes several source datasets, so a generic `sourcedata/raw`
slot still resolves.

The pre-study-first `add-dataset <url>` stays reachable behind its positional
argument, hidden from --help, so the old path keeps working while the rest of the
spine lands. It goes with the e2e re-point.
As written it could be read as sourcedata/raw yielding '+raw'; the spec's own
naming rule says the generic slots carry no id suffix — cross-reference it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With no --babs, the campaign's pyproject declares a plain `babs` dependency
and no [tool.uv.sources] entry, so `uv lock` resolves the latest released babs
and freezes that exact version. The lock is still the pin — of a release
rather than a commit.

--babs URL@REF is unchanged: overrides can be git (a PR branch, a local
checkout), the default is the release. babs main runs far ahead of its last
release, so a real campaign often overrides — reaching for a commit is now the
explicit act.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
… save

`campaign init` writes a `.gitattributes` (`* annex.largefiles=nothing`) as the
first file in the campaign dir, and the save drops `--to-git`.

Everything under the campaign dir is small text a clone must read without
fetching annex content — the lock above all, since rebuilding the environment
from a fresh clone is the whole reproduction story. As an attribute that holds
for every later writer into the dir (and inside a clone), where a per-save flag
holds only for the one save that remembers it — and would mis-route the moment
the shared helper's path scope reached a real subdataset.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
The helper guessed at `<sys.prefix>/bin/datalad` and fell back to a bare
`datalad` on PATH. Under `uvx` there is no `bin/datalad` beside the interpreter,
so the fallback fires and whatever datalad the user's PATH holds runs the
campaign's commit — or nothing does. datalad is a declared dependency, so the
import is guaranteed where PATH is not, and `utils.datalad_save_scope` already
goes through the API.

Failure is read off the result records, and `describe_result` renders datalad's
lazy `(format, *args)` messages so the refusal is legible.

Co-Authored-By: Claude Opus <noreply@anthropic.com>
asmacdo and others added 25 commits September 3, 2026 13:46
A hand-run script from the first shakeouts: given a dataset id it checked that
no OpenNeuroDerivatives/<id>-mriqc repo existed and that the raw dataset was
reachable. Nothing invokes it, it is MRIQC-only and OpenNeuro-only (the two
couplings the CLAUDE.md strip just moved out of the repo), and its own docstring
records the false-pass on git auth/network errors that #10 tracks.
The check it did belongs, if anywhere, in add-dataset's sniff against the
cloned study, which already sees the study's existing derivatives.

Evidence: `git grep preflight` hits only the file and design/openneuro-findings.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
The findings doc is a 2026-03 exploration of the OpenNeuroStudies layout. Its
durable content -- the three-org table and the study/sourcedata/derivatives
shape -- is what the repo docs and the tool now encode, and the rest is our
deployment's, which lives outside the repo since #134.

The ignore lines went with the layouts they hid: processing/ (the pre-campaign
output tree), repronim-containers/ (the shim clone), design/*.png (that doc's
figures, of which none exist).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
select.py listed desc-mechababs_datasets.tsv, the wide ledger state.py kept and
#130 deleted, among the TSVs it is not; the statefile it should
contrast with is sourcedata+derivatives.tsv. test_spine's _iterate docstring
explained itself by the absence of --campaign-path, a flag #127
removed, which tells a reader nothing now.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
It is our curated list of the OpenNeuro datasets we want processed: a fact
about our deployment, not about the tool. Nothing in the package reads it,
and the same reasoning that took the OpenNeuro sections out of CLAUDE.md
(#134) applies to the data behind them. It lives with our
deployment notes now; this repo's history keeps its past.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
Sweep the transition narration out of comments, docstrings and docs:
"this rewrite delivers", "used to", "now that the stamp is gone",
"the whole chunk", "study-first" as a project name, and attributions
of a rule to a person. Each is rewritten as a statement of how the
tool works, which is what a reader after the merge needs.

One test is renamed with its docstring:
test_env_match_no_longer_refuses_a_venv_mechababs_did_not_build ->
test_env_match_accepts_a_venv_mechababs_did_not_build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
Rationale stated both in the module header and at the function that
implements it now lives only at the function; incident narration is
replaced by the rule it produced; the no-override rule is stated once,
where it is enforced.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The pin story, the uv-in-the-lock rule and the env-match mechanics are
stated where they are implemented (campaign.py) and referenced from
here; the three consecutive comment blocks inside init() fold into one
each. The already-exists message drops "config epoch", a term the
glossary does not define.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The uninstalled-member and configured-level rationale each lived in the
docstring and again inline; the inline copies are now pointers. The
flock and clean-in comments keep the reason and drop the restatement.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The module docstring carried the user-facing retire section of
docs/reference.md nearly verbatim; it now summarizes and points there.
The published-object rule is stated once, at require_outside.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The computed-not-stored rollup, the stdout/stderr split and the no-flock
rule each appeared in the module docstring and again in the function
that implements it; the module now points, the function explains. The
UNKNOWN and FROM_LIFECYCLE comments keep the reasoning at half the length.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The module docstring restated the declared-paths contract and the
gitlink-not-recurse rule that campaign_save_scope's docstring carries;
it now points there. require_clean_gitlink and require_clean_shallow
keep their reasons and drop the restatements.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
Six consecutive comment blocks bracketing the nested save scopes
restated the contract each scope already documents; each keeps the one
fact specific to this site. The footprint-ownership docstring says the
rule; the refusal message already carries the detail.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The outer-first save nesting and its reason are stated at add-dataset;
copy_lock_to_member now says "as in add-dataset". The flock and retry
comments keep the site-specific fact.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The two-pins asymmetry is explained in validate.py and pointed at from
the e2e fixtures; the study-lock check is explained at
require_study_lock_match and pointed at from inner.py; the datalad.api
note lives in utils.py; DEFAULT_BABS's comment is the one place that
says why the suite pins babs main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The stamp era, the exit-127 array on a named cluster, and the "first
field failure / N unit tests passed" story are each replaced by the
invariant the test pins. One site name dropped from a docstring.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The term was not in the glossary and every use unpacks to plain words:
a campaign's own label, configs and lock; one lock = one venv; the
bundle and lock are fixed at init.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
…kes --study

`--study <member>` on a member that .gitmodules registers but that is
not installed said "not a member study", which is false, and offered a
re-clone, which is wrong; it now says the member is not installed and
names `datalad get -n <member>`. This is the first thing anyone hits
after a plain clone of a real superstudy.

`campaign init --superstudy`'s next-steps hint printed the study form
of add-dataset; at a superstudy it needs `--study`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The shell expands ~ only at the start of a word, so the second path in a
comma-joined --apps a.yaml,~/b.yaml arrives literal and was reported as
not found.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B86TEMruRYUpr5m7oFmzuK
campaign init: expand a literal ~ in a config path
A study accumulates campaigns, and a cell is (source dataset x app) in
one campaign. Without the label in the name a second campaign could not
produce the same cell until the first campaign's derivative was retired.
The label comes last: <Tool>-<Ver>+<stage>[+<id>]+<label>.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add campaign label to derivative name
@asmacdo

asmacdo commented Sep 4, 2026 •

Copy link
Copy Markdown
Member Author

@lobennett this monstrosity is going in now. ill continue working on it with regular-sized prs ;)

It should be documented well enough that you can try on your own, but hmu if youd like to meet up for discussion/tour/etc.

The thing to watch for on Sherlock will be the glibc issue. CI has the same problem, and campaigns now have env_constraints to help that case. I imagine your env_constraints will look a lot like the CI.

@asmacdo
asmacdo marked this pull request as ready for review September 4, 2026 21:01
@asmacdo
asmacdo merged commit 41324c9 into main Sep 4, 2026
4 checks passed
@asmacdo
asmacdo deleted the study-first-rewrite branch September 4, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment