Repository navigation
Feature branch: study-first rewrite - #114
Merged
Merged
Conversation
Seven user stories that form the requirements spine the study-first direction answers to: sweep many pre-made studies, work in a single study with no campaign, author a study from assorted source datasets, add derivatives later, clone-and-extend (own and foreign-tool studies), and move a run to another cluster. A mix of what works today and what we are building toward; the list is expected to grow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite output_structure.md around the study as the primary unit: - study-<id>/ holds derivatives + a .mechababs/campaigns/<label>/ record (a campaign is a config-epoch run, explicitly NOT a dataset) - superstudy/ is an optional study-of-studies coordinator; homing it is optional (orchestration prov lives in the member studies) - derivative + prov + publishing carried over; prov Bundle now points at the study rather than a separate campaign dataset - open questions flagged with 💬 (create-a-study gap + metadata files, derivative-attempts placement, studies.tsv name overload, prov shape) use_cases.md: add single-study/no-campaign, produce-one-derivative-easily (with the config-reuse + orchestration-provenance framing and a nipoppy nod for the pedagogical case), foreign-tool study, add-dataset-mid-campaign, source-dataset-changes (💬), and crippled-filesystem stories; add the 💬 needs-feedback legend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Walks the intended UX: uvx-from-a-pinned-ref bootstrap (no global install, no invented syntax — plain uv), a per-dataset .mechababs/ env + uv.lock as just-in-time provenance, familiar source-the-venv daily use, cwd-scoped iterate/status, and campaign-by-label. Flags the unresolved campaign-selection and env-enforcement mechanics with a 💬 needs-discussion block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From the discussion with Yarik on mega-studies: studies live in their remote and are installed only while there is work to do on them. - act on one study within a superstudy (scoping, not exclusivity) - release a finished study; mechababs neither pushes nor is disturbed by it - see the state of the set without holding the data - run under a finite disk/inode budget - choose what gets worked next Use cases only, no implementation. The chaining case is deliberately absent -- it turned out not to be about eviction.
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
- QC gate as a prerequisite to the expensive stages, and lab-owned off-OpenNeuro datasets folded into study authoring (both from Logan's Sherlock experience) - consolidating a multi-cluster campaign back into one superstudy - normalize on 'superstudy'; distinct member paths in the superstudy tree
From Yarik's review on PR #101: dashboarding that works at every level, summaries derived from the level beneath rather than maintained beside it, and state files conventional enough for other tools to read. His terminology point (leaf-to-root vs 'bottom-up') is a doc-wide vocabulary question, not a use case -- left for output_structure.md.
Yarik asked for terminology alignment. Marked 💬 -- the definitions are a proposal, not a decision, and the leaf-to-root direction is the part most likely to be argued.
Two boundaries that keep coming up as design questions. Each is paired with the tolerance requirement it implies, since "not our job" alone doesn't tell the design anything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
just a commit to allow a PR to start feature branch
…t stories, split dataset-change cases
…study, mid-sweep release bump, add-shapes, cross-cluster consolidation sharpened
…ents pass stated in place use_cases: terminology settled (root/leaves, up=toward root); selector decided (explicit study argument, working dir gives the level); dataset-change case stays open, stated plainly. output_structure: generic sourcedata slots (raw/rawbids) + derivative +<id> naming; statefile named sourcedata+derivatives.tsv, inclusions/ + env.sh in the campaign footprint; retire moves outside the study (replaces in-study derivative-attempts); superstudy fan-out happens at add-dataset, not campaign init; membership file is studies+sourcedata.tsv (no collision with OpenNeuroStudies' root indexes) + coarse per-member lifecycle; prov/ marked out of scope for the study-first implementation.
…no bootstrap verb), per-campaign venv + env.sh, explicit add-dataset selection, configured-level iterate
The decisions of record for the study-first implementation, distilled from the requirements pass: installer/env (uvx campaign init, one venv per campaign, mutable-via-history lock), layout (existing-study input, +<id> derivative naming), campaign definition (env-var selection, user-provided configs, depends_on input/gate kinds), statefile & reconciler (tall per-study shard, datalad run --explicit transitions, canonical-super/mirror-member, coarse committed lifecycle), CLI surface (mechababs + inner verbs, add-dataset shapes, retire-outside-study), and non-goals. Open items stated in place. quickstart: campaign selection is always the env var (no default-if-one).
Study-first direction — docs RFC, implementation to follow (draft)
… env-match guard Study-first moves the unit of work from a campaign dataset to a study that already exists, with each campaign a config epoch recorded inside it at .mechababs/campaigns/<label>/ (docs/spec.md, docs/output_structure.md). Two modules carry the shapes every later verb needs: - study.require_study_root — one place answering "am I in a study", kept deliberately shallow (a dataset root, not a BIDS validation). - campaign — the campaign dir layout, the tall statefile schema, selection via MECHABABS_CAMPAIGN (always explicit, no default-if-only-one), and the env-match guard. The guard refuses both drift directions: running from some other python, and running from the campaign venv after the committed lock moved. The lock is mutable through git history — that is how a mid-sweep version bump works — so the venv carries a stamp of the lock it was built from, written where it cannot be committed or outlive its environment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…study `mechababs campaign init <label> --apps … --cluster … [--limit N]`, run at the root of an existing study, writes .mechababs/campaigns/<label>/: the user's app and cluster configs copied in, campaign.yaml (bundle order, cluster, limit), the header-only tall statefile, env.sh, pyproject.toml + uv.lock, and the venv the lock builds. This is the one command that runs before the campaign environment exists, so it takes no env-match guard and may run ephemerally from a pinned ref (`uvx --from git+…@<ref> mechababs campaign init …`). Everything after it runs from the venv it builds. The lock is the provenance: `uv lock` resolves mechababs and babs to exact commits, and that committed file — not a vendored code clone — is what says which tools ran. The mechababs pin is read from the running install's PEP 610 direct_url.json, so the campaign records the mechababs actually invoked rather than a ref the user would have to retype; babs, which mechababs only shells out to, is named by --babs URL@REF, carrying over bootstrap.sh's interface and default so a PR branch (or a local checkout) stays a config change. Configs are user-provided by path or URL and copied in — never a bare name resolved against a directory mechababs knows about. A depends_on naming an app outside the bundle is refused here, earlier than the per-cell check at add-dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"campaign 'nprep''s venv" is what a possessive on a quoted label renders as. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pin was already there, unexplained. It is load-bearing for the study-first `datalad run` capture: 1.4 loses the run record when the wrapped command commits its own work, which every scaffold does. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hares `require_selected_campaign` = in a study, with a campaign selected, running that campaign's environment. `campaign init` is the one command that does not take it: it runs before the environment exists, being what creates it. Also clears ruff on the new modules (the repo-wide pass is #106; this only avoids adding to it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g uv to PyPI An unsourced 'mechababs' requirement resolves against PyPI, where the package does not exist (a release is #113) — the resolver error would land far from the cause. Name --mechababs URL@REF as the fix at the point of failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cessors add-dataset needs the same path-scoped `datalad save --to-git` campaign init writes with, so it moves to utils beside the other datalad primitives (the flock, the save scope) rather than being imported out of campaign_init. The tall statefile gains read/write accessors and a per-campaign flock filename. Unlike the wide ledger it replaces, its columns do not vary with the app bundle, so the schema is the module's rather than the file's header. The flock is gitignored from inside the campaign dir: a lock left in the tree would dirty the study, which the transition verbs read as unattributable work.
add-dataset fills a cell's identity columns by sniffing the same per-subject TSV selection reads later — so the reader lives beside it rather than in a second module that would drift. Two things the existing reader did not have to know. A study's TSV covers every source dataset in it, keyed by `source_id`, so a study holding several needs the rows narrowed; a study holding one must NOT match on that key, or a generic sourcedata slot (`sourcedata/raw`, whose directory name is not a dataset id) would find nothing. And `processing_level` comes from the data — a dataset is session-level when its rows name real sessions — not from which of the two filenames the metadata arrived in.
`mechababs add-dataset --sourcedata <path>`: the explicit step that says which data a campaign acts on. It walks UP from the path to the enclosing study — so the same command serves a lone study and a member of a superstudy — sniffs that source dataset in the study's per-subject metadata to fill the cell's identity columns, and writes one cell per app in the campaign's bundle into that study's statefile shard, `depends_on` carried from each app config. It never brings data in: a path that is not there, or not inside a study, is refused rather than installed. Derived columns start empty, which is what routes each cell to scaffold on the next tick; the subject inclusion is scaffold's, not this verb's. A dangling `depends_on` fails here, where the cell is written, rather than parking the cell forever in a reconciler that can never resolve the edge. Adding the whole bundle at once satisfies its own edges. Re-adding leaves existing cells exactly as they are — state and all — and writes only what is missing, so a campaign whose bundle grew needs no second verb; when nothing is missing it refuses rather than committing nothing. Three things the spec did not pin down, now recorded in it: `source_dataset` is the study-relative path (what scaffold needs, and no assumption that sourcedata sits at `sourcedata/<id>`); `processing_level` is read from the metadata rows, not from which filename the study uses; and the `source_id` match applies only when a study describes several source datasets, so a generic `sourcedata/raw` slot still resolves. The pre-study-first `add-dataset <url>` stays reachable behind its positional argument, hidden from --help, so the old path keeps working while the rest of the spine lands. It goes with the e2e re-point.
As written it could be read as sourcedata/raw yielding '+raw'; the spec's own naming rule says the generic slots carry no id suffix — cross-reference it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With no --babs, the campaign's pyproject declares a plain `babs` dependency and no [tool.uv.sources] entry, so `uv lock` resolves the latest released babs and freezes that exact version. The lock is still the pin — of a release rather than a commit. --babs URL@REF is unchanged: overrides can be git (a PR branch, a local checkout), the default is the release. babs main runs far ahead of its last release, so a real campaign often overrides — reaching for a commit is now the explicit act. Co-Authored-By: Claude Opus <noreply@anthropic.com>
… save `campaign init` writes a `.gitattributes` (`* annex.largefiles=nothing`) as the first file in the campaign dir, and the save drops `--to-git`. Everything under the campaign dir is small text a clone must read without fetching annex content — the lock above all, since rebuilding the environment from a fresh clone is the whole reproduction story. As an attribute that holds for every later writer into the dir (and inside a clone), where a per-save flag holds only for the one save that remembers it — and would mis-route the moment the shared helper's path scope reached a real subdataset. Co-Authored-By: Claude Opus <noreply@anthropic.com>
The helper guessed at `<sys.prefix>/bin/datalad` and fell back to a bare `datalad` on PATH. Under `uvx` there is no `bin/datalad` beside the interpreter, so the fallback fires and whatever datalad the user's PATH holds runs the campaign's commit — or nothing does. datalad is a declared dependency, so the import is guaranteed where PATH is not, and `utils.datalad_save_scope` already goes through the API. Failure is read off the result records, and `describe_result` renders datalad's lazy `(format, *args)` messages so the refusal is legible. Co-Authored-By: Claude Opus <noreply@anthropic.com>
A hand-run script from the first shakeouts: given a dataset id it checked that no OpenNeuroDerivatives/<id>-mriqc repo existed and that the raw dataset was reachable. Nothing invokes it, it is MRIQC-only and OpenNeuro-only (the two couplings the CLAUDE.md strip just moved out of the repo), and its own docstring records the false-pass on git auth/network errors that #10 tracks. The check it did belongs, if anywhere, in add-dataset's sniff against the cloned study, which already sees the study's existing derivatives. Evidence: `git grep preflight` hits only the file and design/openneuro-findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
The findings doc is a 2026-03 exploration of the OpenNeuroStudies layout. Its durable content -- the three-org table and the study/sourcedata/derivatives shape -- is what the repo docs and the tool now encode, and the rest is our deployment's, which lives outside the repo since #134. The ignore lines went with the layouts they hid: processing/ (the pre-campaign output tree), repronim-containers/ (the shim clone), design/*.png (that doc's figures, of which none exist). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
select.py listed desc-mechababs_datasets.tsv, the wide ledger state.py kept and #130 deleted, among the TSVs it is not; the statefile it should contrast with is sourcedata+derivatives.tsv. test_spine's _iterate docstring explained itself by the absence of --campaign-path, a flag #127 removed, which tells a reader nothing now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
It is our curated list of the OpenNeuro datasets we want processed: a fact about our deployment, not about the tool. Nothing in the package reads it, and the same reasoning that took the OpenNeuro sections out of CLAUDE.md (#134) applies to the data behind them. It lives with our deployment notes now; this repo's history keeps its past. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pfazng8PuPvbQNN25uzK2Y
Sweep the transition narration out of comments, docstrings and docs: "this rewrite delivers", "used to", "now that the stamp is gone", "the whole chunk", "study-first" as a project name, and attributions of a rule to a person. Each is rewritten as a statement of how the tool works, which is what a reader after the merge needs. One test is renamed with its docstring: test_env_match_no_longer_refuses_a_venv_mechababs_did_not_build -> test_env_match_accepts_a_venv_mechababs_did_not_build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
Rationale stated both in the module header and at the function that implements it now lives only at the function; incident narration is replaced by the rule it produced; the no-override rule is stated once, where it is enforced. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The pin story, the uv-in-the-lock rule and the env-match mechanics are stated where they are implemented (campaign.py) and referenced from here; the three consecutive comment blocks inside init() fold into one each. The already-exists message drops "config epoch", a term the glossary does not define. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The uninstalled-member and configured-level rationale each lived in the docstring and again inline; the inline copies are now pointers. The flock and clean-in comments keep the reason and drop the restatement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The module docstring carried the user-facing retire section of docs/reference.md nearly verbatim; it now summarizes and points there. The published-object rule is stated once, at require_outside. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The computed-not-stored rollup, the stdout/stderr split and the no-flock rule each appeared in the module docstring and again in the function that implements it; the module now points, the function explains. The UNKNOWN and FROM_LIFECYCLE comments keep the reasoning at half the length. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The module docstring restated the declared-paths contract and the gitlink-not-recurse rule that campaign_save_scope's docstring carries; it now points there. require_clean_gitlink and require_clean_shallow keep their reasons and drop the restatements. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
Six consecutive comment blocks bracketing the nested save scopes restated the contract each scope already documents; each keeps the one fact specific to this site. The footprint-ownership docstring says the rule; the refusal message already carries the detail. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The outer-first save nesting and its reason are stated at add-dataset; copy_lock_to_member now says "as in add-dataset". The flock and retry comments keep the site-specific fact. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The two-pins asymmetry is explained in validate.py and pointed at from the e2e fixtures; the study-lock check is explained at require_study_lock_match and pointed at from inner.py; the datalad.api note lives in utils.py; DEFAULT_BABS's comment is the one place that says why the suite pins babs main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The stamp era, the exit-127 array on a named cluster, and the "first field failure / N unit tests passed" story are each replaced by the invariant the test pins. One site name dropped from a docstring. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The term was not in the glossary and every use unpacks to plain words: a campaign's own label, configs and lock; one lock = one venv; the bundle and lock are fixed at init. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
…kes --study `--study <member>` on a member that .gitmodules registers but that is not installed said "not a member study", which is false, and offered a re-clone, which is wrong; it now says the member is not installed and names `datalad get -n <member>`. This is the first thing anyone hits after a plain clone of a real superstudy. `campaign init --superstudy`'s next-steps hint printed the study form of add-dataset; at a superstudy it needs `--study`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XN3jBoNtQXUo53igCkWdRU
The shell expands ~ only at the start of a word, so the second path in a comma-joined --apps a.yaml,~/b.yaml arrives literal and was reported as not found. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B86TEMruRYUpr5m7oFmzuK
campaign init: expand a literal ~ in a config path
A study accumulates campaigns, and a cell is (source dataset x app) in one campaign. Without the label in the name a second campaign could not produce the same cell until the first campaign's derivative was retired. The label comes last: <Tool>-<Ver>+<stage>[+<id>]+<label>. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add campaign label to derivative name
Member
Author
|
@lobennett this monstrosity is going in now. ill continue working on it with regular-sized prs ;) It should be documented well enough that you can try on your own, but hmu if youd like to meet up for discussion/tour/etc. The thing to watch for on Sherlock will be the glibc issue. CI has the same problem, and campaigns now have env_constraints to help that case. I imagine your env_constraints will look a lot like the CI. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tldr:
.mechababs/campaigns/<label>/— configs, pinned env, statefile), not a standalone dataset. The design contract isdocs/spec.md(landed via Study-first direction — docs RFC, implementation to follow (draft) #101);docs/output_structure.mdcarries the target shape.uvx-first:campaign init·add-dataset·iterate·status·jobs·update-env·retire-derivative·test-cluster. The bootstrap.sh /configure/ wide-ledger path is fully replaced.<App>-<Ver>[+<sourcedata-id>]+<label>, so a new campaign on a study never collides with a previous campaign's derivative and never forces a retire (campaign-per-bump is the working posture).datalad run --explicitrecord in the study's own history — scaffold and merge are dispatched with declared outputs; submit is provably non-mutating and stays plain. A study carries the verbatim, re-executable commands that built its derivatives.iteratefans out in catalog order (--batchis the whole tick's budget), and every level commits its own facts — the member its cells, the super its gitlink + catalog row.pyproject.tomldeclares intent (git-ref pins), the committeduv.lockrecords resolution; the guard is location +uv sync --check;campaign update-envis the supported bump, and a detached member supports rerun-reproduction.mechababs test-clusterships the same suite to validate a site config.Closes
Closes #40 —
babs mergenow runs underdatalad run(the dispatched merge verb, with a decide-recheck guard and the RIA pull-back).Closes #50 — the rewrite lands 419 unit tests over the new CLI;
state/constructfrom the issue's untested list no longer exist.Closes #60 —
merge_configre-homed asmechababs/compose.py; the root script is deleted in the dead-code sweep below.Closes #62 — the container shim is retired: babs resolves images natively from a ReproNim/containers clone (babs#399), validated at runtime on Unity;
require_shimis gone and the shim script goes in the sweep.Closes #63 — superseded and delivered by
campaign update-env(edit therev,update-env, or--upgradefor a moved branch pin).Closes #65 — the e2e now builds real isolated campaign venvs in the container (cluster-axis
env_constraintscover the glibc-2.17 wheels);--system-site-packagesexisted only in the deleted bootstrap path.Closes #81 — configs use babs's
$BABS_TMPDIRplaceholder; the relative-w+--pwdfixup is gone, verified on Unity with no nipype trip.Closes #82 — the per-cell recursive save-walk is gone: dispatch declares its outputs (
--explicit), and the once-per-tick clean check is shallow (--ignore-submodules=dirty).Closes #90 — every campaign commits its
uv.lock(and the pyproject that produced it) as the environment record.Closes #91 —
configureno longer exists;campaign initrecords its work as one labeled transition.Closes #106 — repo-wide ruff format + lint sweep, enforced in CI at pinned ruff 0.16.4 with a pinned rule set (landed via #120).
Closes #126 —
campaign update-envexists (resolved differently than filed: the env stamp was dropped entirely in favor ofuv sync --check, see #129).The model: study-first, and what a campaign is now
The old model made the campaign a datalad dataset that contained everything — code submodules, sourcedata, derivatives, a wide ledger. This branch inverts it: the study is the container, and mechababs' entire footprint in one is additive — new derivatives under
derivatives/, and a campaign record under.mechababs/campaigns/<label>/holding the app + cluster configs, the pinned environment (pyproject.toml+uv.lock), the tall statefile (sourcedata+derivatives.tsv, one row per source-dataset × app cell), and the pinned inclusions. A study accumulates campaigns over time and stays operable standalone: clone it, rebuild the venv from the lock, and everything needed to reproduce or extend it is inside.State stays derived, not declared: a pure
route()reads the shard's columns (babs,merged) and bothiterateandstatusroute through it, so there is one reading of what a cell's state is. Identity columns are inputs and never overwritten.The verbs, and the provenance they leave
Outer commands (
iterate,add-dataset,status,retire-derivative,update-env) enforce selection (MECHABABS_CAMPAIGN, set by the campaign'senv.sh), the configured-level rule, and the env-match guard. The mutating transitions are delegated to a hidden inner CLI (mechababs-inner scaffold|submit|merge) dispatched viadatalad run --explicit -d <study>with per-verb declared outputs — so the study's git history holds each transition as a verbatim command thatdatalad reruncan replay, and the inner verbs carry their own cell-state and study-local env checks so a replay validates itself wherever it runs.Per-verb commit shapes are deliberate: scaffold leaves its outputs for the run record to commit (one clean RUNCMD); submit was verified non-mutating and runs plain (
dispatch.plainrefuses if the tree or HEAD moved); merge moves subdataset HEADs mid-command, so it lands as the record-wrapping merge-commit bubble.add-dataset,update-envandretire-derivativeare labeled plain saves — each one's docstring argues why a run record there would be a false promise (resolution against the live world; content deletion; host paths that resolve nowhere else).Chained cells: an app config's
depends_onnames its producer by stem; a dependent cell waits (visibly) until the producer's cell merges, thenresolve_input_originswires the producer's derivative in as a babs input.Superstudy
campaign init --superstudy <path>creates-or-adopts a study-of-studies; its campaign dir carries a membership catalog (studies+sourcedata.tsv) instead of a statefile — per-cell state shards to the members.add-dataset --study <member|url>selects (or clones) a member, writes its campaign footprint (configs seeded once, lock copied down) and a marker recording the super's datalad-id — an identity, so a member cloned elsewhere is correctly refused rather than silently adopted.iteratefans out in catalog order; only installed members advance; one member's drift stops that member, not the fan-out; the single-writer flock lives at the operated level. A campaign is operated only from its configured level — a member refuses, with no override; what a detached member supports is reproduction, which needs no flag.Environment story
campaign initgenerates a campaignpyproject.toml(mechababs self-pinned from its own PEP 610 record; babs defaulting to the latest release,--babs URL@REFto pin a branch; datalad ≥ 1.6 floor) and locks + syncs a private venv. The lock is committed: it is the environment provenance. The guard asks two questions — is this the campaign's venv (location), and does it match the study's committed lock (uv sync --check) — andcampaign update-envis the one verb that runs without it, because its job is exactly to restore that match (fresh clone, wiped site, historical checkout during rerun, or a deliberate bump via a pyproject edit /--upgrade). At a superstudy,update-env --study <member>copies the resulting lock into that member's footprint — the acknowledgment that its remaining work moves onto the new environment.Cluster configs can carry
env_constraints(uvconstraint-dependencies), which is how an old-glibc site (slurm-docker-ci, Sherlock) resolves to versions with usable wheels.Deleted, and what replaced it
bootstrap.sh+configure+construct.py+guard.py→campaign init. The wide ledger (state.py) + olditerate.py/status.py/retire.py→ the tall shard,route(), and the rewritten verbs.merge_config.py→compose.py.run_on_cluster.sh+ the old--campaign-pathtest-cluster →test-cluster --scratch-path, runnable viauvxfrom anywhere. The container shim → native ReproNim/containers resolution (babs#399). Each deletion happened in the chunk that replaced it, with grep evidence in the chunk PR; the stranded roots (bootstrap.sh,merge_config.py, the shim script,preflight.py,validate.clone_ref) went in the dead-code sweep (#140), vulture + grep over package/tests/docs/examples.How it was verified
--app,jobs), Track the lifecycle of member studies in the superstudy #132 (member lifecycle), Docs sweep #133 (docs audit), Claude md strip #134 (CLAUDE.md strip), Flock the operating level (study or superstudy) #136 (flock at the study), iterate: one super commit per tick, recorded before the next cell #137 (per-tick super save), ci: run the unit, uv-contract and e2e suites on every PR #138 (CI), campaign init takes the level's single-writer lock #139 (campaign inittakes the lock), Dead-code sweep: empty the known-dead list and hunt the rest #140 (dead-code sweep), campaign init: expand a literal ~ in a config path #141 (~in--apps), Add campaign label to derivative name #142 (campaign label in the derivative name).Temporary pins and caps (each tracked, none load-bearing forever)
datalad run); reverts to the transitive ≥ 1.6 floor when a release carries it.python <3.14: h5py (via babs → niworkflows → nitransforms) ships no cp314 wheel; lift when it does.Before merge
--derivative→--apprename (iterate,status,jobs) — the flag takes an app config's stem, not a derivative name, and a rename has to happen before the name is public (Status rollup #131)docs/spec.mdline by line against what was built; fix the code or rewrite the linereference.md,overview.md,interventions.md, quickstart, tutorial, install/contributor docs (Docs sweep #133)bootstrap.sh,merge_config.py, the shim script,validate.clone_ref(Dead-code sweep: empty the known-dead list and hunt the rest #140)The pre-rewrite
mainis not tagged; this PR's first parent is the last commit that runs the old substrate.Follow-up issues are filed after the merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01U3EMv3ByezXDHCHJrTcsqN