Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Ordered from the outside in: the objects on disk first, then the campaign that a
A bundle of app configs, a cluster config, and a locked environment pinning mechababs and babs together.
Lives in a campaign dir inside a study or superstudy; it is not a dataset.
- **label**: a campaign's name and identity.
It names the campaign dir `.mechababs/campaigns/<label>/` and the venv inside it.
It names the campaign dir `.mechababs/campaigns/<label>/` and the venv inside it, and ends the name of every derivative the campaign produces.
Selected by sourcing the campaign's `env.sh`, which sets `MECHABABS_CAMPAIGN`.
- **campaign dir**: `.mechababs/campaigns/<label>/` at a level.
Holds the config copies, `pyproject.toml`, `uv.lock`, and the statefile or study catalog.
Expand Down
9 changes: 5 additions & 4 deletions docs/output_structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ study-<id>/
sourcedata/<id>/ # submodule -> raw BIDS dataset (a study may hold more than one)
derivatives/
<Tool>-<Ver>/ # pre-existing derivatives (any tool)
<Tool>-<Ver>+<stage>/ # a new babs derivative (see "a derivative dataset")
<Tool>-<Ver>+<stage>+<id>+<label>/ # a new babs derivative (see "a derivative dataset")
.mechababs/ # mechababs' record — bidsignored, NOT a dataset
campaigns/
<label>/ # one campaign's record in this study
Expand All @@ -51,7 +51,8 @@ study-<id>/
A study can hold **more than one** `sourcedata/<id>`; a campaign selects which to process (the `(study, sourcedata)` pair is the coarse selection; subjects/configs are the fine selection).
When the study holds exactly one raw BIDS dataset, the generic slots `sourcedata/raw` or `sourcedata/rawbids` are preferred; `sourcedata/<id>` covers the multiple-datasets case.
Derivative directory names follow the upstream convention — `<Tool>-<Ver>` in the tool's own casing (`fMRIPrep-25.1.1`, `MRIQC-24.0.2`) — plus `+<stage>` where a run has stages (`fMRIPrep-25.1.1+anat`).
When the sourcedata slot is a generic one (`sourcedata/raw`, `sourcedata/rawbids`) that suffix is enough; otherwise the derivative also carries the source-dataset id — `<Tool>-<Ver>+<stage>+<id>/` (e.g. `fMRIPrep-25.1.1+anat+ds000001`) — since a cell is (source dataset × app config) and the name would collide when a study holds several source datasets.
Unless the sourcedata slot is a generic one (`sourcedata/raw`, `sourcedata/rawbids`) the derivative also carries the source-dataset id, since a cell is (source dataset × app config) and the name would collide when a study holds several source datasets; and it always ends in the campaign label, since a study accumulates campaigns and a new one must be able to produce the same cell beside the old.
So a mechababs derivative is `<Tool>-<Ver>+<stage>[+<id>]+<label>/` (e.g. `fMRIPrep-25.1.1+anat+ds000001+c1`).

**A campaign is a config-epoch run, not a dataset.**
`.mechababs/campaigns/<label>/` is where a study records each campaign that touched it: one pinned environment (`uv.lock` fixes `mechababs` + `babs` by git commit — a fork is just a different URL), one bundle of BIDS-App configs, and the state of that study's cells.
Expand Down Expand Up @@ -99,14 +100,14 @@ A study-of-studies is also a gap in BIDS — BIDS describes no study containing

---

## `<Tool>-<Ver>+<stage>/` — a derivative dataset
## `<Tool>-<Ver>+<stage>[+<id>]+<label>/` — a derivative dataset

The unit of work, one per (source dataset × bids-app-config) cell, tracked by the study's statefile.
This is the babs project: `babs init` targets this path, and its root is the derivative's root.
The BIDS app writes `dataset_description.json` and `sub-*` here.

```
<Tool>-<Ver>+<stage>/ # e.g. fMRIPrep-25.1.1+anat; fMRIPrep-25.1.1+anat+ds000001 when the source dataset is named
<Tool>-<Ver>+<stage>[+<id>]+<label>/ # e.g. fMRIPrep-25.1.1+anat+ds000001+c1; the id only when the source dataset is named
dataset_description.json # DatasetType "derivative"; GeneratedBy [<bids_app>]
.bidsignore # containers/, logs/, prov/
sub-* # unzipped derivative content
Expand Down
4 changes: 2 additions & 2 deletions docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ superstudy/ # optional
study-ds000001/ # a member study
sourcedata/ds000001/ # a source dataset
derivatives/
MRIQC-24.0.2+ds000001/ # a derivative mechababs added
fMRIPrep-25.2.5+anat+ds000001/
MRIQC-24.0.2+ds000001+c1/ # a derivative mechababs added, by campaign c1
fMRIPrep-25.2.5+anat+ds000001+c1/
.mechababs/campaigns/<label>/ # the campaign's record in this study
study-ds000002/
.mechababs/campaigns/<label>/ # the campaign's record at the superstudy
Expand Down
2 changes: 1 addition & 1 deletion docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ Edit them there and commit; a change reaches only cells not yet scaffolded, sinc

### App configs

The filename stem is the app's identity: what `--app` takes, what `depends_on` names, and part of the derivative's name (`MRIQC-24.0.2+ds000001`).
The filename stem is the app's identity: what `--app` takes, what `depends_on` names, and the first part of the derivative's name (`MRIQC-24.0.2+ds000001+c1`: app, source dataset, campaign label).
The `mechababs:` block at the top is read by mechababs and stripped before the config reaches babs; everything else is babs container-config, passed through.

```yaml
Expand Down
4 changes: 2 additions & 2 deletions docs/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Entries marked **Open** are named holes we have not decided.
- **Input is an existing BIDS study.** mechababs operates on a study that already exists on disk (cloned, or authored by another tool); it does not create one in this implementation. Not ruled out later — see the study-authoring gap in [output_structure.md](output_structure.md).
- **The expected study input includes the per-subject metadata TSV** (per-subject `datatypes` / `t1w_num` / `bold_num`) that selection and the `add-dataset` sniff read to apply eligibility rules. For this implementation it is expected present, as OpenNeuroStudies provides. Generating it when absent is follow-up generalization work. Eligibility rules are unchanged.
- **The container dataset is an input, not something mechababs vendors.** An app config's `container.source` is handed to `babs init --container-ds` as-is when it is a URL (what a production config carries), and otherwise resolved as a filesystem path — a relative one against the **study root**, so a config can name a sibling checkout without committing an absolute path. There is no copy-into-the-campaign step: babs installs the container dataset into each derivative, the way it installs any input. This works against a container dataset in its own native layout (ReproNim/containers' `images/<collection>/<app>--<version>.sif`) because babs resolves the image from its datalad-containers registration rather than a fixed path — `PennLINC/babs#399`, which is why the pinned babs matters here.
- **Derivative naming carries the source-dataset id when the sourcedata is named.** A cell is (source dataset × app config), and `<Tool>-<Ver>+<stage>` alone would collide when a study holds several source datasets. When the sourcedata uses a generic slot (`sourcedata/raw`, `sourcedata/rawbids`) no id is needed; otherwise the derivative is `<Tool>-<Ver>+<stage>+<id>/` (e.g. `fMRIPrep-25.1.1+anat+ds000001`).
- **Derivative naming carries the source-dataset id when the sourcedata is named, and always the campaign label.** A cell is (source dataset × app config) in one campaign, and the name carries each part that can differ: `<Tool>-<Ver>+<stage>` alone would collide when a study holds several source datasets, and a study accumulates campaigns, so without the label a new campaign could not produce the same cell until the previous campaign's derivative was retired. When the sourcedata uses a generic slot (`sourcedata/raw`, `sourcedata/rawbids`) no id is needed. So the derivative is `<Tool>-<Ver>+<stage>[+<id>]+<label>/` (e.g. `fMRIPrep-25.1.1+anat+ds000001+c1`), the label last so the upstream `<Tool>-<Ver>` prefix stays and the `+` chain reads app, then source, then campaign.

## Campaign definition

Expand All @@ -55,7 +55,7 @@ Entries marked **Open** are named holes we have not decided.
`app_config` is the **campaign-relative config path** (`bids-app-configs/MRIQC-24.0.2.yaml`), the same identity `campaign.yaml`'s `apps:` list carries; the filename stem is only a derived, human-facing form (display, derivative naming). Internal — the user names configs only at `campaign init`, by full path or URL.
`depends_on` is *declared* in an app config by the producer's stem (the natural name to write) and stored resolved to the producer's path, so both identity columns hold the same kind of thing.
`source_dataset` is the sourcedata's **study-relative path** (`sourcedata/ds000001`), not a bare id: it is what the user named, it is what scaffold wires into babs, and it does not assume the sourcedata sits at `sourcedata/<id>`.
When a derivative carries an `+<id>` suffix, the id is that path's last component — and the generic slots (`sourcedata/raw`, `sourcedata/rawbids`) carry none, per the naming rule in "Layout & input".
When a derivative carries an `+<id>` part, the id is that path's last component — and the generic slots (`sourcedata/raw`, `sourcedata/rawbids`) carry none, per the naming rule in "Layout & input".
`processing_level` is read from the metadata rows themselves — session-level exactly when they name real sessions — rather than from which of the two metadata filenames the study happens to use.
`n_sessions` counts sessions across all subjects, and is blank rather than 0 for a subject-level dataset.
- topology (derived from the app config): `depends_on` (the upstream `app_config`, or empty).
Expand Down
10 changes: 5 additions & 5 deletions mechababs/dispatch.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,18 +101,18 @@ def scaffold_outputs(study, label, source_dataset, app_config):
"""
pin = scaffold_mod.inclusion_pin(study, label, source_dataset, app_config)
return [
scaffold_mod.derivative_path(source_dataset, app_config),
scaffold_mod.derivative_path(source_dataset, app_config, label),
str(campaign_mod.state_path(study, label).relative_to(study)),
str(pin.relative_to(study)),
GITMODULES,
]


def scaffold_message(source_dataset, app_config):
def scaffold_message(source_dataset, app_config, label):
return (
f"mechababs scaffold {source_dataset} "
f"{scaffold_mod.app_stem(app_config)} -> "
f"{scaffold_mod.derivative_path(source_dataset, app_config)}"
f"{scaffold_mod.derivative_path(source_dataset, app_config, label)}"
)


Expand All @@ -127,7 +127,7 @@ def merge_outputs(study, label, source_dataset, app_config):
covers it.)
"""
return [
scaffold_mod.derivative_path(source_dataset, app_config),
scaffold_mod.derivative_path(source_dataset, app_config, label),
str(campaign_mod.state_path(study, label).relative_to(study)),
]

Expand Down Expand Up @@ -249,7 +249,7 @@ def scaffold(study, label, source_dataset, app_config, *, dry_run=False):
study,
inner_command("scaffold", label, source_dataset, app_config),
outputs=scaffold_outputs(study, label, source_dataset, app_config),
message=scaffold_message(source_dataset, app_config),
message=scaffold_message(source_dataset, app_config, label),
dry_run=dry_run,
)

Expand Down
32 changes: 19 additions & 13 deletions mechababs/scaffold.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,21 +49,25 @@ def source_id(source_dataset):
return Path(source_dataset).name


def derivative_name(source_dataset, app_config):
"""The derivative directory: ``<app stem>`` (+``+<id>`` for a named sourcedata).

A cell is (source dataset x app config), so ``<Tool>-<Ver>+<stage>`` alone
would collide the moment a study holds two source datasets. A generic slot
carries no id — its directory name is not one.
def derivative_name(source_dataset, app_config, label):
"""The derivative directory: ``<app stem>[+<id>]+<label>``.

A cell is (source dataset x app config) in one campaign, and the name carries
each part that can differ. ``<Tool>-<Ver>+<stage>`` alone would collide the
moment a study holds two source datasets; a generic slot carries no id, since
its directory name is not one. The campaign label comes last: a study
accumulates campaigns, and without it a new campaign could not produce the
same cell without first retiring the previous campaign's derivative.
"""
stem = app_stem(app_config)
dsid = source_id(source_dataset)
return stem if dsid in GENERIC_SOURCEDATA_SLOTS else f"{stem}+{dsid}"
name = stem if dsid in GENERIC_SOURCEDATA_SLOTS else f"{stem}+{dsid}"
return f"{name}+{label}"


def derivative_path(source_dataset, app_config):
def derivative_path(source_dataset, app_config, label):
"""The derivative's STUDY-RELATIVE path — what the ``babs`` column records."""
return f"derivatives/{derivative_name(source_dataset, app_config)}"
return f"derivatives/{derivative_name(source_dataset, app_config, label)}"


def inclusion_pin(study, label, source_dataset, app_config):
Expand Down Expand Up @@ -257,7 +261,7 @@ def resolve_inclusion(study, label, row, app_config_data, limit):
return pin


def babs_init_command(study, row, app_config_data, inclusion, babs_config):
def babs_init_command(study, row, project, app_config_data, inclusion, babs_config):
"""The ``babs init`` argv for this cell, with study-relative paths.

Study-relative because the run's cwd is the study: the recorded command has to
Expand All @@ -277,7 +281,7 @@ def babs_init_command(study, row, app_config_data, inclusion, babs_config):
cmd = [
campaign_mod.babs_bin(),
"init",
derivative_path(row["source_dataset"], row["app_config"]),
project,
"--container-ds",
resolve_container_ds(study, container),
"--container-name",
Expand Down Expand Up @@ -329,7 +333,7 @@ def scaffold(study, label, source_dataset, app_config):
require_producer_merged(rows, row)
input_origins = resolve_input_origins(study, label, rows, row, app_config_data)

project = derivative_path(source_dataset, app_config)
project = derivative_path(source_dataset, app_config, label)
print(
f"\n=== scaffold {source_dataset} / {app_stem(app_config)} -> {project} ===",
file=sys.stderr,
Expand All @@ -355,7 +359,9 @@ def scaffold(study, label, source_dataset, app_config):
input_origins=input_origins,
campaign_venv=campaign_mod.venv_path(operated_at, label),
)
cmd = babs_init_command(study, row, app_config_data, inclusion, babs_config)
cmd = babs_init_command(
study, row, project, app_config_data, inclusion, babs_config
)
print("+ " + " ".join(str(c) for c in cmd), file=sys.stderr)
subprocess.run([str(c) for c in cmd], cwd=str(study), check=True)

Expand Down
Loading
Loading