Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Don't narrate the transition in a doc ("this used to be X"); write the current s
## Conventions

- **Wrap runs in duct.** Every run should carry con/duct usage/resource logs alongside its outputs; `con-duct` is a dependency of every campaign environment.
The wiring is tracked, not yet built: duct-wrapping each `iterate` step is #53, and duct inside the babs jobs is #16 (tracks `PennLINC/babs#356`).
Every recorded transition is wrapped by `dispatch()`, so a new recorded verb gets it for free; duct inside the babs jobs is #16 (tracks `PennLINC/babs#356`).
- **No untracked-local paths in upstream-facing stuff** (issues, tracked docs).
A gitignored path means nothing to a reader on GitHub β€” **strip the path, keep the intent** (e.g. "the resample question in our fmriprep meeting notes is stale", not the path); remove it at filing time.
- **Configs are the user's, never the tool's.** `examples/` are starters to copy; real site paths never land there, and nothing in the package resolves a config by name.
Expand Down
12 changes: 12 additions & 0 deletions docs/interventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,18 @@ Nothing is written; the flag is this iterate's reading of babs's live counts, an
`mechababs status` shows the cell as `FAILED`, and `mechababs jobs --failed` lists each failed job with its subject, SLURM id, and the directory its logs are in.
Read the log, decide what happened, then pick one of the two repairs below.

A transition itself can also fail or refuse: a `babs init` that dies partway, a member study whose lock copy is behind the superstudy's, a recorded command rerun by hand against a cell that has moved on.
The `datalad run` then saves nothing, and the con/duct records of the attempt are left untracked in the campaign dir's `duct/<cell>/`, so the next `iterate` refuses to start on the dirty study until you have looked.
Its `stderr` and `stdout` files hold what the transition printed; `info.json` holds its exit code, wall time and peak memory.
Keep them, so the study records that the attempt happened, or delete them; either way the reconciler resumes.

```bash
datalad save -r -d . -m "duct records of the refused scaffold" .mechababs/campaigns/<label>/duct/<cell> # at a study
datalad save -r -d . -m "duct records of the refused scaffold" <member>/.mechababs/campaigns/<label>/duct/<cell> # at a superstudy
```

The superstudy form is path-scoped through the member, so it lands one commit in the member and one at the superstudy; a save in the member alone would leave the superstudy's gitlink behind, which the next iterate reads as drift.

## Per-job surgery: repair a derivative in place

When a job failed for a reason a human has to fix, such as an out-of-memory kill needing more memory, the derivative is repaired rather than redone.
Expand Down
7 changes: 7 additions & 0 deletions docs/output_structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ study-<id>/
uv.lock # the resolved, reproducible environment
sourcedata+derivatives.tsv # the statefile: this study's cells for this campaign
inclusions/ # the requested subject list per cell, pinned at scaffold
duct/<cell>/ # con/duct records of each scaffold and merge of that cell
```

A study can hold **more than one** `sourcedata/<id>`; a campaign selects which to process (the `(study, sourcedata)` pair is the coarse selection; subjects/configs are the fine selection).
Expand All @@ -58,6 +59,12 @@ So a mechababs derivative is `<Tool>-<Ver>+<stage>[+<id>]+<label>/` (e.g. `fMRIP
`.mechababs/campaigns/<label>/` is where a study records each campaign that touched it: one pinned environment (`uv.lock` fixes `mechababs` + `babs` by git commit β€” a fork is just a different URL), one bundle of BIDS-App configs, and the state of that study's cells.
A study **accumulates** campaigns over time β€” a set of derivatives now, another a year later with newer tools, each its own `<label>` β€” and because the record is the study's own, the study stays operable standalone: clone it, `uv sync` the campaign's lock, and everything needed to add to it or reproduce it is inside.

**Each transition's cost is recorded beside its provenance.**
`duct/<cell>/` holds the [con/duct](https://github.com/con/duct) records of every scaffold and merge of that cell: `<verb>_<datetime>_{info.json,usage.jsonl,stdout,stderr}`, written inside the transition's `datalad run` and committed by it, so the study carries how long and how heavy each orchestration step was, not only that it happened.
`<cell>` is the same `<sourcedata>_<app>` key that names the inclusion pin.
The timestamp keys attempts: a redone cell or a `datalad rerun` adds its records beside the earlier ones rather than replacing them, so one directory reads as the cell's whole orchestration history.
The records babs's jobs produce, the compute side, live inside the derivative.

**Retired derivatives live outside the study.**
A derivative that had to be redone (a resource change, a tool bug, a config fix) is retired, not deleted: `mechababs retire-derivative` moves it to a required target directory that must be **outside the study** (a destination inside is refused) and resets its cell in the same transition.
The evidence β€” logs, git history, `datalad run` records of *why* it was redone β€” is kept, and because the archive is outside the study, retired attempts never travel with a published study.
Expand Down
1 change: 1 addition & 0 deletions docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ A missed event in an event-driven system is permanent drift; a level-triggered l
## Provenance

Every change-making transition is recorded with `datalad run` at the study, so the study's git history holds the exact command that scaffolded or merged each derivative.
The recorded command runs under [con/duct](https://github.com/con/duct), so the same run also commits how long and how heavy the transition was, in the campaign dir beside the cell's inclusion pin.
The jobs themselves are recorded by babs inside the derivative, as `datalad run` records of each `singularity run`.
Two record homes: orchestration in the study, compute in the derivative.

Expand Down
1 change: 1 addition & 0 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ At a superstudy the iterate runs over every installed member in study catalog or
A member study that has been uninstalled is skipped, never reinstalled.

Scaffold and merge are recorded with `datalad run` at the study, so the study's git history carries the command that produced each derivative; submit changes nothing tracked and is not recorded.
Each recorded transition runs under [con/duct](https://github.com/con/duct), and its usage and resource records are committed by the same run at `.mechababs/campaigns/<label>/duct/<sourcedata>_<app>/<verb>_<datetime>_*`, one directory per cell keyed like its inclusion pin; `con-duct ls` over that directory is how a campaign's orchestration cost is read.
At a superstudy each scaffold or merge is also committed at the superstudy, as the member's moved gitlink, one commit per tick and before the next cell is attempted.
An iterate takes the level's single-writer lock, and refuses to start on a dirty tree.

Expand Down
3 changes: 2 additions & 1 deletion docs/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ Entries marked **Open** are named holes we have not decided.
- **Every save says what happened, not how much.** A save message names the tick it records β€” the cell and its transition β€” and the lifecycle change, if any, that followed from it; never a bare count of cells. A count is what a save site writes when it has been handed a number instead of the work, and it leaves the campaign's own history unreadable to the git-only reader the committed lifecycle exists for.
- `iterate` itself is a plain coordinator, **not** wrapped in a `datalad run` β€” it dispatches one run per advancing cell, so there is no same-level nesting.
- The run is at the **study level** (`-d <study>`), where the derivative and statefile live. At a superstudy, `iterate` dispatches per-member study-level runs β€” the run's `-d` is always the (lone or member) study.
- **Mechanics:** the inner run is `datalad run --explicit -d .` with cwd inside the study. The cwd makes the recorded command study-relative (portable, re-executable records); `--explicit` with each verb's declared output paths (the derivative, the statefile, the inclusion pin) keeps the run from deep-walking `sourcedata/raw` β€” datalad's dirty gate and save discovery ignore all submodule-ignore config, so path scoping is the only route. Because `--explicit` captures only declared outputs, a **shallow clean check** (gitlink compare, no descent into submodule worktrees) is the loud backstop for stray side-writes: once at the top of an iterate, and again before each dispatch, at the study it is about to change. At a superstudy the top-level check covers the super's own tree and **excludes the members** β€” each is asked about separately, by a gitlink compare immediately before it is advanced, so nothing is checked twice, one member's drift stops that member rather than the whole fan-out, and the cost stays flat instead of growing with the number of members.
- **Mechanics:** the inner run is `datalad run --explicit -d .` with cwd inside the study. The cwd makes the recorded command study-relative (portable, re-executable records); `--explicit` with each verb's declared output paths (the derivative, the statefile, the inclusion pin, the cell's duct directory) keeps the run from deep-walking `sourcedata/raw` β€” datalad's dirty gate and save discovery ignore all submodule-ignore config, so path scoping is the only route. Because `--explicit` captures only declared outputs, a **shallow clean check** (gitlink compare, no descent into submodule worktrees) is the loud backstop for stray side-writes: once at the top of an iterate, and again before each dispatch, at the study it is about to change. At a superstudy the top-level check covers the super's own tree and **excludes the members** β€” each is asked about separately, by a gitlink compare immediately before it is advanced, so nothing is checked twice, one member's drift stops that member rather than the whole fan-out, and the cost stays flat instead of growing with the number of members.
- **The dispatched command is `duct … mechababs-inner …`: every recorded transition runs under con/duct, inside the run rather than around it.** The usage and resource records are then outputs of the run and travel with the study the way the inclusion pin does, at `.mechababs/campaigns/<label>/duct/<sourcedata>_<app>/<verb>_<datetime>_*` β€” one directory per cell, keyed like the pin, so scaffold and merge and every later attempt at either read out of one place. The prefix is study-relative (an absolute one would not re-execute elsewhere) and keeps duct's `{datetime}` template, escaped once because datalad format-expands the command it records; so the record holds the template, and a `datalad rerun` or a retire-and-redo adds a timestamped set beside the earlier one instead of clobbering it. The directory rather than the files is what each verb declares, since the names are not known until the run starts. `--fail-time 0` keeps a failed transition's records; the run does not save them, so they sit untracked and the next iterate refuses until a human has looked β€” failures stop, applied to the orchestration itself. Submit is not wrapped, for the reason it is not recorded. The wrap lives in the one dispatch helper, not at the call sites, so a new recorded verb cannot forget it.
- **The scaffold verb declares `.gitmodules` among its outputs.** Registering the new derivative commits the registration mid-command, and datalad's undeclared-committed-path guard (`datalad.run.dirty-committed=error`) would refuse the run; declaring `.gitmodules` satisfies the guard while keeping it armed for genuinely unexpected mid-command commits β€” chosen deliberately over muting it.
- **Two record homes: orchestration β†’ study, compute β†’ derivative.** `datalad run -d <study> <action>` records the **orchestration** in the study (the derivative is an *output subdataset* of that run β€” the study-level diff is "new/updated subdataset + statefile"). `datalad run -d <derivative> singularity …` (the babs job) records the **compute** in the derivative. Different `-d`, different dataset levels, no nesting conflict.
- **Ordering is unchanged β€” no priority mechanism in this implementation.** Advance in default order (at a superstudy, members in catalog order; within a shard, cells in row order), bounded by `--batch`, with `depends_on` the only hard constraint. `--batch N` means the same thing at either level β€” at most N ticks per iterate, full stop; a cell that is passed over is not a tick β€” so at a superstudy it is the *iterate's* budget rather than each member's, spent in catalog order. That is what makes catalog order a priority interface and not only an ordering: `--batch 5` advances the five most important cells in the superstudy, wherever they live. The overrides are `iterate --study` (concentrate on one member) and `--app` (narrow further, to one app's cells).
Expand Down
8 changes: 8 additions & 0 deletions mechababs/campaign.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@
APPS_DIRNAME = "bids-app-configs"
CLUSTERS_DIRNAME = "clusters"
INCLUSIONS_DIRNAME = "inclusions"
# con/duct's usage and resource records of each dispatched transition, one
# subdirectory per cell beside the inclusion pins: the study's own evidence of what
# scaffolding and merging a cell cost, written inside the transition's `datalad run`.
DUCT_DIRNAME = "duct"
ENV_FILENAME = "env.sh"
PYPROJECT_FILENAME = "pyproject.toml"
UV_LOCK_FILENAME = "uv.lock"
Expand Down Expand Up @@ -347,6 +351,10 @@ def inclusions_dir(root, label):
return campaign_dir(root, label) / INCLUSIONS_DIRNAME


def duct_dir(root, label):
return campaign_dir(root, label) / DUCT_DIRNAME


def env_path(root, label):
return campaign_dir(root, label) / ENV_FILENAME

Expand Down
Loading
Loading