Skip to content

Pipeline: add an optional 'if' and 'after' parameters for conditional tasks and better control flow#345

Draft
ljgray wants to merge 5 commits into
mainfrom
ljg/pipeline-if
Draft

Pipeline: add an optional 'if' and 'after' parameters for conditional tasks and better control flow#345
ljgray wants to merge 5 commits into
mainfrom
ljg/pipeline-if

Conversation

@ljgray

@ljgray ljgray commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

if

Useful for templates pipelines where we might want to skip a task under certain conditions, without messing up key inheritance. The pipeline will internally replace the conditional task with a NoOp if the conditional is False, in which case data keys are simply passed through. Naturally, this will fail for a task which produces a different number of outputs than it takes as input, or which does not intend to pass through the input, so this can likely be made to be more robust.

after

Effectively just streamlines the WaitUntil task - adds a set of data product keys (similar to in, out, and requires) which indicate data products that must exist before a task can run, but are not actually consumed by the task.

IO

Adds two extra tasks:

  • Simple task to load all files from a list at once
  • More complicated task to load batches of files together. This ends up being necessary because the pipeline never seems producer-only tasks as being "available", so they are simply run in order when there is nothing else to do. However, the order always starts at the first possible task, so two back-to-back IO tasks will result in the pipeline first loading every file for the first task before proceeding.

@ljgray
ljgray force-pushed the ljg/pipeline-if branch 5 times, most recently from 9763123 to face599 Compare July 14, 2026 19:23
@ljgray
ljgray force-pushed the ljg/pipeline-if branch 2 times, most recently from 510ddc3 to f4c13a3 Compare July 14, 2026 20:24
@ljgray
ljgray force-pushed the ljg/pipeline-if branch from f4c13a3 to e325e4a Compare July 14, 2026 21:09
@ljgray ljgray changed the title Add an optional 'if' parameter for conditional tasks Pipeline: add an optional 'if' and 'after' parameters for conditional tasks and better control flow Jul 14, 2026
@ljgray
ljgray force-pushed the ljg/pipeline-if branch from e325e4a to cabff84 Compare July 14, 2026 21:18
@ljgray
ljgray force-pushed the ljg/pipeline-if branch 3 times, most recently from b53841a to 61e7b6d Compare July 14, 2026 22:58
@ljgray
ljgray marked this pull request as draft July 14, 2026 23:16
@ljgray
ljgray force-pushed the ljg/pipeline-if branch from 61e7b6d to 99b3d17 Compare July 15, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant