Repository navigation
feat(pipeline)!: run a job asynchronously in a notebook; job.run returns a run summary of failed rows (ITL-994, ITL-997) #337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 9 commits into
main
from
eywalker/itl-994-a-pipelinejob-cannot-be-run-with-the-async
Nov 13, 2026
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
a54ba46
test(pipeline): WIP failing tests for a job run in a running loop and…
5c9db2a
Merge origin/main
6433573
fix(pipeline): run a job with the async orchestrator inside a running…
66bbcf0
feat(pipeline)!: job.run returns a run summary and warns when rows fa…
a05064c
Merge origin/main
f28bfab
fix(pipeline): address review of the bridged async run and the run su…
fcad8f2
Merge origin/main
f532523
test(pipeline): leave per-run source-info columns out of the node tab…
f0114fd
fix(pipeline): warn after every failed run; clean up a pod's leftover…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| ### Added | ||
|
|
||
| - **`PipelineJob.run_async`** — `await job.run_async()` runs a job with the | ||
| `AsyncPipelineOrchestrator` (its default) on the caller's event loop, for async code and | ||
| notebook cells. It takes the same arguments as `run`; a `SyncPipelineOrchestrator` raises | ||
| `TypeError`. (ITL-994) | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **`job.run(orchestrator=AsyncPipelineOrchestrator())` works inside a running event loop**, | ||
| such as a Jupyter kernel or async code. It used to raise | ||
| `RuntimeError: asyncio.run() cannot be called from a running event loop`. Inside a loop, the | ||
| run now goes on a separate thread while the call waits, so the same line works in a script | ||
| and in a notebook; it blocks that loop until the run ends (use `await job.run_async()` in | ||
| async code). The interrupt button cancels the run and waits for functions in the middle of | ||
| a row to return; pressing it again stops waiting at once, with a warning naming the node | ||
| whose function has not returned. Nothing more is written to the store when it returns (an | ||
| `async def` function that blocks the event loop is the exception, and the warning says | ||
| so). Stored rows, record ids and hashes are the same as a run outside a loop. (ITL-994) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| - **`PipelineJob.run()` returns a `RunSummary`, not the job.** Results are read from the | ||
| job itself, as before (`job.nodes[...]`). Code written `job = job.run()` becomes | ||
| `job.run()`; code that kept the return value as the job (`completed = job.run()`, | ||
| `completed.nodes[...]`) uses the job instead. `run_async()` returns the same summary. | ||
| (ITL-997) | ||
|
|
||
| ### Added | ||
|
|
||
| - **`RunSummary`** (`orcapod.pipeline`) — what a job run did, per function node: rows | ||
| computed, rows read from the store, rows failed, and the first error (`NodeRunSummary`). | ||
| Shown as a table at the end of a notebook cell and as text by `print`; kept as | ||
| `job.last_run`. `job.last_run.print_traceback("<node>")` prints the full traceback of a | ||
| node's first failed row. (ITL-997) | ||
| - **`RowFailuresWarning`** (`orcapod.errors`) — issued once at the end of a job run in which | ||
| rows failed (with the default `error_policy="continue"`), naming each node with failed | ||
| rows, how many failed out of how many, the first error, and how to see its traceback. A | ||
| run with no failed rows issues none. (ITL-997) | ||
|
|
||
| ### Changed | ||
|
|
||
| - **A failed row no longer logs its traceback at WARNING on the sync orchestrator.** The | ||
| per-row log is at DEBUG, as on the async orchestrator, which logged nothing visible: a | ||
| function failing on every row printed one traceback per row on the sync path and nothing | ||
| at all on the async path. The run summary and `RowFailuresWarning` now report failures the | ||
| same way on both. `error_policy="fail_fast"` is unchanged. (ITL-997) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 9403
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 16423
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 5830
Keep a failure report when
FunctionJobNoderuns outside a job.When
run()orexecute()runs without an observer, each failed row is logged only at DEBUG and skipped.PipelineJobis the only place that issues aRunSummaryorRowFailuresWarning, so this path reports nothing at default log levels.run()returnsNone.execute()returns only the rows that succeeded, so with every row failing it can return an empty list with no warning.Keep a WARNING-level report for failed rows on this path. Lower the level only where the job-level summary reports those failures.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declined. Lowering the sync per-row log to DEBUG was a developer decision (November 12), made so the sync and async paths match. The standalone async path already logged at DEBUG. Keeping WARNING only outside a job would need the node to know whether a job is reporting its failures. The gap you describe is real: a FunctionJobNode run on its own, outside
PipelineJob.run, gives no visible failure signal on either orchestrator. It is reported to the coordinator as a possible follow-up rather than changed here.Posted by a Claude Code agent on behalf of @eywalker.