Repository navigation
fix(async_orchestrator)!: honour error_policy; an async run raises the failing node's own exception (ITL-879) #379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 9 commits into
main
from
eywalker/itl-879-async-execution-ignores-error_policy
Nov 14, 2026
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
3219f5b
test(async_orchestrator): failing tests for error_policy on the async…
14b8182
fix(async_orchestrator)!: honour error_policy; an async run raises th…
d6524c9
docs(function_pods): stopping at the first failed row on both orchest…
9c4a844
Merge remote-tracking branch 'origin/main' into eywalker/itl-879-asyn…
de60388
fix(function_node): a node cancelled by another node's failure commit…
96837aa
fix(function_node): an observer failing in on_node_end does not skip …
9b6eb23
Merge remote-tracking branch 'origin/main' into eywalker/itl-879-asyn…
dbf8843
fix(function_node): a failed or cancelled async node holds no rows; o…
9226786
test(async_error_policy): an aggregate after a fail_fast failure stor…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| - `AsyncPipelineOrchestrator(error_policy="fail_fast")` now stops the run at the first row whose function raises and raises that row's own exception, as `SyncPipelineOrchestrator` does (ITL-879). Before, the async orchestrator ignored the policy: the run skipped the row, computed the others and returned. An async run with `fail_fast` that used to finish now raises; pass `error_policy="continue"` (the default) to keep the old behaviour. Rows run concurrently, so the row raised is the first to fail, which is the first in input order only with `max_concurrency=1`. Every stored row is a correct result of its input row and the failing row is never stored; which other rows are stored before the run stops is not specified (see `docs/concepts/function-pods.md`, "Stopping at the first failed row"). | ||
| - An async run raises the failing node's own exception, never the task group's `ExceptionGroup` (ITL-879). This covers a side-effect pod with `on_error="raise"`, an operator that rejects its inputs, and any other node failure, as store errors already were. When several nodes fail before the run stops, one exception is raised and the others are added to it as notes (`error.__notes__`): a store error (`StoreError`) first, then a stored row that cannot be decoded as its class (`TypeIdentityError`), as before; among the other failures, the one of the node that comes first in the pipeline. Replace `except* SomeError` or `except ExceptionGroup` around `job.run()` / `job.run_async()` / `AsyncPipelineOrchestrator.run()` with `except SomeError`. | ||
| - `FunctionNodeProtocol.async_execute`, `FunctionJobNode.async_execute` and the function pods' `async_execute` take a keyword-only `error_policy` (default `"continue"`), as `execute` already did. A class implementing `FunctionNodeProtocol` must accept it. | ||
|
|
||
| ### Fixed | ||
|
|
||
| - Under the async orchestrator a function node whose run stops on a row failure now calls the observer's `on_node_end` before raising, as the sync orchestrator does (ITL-879). | ||
| - A function node that the async orchestrator cancels because another node failed now commits the results it wrote before the cancellation propagates, as a failing node does, and then holds no rows: it reads as a node with no run, as a node whose run raised (ITL-879, ITL-1093). Before, its ephemeral results were never committed while the job committed its pipeline records, and on a Delta store its results were not committed ahead of its pipeline records. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| ## `src/orcapod/pipeline/async_orchestrator.py` — error policy on the async path | ||
|
|
||
| ### EP1 — The async orchestrator ignored `error_policy`; a node failure reached the caller as an `ExceptionGroup` | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-879 | ||
|
|
||
| `AsyncPipelineOrchestrator` stored `error_policy` and never used it. The pod's per-row | ||
| handler (`_FunctionPodBase.async_execute.process_one`) skipped every row whose function | ||
| raised, so under `error_policy="fail_fast"` an async run computed every other row and | ||
| finished, with a `RunSummary` and a `RowFailuresWarning`, where the sync run raised the row's | ||
| exception after storing the rows before it. Any other node failure that was not a store | ||
| error (a side-effect pod with `on_error="raise"`, an operator that rejected its inputs) | ||
| reached the caller as the task group's `ExceptionGroup`, where the sync run raised the | ||
| exception itself. Under `"continue"` the two orchestrators already agreed: no raise, the | ||
| same summary, one `RowFailuresWarning`, the per-row log at DEBUG on both (the issue's | ||
| "WARNING" premise predates ITL-997), the same observer events and stored rows. | ||
|
|
||
| **Fix:** `error_policy` is a keyword-only argument (default `"continue"`) of | ||
| `FunctionNodeProtocol.async_execute`, `FunctionJobNode.async_execute` and | ||
| `_FunctionPodBase.async_execute`, as of `execute` (coordinator decision, November 13, 2026), | ||
| and the orchestrator passes its own. Under `"fail_fast"` the pod re-raises the row's | ||
| exception after `on_data_crash`; the existing task groups cancel the rows and nodes still | ||
| running and every node closes its output in its existing `finally`; the failing node calls | ||
| `on_node_end` before it raises, as `execute` does. The orchestrator's cancel and close | ||
| machinery is unchanged. Its exception-group handler now always raises ONE exception: a store | ||
| error, then a `TypeIdentityError` (both as before), then the exception of the node earliest | ||
| in topological order (ties: the order the failures happened), the others noted on it | ||
| (`_single_failure`). A function node cancelled by another node's failure commits what it | ||
| wrote (`_commit_after_failure`) before the cancellation propagates, as a failing node does: | ||
| the job's flush commits neither its ephemeral store nor its result table ahead of its | ||
| pipeline records (CodeRabbit, PR #379); it then holds no rows, as a node whose run raised | ||
| (ITL-1093). An observer whose `on_node_end` raises as the node fails does not skip that | ||
| commit either, as on the sync path, and is not called a second time. Rows run concurrently, | ||
| so the row raised is the first to fail, the first in input order only with | ||
| `max_concurrency=1`. Storage guarantee, as documented in | ||
| `docs/concepts/function-pods.md`: every stored row is a correct, complete result of its input | ||
| row; the failing row is never stored; which other rows are stored is not specified. No | ||
| hash, golden value or stored layout changed. Regressions: | ||
| `tests/test_pipeline/test_async_error_policy.py`. Related, not changed: a pod-level error | ||
| policy (F15, ITL-527). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 25397
Commit a cancelled node's pending results before it re-raises.
CancelledErroris aBaseException, so theexcept Exceptionbranch does not catch it. When fail-fast cancels a node that has pending ephemeral results, that node does not commit them. The job-level flush then commits the pipeline records but not the ephemeral results. The result is stored records that reference results that were never stored.Handle cancellation with its own branch. Run the same best-effort commit, then re-raise. Keep the observer behavior of the existing branch unchanged.
Suggested fix
Committable suggestion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accepted and verified, fixed in de60388. A node cancelled because another node failed skipped
_commit_after_failure(CancelledErroris not anException). The job's flush after a failure commits only the pipeline and result databases, never a node's ephemeral store, and table by table in no guaranteed order.test_a_node_cancelled_by_a_failure_commits_its_ephemeral_resultsintests/test_pipeline/test_job_batched_commits.py. Setup: a Delta job; only the downstream nodeincis ephemeral; the upstream fails underfail_fastat its last row withmax_concurrency=1. Before the fix it failed with 29 pipeline records committed and 0 ephemeral results. The test also asserts that the cancelled node did record rows, so it cannot pass vacuously.except asyncio.CancelledErrorbranch, placed before theExceptionbranch, runs_commit_after_failureand re-raises. It makes no observer call; theExceptionbranch is unchanged.fail_fasttest is now parametrized on both orchestrators, since async appliesfail_fastnow.OperatorJobNode(LOG) writes its output in one synchronous call after its task group completes, so a cancellation leaves all of it or none, in one table the job flushes.SideEffectJobNodedelivers and logs each row in one synchronous call, into the pipeline store the job flushes. Neither has this hole.Posted by a Claude Code agent on behalf of @eywalker.