Repository navigation
fix(operator_node): keep an operator node's output after an async run (ITL-896) #302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
247c552
d6dbe76
303de51
b3090e3
475d4ca
4a38fcb
7235f8c
5b5dca8
9af1f02
69af724
91a342c
42d6d25
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| ### Fixed | ||
|
|
||
| #### An operator node returns its rows after an async run (ITL-896) | ||
|
|
||
| After a job ran on `AsyncPipelineOrchestrator`, an operator node (`job.nodes[label]` for | ||
| `Join`, `MergeJoin`, `SemiJoin`, `Batch`, `GroupBy`, `MapTags`, `MapData`, `PolarsFilter` and | ||
| the column-selection operators) returned no rows from `as_table()` and `iter_data()`, without | ||
| an error. It now returns the same rows as after a sync run, whatever `materialize_results` | ||
| is, and a later run replaces those rows even when it produces none. With the default | ||
| `cache_mode=OFF` (the only mode a job sets) an operator node keeps its output in memory and | ||
| stores nothing, so `get_all_records()` returns `None` for it on both orchestrators. | ||
| (ITL-896) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| ## `src/orcapod/core/nodes/operator_node.py` — operator node read-back after an async run | ||
|
|
||
| ### OR1 — An operator node read back empty after an async job run | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-896 | ||
|
|
||
| An operator node keeps its output only in memory; inside a `PipelineJob` every operator runs | ||
| with `cache_mode=OFF` and stores nothing. The sync orchestrator calls `OperatorJobNode.execute()`, | ||
| which always keeps the materialized output on the node. The async orchestrator calls | ||
| `OperatorJobNode.async_execute()`, which kept the rows only in `CacheMode.LOG`; in `OFF` (and with | ||
| no database) it forwarded them downstream and dropped them. `iter_data()` and `as_table()` then | ||
| fell back to the database, which they read only in `REPLAY`, and returned the zero-row | ||
| "not yet run" table without an error. Every operator was affected, on every store. | ||
|
|
||
| **Fix:** `async_execute` collects the rows it forwards in every mode except `REPLAY` and keeps | ||
| them on the node with `StaticOutputOperatorPod._materialize_to_stream`, as the sync path does, | ||
| whatever `materialize_results` is; `LOG` still stores them. Nothing new is stored, and no hash, | ||
| path or stored schema changes. A run with no rows clears the previous run's output, so a node | ||
| never serves stale rows (CodeRabbit on PR #302). `get_all_records()` still returns `None` for an | ||
| operator node in a job, since with the default `cache_mode=OFF` (the only mode a job sets) it | ||
| stores nothing. `clear_cache()` still returns the node to its zero-row "not yet run" | ||
| state on both paths (a function node instead reloads from its store). | ||
| Regressions: `tests/test_pipeline/test_operator_node_async_readback.py`, which compares the sync | ||
| and async reads of every operator. Open, pinned by strict xfails there: a zero-row async result | ||
| reads back the table built from the Python output schema (ITL-858, ITL-865); that table widens | ||
| `int8` to `int64` and changes nullability on both paths (ITL-912); `MergeJoin` after a non-source | ||
| input names its system-tag columns differently on the async path (ITL-906). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1172,8 +1172,11 @@ def run(self) -> None: | |
| def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: | ||
| """Return an iterator over (tag, data) pairs. | ||
|
|
||
| Read-only: never triggers computation. Returns empty before ``run()`` | ||
| or ``execute()`` populates the cache. Call ``node.is_stale`` before | ||
| Read-only: never triggers computation. Yields the rows of the latest | ||
| ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in | ||
| memory (before a run, or after ``clear_cache()``), a node in | ||
| ``CacheMode.REPLAY`` with a database yields the rows stored there; in | ||
| any other mode it yields nothing. Call ``node.is_stale`` before | ||
| iterating if you need to detect outdated cached data. | ||
| """ | ||
| if self._cached_output_stream is not None: | ||
|
|
@@ -1191,8 +1194,11 @@ def as_table( | |
| ) -> "pa.Table": | ||
| """Return the output as a PyArrow table. | ||
|
|
||
| Read-only: never triggers computation. Returns a zero-row table | ||
| with correct schema before ``run()`` or ``execute()`` is called. | ||
| Read-only: never triggers computation. Returns the rows of the latest | ||
| ``run()``, ``execute()`` or ``async_execute()``. With nothing cached in | ||
| memory (before a run, or after ``clear_cache()``), a node in | ||
| ``CacheMode.REPLAY`` with a database returns the rows stored there; in | ||
| any other mode it returns a zero-row table with the node's columns. | ||
| """ | ||
| if self._cached_output_stream is not None: | ||
| return self._cached_output_stream.as_table( | ||
|
|
@@ -1225,7 +1231,8 @@ def get_all_records( | |
|
|
||
| Returns the stored output table with column filtering applied | ||
| per ``ColumnConfig`` conventions. Returns ``None`` when no | ||
| database is attached or no records exist. | ||
| database is attached or no matching records exist. Switching to OFF | ||
| does not remove records stored by an earlier LOG run. | ||
| """ | ||
| if self._pipeline_database is None: | ||
| return None | ||
|
|
@@ -1318,13 +1325,18 @@ async def async_execute( | |
| ) -> None: | ||
| """Async execution with cache mode handling when DB is attached. | ||
|
|
||
| Without a database, delegates to the wrapped operator's | ||
| ``async_execute``. | ||
| Every mode except REPLAY runs the wrapped operator's ``async_execute``, | ||
| forwards its rows downstream and keeps them on the node, exactly as | ||
| ``execute()`` does, so ``iter_data()`` and ``as_table()`` return them | ||
| after the run whichever orchestrator ran the node. | ||
|
|
||
| Without a database, or with one in OFF mode, nothing is stored. | ||
|
|
||
| With a database: | ||
| - REPLAY: emit from DB, close output. | ||
| - OFF: delegate to operator, forward results. | ||
| - LOG: delegate to operator, forward + collect results, then store in DB. | ||
| - OFF: delegate to operator, forward and keep results. | ||
| - LOG: delegate to operator, forward and keep results, then store | ||
| them in the DB. | ||
|
|
||
| Args: | ||
| inputs: Sequence of readable channels from upstream nodes. | ||
|
|
@@ -1338,31 +1350,27 @@ async def async_execute( | |
| node_hash = self.content_hash().to_string() | ||
| ctx_obs = obs.contextualize(*self.node_identity_path) | ||
|
|
||
| if self._pipeline_database is None: | ||
| # Simple delegation without DB | ||
| ctx_obs.on_node_start(node_label, node_hash) | ||
| hashes = [s.pipeline_hash() for s in self._input_streams] | ||
| await self._operator.async_execute( | ||
| inputs, output, input_pipeline_hashes=hashes | ||
| ) | ||
| ctx_obs.on_node_end(node_label, node_hash) | ||
| return | ||
|
|
||
| try: | ||
| ctx_obs.on_node_start(node_label, node_hash) | ||
|
|
||
| if self._cache_mode == CacheMode.REPLAY: | ||
| if ( | ||
| self._pipeline_database is not None | ||
| and self._cache_mode == CacheMode.REPLAY | ||
| ): | ||
| self._replay_from_cache() | ||
| assert self._cached_output_stream is not None | ||
| for tag, data in self._cached_output_stream.iter_data(): | ||
| await output.send((tag, data)) | ||
| ctx_obs.on_node_end(node_label, node_hash) | ||
| return # finally block closes output | ||
|
|
||
| # OFF or LOG: delegate to operator, forward results downstream | ||
| # Delegate to the operator, forward its rows downstream and keep them. | ||
| intermediate: Channel[tuple[TagProtocol, DataProtocol]] = Channel() | ||
| should_collect = self._cache_mode == CacheMode.LOG | ||
| if should_collect: | ||
| should_store = ( | ||
| self._pipeline_database is not None | ||
| and self._cache_mode == CacheMode.LOG | ||
| ) | ||
| if should_store: | ||
| # Rows are forwarded while the operator runs, so an input this node | ||
| # will refuse to store must be refused before the first row leaves. | ||
| for input_stream in self._input_streams: | ||
|
|
@@ -1373,26 +1381,37 @@ async def async_execute( | |
|
|
||
| async def forward() -> None: | ||
| async for item in intermediate.reader: | ||
| if should_collect: | ||
| collected.append(item) | ||
| collected.append(item) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Avoid unbounded retention during streaming execution. In OFF mode and without a database, this now retains every forwarded datagram until the operator finishes. Previously those paths did not collect the output. Large or unbounded streams can exhaust memory despite the bounded channels, and each operator retains its own output. Use a bounded or disk-backed retention strategy, or provide an explicit way to disable retention.
|
||
| await output.send(item) | ||
|
|
||
| hashes = [s.pipeline_hash() for s in self._input_streams] | ||
| async with asyncio.TaskGroup() as tg: | ||
| tg.create_task( | ||
| self._operator.async_execute( | ||
| inputs, | ||
| intermediate.writer, | ||
| input_pipeline_hashes=hashes, | ||
| operator = cast(StaticOutputOperatorPod, self._operator) | ||
| try: | ||
| async with asyncio.TaskGroup() as tg: | ||
| tg.create_task( | ||
| operator.async_execute( | ||
| inputs, | ||
| intermediate.writer, | ||
| input_pipeline_hashes=hashes, | ||
| ) | ||
| ) | ||
| ) | ||
| tg.create_task(forward()) | ||
|
|
||
| # TaskGroup has completed — store if LOG mode (sync DB write, post-hoc) | ||
| if should_collect and collected: | ||
| tg.create_task(forward()) | ||
| except ExceptionGroup as exc: | ||
| # Preserve direct operator errors; concurrent failures stay grouped. | ||
| if len(exc.exceptions) == 1: | ||
| raise exc.exceptions[0] from exc | ||
| raise | ||
|
|
||
| # TaskGroup has completed. Keep the output on the node, as execute() | ||
| # does, and store it in LOG mode (sync DB write, post-hoc). A run | ||
| # with no rows replaces an earlier run's output too: the node then | ||
| # reads as a zero-row table. | ||
| self._cached_output_stream = None | ||
| if collected: | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 475d4ca. Reproduced through the public interface first: run a job with a Posted by a Claude Code agent on behalf of @eywalker.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Declined in this PR; the finding itself is correct. Posted by a Claude Code agent on behalf of @eywalker. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| stream = StaticOutputOperatorPod._materialize_to_stream(collected) | ||
| self._cached_output_stream = stream | ||
| self._store_output_stream(stream) | ||
| if should_store: | ||
| self._store_output_stream(stream) | ||
|
|
||
| self._update_modified_time() | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 41880
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 26951
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 28721
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 11845
Preserve direct operator exceptions for no-database calls.
If an
Indexnode hasfail_on_miss=Trueand receives an out-of-bounds row,OperatorJobNode.async_execute()now wraps itsRuntimeErrorin anExceptionGroup. A caller that catchesRuntimeErrorwill miss it. Preserve the direct exception when it is the sole failure, without discarding concurrent failures.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 91a342c. The node unwraps only the TaskGroup wrapper containing a single exception, preserving the original type and message. A public Index failure regression failed in no-database, OFF and LOG modes before the fix and now passes against the sync exception. A separate simultaneous operator/forwarding failure test verifies that both errors remain grouped.
Posted by a Claude Code agent on behalf of @eywalker.