Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog.d/ITL-1075.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
### Fixed

- An async job run no longer hangs when one node feeds both inputs of a `SemiJoin` (ITL-1075). `SemiJoin` read its left input only after its right input had ended. When a node fed both inputs (for example `SemiJoin(C, A)`, with `C` computed from `A`), the bounded channel into the unread left input filled, every node between `A` and that channel blocked in turn, `A` could not finish, and the run waited forever. A computing function node absorbs its whole input, which hid this on a first run: the hang showed on a run that found every row stored, or on any run of a graph of operators only. `SemiJoin` now reads both inputs at once. It keeps the left rows that arrive before the right input ends (at most the whole left input, which the sync path and `MergeJoin` already hold), and then streams the rest of the left input as before. Output rows, their order, system tags and hashes are unchanged.
- An async job run no longer hangs on an operator node in `CacheMode.REPLAY` with more upstream rows than a channel holds (ITL-1075). The node served its stored rows without reading its input, so its upstream blocked on a full channel. It now reads and discards its input while it replays, so the upstream runs over every row, as it does with the sync orchestrator.
33 changes: 33 additions & 0 deletions design_issues/ITL-1075-AH1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## `src/orcapod/core/operators/` — Async execution

### AH1 — An async run deadlocked when a consumer stopped reading one of its inputs
**Status:** resolved
**Severity:** high
**Issue:** ITL-1075

The async orchestrator joins nodes with bounded channels, so every node must keep reading
each input channel to its end: a consumer that waits on something before it reads an input
blocks that input's producer once the channel is full. Two consumers did:

- `SemiJoin.async_execute` collected its right input before it read its left one. When one
node `A` fed both inputs (`SemiJoin(C, A)` with `C` computed from `A`, or `SemiJoin(A, C)`,
or `SemiJoin(A, B)` with `A` and `B` from one source), the channel into the unread left
input filled, every node between `A` and it blocked, `A` could not finish its right-side
rows, and the run waited forever. A computing function node spawns one task per input row
without a limit (no `max_concurrency`) and so absorbs its whole input, which hid the cycle
on a first run; a run that found every row stored, or a graph of operators only, hung once
the rows exceeded what the channels in the cycle hold (about 3 x `buffer_size` + 130).
- `OperatorNode.async_execute` in `CacheMode.REPLAY` served its stored rows without reading
its inputs, so any upstream with more rows than a channel holds blocked forever.

Every other `async_execute` (the unary operators, `Join`, `MergeJoin` and the barrier
default, function, side-effect and source nodes) reads every input to its end.

**Fix:** `SemiJoin` reads its left input while a task collects its right one
(`_collect_right_reading_left`), keeping the left rows that arrive before the right input
ends (at most the whole left input, as the sync path holds); a `REPLAY` operator node
reads and discards its inputs while it sends its stored rows. Each left receive is
awaited together with the right task, so the end or failure of the right input is acted on
at once; a receive still pending then is handed to the left rows, never cancelled while its
row may be read. Regression tests in `tests/test_pipeline/test_async_fan_out_does_not_hang.py`
and `tests/test_channels/test_native_async_operators.py`.
34 changes: 29 additions & 5 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,16 @@
pc = LazyModule("pyarrow.compute")


async def _discard(channel: ReadableChannel[Any]) -> None:
"""Read ``channel`` to its end, keeping nothing.

Args:
channel: The channel to drain.
"""
async for _ in channel:
pass


# ---------------------------------------------------------------------------
# OperatorNodeBase — shared base (no DB)
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -1680,7 +1690,8 @@ async def async_execute(
Without a database, or with one in OFF mode, nothing is stored.

With a database:
- REPLAY: emit from DB, close output.
- REPLAY: emit from DB while reading and discarding every input, so
the upstream nodes run as with the sync orchestrator; close output.
- OFF: delegate to operator, forward and keep results.
- LOG: delegate to operator, forward and keep results, then store
them in the DB.
Expand Down Expand Up @@ -1708,10 +1719,23 @@ async def async_execute(
self._pipeline_database is not None
and self._cache_mode == CacheMode.REPLAY
):
self._replay_from_cache()
assert self._cached_output_stream is not None
for tag, data in self._cached_output_stream.iter_data():
await output.send((tag, data))
# Read and discard every input while the stored rows are sent: the
# upstream nodes run over every row, as with the sync orchestrator,
# and an input no one reads blocks its producer, and the run, once
# its channel is full (ITL-1075).
discards = [asyncio.create_task(_discard(ch)) for ch in inputs]
try:
self._replay_from_cache()
assert self._cached_output_stream is not None
for tag, data in self._cached_output_stream.iter_data():
await output.send((tag, data))
_ = await asyncio.gather(*discards)
finally:
for discard in discards:
_ = discard.cancel()
# Wait for the readers to end; a failure of the replay stays the
# error raised.
_ = await asyncio.gather(*discards, return_exceptions=True)
ctx_obs.on_node_end(node_label, node_hash)
return # finally block closes output

Expand Down
170 changes: 156 additions & 14 deletions src/orcapod/core/operators/semijoin.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
import asyncio
import contextlib
import warnings
from collections.abc import Collection, Sequence
from typing import TYPE_CHECKING, Any
from collections import deque
from collections.abc import AsyncGenerator, AsyncIterator, Collection, Sequence
from typing import TYPE_CHECKING, Any, final

from orcapod.channels import ReadableChannel, WritableChannel
from orcapod.channels import ChannelClosed, ReadableChannel, WritableChannel
from orcapod.core.operators.base import BinaryOperator, require_shared_arrow_type
from orcapod.core.operators.key_nullability import reconcile_key_nullability
from orcapod.core.operators.match_keys import frozen_scalar
Expand Down Expand Up @@ -293,9 +296,16 @@ async def async_execute(
"""Build-probe: collect right input, then stream left through a hash lookup.

Phase 1 — Build: collect all rows from the right (filter) channel and
index them by the common-key values.
Phase 2 — Probe: stream left rows one at a time; for each row whose
common-key values appear in the right-side index, emit immediately.
index them by the common-key values. The left channel is read at the
same time (ITL-1075): the left rows that arrive before the right input
ends are kept, at most the whole left input, as the sync path and
``MergeJoin``'s barrier path hold their inputs. Reading the left input
only after the right one ended deadlocked a run in which one node fed
both inputs.
Phase 2 — Probe: the kept left rows, then the rest of the left input
one at a time with backpressure; for each row whose common-key values
appear in the right-side index, emit immediately. Left rows keep their
arrival order.

Falls back to barrier mode when the right input is empty (schema
cannot be inferred from data) or when there are no common keys.
Expand All @@ -312,11 +322,12 @@ async def async_execute(
their system-tag names, so no column is named after an
input's pipeline hash.
"""
left_rows: _LeftRows | None = None
try:
left_ch, right_ch = inputs[0], inputs[1]

# Phase 1: Build right-side lookup
right_rows = await right_ch.collect()
# Phase 1: Build right-side lookup, reading the left input meanwhile
right_rows, left_rows = await _collect_right_reading_left(
inputs[0], inputs[1]
)

if not right_rows:
# Empty right: determine common keys from the validated
Expand All @@ -325,10 +336,11 @@ async def async_execute(
common = self._common_keys_from_schema()
if common:
# Drain left channel (discard) — result is empty
await left_ch.collect()
async for _ in left_rows:
pass
return
# No common keys — pass all left rows through unchanged
async for tag, data in left_ch:
async for tag, data in left_rows:
await output.send((tag, data))
return

Expand All @@ -339,7 +351,7 @@ async def async_execute(
common_keys: tuple[str, ...] | None = None
right_lookup: set[tuple[object, ...]] = set()

async for tag, data in left_ch:
async for tag, data in left_rows:
if common_keys is None:
# First left row — determine common keys (the shared tag
# columns, ITL-1088) and build index
Expand All @@ -359,7 +371,7 @@ async def async_execute(
if not common_keys:
# No common keys — pass all left rows through
await output.send((tag, data))
async for t, p in left_ch:
async for t, p in left_rows:
await output.send((t, p))
return

Expand All @@ -384,6 +396,8 @@ async def async_execute(
await output.send((tag, data))
finally:
await output.close()
if left_rows is not None:
await left_rows.aclose()

@staticmethod
def _key_table(tag: TagProtocol, data: DataProtocol, key: str) -> "pa.Table":
Expand Down Expand Up @@ -431,3 +445,131 @@ def _match_key(

def identity_structure(self) -> Any:
return self.__class__.__name__


_Row = tuple[TagProtocol, DataProtocol]


async def _collect_right_reading_left(
left: ReadableChannel[_Row],
right: ReadableChannel[_Row],
) -> tuple[list[_Row], "_LeftRows"]:
"""Collect ``right`` while reading ``left``, so neither input stops being read.

A task collects ``right``; meanwhile the rows of ``left`` are read and kept, until
``right`` ends or ``left`` does. A semi-join that waited for its right input before
reading its left one blocked a node feeding both inputs: the channel into the left
input filled, that node could not finish its right-side rows, and the run waited
forever (ITL-1075).

Each left receive runs as a task, awaited together with the right one, so the end
(or failure) of ``right`` is acted on at once, not when ``left`` next sends or
closes. A receive still pending then is handed to the returned rows, which await
it in its place: it is never cancelled while its row may still be read.

Args:
left: The left input channel.
right: The right input channel.

Returns:
The rows of ``right``, and every row of ``left`` in arrival order: the rows
kept while ``right`` was collected, the pending receive's row, then the rest
of ``left`` as it arrives.

Raises:
Exception: Whatever collecting ``right`` raises. A pending left receive is
then cancelled, since the left rows are not going to be read.
"""
right_task = asyncio.create_task(right.collect())
kept: deque[_Row] = deque()
receive: asyncio.Task[_Row] | None = None
left_ended = False
try:
while not right_task.done():
if receive is None:
receive = asyncio.create_task(left.receive())
_ = await asyncio.wait(
(receive, right_task), return_when=asyncio.FIRST_COMPLETED
)
if receive.done():
finished, receive = receive, None
try:
kept.append(finished.result())
except ChannelClosed:
left_ended = True
break
right_rows = await right_task
except BaseException:
if receive is not None:
await _cancel_receive(receive)
raise
finally:
if not right_task.done():
_ = right_task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await right_task
return right_rows, _LeftRows(left, kept, receive, left_ended)


async def _cancel_receive(receive: "asyncio.Task[_Row]") -> None:
"""Cancel a left receive whose row is not going to be read, and wait for it to end.

Args:
receive: The receive task; it may already be done.
"""
_ = receive.cancel()
_ = await asyncio.wait((receive,))
if not receive.cancelled():
# Retrieve its outcome (a row, or ``ChannelClosed``) so it is not reported.
_ = receive.exception()


@final
class _LeftRows:
"""Every row of the left input in arrival order, for one pass.

The rows kept while the right input was collected come first, then the row of
the receive that was pending when it ended, then the rest of the left channel.
"""

def __init__(
self,
left: ReadableChannel[_Row],
kept: deque[_Row],
pending: "asyncio.Task[_Row] | None",
left_ended: bool,
) -> None:
self._left = left
self._kept = kept
self._pending = pending
self._left_ended = left_ended
self._rows = self._iterate()

def __aiter__(self) -> AsyncIterator[_Row]:
return self._rows

async def _iterate(self) -> AsyncGenerator[_Row, None]:
while self._kept:
yield self._kept.popleft()
if self._pending is not None:
try:
row = await self._pending
except ChannelClosed:
return
finally:
self._pending = None
yield row
if not self._left_ended:
async for row in self._left:
yield row

async def aclose(self) -> None:
"""Release the left input when its rows are not all read.

A pending receive is cancelled. After a complete pass there is none, so this
does nothing.
"""
await self._rows.aclose()
if self._pending is not None:
pending, self._pending = self._pending, None
await _cancel_receive(pending)
Loading
Loading