Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -567,6 +567,14 @@ predicate on values, and DropNulls reads only whether a value is present (its va
never the value. A filter decides which rows exist and never synthesizes a value; no other
operator reads data content.

No operator carries its inputs' Data meta (`__`) columns (developer, November 13, 2026,
ITL-1080): on the sync and the async path, in rows and in the predicted output schema, for a
raw ArrowTableStream's `__` columns and for the meta a cached pod's or function node's rows
carry. One rule, core/operators/meta_columns.py. A carry-forward mode is ITL-867.
The meta prefix (`__`) is reserved: ArrowTableStream and the source builder refuse a tag named
with it, and MapTags / MapData (targets) and Pick / Index (out) refuse it as a new name
(schema_utils.require_names_not_meta, at construction).

Operator or derived source? (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
Expand Down
9 changes: 9 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,15 @@ predicate on values, and `DropNulls` reads only whether a value is present (its
never the value. A filter decides which rows exist and never synthesizes a value; no other
operator reads data content.

No operator carries its inputs' `Data` meta (`__`) columns (developer, November 13, 2026,
ITL-1080): on the sync and the async path, in rows and in the predicted output schema, for a
raw `ArrowTableStream`'s `__` columns and for the meta a cached pod's or function node's rows
carry. The rule is in one place, `core/operators/meta_columns.py` (sync results through the
operator base classes, async rows through `WithoutMetaWriter`). A carry-forward mode is ITL-867.
The meta prefix (`__`) is reserved: `ArrowTableStream` and the source builder refuse a tag named
with it, and `MapTags` / `MapData` (targets) and `Pick` / `Index` (`out`) refuse it as a new name
(`schema_utils.require_names_not_meta`, at construction).

**Operator or derived source?** (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
Expand Down
40 changes: 40 additions & 0 deletions changelog.d/ITL-1080.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
### Changed

- No operator carries its inputs' `Data` meta (`__`-prefixed) columns anymore, on the sync and
the async path alike, as `Join` already did (ITL-1080). An operator's output rows and its
predicted output schema hold none of them, whether they came from a raw `ArrowTableStream`
whose table has `__` columns or from the rows of a cached function pod or function node
(`__computed`, `__pf_*`, `__pod_ts`). Before, `MergeJoin`, `Batch`, `GroupBy`, `PolarsFilter`,
`SemiJoin`, `DropNulls`, `MapTags`, `MapData` and the column selections kept them on one path
or both. No row content hash, record id or stored table changes: meta columns were never
hashed, and job runs already stored operator rows without them. A mode that carries meta
forward is ITL-867.
- A tag column whose name starts with `__`, the prefix reserved for meta columns, is refused with
`InputValidationError` where it would be made: building an `ArrowTableStream`, building a
source with such a name in `tag_columns` (before, a "tag_columns not found" error, since the
source drops `__` columns), and `MapTags` renaming a tag to such a name (ITL-1080). `MapData`
likewise refuses to rename a data column to such a name, and `Pick` and `Index` refuse such an
`out` name (before, the picked column was silently dropped). Before, a raw stream could carry such
a tag, and every path that rebuilt rows silently lost it, so sync and async runs differed.

### Fixed

- An operator node whose async run produces no row keeps the sync run's columns and Arrow types
(`int8` stays `int8`; system-tag and source-info columns are present), instead of a table
built from the Python output schema (ITL-1080). When an input itself delivered no row, that
input is rebuilt from its upstream's Python output schema on both paths alike (ITL-912), so a
`Join` of such an input with a non-empty one whose key is, say, `int8` now raises
`InputValidationError` on an async run as it already did on a sync run, instead of finishing
with a mistyped empty result.
- `DropDataColumns` and `SelectDataColumns` raise `InputValidationError` when their result would
have no data column (every data column dropped, or `strict=False` matching none), when the
operator is applied on the sync path and on the first row on the async path, with one message
(ITL-1080). Before, the sync path failed only when the rows were read, with a `ValueError`, and
the async path emitted rows with empty data.
- `MapData(..., drop_unmapped=True)` whose sources are not in the stream would leave no data
column: it now raises the same `InputValidationError` on both paths, where both paths used to
keep every data column while the prediction listed none. `MapTags(..., drop_unmapped=True)` in
the same case drops every tag on both paths, as its prediction says; the async path used to
keep them. A meta (`__`) source of `MapData` is never renamed and no longer reports a false
collision, and a strict `SelectDataColumns` / `DropDataColumns` naming a meta column reports it
missing on both paths (ITL-1080).
44 changes: 44 additions & 0 deletions design_issues/ITL-1080-SA1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
## `src/orcapod/core/operators/` — Async execution

### SA1 — Sync and async runs gave different rows: meta columns, and a zero-row async operator node
**Status:** resolved
**Severity:** high
**Issue:** ITL-1080

The same pipeline run with the sync and the async orchestrator must give the same rows
(developer, November 13, 2026). Nine strict expected failures pinned differences:

* Eight operators disagreed on the inputs' `Data` meta (`__`) columns. Over a raw
`ArrowTableStream` that lists `__` columns as data, the sync path of `MergeJoin`, `Batch`,
`GroupBy` and `PolarsFilter` kept them while their barrier async path (`_rows_to_table`)
dropped them; `SelectDataColumns`, `Index` and `Pick` did the opposite. Over a cached function
pod's stream (rows carry `__computed`, `as_table()` hides it), `MapTags`, `MapData` and
`DropNulls` dropped it on sync and kept it on async. `Join` already dropped meta on both paths
(ITL-854, ITL-857). Row content hashes never differed: `Data.content_hash()` does not cover
meta, and both orchestrators already stored operator rows without meta.
* An async operator node whose run produced no row read back a table built from the Python
output schema (`int8` became `int64`; system-tag and source-info columns missing); the sync
run kept the operator's own result.

**Fix:** the rule (developer, November 13, 2026): no operator carries its inputs' meta
columns, on either path, in rows and in the prediction. It lives in
`core/operators/meta_columns.py`: the unary and binary operator base classes (and `Join`) pass
their sync result through `without_input_meta` and their prediction through
`without_meta_fields`; every operator with its own `async_execute` sends its rows through
`WithoutMetaWriter`. A mode that carries meta forward across operators is ITL-867.
`OperatorJobNode.async_execute` taps its input channels (the first row, and the first row with
a `None` source-info token per column) and, when no row comes out, applies the operator to its
inputs rebuilt with no rows, exactly as the sync orchestrator rebuilds them
(`StaticOutputOperatorPod._materialize_upstream`, shared by both). No row content hash, record
id, pipeline hash or golden value moves.

A further divergence (found by the ITL-1084 worker): `DropDataColumns` of every data column, or
`SelectDataColumns(strict=False)` matching none, failed on the sync path only when rows were read
and emitted rows with empty `Data` on the async path. Both now raise one `InputValidationError`
(`column_selection._require_a_data_column`): at `op(stream)` on sync, on the first row on async.

A tag named with the meta prefix (`__g`) was a meta column to the `Tag` datagram, so every path
that rebuilds rows (every `async_execute`, `Index`/`Pick` on sync, both orchestrators) lost it
while table-based sync paths kept it, and `Join` failed on it outright. Such a tag is now
refused where it would be made (`schema_utils.require_names_not_meta`: `ArrowTableStream`,
`SourceStreamBuilder.build`, `MapTags`; `MapData` refuses it as a new data name), and the meta helpers strip only non-tag `__` columns.
2 changes: 1 addition & 1 deletion design_issues/ITL-837-JA1.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,4 @@ drop them. Nothing downstream of a `Join` reads or stores them, and meta is not
two inputs that share only a meta name join. How meta columns should be handled across
operators and pods is a separate design question (ITL-867). `MergeJoin` still carries input
meta on its sync path (colliding names are merged into lists like data) and drops it on its
barrier async path (ITL-846).
barrier async path (ITL-846). Since ITL-1080 no operator carries input meta, on either path (SA1).
16 changes: 12 additions & 4 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ The filters are the operators that read data columns: `PolarsFilter` evaluates i
on values, and `DropNulls` reads only whether a value is present (its validity), never the
value. Both decide which rows exist; neither synthesizes a value.

No operator carries its inputs' meta columns: the `__`-prefixed columns, such as a cached
function pod's `__computed`, that describe how an input row was produced. An operator's
output row is produced by the operator, so its `Data` holds none of them and its predicted
output schema lists none, on the sync and the async path alike. A way to carry them forward
across operators is an open design question (ITL-867). The `__` prefix is reserved for meta
columns: a stream and a source refuse a tag column named with it, and `MapTags` and `MapData`
refuse it as the new name of a tag or data column.

## Operator categories

### `UnaryOperator` -- single input
Expand All @@ -58,9 +66,8 @@ Takes one or more streams. Used for `Join`, which performs an N-ary inner join.

N-ary inner join on shared tag columns. Requires that input streams have non-overlapping
data columns (raises `InputValidationError` on collision). Join is **commutative** -- the
order of input streams does not affect the result. Join drops its inputs' meta
(`__`-prefixed) columns, such as a cached function pod's `__computed`, so they never collide
and never appear in its output. Rows match on equality of every shared tag column, whatever
order of input streams does not affect the result. Like every operator, Join drops its
inputs' meta (`__`-prefixed) columns, so they never collide. Rows match on equality of every shared tag column, whatever
its type (lists and structs compare element by element). A row with a null in any shared tag
column matches nothing, a null list or struct included, and rows with null on both sides do
not match each other (the SQL convention for nulls; `MergeJoin` and `SemiJoin` follow it too, on the sync and the async path, while `GroupBy` is the exception by design and puts null keys in one group); a null *inside* a list or struct is an ordinary element that equals
Expand Down Expand Up @@ -543,7 +550,8 @@ print(by_method.pipeline_hash() == by_operator.pipeline_hash())

`DropNulls()` and `stream.drop_nulls()`, with no names, drop every row that has a null in
**any** tag or data column, and make every tag and data column non-nullable. System-tag,
source-info, context and meta columns are never tested or changed.
source-info, context and meta columns are never tested (meta columns are not carried
at all, as by every operator).

```python
visits = ArrowTableSource(
Expand Down
77 changes: 68 additions & 9 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from typing import TYPE_CHECKING, Any, Literal, cast

from orcapod import contexts
from orcapod.channels import Channel, ReadableChannel, WritableChannel
from orcapod.channels import Channel, ChannelClosed, ReadableChannel, WritableChannel
from orcapod.config import OrcapodConfig
from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod
from orcapod.core.streams.arrow_table_stream import ArrowTableStream
Expand Down Expand Up @@ -89,6 +89,55 @@ async def _discard(channel: ReadableChannel[Any]) -> None:
pass


class _InputTap:
"""An input channel reader that keeps the rows needed to rebuild its columns.

An async operator node hands its input channels to the operator, so it sees
no input row itself. When the run produces no output row, the node rebuilds
each input with no rows and applies the operator to those, as the sync path
applies it to the rebuilt upstream rows (ITL-1080). Those columns and types
are what ``StaticOutputOperatorPod._rows_to_table`` makes of the input's
rows, and the only thing in them that depends on more than one row is the
nullability of a source-info column (nullable when some row's token is
``None``). So the tap keeps the first row, and the first row with a ``None``
token for each source-info column, never the whole input.

Attributes:
rows: The kept rows, in arrival order.
"""

def __init__(self, reader: ReadableChannel[tuple[TagProtocol, DataProtocol]]):
self._reader = reader
self.rows: list[tuple[TagProtocol, DataProtocol]] = []
self._null_sources: set[str] = set()

def _keep(self, item: tuple[TagProtocol, DataProtocol]) -> None:
"""Keep ``item`` if it is the first row or has a new ``None`` token."""
nulls = {name for name, token in item[1].source_info().items() if token is None}
if not self.rows or not nulls <= self._null_sources:
self.rows.append(item)
self._null_sources |= nulls

async def receive(self) -> tuple[TagProtocol, DataProtocol]:
"""Receive the next row from the wrapped reader, keeping it if needed."""
item = await self._reader.receive()
self._keep(item)
return item

def __aiter__(self) -> "_InputTap":
return self

async def __anext__(self) -> tuple[TagProtocol, DataProtocol]:
try:
return await self.receive()
except ChannelClosed:
raise StopAsyncIteration

async def collect(self) -> list[tuple[TagProtocol, DataProtocol]]:
"""Drain every remaining row."""
return [item async for item in self]


# ---------------------------------------------------------------------------
# OperatorNodeBase — shared base (no DB)
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -1761,11 +1810,12 @@ async def forward() -> None:

hashes = [s.pipeline_hash() for s in self._input_streams]
operator = cast(StaticOutputOperatorPod, self._operator)
taps = [_InputTap(reader) for reader in inputs]
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(
operator.async_execute(
inputs,
taps,
intermediate.writer,
input_pipeline_hashes=hashes,
)
Expand All @@ -1787,14 +1837,23 @@ async def forward() -> None:
if should_store:
self._store_output_stream(stream)
else:
# A completed run with no rows keeps a typed zero-row result, as the
# sync path does, so the node reads as computed and empty (not as
# never run). Nothing is stored for it.
self._cached_output_stream = ArrowTableStream(
self._make_empty_table(),
tag_columns=self.keys()[0],
data_context=self.data_context,
# A completed run with no rows keeps the operator's result over its
# inputs rebuilt with no rows, exactly as the sync orchestrator
# rebuilds them, so the node reads as computed and empty with the
# sync path's columns and Arrow types (ITL-1080). An input that
# delivered no row is rebuilt from its Python output schema, as on
# the sync path. Nothing is stored for it.
empty_result = operator.process(
*(
StaticOutputOperatorPod._materialize_upstream(
tap.rows, upstream, zero_rows=True
)
for tap, upstream in zip(taps, self._input_streams, strict=True)
)
)
# Computed, as execute() computes it, so the node reads as run.
_ = list(empty_result.iter_data())
self._cached_output_stream = empty_result

self._update_modified_time()

Expand Down
22 changes: 18 additions & 4 deletions src/orcapod/core/operators/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
from collections.abc import Collection, Sequence
from typing import TYPE_CHECKING, Any

from orcapod.core.operators.meta_columns import (
without_input_meta,
without_meta_fields,
)
from orcapod.channels import ReadableChannel, WritableChannel
from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod
from orcapod.errors import InputValidationError
Expand Down Expand Up @@ -136,7 +140,8 @@ def validate_inputs(self, *streams: StreamProtocol) -> None:
def static_process(self, *streams: StreamProtocol) -> StreamProtocol:
"""Forward to ``unary_static_process`` with the single input stream."""
stream = streams[0]
return self.unary_static_process(stream)
# No operator carries its input's Data meta columns (ITL-1080).
return without_input_meta(self.unary_static_process(stream), streams)

def output_schema(
self,
Expand All @@ -145,7 +150,11 @@ def output_schema(
all_info: bool = False,
) -> tuple[Schema, Schema]:
stream = streams[0]
return self.unary_output_schema(stream, columns=columns, all_info=all_info)
tag_schema, data_schema = self.unary_output_schema(
stream, columns=columns, all_info=all_info
)
# The prediction of ``without_input_meta`` in ``static_process``.
return tag_schema, without_meta_fields(data_schema)

def argument_symmetry(self, streams: Collection[StreamProtocol]) -> ArgumentGroup:
# return single stream as a tuple
Expand Down Expand Up @@ -231,7 +240,10 @@ def is_commutative(self) -> bool:
def static_process(self, *streams: StreamProtocol) -> StreamProtocol:
"""Forward to ``binary_static_process`` with two input streams."""
left_stream, right_stream = streams
return self.binary_static_process(left_stream, right_stream)
# No operator carries its inputs' Data meta columns (ITL-1080).
return without_input_meta(
self.binary_static_process(left_stream, right_stream), streams
)

def output_schema(
self,
Expand All @@ -240,9 +252,11 @@ def output_schema(
all_info: bool = False,
) -> tuple[Schema, Schema]:
left_stream, right_stream = streams
return self.binary_output_schema(
tag_schema, data_schema = self.binary_output_schema(
left_stream, right_stream, columns=columns, all_info=all_info
)
# The prediction of ``without_input_meta`` in ``static_process``.
return tag_schema, without_meta_fields(data_schema)

def validate_inputs(self, *streams: StreamProtocol) -> None:
if len(streams) != 2:
Expand Down
2 changes: 2 additions & 0 deletions src/orcapod/core/operators/batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from collections.abc import Sequence
from typing import TYPE_CHECKING, Any

from orcapod.core.operators.meta_columns import WithoutMetaWriter
from orcapod.channels import ReadableChannel, WritableChannel
from orcapod.core.operators.base import UnaryOperator
from orcapod.core.streams import ArrowTableStream
Expand Down Expand Up @@ -173,6 +174,7 @@ async def async_execute(
ValueError: If ``input_pipeline_hashes`` does not have one hash per input.
"""
try:
output = WithoutMetaWriter(output)
if len(input_pipeline_hashes) != len(inputs):
raise ValueError(
f"input_pipeline_hashes length ({len(input_pipeline_hashes)}) "
Expand Down
Loading
Loading