Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
1b02228
test(operator_node): failing tests for typed zero-row operator node t…
Nov 14, 2026
e5c22f9
fix(operator_node): type an operator node's zero-row table from its i…
Nov 14, 2026
2aacb95
fix(function_pod): type a function pod's zero-row table from its inpu…
Nov 14, 2026
a88ba03
fix(sources): cached, polling and proxy sources give their typed zero…
Nov 14, 2026
61d9bf5
docs: zero-row table rule, changelog fragment and design issue ZR1 (I…
Nov 14, 2026
24400da
docs: cite ITL-1089 for the remaining zero-row fallbacks (ITL-912)
Nov 14, 2026
ac00d19
fix(side_effects): a side-effect pod's zero-row table is its input's;…
Nov 14, 2026
29fe5ed
Merge remote-tracking branch 'origin/main' into eywalker/itl-912-an-o…
Nov 14, 2026
c89a89a
chore: merge origin/main into ITL-912 branch; PipelineJob takes datab…
Nov 14, 2026
06d7c2d
Merge remote-tracking branch 'origin/main' into eywalker/itl-912-an-o…
Nov 14, 2026
ad709f7
chore: merge origin/main (6a379238) into ITL-912 branch
Nov 14, 2026
74365fe
fix: zero-row tables take the upstream's data context, never carry me…
Nov 14, 2026
033de4c
Merge commit 'b4fde330' into eywalker/itl-912-an-operator-nodes-not-y…
Nov 14, 2026
1f909b0
Merge commit '5537291f' into eywalker/itl-912-an-operator-nodes-not-y…
Nov 14, 2026
51c0792
Merge origin/main (3ac7f898, #369) into ITL-912
Nov 14, 2026
40a3dda
Merge origin/main (b746c630, #377) into ITL-912
Nov 14, 2026
54d5ab4
docs(changelog): a Join whose input delivered no row completes with t…
Nov 14, 2026
2ea739f
Merge remote-tracking branch 'origin/main' into eywalker/itl-912-an-o…
Nov 14, 2026
e0fd773
fix(streams): a head with no rows has the types of the stream's rows …
Nov 14, 2026
5dab609
fix(operators): head() never calls user code to learn column types (I…
Nov 14, 2026
0c1e227
fix(streams): CodeRabbit round on 5dab6091 (ITL-912)
Nov 14, 2026
4dc0245
test(streams): bound the memo thread test and document its helpers (I…
Nov 14, 2026
45698b4
Merge origin/main (edc17c57, #379) into ITL-912
Nov 14, 2026
7d92767
fix(streams): zero-row tables take non-large input types in the large…
Nov 14, 2026
38ec896
fix(streams): CodeRabbit round on 7d92767f: no-op filter, cached sour…
Nov 14, 2026
ead4a12
docs(streams): head() applies a user-defined operator to zero-row tab…
Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,11 @@ Key methods: output_schema(), keys(), iter_data(), as_table().
Every stream, source and node has run() with no arguments (developer, November 13, 2026,
ITL-1081): it does the work as_table() does (a job node runs as in a job) and returns a
StreamView of every row; the view's head/tail/sort_by_tags choose what is shown.
head() and the notebook preview never compute. Never give run() arguments.
head() and the notebook preview never compute (ITL-991, ITL-912): they never compute, poll or deliver a row (they read only what is held or stored, or a bounded table read), never call user code (a function, a predicate, a side effect) and never
write. With no rows to show they learn the column types by applying operators to zero-row
tables, which processes no data; an operator whose output schema is its input's
(OUTPUT_SCHEMA_IS_INPUT_SCHEMA, PolarsFilter) is not applied at all. A user-defined operator's static_process is applied to zero-row tables and must not have effects. Never give run()
arguments.

Source (core/sources/) — produces a stream from external data. ArrowTableSource is the core
implementation; CSV/Delta/DataFrame/Dict/List sources all delegate to it internally. Each
Expand Down Expand Up @@ -716,6 +720,21 @@ pa.Table.from_pylist(rows) without a type for a column whose input type is known
inference widens int8 to int64, float32 to double, and takes decimal precision from the
data. The rules:

- The Arrow schema is never computed (developer, November 13, 2026, ITL-912): "Generally
Orcapod should never have to compute anything to get the arrow schema, all function pods and
operator pods can deterministically derive the output stream schema from the input stream
schemas, and ultimately no source can be created without having schema specified somehow."
A table with no rows (before a run, after clear_cache(), a zero-row result or input) has
exactly the types of the rows: get it from streams/base.typed_zero_row_table, never from
the Python schema (except when a stream cannot type itself without rows: an unbound input or a
PollingSource not yet polled, ITL-1089, which fall back to the declared schema), and never by
computing, reading or polling a row or calling a function.
An operator's is its result over its inputs' zero-row tables; a function pod's is the
input's tags plus the declared outputs (function_zero_row_table); a source's is the table
or schema it holds. A new stream kind implements _typed_zero_row_table. A head with no
rows is built in one place, StreamBase._head_without_rows, from the stream's
_zero_row_table(config), which equals as_table(columns=config) with no rows; a kind
whose as_table shapes its table its own way overrides _zero_row_table with that shaping.
- Type strictness (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
21 changes: 20 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,7 +516,11 @@ Key methods: `output_schema()`, `keys()`, `iter_data()`, `as_table()`.
Every stream, source and node has `run()` with no arguments (developer, November 13, 2026,
ITL-1081): it does the work `as_table()` does (a job node runs as in a job) and returns a
`StreamView` of every row; the view's `head`/`tail`/`sort_by_tags` choose what is shown.
`head()` and the notebook preview never compute. Never give `run()` arguments.
`head()` and the notebook preview never compute (ITL-991, ITL-912): they never compute, poll or deliver a row (they read only what is held or stored, or a bounded table read), never call user code (a function, a predicate, a side effect) and never
write. With no rows to show they learn the column types by applying operators to zero-row
tables, which processes no data; an operator whose output schema is its input's
(`OUTPUT_SCHEMA_IS_INPUT_SCHEMA`, `PolarsFilter`) is not applied at all. A user-defined operator's `static_process` is applied to zero-row tables and must not have effects. Never give `run()`
arguments.

**Source** (`core/sources/`) — produces a stream from external data. `ArrowTableSource` is the
core implementation; CSV/Delta/DataFrame/Dict/List sources all delegate to it internally. Each
Expand Down Expand Up @@ -758,6 +762,21 @@ then through the data context's type converter — never call `pa.array(values)`
inference widens `int8` to `int64`, `float32` to `double`, and takes decimal precision from the
data. The rules:

- **The Arrow schema is never computed** (developer, November 13, 2026, ITL-912): "Generally
Orcapod should never have to compute anything to get the arrow schema, all function pods and
operator pods can deterministically derive the output stream schema from the input stream
schemas, and ultimately no source can be created without having schema specified somehow."
A table with no rows (before a run, after `clear_cache()`, a zero-row result or input) has
exactly the types of the rows: get it from `streams/base.typed_zero_row_table`, never from
the Python schema (except when a stream cannot type itself without rows: an unbound input or a
`PollingSource` not yet polled, ITL-1089, which fall back to the declared schema), and never by
computing, reading or polling a row or calling a function.
An operator's is its result over its inputs' zero-row tables; a function pod's is the
input's tags plus the declared outputs (`function_zero_row_table`); a source's is the table
or schema it holds. A new stream kind implements `_typed_zero_row_table`. A head with no
rows is built in one place, `StreamBase._head_without_rows`, from the stream's
`_zero_row_table(config)`, which equals `as_table(columns=config)` with no rows; a kind
whose `as_table` shapes its table its own way overrides `_zero_row_table` with that shaping.
- **Type strictness** (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
6 changes: 2 additions & 4 deletions changelog.d/ITL-1080.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,8 @@
- An operator node whose async run produces no row keeps the sync run's columns and Arrow types
(`int8` stays `int8`; system-tag and source-info columns are present), instead of a table
built from the Python output schema (ITL-1080). When an input itself delivered no row, that
input is rebuilt from its upstream's Python output schema on both paths alike (ITL-912), so a
`Join` of such an input with a non-empty one whose key is, say, `int8` now raises
`InputValidationError` on an async run as it already did on a sync run, instead of finishing
with a mistyped empty result.
input is rebuilt the same way on both paths alike, from its upstream's zero-row table
(ITL-912).
- `DropDataColumns` and `SelectDataColumns` raise `InputValidationError` when their result would
have no data column (every data column dropped, or `strict=False` matching none), when the
operator is applied on the sync path and on the first row on the async path, with one message
Expand Down
62 changes: 62 additions & 0 deletions changelog.d/ITL-912.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
### Changed

- `head()` contract (ITL-912): `head()` and the notebook preview never compute, poll or deliver a row (they read only what is held or stored, or make a bounded table read), never call user code (a function, a predicate, a side effect) and never write.
With no rows to show, they now give the columns and Arrow types the rows would have: they
apply each operator to empty (zero-row) tables, which processes no data (a chain of N
operators applies up to N operators to empty tables, about 60 ms for 20), and a filter is not
applied at all, since its columns are its input's. A user-defined operator's `static_process` is applied to zero-row tables and must not have effects. So a pipeline whose inputs disagree on a
column's Arrow type (`int8` against `int64`) now raises from `head()` the
`InputValidationError` its run raises; the notebook preview shows it as "could not read". A
filter whose predicate does not fit its columns shows a table in `head()` and raises when it
runs.

### Fixed

#### A table with no rows has the types of the rows (ITL-912)

An operator node read before its run, after `clear_cache()`, or after a run that produced no
row, returned a zero-row table built from its Python output schema: `int8` became `int64`,
a non-nullable list item became nullable, source-info columns changed nullability and the
system-tag columns were missing. A function pod's stream and a function node did the same for
their tag columns, and left out the `_context_key` and source-info columns; a side-effect
pod's stream or node that delivered no row did the same, and so did `head()` and the notebook
preview of a stream or node with no rows to show. Both orchestrators built an
operator node's input the same way when an upstream node delivered no row, so a `Join` of
such an input with a non-empty one whose key is, say, `int8` raised `InputValidationError`.

Each of these tables now has exactly the columns, Arrow types and nullability of the rows,
on both orchestrators, and that `Join` completes with typed zero rows on both paths. Nothing
is computed, read or called to get them: an operator applies
itself to its inputs' zero-row tables, a function pod combines its input's tags with its
declared output types, a side-effect pod gives its input's, and a source gives the table or
schema it already holds. Nothing is
stored for an empty table, so no hash, stored table or path changes.

### Known limitations

- A zero-row table is still built from the Python output schema (`int8` reads as `int64`,
system-tag columns are absent) when an upstream cannot say its Arrow types without rows: a
`PollingSource` before its first poll and an unbound input slot or `SourceProxy` (both are
declared with a Python schema only), a `DerivedSource`, and a loaded job node with no live
upstream and no stored table. `Index` and `Pick` over zero rows still lack the system-tag
columns (ITL-985). These cases are tracked in ITL-1089. (ITL-912)
- A source-info column (`_source_*`) of an operator node's rows is nullable only when some row's
provenance token is `None`, a rule that depends on the rows. A zero-row table has no token,
so it declares the column non-nullable, as rows with a known token do; when the rows have an
unknown (`None`) token, the field's nullability differs between the zero-row table and the
rows. A row-independent rule would change the operator node's stored record hashes. Tracked in
ITL-1089. (ITL-912)

#### Zero-row tables over non-large inputs (ITL-912)

A source holding `string`, `binary` or `list` columns gave a function node's or operator
node's table with no rows the non-large types, while the table with rows has the large ones
(`large_string`, ...). The zero-row table now takes its input's types in the large-normalized
form, as the populated table does, and an operator node's zero-row table lists its columns in
the order its rows do (a join's system tags before its data). A source's own table is not
changed.

A side-effect pod's stream takes its input's zero-row types in the same large form; a
`PolarsFilter()` with no predicate shows its input's types, as it passes the input through; and
a `CachedSource` over a source with no rows, with nothing stored, reads with the source's own
Arrow types and system-tag columns, as its `head()` shows them.
4 changes: 2 additions & 2 deletions changelog.d/ITL-991.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ the operator, displaying a side-effect stream delivered every side effect, and d
- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a
`StreamHead` (`orcapod.streams`): at most `n` rows as a PyArrow table with their exact
types, the row count when it is known without computing, and a `status` (`HeadStatus`) that
tells an empty result from one that is not computed. It never calls a function, runs an
operator, delivers a side effect, polls a source or writes. The notebook preview is
tells an empty result from one that is not computed. It never computes, polls or delivers a row (it reads only what is held or stored, or makes a bounded table read), never calls user code and never writes (see ITL-912 for how a head with no
rows learns its column types). The notebook preview is
`head()` rendered, with large cells (arrays, bytes, long text, long lists) summarised. A job
node with nothing in memory reads its whole stored slice to show its first rows. (ITL-991)

Expand Down
45 changes: 45 additions & 0 deletions design_issues/ITL-912-ZR1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
## Cross-cutting

### ZR1 — Zero-row tables were typed from the Python schema
**Status:** resolved
**Severity:** high
**Issue:** ITL-912

The table a stream or node gives when it has no row was built from its Python output schema
(`arrow_utils.make_empty_table`): an operator node before its run, after `clear_cache()`, with
no stored table, or after a zero-row async run (`OperatorJobNode._make_empty_table`); the
sync orchestrator's input for an upstream that produced no row
(`SyncPipelineOrchestrator._materialize_as_stream`); and a function pod's stream or node with
no row (`_table_from_pairs([])`); and a side-effect pod's stream or node that delivered no row
(`side_effects._table_of_rows`); and the table of a `head()` with no rows to show
(`StreamBase._head_without_rows`). So `int8` read as `int64`, list items and source-info
columns changed nullability, system-tag columns went missing (operator nodes) and a function
pod lost its input's tag types and its context and source-info columns. Zero-row and
non-empty results must have the same types, and the developer's rule (November 13, 2026) is
that an Arrow schema is never computed: every pod derives it from its inputs' schemas and every
source has one.

**Fix:** one function, `streams/base.typed_zero_row_table(stream)`, asks a stream for
`_typed_zero_row_table()`: zero rows, user tags, system tags, data and source info, with the
types its rows have, or `None` when it cannot know them without rows. An `ArrowTableStream`
cuts its table; a source gives the table or schema it holds (`_stream`; cached sources
delegate to the wrapped source, a polling source to what it polled, a proxy to its delegate);
an operator node or `DynamicPodStream` applies its operator (`static_process`) to its inputs'
zero-row tables; a function pod's stream or node joins its input's tag columns to the zero-row
form of its output `Data` (`function_zero_row_table`, `Data.function_output_zero_row_table`,
built from the same converter call and field builders a computed row uses); a side-effect
pod's stream or node gives its input's (it passes rows through), with the context column. Inputs a job
builds with no rows use `StaticOutputOperatorPod._zero_row_input` (row-built column order,
`_rows_to_table`'s source-info nullability rule), through `_materialize_upstream` on both
orchestrators. A head with no rows is built in one place, `StreamBase._head_without_rows`,
from the stream's `_zero_row_table(config)`: `as_table(columns=config)` with no rows. Nothing is stored, so no hash, stored schema or
path changes; nothing is computed, read, polled or called. Regressions:
`tests/test_pipeline/test_operator_node_empty_table_types.py` and the un-marked xfails in
`tests/test_pipeline/test_operator_node_async_readback.py`.

Still built from the Python schema (no Arrow schema exists without rows): a `PollingSource`
before its first poll and an unbound input slot or `SourceProxy` (declared with a Python
`Schema` only; reported to the developer as a design gap), a `DerivedSource` (its
`_empty_origin_table` is left as it is), and a loaded node with no live upstream and no stored
table. `Index` and `Pick` over zero rows lack system-tag columns (ITL-985). These remaining
cases are tracked in ITL-1089.
4 changes: 2 additions & 2 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -653,8 +653,8 @@ silent widening; convert explicitly.
the outputs of other operators and function pods. An input that delivers no rows gives no
rows on both paths. One limit remains: an async channel carries rows, not a schema, so
where the sync path returns an empty stream with typed columns the async path emits
nothing, and an operator node that received no rows reads back columns typed from its
Python output schema (ITL-858, ITL-912).
nothing. An operator node that received no rows still reads back the sync path's columns
and Arrow types: its operator applied to its inputs' zero-row tables (ITL-858, ITL-912).

```python
import pyarrow as pa
Expand Down
16 changes: 13 additions & 3 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,19 @@ In a notebook, a stream, source or job node shown at the end of a cell renders a
table: its first rows in tag order, with a line saying what they are. That preview is
`head()` asked for `sort_by_tags` and rendered, and `head(n, columns={"sort_by_tags": True})`
gives the same rows to work with. Neither ever computes anything: they show
only what is readily available, and never call a function, run an operator, deliver a side
effect, poll a source or write to a database. `run()` computes (see "Computing a stream
outside a job" below).
only what is readily available, never compute, poll or deliver a row (they read only what is held or stored, or make a bounded table read), never call your
code (a function, a predicate, a side effect) and never write to a database. `run()` computes
(see "Computing a stream outside a job" below).

With no rows to show, a head still has the columns and Arrow types the rows would have. To
learn them it applies each operator to empty (zero-row) tables, which processes no data: a
chain of N operators applies up to N operators to empty tables (about 60 ms for 20). A filter
is not applied at all, since its columns are its input's. A user-defined operator's
`static_process` is applied to zero-row tables and must not have effects. Two consequences: a pipeline whose
inputs disagree on a column's Arrow type (say `int8` against `int64`) raises from `head()`
the same `InputValidationError` its run raises (the notebook preview shows it as "could not
read"), and a filter whose predicate does not fit its columns shows a table in `head()` and
raises when it runs.

```python
from orcapod import function_pod
Expand Down
17 changes: 15 additions & 2 deletions src/orcapod/core/cached_function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@
from typing import TYPE_CHECKING, Any

from orcapod.core.datagrams import Datagram
from orcapod.core.function_pod import FunctionPodStream, WrappedFunctionPod
from orcapod.core.function_pod import (
FunctionPodStream,
WrappedFunctionPod,
function_zero_row_table,
)
from orcapod.core.streams.view import StreamView
from orcapod.system_constants import constants
from orcapod.types import ColumnConfig
from orcapod.types import ColumnConfig, Schema
from orcapod.contexts.stored import StaleTables
from orcapod.core.result_cache import ResultCache
from orcapod.hooks import InvocationStatus
Expand Down Expand Up @@ -458,6 +462,15 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]:
for tag, data in super().iter_data():
yield tag, _without_stored_meta(data)

def _zero_row_output_table(self) -> pa.Table | None:
"""The table of this stream's rows with no row, ``__computed`` included."""
return function_zero_row_table(
self._input_stream,
self.output_schema()[1],
self.data_context,
Schema({CachedFunctionPod.RESULT_COMPUTED_FLAG: bool}),
)

def _computed_prefix(
self, limit: int, contiguous: bool = True
) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]:
Expand Down
Loading
Loading