Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,15 @@ data. The rules:
rows is built in one place, StreamBase._head_without_rows, from the stream's
_zero_row_table(config), which equals as_table(columns=config) with no rows; a kind
whose as_table shapes its table its own way overrides _zero_row_table with that shaping.
The zero-row table has the populated table's columns in the populated order too (ITL-1131).
A table built from rows (and a node's or operator input's table with no rows) is in row
order, user tags, system tags, data, source info, stated once in
StaticOutputOperatorPod._in_row_order; a node that keeps a run with no rows keeps
_zero_rows_in_row_order, and an operator's own empty result built from its input
goes through _as_row_built_table. An operator node keeps and stores the row order on every
path (_kept_output: a job run, sync or async, and node.run() on its own; ITL-1133),
and reads a stored operator cache in row order (_in_row_order_on_read). Never change
the order of any other populated table: stored tables depend on it.
- Type strictness (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
9 changes: 9 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -777,6 +777,15 @@ data. The rules:
rows is built in one place, `StreamBase._head_without_rows`, from the stream's
`_zero_row_table(config)`, which equals `as_table(columns=config)` with no rows; a kind
whose `as_table` shapes its table its own way overrides `_zero_row_table` with that shaping.
The zero-row table has the populated table's columns in the populated order too (ITL-1131).
A table built from rows (and a node's or operator input's table with no rows) is in row
order, user tags, system tags, data, source info, stated once in
`StaticOutputOperatorPod._in_row_order`; a node that keeps a run with no rows keeps
`_zero_rows_in_row_order`, and an operator's own empty result built from its input
goes through `_as_row_built_table`. An operator node keeps and stores the row order on every
path (`_kept_output`: a job run, sync or async, and `node.run()` on its own; ITL-1133),
and reads a stored operator cache in row order (`_in_row_order_on_read`). Never change
the order of any other populated table: stored tables depend on it.
- **Type strictness** (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
20 changes: 20 additions & 0 deletions changelog.d/ITL-1131.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
### Fixed

- A table with no rows has its populated table's columns in the same order (ITL-1131).
A `Join` or `MergeJoin` job node after a run whose input had no rows (sync and async)
listed its data columns before its system tags, where the same node after a run with rows
lists them after. `Index` and `Pick` over an input with no rows, or whose rows were all
skipped, dropped the system-tag columns, and `Pick` also gave its source-info columns as
nullable and lost the token column of an `out=` column. Every operator's zero-row table now
equals its populated table in column names, order, Arrow types and nullability.
- An operator node has one column order on every path, in memory and stored (ITL-1133): the
row order of a job run (user tags, system tags, data, source info). A node run on its own
(`node.run()`, without a store or in `LOG`) kept and stored its operator's own order (a join
lists its data before its system tags), so its table differed from the same node's in a job
and from its own table before a run, and a store written by both refused the second append.
An operator cache stored in the operator's order by an earlier build reads in row order and
accepts new rows; it is not rewritten and its path does not change. No hash or record id
changes: a record id does not depend on column order.
- `DropTagColumns` predicts the system-tag columns its result has (ITL-1135):
`output_schema(columns={"system_tags": True})` and `output_schema(all_info=True)` left them
out, though the computed table keeps them. No hash, record id or stored path changes.
28 changes: 28 additions & 0 deletions design_issues/ITL-1131-ZR2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
## Cross-cutting

### ZR2 — A zero-row table had a different column order from its populated table
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1131

ITL-912 (ZR1) gave every table with no rows the Arrow types and nullability of the rows, but
not always their column order or all their columns. A `Join` or `MergeJoin` job node after a
run whose input had no rows (sync `OperatorJobNode.execute()` and the async empty branch)
kept the operator's own result, which lists data before system tags, while a run with rows
keeps `_materialize_to_stream` (row order: user tags, system tags, data, source info).
`Index` and `Pick` built their empty result from the input's table without system tags
(`Pick` also without source info, so its source-info columns came out nullable, and without
the token of an `out=` column). The row order was restated in two places
(`StaticOutputOperatorPod._zero_row_input`, `OperatorJobNode._typed_zero_row_table`).

**Fix:** the row order has one owner, `StaticOutputOperatorPod._in_row_order`, used by
`_zero_row_input`, `OperatorJobNode._typed_zero_row_table`, `_zero_rows_in_row_order` (what a
node keeps after a sync or async run with no rows) and `_as_row_built_table` (row order plus
source-info flags; `Index` and `Pick` build their empty result from the input with system
tags and source info through it). `tests/test_pipeline/test_zero_row_column_order.py` checks
every operator and a function pod, plain stream and job node, sync and async, before a run
and after a zero-input run, with one and two system-tag pairs. No populated or stored order
changed.

Found alongside it and fixed in the same change: an operator node run on its own kept and
stored its operator's order (ZR3, ITL-1133).
27 changes: 27 additions & 0 deletions design_issues/ITL-1133-ZR3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## Cross-cutting

### ZR3 — An operator node kept and stored two column orders
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1133

Found during ITL-1131 (ZR2). A job run keeps and stores an operator node's rows in row order
(`_materialize_to_stream`: user tags, system tags, data, source info), but a node run on its
own (`OperatorJobNode.run()`, `_compute_and_store`) kept its operator's result and stored it in
the operator's own order (a join lists its data before its system tags; every operator that
keeps its input's order did the same over a source, which lists system tags after data). So one
node had two populated orders, its table before a run (row order) matched only one of them, and
a store holding rows of both refused the second append (`StoreWriteError`, schema mismatch, in
the in-memory store and in a Delta store with a pending batch). Only the operator cache
(`odb_v1`) was affected: function pipeline and result tables, side-effect logs and cached
sources store one order on every path. `odb_v1` dates from 2026-11-12 (ITL-967) and was never
released.

**Fix:** every way a node runs its operator keeps `StaticOutputOperatorPod._kept_output` (the
rows as `_materialize_to_stream` builds them, or the zero-row result in row order), so a node
run on its own keeps and stores the row order. The one operator-cache read path
(`OperatorJobNode._read_current_context_records`) puts the user columns of a stored table in
row order (`_in_row_order_on_read`; bookkeeping columns stay in place), so a table an earlier
build stored in the operator's order reads like any other and accepts row-order appends
(developer, November 14, 2026: option B, no layout bump). Record ids do not depend on column
order, so no hash moves. Reproduction: `superpowers/scripts/itl_1133_store_column_order.py`.
19 changes: 19 additions & 0 deletions design_issues/ITL-1135-SP1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## Cross-cutting

### SP1 — DropTagColumns predicted no system-tag columns
**Status:** resolved
**Severity:** low
**Issue:** ITL-1135

Found during ITL-1131. `DropTagColumns.unary_output_schema` kept only the tag-schema keys
that were remaining user tags, so a prediction that asked for system tags
(`columns={"system_tags": True}`, `all_info=True`) left them out, while the computed result
keeps them; `SelectTagColumns` was right. The golden stream summary of `DropTagColumns`
pinned that wrong prediction (`system_tag_columns: []`).

**Fix:** drop only the named user tags from the predicted tag schema. A sweep checks every
operator's prediction against `static_process(...).output_schema(...)` for every column
config, plain stream and job node, with one and two system-tag pairs. It found one more
disagreement, not fixed here: `Batch` and `GroupBy` predict `list[T | None]` for a nullable
data column whose computed table reads as `list[T]` (strict xfail in the sweep): the
Arrow-to-Python reading drops list item nullability (ITL-1136).
83 changes: 57 additions & 26 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -1251,7 +1251,36 @@ def _read_current_context_records(self) -> "pa.Table | None":
self._pipeline_database.base_path + self._stored_table_path,
stored_rows - records.num_rows,
)
return records.drop([constants.DATA_CONTEXT_COL])
return self._in_row_order_on_read(records.drop([constants.DATA_CONTEXT_COL]))

def _in_row_order_on_read(self, records: "pa.Table") -> "pa.Table":
"""Put a stored table's user columns in row order; bookkeeping stays in place.

A node run on its own stored its rows in its operator's column order before
ITL-1133 (a join lists its data before its system tags); every writer now
stores the row order (``StaticOutputOperatorPod._in_row_order``). Read in row
order, such a table gives the columns of every other path, and a table in row
order is returned unchanged. Only the order changes: no column is added,
dropped, renamed or retyped, and the bookkeeping columns (``__``, and
``HASH_COLUMN_NAME``) keep their positions.
"""
names = records.column_names

def bookkeeping(name: str) -> bool:
return (
name.startswith(constants.META_PREFIX) or name == self.HASH_COLUMN_NAME
)

user = [n for n in names if not bookkeeping(n)]
ordered = iter(
StaticOutputOperatorPod._in_row_order(
records.select(user), self.keys()[0]
).column_names
)
order = [n if bookkeeping(n) else next(ordered) for n in names]
if order == names:
return records
return records.select(order)

def _typed_zero_row_table(self) -> "pa.Table | None":
"""This node's zero-row table: its operator applied to zero-row inputs.
Expand Down Expand Up @@ -1281,16 +1310,11 @@ def _typed_zero_row_table(self) -> "pa.Table | None":
# table is read.
operator = cast(StaticOutputOperatorPod, self._operator)
result = operator.zero_row_result(*inputs)
table = result.as_table(columns=ZERO_ROW_INPUT_COLUMNS)
# The column order of the node's rows (``_rows_to_table``): user tags, system
# tags, data, source info; an operator's own table may order them otherwise
# (a join lists its data before its system tags).
tag_keys, data_keys = result.keys()
names = table.column_names
system_tags = [n for n in names if n.startswith(constants.SYSTEM_TAG_PREFIX)]
source_info = [n for n in names if n.startswith(constants.SOURCE_PREFIX)]
data = [n for n in data_keys if n in names]
return table.select([*tag_keys, *system_tags, *data, *source_info])
# In the column order of the node's rows; an operator's own table may order
# them otherwise (a join lists its data before its system tags).
return StaticOutputOperatorPod._in_row_order(
result.as_table(columns=ZERO_ROW_INPUT_COLUMNS), result.keys()[0]
)

def _make_empty_table(self) -> "pa.Table":
"""Build the zero-row table this node reads as when it holds no rows.
Expand Down Expand Up @@ -1478,13 +1502,10 @@ def execute(
# Materialize
output = list(result_stream.iter_data())

# Cache
if output:
self._cached_output_stream = (
StaticOutputOperatorPod._materialize_to_stream(output)
)
else:
self._cached_output_stream = result_stream
# Cache, in row order with or without rows (ITL-1131, ITL-1133)
self._cached_output_stream = StaticOutputOperatorPod._kept_output(
result_stream, output
)

self._update_modified_time()

Expand All @@ -1504,15 +1525,23 @@ def execute(
self.clear_cache()
raise

def _compute_output(self) -> StreamProtocol:
"""Run the operator over the node's inputs and return what the node keeps.

The rows in row order, as a job run keeps them (``_kept_output``), so a node
run on its own has the columns, in memory and stored, of the same node in a
job (ITL-1133). The rows are computed here, not when first read.
"""
result = self._operator.process(*self._input_streams)
return StaticOutputOperatorPod._kept_output(result, list(result.iter_data()))

def _compute_and_store(self, *, store: bool = True) -> None:
"""Compute operator output, optionally store in DB.

Args:
store: Whether a ``LOG`` node stores the output (not in a load-only run).
"""
self._cached_output_stream = self._operator.process(
*self._input_streams,
)
self._cached_output_stream = self._compute_output()

if self._cache_mode == CacheMode.OFF or not store:
self._update_modified_time()
Expand Down Expand Up @@ -1625,9 +1654,7 @@ def run(self, *, load_only: bool = False) -> StreamView:
commit_run([self._pipeline_database], [self])
self._run_load_only = load_only
else:
self._cached_output_stream = self._operator.process(
*self._input_streams,
)
self._cached_output_stream = self._compute_output()
self._update_modified_time()
computed = True
except BaseException:
Expand Down Expand Up @@ -2016,9 +2043,13 @@ async def forward() -> None:
for tap, upstream in zip(taps, self._input_streams, strict=True)
)
)
# Computed, as execute() computes it, so the node reads as run.
# Computed, as execute() computes it, so the node reads as run, and
# kept in the column order of a run with rows, as execute() keeps it
# (ITL-1131).
_ = list(empty_result.iter_data())
self._cached_output_stream = empty_result
self._cached_output_stream = (
StaticOutputOperatorPod._zero_rows_in_row_order(empty_result)
)

self._update_modified_time()

Expand Down
7 changes: 3 additions & 4 deletions src/orcapod/core/operators/column_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -387,10 +387,9 @@ def unary_output_schema(
tag_schema, data_schema = stream.output_schema(
columns=columns, all_info=all_info
)
tag_columns, _ = stream.keys()
new_tag_columns = [c for c in tag_columns if c not in self.columns]

new_tag_schema = {k: v for k, v in tag_schema.items() if k in new_tag_columns}
# Only the named user tags go: the system-tag columns stay, as in the
# computed result (ITL-1135).
new_tag_schema = {k: v for k, v in tag_schema.items() if k not in self.columns}

return Schema(new_tag_schema), data_schema

Expand Down
6 changes: 4 additions & 2 deletions src/orcapod/core/operators/index.py
Original file line number Diff line number Diff line change
Expand Up @@ -371,8 +371,10 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
# The source-info columns are part of it: the projected column's token
# type is the input's with one list level removed, as for a non-empty
# result (ITL-1046), and every other token column keeps its input type.
# The system tags are part of it too: a result with rows keeps each row's
# (ITL-1131).
input_schema = arrow_utils.normalize_schema_to_large_types(
stream.as_table(columns={"source": True}).schema
stream.as_table(columns={"source": True, "system_tags": True}).schema
)
element_field = pa.field(
self.out or self.column,
Expand Down Expand Up @@ -404,7 +406,7 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
output_schema = input_schema.append(element_field) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
output_schema = output_schema.append(token_field) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
return ArrowTableStream(
output_schema.empty_table(),
self._as_row_built_table(output_schema.empty_table(), tag_columns),
tag_columns=tag_columns,
data_context=stream.data_context,
)
Expand Down
17 changes: 15 additions & 2 deletions src/orcapod/core/operators/pick.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from orcapod.core.operators.base import UnaryOperator
from orcapod.core.streams.arrow_table_stream import ArrowTableStream
from orcapod.errors import InputValidationError
from orcapod.system_constants import constants
from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol
from orcapod.types import ColumnConfig, ContentHash, Schema
from orcapod.utils import arrow_utils, schema_utils
Expand Down Expand Up @@ -373,8 +374,10 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
# The schema is the input's Arrow schema (large-normalized, as for
# a non-empty result) with only the projected column changed.
tag_columns, _ = stream.keys()
# The system tags and source info are part of it: a result with rows keeps
# each row's (ITL-1131).
input_schema = arrow_utils.normalize_schema_to_large_types(
stream.as_table().schema
stream.as_table(columns={"source": True, "system_tags": True}).schema
)
value_field = pa.field(
self.out or self.column,
Expand All @@ -383,13 +386,23 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
),
nullable=self._output_nullable(),
)
# A picked value's token is a string (``_process_one``), typed by the
# source-info type rule.
token_name = f"{constants.SOURCE_PREFIX}{self.out or self.column}"
token_field = pa.field(
token_name, arrow_utils.source_info_arrow_type(token_name)
)
if self.out is None:
position = input_schema.get_field_index(self.column)
output_schema = input_schema.set(position, value_field) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
output_schema = output_schema.set(
output_schema.get_field_index(token_name), token_field
) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
else:
output_schema = input_schema.append(value_field) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
output_schema = output_schema.append(token_field) # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
return ArrowTableStream(
output_schema.empty_table(),
self._as_row_built_table(output_schema.empty_table(), tag_columns),
tag_columns=tag_columns,
data_context=stream.data_context,
)
Expand Down
Loading
Loading