Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
227f1b6
test(pipeline): bind refuses a source whose column types differ (fail…
Nov 14, 2026
715b362
fix(pipeline)!: bind and bound_source refuse a source whose column ty…
Nov 14, 2026
e024d40
docs: changelog fragment, design issue BT1 and bind docs sentence (IT…
Nov 14, 2026
4a92072
style(tests): ruff format the bind type check tests
Nov 14, 2026
9116458
Merge remote-tracking branch 'origin/main' into eywalker/itl-1112-bin…
Nov 14, 2026
582049a
docs(plan): ITL-1112 round-2 decisions (narrower bind, cast, load)
Nov 14, 2026
0ced55c
test(pipeline): a slot of T | None accepts a source of T, read as dec…
Nov 14, 2026
0484c23
feat(pipeline): a slot of T | None accepts a source of T and gives it…
Nov 14, 2026
acc423b
docs: changelog, design issue and bind docs for the narrower-source r…
Nov 14, 2026
091bf52
docs(source_node): docstrings of the read paths name the declared-sch…
Nov 14, 2026
5c9cfb9
test(pipeline): no accepted bind changes a node's path or table schem…
Nov 14, 2026
493252c
test(pipeline): narrow the first snapshot before the loaded-job check…
Nov 14, 2026
b5e6914
fix(source_node): columns in declared order, a stream's system tags t…
Nov 14, 2026
390c5d2
fix(source_node): a narrower bind needs exactly one system-tag pair; …
Nov 14, 2026
2ee833d
Merge origin/main (ITL-912 typed zero-row tables) into ITL-1112
Nov 14, 2026
de1ed25
fix(source_node): a slot's zero-row table is its bound source's in th…
Nov 14, 2026
c6dfdc1
fix(source_node)!: every bind refuses a source without exactly one sy…
Nov 14, 2026
9b842de
fix(source_node): a pipeline's own leaf at compile is taken as it is,…
Nov 14, 2026
0d9fb60
fix(source_node)!: a pipeline built over a raw stream is refused too …
Nov 14, 2026
1ee8490
test(polling_source): bind a source, not a raw stream, to a source no…
Nov 14, 2026
882d89e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1112-bin…
Nov 14, 2026
e83bcb5
fix(source_node): a slot predicts its record id as bytes; #387's rebi…
Nov 14, 2026
47c5bad
fix(source_node): structural classes of one identity fit; a bound sou…
Nov 14, 2026
3ed3b6f
Merge origin/main (ITL-1095 job node shows its current run) into ITL-…
Nov 14, 2026
4ffdef0
test(current_run): bind a source, not a raw stream, to a source node …
Nov 14, 2026
5c9ac6c
Merge remote-tracking branch 'origin/main' into eywalker/itl-1112-bin…
Nov 14, 2026
e5155e3
test(large_normalization): a raw ArrowTableStream is refused as a job…
Nov 14, 2026
ffcdcf6
fix(source_node): a bound source's system-tag pair is given source id…
Nov 14, 2026
f2aaf91
docs(source_node): column order is not part of the bind invariant (IT…
Nov 14, 2026
777afbd
Merge remote-tracking branch 'origin/main' into eywalker/itl-1112-bin…
Nov 14, 2026
fd1278d
fix(source_node): a slot shows its declared types for a structural cl…
Nov 14, 2026
fc2e177
Merge remote-tracking branch 'origin/main' into eywalker/itl-1112-bin…
Nov 14, 2026
af162ef
test(node_view): bind source slots to streams over a source, not a ra…
Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -816,6 +816,29 @@ data. The rules:
### Important implementation details

- ArrowTableSource raises ValueError if any tag_columns are not in the table.
- The pipeline determines every node's schema; a bound source never changes it (developer,
November 14, 2026, ITL-1112). No accepted bind changes any node's stored path or the
columns of any node's or stored table (names, tag or data roles, types, nullable flags,
system-tag column names); only values differ. Column order is not part of it: matching a
source to a slot, columns match by name within the tag group and within the data group,
and their order never decides acceptance (developer, November 14, 2026). Do not reorder
columns without a reason. SourceNodeBase.validate is the one check, used by
PipelineJob.bind, the bound_source setter (and SourceJobNode.__init__) and
PipelineJob.load: per column the declared Python type (structural classes of one identity
are one type, equivalent_hints), or T where the slot declares T | None; and exactly
one source-id (str) and one record-id (bytes) system-tag column, no other. Anything
else raises SourceSpecMismatchError before anything is bound. The
source's rows are given as the declared schema at one place, _DeclaredForm (nullable
fields over the same arrays; the system-tag column NAMES of the declared schema hash,
taken from the source's own pair by prefix, the VALUES the source's own; as a stable
convenience, not a guarantee, the declared column order, source id before record id). Limits: the check compares Python types, so a difference in Arrow width only
(int32 against int64) is not yet refused (ITL-949). A source whose system-tag columns
are not exactly one source-id and one record-id column (a joined or merged stream, a raw
stream with none) is refused by the same check (developer, November 14, 2026, ITL-1126);
binding one properly is ITL-1127. That includes a pipeline's input: Pipeline.compile
binds each leaf through SourceJobNode.from_stream, so a pipeline built directly over a
raw ArrowTableStream is refused (developer, November 14, 2026; wrap the table as a
source). Never pass a bound stream's own system tags through.
- A source id is the source's name (developer, November 11, 2026; ITL-957). A default id
built from a hash is <source kind>:<algorithm label>:<hex> and is built only by
core/sources/source_ids.hash_source_id: never format one by hand, and never write a label
Expand Down
23 changes: 23 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -858,6 +858,29 @@ data. The rules:
### Important implementation details

- `ArrowTableSource.__init__` raises `ValueError` if any `tag_columns` are not in the table.
- The pipeline determines every node's schema; a bound source never changes it (developer,
November 14, 2026, ITL-1112). No accepted bind changes any node's stored path or the
columns of any node's or stored table (names, tag or data roles, types, nullable flags,
system-tag column names); only values differ. Column order is not part of it: matching a
source to a slot, columns match by name within the tag group and within the data group,
and their order never decides acceptance (developer, November 14, 2026). Do not reorder
columns without a reason. `SourceNodeBase.validate` is the one check, used by
`PipelineJob.bind`, the `bound_source` setter (and `SourceJobNode.__init__`) and
`PipelineJob.load`: per column the declared Python type (structural classes of one identity
are one type, `equivalent_hints`), or `T` where the slot declares `T | None`; and exactly
one source-id (`str`) and one record-id (`bytes`) system-tag column, no other. Anything
else raises `SourceSpecMismatchError` before anything is bound. The
source's rows are given as the declared schema at one place, `_DeclaredForm` (nullable
fields over the same arrays; the system-tag column NAMES of the declared schema hash,
taken from the source's own pair by prefix, the VALUES the source's own; as a stable
convenience, not a guarantee, the declared column order, source id before record id). Limits: the check compares Python types, so a difference in Arrow width only
(`int32` against `int64`) is not yet refused (ITL-949). A source whose system-tag columns
are not exactly one source-id and one record-id column (a joined or merged stream, a raw
stream with none) is refused by the same check (developer, November 14, 2026, ITL-1126);
binding one properly is ITL-1127. That includes a pipeline's input: `Pipeline.compile`
binds each leaf through `SourceJobNode.from_stream`, so a pipeline built directly over a
raw `ArrowTableStream` is refused (developer, November 14, 2026; wrap the table as a
source). Never pass a bound stream's own system tags through.
- A source id is the source's name (developer, November 11, 2026; ITL-957). A default id
built from a hash is `<source kind>:<algorithm label>:<hex>` and is built only by
`core/sources/source_ids.hash_source_id`: never format one by hand, and never write a label
Expand Down
2 changes: 1 addition & 1 deletion changelog.d/ITL-1111.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
### Fixed

- Defining a pipeline whose stages split a stream into two branches and join them again no longer takes time exponential in the number of stages (about 4x per stage before: 9 stages took 13 seconds to build, with no row computed; 20 stages now build in well under a second, as streams and inside a `PipelineJob`) (ITL-1111). Three graph walks asked a shared upstream once per path: every operator stream, function pod stream and node predicted its output schema again from its upstreams on each call, and a join asks each input several times; `head()` and the notebook preview walked every path to check whether reading the schema would poll a source; and each operator or function pod call formatted its input streams, whose description nests every upstream, into a debug log message even with debug logging off. A stream or node now keeps its predicted output schema per column selection (`output_schema()`, `keys()`), the poll check visits each upstream once, and the log message is formatted only when debug logging is on (see Changed for the description). Job compilation, `save`, `PipelineJob.load` and a run's schema queries are linear in the nodes as well: each node is asked a bounded number of times. A schema that includes the system tags is still as wide as the joins make it, because each join keeps both inputs' system-tag columns: stacked split-and-join stages over one source double their number per stage (ITL-1114). `head()` and the notebook preview of a stream that has not run build their table of no rows from a table that carries the system tags whatever columns are shown, so over such stacked stages they are as wide as the joins make them (ITL-1132). No hash, stored table or golden value changes. `PipelineJob.bind(sources=...)` makes every node of the job predict its schema again, so binding a source whose column has another type still changes the downstream schemas; assigning `bound_source` on a source node directly, or setting a node's `upstreams`, does not reach the nodes downstream of it, so use `bind`.
- Defining a pipeline whose stages split a stream into two branches and join them again no longer takes time exponential in the number of stages (about 4x per stage before: 9 stages took 13 seconds to build, with no row computed; 20 stages now build in well under a second, as streams and inside a `PipelineJob`) (ITL-1111). Three graph walks asked a shared upstream once per path: every operator stream, function pod stream and node predicted its output schema again from its upstreams on each call, and a join asks each input several times; `head()` and the notebook preview walked every path to check whether reading the schema would poll a source; and each operator or function pod call formatted its input streams, whose description nests every upstream, into a debug log message even with debug logging off. A stream or node now keeps its predicted output schema per column selection (`output_schema()`, `keys()`), the poll check visits each upstream once, and the log message is formatted only when debug logging is on (see Changed for the description). Job compilation, `save`, `PipelineJob.load` and a run's schema queries are linear in the nodes as well: each node is asked a bounded number of times. A schema that includes the system tags is still as wide as the joins make it, because each join keeps both inputs' system-tag columns: stacked split-and-join stages over one source double their number per stage (ITL-1114). `head()` and the notebook preview of a stream that has not run build their table of no rows from a table that carries the system tags whatever columns are shown, so over such stacked stages they are as wide as the joins make them (ITL-1132). No hash, stored table or golden value changes. `PipelineJob.bind(sources=...)` makes every node of the job predict its schema again (a source whose column has another type is refused, ITL-1112); assigning `bound_source` on a source node directly, or setting a node's `upstreams`, does not reach the nodes downstream of it, so use `bind`.

### Changed

Expand Down
85 changes: 85 additions & 0 deletions changelog.d/ITL-1112.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
### Breaking changes and migration

#### Binding refuses a source whose column types the slot does not declare (ITL-1112)

`PipelineJob.bind(sources=...)` checked a source against its slot by column names only;
assigning `job.nodes[name].bound_source` and the file sources `PipelineJob.load` rebuilds were
not checked at all. A source with other column types was accepted: every pipeline hash, built
from the slot's declared schema, stayed the same, so a run wrote rows of the new types to the
stored paths of the declared ones. Every way of binding now compares the source's tag and data
columns with the slot's: names, role (tag or data) and Python type, the inputs of the pipeline
hash. A column must have the declared type (two `orcapod.dataclass` classes of one identity are
one type), or `T` where the slot declares `T | None`. Anything
else raises `SourceSpecMismatchError` naming the slot, the column and both types, before
anything is bound; a refused `bind` leaves the job as it was, its database included.

```python
slot = SourceNode(name="src", tag_schema={"id": int}, data_schema={"x": int})
job.bind(sources={"src": DictSource([{"id": 0, "x": "a"}], tag_columns=["id"])})
# v0.2.0: accepted; rows with x: str were stored under the path of x: int
# now: SourceSpecMismatchError: ... data column 'x' is declared int but the source has str
```

Refused now, accepted before: a column of another type, and a nullable column (`int | None`)
for a slot that declares it non-null (`int`). A slot recorded from data without a null (a
`DictSource`, `ListSource`, `CSVSource` or `DataFrameSource` infers `int` then) refuses a
later source whose data has one; a `pa.table` and a Delta table written from one are nullable
(`int | None`) in every column, tags included, unless their schema says otherwise.
`SourceSpecMismatchError` is now an `InputValidationError` as well as a `ValueError`. A file
source that `PipelineJob.load` rebuilds and its slot refuses is handled like one that could not
be rebuilt: a warning names the slot and the slot stays unbound.

Migration: bind a source whose columns have the slot's types (convert them, or give the
Arrow table a schema with the slot's nullability), or record the pipeline with a slot of the
source's types, which gives it its own stored paths.

#### Binding refuses a joined or merged stream, and a stream without system tags (ITL-1126)

A slot's rows carry one source-id and one record-id system-tag column, under the slot's names.
A stream with one pair per input (the result of a `Join` or `MergeJoin`), or with none (an
`ArrowTableStream` over a table without them), cannot give its rows that way; it was bound
before with its own system-tag columns, so the system-tag columns of the slot's table, and
of every table after it, depended on what was bound. Every way of binding (`bind`, the
`bound_source` setter, `SourceJobNode(..., bound_source=...)`, `PipelineJob.load`) now refuses
it with `SourceSpecMismatchError`, naming the slot and the system-tag columns found, whatever
its column types; so is a stream with any other system-tag column, or whose source id is not
`str` or record id not `bytes`. So does building a pipeline or job directly over such a stream
(`with job: op(ArrowTableStream(...))`), which binds it to the pipeline's input slot.

Migration: wrap a raw stream's table as a source,
`ArrowTableSource(stream.as_table(), tag_columns=[...])`; make a join part of the pipeline
(bind its inputs to slots of their own), or store the joined stream and bind the stored
source. Binding such a stream properly is planned (ITL-1127).

### Changed

- A slot that declares `x: int | None` accepts a source whose `x` is `int` (ITL-1112). The slot
gives that source's rows as rows of its declared schema: `output_schema()`, `as_table()`,
`iter_data()`, `head()` and the async path show `x` nullable, and the system-tag columns
carry the declared schema's names. The rows are exactly those of a source with a nullable
`x`, the same values and the same source id (row hashes and record ids included), so both
write to the same stored tables. Nothing downstream changes: every pipeline hash and
predicted schema is the declared one. The source node's `content_hash` and source id stay
the bound source's own.
- The pipeline determines every node's schema (ITL-1112): no accepted bind changes a node's
stored path or the columns of any node's or stored table (names, tag or data roles, types,
nullable flags, system-tag column names). A stream bound to a slot (a source passed
through a filter or a rename, with one system-tag pair) gives its system-tag columns the
slot's names; their values stay the stream's own. It passed through as it was before.
Column order is not part of that: a source matches its slot by column name within the tag
group and within the data group, whatever their order. As a convenience, a source whose
columns come in another order is shown in the slot's declared order, its system tags
source id first.

### Fixed

- A source slot's predicted schema with system tags (`output_schema(columns={"system_tags": True})`,
`all_info=True`) gave the record-id column as `str`; every source's rows have it as `bytes`
(Arrow `large_binary`). It is `bytes` now, for an unbound slot and for every bind, so the
slot's prediction and every prediction after it are what the rows are, whatever is bound
(ITL-1112).

### Known limitations

- The bind check compares Python types, so a source that differs from its slot only in
Arrow width (`int32` against `int64`) is still accepted (ITL-949).
3 changes: 2 additions & 1 deletion design_issues/ITL-1111-GW1.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ Call-count tests over 20 stages: `tests/test_pipeline/test_graph_walk_cost.py`.

Left as they are: assigning `SourceJobNode.bound_source` directly, or setting a node's
`upstreams`, clears no memo of the nodes downstream of it, which keep their schema (their
pipeline hashes are equally stale today; strict xfail tests record both). The risk is bounded:
pipeline hashes are equally stale today; a strict xfail test records the `upstreams` case, and a
`bound_source` of another column type is refused, ITL-1112). The risk is bounded:
nothing in `src` calls the `upstreams` setter (the loaders build a node with `__new__` and set
`_input_streams` before any schema is asked; only tests call it), and after ITL-1112 a bind
cannot change a slot's schema. Cascading the clear (a registry of dependents, or a setter that
Expand Down
44 changes: 44 additions & 0 deletions design_issues/ITL-1112-BT1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
## `src/orcapod/core/nodes/source_node.py`

### BT1 — Binding a source of other column types left every pipeline hash unchanged
**Status:** resolved
**Severity:** high
**Issue:** ITL-1112

`PipelineJob.bind(sources=...)` checked a source against its slot by column names only;
assigning a source node's `bound_source`, and the file sources `PipelineJob.load` rebuilds, were
not checked at all. A source whose `x` is `str`, bound to a slot declaring `x: int`, was
accepted on a fresh and on a loaded job. Every pipeline hash is built from the slot's declared
schema, so none moved, while `output_schema()` passed the bound source through: a downstream
`MapData` reported `y: str` under the pipeline hash of `y: int`. A run then sent `str` values to
a function typed `int` and wrote its rows to the stored paths of the declared types
(`schema:<pipeline hash of y: int>/pdb_v2` and the pod's result table). The same held for a
nullability difference (`int` against `int | None`), which has its own pipeline hash.

**Fix:** `SourceNodeBase.validate` compares, column by column, the source's tag and data schemas
with the declared ones: names, role (tag or data) and Python type, the inputs of the pipeline
hash. A column must have the declared type or, where the slot declares `T | None`, `T`
(developer, November 14, 2026); nothing else fits. Column order and `optional_fields` are not
compared; neither enters the pipeline hash. The `bound_source` setter (which `__init__` now goes
through) calls the same check before it assigns, `bind` checks every source before it changes
anything, its database included, and `load` handles a rebuilt source its slot refuses like one
it could not rebuild. `SourceSpecMismatchError` names the slot, the column and both types and
is now an `InputValidationError` as well as a `ValueError`.

A narrower source is given as the declared schema at one place, `_DeclaredForm`: the same
arrays under nullable fields, and the two system-tag columns renamed from the source's schema
hash to the declared one (their names carry the schema hash). `iter_data`, `async_iter_data`,
`as_table`, `head` and `output_schema` go through it, so the rows equal those of a nullable
source with the same values and source id, row hashes and record ids included, and both append
to one stored table. The node's `content_hash` and source id stay the source's own. The same
form gives a stream's one system-tag pair the slot's names (taken from the stream's own
columns by prefix), so no accepted bind changes a node's path or its columns (names, roles,
types, nullable flags, system-tag names; column order is not part of that, and the form's
declared order is a convenience). Two Arrow types that are one Python type (`int32` and `int64`)
are still not told apart (ITL-949). A source whose system-tag columns are not exactly one
source-id and one record-id column (a joined or merged stream, a raw stream with none) is
refused by the same check (developer, November 14, 2026, ITL-1126; binding one properly is
ITL-1127), also as a pipeline's input: a pipeline built directly over a raw stream is
refused at compile. The slot's own prediction types the record-id system-tag column `bytes`,
as every source's rows have it (it said `str`), so a slot's prediction is the same unbound
and under every accepted bind. No hash, golden value or stored path changed.
Loading
Loading