Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -800,6 +800,21 @@ data. The rules:
async path) and never stored, so its identity is the fixed ("DropNulls", None),
different from any list of names. System-tag, source-info, context and meta columns are
never included. An explicitly empty collection is not "no argument": it raises ValueError.
- `SemiJoin`'s match keys are exactly the names that are tag columns of both inputs
(developer, November 13, 2026, ITL-1088), from one helper, `SemiJoin._match_keys`, used by
validation, the sync path and the async path. A shared data column name, or a name that is a
tag on one side and data on the other, is ignored (by name only); the right input's data
columns are never read. No shared tag column: every left row is returned. Ignored names
raise `SemiJoinIgnoredColumnWarning` once per operator and set of names (provisional in
v0.3: may become an error); the async path warns only when no validation ran.
- `Join`, `MergeJoin` and `SemiJoin` accept a shared TAG key whose types differ only in
nullability (`T` against `T | None`), because a null key matches nothing (developer,
November 13, 2026, ITL-829). `Join` and `MergeJoin` give the key `T` when any input has it
`T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output is the left
input's schema as it is. The rule is `core/operators/key_nullability.py`
(`reconcile_key_nullability` on tag schemas before they are combined, `key_nullable` for the
Arrow field flag, set by re-wrapping arrays, never by a cast). Equal nullability takes the
unchanged path; any other type difference, and every data column, behaves as before.
- GroupBy requires every column in `by` to be a scalar tag column; raises InputValidationError
for unknown, data, or list-valued columns (list-valued usually means the stream came from
Batch — group before batching, not after). Members sort by non-key tag values with system
Expand Down
15 changes: 15 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -844,6 +844,21 @@ data. The rules:
async path) and never stored, so its identity is the fixed `("DropNulls", None)`,
different from any list of names. System-tag, source-info, context and meta columns are
never included. An explicitly empty collection is not "no argument": it raises `ValueError`.
- `SemiJoin`'s match keys are exactly the names that are tag columns of both inputs
(developer, November 13, 2026, ITL-1088), from one helper, `SemiJoin._match_keys`, used by
validation, the sync path and the async path. A shared data column name, or a name that is a
tag on one side and data on the other, is ignored (by name only); the right input's data
columns are never read. No shared tag column: every left row is returned. Ignored names
raise `SemiJoinIgnoredColumnWarning` once per operator and set of names (provisional in
v0.3: may become an error); the async path warns only when no validation ran.
- `Join`, `MergeJoin` and `SemiJoin` accept a shared TAG key whose types differ only in
nullability (`T` against `T | None`), because a null key matches nothing (developer,
November 13, 2026, ITL-829). `Join` and `MergeJoin` give the key `T` when any input has it
`T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output is the left
input's schema as it is. The rule is `core/operators/key_nullability.py`
(`reconcile_key_nullability` on tag schemas before they are combined, `key_nullable` for the
Arrow field flag, set by re-wrapping arrays, never by a cast). Equal nullability takes the
unchanged path; any other type difference, and every data column, behaves as before.
- `GroupBy` requires every column in `by` to be a scalar tag column; it raises
`InputValidationError` for unknown, data, or list-valued columns (the last usually means the
stream came from `Batch` — group before batching, not after). Members are sorted by non-key
Expand Down
45 changes: 45 additions & 0 deletions changelog.d/ITL-829.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
### Breaking changes and migration

#### `SemiJoin` keys on the shared tag columns only (ITL-1088)

`SemiJoin` used every name its two inputs share, tag or data, as a match key. A left row was
kept only when its tags **and** its values in every shared data column equalled those of some
right row, and a name that was a tag on one side and a data column on the other was a key too.
So the operator read data values, which only filters may do. The match keys are now exactly the
columns that are tag columns of both inputs. Any other shared name (data on both sides, or a
tag on one side and data on the other) is ignored, and the right input's data columns are
never read.

```python
left = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 2}], tag_columns=["id"])
right = DictSource(data=[{"id": 1, "y": 1}, {"id": 2, "y": 9}], tag_columns=["id"])
SemiJoin()(left, right)
# v0.2.0: only id 1 (id and y must both match)
# now: id 1 and id 2 (only the shared tag id decides), and a SemiJoinIgnoredColumnWarning
# naming 'y'
```

- **What changes.** If the two `SemiJoin` inputs share a data column name, the result now keeps
every left row whose shared tags match, whatever the data values. Before, rows whose shared
data values differed were dropped, so the new result always contains the old one. Filtering a
stream by a subset of itself (every data name shared, equal values) gives the same rows as
before. A shared data column whose type or nullability differs no longer raises, because it
is not a key.
- **The warning.** Each ignored shared name is listed in a `SemiJoinIgnoredColumnWarning`
(new, in `orcapod.errors`), once per operator and set of names. Silence it by renaming or
dropping the column on the right input before the semi join, or with
`warnings.filterwarnings("ignore", category=SemiJoinIgnoredColumnWarning)`. This behaviour is
provisional in v0.3 and may become an error.
- **Unchanged.** `SemiJoin`'s identity and pipeline hash do not change. The output schema is
still the left input's. With no shared column at all, every left row is still returned.
- **Stored operator caches.** A cached `SemiJoin` node keeps its table path, so what it stored
before the upgrade is still there. With caching off it computes the new result. With
`CacheMode.LOG` it computes the new result on its next run, passes it downstream, and adds
the rows it had not stored yet. With `CacheMode.REPLAY`, and for a pipeline loaded from a
file, it serves the stored rows, which are the old, smaller result. To get the new result
there, run the pipeline once with the operator cache in `LOG` mode, or clear the operator's
cache.

### Fixed

- `Join`, `MergeJoin` and `SemiJoin` accept a shared tag key that is non-nullable in one input and nullable in another (ITL-829). They raised `InputValidationError` ("Type conflict for key 'id': <class 'int'> vs int | None", or "Types int | None and <class 'int'> are not compatible") when the graph was built. A null key matches nothing, so `Join` and `MergeJoin` now give the key `T` when any input has it `T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output keeps the left input's schema as it is. Output prediction and computed Arrow fields agree on the sync and async paths. Any other type difference still raises, a data column that differs in nullability behaves as before in `Join` and `MergeJoin`, and every pipeline that worked before gives the same rows, schema and hashes.
27 changes: 27 additions & 0 deletions design_issues/ITL-1088-SJ3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## `src/orcapod/core/operators/semijoin.py` — nested columns and string/large_string keys

### SJ3 — `SemiJoin` matched on shared data columns
**Status:** resolved
**Severity:** high
**Issue:** ITL-1088

`SemiJoin` took every name its inputs share, tag or data, as a match key. A shared data column
`y` filtered the left rows by data value, and a name that was a tag on one side and a data column
on the other was matched across the two roles. The operator therefore read data content,
which only filters may do, and a shared data column of another type or nullability made the
graph fail to build.

**Fix:** developer decision, November 13, 2026: "Semijoin should key on shared tags only". The
keys come from one helper, `SemiJoin._match_keys(left_tag_keys, right_tag_keys)`, used by
validation, `binary_static_process` and `async_execute`: the names that are tag columns of both
inputs. Every other shared name is ignored (by name only), and the sync path reads only the
right input's key columns. With no shared tag every left row is returned, as before. Ignored
names raise `SemiJoinIgnoredColumnWarning` (from `_ignored_shared_names`) once per operator and
set of names. The developer considered making them an error and settled, for v0.3, on the
warning, because a stream semi-joined with a filtered copy of itself shares every data name;
the final behaviour is to be decided after use. Developer's note for that decision: once
referential tags are available, the suggestion should be to promote the data column to a tag
on both sides so that it is matched. The identity is unchanged; a cached node in `REPLAY`
serves its stored (old) rows until a `LOG` run stores the new result. Tests in
`tests/test_core/operators/test_semijoin_tag_keys.py`; the nested-key tests in
`test_semijoin_nested.py` now use a shared tag key.
26 changes: 26 additions & 0 deletions design_issues/ITL-829-SJ2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## `src/orcapod/core/operators/semijoin.py` — nested columns and string/large_string keys

### SJ2 — Joins rejected a tag key that differs only in nullability
**Status:** resolved
**Severity:** medium
**Issue:** ITL-829 (second case)

`SemiJoin`, `Join` and `MergeJoin` raised `InputValidationError` when a shared tag key was `T`
in one input and `T | None` in another, e.g. a stream with a declared non-nullable key against
a table built without a schema ("Type conflict for key 'id': <class 'int'> vs int | None" from
`schema_utils.intersection_schemas`, or "Types int | None and <class 'int'> are not
compatible" from `union_schemas`). The graph could not be built, although a null key matches
nothing and the result is clear. The strict xfail `test_binary_op_delegation` in
`tests/test_channels/test_node_async_execute.py` recorded it for `SemiJoin`.

**Fix:** developer decision, November 13, 2026: the three joins accept such a key.
`core/operators/key_nullability.py` holds the rule: `reconcile_key_nullability` gives both tag
schemas the non-nullable type for a key whose types are equal once `None` is removed (only tag
schemas; any other difference is left to raise as before), and `key_nullable` makes the output
key nullable only when every input has it nullable. `Join` (prediction, sync field flags after
`restore_schema_nullability`, async `_merge_tag_tables`) and `MergeJoin` (prediction, sync
`output_nullable`) give the key `T` when any input has it `T`; `SemiJoin` validates with the
reconciled schemas and keeps the left input's schema. Flags are set by re-wrapping arrays,
never by a cast. Inputs with equal nullability take the unchanged path, so no stored value or
golden hash moves. Tests in `tests/test_core/operators/test_join_key_nullability.py`, cases in
`test_async_sync_equivalence.py`, golden entries `pipeline/join_key_nullability`.
25 changes: 22 additions & 3 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,13 @@ not match each other (the SQL convention for nulls; `MergeJoin` and `SemiJoin` f
another null, an empty list is not a null, and NaN equals NaN. The async path gives the same
rows.

Because a null key matches nothing, a shared tag column may differ between the inputs in
nullability alone: `T` in one input and `T | None` in another is accepted by `Join`, `MergeJoin`
and `SemiJoin`. In the output of `Join` and `MergeJoin` the column is `T` when any input has it
`T`, and `T | None` only when every input has it nullable; `SemiJoin`'s output keeps the left
input's type. Any other type difference is still an error, and a data column is never
reconciled this way.

A name is a tag or a data column, never both: if one input has `x` as a tag column and another has it as a data column, `Join` (and `MergeJoin`) raises `InputValidationError` when the graph is built; rename one with `map_tags` or `map_data`.

Column order follows each operator's argument symmetry: commutative operators (`Join`,
Expand Down Expand Up @@ -145,6 +152,17 @@ stream. The right stream's data columns are discarded. SemiJoin is **not commuta
the order of inputs matters. The first stream is the one being filtered; the second stream
provides the set of matching tags.

The match keys are exactly the columns that are tag columns of both streams. A name the two
streams share as a data column, or a tag in one and a data column in the other, is not a key
and is ignored: the right stream's data columns are never read. If the streams share no tag
column, every left row is returned. Because such names were match keys before v0.3, SemiJoin
names them in a `SemiJoinIgnoredColumnWarning` (from `orcapod.errors`). This behaviour is
provisional in v0.3 and may become an error. To silence the warning, rename or drop the
column on the right stream before the semi join (`map_data`, `select_data_columns`,
`drop_data_columns`, or `map_tags` for a tag). A right stream must keep at least one data
column, so a filtered copy of the left stream (which shares every data name) can be reduced
to one renamed column, e.g. `subset.select_data_columns(["age"]).map_data({"age": "_age"})`.

```python
from orcapod.sources import DictSource
from orcapod.operators import SemiJoin
Expand Down Expand Up @@ -509,9 +527,10 @@ The rules:
was). The new hash is the one the row has in a stream that declared the column
non-nullable from the start.

Tag columns work the same way. A stream whose key column is `str | None` cannot be joined
with one whose key is `str` (`Join` never reconciles types), so narrow the key first. The
stream method is `drop_nulls`:
Tag columns work the same way. A stream whose key column is `str | None` can be joined with
one whose key is `str`: a null key matches nothing, so the joined key comes out `str` (see
`Join`). Narrowing still matters wherever a column must be `T` before it is consumed, such
as a pod parameter typed `str`. The stream method is `drop_nulls`:

```python
by_method = recordings.drop_nulls(["session", "start_ms"])
Expand Down
67 changes: 64 additions & 3 deletions src/orcapod/core/operators/join.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
require_shared_arrow_type,
require_tag_data_roles_agree,
)
from orcapod.core.operators.key_nullability import (
key_nullable,
reconcile_key_nullability,
)
from orcapod.core.streams import ArrowTableStream
from orcapod.errors import InputValidationError
from orcapod.protocols.core_protocols import (
Expand Down Expand Up @@ -111,7 +115,11 @@ def _joined_user_schemas(self, *streams: StreamProtocol) -> tuple[Schema, Schema
for other_stream in ordered_streams[1:]:
other_tag_schema, other_data_schema = other_stream.output_schema()
other_data_schema = self._without_meta_fields(other_data_schema)
tag_schema = schema_utils.union_schemas(tag_schema, other_tag_schema)
# A shared tag key that differs in nullability only is non-nullable
# (ITL-829): a null key matches nothing.
tag_schema = schema_utils.union_schemas(
*reconcile_key_nullability(tag_schema, other_tag_schema)
)
intersection_data_schema = schema_utils.intersection_schemas(
data_schema, other_data_schema
)
Expand Down Expand Up @@ -250,6 +258,9 @@ def static_process(self, *streams: StreamProtocol) -> StreamProtocol:
table = arrow_utils.restore_schema_nullability(
table, table_ref_schema, next_ref_schema
)
table = self._shared_key_nullability(
table, table_ref_schema, next_ref_schema, common_tag_keys
)

tag_keys.update(next_tag_keys)

Expand Down Expand Up @@ -278,6 +289,49 @@ def static_process(self, *streams: StreamProtocol) -> StreamProtocol:
tag_columns=tuple(tag_order),
)

@staticmethod
def _shared_key_nullability(
table: "pa.Table",
left_schema: "pa.Schema",
right_schema: "pa.Schema",
keys: Collection[str],
) -> "pa.Table":
"""Flag each shared tag key nullable only when both inputs have it nullable.

A null key matches nothing, so a key non-nullable in either input holds
no null after the join (ITL-829). The same arrays are re-wrapped under the
new field flags: no value or type changes. Equal flags keep the flag.

Args:
table: The joined table.
left_schema: The left input table's schema.
right_schema: The right input table's schema.
keys: The join keys; names absent from either schema are skipped.

Returns:
The table with each shared key's nullable flag decided by ``key_nullable``.
"""
schema = table.schema
changed = False
for name in keys:
left_index = left_schema.get_field_index(name)
right_index = right_schema.get_field_index(name)
index = schema.get_field_index(name)
if left_index < 0 or right_index < 0 or index < 0:
continue
nullable = key_nullable(
left_schema.field(left_index).nullable,
right_schema.field(right_index).nullable,
)
if schema.field(index).nullable != nullable:
changed = True
schema = schema.set( # pyright: ignore[reportUnknownMemberType] # stubs leave Field unparametrized
index, schema.field(index).with_nullable(nullable)
)
if not changed:
return table
return pa.Table.from_arrays(table.columns, schema=schema)

@staticmethod
def _without_meta(table: "pa.Table") -> "pa.Table":
"""Drop the meta (``__``-prefixed) columns from an input table.
Expand Down Expand Up @@ -757,7 +811,8 @@ def _merge_tag_tables(
left lacks. The async path joins in canonical order, so this is the
order ``_joined_user_schemas`` defines. A shared (join-key) column
takes the right tag's field, as ``static_process`` does
(``restore_schema_nullability`` lets the last reference win), and the
(``restore_schema_nullability`` lets the last reference win), nullable
only when both tags have it nullable (``key_nullable``), and the
left tag's values, as its Polars join does (they are equal because the
rows matched on it, except ``0.0`` against ``-0.0``).

Expand Down Expand Up @@ -787,10 +842,16 @@ def _merge_tag_tables(
# A tag without user columns is a zero-column, zero-row table.
for index, name in enumerate(table.column_names):
arrow_type = table.schema.types[index]
field = table.schema.field(index)
if name in types:
_ = require_shared_arrow_type("Join", name, types[name], arrow_type)
# A shared key is nullable only when both tags have it
# nullable: a null key matches nothing (ITL-829).
field = field.with_nullable(
key_nullable(fields[name].nullable, field.nullable)
)
types[name] = arrow_type
fields[name] = table.schema.field(index)
fields[name] = field
# A shared column keeps the left tag's values (the sync join's
# Polars join does): rows that match on ``0.0`` and ``-0.0``
# keep the left spelling.
Expand Down
Loading
Loading