Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion changelog.d/ITL-1023.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@

### Known limitations

- `MergeJoin` and `MapColumns` raise "not supported with referential tags yet"
- `MapColumns` raises "not supported with referential tags yet"
(operator-node storage and `DerivedSource` over a referential stream are supported,
see ITL-1019 above).
`PollingSource` keeps its tags plain. The identities of referential streams may still
Expand Down
10 changes: 10 additions & 0 deletions changelog.d/ITL-1194.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
### Added

- `MergeJoin` handles referential tags (ITL-1194), following `Join`: a tag referential in
both inputs is one scalar referential column, not a two-element list, and its source
info is one merged token, the `join_v1` fold of the two inputs' tokens (sorted), as
`Join` gives. A tag referential in one input stays referential with its own token when
the other input has it as a plain tag or not at all; a data-only column of that name in
the other input is refused. A shared key that is `T` in one input and `T | None` in the
other is `T` in both roles (ITL-829). Every other colliding data column is merged into a
sorted list, as before; inputs without referential tags merge exactly as before.
31 changes: 25 additions & 6 deletions docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ does not make it compute again.
| `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. |
| Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the reference. |
| `Batch` | Keeps every tag, list-valued, as for plain tags: `x` becomes the same list in both roles and stays referential. Its source info is one token per member, like any data column: nothing is merged. |
| `MergeJoin` | Not supported with referential tags yet: it raises. |
| `MergeJoin` | Follows `Join`. Referential in both inputs: **one** scalar referential column `x`, not a two-element list (the join equated the values), with one merged token as its source info, the same fold as `Join`'s (below). Referential in one input, and a plain tag or no column of that name in the other: `x` stays referential with its own source info. A data-only column `x` in the other input is refused. A key that is `T` in one input and `T \| None` in the other is `T` in both roles. Every other colliding data column is merged into a sorted list, as before. |

```python
units = op.sources.DictSource(
Expand All @@ -100,6 +100,25 @@ print(joined.keys())
# (('unit', 'time'), ('unit', 'area', 'time', 'amp'))
```

`MergeJoin` merges every other colliding data column into a sorted list, and keeps each
referential tag both inputs share as one column:

```python
more = op.sources.DictSource(
[{"unit": 1, "time": 10, "amp": 0.75}, {"unit": 2, "time": 20, "amp": 0.125}],
tag_columns=["unit", "time"],
source_id="more_spikes",
)
merged = op.operators.MergeJoin()(spikes, more)
print(merged.keys())
# (('unit', 'time'), ('unit', 'time', 'amp'))
# A join does not promise a row order: sort the rows before showing them.
for tag, data in sorted(merged.iter_data(), key=lambda row: row[0]["time"]):
print(tag["time"], data["time"], data["amp"])
# 10 10 [0.5, 0.75]
# 20 20 [0.125, 0.25]
```

```python
print(spikes.select_columns(["time", "amp"]).keys())
# (('time',), ('time', 'amp'))
Expand Down Expand Up @@ -142,15 +161,15 @@ print(renamed.keys())

A source-info column belongs to its data column and has its shape: a scalar value has one
token, a list one token per element. When a value is the merge of several equal elements
that are different entries (the key of a `GroupBy`, or the key a `Join` merges from
several referential inputs), no single member's token is true for it. Its source info is
that are different entries (the key of a `GroupBy`, or the key a `Join` or `MergeJoin`
merges from several referential inputs), no single member's token is true for it. Its source info is
then one **merged token**, the fold of the members' tokens by the operation that merged
them (the SHA-256 of their canonical JSON list, after the operation's label), as the
system tags of a group are folded:

- `GroupBy`: `group_v1:<hex>` over the members' tokens in member order (the order of the
group's data lists, which does not depend on row order);
- `Join`: `join_v1:<hex>` over the members' tokens sorted, so it does not depend on the
- `Join` and `MergeJoin`: `join_v1:<hex>` over the members' tokens sorted, so it does not depend on the
order of the inputs.

It is a merged token for a group of one member too, and a merged token can itself be a
Expand Down Expand Up @@ -257,8 +276,8 @@ the file is checked against the stored tables when a node reads them.

## Not supported yet

These raise "not supported with referential tags yet" when a referential tag reaches them:
`MergeJoin` and `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`).
`MapColumns` (`map_columns`) raises "not supported with referential tags yet" when a
referential tag reaches it; rename with `map_tags` and `map_data`.
`PollingSource` keeps its tags plain. Each error names its issue and the way out: build the
source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not
data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/design_issues/ITL-1023-RT1.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
**Severity:** medium
**Issue:** ITL-1023

The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
141 changes: 128 additions & 13 deletions src/orcapod/core/operators/merge_join.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,19 @@
require_shared_arrow_type,
require_tag_data_roles_agree,
)
from orcapod.core.operators.join import (
_MERGED_SOURCE_PREFIX, # pyright: ignore[reportPrivateUsage] # one definition, shared with Join
Join,
)
from orcapod.core.operators.key_nullability import (
key_nullable,
reconcile_key_nullability,
)
from orcapod.core.referential_tags import (
check_namespaces,
reference_names,
stream_references,
)
from orcapod.core.streams import ArrowTableStream
from orcapod.errors import InputValidationError
from orcapod.protocols.core_protocols import StreamProtocol
Expand Down Expand Up @@ -157,10 +166,21 @@ class MergeJoin(BinaryOperator):

MergeJoin is commutative: MergeJoin(A, B) produces the same result as
MergeJoin(B, A), achieved by sorting merged values and system tag values.

Referential tags (ITL-1194) follow ``Join``, with ``Join``'s helpers. A tag
``x`` referential in both inputs is a join key, so its two values are equal: the
output has ONE scalar referential column ``x``, not a two-element list, and its
source info is one token, the ``join_v1`` fold of the two inputs' tokens
(sorted). A tag referential in one input stays referential, with its own token,
whether the other input has it as a plain tag or not at all. A referential tag
of one input named like a data-only column of the other is refused (the tag and
data role check). A shared key that is ``T`` in one input and ``T | None`` in
the other is ``T`` in both roles (ITL-829). Every other colliding data column is
merged into a sorted list, as for plain inputs.
"""

# Referential tags (ITL-1023): not handled yet.
REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = "ITL-1018"
# Referential tags (ITL-1023) are handled (ITL-1194).
REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = None

@property
def kernel_id(self) -> tuple[str, ...]:
Expand All @@ -172,13 +192,18 @@ def is_commutative(self) -> bool:
def validate_binary_inputs(
self, left_stream: StreamProtocol, right_stream: StreamProtocol
) -> None:
_, left_data_schema = left_stream.output_schema()
_, right_data_schema = right_stream.output_schema()
self._require_tag_data_roles_agree(left_stream, right_stream)
left_data_schema, right_data_schema, merged = self._input_data_schemas(
left_stream, right_stream
)

# Colliding data columns must have identical types since they are
# merged into list[T] — both sides must contribute the same T.
colliding_keys = set(left_data_schema.keys()) & set(right_data_schema.keys())
for key in colliding_keys:
# merged into list[T] — both sides must contribute the same T. A merged
# referential column is one value, checked in ``binary_output_schema``.
colliding_keys = (
set(left_data_schema.keys()) & set(right_data_schema.keys())
) - merged
for key in sorted(colliding_keys):
left_type = left_data_schema[key]
right_type = right_data_schema[key]
if left_type != right_type:
Expand Down Expand Up @@ -220,18 +245,31 @@ def binary_output_schema(
(first_stream, _), (second_stream, _) = self._canonical_order(
left_stream, right_stream
)
left_tag_schema, left_data_schema = first_stream.output_schema()
right_tag_schema, right_data_schema = second_stream.output_schema()
left_tag_schema, _ = first_stream.output_schema()
right_tag_schema, _ = second_stream.output_schema()
left_data_schema, right_data_schema, merged = self._input_data_schemas(
first_stream, second_stream
)

# Tag schema: union of both tag schemas. A shared tag key that differs in
# nullability only is non-nullable (ITL-829): a null key matches nothing.
tag_schema = schema_utils.union_schemas(
*reconcile_key_nullability(left_tag_schema, right_tag_schema)
)

# A referential tag of both inputs is one column, one value (ITL-1194).
for name in sorted(merged):
if left_data_schema[name] != right_data_schema[name]:
raise InputValidationError(
f"MergeJoin: referential tag {name!r} has type "
f"{left_data_schema[name]} in one input and "
f"{right_data_schema[name]} in the other"
)

# Data schema: colliding columns become list[T], non-colliding stay scalar
colliding_schema = schema_utils.intersection_schemas(
left_data_schema, right_data_schema
{k: v for k, v in left_data_schema.items() if k not in merged},
{k: v for k, v in right_data_schema.items() if k not in merged},
)

merged_data_schema = {}
Expand All @@ -246,6 +284,21 @@ def binary_output_schema(
else:
merged_data_schema[key] = right_data_schema[key]

# A referential tag's column is laid out once, among the tags, and a
# stream takes its data order from the layout: the referential columns
# lead the data, in tag order. Without referential tags the order is as
# before.
references = set(reference_names(tag_schema, merged_data_schema))
check_namespaces(tag_schema, merged_data_schema, references)
merged_data_schema = {
**{
name: merged_data_schema[name]
for name in tag_schema
if name in references
},
**{k: v for k, v in merged_data_schema.items() if k not in references},
}

# Add system tag columns if requested
if columns_config.system_tags:
system_tag_schema = self._predict_system_tag_schema(
Expand All @@ -255,6 +308,44 @@ def binary_output_schema(

return tag_schema, Schema(merged_data_schema)

@staticmethod
def _input_data_schemas(
first_stream: StreamProtocol, second_stream: StreamProtocol
) -> tuple[Schema, Schema, set[str]]:
"""Return the inputs' data schemas as the join sees them, and the merged names.

As in ``Join``: a referential key the join narrows to non-nullable (ITL-829)
has that type in its data role too (``Join._narrow_references``), and the
merged columns are the referential tags of both inputs
(``Join._merged_references``).

Args:
first_stream: One input.
second_stream: The other input.

Returns:
``(first data schema, second data schema, merged names)``.
"""
first_tags, first_data = first_stream.output_schema()
second_tags, second_data = second_stream.output_schema()
first_refs = set(reference_names(first_tags, first_data))
second_refs = set(reference_names(second_tags, second_data))
merged = Join._merged_references( # pyright: ignore[reportPrivateUsage] # Join's one rule
first_tags, first_refs, second_tags, second_refs
)
first_reconciled, second_reconciled = reconcile_key_nullability(
first_tags, second_tags
)
return (
Join._narrow_references( # pyright: ignore[reportPrivateUsage] # Join's one rule
first_data, first_tags, first_reconciled, first_refs
),
Join._narrow_references( # pyright: ignore[reportPrivateUsage] # Join's one rule
second_data, second_tags, second_reconciled, second_refs
),
merged,
)

@staticmethod
def _require_tag_data_roles_agree(
left_stream: StreamProtocol, right_stream: StreamProtocol
Expand Down Expand Up @@ -355,8 +446,27 @@ def binary_static_process(
right_tag_keys, right_data_keys = second_stream.keys()
shared_tag_keys = set(left_tag_keys) & set(right_tag_keys)

# Find colliding data columns
colliding_keys = set(left_data_keys) & set(right_data_keys)
# A referential tag of both inputs is a join key: the join keeps one column,
# and its two source-info tokens become one (ITL-1194, as ``Join``). The
# second input's token is set aside under a temporary name until then.
merged = Join._merged_references( # pyright: ignore[reportPrivateUsage] # Join's one rule
left_tag_keys,
stream_references(first_stream),
right_tag_keys,
stream_references(second_stream),
)
right_source = {
f"{constants.SOURCE_PREFIX}{name}": f"{_MERGED_SOURCE_PREFIX}{name}"
for name in sorted(merged)
if f"{constants.SOURCE_PREFIX}{name}" in right_table.column_names
}
if right_source:
right_table = right_table.rename_columns(
[right_source.get(c, c) for c in right_table.column_names]
)

# Find colliding data columns (a merged referential column is not one)
colliding_keys = (set(left_data_keys) & set(right_data_keys)) - merged

# Snapshot the inputs' Arrow types BEFORE the Polars round-trip, which
# stores strings, binaries and lists in their large layouts, drops
Expand Down Expand Up @@ -434,6 +544,9 @@ def binary_static_process(
.to_arrow()
)
joined = joined.drop(COMMON_JOIN_KEY)
for left_name, right_name in right_source.items():
joined = Join._merge_source_columns(joined, left_name, right_name) # pyright: ignore[reportPrivateUsage] # Join's one rule
joined = Join._fold_source_column(joined, left_name) # pyright: ignore[reportPrivateUsage] # Join's one rule

# Use the left stream's type converter — not the default context — so that a
# MergeJoin over streams built with a non-default DataContext reconstructs the
Expand Down Expand Up @@ -534,7 +647,8 @@ def binary_static_process(
# so both must follow it. No hash or record ID depends on column order.
tag_schema, data_schema = self.binary_output_schema(left_stream, right_stream)
tag_cols = list(tag_schema.keys())
data_cols = list(data_schema.keys())
# A referential tag's column is laid out once, among the tags.
data_cols = [name for name in data_schema if name not in tag_schema]
listed = {*tag_cols, *data_cols}
other_cols = [c for c in joined.column_names if c not in listed]
joined = joined.select(tag_cols + data_cols + other_cols)
Expand All @@ -561,6 +675,7 @@ def binary_static_process(
return ArrowTableStream(
joined,
tag_columns=tuple(tag_cols),
reference_tags=reference_names(tag_schema, data_schema),
)

def identity_structure(self) -> Any:
Expand Down
49 changes: 48 additions & 1 deletion tests/test_core/operators/test_async_sync_equivalence.py
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,9 @@ def source_shared_data() -> ArrowTableSource:
)


def merge_source(source_id: str, offset: int) -> ArrowTableSource:
def merge_source(
source_id: str, offset: int, referential: bool = False
) -> ArrowTableSource:
"""A ``MergeJoin`` input; two of them collide on every data column but one."""
only = f"only_{source_id}"
return _source(
Expand All @@ -224,6 +226,7 @@ def merge_source(source_id: str, offset: int) -> ArrowTableSource:
tags=("k",),
source_id=source_id,
nullable=("xs",),
referential=referential,
)


Expand Down Expand Up @@ -500,6 +503,50 @@ class Case:
),
),
),
# MergeJoin on referential tags (ITL-1194): one merged scalar column for a tag
# referential in both inputs; the other colliding columns merged into lists.
Case(
"ref_merge_join_2",
MergeJoin,
lambda: (
merge_source("m", 0, referential=True),
merge_source("n", 10, referential=True),
),
),
Case(
"ref_merge_join_plain_left",
MergeJoin,
lambda: (merge_source("m", 0), merge_source("n", 10, referential=True)),
),
Case(
"ref_merge_join_plain_right",
MergeJoin,
lambda: (merge_source("m", 0, referential=True), merge_source("n", 10)),
),
Case(
# ``tc`` is referential on one input and absent from the other.
"ref_merge_join_absent",
MergeJoin,
lambda: (source_c(referential=True), merge_source("n", 10, referential=True)),
),
Case(
# A referential key T on one side and T | None on the other (ITL-829).
"ref_merge_join_key_nullability",
MergeJoin,
lambda: (
merge_source("m", 0, referential=True),
_source(
{
"k": pa.array([1, None, 3], pa.int16()),
"x8": pa.array([7, 8, 9], pa.int8()),
},
("k",),
"n",
nullable=("k",),
referential=True,
),
),
),
Case(
"ref_semijoin",
SemiJoin,
Expand Down
Loading
Loading