diff --git a/changelog.d/ITL-1023.md b/changelog.d/ITL-1023.md index de5a8271b..7a5381b41 100644 --- a/changelog.d/ITL-1023.md +++ b/changelog.d/ITL-1023.md @@ -115,7 +115,7 @@ ### Known limitations -- `MergeJoin` and `MapColumns` raise "not supported with referential tags yet" +- `MapColumns` raises "not supported with referential tags yet" (operator-node storage and `DerivedSource` over a referential stream are supported, see ITL-1019 above). `PollingSource` keeps its tags plain. The identities of referential streams may still diff --git a/changelog.d/ITL-1194.md b/changelog.d/ITL-1194.md new file mode 100644 index 000000000..76e53c826 --- /dev/null +++ b/changelog.d/ITL-1194.md @@ -0,0 +1,10 @@ +### Added + +- `MergeJoin` handles referential tags (ITL-1194), following `Join`: a tag referential in + both inputs is one scalar referential column, not a two-element list, and its source + info is one merged token, the `join_v1` fold of the two inputs' tokens (sorted), as + `Join` gives. A tag referential in one input stays referential with its own token when + the other input has it as a plain tag or not at all; a data-only column of that name in + the other input is refused. A shared key that is `T` in one input and `T | None` in the + other is `T` in both roles (ITL-829). Every other colliding data column is merged into a + sorted list, as before; inputs without referential tags merge exactly as before. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index a8687b6be..2d4dadb1d 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -87,7 +87,7 @@ does not make it compute again. | `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | | Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the reference. | | `Batch` | Keeps every tag, list-valued, as for plain tags: `x` becomes the same list in both roles and stays referential. Its source info is one token per member, like any data column: nothing is merged. | -| `MergeJoin` | Not supported with referential tags yet: it raises. | +| `MergeJoin` | Follows `Join`. Referential in both inputs: **one** scalar referential column `x`, not a two-element list (the join equated the values), with one merged token as its source info, the same fold as `Join`'s (below). Referential in one input, and a plain tag or no column of that name in the other: `x` stays referential with its own source info. A data-only column `x` in the other input is refused. A key that is `T` in one input and `T \| None` in the other is `T` in both roles. Every other colliding data column is merged into a sorted list, as before. | ```python units = op.sources.DictSource( @@ -100,6 +100,25 @@ print(joined.keys()) # (('unit', 'time'), ('unit', 'area', 'time', 'amp')) ``` +`MergeJoin` merges every other colliding data column into a sorted list, and keeps each +referential tag both inputs share as one column: + +```python +more = op.sources.DictSource( + [{"unit": 1, "time": 10, "amp": 0.75}, {"unit": 2, "time": 20, "amp": 0.125}], + tag_columns=["unit", "time"], + source_id="more_spikes", +) +merged = op.operators.MergeJoin()(spikes, more) +print(merged.keys()) +# (('unit', 'time'), ('unit', 'time', 'amp')) +# A join does not promise a row order: sort the rows before showing them. +for tag, data in sorted(merged.iter_data(), key=lambda row: row[0]["time"]): + print(tag["time"], data["time"], data["amp"]) +# 10 10 [0.5, 0.75] +# 20 20 [0.125, 0.25] +``` + ```python print(spikes.select_columns(["time", "amp"]).keys()) # (('time',), ('time', 'amp')) @@ -142,15 +161,15 @@ print(renamed.keys()) A source-info column belongs to its data column and has its shape: a scalar value has one token, a list one token per element. When a value is the merge of several equal elements -that are different entries (the key of a `GroupBy`, or the key a `Join` merges from -several referential inputs), no single member's token is true for it. Its source info is +that are different entries (the key of a `GroupBy`, or the key a `Join` or `MergeJoin` +merges from several referential inputs), no single member's token is true for it. Its source info is then one **merged token**, the fold of the members' tokens by the operation that merged them (the SHA-256 of their canonical JSON list, after the operation's label), as the system tags of a group are folded: - `GroupBy`: `group_v1:` over the members' tokens in member order (the order of the group's data lists, which does not depend on row order); -- `Join`: `join_v1:` over the members' tokens sorted, so it does not depend on the +- `Join` and `MergeJoin`: `join_v1:` over the members' tokens sorted, so it does not depend on the order of the inputs. It is a merged token for a group of one member too, and a merged token can itself be a @@ -257,8 +276,8 @@ the file is checked against the stored tables when a node reads them. ## Not supported yet -These raise "not supported with referential tags yet" when a referential tag reaches them: -`MergeJoin` and `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`). +`MapColumns` (`map_columns`) raises "not supported with referential tags yet" when a +referential tag reaches it; rename with `map_tags` and `map_data`. `PollingSource` keeps its tags plain. Each error names its issue and the way out: build the source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer diff --git a/docs/dev/design_issues/ITL-1023-RT1.md b/docs/dev/design_issues/ITL-1023-RT1.md index 371aa9b7d..4b8f026ab 100644 --- a/docs/dev/design_issues/ITL-1023-RT1.md +++ b/docs/dev/design_issues/ITL-1023-RT1.md @@ -5,4 +5,4 @@ **Severity:** medium **Issue:** ITL-1023 -The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. +The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. diff --git a/src/orcapod/core/operators/merge_join.py b/src/orcapod/core/operators/merge_join.py index 893203b0f..891e3a095 100644 --- a/src/orcapod/core/operators/merge_join.py +++ b/src/orcapod/core/operators/merge_join.py @@ -6,10 +6,19 @@ require_shared_arrow_type, require_tag_data_roles_agree, ) +from orcapod.core.operators.join import ( + _MERGED_SOURCE_PREFIX, # pyright: ignore[reportPrivateUsage] # one definition, shared with Join + Join, +) from orcapod.core.operators.key_nullability import ( key_nullable, reconcile_key_nullability, ) +from orcapod.core.referential_tags import ( + check_namespaces, + reference_names, + stream_references, +) from orcapod.core.streams import ArrowTableStream from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import StreamProtocol @@ -157,10 +166,21 @@ class MergeJoin(BinaryOperator): MergeJoin is commutative: MergeJoin(A, B) produces the same result as MergeJoin(B, A), achieved by sorting merged values and system tag values. + + Referential tags (ITL-1194) follow ``Join``, with ``Join``'s helpers. A tag + ``x`` referential in both inputs is a join key, so its two values are equal: the + output has ONE scalar referential column ``x``, not a two-element list, and its + source info is one token, the ``join_v1`` fold of the two inputs' tokens + (sorted). A tag referential in one input stays referential, with its own token, + whether the other input has it as a plain tag or not at all. A referential tag + of one input named like a data-only column of the other is refused (the tag and + data role check). A shared key that is ``T`` in one input and ``T | None`` in + the other is ``T`` in both roles (ITL-829). Every other colliding data column is + merged into a sorted list, as for plain inputs. """ - # Referential tags (ITL-1023): not handled yet. - REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = "ITL-1018" + # Referential tags (ITL-1023) are handled (ITL-1194). + REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = None @property def kernel_id(self) -> tuple[str, ...]: @@ -172,13 +192,18 @@ def is_commutative(self) -> bool: def validate_binary_inputs( self, left_stream: StreamProtocol, right_stream: StreamProtocol ) -> None: - _, left_data_schema = left_stream.output_schema() - _, right_data_schema = right_stream.output_schema() + self._require_tag_data_roles_agree(left_stream, right_stream) + left_data_schema, right_data_schema, merged = self._input_data_schemas( + left_stream, right_stream + ) # Colliding data columns must have identical types since they are - # merged into list[T] — both sides must contribute the same T. - colliding_keys = set(left_data_schema.keys()) & set(right_data_schema.keys()) - for key in colliding_keys: + # merged into list[T] — both sides must contribute the same T. A merged + # referential column is one value, checked in ``binary_output_schema``. + colliding_keys = ( + set(left_data_schema.keys()) & set(right_data_schema.keys()) + ) - merged + for key in sorted(colliding_keys): left_type = left_data_schema[key] right_type = right_data_schema[key] if left_type != right_type: @@ -220,8 +245,11 @@ def binary_output_schema( (first_stream, _), (second_stream, _) = self._canonical_order( left_stream, right_stream ) - left_tag_schema, left_data_schema = first_stream.output_schema() - right_tag_schema, right_data_schema = second_stream.output_schema() + left_tag_schema, _ = first_stream.output_schema() + right_tag_schema, _ = second_stream.output_schema() + left_data_schema, right_data_schema, merged = self._input_data_schemas( + first_stream, second_stream + ) # Tag schema: union of both tag schemas. A shared tag key that differs in # nullability only is non-nullable (ITL-829): a null key matches nothing. @@ -229,9 +257,19 @@ def binary_output_schema( *reconcile_key_nullability(left_tag_schema, right_tag_schema) ) + # A referential tag of both inputs is one column, one value (ITL-1194). + for name in sorted(merged): + if left_data_schema[name] != right_data_schema[name]: + raise InputValidationError( + f"MergeJoin: referential tag {name!r} has type " + f"{left_data_schema[name]} in one input and " + f"{right_data_schema[name]} in the other" + ) + # Data schema: colliding columns become list[T], non-colliding stay scalar colliding_schema = schema_utils.intersection_schemas( - left_data_schema, right_data_schema + {k: v for k, v in left_data_schema.items() if k not in merged}, + {k: v for k, v in right_data_schema.items() if k not in merged}, ) merged_data_schema = {} @@ -246,6 +284,21 @@ def binary_output_schema( else: merged_data_schema[key] = right_data_schema[key] + # A referential tag's column is laid out once, among the tags, and a + # stream takes its data order from the layout: the referential columns + # lead the data, in tag order. Without referential tags the order is as + # before. + references = set(reference_names(tag_schema, merged_data_schema)) + check_namespaces(tag_schema, merged_data_schema, references) + merged_data_schema = { + **{ + name: merged_data_schema[name] + for name in tag_schema + if name in references + }, + **{k: v for k, v in merged_data_schema.items() if k not in references}, + } + # Add system tag columns if requested if columns_config.system_tags: system_tag_schema = self._predict_system_tag_schema( @@ -255,6 +308,44 @@ def binary_output_schema( return tag_schema, Schema(merged_data_schema) + @staticmethod + def _input_data_schemas( + first_stream: StreamProtocol, second_stream: StreamProtocol + ) -> tuple[Schema, Schema, set[str]]: + """Return the inputs' data schemas as the join sees them, and the merged names. + + As in ``Join``: a referential key the join narrows to non-nullable (ITL-829) + has that type in its data role too (``Join._narrow_references``), and the + merged columns are the referential tags of both inputs + (``Join._merged_references``). + + Args: + first_stream: One input. + second_stream: The other input. + + Returns: + ``(first data schema, second data schema, merged names)``. + """ + first_tags, first_data = first_stream.output_schema() + second_tags, second_data = second_stream.output_schema() + first_refs = set(reference_names(first_tags, first_data)) + second_refs = set(reference_names(second_tags, second_data)) + merged = Join._merged_references( # pyright: ignore[reportPrivateUsage] # Join's one rule + first_tags, first_refs, second_tags, second_refs + ) + first_reconciled, second_reconciled = reconcile_key_nullability( + first_tags, second_tags + ) + return ( + Join._narrow_references( # pyright: ignore[reportPrivateUsage] # Join's one rule + first_data, first_tags, first_reconciled, first_refs + ), + Join._narrow_references( # pyright: ignore[reportPrivateUsage] # Join's one rule + second_data, second_tags, second_reconciled, second_refs + ), + merged, + ) + @staticmethod def _require_tag_data_roles_agree( left_stream: StreamProtocol, right_stream: StreamProtocol @@ -355,8 +446,27 @@ def binary_static_process( right_tag_keys, right_data_keys = second_stream.keys() shared_tag_keys = set(left_tag_keys) & set(right_tag_keys) - # Find colliding data columns - colliding_keys = set(left_data_keys) & set(right_data_keys) + # A referential tag of both inputs is a join key: the join keeps one column, + # and its two source-info tokens become one (ITL-1194, as ``Join``). The + # second input's token is set aside under a temporary name until then. + merged = Join._merged_references( # pyright: ignore[reportPrivateUsage] # Join's one rule + left_tag_keys, + stream_references(first_stream), + right_tag_keys, + stream_references(second_stream), + ) + right_source = { + f"{constants.SOURCE_PREFIX}{name}": f"{_MERGED_SOURCE_PREFIX}{name}" + for name in sorted(merged) + if f"{constants.SOURCE_PREFIX}{name}" in right_table.column_names + } + if right_source: + right_table = right_table.rename_columns( + [right_source.get(c, c) for c in right_table.column_names] + ) + + # Find colliding data columns (a merged referential column is not one) + colliding_keys = (set(left_data_keys) & set(right_data_keys)) - merged # Snapshot the inputs' Arrow types BEFORE the Polars round-trip, which # stores strings, binaries and lists in their large layouts, drops @@ -434,6 +544,9 @@ def binary_static_process( .to_arrow() ) joined = joined.drop(COMMON_JOIN_KEY) + for left_name, right_name in right_source.items(): + joined = Join._merge_source_columns(joined, left_name, right_name) # pyright: ignore[reportPrivateUsage] # Join's one rule + joined = Join._fold_source_column(joined, left_name) # pyright: ignore[reportPrivateUsage] # Join's one rule # Use the left stream's type converter — not the default context — so that a # MergeJoin over streams built with a non-default DataContext reconstructs the @@ -534,7 +647,8 @@ def binary_static_process( # so both must follow it. No hash or record ID depends on column order. tag_schema, data_schema = self.binary_output_schema(left_stream, right_stream) tag_cols = list(tag_schema.keys()) - data_cols = list(data_schema.keys()) + # A referential tag's column is laid out once, among the tags. + data_cols = [name for name in data_schema if name not in tag_schema] listed = {*tag_cols, *data_cols} other_cols = [c for c in joined.column_names if c not in listed] joined = joined.select(tag_cols + data_cols + other_cols) @@ -561,6 +675,7 @@ def binary_static_process( return ArrowTableStream( joined, tag_columns=tuple(tag_cols), + reference_tags=reference_names(tag_schema, data_schema), ) def identity_structure(self) -> Any: diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 60fbbbb75..a015bd90f 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -209,7 +209,9 @@ def source_shared_data() -> ArrowTableSource: ) -def merge_source(source_id: str, offset: int) -> ArrowTableSource: +def merge_source( + source_id: str, offset: int, referential: bool = False +) -> ArrowTableSource: """A ``MergeJoin`` input; two of them collide on every data column but one.""" only = f"only_{source_id}" return _source( @@ -224,6 +226,7 @@ def merge_source(source_id: str, offset: int) -> ArrowTableSource: tags=("k",), source_id=source_id, nullable=("xs",), + referential=referential, ) @@ -500,6 +503,50 @@ class Case: ), ), ), + # MergeJoin on referential tags (ITL-1194): one merged scalar column for a tag + # referential in both inputs; the other colliding columns merged into lists. + Case( + "ref_merge_join_2", + MergeJoin, + lambda: ( + merge_source("m", 0, referential=True), + merge_source("n", 10, referential=True), + ), + ), + Case( + "ref_merge_join_plain_left", + MergeJoin, + lambda: (merge_source("m", 0), merge_source("n", 10, referential=True)), + ), + Case( + "ref_merge_join_plain_right", + MergeJoin, + lambda: (merge_source("m", 0, referential=True), merge_source("n", 10)), + ), + Case( + # ``tc`` is referential on one input and absent from the other. + "ref_merge_join_absent", + MergeJoin, + lambda: (source_c(referential=True), merge_source("n", 10, referential=True)), + ), + Case( + # A referential key T on one side and T | None on the other (ITL-829). + "ref_merge_join_key_nullability", + MergeJoin, + lambda: ( + merge_source("m", 0, referential=True), + _source( + { + "k": pa.array([1, None, 3], pa.int16()), + "x8": pa.array([7, 8, 9], pa.int8()), + }, + ("k",), + "n", + nullable=("k",), + referential=True, + ), + ), + ), Case( "ref_semijoin", SemiJoin, diff --git a/tests/test_core/referential_tags/test_error_messages.py b/tests/test_core/referential_tags/test_error_messages.py index 60bc81566..243216f8b 100644 --- a/tests/test_core/referential_tags/test_error_messages.py +++ b/tests/test_core/referential_tags/test_error_messages.py @@ -17,7 +17,7 @@ from orcapod.core.cached_function_pod import CachedFunctionPod from orcapod.core.function_pod import FunctionPod from orcapod.core.nodes.source_node import SourceNode -from orcapod.core.operators import DropNulls, MergeJoin +from orcapod.core.operators import DropNulls, MapColumns from orcapod.databases import InMemoryArrowDatabase from orcapod.errors import SourceSpecMismatchError from orcapod.types import Schema @@ -253,10 +253,10 @@ def test_the_exact_form_on_plain_tags_says_nothing_about_referential_tags() -> N def test_not_supported_yet_names_tag_only_columns_true_and_the_issue() -> None: - # MergeJoin is the operator not supported yet (Batch is, since ITL-1017). + # MapColumns is an operator not supported yet (MergeJoin is, since ITL-1194). source = _source(["k", "x"]) with pytest.raises(NotImplementedError) as excinfo: - _ = MergeJoin()(source, source) + _ = MapColumns({"x": "z"})(source) message = str(excinfo.value) - assert "ITL-1018" in message + assert "ITL-1022" in message assert "tag_only_columns=True" in message diff --git a/tests/test_core/referential_tags/test_merge_join.py b/tests/test_core/referential_tags/test_merge_join.py new file mode 100644 index 000000000..4605a9815 --- /dev/null +++ b/tests/test_core/referential_tags/test_merge_join.py @@ -0,0 +1,251 @@ +"""``MergeJoin`` on referential tags (ITL-1194). + +``MergeJoin`` follows ``Join`` (developer-approved design, November 17, 2026): + +1. A tag ``x`` referential in both inputs is a join key, so its two values are equal: + the output has ONE scalar referential column ``x``, and its source info is one + token, the ``join_v1`` fold of the two inputs' tokens (sorted). +2. ``x`` referential in one input and a plain tag in the other stays referential, + with its own token. +3. ``x`` referential in one input and absent from the other stays referential, with + its own token. +4. ``x`` referential in one input and a data-only column in the other is refused (the + tag and data role check). + +Every other colliding data column is merged into a sorted list, as for plain inputs. +A shared key that is ``T`` in one input and ``T | None`` in the other is ``T`` in +both roles (ITL-829). Every operator is driven through its public call and read +through the stream's public methods; a schema prediction is checked against +``static_process``. +""" + +from __future__ import annotations + +import re +from collections.abc import Sequence +from typing import Any + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.core.operators import Join, MapData, MergeJoin +from orcapod.core.operators.join import JOIN_FOLD_LABEL +from orcapod.core.referential_tags import stream_references +from orcapod.core.sources import ArrowTableSource +from orcapod.errors import InputValidationError +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.system_constants import constants +from orcapod.utils import arrow_utils + +JOIN_TOKEN = re.compile(rf"^{JOIN_FOLD_LABEL}:[0-9a-f]{{64}}$") + + +def join_fold(tokens: Sequence[str | None]) -> str: + """The token of a key referential in both inputs: the tokens sorted, then folded.""" + ordered = sorted(tokens, key=lambda token: (token is None, token or "")) + return arrow_utils.fold_system_tag_values(ordered, label=JOIN_FOLD_LABEL) + + +def _source( + sid: str, values: Sequence[float], tags: Sequence[str] = ("k", "x"), **kw: Any +) -> op.sources.DictSource: + """Tags ``k`` and ``x`` (referential unless ``tag_only_columns=True``); data ``v``.""" + rows = [ + {"k": k, "x": 10 * k, "v": v} for k, v in zip(range(1, len(values) + 1), values) + ] + return op.sources.DictSource(rows, tag_columns=list(tags), source_id=sid, **kw) + + +def _without_x(sid: str, values: Sequence[float]) -> op.sources.DictSource: + """Plain tag ``k``, data ``v``: no column ``x`` at all.""" + rows = [{"k": k, "v": v} for k, v in zip(range(1, len(values) + 1), values)] + return op.sources.DictSource( + rows, tag_columns=["k"], source_id=sid, tag_only_columns=True + ) + + +def _a() -> op.sources.DictSource: + return _source("A", [1.0, 2.0]) + + +def _b() -> op.sources.DictSource: + return _source("B", [3.0, 0.5]) + + +def _by_k(stream: StreamProtocol) -> dict[int, dict[str, Any]]: + table = stream.as_table(all_info=True) + return {row["k"]: row for row in table.to_pylist()} + + +# ── case 1: referential in both inputs ────────────────────────────────────── + + +def test_a_tag_referential_in_both_is_one_scalar_referential_column() -> None: + merged = MergeJoin()(_a(), _b()) + assert merged.keys() == (("k", "x"), ("k", "x", "v")) + assert stream_references(merged) == ("k", "x") + rows = _by_k(merged) + # The key is one value, not a two-element list; the other collision is a list. + assert {k: row["x"] for k, row in rows.items()} == {1: 10, 2: 20} + assert {k: row["v"] for k, row in rows.items()} == {1: [1.0, 3.0], 2: [0.5, 2.0]} + for tag, data in merged.iter_data(): + assert tag["x"] == data["x"] + + +def test_its_source_info_is_one_join_token_of_both_tokens() -> None: + rows = _by_k(MergeJoin()(_a(), _b())) + source_x = f"{constants.SOURCE_PREFIX}x" + for k, row in rows.items(): + assert JOIN_TOKEN.match(row[source_x]) + assert row[source_x] == join_fold([f"A::row_{k - 1}::x", f"B::row_{k - 1}::x"]) + # A merged list keeps one token per member, in the order of its values. + assert rows[2][f"{constants.SOURCE_PREFIX}v"] == ["B::row_1::v", "A::row_1::v"] + + +def test_the_token_is_the_one_join_gives() -> None: + """One fold, ``Join``'s: the same inputs joined by ``Join`` give the same token.""" + source_x = f"{constants.SOURCE_PREFIX}x" + merged = _by_k(MergeJoin()(_a(), _b())) + joined = _by_k(Join()(_a(), MapData({"v": "w"})(_b()))) + assert {k: row[source_x] for k, row in merged.items()} == { + k: row[source_x] for k, row in joined.items() + } + + +def test_the_inputs_order_does_not_matter() -> None: + forward = MergeJoin()(_a(), _b()).as_table(all_info=True) + backward = MergeJoin()(_b(), _a()).as_table(all_info=True) + assert forward.equals(backward) + + +# ── cases 2 and 3: referential in one input only ──────────────────────────── + + +@pytest.mark.parametrize( + "other", + [ + lambda: _source("B", [3.0, 0.5], tag_only_columns=True), + lambda: _without_x("B", [3.0, 0.5]), + ], + ids=["plain_tag", "absent"], +) +def test_a_tag_referential_in_one_input_keeps_its_own_token(other: Any) -> None: + for left, right in ((_a(), other()), (other(), _a())): + merged = MergeJoin()(left, right) + assert merged.keys() == (("k", "x"), ("k", "x", "v")) + assert stream_references(merged) == ("k", "x") + rows = _by_k(merged) + for k, row in rows.items(): + assert row["x"] == 10 * k + assert row[f"{constants.SOURCE_PREFIX}x"] == f"A::row_{k - 1}::x" + + +# ── case 4: referential against data-only ─────────────────────────────────── + + +def test_a_referential_tag_against_a_data_only_column_is_refused() -> None: + data_only = _source("D", [3.0, 0.5], tags=("k",), tag_only_columns=True) + for left, right in ((_a(), data_only), (data_only, _a())): + with pytest.raises(InputValidationError, match="column 'x' is a tag"): + _ = MergeJoin()(left, right) + + +def test_a_referential_tag_of_two_types_is_refused() -> None: + other = op.sources.DictSource( + [{"k": 1, "x": "ten", "v": 3.0}], tag_columns=["k", "x"], source_id="S" + ) + # Refused where the tag schemas meet, as for a plain tag of two types. + with pytest.raises(InputValidationError, match="not compatible"): + _ = MergeJoin()(_a(), other) + + +# ── nullability (ITL-829) ─────────────────────────────────────────────────── + + +def _arrow(sid: str, x_nullable: bool, xs: list[int | None]) -> ArrowTableSource: + schema = pa.schema( + [ + pa.field("x", pa.int64(), nullable=x_nullable), + pa.field("v", pa.float64(), nullable=False), + ] + ) + table = pa.table({"x": xs, "v": [float(i) for i in range(len(xs))]}, schema=schema) + return ArrowTableSource(table, tag_columns=["x"], source_id=sid) + + +@pytest.mark.parametrize( + ("left_nullable", "right_nullable", "expected"), + [(False, True, int), (True, False, int), (True, True, int | None)], +) +def test_a_merged_key_takes_the_join_nullability_in_both_roles( + left_nullable: bool, right_nullable: bool, expected: Any +) -> None: + left = _arrow("L", left_nullable, [1, 2]) + right = _arrow("R", right_nullable, [1, 2] if not right_nullable else [1, None]) + merged = MergeJoin()(left, right) + tag_schema, data_schema = merged.output_schema() + assert tag_schema["x"] == expected + assert data_schema["x"] == expected + computed = MergeJoin().static_process(left, right) + assert computed.output_schema() == merged.output_schema() + assert computed.as_table().schema.field("x").nullable == (expected is not int) + + +# ── schema prediction, zero rows ──────────────────────────────────────────── + + +@pytest.mark.parametrize("all_info", [False, True]) +@pytest.mark.parametrize( + "inputs", + [ + lambda: (_a(), _b()), + lambda: (_a(), _source("B", [3.0, 0.5], tag_only_columns=True)), + lambda: (_without_x("B", [3.0, 0.5]), _a()), + ], + ids=["both", "plain_tag", "absent"], +) +def test_the_schema_is_predicted(inputs: Any, all_info: bool) -> None: + left, right = inputs() + merge_join = MergeJoin() + predicted = merge_join.output_schema(left, right, all_info=all_info) + computed = merge_join.static_process(left, right) + assert predicted == computed.output_schema(all_info=all_info) + # The order too: the referential columns lead the data, in tag order. + assert [list(schema) for schema in predicted] == [ + list(schema) for schema in computed.output_schema(all_info=all_info) + ] + assert computed.keys() == merge_join(left, right).keys() + + +def test_zero_rows_have_the_types_of_rows() -> None: + full = MergeJoin()(_a(), _b()).as_table(all_info=True) + empty = MergeJoin()(_a().polars_filter(k=99), _b()).as_table(all_info=True) + assert empty.num_rows == 0 + + # The filter is a step of its own, so only the system-tag NAMES differ. + def fields(schema: pa.Schema) -> list[Any]: + return [ + ("system tag", f.type, f.nullable) + if f.name.startswith(constants.SYSTEM_TAG_PREFIX) + else f + for f in schema + ] + + assert fields(empty.schema) == fields(full.schema) + + +# ── plain inputs unchanged ────────────────────────────────────────────────── + + +def test_plain_inputs_merge_as_before() -> None: + merged = MergeJoin()( + _source("A", [1.0, 2.0], tag_only_columns=True), + _source("B", [3.0, 0.5], tag_only_columns=True), + ) + assert merged.keys() == (("k", "x"), ("v",)) + assert stream_references(merged) == () + assert {k: row["v"] for k, row in _by_k(merged).items()} == { + 1: [1.0, 3.0], + 2: [0.5, 2.0], + } diff --git a/tests/test_core/referential_tags/test_operators.py b/tests/test_core/referential_tags/test_operators.py index 55d1b8d8a..514558bed 100644 --- a/tests/test_core/referential_tags/test_operators.py +++ b/tests/test_core/referential_tags/test_operators.py @@ -30,9 +30,9 @@ GroupBy, Index, Join, + MapColumns, MapData, MapTags, - MergeJoin, Pick, PolarsFilter, SelectColumns, @@ -522,8 +522,9 @@ def test_extraction_named_like_a_tag_raises() -> None: def test_uncovered_operators_raise() -> None: - with pytest.raises(NotImplementedError, match="ITL-1018"): - MergeJoin()(_a(), _b()) + # MergeJoin handles referential tags since ITL-1194 (test_merge_join.py). + with pytest.raises(NotImplementedError, match="ITL-1022"): + MapColumns({"x": "z"})(_a()) # ── renames (developer decision, November 12, 2026) ───────────────────────── @@ -629,6 +630,5 @@ def test_map_tags_on_plain_tags_is_unchanged() -> None: def test_uncovered_operators_still_work_on_plain_tags() -> None: - plain = MergeJoin()(_a(tag_only_columns=True), _b(tag_only_columns=True)) - tags, data = plain.keys() - assert tags == ("x",) and sorted(data) == ["a", "b"] + plain = MapColumns({"x": "z"})(_a(tag_only_columns=True)) + assert plain.keys() == (("z",), ("a",))