From 8b73ba71af9ab84ccb2244f301857caee3aad533 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:36:05 -0800 Subject: [PATCH 01/10] feat(operators): TagBy / stream.tag_by makes data columns referential tags (ITL-1175) TagBy gives existing data columns the tag role, as referential tags: one column in both roles, values and source info unchanged. The new tags follow the input's tags in the order given; the order is part of the identity ("TagBy", [names]). System tags, record ids and data are unchanged. Refused, naming the column: not a data column, already a tag, a reserved name, a type that cannot be a tag (referential_tags.require_taggable_type; nullable points to drop_nulls). Async uses the base barrier path; cases added to the async/sync equivalence tests. Registered for saved jobs; docs, changelog fragment and dev spec. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-1175.md | 18 + docs/concepts/referential-tags.md | 46 ++ docs/dev/design_issues/ITL-1023-RT1.md | 2 +- docs/dev/specs/ITL-1175-tag-by.md | 116 +++++ src/orcapod/core/operators/__init__.py | 2 + src/orcapod/core/operators/base.py | 4 +- src/orcapod/core/operators/tag_by.py | 213 ++++++++++ src/orcapod/core/referential_tags.py | 54 +++ src/orcapod/core/streams/base.py | 44 +- src/orcapod/pipeline/serialization.py | 2 + .../protocols/core_protocols/streams.py | 12 + .../operators/test_async_sync_equivalence.py | 6 + .../test_core/referential_tags/test_tag_by.py | 400 ++++++++++++++++++ 13 files changed, 915 insertions(+), 4 deletions(-) create mode 100644 changelog.d/ITL-1175.md create mode 100644 docs/dev/specs/ITL-1175-tag-by.md create mode 100644 src/orcapod/core/operators/tag_by.py create mode 100644 tests/test_core/referential_tags/test_tag_by.py diff --git a/changelog.d/ITL-1175.md b/changelog.d/ITL-1175.md new file mode 100644 index 000000000..63d9086a0 --- /dev/null +++ b/changelog.d/ITL-1175.md @@ -0,0 +1,18 @@ +### Added + +- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as + referential tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each + named column becomes one column in both roles, with its values and source info + unchanged; the new tags follow the stream's tags in the order given, and that order is + part of the operator's identity. System tags, record ids and every data column are + unchanged. A column a function computed may become a tag and then key a join or a + group. Refused, naming the column: a name that is not a data column of the input, a + name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is + one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a + nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered + for saved jobs and pipelines. + +### Changed + +- The error for a name that is a tag in one join input and a data column in the other + also offers `tag_by` as a way out. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index 411c6efad..d1183047e 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -74,10 +74,56 @@ A function output named like a tag is an error when the pipeline is built. A fun that does not read a referenced column keeps its result identity: retagging a source does not make it compute again. +## Making a data column a tag: `tag_by` + +A source declares its tags, but a value a later step should join, group or sort by is +often computed on the way: a session label, a trial index, a phase. `tag_by` gives +existing data columns the tag role: each becomes a referential tag, one column that is +a tag and the data column it was, with its values and source info unchanged. + +```python +@op.function_pod(output_keys=["phase"]) +def phase_of(time: int) -> str: + return "early" if time < 15 else "late" + + +phased = phase_of.pod(spikes).tag_by("phase") +print(phased.keys()) +# (('unit', 'time', 'phase'), ('unit', 'time', 'phase')) + +by_phase = phased.group_by(["phase"]) +print(by_phase.keys()) +# (('phase',), ('unit', 'time', 'phase')) +``` + +- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's + tags in the order given, and that order is part of the operator (`TagBy`): the names + are a list, never a set. +- Nothing else changes: no row is added, removed or reordered, the data columns keep + their names, types and values, every data column keeps its own source-info token, + and the system tags keep their names and values. +- No data element comes from a tag value: the operator adds a tag role to a column + that is already data, and creates or moves no value. Offering a tag's value as data + stays with `DerivedSource(tags_as_data=...)`. +- A computed column may become a tag, and then key a join or a group, as `phase` does + above. + +`tag_by` refuses, naming the column: a name that is not a data column of the stream; a +name that is already a tag (plain or referential); a reserved name (the `__` prefix, a +system column, a name holding `::`); and a column whose type cannot be a tag. A tag is +one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`. +A nullable column (`T | None`) is refused: drop the rows with a null first, +`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, +your own classes and logical types (`Path`, `UUID`, arrays) are refused. + +`tag_by` is one of three moves between the roles of a column. The other two, which turn +a referential tag into plain data or into a plain tag, are the subject of ITL-1195. + ## Operators | Operator | Referential tag `x` | |---|---| +| `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. | | `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. | diff --git a/docs/dev/design_issues/ITL-1023-RT1.md b/docs/dev/design_issues/ITL-1023-RT1.md index b8750dc62..0fcb4b1ad 100644 --- a/docs/dev/design_issues/ITL-1023-RT1.md +++ b/docs/dev/design_issues/ITL-1023-RT1.md @@ -5,4 +5,4 @@ **Severity:** medium **Issue:** ITL-1023 -The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. +The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` and `TagBy` (`tag_by`, ITL-1175), which gives a data column the tag role. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md new file mode 100644 index 000000000..057426806 --- /dev/null +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -0,0 +1,116 @@ +# ITL-1175: mark existing data columns as referential tags (`tag_by`) + +Status: approved by the coordinator on 2026-11-16, with two developer decisions (the +name `tag_by` / `TagBy`; a computed column may become a tag). Stacked on the referential +tags slice (ITL-1023, PR #342). The sections below are the design as sent; "Decisions" +records what changed. + +## Decisions + +- Name: `tag_by` (developer); the class follows the operator naming: `TagBy`. +- A computed column may become a tag and key a join or a group (developer); no lineage + check. +- Type rule: the slice has no type rule for a referential tag declared at a source (a + source accepts any type), so `TagBy` uses its own rule, in one function + (`referential_tags.require_taggable_type`) that the source path could adopt: `int`, + `float`, `bool`, `str`, `bytes`, `datetime`, `date`. `Decimal`, time of day and + duration are not in it: the type converter reads them as no Python type (`Any`), and + `Any` is refused. The source path is not changed here; the inconsistency is reported + in the PR. +- Removing the data role afterwards is the subject of ITL-1195 (explicit operators); the + docs leave a placeholder for it instead of describing the implicit path. + +## Public API + +- Operator `orcapod.core.operators.TagBy(columns)`; stream method + `stream.tag_by(*columns, label=None)`. `s.tag_by("x", "y")` equals + `s.tag_by(["x", "y"])` (the convention of `select_columns`). +- The name is a choice for the developer. It lives in one place per surface: the class + name, the stream method, the serialization registry entry, the docs. Not + `reference_tags`: `ArrowTableStream.reference_tags` is already a property. + Alternatives: `as_tags` / `AsTags`, `mark_tags` / `MarkTags`. +- No `strict` flag: a name that cannot be marked is always an error. + +## Order rule and identity + +- The names are ordered: the output's user tags are the input's user tags, in their + order, then the named columns in the order given. So the order is part of the + operator, and its identity is `("TagBy", [names in the order given])`; its pipeline + identity is the same. A `set` / `frozenset` is refused (`TypeError`, unordered, as for + `record_id_columns`), as is an empty list and a repeated name (`ValueError`). + +## Refusals + +At construction (`ValueError` / `TypeError`, naming the column): no name; a non-string; +a repeated name; a set; the meta prefix (`require_names_not_meta`); a system column name +(`require_names_not_system`); a name holding `::` (`require_names_without_separator`). + +At validation (`InputValidationError`, naming the column, raised when the pipeline is +built, sync and async alike): + +- not a data column of the input (lists the input's data columns); +- already a tag: a referential tag ("already a referential tag"); a plain tag is never a + data column, so it is reported as a tag, not a data column; +- a type that cannot be a tag. Recommended rule: the column's Python type must be one of + `int`, `float`, `bool`, `str`, `bytes`, `datetime`, `date` (see Decisions). Refused: `T | None` (the error says to apply + `drop_nulls("x")` first, which narrows it to `T`), lists, sets, dicts, tuples, structs, + user classes and every logical type (`Path`, `UUID`, arrays...). Sources and function + outputs are non-nullable by default, so this costs nothing in the common case. + Conservative variant: also refuse `float` (equality on floats as a key). + +## What the output is + +- Tags: input user tags, then the new names; system tags unchanged (rule 1, + name-preserving: same column names, same values; record ids unchanged). +- Data: unchanged: same names, order, types, nullability, values; each column keeps its + own source-info token. No token is created or merged. +- Table layout: the column is stored once and moves into the user tag group (row order: + user tags, system tags, data, source info), as a source places a referential tag. +- Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with + their data types, after the user tags and before the system tags; the data schema is + unchanged. Verified against `static_process(...).output_schema(...)`. +- Sync: the input table re-wrapped as an `ArrowTableStream` with the extra tags in + `tag_columns` and `reference_tags`: no Polars round trip, no cast. +- Async: no own `async_execute`: the base barrier path collects the rows and applies the + sync rule, so the rows are identical by construction. Added to + `test_async_sync_equivalence.py`. A streaming version can come later. + +## Why "no data element comes from a tag value" holds + +The operator adds a tag role to a data column; it creates no value and moves none. Every +output data column is the input data column of that name, with the same values and +source info, and no input tag becomes data. The column stays one physical column, as a +source's referential tag is. The direction the rule forbids (a tag value offered as data) +stays with `DerivedSource(tags_as_data=...)`. + +## Downstream + +- `Join` with a stream where `x` is a plain tag or a referential tag: the slice's rules + (one referential tag, one data column, merged `join_v1` token when referential on + several inputs). With a stream where `x` is data only: the existing tag/data error (its + remedy text gains "or mark it with `tag_by`"). +- `GroupBy(["x"])`, `Batch`, `MapTags`, select/drop: the slice's rules for any + referential tag. +- Removing the data role afterwards (`tag_by("x").drop_data_columns("x")`) leaves a plain + tag holding the data value, with the `ReferenceMaterializationWarning`, as for any + referential tag. That is the data-to-tag direction (the rule holds), but it makes the + issue's out-of-scope "plain tag from a data column" reachable in two steps. + Recommendation: allow (existing slice behaviour, warned). Conservative variant: none + that is small; it would need tracking where a tag came from. +- Jobs: an operator node runs it in the default cache mode; `LOG` / `REPLAY` refuse + referential tags as for every operator (slice limitation, unchanged). A function node + downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and + the class is registered in `pipeline/serialization.py`. + +## Open choice: may a computed column key a join? + +Recommendation: yes. It is the issue's own typical use (a function computes a session +label that later steps join, group or sort by), and nothing in hashing or record +identity depends on where a tag's value came from. Conservative variant: `TagBy` refuses +a column a function pod produced. That needs a lineage walk upstream (through renames and +joins) in one helper called from validation; it is not small, and I recommend against it. + +## Not in scope + +Golden pins (identities provisional in 0.4.0rc1); a plain-tag variant; the reverse +direction; `MergeJoin` (ITL-1194). diff --git a/src/orcapod/core/operators/__init__.py b/src/orcapod/core/operators/__init__.py index 032cd0a60..838f6eb7c 100644 --- a/src/orcapod/core/operators/__init__.py +++ b/src/orcapod/core/operators/__init__.py @@ -16,6 +16,7 @@ from .merge_join import MergeJoin from .pick import Pick from .semijoin import SemiJoin +from .tag_by import TagBy __all__ = [ "Join", @@ -36,4 +37,5 @@ "DropNulls", "Pick", "Index", + "TagBy", ] diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index eb65ae1ce..bd6851502 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -101,8 +101,8 @@ def _tag_data_error(name: str, tag_side: str, data_side: str) -> InputValidation return InputValidationError( f"column {name!r} is a tag in the {tag_side} input and a data column in the " + f"{data_side}, and a name cannot be both. Rename it in one input with " - + "map_tags or map_data, or declare it a tag at the source that has it as " - + "a data column." + + "map_tags or map_data, or make it a tag of the input that has it as a data column (declare it " + + "a tag at its source, or mark it with tag_by)." ) diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py new file mode 100644 index 000000000..fcfeee00b --- /dev/null +++ b/src/orcapod/core/operators/tag_by.py @@ -0,0 +1,213 @@ +"""TagBy operator: give existing data columns the tag role, as referential tags (ITL-1175).""" + +from __future__ import annotations + +from collections.abc import Sequence, Set +from typing import Any, ClassVar + +from typing_extensions import override + +from orcapod.core.operators.base import UnaryOperator +from orcapod.core.referential_tags import require_taggable_type, stream_references +from orcapod.core.streams import ArrowTableStream +from orcapod.errors import InputValidationError +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.system_constants import constants +from orcapod.types import ColumnConfig, Schema +from orcapod.utils import schema_utils + + +def _tag_by_names(columns: str | Sequence[str]) -> tuple[str, ...]: + """Return the names ``TagBy`` marks, in the order given. + + Args: + columns: One name, or a sequence of names. + + Returns: + The names, in the order given. + + Raises: + TypeError: If ``columns`` is a set (unordered: the order of the names is the + order of the new tags) or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + """ + if isinstance(columns, str): + return (columns,) + if isinstance(columns, Set): + raise TypeError( + "TagBy: give the column names as a list or a tuple, not a set: their " + + "order is the order of the new tags" + ) + names = tuple(columns) + if not names: + raise ValueError("TagBy: expected at least one column name") + if not_strings := [name for name in names if not isinstance(name, str)]: + raise TypeError(f"TagBy: column names must be strings, got {not_strings!r}") + if repeated := sorted({name for name in names if names.count(name) > 1}): + raise ValueError(f"TagBy: columns {repeated} are named more than once") + return names + + +class TagBy(UnaryOperator): + """Give existing data columns of a stream the tag role, as referential tags. + + Each named data column becomes a referential tag (ITL-1023): one column in + both roles, a tag and the data column it already was. Nothing else changes: + the data columns keep their names, order, types, values and source info, + the system tags keep their names and values (rule 1, name-preserving), and + no row is added, removed or reordered. The new tags follow the input's user + tags, in the order given, and the column is stored once, among the tags. + + The operator creates no value and moves none: every output data column is the + input data column of that name, and no tag becomes data. A data column that a + function computed may become a tag (developer, November 16, 2026), and then + key a join or a group. + + Args: + columns: One data column name, or a sequence of them. Their order is the + order of the new tags, and part of the operator's identity. + + Raises: + TypeError: If ``columns`` is a set or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + InputValidationError: If a name is reserved: the meta prefix, a system + column name, or one holding ``::``. + """ + + # Referential tags (ITL-1023): handled; this operator makes them. + REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = None + + def __init__(self, columns: str | Sequence[str], **kwargs: Any) -> None: + names = _tag_by_names(columns) + schema_utils.require_names_not_meta(names) + schema_utils.require_names_not_system(names) + schema_utils.require_names_without_separator(names) + self.columns: tuple[str, ...] = names + super().__init__(**kwargs) + + @override + def identity_structure(self) -> Any: + """Return the operator's identity: its class name and the names, in order. + + Returns: + ``("TagBy", [names in the order given])``. + """ + return (self.__class__.__name__, list(self.columns)) + + @override + def to_config(self) -> dict[str, Any]: + """Serialize this operator to a config dict. + + Returns: + A dict with ``class_name``, ``module_path`` and ``config`` keys, where + ``config`` holds ``columns``, the names in order. + """ + config = super().to_config() + config["config"] = {"columns": list(self.columns)} + return config + + @override + def __repr__(self) -> str: + return f"TagBy({list(self.columns)!r})" + + @override + def validate_unary_input(self, stream: StreamProtocol) -> None: + """Check that every name is a data column of ``stream`` that can be a tag. + + Args: + stream: The input stream. + + Raises: + InputValidationError: Naming the column, if it is already a tag + (referential or plain), is not a data column of ``stream``, or has a + type that cannot be a tag (``referential_tags.require_taggable_type``). + """ + tag_columns, data_columns = stream.keys() + references = stream_references(stream) + _, data_schema = stream.output_schema() + for name in self.columns: + if name in references: + raise InputValidationError( + f"TagBy: column {name!r} is already a referential tag" + ) + if name in tag_columns: + raise InputValidationError( + f"TagBy: column {name!r} is a plain tag, not a data column: only " + + "a data column can be given the tag role" + ) + user_data = [ + c for c in data_columns if not c.startswith(constants.META_PREFIX) + ] + if name not in user_data: + raise InputValidationError( + f"TagBy: column {name!r} is not a data column of the input; its " + + f"data columns are {user_data}" + ) + try: + require_taggable_type(data_schema[name]) + except InputValidationError as exc: + raise InputValidationError( + f"TagBy: column {name!r} cannot be a tag: {exc}" + ) from exc + + @override + def unary_output_schema( + self, + stream: StreamProtocol, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> tuple[Schema, Schema]: + """Return the input schemas with the named columns added to the tags. + + The tag schema is the input's user tags, then the named columns with their + data types, then the system tags (when ``columns`` includes them); the data + schema is the input's. + + Args: + stream: The input stream. + columns: Column inclusion config passed through to ``output_schema``. + all_info: Whether to include all info columns. + + Returns: + A ``(tag_schema, data_schema)`` tuple. + """ + tag_schema, data_schema = stream.output_schema( + columns=columns, all_info=all_info + ) + user_tags = { + name: python_type + for name, python_type in tag_schema.items() + if not name.startswith(constants.SYSTEM_TAG_PREFIX) + } + system_tags = { + name: python_type + for name, python_type in tag_schema.items() + if name.startswith(constants.SYSTEM_TAG_PREFIX) + } + new_tags = {name: data_schema[name] for name in self.columns} + return Schema({**user_tags, **new_tags, **system_tags}), data_schema + + @override + def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: + """Re-wrap the input's rows with the named columns among the tags. + + The same arrays, laid out with the tags first: no cast, no value changed. + + Args: + stream: The input stream. + + Returns: + The stream with the named columns as referential tags. + """ + tag_columns, _ = stream.keys() + table = stream.as_table( + columns={"source": True, "system_tags": True, "sort_by_tags": False} + ) + new_tag_columns = [*tag_columns, *self.columns] + rest = [name for name in table.column_names if name not in new_tag_columns] + return ArrowTableStream( + table.select([*new_tag_columns, *rest]), + tag_columns=new_tag_columns, + reference_tags=[*stream_references(stream), *self.columns], + ) diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 7c508d813..09a00d4d8 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -19,6 +19,8 @@ from __future__ import annotations +import datetime +import types import typing import warnings import weakref @@ -313,6 +315,58 @@ def without_reference_marks(table: "pa.Table") -> "pa.Table": return mark_references(table, ()) +# The Python types a column may have to become a referential tag by ``tag_by`` +# (ITL-1175): one value per row, of a type with plain equality and order. A column +# whose Arrow type the type converter reads as no Python type (``Any``: decimal, time +# of day, duration) is refused. Sources do not apply this rule yet: a source accepts +# any type for a declared tag. +TAGGABLE_TYPES: tuple[type, ...] = ( + int, + float, + bool, + str, + bytes, + datetime.datetime, + datetime.date, +) + + +def require_taggable_type(python_type: object) -> None: + """Refuse a column type that cannot become a referential tag (ITL-1175). + + The rule ``tag_by`` applies: the type must be one of ``TAGGABLE_TYPES``. A + nullable type (``T | None``) is refused, with the way out: ``drop_nulls`` narrows + the column to ``T``. Lists, sets, dicts, tuples, structs, user classes and every + logical type (``Path``, ``UUID``, arrays) are refused, and so is a column the type + converter gives no Python type (``Any``). + + Args: + python_type: The column's Python type, as ``output_schema()`` gives it. + + Raises: + InputValidationError: If the type cannot be a tag. The caller names the + column. + """ + if python_type in TAGGABLE_TYPES: + return + members = typing.get_args(python_type) + if typing.get_origin(python_type) in (typing.Union, types.UnionType) and ( + type(None) in members + ): + rest = [member for member in members if member is not type(None)] + if len(rest) == 1 and rest[0] in TAGGABLE_TYPES: + raise InputValidationError( + f"its type {python_type} is nullable, and a tag needs a value in " + + "every row: drop the rows with a null first with drop_nulls, which " + + f"narrows the column to {rest[0].__name__}" + ) + names = ", ".join(t.__name__ for t in TAGGABLE_TYPES) + raise InputValidationError( + f"its type {python_type} cannot be a tag: a tag is one value of one of " + + f"these types: {names}" + ) + + def not_supported_yet(operation: str, issue: str) -> NotImplementedError: """Return the error for an operation that does not handle referential tags yet. diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index a0016940d..a2f695b69 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -2,7 +2,14 @@ import logging from abc import abstractmethod -from collections.abc import AsyncIterator, Callable, Collection, Iterator, Mapping +from collections.abc import ( + AsyncIterator, + Callable, + Collection, + Iterator, + Mapping, + Sequence, +) from contextvars import ContextVar from datetime import datetime from typing import TYPE_CHECKING, Any, cast @@ -565,6 +572,41 @@ def drop_nulls( return DropNulls(columns)(self, label=label) + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> StreamBase: + """Give the named data columns the tag role, as referential tags. + + ``s.tag_by("x", "y")`` equals ``s.tag_by(["x", "y"])``. Each named data + column becomes a referential tag: one column, both a tag and the data column + it was, with its values and source info unchanged. The new tags follow this + stream's tags, in the order given. See ``orcapod.core.operators.TagBy``. + + Args: + *columns: One or more data column names, or one list or tuple of names + as the only positional argument (not a set: the order matters). + label: Optional node label for the pipeline graph. + + Returns: + The stream with the named columns as referential tags. + + Raises: + TypeError: When no name is given, the names are a set, or a name is not + a string. + ValueError: When a name is repeated. + InputValidationError: When a name is reserved, is not a data column of + this stream, is already a tag, or has a type that cannot be a tag. + """ + from orcapod.core.operators import TagBy + + if len(columns) == 1 and not isinstance(columns[0], str): + names: str | Sequence[str] = columns[0] + else: + names = cast("tuple[str, ...]", columns) + return TagBy(names)(self, label=label) + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/src/orcapod/pipeline/serialization.py b/src/orcapod/pipeline/serialization.py index ee71b74c3..af73be040 100644 --- a/src/orcapod/pipeline/serialization.py +++ b/src/orcapod/pipeline/serialization.py @@ -205,6 +205,7 @@ def _build_operator_registry() -> dict[str, type]: SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) return { @@ -224,6 +225,7 @@ def _build_operator_registry() -> dict[str, type]: "MapColumns": MapColumns, "PolarsFilter": PolarsFilter, "DropNulls": DropNulls, + "TagBy": TagBy, } diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index a329b3e16..67292e92c 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -421,6 +421,18 @@ def drop_nulls( """ ... + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> "StreamProtocol": + """ + Give the named data columns the tag role, as referential tags: each stays + one column, a tag and the data column it was, with its values and source + info unchanged. The new tags follow the stream's tags, in the order given. + """ + ... + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 60fbbbb75..12a8ab44b 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -47,6 +47,7 @@ SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod from orcapod.core.sources import ArrowTableSource @@ -585,6 +586,11 @@ class Case: lambda: MapData({"ta": "tb"}), lambda: (source_a(referential=True),), ), + # TagBy gives data columns the tag role, as referential tags (ITL-1175). + Case("tag_by", lambda: TagBy(["i8", "bn"]), lambda: (source_a(),)), + Case("tag_by_reversed", lambda: TagBy(["bn", "i8"]), lambda: (source_a(),)), + Case("tag_by_b", lambda: TagBy(["u8", "ls"]), lambda: (source_b(),)), + Case("ref_tag_by", lambda: TagBy("i8"), lambda: (source_a(referential=True),)), ] CASE = pytest.mark.parametrize("case", CASES, ids=[c.label for c in CASES]) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py new file mode 100644 index 000000000..a2116c3b5 --- /dev/null +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -0,0 +1,400 @@ +"""``TagBy`` / ``stream.tag_by``: data columns become referential tags (ITL-1175). + +A named data column becomes a referential tag: one column, a tag and the data column it +was, with its values and source info unchanged. The new tags follow the input's tags in +the order given. Every operator is driven through its public call and read through the +stream's public methods; a schema prediction is checked against ``static_process``. +""" + +from __future__ import annotations + +import datetime +import decimal +import warnings +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.core.operators import TagBy +from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, + stored_references, + stream_references, +) +from orcapod.databases import InMemoryArrowDatabase +from orcapod.errors import InputValidationError +from orcapod.pipeline.serialization import resolve_operator_from_config +from orcapod.pipeline import ( + AsyncPipelineOrchestrator, + PipelineJob, + SyncPipelineOrchestrator, +) +from orcapod.system_constants import constants + +ORCHESTRATORS = pytest.mark.parametrize( + "orchestrator", + [SyncPipelineOrchestrator, AsyncPipelineOrchestrator], + ids=["sync", "async"], +) + +ROWS = [ + {"unit": 1, "trial": 10, "amp": 0.5, "label": "a"}, + {"unit": 2, "trial": 20, "amp": 1.5, "label": "b"}, +] + + +def _plain() -> op.sources.DictSource: + """``unit`` a plain tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource( + ROWS, tag_columns=["unit"], tag_only_columns=True, source_id="spikes" + ) + + +def _referential() -> op.sources.DictSource: + """``unit`` a referential tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource(ROWS, tag_columns=["unit"], source_id="spikes") + + +@op.function_pod(output_keys=["session"]) +def session_of(trial: int) -> str: + """A computed value later steps key on.""" + return f"s{trial // 20}" + + +def _sorted_rows(table: pa.Table) -> list[dict[str, Any]]: + return sorted(table.to_pylist(), key=lambda row: repr(sorted(row.items()))) + + +# ── what the output is ────────────────────────────────────────────────────── + + +def test_named_data_columns_become_referential_tags_in_the_order_given() -> None: + tagged = _plain().tag_by("label", "trial") + assert tagged.keys() == ( + ("unit", "label", "trial"), + ("trial", "amp", "label"), + ) + assert stream_references(tagged) == ("label", "trial") + for tag, data in tagged.iter_data(): + assert tag["label"] == data["label"] and tag["trial"] == data["trial"] + + +def test_the_column_is_stored_once_among_the_tags() -> None: + table = _plain().tag_by("trial").as_table() + assert table.column_names == ["unit", "trial", "amp", "label"] + + +def test_values_system_tags_and_source_info_are_unchanged() -> None: + source = _plain() + tagged = source.tag_by("trial") + before = source.as_table(all_info=True) + after = tagged.as_table(all_info=True) + unchanged = [ + name + for name in before.column_names + if name.startswith((constants.SYSTEM_TAG_PREFIX, constants.SOURCE_PREFIX)) + or name in ("unit", "trial", "amp", "label") + ] + assert set(unchanged) <= set(after.column_names) + assert _sorted_rows(after.select(unchanged)) == _sorted_rows( + before.select(unchanged) + ) + # The data's types and nullability are the input's. + for name in ("trial", "amp", "label"): + assert after.schema.field(name) == before.schema.field(name) + + +def test_an_existing_reference_is_kept() -> None: + tagged = _referential().tag_by("trial") + assert stream_references(tagged) == ("unit", "trial") + assert tagged.keys() == (("unit", "trial"), ("unit", "trial", "amp", "label")) + + +def test_names_as_strings_or_one_sequence_are_one_operator() -> None: + source = _plain() + by_strings = source.tag_by("trial", "label") + by_list = source.tag_by(["trial", "label"]) + by_tuple = source.tag_by(("trial", "label")) + assert ( + by_strings.content_hash() == by_list.content_hash() == by_tuple.content_hash() + ) + assert by_strings.pipeline_hash() == by_list.pipeline_hash() + + +def test_the_order_of_the_names_is_part_of_the_identity() -> None: + source = _plain() + forward = source.tag_by("trial", "label") + backward = source.tag_by("label", "trial") + assert forward.keys()[0] == ("unit", "trial", "label") + assert backward.keys()[0] == ("unit", "label", "trial") + assert forward.content_hash() != backward.content_hash() + assert forward.pipeline_hash() != backward.pipeline_hash() + assert TagBy(["trial", "label"]).identity_structure() == ( + "TagBy", + ["trial", "label"], + ) + + +@pytest.mark.parametrize("all_info", [False, True]) +def test_the_schema_is_predicted(all_info: bool) -> None: + source = _referential() + tag_by = TagBy(["label", "trial"]) + predicted = tag_by.output_schema(source, all_info=all_info) + assert predicted == tag_by.static_process(source).output_schema(all_info=all_info) + tag_schema, data_schema = predicted + assert list(tag_schema)[:3] == ["unit", "label", "trial"] + assert data_schema == source.output_schema()[1] + + +def test_zero_rows_have_the_types_of_rows() -> None: + source = _plain() + tagged = source.tag_by("trial") + empty = source.polars_filter(unit=99).tag_by("trial") + assert empty.as_table().schema == tagged.as_table().schema + assert empty.as_table().num_rows == 0 + + +def test_the_config_round_trips_and_is_registered_for_loading() -> None: + tag_by = TagBy(["label", "trial"]) + config = tag_by.to_config() + assert config["class_name"] == "TagBy" + assert config["config"] == {"columns": ["label", "trial"]} + for loaded in (TagBy.from_config(config), resolve_operator_from_config(config)): + assert isinstance(loaded, TagBy) + assert loaded.columns == ("label", "trial") + assert loaded.content_hash() == tag_by.content_hash() + assert loaded.pipeline_hash() == tag_by.pipeline_hash() + + +# ── after a function pod: a computed column keys a join and a group ───────── + + +def test_a_computed_column_becomes_a_tag_and_keys_a_join() -> None: + computed = TagBy("session")(session_of.pod(_referential())) + assert computed.keys() == (("unit", "session"), ("unit", "session")) + sessions = op.sources.DictSource( + [{"session": "s0", "room": "A"}, {"session": "s1", "room": "B"}], + tag_columns=["session"], + source_id="sessions", + ) + joined = computed.join(sessions) + assert set(stream_references(joined)) == {"unit", "session"} + rows = sorted(joined.as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["room"]) for r in rows] == [ + (1, "s0", "A"), + (2, "s1", "B"), + ] + + +def test_a_computed_column_becomes_a_tag_and_keys_a_group() -> None: + grouped = TagBy("session")(session_of.pod(_referential())).group_by(["session"]) + assert grouped.keys() == (("session",), ("unit", "session")) + rows = sorted(grouped.as_table().to_pylist(), key=lambda r: r["session"]) + assert [(r["session"], r["unit"]) for r in rows] == [("s0", [1]), ("s1", [2])] + + +def test_a_join_with_a_data_only_column_of_that_name_is_refused() -> None: + other = op.sources.DictSource( + [{"k": 1, "label": "a"}], + tag_columns=["k"], + tag_only_columns=True, + source_id="o", + ) + with pytest.raises(InputValidationError, match="tag_by"): + _ = _plain().tag_by("label").join(other) + + +def test_removing_the_data_role_afterwards_leaves_a_plain_tag_with_the_warning() -> ( + None +): + tagged = _plain().tag_by("label") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + plain = tagged.drop_data_columns(["label"]) + assert any(w.category is ReferenceMaterializationWarning for w in caught) + assert plain.keys() == (("unit", "label"), ("trial", "amp")) + + +def test_a_nullable_column_is_tagged_after_drop_nulls() -> None: + table = pa.table( + {"unit": pa.array([1, 2]), "label": pa.array(["a", None], pa.large_string())} + ) + source = op.sources.ArrowTableSource( + table, tag_columns=["unit"], tag_only_columns=True, source_id="n" + ) + with pytest.raises(InputValidationError, match="drop_nulls"): + _ = source.tag_by("label") + tagged = source.drop_nulls("label").tag_by("label") + assert tagged.as_table().column("label").to_pylist() == ["a"] + + +# ── refusals ──────────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize( + ("columns", "error"), + [ + ({"trial", "label"}, TypeError), + (frozenset({"trial"}), TypeError), + ([], ValueError), + (["trial", "trial"], ValueError), + ([1], TypeError), + (["__meta"], InputValidationError), + ([f"{constants.SYSTEM_TAG_PREFIX}x"], InputValidationError), + ([f"{constants.SOURCE_PREFIX}x"], InputValidationError), + (["a::b"], InputValidationError), + ], + ids=[ + "set", + "frozenset", + "empty", + "repeated", + "not_a_string", + "meta", + "system_tag", + "source_info", + "separator", + ], +) +def test_bad_names_are_refused_at_construction(columns: Any, error: type) -> None: + with pytest.raises(error): + _ = TagBy(columns) + + +def test_the_stream_method_refuses_no_name_and_a_set() -> None: + with pytest.raises(TypeError): + _ = _plain().tag_by({"trial"}) # pyright: ignore[reportArgumentType] # the refusal is the test + with pytest.raises(ValueError, match="at least one"): + _ = _plain().tag_by([]) + + +@pytest.mark.parametrize( + ("name", "match"), + [ + ("missing", "'missing' is not a data column"), + ("unit", "'unit' is a plain tag"), + ], +) +def test_a_name_that_is_not_a_data_column_is_refused(name: str, match: str) -> None: + with pytest.raises(InputValidationError, match=match): + _ = _plain().tag_by(name) + + +def test_an_existing_referential_tag_is_refused() -> None: + with pytest.raises(InputValidationError, match="'unit' is already a referential"): + _ = _referential().tag_by("unit") + + +def _typed_source(column: pa.Array[Any] | pa.ChunkedArray[Any]) -> Any: + """Tag ``unit``; data ``v`` of the column's type, non-nullable.""" + schema = pa.schema( + [ + pa.field("unit", pa.int64(), nullable=False), + pa.field("v", column.type, False), + ] + ) + return op.sources.ArrowTableSource( + pa.Table.from_arrays([pa.array([1, 2], pa.int64()), column], schema=schema), + tag_columns=["unit"], + tag_only_columns=True, + source_id="typed", + ) + + +@pytest.mark.parametrize( + "column", + [ + pa.array([[1], [2]], pa.large_list(pa.int64())), + pa.array([{"a": 1}, {"a": 2}], pa.struct([("a", pa.int64())])), + pa.array([decimal.Decimal("1.5"), decimal.Decimal("2.5")]), + pa.array([datetime.time(1), datetime.time(2)]), + pa.array([datetime.timedelta(1), datetime.timedelta(2)]), + ], + ids=["list", "struct", "decimal", "time", "duration"], +) +def test_a_nested_or_untyped_type_is_refused(column: pa.Array[Any]) -> None: + with pytest.raises(InputValidationError, match="'v' cannot be a tag"): + _ = _typed_source(column).tag_by("v") + + +def test_a_logical_type_is_refused() -> None: + source = op.sources.DictSource( + [{"unit": 1, "path": Path("/a")}, {"unit": 2, "path": Path("/b")}], + tag_columns=["unit"], + tag_only_columns=True, + source_id="paths", + ) + with pytest.raises(InputValidationError, match="'path' cannot be a tag"): + _ = source.tag_by("path") + + +@pytest.mark.parametrize( + "column", + [ + pa.array([1, 2], pa.int8()), + pa.array([0.5, 1.5], pa.float32()), + pa.array([True, False]), + pa.array(["a", "b"], pa.string()), + pa.array([b"a", b"b"], pa.binary()), + pa.array([datetime.datetime(2026, 11, 16), datetime.datetime(2026, 11, 17)]), + pa.array([datetime.date(2026, 11, 16), datetime.date(2026, 11, 17)]), + ], + ids=[ + "int8", + "float32", + "bool", + "string", + "binary", + "timestamp", + "date", + ], +) +def test_a_scalar_type_is_accepted(column: pa.Array[Any]) -> None: + source = _typed_source(column) + tagged = source.tag_by("v") + assert stream_references(tagged) == ("v",) + assert tagged.as_table().schema.field("v") == source.as_table().schema.field("v") + + +# ── in a job ──────────────────────────────────────────────────────────────── + + +@ORCHESTRATORS +def test_in_a_job_a_function_node_stores_the_new_tag_marked( + tmp_path: Path, + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(database=database) + with job: + computed = session_of.pod(_referential(), label="session") + tagged = TagBy("session")(computed, label="tag_by") + double.pod(tagged, label="double") + job.run(orchestrator=orchestrator()) + assert job.nodes["tag_by"].keys() == (("unit", "session"), ("unit", "session")) + assert job.nodes["double"].keys() == (("unit", "session"), ("unit", "session", "y")) + rows = sorted(job.nodes["double"].as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["y"]) for r in rows] == [ + (1, "s0", "s0s0"), + (2, "s1", "s1s1"), + ] + marked = { + stored_references(database.get_all_records(path).schema) # pyright: ignore[reportOptionalMemberAccess] # every listed table has records + for path in database.list_table_paths() + if "_result" not in path + } + assert ("unit", "session") in marked + + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json") + assert loaded.nodes["tag_by"].keys() == job.nodes["tag_by"].keys() + assert loaded.nodes["tag_by"].content_hash() == job.nodes["tag_by"].content_hash() + + +@op.function_pod(output_keys=["y"]) +def double(session: str) -> str: + """Reads the new tag's data column; ``unit`` falls through.""" + return session * 2 From 6aa23146826c0644baf5a7f1f4f3a6868a1b5b31 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:38:23 -0800 Subject: [PATCH 02/10] docs(referential-tags): describe the three moves by role after tag_by (ITL-1175) Developer decision (ITL-1195): dropping the tag or the data of a referential tag removes that role only; dropping the column removes both. Co-Authored-By: Claude Opus 5.5 --- docs/concepts/referential-tags.md | 6 ++++-- docs/dev/specs/ITL-1175-tag-by.md | 7 +++++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index d1183047e..29cc256b4 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -116,8 +116,10 @@ A nullable column (`T | None`) is refused: drop the rows with a null first, `s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, your own classes and logical types (`Path`, `UUID`, arrays) are refused. -`tag_by` is one of three moves between the roles of a column. The other two, which turn -a referential tag into plain data or into a plain tag, are the subject of ITL-1195. +`tag_by` adds the tag role; the drop operators remove a role. For a referential tag `x`, +dropping the tag (`drop_tag_columns(["x"])`) leaves `x` plain data, dropping the data +(`drop_data_columns(["x"])`) leaves `x` a plain tag, and dropping the column +(`drop_columns(["x"])`) removes it from both roles. ## Operators diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index 057426806..fa79c055b 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -17,8 +17,11 @@ records what changed. duration are not in it: the type converter reads them as no Python type (`Any`), and `Any` is refused. The source path is not changed here; the inconsistency is reported in the PR. -- Removing the data role afterwards is the subject of ITL-1195 (explicit operators); the - docs leave a placeholder for it instead of describing the implicit path. +- Removing a role afterwards (developer, November 16, 2026, ITL-1195): no new operators. + For a referential tag, dropping the tag leaves plain data, dropping the data leaves a + plain tag, dropping the column removes both roles. So `tag_by("x")` followed by + `drop_data_columns(["x"])` leaving a plain tag is intended behaviour. The docs + describe the three moves by role in one paragraph; final wording comes with ITL-1195. ## Public API From eb32ab136360616dee6ce8f4a48d87e7d1e4beb3 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:42:04 -0800 Subject: [PATCH 03/10] test(tag_by): a logged TagBy operator node stores its new tag marked (ITL-1175) Co-Authored-By: Claude Opus 5.5 --- .../test_core/referential_tags/test_tag_by.py | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index a2116c3b5..35fa5865b 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -32,7 +32,8 @@ PipelineJob, SyncPipelineOrchestrator, ) -from orcapod.system_constants import constants +from orcapod.system_constants import OPERATOR_DB_SCHEMA_VERSION, constants +from orcapod.types import CacheMode ORCHESTRATORS = pytest.mark.parametrize( "orchestrator", @@ -398,3 +399,24 @@ def test_in_a_job_a_function_node_stores_the_new_tag_marked( def double(session: str) -> str: """Reads the new tag's data column; ``unit`` falls through.""" return session * 2 + + +@ORCHESTRATORS +def test_a_logged_tag_by_node_stores_its_new_tag_marked( + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(name="ops", database=database) + with job: + _ = TagBy("trial")(_referential(), label="tag_by") + job.nodes["tag_by"].attach_databases( + pipeline_database=database.at("ops"), cache_mode=CacheMode.LOG + ) + job.run(orchestrator=orchestrator()) + paths = [ + p for p in database.list_table_paths() if p[-1] == OPERATOR_DB_SCHEMA_VERSION + ] + assert len(paths) == 1 + stored = database.get_all_records(paths[0]) + assert stored is not None + assert stored_references(stored.schema) == ("unit", "trial") From a6b4a17a1c523945bc6bb86c14b84aeed3a68dbf Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:43:43 -0800 Subject: [PATCH 04/10] docs(referential-tags): select and drop act on a referential tag by role (ITL-1195) Developer decision (November 16, 2026): for a referential tag, dropping the tag removes the tag role only (plain data), dropping the data removes the data role only (plain tag), dropping the column removes both; the select operators follow the same rule. The slice already behaved so: this states the rule in "Moving a column between roles" and tests it by role (keys, references, values, system tags, source info, schema predictions against static_process; the missing async case). No new operator or alias. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-1195.md | 11 ++ docs/concepts/referential-tags.md | 49 ++++- .../operators/test_async_sync_equivalence.py | 6 + .../test_core/referential_tags/test_roles.py | 168 ++++++++++++++++++ 4 files changed, 226 insertions(+), 8 deletions(-) create mode 100644 changelog.d/ITL-1195.md create mode 100644 tests/test_core/referential_tags/test_roles.py diff --git a/changelog.d/ITL-1195.md b/changelog.d/ITL-1195.md new file mode 100644 index 000000000..847be53fd --- /dev/null +++ b/changelog.d/ITL-1195.md @@ -0,0 +1,11 @@ +### Changed + +- The select and drop operators act on a referential tag by role (developer decision, + November 16, 2026, ITL-1195), as documented in "Moving a column between roles": for a + referential tag `x`, dropping the tag (`drop_tag_columns`) removes the tag role only + and leaves `x` plain data with its source info; dropping the data (`drop_data_columns`) + removes the data role only and leaves `x` a plain tag with the same values; dropping + the column (`drop_columns`) removes it from both roles. `select_tag_columns`, + `select_data_columns` and `select_columns` follow the same rule for the names they do + not select. This was already the slice's behaviour; it is now the stated rule, with + tests by role. No second name (such as `untag`) is added. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index c1aff9542..a15938c63 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -74,7 +74,45 @@ A function output named like a tag is an error when the pipeline is built. A fun that does not read a referenced column keeps its result identity: retagging a source does not make it compute again. -## Making a data column a tag: `tag_by` +## Moving a column between roles + +A referential tag is one column in two roles, a tag and a data column. Four steps move a +column between the roles, and none of them changes, copies or creates a value: + +| Step | Data column `x` | Referential tag `x` | Plain tag `x` | +|---|---|---|---| +| `tag_by("x")` | becomes a referential tag | refused: already a tag | refused: not a data column | +| `drop_tag_columns(["x"])` | refused: not a tag | the tag role goes: `x` is plain data | removed | +| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag | refused: not a data column | +| `drop_columns(["x"])` | removed | removed from both roles | removed | + +The select operators follow the same rule by role: `select_tag_columns` removes the tag +role of every tag it does not name, `select_data_columns` the data role of every data +column it does not name, and `select_columns` keeps exactly the columns it names, in +both roles. (The drop and select refusals are those of a missing name: `strict=False` +ignores the name instead.) + +```python +print(spikes.drop_tag_columns(["time"]).keys()) +# (('unit',), ('unit', 'time', 'amp')) +print(spikes.drop_data_columns(["time"]).keys()) +# (('unit', 'time'), ('unit', 'amp')) +print(spikes.drop_columns(["time"]).keys()) +# (('unit',), ('unit', 'amp')) +``` + +- System tags and record ids never change. +- Source info belongs to the data role: dropping the tag keeps the column's source-info + token, dropping the data removes it (a plain tag carries none). The row's provenance + is still its system tags (its source id and record id). +- No data element comes from a tag value: dropping the tag leaves a data column that was + data before; dropping the data leaves a tag whose values came from a data column. +- A stream keeps at least one data column: dropping the data role of the last one is + refused. +- A stored table marks exactly the referential tags of what it stores, so a column moved + out of one role is stored unmarked. + +### Adding the tag role: `tag_by` A source declares its tags, but a value a later step should join, group or sort by is often computed on the way: a session label, a trial index, a phase. `tag_by` gives @@ -116,11 +154,6 @@ A nullable column (`T | None`) is refused: drop the rows with a null first, `s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, your own classes and logical types (`Path`, `UUID`, arrays) are refused. -`tag_by` adds the tag role; the drop operators remove a role. For a referential tag `x`, -dropping the tag (`drop_tag_columns(["x"])`) leaves `x` plain data, dropping the data -(`drop_data_columns(["x"])`) leaves `x` a plain tag, and dropping the column -(`drop_columns(["x"])`) removes it from both roles. - ## Operators | Operator | Referential tag `x` | @@ -128,8 +161,8 @@ dropping the tag (`drop_tag_columns(["x"])`) leaves `x` plain data, dropping the | `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. | -| `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. | -| `SelectTagColumns`, `DropTagColumns` | Removing the tag leaves `x` a data column. | +| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `ReferenceMaterializationWarning`. | +| `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. | | `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. | | `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. | | `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 12a8ab44b..7f9bf0b70 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -545,6 +545,12 @@ class Case: lambda: DropTagColumns(["ta"]), lambda: (source_a(referential=True),), ), + # Not selecting a referential tag removes its tag role only (ITL-1195). + Case( + "ref_select_tag_columns", + lambda: SelectTagColumns(["k"]), + lambda: (source_a(referential=True),), + ), Case( "ref_drop_data_columns", lambda: DropDataColumns(["k"]), diff --git a/tests/test_core/referential_tags/test_roles.py b/tests/test_core/referential_tags/test_roles.py new file mode 100644 index 000000000..d871261eb --- /dev/null +++ b/tests/test_core/referential_tags/test_roles.py @@ -0,0 +1,168 @@ +"""The select and drop operators act on a referential tag by role (ITL-1195). + +Developer decision, November 16, 2026: for a referential tag ``x`` (one column in two +roles), dropping the tag removes the tag role only and leaves ``x`` plain data; dropping +the data removes the data role only and leaves ``x`` a plain tag; dropping the column +removes it from both roles. The select operators follow the same rule: selecting tags +or data drops the others by role. Plain tags and plain data columns behave as before. + +Every operator is driven through its public call and read through the stream's public +methods; a schema prediction is checked against ``static_process``. +""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import Any + +import pytest + +import orcapod as op +from orcapod.core.operators import ( + DropColumns, + DropDataColumns, + DropTagColumns, + SelectColumns, + SelectDataColumns, + SelectTagColumns, + TagBy, +) +from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod +from orcapod.core.referential_tags import stream_references +from orcapod.errors import InputValidationError +from orcapod.system_constants import constants + +ROWS = [{"k": 1, "x": 10, "y": 0.5}, {"k": 2, "x": 20, "y": 1.5}] + + +def _source() -> op.sources.DictSource: + """Referential tags ``k`` and ``x``; data ``y``.""" + return op.sources.DictSource(ROWS, tag_columns=["k", "x"], source_id="R") + + +def _system_tags(stream: Any) -> list[dict[str, Any]]: # pyright: ignore[reportExplicitAny] # any stream + table = stream.as_table(columns={"system_tags": True}) + names = [c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX)] + return sorted(table.select(names).to_pylist(), key=repr) + + +def _source_info(stream: Any) -> dict[str, list[Any]]: # pyright: ignore[reportExplicitAny] # any stream + table = stream.as_table(columns={"source": True}) + return { + name.removeprefix(constants.SOURCE_PREFIX): table.column(name).to_pylist() + for name in table.column_names + if name.startswith(constants.SOURCE_PREFIX) + } + + +# (operator, expected keys, expected referential tags): ``x`` is the one moved. +ROLE_CASES: list[tuple[str, Callable[[], StaticOutputOperatorPod], Any, Any]] = [ + # Dropping or not selecting the tag: the tag role goes, x is plain data. + ("drop_tag", lambda: DropTagColumns(["x"]), (("k",), ("k", "x", "y")), ("k",)), + ("select_tag", lambda: SelectTagColumns(["k"]), (("k",), ("k", "x", "y")), ("k",)), + # Dropping or not selecting the data: the data role goes, x is a plain tag. + ("drop_data", lambda: DropDataColumns(["x"]), (("k", "x"), ("k", "y")), ("k",)), + ( + "select_data", + lambda: SelectDataColumns(["k", "y"]), + (("k", "x"), ("k", "y")), + ("k",), + ), + # Dropping or not selecting the column: both roles go. + ("drop_column", lambda: DropColumns(["x"]), (("k",), ("k", "y")), ("k",)), + ("select_column", lambda: SelectColumns(["k", "y"]), (("k",), ("k", "y")), ("k",)), +] +ROLE_IDS = [case[0] for case in ROLE_CASES] + + +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +def test_each_operator_acts_by_role( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # expected keys + references: tuple[str, ...], +) -> None: + source = _source() + result = make_op()(source) + assert result.keys() == keys + assert stream_references(result) == references + # No value changes: every remaining column holds the source's values. + for row in result.as_table().to_pylist(): + assert row == {name: _by_k(row["k"])[name] for name in row} + # System tags and record ids never change. + assert _system_tags(result) == _system_tags(source) + # Source info belongs to the data role. + info = _source_info(result) + _, data = keys + assert set(info) == set(data) + if "x" in data: + assert info["x"] == _source_info(source)["x"] + _ = label + + +def _by_k(k: int) -> dict[str, Any]: # pyright: ignore[reportExplicitAny] # row values + return next(row for row in ROWS if row["k"] == k) + + +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +@pytest.mark.parametrize("all_info", [False, True]) +def test_the_schema_is_predicted_by_role( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # expected keys + references: tuple[str, ...], + all_info: bool, +) -> None: + source = _source() + operator = make_op() + predicted = operator.output_schema(source, all_info=all_info) + assert predicted == operator.static_process(source).output_schema(all_info=all_info) + _ = (label, keys, references) + + +def test_dropping_the_data_role_of_the_last_data_column_is_refused() -> None: + only = op.sources.DictSource( + [{"k": 1, "x": 10}], + tag_columns=["k", "x"], + tag_only_columns=["k"], + source_id="O", + ) + assert only.keys() == (("k", "x"), ("x",)) + with pytest.raises(InputValidationError, match="no data column would remain"): + _ = DropDataColumns(["x"])(only) + + +def test_a_role_that_is_not_there_is_a_missing_name() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k", "x"], tag_only_columns=["x"], source_id="P" + ) + # A plain tag has no data role; a data-only column has no tag role. + with pytest.raises(InputValidationError, match="Missing data columns"): + _ = DropDataColumns(["x"])(plain) + with pytest.raises(InputValidationError, match="Missing tag columns"): + _ = DropTagColumns(["y"])(plain) + + +def test_plain_tags_and_data_are_dropped_as_before() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k", "x"], tag_only_columns=True, source_id="P" + ) + assert DropTagColumns(["x"])(plain).keys() == (("k",), ("y",)) + assert DropColumns(["x"])(plain).keys() == (("k",), ("y",)) + assert SelectTagColumns(["k"])(plain).keys() == (("k",), ("y",)) + + +def test_tag_by_then_drop_tag_gives_back_the_rows() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k"], tag_only_columns=True, source_id="P" + ) + back = DropTagColumns(["x"])(TagBy("x")(plain)) + assert back.keys() == plain.keys() + assert sorted(back.as_table().to_pylist(), key=repr) == sorted( + plain.as_table().to_pylist(), key=repr + ) + assert _source_info(back) == _source_info(plain) From 3ab814eb4518bd75fe1367da56c0db09912ce4c7 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:47:30 -0800 Subject: [PATCH 05/10] refactor(referential_tags): remove ReferenceMaterializationWarning (ITL-1195) Dropping the data of a referential tag removes its data role only: that is the defined meaning of the call (developer, November 16, 2026), so the warning that guarded against it goes, with what existed only for it: the reference_materialization setting, warn_materialized and materialization_diagnostic_off (and its use around the function pod call's automatic select). Tests, docs and the slice's changelog fragment updated. One commit on purpose: reverting it restores the warning (the conservative variant, to keep it where the data role is removed by omission). Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-1023.md | 3 +- changelog.d/ITL-1195.md | 3 + docs/concepts/referential-tags.md | 14 +--- src/orcapod/core/function_pod.py | 6 +- .../core/operators/column_selection.py | 10 +-- src/orcapod/core/referential_tags.py | 81 ++----------------- .../referential_tags/test_derived_source.py | 6 +- .../referential_tags/test_function_pods.py | 16 ++-- .../test_operator_node_storage.py | 5 +- .../referential_tags/test_operators.py | 27 +------ .../test_core/referential_tags/test_tag_by.py | 12 +-- 11 files changed, 27 insertions(+), 156 deletions(-) diff --git a/changelog.d/ITL-1023.md b/changelog.d/ITL-1023.md index 2575718cb..5a2f406aa 100644 --- a/changelog.d/ITL-1023.md +++ b/changelog.d/ITL-1023.md @@ -25,8 +25,7 @@ - `MapTags` renames a referential tag in both roles; `tag_only=True` renames the tag only. `MapData` is a data-only move. - Removing the data column of a referential tag (`SelectDataColumns`, `DropDataColumns`) - leaves a plain tag and emits a `ReferenceMaterializationWarning` - (`orcapod.core.referential_tags.reference_materialization = "silent"` hides it). + removes its data role only and leaves a plain tag (ITL-1195). - A function output named like a tag of its input is an error when the pipeline is built. - Calling a function pod, `pod(stream)`, lets the input's referential tags fall through: the output rows carry the function's outputs and the row's own referenced columns, so diff --git a/changelog.d/ITL-1195.md b/changelog.d/ITL-1195.md index 847be53fd..910399859 100644 --- a/changelog.d/ITL-1195.md +++ b/changelog.d/ITL-1195.md @@ -9,3 +9,6 @@ `select_data_columns` and `select_columns` follow the same rule for the names they do not select. This was already the slice's behaviour; it is now the stated rule, with tests by role. No second name (such as `untag`) is added. +- Removing the data role of a referential tag emits no warning: it is the defined meaning + of the call. `ReferenceMaterializationWarning`, the + `referential_tags.reference_materialization` setting and their helpers are removed. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index a15938c63..0835e7829 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -161,9 +161,9 @@ your own classes and logical types (`Path`, `UUID`, arrays) are refused. | `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. | -| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `ReferenceMaterializationWarning`. | +| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info. | | `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. | -| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. | +| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split. | | `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. | | `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | | Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the reference. | @@ -252,16 +252,6 @@ print(grouped.column("_source_v").to_pylist()) # [['sessions::row_0::v', 'sessions::row_1::v']] ``` -The materialization warning can be turned off; it never changes a result or a hash: - -```python -from orcapod.core import referential_tags - -referential_tags.reference_materialization = "silent" -# ... build the pipeline ... -referential_tags.reference_materialization = "warn" # the default -``` - ## Stored tables and saved jobs A stored table states which of its tags are referential. A function node stores each diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index ec5fdba4b..cce9ca395 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -16,7 +16,6 @@ from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.datagrams import Data from orcapod.core.referential_tags import ( - materialization_diagnostic_off, reference_names, stream_references, ) @@ -847,10 +846,7 @@ def __call__( self.label, dropped, ) - # The select is Orcapod's own step: the referential tags it turns into - # plain tags are not reported (ITL-1023). - with materialization_diagnostic_off(): - input_stream = SelectDataColumns(columns=kept)(input_stream) + input_stream = SelectDataColumns(columns=kept)(input_stream) if fall_through_pod is not None and references: return ReferenceFallThroughPod(fall_through_pod, references).process( input_stream, label=label diff --git a/src/orcapod/core/operators/column_selection.py b/src/orcapod/core/operators/column_selection.py index 7a8b54495..e5b5bd399 100644 --- a/src/orcapod/core/operators/column_selection.py +++ b/src/orcapod/core/operators/column_selection.py @@ -6,7 +6,7 @@ from orcapod.core.operators.meta_columns import WithoutMetaWriter from orcapod.channels import ReadableChannel, WritableChannel from orcapod.core.operators.base import UnaryOperator -from orcapod.core.referential_tags import stream_references, warn_materialized +from orcapod.core.referential_tags import stream_references from orcapod.core.streams import ArrowTableStream from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol @@ -265,10 +265,6 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: raise InputValidationError( f"Missing data columns: {missing_columns}. Make sure all specified columns to select are present or use strict=False to ignore missing columns" ) - warn_materialized( - self, - [c for c in stream_references(stream) if c not in self.columns], - ) require_a_data_column([c for c in data_columns if c in columns_to_select]) def unary_output_schema( @@ -549,10 +545,6 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: raise InputValidationError( f"Missing data columns: {missing_columns}. Make sure all specified columns to drop are present or use strict=False to ignore missing columns" ) - warn_materialized( - self, - [c for c in stream_references(stream) if c in self.columns], - ) require_a_data_column([c for c in data_columns if c not in self.columns]) def unary_output_schema( diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 52bc7d465..7f906e4aa 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -9,8 +9,9 @@ schema field records it. This module holds what more than one operator needs: finding the references of a stream, -the namespace validator (sections 4 and 13.4), and the diagnostic emitted when a structural -operator turns a reference into a plain tag (section 7). +the namespace validator (sections 4 and 13.4), the stored marker, and the type rule of +``tag_by``. Removing a role is no diagnostic: dropping the tag or the data of a +referential tag removes that role only (developer, November 16, 2026, ITL-1195). The identities of referential streams are not frozen yet: on the integration branch of ITL-1023 they may still change. Plain (``tag_only_columns``) streams keep their v0.3 @@ -22,12 +23,8 @@ import datetime import types import typing -import warnings -import weakref -from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence -from contextlib import contextmanager -from contextvars import ContextVar -from typing import TYPE_CHECKING, Any, Literal +from collections.abc import Callable, Iterable, Mapping, Sequence +from typing import TYPE_CHECKING, Any from orcapod.errors import InputValidationError from orcapod.types import SourceInfoValue @@ -39,45 +36,6 @@ else: pa = LazyModule("pyarrow") -# Diagnostic emitted when a structural operator materializes a reference (section 7.4). -# ``"warn"`` emits a ``ReferenceMaterializationWarning``; ``"silent"`` emits nothing. The -# setting never changes a result, a schema or a hash. A module-level default for now. -reference_materialization: Literal["warn", "silent"] = "warn" - - -class ReferenceMaterializationWarning(UserWarning): - """A structural operator turned a referential tag into a plain tag (section 7).""" - - -# Set while Orcapod itself adds a structural step the user did not write (the select -# of a function pod call): its materializations are not reported. -_diagnostic_off: ContextVar[bool] = ContextVar( - "reference_materialization_off", default=False -) - - -# The operations that already reported, by id, holding the object so the id is not -# reused while the entry exists. -_reported: "weakref.WeakValueDictionary[int, object]" = weakref.WeakValueDictionary() - - -@contextmanager -def materialization_diagnostic_off() -> Iterator[None]: - """Build the steps in this block without the materialization diagnostic. - - Used for steps Orcapod adds by itself, such as the automatic select of a function - pod call, which would otherwise report on every call. The steps and their results - are exactly the same. - - Yields: - Nothing. - """ - token = _diagnostic_off.set(True) - try: - yield - finally: - _diagnostic_off.reset(token) - def reference_names( tag_names: Iterable[str], data_names: Iterable[str] @@ -143,35 +101,6 @@ def check_namespaces( ) -def warn_materialized(operation: object, names: Iterable[str]) -> None: - """Report that ``operation`` turned the referential tags ``names`` into plain tags. - - Its job is the diagnostic of section 7.4, so it takes the operation. It reports - once per operation object: a job validates its operators again when it builds and - runs its nodes, and that is not a new structural step. An operation built while - ``materialization_diagnostic_off`` is active (Orcapod's own step) never reports. - Emits nothing when ``names`` is empty or ``reference_materialization`` is - ``"silent"``. - - Args: - operation: The structural operation (an operator instance). - names: The tags that became plain. - """ - names = tuple(names) - if not names or id(operation) in _reported: - return - _reported[id(operation)] = operation - if reference_materialization == "silent" or _diagnostic_off.get(): - return - warnings.warn( - f"{type(operation).__name__}: the data column of referential tag(s) {list(names)} was removed; " - "they are now plain tags with the same values (set " - "orcapod.core.referential_tags.reference_materialization = 'silent' to hide this)", - ReferenceMaterializationWarning, - stacklevel=3, - ) - - # The marker of a referential tag in a stored table (developer, November 16, 2026, # ITL-1019): Arrow field metadata on the stored tag column, at the level of a logical # type's extension metadata. Only a referential tag carries it; a plain tag carries no diff --git a/tests/test_core/referential_tags/test_derived_source.py b/tests/test_core/referential_tags/test_derived_source.py index 1a7176ba8..c67b1793c 100644 --- a/tests/test_core/referential_tags/test_derived_source.py +++ b/tests/test_core/referential_tags/test_derived_source.py @@ -9,7 +9,6 @@ from __future__ import annotations -import warnings from collections.abc import Callable from pathlib import Path @@ -20,7 +19,6 @@ from orcapod.core.operators import Join from orcapod.core.referential_tags import ( REFERENTIAL_TAG_METADATA_KEY, - ReferenceMaterializationWarning, mark_references, without_reference_marks, ) @@ -153,9 +151,7 @@ def test_as_source_of_an_operator_node_keeps_its_bindings( make_store: Callable[[Path], ArrowDatabaseProtocol], orchestrator: Orchestrator, ) -> None: - with warnings.catch_warnings(): - warnings.simplefilter("ignore", ReferenceMaterializationWarning) - job = _operator_job(make_store(tmp_path), orchestrator) + job = _operator_job(make_store(tmp_path), orchestrator) node = job.nodes["joined"] source = node.as_source() assert source.keys() == node.keys() diff --git a/tests/test_core/referential_tags/test_function_pods.py b/tests/test_core/referential_tags/test_function_pods.py index d2bb91086..f097b7cb3 100644 --- a/tests/test_core/referential_tags/test_function_pods.py +++ b/tests/test_core/referential_tags/test_function_pods.py @@ -20,7 +20,6 @@ import orcapod as op from orcapod.core.operators import SelectDataColumns from orcapod.core.referential_tags import ( - ReferenceMaterializationWarning, stream_references, ) from orcapod.databases import InMemoryArrowDatabase @@ -113,9 +112,7 @@ def half(amp: float) -> float: ) # The plain pod (``t`` becomes a plain tag), joined back on ``t`` and ``row`` with # the input's referenced column: plain with referential gives the reference back. - with warnings.catch_warnings(): - warnings.simplefilter("ignore", ReferenceMaterializationWarning) - computed = half.pod.process(with_row.select_data_columns(["amp"])) + computed = half.pod.process(with_row.select_data_columns(["amp"])) joined = op.operators.Join()(computed, with_row.select_data_columns(["t"])) composed = sorted( (tag["t"], data["t"], data["half"]) for tag, data in joined.iter_data() @@ -161,17 +158,18 @@ def halve(amp: float) -> float: assert out.keys() == (("t",), ("t", "amp")) -def test_the_call_reports_no_materialization() -> None: +def test_removing_a_data_role_emits_no_warning() -> None: + """Dropping the data of a referential tag is the defined meaning (ITL-1195).""" + @op.function_pod(output_keys=["half"]) def half(amp: float) -> float: return amp / 2 - with warnings.catch_warnings(): - warnings.simplefilter("error", ReferenceMaterializationWarning) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") half.pod(_source()) - # A select written by hand that drops the referenced column does report. - with pytest.warns(ReferenceMaterializationWarning): SelectDataColumns(["amp"])(_source()) + assert [str(w.message) for w in caught] == [] CALLS: list[float] = [] diff --git a/tests/test_core/referential_tags/test_operator_node_storage.py b/tests/test_core/referential_tags/test_operator_node_storage.py index 0a1df68dd..69f089a5e 100644 --- a/tests/test_core/referential_tags/test_operator_node_storage.py +++ b/tests/test_core/referential_tags/test_operator_node_storage.py @@ -11,7 +11,6 @@ from __future__ import annotations import dataclasses -import warnings from collections.abc import Callable from pathlib import Path from typing import cast @@ -25,7 +24,6 @@ from orcapod.core.operators import Join from orcapod.core.referential_tags import ( REFERENTIAL_TAG_METADATA_KEY, - ReferenceMaterializationWarning, stored_references, without_reference_marks, ) @@ -163,8 +161,7 @@ def test_a_reference_the_operator_materializes_is_not_marked() -> None: """The marker is the operator's output binding, never one that came from upstream.""" store = InMemoryArrowDatabase() job = PipelineJob(name=JOB, database=store) - with job, warnings.catch_warnings(): - warnings.simplefilter("ignore", ReferenceMaterializationWarning) + with job: _ = _source("a", "x", [1, 2]).select_data_columns(["x"], label="selected") node = job.nodes["selected"] node.attach_databases(pipeline_database=store.at(JOB), cache_mode=CacheMode.LOG) diff --git a/tests/test_core/referential_tags/test_operators.py b/tests/test_core/referential_tags/test_operators.py index bf6fe5251..8e7701a8b 100644 --- a/tests/test_core/referential_tags/test_operators.py +++ b/tests/test_core/referential_tags/test_operators.py @@ -41,7 +41,6 @@ SemiJoin, ) from orcapod.core.referential_tags import ( - ReferenceMaterializationWarning, stream_references, ) from orcapod.errors import InputValidationError, SemiJoinIgnoredColumnWarning @@ -276,8 +275,7 @@ def test_group_by_prediction_matches_the_result() -> None: def test_selecting_away_the_referenced_data_leaves_a_plain_tag() -> None: source = _ref([{"x": 1, "y": 2}], ["x"], "S") - with pytest.warns(ReferenceMaterializationWarning): - selected = SelectDataColumns(["y"])(source) + selected = SelectDataColumns(["y"])(source) assert selected.keys() == (("x",), ("y",)) assert stream_references(selected) == () assert selected.as_table().to_pylist() == [{"x": 1, "y": 2}] @@ -286,26 +284,10 @@ def test_selecting_away_the_referenced_data_leaves_a_plain_tag() -> None: def test_dropping_the_referenced_data_leaves_a_plain_tag() -> None: source = _ref([{"x": 1, "y": 2}], ["x"], "S") - with pytest.warns(ReferenceMaterializationWarning): - dropped = DropDataColumns(["x"])(source) + dropped = DropDataColumns(["x"])(source) assert dropped.keys() == (("x",), ("y",)) -def test_silent_materialization_changes_nothing_but_the_warning( - monkeypatch: pytest.MonkeyPatch, -) -> None: - source = _ref([{"x": 1, "y": 2}], ["x"], "S") - with pytest.warns(ReferenceMaterializationWarning): - warned = SelectDataColumns(["y"])(source) - monkeypatch.setattr(referential_tags, "reference_materialization", "silent") - with warnings.catch_warnings(): - warnings.simplefilter("error") - silent = SelectDataColumns(["y"])(source) - assert silent.content_hash() == warned.content_hash() - assert silent.pipeline_hash() == warned.pipeline_hash() - assert silent.as_table().to_pylist() == warned.as_table().to_pylist() - - @pytest.mark.parametrize( "drop", [lambda: DropTagColumns(["x"]), lambda: SelectTagColumns(["k"])] ) @@ -350,10 +332,7 @@ def _xky(**kw: Any): def test_select_and_drop_columns_act_on_the_one_column_in_both_roles( make_op, keys, references ) -> None: - with warnings.catch_warnings(): - # Neither splits the roles, so nothing is materialized. - warnings.simplefilter("error", ReferenceMaterializationWarning) - result = make_op()(_xky()) + result = make_op()(_xky()) assert result.keys() == keys assert stream_references(result) == references table = result.as_table(columns={"source": True}) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index 35fa5865b..426aee03f 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -10,7 +10,6 @@ import datetime import decimal -import warnings from pathlib import Path from typing import Any @@ -20,7 +19,6 @@ import orcapod as op from orcapod.core.operators import TagBy from orcapod.core.referential_tags import ( - ReferenceMaterializationWarning, stored_references, stream_references, ) @@ -208,14 +206,8 @@ def test_a_join_with_a_data_only_column_of_that_name_is_refused() -> None: _ = _plain().tag_by("label").join(other) -def test_removing_the_data_role_afterwards_leaves_a_plain_tag_with_the_warning() -> ( - None -): - tagged = _plain().tag_by("label") - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - plain = tagged.drop_data_columns(["label"]) - assert any(w.category is ReferenceMaterializationWarning for w in caught) +def test_dropping_the_data_role_afterwards_leaves_a_plain_tag() -> None: + plain = _plain().tag_by("label").drop_data_columns(["label"]) assert plain.keys() == (("unit", "label"), ("trial", "amp")) From 8cd398fd0e22763cfa2b3a538e1e91f256430467 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:05:48 -0800 Subject: [PATCH 06/10] fix(tag_by): keep the input's column layout and data context (ITL-1175) TagBy moved the named columns in front of the other data columns, so the computed stream's data order differed from the predicted (input) order. The columns now stay where they are; the rows keep the input's data context, as SelectColumns does. Co-Authored-By: Claude Opus 5.5 --- docs/dev/specs/ITL-1175-tag-by.md | 6 ++++-- src/orcapod/core/operators/tag_by.py | 15 +++++++++------ .../test_core/referential_tags/test_tag_by.py | 18 ++++++++++++++++++ 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index fa79c055b..7ecd99c64 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -67,8 +67,10 @@ built, sync and async alike): name-preserving: same column names, same values; record ids unchanged). - Data: unchanged: same names, order, types, nullability, values; each column keeps its own source-info token. No token is created or merged. -- Table layout: the column is stored once and moves into the user tag group (row order: - user tags, system tags, data, source info), as a source places a referential tag. +- Table layout: the column is stored once and stays where it is, as a source leaves a + referential tag at its place in the table: the data order is the input's (the + predicted one). (The first version moved it into the user tag group, which put the + new tags first among the data columns, unlike the prediction: CodeRabbit on PR #420.) - Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with their data types, after the user tags and before the system tags; the data schema is unchanged. Verified against `static_process(...).output_schema(...)`. diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py index fcfeee00b..e138b2121 100644 --- a/src/orcapod/core/operators/tag_by.py +++ b/src/orcapod/core/operators/tag_by.py @@ -56,7 +56,7 @@ class TagBy(UnaryOperator): the data columns keep their names, order, types, values and source info, the system tags keep their names and values (rule 1, name-preserving), and no row is added, removed or reordered. The new tags follow the input's user - tags, in the order given, and the column is stored once, among the tags. + tags, in the order given, and the column is stored once, where it was. The operator creates no value and moves none: every output data column is the input data column of that name, and no tag becomes data. A data column that a @@ -192,7 +192,11 @@ def unary_output_schema( def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: """Re-wrap the input's rows with the named columns among the tags. - The same arrays, laid out with the tags first: no cast, no value changed. + The input's table as it is laid out: no column moved, no cast, no value + changed. The data columns are taken in table order, so leaving each named + column where it is keeps the input's data order (the predicted one); the + tag order is the ``tag_columns`` given. The rows keep the input's data + context. Args: stream: The input stream. @@ -204,10 +208,9 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: table = stream.as_table( columns={"source": True, "system_tags": True, "sort_by_tags": False} ) - new_tag_columns = [*tag_columns, *self.columns] - rest = [name for name in table.column_names if name not in new_tag_columns] return ArrowTableStream( - table.select([*new_tag_columns, *rest]), - tag_columns=new_tag_columns, + table, + tag_columns=[*tag_columns, *self.columns], reference_tags=[*stream_references(stream), *self.columns], + data_context=stream.data_context_key, ) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index 35fa5865b..b9bdd3ccb 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -150,6 +150,24 @@ def test_the_schema_is_predicted(all_info: bool) -> None: assert data_schema == source.output_schema()[1] +def test_the_data_order_is_the_inputs_and_the_predicted_one() -> None: + """The named columns stay where they are: tagging reorders no data column. + + The order of ``keys()`` and of the data schema of the computed stream + (``static_process``) equals the input's and the predicted one, also when the + names are given in another order than the data (CodeRabbit on PR #420). + """ + source = _referential() + tag_by = TagBy(["label", "trial"]) + computed = tag_by.static_process(source) + _, input_data = source.keys() + assert input_data == ("unit", "trial", "amp", "label") + assert computed.keys() == (("unit", "label", "trial"), input_data) + assert list(computed.output_schema()[1]) == list(input_data) + assert list(tag_by.output_schema(source)[1]) == list(input_data) + assert source.tag_by("label", "trial").keys() == computed.keys() + + def test_zero_rows_have_the_types_of_rows() -> None: source = _plain() tagged = source.tag_by("trial") From 5292a60e6861877f8a62b4c4c6f2646da687640d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:07:57 -0800 Subject: [PATCH 07/10] revert(referential_tags): keep ReferenceMaterializationWarning (ITL-1195) Reverts 3ab814eb: developer decision, November 17, 2026: "Let's keep the warning on select data columns for now." The warning, its setting and the silenced automatic select of pod(stream) are as on the slice, and so is the slice's changelog bullet. The docs now say plainly that dropping or selecting away the data of a referential tag leaves a plain tag with the warning, while dropping the tag or the column is silent; a role test checks which removal warns. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-1023.md | 3 +- changelog.d/ITL-1195.md | 7 +- docs/concepts/referential-tags.md | 24 +++++- src/orcapod/core/function_pod.py | 6 +- .../core/operators/column_selection.py | 10 ++- src/orcapod/core/referential_tags.py | 81 +++++++++++++++++-- .../referential_tags/test_derived_source.py | 6 +- .../referential_tags/test_function_pods.py | 16 ++-- .../test_operator_node_storage.py | 5 +- .../referential_tags/test_operators.py | 27 ++++++- .../test_core/referential_tags/test_roles.py | 29 ++++++- .../test_core/referential_tags/test_tag_by.py | 12 ++- 12 files changed, 194 insertions(+), 32 deletions(-) diff --git a/changelog.d/ITL-1023.md b/changelog.d/ITL-1023.md index 136e2ddc5..827ee28fe 100644 --- a/changelog.d/ITL-1023.md +++ b/changelog.d/ITL-1023.md @@ -25,7 +25,8 @@ - `MapTags` renames a referential tag in both roles; `tag_only=True` renames the tag only. `MapData` is a data-only move. - Removing the data column of a referential tag (`SelectDataColumns`, `DropDataColumns`) - removes its data role only and leaves a plain tag (ITL-1195). + leaves a plain tag and emits a `ReferenceMaterializationWarning` + (`orcapod.core.referential_tags.reference_materialization = "silent"` hides it). - A function output named like a tag of its input is an error when the pipeline is built. - Calling a function pod, `pod(stream)`, lets the input's referential tags fall through: the output rows carry the function's outputs and the row's own referenced columns, so diff --git a/changelog.d/ITL-1195.md b/changelog.d/ITL-1195.md index 910399859..2028df0f2 100644 --- a/changelog.d/ITL-1195.md +++ b/changelog.d/ITL-1195.md @@ -8,7 +8,6 @@ the column (`drop_columns`) removes it from both roles. `select_tag_columns`, `select_data_columns` and `select_columns` follow the same rule for the names they do not select. This was already the slice's behaviour; it is now the stated rule, with - tests by role. No second name (such as `untag`) is added. -- Removing the data role of a referential tag emits no warning: it is the defined meaning - of the call. `ReferenceMaterializationWarning`, the - `referential_tags.reference_materialization` setting and their helpers are removed. + tests by role. Removing the data role still emits `ReferenceMaterializationWarning`, + as on the slice; removing the tag role or the column is silent. No second name (such as + `untag`) is added. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index 0835e7829..d5a9616e4 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -83,7 +83,7 @@ column between the roles, and none of them changes, copies or creates a value: |---|---|---|---| | `tag_by("x")` | becomes a referential tag | refused: already a tag | refused: not a data column | | `drop_tag_columns(["x"])` | refused: not a tag | the tag role goes: `x` is plain data | removed | -| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag | refused: not a data column | +| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag, with a `ReferenceMaterializationWarning` | refused: not a data column | | `drop_columns(["x"])` | removed | removed from both roles | removed | The select operators follow the same rule by role: `select_tag_columns` removes the tag @@ -92,6 +92,14 @@ column it does not name, and `select_columns` keeps exactly the columns it names both roles. (The drop and select refusals are those of a missing name: `strict=False` ignores the name instead.) +For a referential tag `x`, in plain words: + +- Dropping or selecting away the **data** of `x` leaves a plain tag `x`, and issues a + `ReferenceMaterializationWarning`: the tag no longer refers to a data column, so its + values now live only in the tag (see below to turn the warning off). +- Dropping or selecting away the **tag** `x` leaves plain data `x`, silently. +- Dropping the **column** `x` removes it from both roles, silently. + ```python print(spikes.drop_tag_columns(["time"]).keys()) # (('unit',), ('unit', 'time', 'amp')) @@ -161,9 +169,9 @@ your own classes and logical types (`Path`, `UUID`, arrays) are refused. | `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. | -| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info. | +| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `ReferenceMaterializationWarning`. | | `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. | -| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split. | +| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. | | `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. | | `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | | Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the reference. | @@ -252,6 +260,16 @@ print(grouped.column("_source_v").to_pylist()) # [['sessions::row_0::v', 'sessions::row_1::v']] ``` +The materialization warning can be turned off; it never changes a result or a hash: + +```python +from orcapod.core import referential_tags + +referential_tags.reference_materialization = "silent" +# ... build the pipeline ... +referential_tags.reference_materialization = "warn" # the default +``` + ## Stored tables and saved jobs A stored table states which of its tags are referential. A function node stores each diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index cce9ca395..ec5fdba4b 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -16,6 +16,7 @@ from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.datagrams import Data from orcapod.core.referential_tags import ( + materialization_diagnostic_off, reference_names, stream_references, ) @@ -846,7 +847,10 @@ def __call__( self.label, dropped, ) - input_stream = SelectDataColumns(columns=kept)(input_stream) + # The select is Orcapod's own step: the referential tags it turns into + # plain tags are not reported (ITL-1023). + with materialization_diagnostic_off(): + input_stream = SelectDataColumns(columns=kept)(input_stream) if fall_through_pod is not None and references: return ReferenceFallThroughPod(fall_through_pod, references).process( input_stream, label=label diff --git a/src/orcapod/core/operators/column_selection.py b/src/orcapod/core/operators/column_selection.py index e5b5bd399..7a8b54495 100644 --- a/src/orcapod/core/operators/column_selection.py +++ b/src/orcapod/core/operators/column_selection.py @@ -6,7 +6,7 @@ from orcapod.core.operators.meta_columns import WithoutMetaWriter from orcapod.channels import ReadableChannel, WritableChannel from orcapod.core.operators.base import UnaryOperator -from orcapod.core.referential_tags import stream_references +from orcapod.core.referential_tags import stream_references, warn_materialized from orcapod.core.streams import ArrowTableStream from orcapod.errors import InputValidationError from orcapod.protocols.core_protocols import DataProtocol, StreamProtocol, TagProtocol @@ -265,6 +265,10 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: raise InputValidationError( f"Missing data columns: {missing_columns}. Make sure all specified columns to select are present or use strict=False to ignore missing columns" ) + warn_materialized( + self, + [c for c in stream_references(stream) if c not in self.columns], + ) require_a_data_column([c for c in data_columns if c in columns_to_select]) def unary_output_schema( @@ -545,6 +549,10 @@ def validate_unary_input(self, stream: StreamProtocol) -> None: raise InputValidationError( f"Missing data columns: {missing_columns}. Make sure all specified columns to drop are present or use strict=False to ignore missing columns" ) + warn_materialized( + self, + [c for c in stream_references(stream) if c in self.columns], + ) require_a_data_column([c for c in data_columns if c not in self.columns]) def unary_output_schema( diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 7f906e4aa..52bc7d465 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -9,9 +9,8 @@ schema field records it. This module holds what more than one operator needs: finding the references of a stream, -the namespace validator (sections 4 and 13.4), the stored marker, and the type rule of -``tag_by``. Removing a role is no diagnostic: dropping the tag or the data of a -referential tag removes that role only (developer, November 16, 2026, ITL-1195). +the namespace validator (sections 4 and 13.4), and the diagnostic emitted when a structural +operator turns a reference into a plain tag (section 7). The identities of referential streams are not frozen yet: on the integration branch of ITL-1023 they may still change. Plain (``tag_only_columns``) streams keep their v0.3 @@ -23,8 +22,12 @@ import datetime import types import typing -from collections.abc import Callable, Iterable, Mapping, Sequence -from typing import TYPE_CHECKING, Any +import warnings +import weakref +from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence +from contextlib import contextmanager +from contextvars import ContextVar +from typing import TYPE_CHECKING, Any, Literal from orcapod.errors import InputValidationError from orcapod.types import SourceInfoValue @@ -36,6 +39,45 @@ else: pa = LazyModule("pyarrow") +# Diagnostic emitted when a structural operator materializes a reference (section 7.4). +# ``"warn"`` emits a ``ReferenceMaterializationWarning``; ``"silent"`` emits nothing. The +# setting never changes a result, a schema or a hash. A module-level default for now. +reference_materialization: Literal["warn", "silent"] = "warn" + + +class ReferenceMaterializationWarning(UserWarning): + """A structural operator turned a referential tag into a plain tag (section 7).""" + + +# Set while Orcapod itself adds a structural step the user did not write (the select +# of a function pod call): its materializations are not reported. +_diagnostic_off: ContextVar[bool] = ContextVar( + "reference_materialization_off", default=False +) + + +# The operations that already reported, by id, holding the object so the id is not +# reused while the entry exists. +_reported: "weakref.WeakValueDictionary[int, object]" = weakref.WeakValueDictionary() + + +@contextmanager +def materialization_diagnostic_off() -> Iterator[None]: + """Build the steps in this block without the materialization diagnostic. + + Used for steps Orcapod adds by itself, such as the automatic select of a function + pod call, which would otherwise report on every call. The steps and their results + are exactly the same. + + Yields: + Nothing. + """ + token = _diagnostic_off.set(True) + try: + yield + finally: + _diagnostic_off.reset(token) + def reference_names( tag_names: Iterable[str], data_names: Iterable[str] @@ -101,6 +143,35 @@ def check_namespaces( ) +def warn_materialized(operation: object, names: Iterable[str]) -> None: + """Report that ``operation`` turned the referential tags ``names`` into plain tags. + + Its job is the diagnostic of section 7.4, so it takes the operation. It reports + once per operation object: a job validates its operators again when it builds and + runs its nodes, and that is not a new structural step. An operation built while + ``materialization_diagnostic_off`` is active (Orcapod's own step) never reports. + Emits nothing when ``names`` is empty or ``reference_materialization`` is + ``"silent"``. + + Args: + operation: The structural operation (an operator instance). + names: The tags that became plain. + """ + names = tuple(names) + if not names or id(operation) in _reported: + return + _reported[id(operation)] = operation + if reference_materialization == "silent" or _diagnostic_off.get(): + return + warnings.warn( + f"{type(operation).__name__}: the data column of referential tag(s) {list(names)} was removed; " + "they are now plain tags with the same values (set " + "orcapod.core.referential_tags.reference_materialization = 'silent' to hide this)", + ReferenceMaterializationWarning, + stacklevel=3, + ) + + # The marker of a referential tag in a stored table (developer, November 16, 2026, # ITL-1019): Arrow field metadata on the stored tag column, at the level of a logical # type's extension metadata. Only a referential tag carries it; a plain tag carries no diff --git a/tests/test_core/referential_tags/test_derived_source.py b/tests/test_core/referential_tags/test_derived_source.py index c67b1793c..1a7176ba8 100644 --- a/tests/test_core/referential_tags/test_derived_source.py +++ b/tests/test_core/referential_tags/test_derived_source.py @@ -9,6 +9,7 @@ from __future__ import annotations +import warnings from collections.abc import Callable from pathlib import Path @@ -19,6 +20,7 @@ from orcapod.core.operators import Join from orcapod.core.referential_tags import ( REFERENTIAL_TAG_METADATA_KEY, + ReferenceMaterializationWarning, mark_references, without_reference_marks, ) @@ -151,7 +153,9 @@ def test_as_source_of_an_operator_node_keeps_its_bindings( make_store: Callable[[Path], ArrowDatabaseProtocol], orchestrator: Orchestrator, ) -> None: - job = _operator_job(make_store(tmp_path), orchestrator) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", ReferenceMaterializationWarning) + job = _operator_job(make_store(tmp_path), orchestrator) node = job.nodes["joined"] source = node.as_source() assert source.keys() == node.keys() diff --git a/tests/test_core/referential_tags/test_function_pods.py b/tests/test_core/referential_tags/test_function_pods.py index f097b7cb3..d2bb91086 100644 --- a/tests/test_core/referential_tags/test_function_pods.py +++ b/tests/test_core/referential_tags/test_function_pods.py @@ -20,6 +20,7 @@ import orcapod as op from orcapod.core.operators import SelectDataColumns from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, stream_references, ) from orcapod.databases import InMemoryArrowDatabase @@ -112,7 +113,9 @@ def half(amp: float) -> float: ) # The plain pod (``t`` becomes a plain tag), joined back on ``t`` and ``row`` with # the input's referenced column: plain with referential gives the reference back. - computed = half.pod.process(with_row.select_data_columns(["amp"])) + with warnings.catch_warnings(): + warnings.simplefilter("ignore", ReferenceMaterializationWarning) + computed = half.pod.process(with_row.select_data_columns(["amp"])) joined = op.operators.Join()(computed, with_row.select_data_columns(["t"])) composed = sorted( (tag["t"], data["t"], data["half"]) for tag, data in joined.iter_data() @@ -158,18 +161,17 @@ def halve(amp: float) -> float: assert out.keys() == (("t",), ("t", "amp")) -def test_removing_a_data_role_emits_no_warning() -> None: - """Dropping the data of a referential tag is the defined meaning (ITL-1195).""" - +def test_the_call_reports_no_materialization() -> None: @op.function_pod(output_keys=["half"]) def half(amp: float) -> float: return amp / 2 - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") + with warnings.catch_warnings(): + warnings.simplefilter("error", ReferenceMaterializationWarning) half.pod(_source()) + # A select written by hand that drops the referenced column does report. + with pytest.warns(ReferenceMaterializationWarning): SelectDataColumns(["amp"])(_source()) - assert [str(w.message) for w in caught] == [] CALLS: list[float] = [] diff --git a/tests/test_core/referential_tags/test_operator_node_storage.py b/tests/test_core/referential_tags/test_operator_node_storage.py index 69f089a5e..0a1df68dd 100644 --- a/tests/test_core/referential_tags/test_operator_node_storage.py +++ b/tests/test_core/referential_tags/test_operator_node_storage.py @@ -11,6 +11,7 @@ from __future__ import annotations import dataclasses +import warnings from collections.abc import Callable from pathlib import Path from typing import cast @@ -24,6 +25,7 @@ from orcapod.core.operators import Join from orcapod.core.referential_tags import ( REFERENTIAL_TAG_METADATA_KEY, + ReferenceMaterializationWarning, stored_references, without_reference_marks, ) @@ -161,7 +163,8 @@ def test_a_reference_the_operator_materializes_is_not_marked() -> None: """The marker is the operator's output binding, never one that came from upstream.""" store = InMemoryArrowDatabase() job = PipelineJob(name=JOB, database=store) - with job: + with job, warnings.catch_warnings(): + warnings.simplefilter("ignore", ReferenceMaterializationWarning) _ = _source("a", "x", [1, 2]).select_data_columns(["x"], label="selected") node = job.nodes["selected"] node.attach_databases(pipeline_database=store.at(JOB), cache_mode=CacheMode.LOG) diff --git a/tests/test_core/referential_tags/test_operators.py b/tests/test_core/referential_tags/test_operators.py index 8e7701a8b..bf6fe5251 100644 --- a/tests/test_core/referential_tags/test_operators.py +++ b/tests/test_core/referential_tags/test_operators.py @@ -41,6 +41,7 @@ SemiJoin, ) from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, stream_references, ) from orcapod.errors import InputValidationError, SemiJoinIgnoredColumnWarning @@ -275,7 +276,8 @@ def test_group_by_prediction_matches_the_result() -> None: def test_selecting_away_the_referenced_data_leaves_a_plain_tag() -> None: source = _ref([{"x": 1, "y": 2}], ["x"], "S") - selected = SelectDataColumns(["y"])(source) + with pytest.warns(ReferenceMaterializationWarning): + selected = SelectDataColumns(["y"])(source) assert selected.keys() == (("x",), ("y",)) assert stream_references(selected) == () assert selected.as_table().to_pylist() == [{"x": 1, "y": 2}] @@ -284,10 +286,26 @@ def test_selecting_away_the_referenced_data_leaves_a_plain_tag() -> None: def test_dropping_the_referenced_data_leaves_a_plain_tag() -> None: source = _ref([{"x": 1, "y": 2}], ["x"], "S") - dropped = DropDataColumns(["x"])(source) + with pytest.warns(ReferenceMaterializationWarning): + dropped = DropDataColumns(["x"])(source) assert dropped.keys() == (("x",), ("y",)) +def test_silent_materialization_changes_nothing_but_the_warning( + monkeypatch: pytest.MonkeyPatch, +) -> None: + source = _ref([{"x": 1, "y": 2}], ["x"], "S") + with pytest.warns(ReferenceMaterializationWarning): + warned = SelectDataColumns(["y"])(source) + monkeypatch.setattr(referential_tags, "reference_materialization", "silent") + with warnings.catch_warnings(): + warnings.simplefilter("error") + silent = SelectDataColumns(["y"])(source) + assert silent.content_hash() == warned.content_hash() + assert silent.pipeline_hash() == warned.pipeline_hash() + assert silent.as_table().to_pylist() == warned.as_table().to_pylist() + + @pytest.mark.parametrize( "drop", [lambda: DropTagColumns(["x"]), lambda: SelectTagColumns(["k"])] ) @@ -332,7 +350,10 @@ def _xky(**kw: Any): def test_select_and_drop_columns_act_on_the_one_column_in_both_roles( make_op, keys, references ) -> None: - result = make_op()(_xky()) + with warnings.catch_warnings(): + # Neither splits the roles, so nothing is materialized. + warnings.simplefilter("error", ReferenceMaterializationWarning) + result = make_op()(_xky()) assert result.keys() == keys assert stream_references(result) == references table = result.as_table(columns={"source": True}) diff --git a/tests/test_core/referential_tags/test_roles.py b/tests/test_core/referential_tags/test_roles.py index d871261eb..dab4bd0e5 100644 --- a/tests/test_core/referential_tags/test_roles.py +++ b/tests/test_core/referential_tags/test_roles.py @@ -2,8 +2,8 @@ Developer decision, November 16, 2026: for a referential tag ``x`` (one column in two roles), dropping the tag removes the tag role only and leaves ``x`` plain data; dropping -the data removes the data role only and leaves ``x`` a plain tag; dropping the column -removes it from both roles. The select operators follow the same rule: selecting tags +the data removes the data role only and leaves ``x`` a plain tag, with a +``ReferenceMaterializationWarning``; dropping the column removes it from both roles. The select operators follow the same rule: selecting tags or data drops the others by role. Plain tags and plain data columns behave as before. Every operator is driven through its public call and read through the stream's public @@ -13,6 +13,7 @@ from __future__ import annotations from collections.abc import Callable +import warnings from typing import Any import pytest @@ -28,7 +29,10 @@ TagBy, ) from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod -from orcapod.core.referential_tags import stream_references +from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, + stream_references, +) from orcapod.errors import InputValidationError from orcapod.system_constants import constants @@ -102,6 +106,25 @@ def test_each_operator_acts_by_role( _ = label +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +def test_only_removing_the_data_role_warns( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # unused case field + references: Any, # pyright: ignore[reportExplicitAny] # unused case field +) -> None: + """Dropping or selecting away the data of ``x`` leaves a plain tag and warns + (``ReferenceMaterializationWarning``); removing the tag role or the whole column + is silent (developer, November 17, 2026: keep the warning for now).""" + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + _ = make_op()(_source()).keys() + warned = any(w.category is ReferenceMaterializationWarning for w in caught) + assert warned == (label in ("drop_data", "select_data")) + + def _by_k(k: int) -> dict[str, Any]: # pyright: ignore[reportExplicitAny] # row values return next(row for row in ROWS if row["k"] == k) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index 445ea488c..b9bdd3ccb 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -10,6 +10,7 @@ import datetime import decimal +import warnings from pathlib import Path from typing import Any @@ -19,6 +20,7 @@ import orcapod as op from orcapod.core.operators import TagBy from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, stored_references, stream_references, ) @@ -224,8 +226,14 @@ def test_a_join_with_a_data_only_column_of_that_name_is_refused() -> None: _ = _plain().tag_by("label").join(other) -def test_dropping_the_data_role_afterwards_leaves_a_plain_tag() -> None: - plain = _plain().tag_by("label").drop_data_columns(["label"]) +def test_removing_the_data_role_afterwards_leaves_a_plain_tag_with_the_warning() -> ( + None +): + tagged = _plain().tag_by("label") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + plain = tagged.drop_data_columns(["label"]) + assert any(w.category is ReferenceMaterializationWarning for w in caught) assert plain.keys() == (("unit", "label"), ("trial", "amp")) From 83ff606ff15d46b6e7224097c77056c8fe55086f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:17:29 -0800 Subject: [PATCH 08/10] fix(tag_by): keep each row's data context key; correct the spec on LOG (ITL-1175) Co-Authored-By: Claude Opus 5.5 --- docs/dev/specs/ITL-1175-tag-by.md | 6 +++--- src/orcapod/core/operators/tag_by.py | 12 +++++++++--- tests/test_core/referential_tags/test_tag_by.py | 11 +++++++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index 7ecd99c64..350d9fbfc 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -102,9 +102,9 @@ stays with `DerivedSource(tags_as_data=...)`. issue's out-of-scope "plain tag from a data column" reachable in two steps. Recommendation: allow (existing slice behaviour, warned). Conservative variant: none that is small; it would need tracking where a tag came from. -- Jobs: an operator node runs it in the default cache mode; `LOG` / `REPLAY` refuse - referential tags as for every operator (slice limitation, unchanged). A function node - downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and +- Jobs: an operator node runs it in every cache mode; in `LOG` mode it stores its output + with the new tag marked (operator storage of referential tags came with #407, ITL-1019; + tested). A function node downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and the class is registered in `pipeline/serialization.py`. ## Open choice: may a computed column key a join? diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py index e138b2121..6072be8d6 100644 --- a/src/orcapod/core/operators/tag_by.py +++ b/src/orcapod/core/operators/tag_by.py @@ -195,8 +195,8 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: The input's table as it is laid out: no column moved, no cast, no value changed. The data columns are taken in table order, so leaving each named column where it is keeps the input's data order (the predicted one); the - tag order is the ``tag_columns`` given. The rows keep the input's data - context. + tag order is the ``tag_columns`` given. The stream keeps the input's data + context, and each row its own data context key (the context column). Args: stream: The input stream. @@ -205,8 +205,14 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: The stream with the named columns as referential tags. """ tag_columns, _ = stream.keys() + # The context column too, so each row keeps its own data context key. table = stream.as_table( - columns={"source": True, "system_tags": True, "sort_by_tags": False} + columns={ + "source": True, + "system_tags": True, + "context": True, + "sort_by_tags": False, + } ) return ArrowTableStream( table, diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index b9bdd3ccb..9e5faa6c3 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -168,6 +168,17 @@ def test_the_data_order_is_the_inputs_and_the_predicted_one() -> None: assert source.tag_by("label", "trial").keys() == computed.keys() +def test_each_row_keeps_its_data_context_key() -> None: + """The context column passes through: no row takes the default key instead.""" + source = _plain() + before = source.as_table(columns={"context": True}) + after = TagBy(["trial"]).static_process(source).as_table(columns={"context": True}) + assert ( + after.column(constants.CONTEXT_KEY).to_pylist() + == before.column(constants.CONTEXT_KEY).to_pylist() + ) + + def test_zero_rows_have_the_types_of_rows() -> None: source = _plain() tagged = source.tag_by("trial") From 01c3b729ed37e01a82309a5bfb45f8c616c39d96 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:19:01 -0800 Subject: [PATCH 09/10] docs(design_issues): log ST5, the two tag orders of ArrowTableStream (ITL-1198) Co-Authored-By: Claude Opus 5.5 --- docs/dev/design_issues/ITL-1198-ST5.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 docs/dev/design_issues/ITL-1198-ST5.md diff --git a/docs/dev/design_issues/ITL-1198-ST5.md b/docs/dev/design_issues/ITL-1198-ST5.md new file mode 100644 index 000000000..fdbc6d763 --- /dev/null +++ b/docs/dev/design_issues/ITL-1198-ST5.md @@ -0,0 +1,23 @@ +## `src/orcapod/core/streams/arrow_table_stream.py` — tag order + +### ST5 — `keys()` lists tags in `tag_columns` order, `output_schema()` in physical order +**Status:** open +**Severity:** low +**Issue:** ITL-1198 + +`ArrowTableStream.keys()` gives the tag columns in the order of its `tag_columns` argument, +while `output_schema()` gives them in the table's physical column order (`_tag_schema` is the +table schema filtered to the tags). The data keys and the data schema both follow the physical +order and agree. A plain source shows the difference: over a table with columns `b, a, v` and +`tag_columns=["a", "b"]`, `keys()` lists `a, b` and `output_schema()` lists `b, a`. + +`TagBy` (ITL-1175) meets it: it keeps the input's layout, so the data order stays the input's +(the predicted one), and its predicted tag order is the requested one, which `keys()` reports. +For names given in another order than the table's, `output_schema()` of the computed stream +lists the new tags in table order. No layout can give both orders, since one physical order +decides both schemas. + +No hash depends on either order (`Schema` equality and the pipeline and content hashes are +order-insensitive), and column order is not part of a stream's identity (ITL-1112). The fix +is to decide one order for both methods in `ArrowTableStream`, without moving any stored +table's layout. From e43c60943e56d2d9a7401039ad6c14c494d65412 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 02:35:09 -0800 Subject: [PATCH 10/10] docs(design_issues): log AB1, the async barrier input's column order and data context (ITL-1205) Co-Authored-By: Claude Opus 5.5 --- docs/dev/design_issues/ITL-1205-AB1.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 docs/dev/design_issues/ITL-1205-AB1.md diff --git a/docs/dev/design_issues/ITL-1205-AB1.md b/docs/dev/design_issues/ITL-1205-AB1.md new file mode 100644 index 000000000..306669fa4 --- /dev/null +++ b/docs/dev/design_issues/ITL-1205-AB1.md @@ -0,0 +1,22 @@ +## `src/orcapod/core/operators/static_output_pod.py` — async barrier input + +### AB1 — The async barrier path rebuilds its input with data-tags first and in the default data context +**Status:** open +**Severity:** medium +**Issue:** ITL-1205 + +An operator without its own `async_execute` collects its input rows and rebuilds them as a +stream (`StaticOutputOperatorPod._materialize_input_stream`, through `_rows_to_table`), then +applies its sync rule. Two properties of that rebuild make the async result differ from the +sync one, for every operator on this path. + +1. Data order: `_rows_to_table` takes a data-tag (a tag that is also a data column, today a + referential tag) from the tags and lays it out with them, so it leads the data columns. + On the sync path a data-tag stays where the input table has it. For an input with data + `a, x, y` and `x` a data-tag, `PolarsFilter` (and `TagBy`) give data `a, x, y` on the sync + path and `x, a, y` on the async path. +2. Data context: the rebuilt input has neither the rows' data context nor a context column, + so it takes the default context. Only one data context is registered today, so this is + not observable yet. + +The fix must keep the row-built layout of stored tables (`_in_row_order`) unchanged.