From 8b73ba71af9ab84ccb2244f301857caee3aad533 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:36:05 -0800 Subject: [PATCH 1/6] feat(operators): TagBy / stream.tag_by makes data columns referential tags (ITL-1175) TagBy gives existing data columns the tag role, as referential tags: one column in both roles, values and source info unchanged. The new tags follow the input's tags in the order given; the order is part of the identity ("TagBy", [names]). System tags, record ids and data are unchanged. Refused, naming the column: not a data column, already a tag, a reserved name, a type that cannot be a tag (referential_tags.require_taggable_type; nullable points to drop_nulls). Async uses the base barrier path; cases added to the async/sync equivalence tests. Registered for saved jobs; docs, changelog fragment and dev spec. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-1175.md | 18 + docs/concepts/referential-tags.md | 46 ++ docs/dev/design_issues/ITL-1023-RT1.md | 2 +- docs/dev/specs/ITL-1175-tag-by.md | 116 +++++ src/orcapod/core/operators/__init__.py | 2 + src/orcapod/core/operators/base.py | 4 +- src/orcapod/core/operators/tag_by.py | 213 ++++++++++ src/orcapod/core/referential_tags.py | 54 +++ src/orcapod/core/streams/base.py | 44 +- src/orcapod/pipeline/serialization.py | 2 + .../protocols/core_protocols/streams.py | 12 + .../operators/test_async_sync_equivalence.py | 6 + .../test_core/referential_tags/test_tag_by.py | 400 ++++++++++++++++++ 13 files changed, 915 insertions(+), 4 deletions(-) create mode 100644 changelog.d/ITL-1175.md create mode 100644 docs/dev/specs/ITL-1175-tag-by.md create mode 100644 src/orcapod/core/operators/tag_by.py create mode 100644 tests/test_core/referential_tags/test_tag_by.py diff --git a/changelog.d/ITL-1175.md b/changelog.d/ITL-1175.md new file mode 100644 index 000000000..63d9086a0 --- /dev/null +++ b/changelog.d/ITL-1175.md @@ -0,0 +1,18 @@ +### Added + +- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as + referential tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each + named column becomes one column in both roles, with its values and source info + unchanged; the new tags follow the stream's tags in the order given, and that order is + part of the operator's identity. System tags, record ids and every data column are + unchanged. A column a function computed may become a tag and then key a join or a + group. Refused, naming the column: a name that is not a data column of the input, a + name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is + one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a + nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered + for saved jobs and pipelines. + +### Changed + +- The error for a name that is a tag in one join input and a data column in the other + also offers `tag_by` as a way out. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index 411c6efad..d1183047e 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -74,10 +74,56 @@ A function output named like a tag is an error when the pipeline is built. A fun that does not read a referenced column keeps its result identity: retagging a source does not make it compute again. +## Making a data column a tag: `tag_by` + +A source declares its tags, but a value a later step should join, group or sort by is +often computed on the way: a session label, a trial index, a phase. `tag_by` gives +existing data columns the tag role: each becomes a referential tag, one column that is +a tag and the data column it was, with its values and source info unchanged. + +```python +@op.function_pod(output_keys=["phase"]) +def phase_of(time: int) -> str: + return "early" if time < 15 else "late" + + +phased = phase_of.pod(spikes).tag_by("phase") +print(phased.keys()) +# (('unit', 'time', 'phase'), ('unit', 'time', 'phase')) + +by_phase = phased.group_by(["phase"]) +print(by_phase.keys()) +# (('phase',), ('unit', 'time', 'phase')) +``` + +- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's + tags in the order given, and that order is part of the operator (`TagBy`): the names + are a list, never a set. +- Nothing else changes: no row is added, removed or reordered, the data columns keep + their names, types and values, every data column keeps its own source-info token, + and the system tags keep their names and values. +- No data element comes from a tag value: the operator adds a tag role to a column + that is already data, and creates or moves no value. Offering a tag's value as data + stays with `DerivedSource(tags_as_data=...)`. +- A computed column may become a tag, and then key a join or a group, as `phase` does + above. + +`tag_by` refuses, naming the column: a name that is not a data column of the stream; a +name that is already a tag (plain or referential); a reserved name (the `__` prefix, a +system column, a name holding `::`); and a column whose type cannot be a tag. A tag is +one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`. +A nullable column (`T | None`) is refused: drop the rows with a null first, +`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, +your own classes and logical types (`Path`, `UUID`, arrays) are refused. + +`tag_by` is one of three moves between the roles of a column. The other two, which turn +a referential tag into plain data or into a plain tag, are the subject of ITL-1195. + ## Operators | Operator | Referential tag `x` | |---|---| +| `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. | | `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. | diff --git a/docs/dev/design_issues/ITL-1023-RT1.md b/docs/dev/design_issues/ITL-1023-RT1.md index b8750dc62..0fcb4b1ad 100644 --- a/docs/dev/design_issues/ITL-1023-RT1.md +++ b/docs/dev/design_issues/ITL-1023-RT1.md @@ -5,4 +5,4 @@ **Severity:** medium **Issue:** ITL-1023 -The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. +The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` and `TagBy` (`tag_by`, ITL-1175), which gives a data column the tag role. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md new file mode 100644 index 000000000..057426806 --- /dev/null +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -0,0 +1,116 @@ +# ITL-1175: mark existing data columns as referential tags (`tag_by`) + +Status: approved by the coordinator on 2026-11-16, with two developer decisions (the +name `tag_by` / `TagBy`; a computed column may become a tag). Stacked on the referential +tags slice (ITL-1023, PR #342). The sections below are the design as sent; "Decisions" +records what changed. + +## Decisions + +- Name: `tag_by` (developer); the class follows the operator naming: `TagBy`. +- A computed column may become a tag and key a join or a group (developer); no lineage + check. +- Type rule: the slice has no type rule for a referential tag declared at a source (a + source accepts any type), so `TagBy` uses its own rule, in one function + (`referential_tags.require_taggable_type`) that the source path could adopt: `int`, + `float`, `bool`, `str`, `bytes`, `datetime`, `date`. `Decimal`, time of day and + duration are not in it: the type converter reads them as no Python type (`Any`), and + `Any` is refused. The source path is not changed here; the inconsistency is reported + in the PR. +- Removing the data role afterwards is the subject of ITL-1195 (explicit operators); the + docs leave a placeholder for it instead of describing the implicit path. + +## Public API + +- Operator `orcapod.core.operators.TagBy(columns)`; stream method + `stream.tag_by(*columns, label=None)`. `s.tag_by("x", "y")` equals + `s.tag_by(["x", "y"])` (the convention of `select_columns`). +- The name is a choice for the developer. It lives in one place per surface: the class + name, the stream method, the serialization registry entry, the docs. Not + `reference_tags`: `ArrowTableStream.reference_tags` is already a property. + Alternatives: `as_tags` / `AsTags`, `mark_tags` / `MarkTags`. +- No `strict` flag: a name that cannot be marked is always an error. + +## Order rule and identity + +- The names are ordered: the output's user tags are the input's user tags, in their + order, then the named columns in the order given. So the order is part of the + operator, and its identity is `("TagBy", [names in the order given])`; its pipeline + identity is the same. A `set` / `frozenset` is refused (`TypeError`, unordered, as for + `record_id_columns`), as is an empty list and a repeated name (`ValueError`). + +## Refusals + +At construction (`ValueError` / `TypeError`, naming the column): no name; a non-string; +a repeated name; a set; the meta prefix (`require_names_not_meta`); a system column name +(`require_names_not_system`); a name holding `::` (`require_names_without_separator`). + +At validation (`InputValidationError`, naming the column, raised when the pipeline is +built, sync and async alike): + +- not a data column of the input (lists the input's data columns); +- already a tag: a referential tag ("already a referential tag"); a plain tag is never a + data column, so it is reported as a tag, not a data column; +- a type that cannot be a tag. Recommended rule: the column's Python type must be one of + `int`, `float`, `bool`, `str`, `bytes`, `datetime`, `date` (see Decisions). Refused: `T | None` (the error says to apply + `drop_nulls("x")` first, which narrows it to `T`), lists, sets, dicts, tuples, structs, + user classes and every logical type (`Path`, `UUID`, arrays...). Sources and function + outputs are non-nullable by default, so this costs nothing in the common case. + Conservative variant: also refuse `float` (equality on floats as a key). + +## What the output is + +- Tags: input user tags, then the new names; system tags unchanged (rule 1, + name-preserving: same column names, same values; record ids unchanged). +- Data: unchanged: same names, order, types, nullability, values; each column keeps its + own source-info token. No token is created or merged. +- Table layout: the column is stored once and moves into the user tag group (row order: + user tags, system tags, data, source info), as a source places a referential tag. +- Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with + their data types, after the user tags and before the system tags; the data schema is + unchanged. Verified against `static_process(...).output_schema(...)`. +- Sync: the input table re-wrapped as an `ArrowTableStream` with the extra tags in + `tag_columns` and `reference_tags`: no Polars round trip, no cast. +- Async: no own `async_execute`: the base barrier path collects the rows and applies the + sync rule, so the rows are identical by construction. Added to + `test_async_sync_equivalence.py`. A streaming version can come later. + +## Why "no data element comes from a tag value" holds + +The operator adds a tag role to a data column; it creates no value and moves none. Every +output data column is the input data column of that name, with the same values and +source info, and no input tag becomes data. The column stays one physical column, as a +source's referential tag is. The direction the rule forbids (a tag value offered as data) +stays with `DerivedSource(tags_as_data=...)`. + +## Downstream + +- `Join` with a stream where `x` is a plain tag or a referential tag: the slice's rules + (one referential tag, one data column, merged `join_v1` token when referential on + several inputs). With a stream where `x` is data only: the existing tag/data error (its + remedy text gains "or mark it with `tag_by`"). +- `GroupBy(["x"])`, `Batch`, `MapTags`, select/drop: the slice's rules for any + referential tag. +- Removing the data role afterwards (`tag_by("x").drop_data_columns("x")`) leaves a plain + tag holding the data value, with the `ReferenceMaterializationWarning`, as for any + referential tag. That is the data-to-tag direction (the rule holds), but it makes the + issue's out-of-scope "plain tag from a data column" reachable in two steps. + Recommendation: allow (existing slice behaviour, warned). Conservative variant: none + that is small; it would need tracking where a tag came from. +- Jobs: an operator node runs it in the default cache mode; `LOG` / `REPLAY` refuse + referential tags as for every operator (slice limitation, unchanged). A function node + downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and + the class is registered in `pipeline/serialization.py`. + +## Open choice: may a computed column key a join? + +Recommendation: yes. It is the issue's own typical use (a function computes a session +label that later steps join, group or sort by), and nothing in hashing or record +identity depends on where a tag's value came from. Conservative variant: `TagBy` refuses +a column a function pod produced. That needs a lineage walk upstream (through renames and +joins) in one helper called from validation; it is not small, and I recommend against it. + +## Not in scope + +Golden pins (identities provisional in 0.4.0rc1); a plain-tag variant; the reverse +direction; `MergeJoin` (ITL-1194). diff --git a/src/orcapod/core/operators/__init__.py b/src/orcapod/core/operators/__init__.py index 032cd0a60..838f6eb7c 100644 --- a/src/orcapod/core/operators/__init__.py +++ b/src/orcapod/core/operators/__init__.py @@ -16,6 +16,7 @@ from .merge_join import MergeJoin from .pick import Pick from .semijoin import SemiJoin +from .tag_by import TagBy __all__ = [ "Join", @@ -36,4 +37,5 @@ "DropNulls", "Pick", "Index", + "TagBy", ] diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index eb65ae1ce..bd6851502 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -101,8 +101,8 @@ def _tag_data_error(name: str, tag_side: str, data_side: str) -> InputValidation return InputValidationError( f"column {name!r} is a tag in the {tag_side} input and a data column in the " + f"{data_side}, and a name cannot be both. Rename it in one input with " - + "map_tags or map_data, or declare it a tag at the source that has it as " - + "a data column." + + "map_tags or map_data, or make it a tag of the input that has it as a data column (declare it " + + "a tag at its source, or mark it with tag_by)." ) diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py new file mode 100644 index 000000000..fcfeee00b --- /dev/null +++ b/src/orcapod/core/operators/tag_by.py @@ -0,0 +1,213 @@ +"""TagBy operator: give existing data columns the tag role, as referential tags (ITL-1175).""" + +from __future__ import annotations + +from collections.abc import Sequence, Set +from typing import Any, ClassVar + +from typing_extensions import override + +from orcapod.core.operators.base import UnaryOperator +from orcapod.core.referential_tags import require_taggable_type, stream_references +from orcapod.core.streams import ArrowTableStream +from orcapod.errors import InputValidationError +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.system_constants import constants +from orcapod.types import ColumnConfig, Schema +from orcapod.utils import schema_utils + + +def _tag_by_names(columns: str | Sequence[str]) -> tuple[str, ...]: + """Return the names ``TagBy`` marks, in the order given. + + Args: + columns: One name, or a sequence of names. + + Returns: + The names, in the order given. + + Raises: + TypeError: If ``columns`` is a set (unordered: the order of the names is the + order of the new tags) or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + """ + if isinstance(columns, str): + return (columns,) + if isinstance(columns, Set): + raise TypeError( + "TagBy: give the column names as a list or a tuple, not a set: their " + + "order is the order of the new tags" + ) + names = tuple(columns) + if not names: + raise ValueError("TagBy: expected at least one column name") + if not_strings := [name for name in names if not isinstance(name, str)]: + raise TypeError(f"TagBy: column names must be strings, got {not_strings!r}") + if repeated := sorted({name for name in names if names.count(name) > 1}): + raise ValueError(f"TagBy: columns {repeated} are named more than once") + return names + + +class TagBy(UnaryOperator): + """Give existing data columns of a stream the tag role, as referential tags. + + Each named data column becomes a referential tag (ITL-1023): one column in + both roles, a tag and the data column it already was. Nothing else changes: + the data columns keep their names, order, types, values and source info, + the system tags keep their names and values (rule 1, name-preserving), and + no row is added, removed or reordered. The new tags follow the input's user + tags, in the order given, and the column is stored once, among the tags. + + The operator creates no value and moves none: every output data column is the + input data column of that name, and no tag becomes data. A data column that a + function computed may become a tag (developer, November 16, 2026), and then + key a join or a group. + + Args: + columns: One data column name, or a sequence of them. Their order is the + order of the new tags, and part of the operator's identity. + + Raises: + TypeError: If ``columns`` is a set or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + InputValidationError: If a name is reserved: the meta prefix, a system + column name, or one holding ``::``. + """ + + # Referential tags (ITL-1023): handled; this operator makes them. + REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = None + + def __init__(self, columns: str | Sequence[str], **kwargs: Any) -> None: + names = _tag_by_names(columns) + schema_utils.require_names_not_meta(names) + schema_utils.require_names_not_system(names) + schema_utils.require_names_without_separator(names) + self.columns: tuple[str, ...] = names + super().__init__(**kwargs) + + @override + def identity_structure(self) -> Any: + """Return the operator's identity: its class name and the names, in order. + + Returns: + ``("TagBy", [names in the order given])``. + """ + return (self.__class__.__name__, list(self.columns)) + + @override + def to_config(self) -> dict[str, Any]: + """Serialize this operator to a config dict. + + Returns: + A dict with ``class_name``, ``module_path`` and ``config`` keys, where + ``config`` holds ``columns``, the names in order. + """ + config = super().to_config() + config["config"] = {"columns": list(self.columns)} + return config + + @override + def __repr__(self) -> str: + return f"TagBy({list(self.columns)!r})" + + @override + def validate_unary_input(self, stream: StreamProtocol) -> None: + """Check that every name is a data column of ``stream`` that can be a tag. + + Args: + stream: The input stream. + + Raises: + InputValidationError: Naming the column, if it is already a tag + (referential or plain), is not a data column of ``stream``, or has a + type that cannot be a tag (``referential_tags.require_taggable_type``). + """ + tag_columns, data_columns = stream.keys() + references = stream_references(stream) + _, data_schema = stream.output_schema() + for name in self.columns: + if name in references: + raise InputValidationError( + f"TagBy: column {name!r} is already a referential tag" + ) + if name in tag_columns: + raise InputValidationError( + f"TagBy: column {name!r} is a plain tag, not a data column: only " + + "a data column can be given the tag role" + ) + user_data = [ + c for c in data_columns if not c.startswith(constants.META_PREFIX) + ] + if name not in user_data: + raise InputValidationError( + f"TagBy: column {name!r} is not a data column of the input; its " + + f"data columns are {user_data}" + ) + try: + require_taggable_type(data_schema[name]) + except InputValidationError as exc: + raise InputValidationError( + f"TagBy: column {name!r} cannot be a tag: {exc}" + ) from exc + + @override + def unary_output_schema( + self, + stream: StreamProtocol, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> tuple[Schema, Schema]: + """Return the input schemas with the named columns added to the tags. + + The tag schema is the input's user tags, then the named columns with their + data types, then the system tags (when ``columns`` includes them); the data + schema is the input's. + + Args: + stream: The input stream. + columns: Column inclusion config passed through to ``output_schema``. + all_info: Whether to include all info columns. + + Returns: + A ``(tag_schema, data_schema)`` tuple. + """ + tag_schema, data_schema = stream.output_schema( + columns=columns, all_info=all_info + ) + user_tags = { + name: python_type + for name, python_type in tag_schema.items() + if not name.startswith(constants.SYSTEM_TAG_PREFIX) + } + system_tags = { + name: python_type + for name, python_type in tag_schema.items() + if name.startswith(constants.SYSTEM_TAG_PREFIX) + } + new_tags = {name: data_schema[name] for name in self.columns} + return Schema({**user_tags, **new_tags, **system_tags}), data_schema + + @override + def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: + """Re-wrap the input's rows with the named columns among the tags. + + The same arrays, laid out with the tags first: no cast, no value changed. + + Args: + stream: The input stream. + + Returns: + The stream with the named columns as referential tags. + """ + tag_columns, _ = stream.keys() + table = stream.as_table( + columns={"source": True, "system_tags": True, "sort_by_tags": False} + ) + new_tag_columns = [*tag_columns, *self.columns] + rest = [name for name in table.column_names if name not in new_tag_columns] + return ArrowTableStream( + table.select([*new_tag_columns, *rest]), + tag_columns=new_tag_columns, + reference_tags=[*stream_references(stream), *self.columns], + ) diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 7c508d813..09a00d4d8 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -19,6 +19,8 @@ from __future__ import annotations +import datetime +import types import typing import warnings import weakref @@ -313,6 +315,58 @@ def without_reference_marks(table: "pa.Table") -> "pa.Table": return mark_references(table, ()) +# The Python types a column may have to become a referential tag by ``tag_by`` +# (ITL-1175): one value per row, of a type with plain equality and order. A column +# whose Arrow type the type converter reads as no Python type (``Any``: decimal, time +# of day, duration) is refused. Sources do not apply this rule yet: a source accepts +# any type for a declared tag. +TAGGABLE_TYPES: tuple[type, ...] = ( + int, + float, + bool, + str, + bytes, + datetime.datetime, + datetime.date, +) + + +def require_taggable_type(python_type: object) -> None: + """Refuse a column type that cannot become a referential tag (ITL-1175). + + The rule ``tag_by`` applies: the type must be one of ``TAGGABLE_TYPES``. A + nullable type (``T | None``) is refused, with the way out: ``drop_nulls`` narrows + the column to ``T``. Lists, sets, dicts, tuples, structs, user classes and every + logical type (``Path``, ``UUID``, arrays) are refused, and so is a column the type + converter gives no Python type (``Any``). + + Args: + python_type: The column's Python type, as ``output_schema()`` gives it. + + Raises: + InputValidationError: If the type cannot be a tag. The caller names the + column. + """ + if python_type in TAGGABLE_TYPES: + return + members = typing.get_args(python_type) + if typing.get_origin(python_type) in (typing.Union, types.UnionType) and ( + type(None) in members + ): + rest = [member for member in members if member is not type(None)] + if len(rest) == 1 and rest[0] in TAGGABLE_TYPES: + raise InputValidationError( + f"its type {python_type} is nullable, and a tag needs a value in " + + "every row: drop the rows with a null first with drop_nulls, which " + + f"narrows the column to {rest[0].__name__}" + ) + names = ", ".join(t.__name__ for t in TAGGABLE_TYPES) + raise InputValidationError( + f"its type {python_type} cannot be a tag: a tag is one value of one of " + + f"these types: {names}" + ) + + def not_supported_yet(operation: str, issue: str) -> NotImplementedError: """Return the error for an operation that does not handle referential tags yet. diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index a0016940d..a2f695b69 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -2,7 +2,14 @@ import logging from abc import abstractmethod -from collections.abc import AsyncIterator, Callable, Collection, Iterator, Mapping +from collections.abc import ( + AsyncIterator, + Callable, + Collection, + Iterator, + Mapping, + Sequence, +) from contextvars import ContextVar from datetime import datetime from typing import TYPE_CHECKING, Any, cast @@ -565,6 +572,41 @@ def drop_nulls( return DropNulls(columns)(self, label=label) + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> StreamBase: + """Give the named data columns the tag role, as referential tags. + + ``s.tag_by("x", "y")`` equals ``s.tag_by(["x", "y"])``. Each named data + column becomes a referential tag: one column, both a tag and the data column + it was, with its values and source info unchanged. The new tags follow this + stream's tags, in the order given. See ``orcapod.core.operators.TagBy``. + + Args: + *columns: One or more data column names, or one list or tuple of names + as the only positional argument (not a set: the order matters). + label: Optional node label for the pipeline graph. + + Returns: + The stream with the named columns as referential tags. + + Raises: + TypeError: When no name is given, the names are a set, or a name is not + a string. + ValueError: When a name is repeated. + InputValidationError: When a name is reserved, is not a data column of + this stream, is already a tag, or has a type that cannot be a tag. + """ + from orcapod.core.operators import TagBy + + if len(columns) == 1 and not isinstance(columns[0], str): + names: str | Sequence[str] = columns[0] + else: + names = cast("tuple[str, ...]", columns) + return TagBy(names)(self, label=label) + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/src/orcapod/pipeline/serialization.py b/src/orcapod/pipeline/serialization.py index ee71b74c3..af73be040 100644 --- a/src/orcapod/pipeline/serialization.py +++ b/src/orcapod/pipeline/serialization.py @@ -205,6 +205,7 @@ def _build_operator_registry() -> dict[str, type]: SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) return { @@ -224,6 +225,7 @@ def _build_operator_registry() -> dict[str, type]: "MapColumns": MapColumns, "PolarsFilter": PolarsFilter, "DropNulls": DropNulls, + "TagBy": TagBy, } diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index a329b3e16..67292e92c 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -421,6 +421,18 @@ def drop_nulls( """ ... + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> "StreamProtocol": + """ + Give the named data columns the tag role, as referential tags: each stays + one column, a tag and the data column it was, with its values and source + info unchanged. The new tags follow the stream's tags, in the order given. + """ + ... + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 60fbbbb75..12a8ab44b 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -47,6 +47,7 @@ SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod from orcapod.core.sources import ArrowTableSource @@ -585,6 +586,11 @@ class Case: lambda: MapData({"ta": "tb"}), lambda: (source_a(referential=True),), ), + # TagBy gives data columns the tag role, as referential tags (ITL-1175). + Case("tag_by", lambda: TagBy(["i8", "bn"]), lambda: (source_a(),)), + Case("tag_by_reversed", lambda: TagBy(["bn", "i8"]), lambda: (source_a(),)), + Case("tag_by_b", lambda: TagBy(["u8", "ls"]), lambda: (source_b(),)), + Case("ref_tag_by", lambda: TagBy("i8"), lambda: (source_a(referential=True),)), ] CASE = pytest.mark.parametrize("case", CASES, ids=[c.label for c in CASES]) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py new file mode 100644 index 000000000..a2116c3b5 --- /dev/null +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -0,0 +1,400 @@ +"""``TagBy`` / ``stream.tag_by``: data columns become referential tags (ITL-1175). + +A named data column becomes a referential tag: one column, a tag and the data column it +was, with its values and source info unchanged. The new tags follow the input's tags in +the order given. Every operator is driven through its public call and read through the +stream's public methods; a schema prediction is checked against ``static_process``. +""" + +from __future__ import annotations + +import datetime +import decimal +import warnings +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.core.operators import TagBy +from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, + stored_references, + stream_references, +) +from orcapod.databases import InMemoryArrowDatabase +from orcapod.errors import InputValidationError +from orcapod.pipeline.serialization import resolve_operator_from_config +from orcapod.pipeline import ( + AsyncPipelineOrchestrator, + PipelineJob, + SyncPipelineOrchestrator, +) +from orcapod.system_constants import constants + +ORCHESTRATORS = pytest.mark.parametrize( + "orchestrator", + [SyncPipelineOrchestrator, AsyncPipelineOrchestrator], + ids=["sync", "async"], +) + +ROWS = [ + {"unit": 1, "trial": 10, "amp": 0.5, "label": "a"}, + {"unit": 2, "trial": 20, "amp": 1.5, "label": "b"}, +] + + +def _plain() -> op.sources.DictSource: + """``unit`` a plain tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource( + ROWS, tag_columns=["unit"], tag_only_columns=True, source_id="spikes" + ) + + +def _referential() -> op.sources.DictSource: + """``unit`` a referential tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource(ROWS, tag_columns=["unit"], source_id="spikes") + + +@op.function_pod(output_keys=["session"]) +def session_of(trial: int) -> str: + """A computed value later steps key on.""" + return f"s{trial // 20}" + + +def _sorted_rows(table: pa.Table) -> list[dict[str, Any]]: + return sorted(table.to_pylist(), key=lambda row: repr(sorted(row.items()))) + + +# ── what the output is ────────────────────────────────────────────────────── + + +def test_named_data_columns_become_referential_tags_in_the_order_given() -> None: + tagged = _plain().tag_by("label", "trial") + assert tagged.keys() == ( + ("unit", "label", "trial"), + ("trial", "amp", "label"), + ) + assert stream_references(tagged) == ("label", "trial") + for tag, data in tagged.iter_data(): + assert tag["label"] == data["label"] and tag["trial"] == data["trial"] + + +def test_the_column_is_stored_once_among_the_tags() -> None: + table = _plain().tag_by("trial").as_table() + assert table.column_names == ["unit", "trial", "amp", "label"] + + +def test_values_system_tags_and_source_info_are_unchanged() -> None: + source = _plain() + tagged = source.tag_by("trial") + before = source.as_table(all_info=True) + after = tagged.as_table(all_info=True) + unchanged = [ + name + for name in before.column_names + if name.startswith((constants.SYSTEM_TAG_PREFIX, constants.SOURCE_PREFIX)) + or name in ("unit", "trial", "amp", "label") + ] + assert set(unchanged) <= set(after.column_names) + assert _sorted_rows(after.select(unchanged)) == _sorted_rows( + before.select(unchanged) + ) + # The data's types and nullability are the input's. + for name in ("trial", "amp", "label"): + assert after.schema.field(name) == before.schema.field(name) + + +def test_an_existing_reference_is_kept() -> None: + tagged = _referential().tag_by("trial") + assert stream_references(tagged) == ("unit", "trial") + assert tagged.keys() == (("unit", "trial"), ("unit", "trial", "amp", "label")) + + +def test_names_as_strings_or_one_sequence_are_one_operator() -> None: + source = _plain() + by_strings = source.tag_by("trial", "label") + by_list = source.tag_by(["trial", "label"]) + by_tuple = source.tag_by(("trial", "label")) + assert ( + by_strings.content_hash() == by_list.content_hash() == by_tuple.content_hash() + ) + assert by_strings.pipeline_hash() == by_list.pipeline_hash() + + +def test_the_order_of_the_names_is_part_of_the_identity() -> None: + source = _plain() + forward = source.tag_by("trial", "label") + backward = source.tag_by("label", "trial") + assert forward.keys()[0] == ("unit", "trial", "label") + assert backward.keys()[0] == ("unit", "label", "trial") + assert forward.content_hash() != backward.content_hash() + assert forward.pipeline_hash() != backward.pipeline_hash() + assert TagBy(["trial", "label"]).identity_structure() == ( + "TagBy", + ["trial", "label"], + ) + + +@pytest.mark.parametrize("all_info", [False, True]) +def test_the_schema_is_predicted(all_info: bool) -> None: + source = _referential() + tag_by = TagBy(["label", "trial"]) + predicted = tag_by.output_schema(source, all_info=all_info) + assert predicted == tag_by.static_process(source).output_schema(all_info=all_info) + tag_schema, data_schema = predicted + assert list(tag_schema)[:3] == ["unit", "label", "trial"] + assert data_schema == source.output_schema()[1] + + +def test_zero_rows_have_the_types_of_rows() -> None: + source = _plain() + tagged = source.tag_by("trial") + empty = source.polars_filter(unit=99).tag_by("trial") + assert empty.as_table().schema == tagged.as_table().schema + assert empty.as_table().num_rows == 0 + + +def test_the_config_round_trips_and_is_registered_for_loading() -> None: + tag_by = TagBy(["label", "trial"]) + config = tag_by.to_config() + assert config["class_name"] == "TagBy" + assert config["config"] == {"columns": ["label", "trial"]} + for loaded in (TagBy.from_config(config), resolve_operator_from_config(config)): + assert isinstance(loaded, TagBy) + assert loaded.columns == ("label", "trial") + assert loaded.content_hash() == tag_by.content_hash() + assert loaded.pipeline_hash() == tag_by.pipeline_hash() + + +# ── after a function pod: a computed column keys a join and a group ───────── + + +def test_a_computed_column_becomes_a_tag_and_keys_a_join() -> None: + computed = TagBy("session")(session_of.pod(_referential())) + assert computed.keys() == (("unit", "session"), ("unit", "session")) + sessions = op.sources.DictSource( + [{"session": "s0", "room": "A"}, {"session": "s1", "room": "B"}], + tag_columns=["session"], + source_id="sessions", + ) + joined = computed.join(sessions) + assert set(stream_references(joined)) == {"unit", "session"} + rows = sorted(joined.as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["room"]) for r in rows] == [ + (1, "s0", "A"), + (2, "s1", "B"), + ] + + +def test_a_computed_column_becomes_a_tag_and_keys_a_group() -> None: + grouped = TagBy("session")(session_of.pod(_referential())).group_by(["session"]) + assert grouped.keys() == (("session",), ("unit", "session")) + rows = sorted(grouped.as_table().to_pylist(), key=lambda r: r["session"]) + assert [(r["session"], r["unit"]) for r in rows] == [("s0", [1]), ("s1", [2])] + + +def test_a_join_with_a_data_only_column_of_that_name_is_refused() -> None: + other = op.sources.DictSource( + [{"k": 1, "label": "a"}], + tag_columns=["k"], + tag_only_columns=True, + source_id="o", + ) + with pytest.raises(InputValidationError, match="tag_by"): + _ = _plain().tag_by("label").join(other) + + +def test_removing_the_data_role_afterwards_leaves_a_plain_tag_with_the_warning() -> ( + None +): + tagged = _plain().tag_by("label") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + plain = tagged.drop_data_columns(["label"]) + assert any(w.category is ReferenceMaterializationWarning for w in caught) + assert plain.keys() == (("unit", "label"), ("trial", "amp")) + + +def test_a_nullable_column_is_tagged_after_drop_nulls() -> None: + table = pa.table( + {"unit": pa.array([1, 2]), "label": pa.array(["a", None], pa.large_string())} + ) + source = op.sources.ArrowTableSource( + table, tag_columns=["unit"], tag_only_columns=True, source_id="n" + ) + with pytest.raises(InputValidationError, match="drop_nulls"): + _ = source.tag_by("label") + tagged = source.drop_nulls("label").tag_by("label") + assert tagged.as_table().column("label").to_pylist() == ["a"] + + +# ── refusals ──────────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize( + ("columns", "error"), + [ + ({"trial", "label"}, TypeError), + (frozenset({"trial"}), TypeError), + ([], ValueError), + (["trial", "trial"], ValueError), + ([1], TypeError), + (["__meta"], InputValidationError), + ([f"{constants.SYSTEM_TAG_PREFIX}x"], InputValidationError), + ([f"{constants.SOURCE_PREFIX}x"], InputValidationError), + (["a::b"], InputValidationError), + ], + ids=[ + "set", + "frozenset", + "empty", + "repeated", + "not_a_string", + "meta", + "system_tag", + "source_info", + "separator", + ], +) +def test_bad_names_are_refused_at_construction(columns: Any, error: type) -> None: + with pytest.raises(error): + _ = TagBy(columns) + + +def test_the_stream_method_refuses_no_name_and_a_set() -> None: + with pytest.raises(TypeError): + _ = _plain().tag_by({"trial"}) # pyright: ignore[reportArgumentType] # the refusal is the test + with pytest.raises(ValueError, match="at least one"): + _ = _plain().tag_by([]) + + +@pytest.mark.parametrize( + ("name", "match"), + [ + ("missing", "'missing' is not a data column"), + ("unit", "'unit' is a plain tag"), + ], +) +def test_a_name_that_is_not_a_data_column_is_refused(name: str, match: str) -> None: + with pytest.raises(InputValidationError, match=match): + _ = _plain().tag_by(name) + + +def test_an_existing_referential_tag_is_refused() -> None: + with pytest.raises(InputValidationError, match="'unit' is already a referential"): + _ = _referential().tag_by("unit") + + +def _typed_source(column: pa.Array[Any] | pa.ChunkedArray[Any]) -> Any: + """Tag ``unit``; data ``v`` of the column's type, non-nullable.""" + schema = pa.schema( + [ + pa.field("unit", pa.int64(), nullable=False), + pa.field("v", column.type, False), + ] + ) + return op.sources.ArrowTableSource( + pa.Table.from_arrays([pa.array([1, 2], pa.int64()), column], schema=schema), + tag_columns=["unit"], + tag_only_columns=True, + source_id="typed", + ) + + +@pytest.mark.parametrize( + "column", + [ + pa.array([[1], [2]], pa.large_list(pa.int64())), + pa.array([{"a": 1}, {"a": 2}], pa.struct([("a", pa.int64())])), + pa.array([decimal.Decimal("1.5"), decimal.Decimal("2.5")]), + pa.array([datetime.time(1), datetime.time(2)]), + pa.array([datetime.timedelta(1), datetime.timedelta(2)]), + ], + ids=["list", "struct", "decimal", "time", "duration"], +) +def test_a_nested_or_untyped_type_is_refused(column: pa.Array[Any]) -> None: + with pytest.raises(InputValidationError, match="'v' cannot be a tag"): + _ = _typed_source(column).tag_by("v") + + +def test_a_logical_type_is_refused() -> None: + source = op.sources.DictSource( + [{"unit": 1, "path": Path("/a")}, {"unit": 2, "path": Path("/b")}], + tag_columns=["unit"], + tag_only_columns=True, + source_id="paths", + ) + with pytest.raises(InputValidationError, match="'path' cannot be a tag"): + _ = source.tag_by("path") + + +@pytest.mark.parametrize( + "column", + [ + pa.array([1, 2], pa.int8()), + pa.array([0.5, 1.5], pa.float32()), + pa.array([True, False]), + pa.array(["a", "b"], pa.string()), + pa.array([b"a", b"b"], pa.binary()), + pa.array([datetime.datetime(2026, 11, 16), datetime.datetime(2026, 11, 17)]), + pa.array([datetime.date(2026, 11, 16), datetime.date(2026, 11, 17)]), + ], + ids=[ + "int8", + "float32", + "bool", + "string", + "binary", + "timestamp", + "date", + ], +) +def test_a_scalar_type_is_accepted(column: pa.Array[Any]) -> None: + source = _typed_source(column) + tagged = source.tag_by("v") + assert stream_references(tagged) == ("v",) + assert tagged.as_table().schema.field("v") == source.as_table().schema.field("v") + + +# ── in a job ──────────────────────────────────────────────────────────────── + + +@ORCHESTRATORS +def test_in_a_job_a_function_node_stores_the_new_tag_marked( + tmp_path: Path, + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(database=database) + with job: + computed = session_of.pod(_referential(), label="session") + tagged = TagBy("session")(computed, label="tag_by") + double.pod(tagged, label="double") + job.run(orchestrator=orchestrator()) + assert job.nodes["tag_by"].keys() == (("unit", "session"), ("unit", "session")) + assert job.nodes["double"].keys() == (("unit", "session"), ("unit", "session", "y")) + rows = sorted(job.nodes["double"].as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["y"]) for r in rows] == [ + (1, "s0", "s0s0"), + (2, "s1", "s1s1"), + ] + marked = { + stored_references(database.get_all_records(path).schema) # pyright: ignore[reportOptionalMemberAccess] # every listed table has records + for path in database.list_table_paths() + if "_result" not in path + } + assert ("unit", "session") in marked + + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json") + assert loaded.nodes["tag_by"].keys() == job.nodes["tag_by"].keys() + assert loaded.nodes["tag_by"].content_hash() == job.nodes["tag_by"].content_hash() + + +@op.function_pod(output_keys=["y"]) +def double(session: str) -> str: + """Reads the new tag's data column; ``unit`` falls through.""" + return session * 2 From 6aa23146826c0644baf5a7f1f4f3a6868a1b5b31 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:38:23 -0800 Subject: [PATCH 2/6] docs(referential-tags): describe the three moves by role after tag_by (ITL-1175) Developer decision (ITL-1195): dropping the tag or the data of a referential tag removes that role only; dropping the column removes both. Co-Authored-By: Claude Opus 5.5 --- docs/concepts/referential-tags.md | 6 ++++-- docs/dev/specs/ITL-1175-tag-by.md | 7 +++++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index d1183047e..29cc256b4 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -116,8 +116,10 @@ A nullable column (`T | None`) is refused: drop the rows with a null first, `s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, your own classes and logical types (`Path`, `UUID`, arrays) are refused. -`tag_by` is one of three moves between the roles of a column. The other two, which turn -a referential tag into plain data or into a plain tag, are the subject of ITL-1195. +`tag_by` adds the tag role; the drop operators remove a role. For a referential tag `x`, +dropping the tag (`drop_tag_columns(["x"])`) leaves `x` plain data, dropping the data +(`drop_data_columns(["x"])`) leaves `x` a plain tag, and dropping the column +(`drop_columns(["x"])`) removes it from both roles. ## Operators diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index 057426806..fa79c055b 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -17,8 +17,11 @@ records what changed. duration are not in it: the type converter reads them as no Python type (`Any`), and `Any` is refused. The source path is not changed here; the inconsistency is reported in the PR. -- Removing the data role afterwards is the subject of ITL-1195 (explicit operators); the - docs leave a placeholder for it instead of describing the implicit path. +- Removing a role afterwards (developer, November 16, 2026, ITL-1195): no new operators. + For a referential tag, dropping the tag leaves plain data, dropping the data leaves a + plain tag, dropping the column removes both roles. So `tag_by("x")` followed by + `drop_data_columns(["x"])` leaving a plain tag is intended behaviour. The docs + describe the three moves by role in one paragraph; final wording comes with ITL-1195. ## Public API From eb32ab136360616dee6ce8f4a48d87e7d1e4beb3 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Mon, 16 Nov 2026 23:42:04 -0800 Subject: [PATCH 3/6] test(tag_by): a logged TagBy operator node stores its new tag marked (ITL-1175) Co-Authored-By: Claude Opus 5.5 --- .../test_core/referential_tags/test_tag_by.py | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index a2116c3b5..35fa5865b 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -32,7 +32,8 @@ PipelineJob, SyncPipelineOrchestrator, ) -from orcapod.system_constants import constants +from orcapod.system_constants import OPERATOR_DB_SCHEMA_VERSION, constants +from orcapod.types import CacheMode ORCHESTRATORS = pytest.mark.parametrize( "orchestrator", @@ -398,3 +399,24 @@ def test_in_a_job_a_function_node_stores_the_new_tag_marked( def double(session: str) -> str: """Reads the new tag's data column; ``unit`` falls through.""" return session * 2 + + +@ORCHESTRATORS +def test_a_logged_tag_by_node_stores_its_new_tag_marked( + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(name="ops", database=database) + with job: + _ = TagBy("trial")(_referential(), label="tag_by") + job.nodes["tag_by"].attach_databases( + pipeline_database=database.at("ops"), cache_mode=CacheMode.LOG + ) + job.run(orchestrator=orchestrator()) + paths = [ + p for p in database.list_table_paths() if p[-1] == OPERATOR_DB_SCHEMA_VERSION + ] + assert len(paths) == 1 + stored = database.get_all_records(paths[0]) + assert stored is not None + assert stored_references(stored.schema) == ("unit", "trial") From 8cd398fd0e22763cfa2b3a538e1e91f256430467 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:05:48 -0800 Subject: [PATCH 4/6] fix(tag_by): keep the input's column layout and data context (ITL-1175) TagBy moved the named columns in front of the other data columns, so the computed stream's data order differed from the predicted (input) order. The columns now stay where they are; the rows keep the input's data context, as SelectColumns does. Co-Authored-By: Claude Opus 5.5 --- docs/dev/specs/ITL-1175-tag-by.md | 6 ++++-- src/orcapod/core/operators/tag_by.py | 15 +++++++++------ .../test_core/referential_tags/test_tag_by.py | 18 ++++++++++++++++++ 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index fa79c055b..7ecd99c64 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -67,8 +67,10 @@ built, sync and async alike): name-preserving: same column names, same values; record ids unchanged). - Data: unchanged: same names, order, types, nullability, values; each column keeps its own source-info token. No token is created or merged. -- Table layout: the column is stored once and moves into the user tag group (row order: - user tags, system tags, data, source info), as a source places a referential tag. +- Table layout: the column is stored once and stays where it is, as a source leaves a + referential tag at its place in the table: the data order is the input's (the + predicted one). (The first version moved it into the user tag group, which put the + new tags first among the data columns, unlike the prediction: CodeRabbit on PR #420.) - Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with their data types, after the user tags and before the system tags; the data schema is unchanged. Verified against `static_process(...).output_schema(...)`. diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py index fcfeee00b..e138b2121 100644 --- a/src/orcapod/core/operators/tag_by.py +++ b/src/orcapod/core/operators/tag_by.py @@ -56,7 +56,7 @@ class TagBy(UnaryOperator): the data columns keep their names, order, types, values and source info, the system tags keep their names and values (rule 1, name-preserving), and no row is added, removed or reordered. The new tags follow the input's user - tags, in the order given, and the column is stored once, among the tags. + tags, in the order given, and the column is stored once, where it was. The operator creates no value and moves none: every output data column is the input data column of that name, and no tag becomes data. A data column that a @@ -192,7 +192,11 @@ def unary_output_schema( def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: """Re-wrap the input's rows with the named columns among the tags. - The same arrays, laid out with the tags first: no cast, no value changed. + The input's table as it is laid out: no column moved, no cast, no value + changed. The data columns are taken in table order, so leaving each named + column where it is keeps the input's data order (the predicted one); the + tag order is the ``tag_columns`` given. The rows keep the input's data + context. Args: stream: The input stream. @@ -204,10 +208,9 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: table = stream.as_table( columns={"source": True, "system_tags": True, "sort_by_tags": False} ) - new_tag_columns = [*tag_columns, *self.columns] - rest = [name for name in table.column_names if name not in new_tag_columns] return ArrowTableStream( - table.select([*new_tag_columns, *rest]), - tag_columns=new_tag_columns, + table, + tag_columns=[*tag_columns, *self.columns], reference_tags=[*stream_references(stream), *self.columns], + data_context=stream.data_context_key, ) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index 35fa5865b..b9bdd3ccb 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -150,6 +150,24 @@ def test_the_schema_is_predicted(all_info: bool) -> None: assert data_schema == source.output_schema()[1] +def test_the_data_order_is_the_inputs_and_the_predicted_one() -> None: + """The named columns stay where they are: tagging reorders no data column. + + The order of ``keys()`` and of the data schema of the computed stream + (``static_process``) equals the input's and the predicted one, also when the + names are given in another order than the data (CodeRabbit on PR #420). + """ + source = _referential() + tag_by = TagBy(["label", "trial"]) + computed = tag_by.static_process(source) + _, input_data = source.keys() + assert input_data == ("unit", "trial", "amp", "label") + assert computed.keys() == (("unit", "label", "trial"), input_data) + assert list(computed.output_schema()[1]) == list(input_data) + assert list(tag_by.output_schema(source)[1]) == list(input_data) + assert source.tag_by("label", "trial").keys() == computed.keys() + + def test_zero_rows_have_the_types_of_rows() -> None: source = _plain() tagged = source.tag_by("trial") From 83ff606ff15d46b6e7224097c77056c8fe55086f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:17:29 -0800 Subject: [PATCH 5/6] fix(tag_by): keep each row's data context key; correct the spec on LOG (ITL-1175) Co-Authored-By: Claude Opus 5.5 --- docs/dev/specs/ITL-1175-tag-by.md | 6 +++--- src/orcapod/core/operators/tag_by.py | 12 +++++++++--- tests/test_core/referential_tags/test_tag_by.py | 11 +++++++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md index 7ecd99c64..350d9fbfc 100644 --- a/docs/dev/specs/ITL-1175-tag-by.md +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -102,9 +102,9 @@ stays with `DerivedSource(tags_as_data=...)`. issue's out-of-scope "plain tag from a data column" reachable in two steps. Recommendation: allow (existing slice behaviour, warned). Conservative variant: none that is small; it would need tracking where a tag came from. -- Jobs: an operator node runs it in the default cache mode; `LOG` / `REPLAY` refuse - referential tags as for every operator (slice limitation, unchanged). A function node - downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and +- Jobs: an operator node runs it in every cache mode; in `LOG` mode it stores its output + with the new tag marked (operator storage of referential tags came with #407, ITL-1019; + tested). A function node downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and the class is registered in `pipeline/serialization.py`. ## Open choice: may a computed column key a join? diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py index e138b2121..6072be8d6 100644 --- a/src/orcapod/core/operators/tag_by.py +++ b/src/orcapod/core/operators/tag_by.py @@ -195,8 +195,8 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: The input's table as it is laid out: no column moved, no cast, no value changed. The data columns are taken in table order, so leaving each named column where it is keeps the input's data order (the predicted one); the - tag order is the ``tag_columns`` given. The rows keep the input's data - context. + tag order is the ``tag_columns`` given. The stream keeps the input's data + context, and each row its own data context key (the context column). Args: stream: The input stream. @@ -205,8 +205,14 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: The stream with the named columns as referential tags. """ tag_columns, _ = stream.keys() + # The context column too, so each row keeps its own data context key. table = stream.as_table( - columns={"source": True, "system_tags": True, "sort_by_tags": False} + columns={ + "source": True, + "system_tags": True, + "context": True, + "sort_by_tags": False, + } ) return ArrowTableStream( table, diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py index b9bdd3ccb..9e5faa6c3 100644 --- a/tests/test_core/referential_tags/test_tag_by.py +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -168,6 +168,17 @@ def test_the_data_order_is_the_inputs_and_the_predicted_one() -> None: assert source.tag_by("label", "trial").keys() == computed.keys() +def test_each_row_keeps_its_data_context_key() -> None: + """The context column passes through: no row takes the default key instead.""" + source = _plain() + before = source.as_table(columns={"context": True}) + after = TagBy(["trial"]).static_process(source).as_table(columns={"context": True}) + assert ( + after.column(constants.CONTEXT_KEY).to_pylist() + == before.column(constants.CONTEXT_KEY).to_pylist() + ) + + def test_zero_rows_have_the_types_of_rows() -> None: source = _plain() tagged = source.tag_by("trial") From 01c3b729ed37e01a82309a5bfb45f8c616c39d96 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 17 Nov 2026 00:19:01 -0800 Subject: [PATCH 6/6] docs(design_issues): log ST5, the two tag orders of ArrowTableStream (ITL-1198) Co-Authored-By: Claude Opus 5.5 --- docs/dev/design_issues/ITL-1198-ST5.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 docs/dev/design_issues/ITL-1198-ST5.md diff --git a/docs/dev/design_issues/ITL-1198-ST5.md b/docs/dev/design_issues/ITL-1198-ST5.md new file mode 100644 index 000000000..fdbc6d763 --- /dev/null +++ b/docs/dev/design_issues/ITL-1198-ST5.md @@ -0,0 +1,23 @@ +## `src/orcapod/core/streams/arrow_table_stream.py` — tag order + +### ST5 — `keys()` lists tags in `tag_columns` order, `output_schema()` in physical order +**Status:** open +**Severity:** low +**Issue:** ITL-1198 + +`ArrowTableStream.keys()` gives the tag columns in the order of its `tag_columns` argument, +while `output_schema()` gives them in the table's physical column order (`_tag_schema` is the +table schema filtered to the tags). The data keys and the data schema both follow the physical +order and agree. A plain source shows the difference: over a table with columns `b, a, v` and +`tag_columns=["a", "b"]`, `keys()` lists `a, b` and `output_schema()` lists `b, a`. + +`TagBy` (ITL-1175) meets it: it keeps the input's layout, so the data order stays the input's +(the predicted one), and its predicted tag order is the requested one, which `keys()` reports. +For names given in another order than the table's, `output_schema()` of the computed stream +lists the new tags in table order. No layout can give both orders, since one physical order +decides both schemas. + +No hash depends on either order (`Schema` equality and the pipeline and content hashes are +order-insensitive), and column order is not part of a stream's identity (ITL-1112). The fix +is to decide one order for both methods in `ArrowTableStream`, without moving any stored +table's layout.