diff --git a/changelog.d/ITL-1175.md b/changelog.d/ITL-1175.md new file mode 100644 index 000000000..63d9086a0 --- /dev/null +++ b/changelog.d/ITL-1175.md @@ -0,0 +1,18 @@ +### Added + +- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as + referential tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each + named column becomes one column in both roles, with its values and source info + unchanged; the new tags follow the stream's tags in the order given, and that order is + part of the operator's identity. System tags, record ids and every data column are + unchanged. A column a function computed may become a tag and then key a join or a + group. Refused, naming the column: a name that is not a data column of the input, a + name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is + one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a + nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered + for saved jobs and pipelines. + +### Changed + +- The error for a name that is a tag in one join input and a data column in the other + also offers `tag_by` as a way out. diff --git a/changelog.d/ITL-1195.md b/changelog.d/ITL-1195.md new file mode 100644 index 000000000..2028df0f2 --- /dev/null +++ b/changelog.d/ITL-1195.md @@ -0,0 +1,13 @@ +### Changed + +- The select and drop operators act on a referential tag by role (developer decision, + November 16, 2026, ITL-1195), as documented in "Moving a column between roles": for a + referential tag `x`, dropping the tag (`drop_tag_columns`) removes the tag role only + and leaves `x` plain data with its source info; dropping the data (`drop_data_columns`) + removes the data role only and leaves `x` a plain tag with the same values; dropping + the column (`drop_columns`) removes it from both roles. `select_tag_columns`, + `select_data_columns` and `select_columns` follow the same rule for the names they do + not select. This was already the slice's behaviour; it is now the stated rule, with + tests by role. Removing the data role still emits `ReferenceMaterializationWarning`, + as on the slice; removing the tag role or the column is silent. No second name (such as + `untag`) is added. diff --git a/docs/concepts/referential-tags.md b/docs/concepts/referential-tags.md index 2d4dadb1d..18cc36194 100644 --- a/docs/concepts/referential-tags.md +++ b/docs/concepts/referential-tags.md @@ -74,14 +74,103 @@ A function output named like a tag is an error when the pipeline is built. A fun that does not read a referenced column keeps its result identity: retagging a source does not make it compute again. +## Moving a column between roles + +A referential tag is one column in two roles, a tag and a data column. Four steps move a +column between the roles, and none of them changes, copies or creates a value: + +| Step | Data column `x` | Referential tag `x` | Plain tag `x` | +|---|---|---|---| +| `tag_by("x")` | becomes a referential tag | refused: already a tag | refused: not a data column | +| `drop_tag_columns(["x"])` | refused: not a tag | the tag role goes: `x` is plain data | removed | +| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag, with a `ReferenceMaterializationWarning` | refused: not a data column | +| `drop_columns(["x"])` | removed | removed from both roles | removed | + +The select operators follow the same rule by role: `select_tag_columns` removes the tag +role of every tag it does not name, `select_data_columns` the data role of every data +column it does not name, and `select_columns` keeps exactly the columns it names, in +both roles. (The drop and select refusals are those of a missing name: `strict=False` +ignores the name instead.) + +For a referential tag `x`, in plain words: + +- Dropping or selecting away the **data** of `x` leaves a plain tag `x`, and issues a + `ReferenceMaterializationWarning`: the tag no longer refers to a data column, so its + values now live only in the tag (see below to turn the warning off). +- Dropping or selecting away the **tag** `x` leaves plain data `x`, silently. +- Dropping the **column** `x` removes it from both roles, silently. + +```python +print(spikes.drop_tag_columns(["time"]).keys()) +# (('unit',), ('unit', 'time', 'amp')) +print(spikes.drop_data_columns(["time"]).keys()) +# (('unit', 'time'), ('unit', 'amp')) +print(spikes.drop_columns(["time"]).keys()) +# (('unit',), ('unit', 'amp')) +``` + +- System tags and record ids never change. +- Source info belongs to the data role: dropping the tag keeps the column's source-info + token, dropping the data removes it (a plain tag carries none). The row's provenance + is still its system tags (its source id and record id). +- No data element comes from a tag value: dropping the tag leaves a data column that was + data before; dropping the data leaves a tag whose values came from a data column. +- A stream keeps at least one data column: dropping the data role of the last one is + refused. +- A stored table marks exactly the referential tags of what it stores, so a column moved + out of one role is stored unmarked. + +### Adding the tag role: `tag_by` + +A source declares its tags, but a value a later step should join, group or sort by is +often computed on the way: a session label, a trial index, a phase. `tag_by` gives +existing data columns the tag role: each becomes a referential tag, one column that is +a tag and the data column it was, with its values and source info unchanged. + +```python +@op.function_pod(output_keys=["phase"]) +def phase_of(time: int) -> str: + return "early" if time < 15 else "late" + + +phased = phase_of.pod(spikes).tag_by("phase") +print(phased.keys()) +# (('unit', 'time', 'phase'), ('unit', 'time', 'phase')) + +by_phase = phased.group_by(["phase"]) +print(by_phase.keys()) +# (('phase',), ('unit', 'time', 'phase')) +``` + +- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's + tags in the order given, and that order is part of the operator (`TagBy`): the names + are a list, never a set. +- Nothing else changes: no row is added, removed or reordered, the data columns keep + their names, types and values, every data column keeps its own source-info token, + and the system tags keep their names and values. +- No data element comes from a tag value: the operator adds a tag role to a column + that is already data, and creates or moves no value. Offering a tag's value as data + stays with `DerivedSource(tags_as_data=...)`. +- A computed column may become a tag, and then key a join or a group, as `phase` does + above. + +`tag_by` refuses, naming the column: a name that is not a data column of the stream; a +name that is already a tag (plain or referential); a reserved name (the `__` prefix, a +system column, a name holding `::`); and a column whose type cannot be a tag. A tag is +one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`. +A nullable column (`T | None`) is refused: drop the rows with a null first, +`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, +your own classes and logical types (`Path`, `UUID`, arrays) are refused. + ## Operators | Operator | Referential tag `x` | |---|---| +| `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. | | `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | | `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tuple of the input's non-key tags (then record ids), the same order for every data list of the group, so all of them stay aligned. | -| `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. | -| `SelectTagColumns`, `DropTagColumns` | Removing the tag leaves `x` a data column. | +| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `ReferenceMaterializationWarning`. | +| `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. | | `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. | | `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. | | `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | diff --git a/docs/dev/design_issues/ITL-1023-RT1.md b/docs/dev/design_issues/ITL-1023-RT1.md index 4b8f026ab..8501fe622 100644 --- a/docs/dev/design_issues/ITL-1023-RT1.md +++ b/docs/dev/design_issues/ITL-1023-RT1.md @@ -5,4 +5,4 @@ **Severity:** medium **Issue:** ITL-1023 -The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. +The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` and `TagBy` (`tag_by`, ITL-1175), which gives a data column the tag role. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry. diff --git a/docs/dev/design_issues/ITL-1198-ST5.md b/docs/dev/design_issues/ITL-1198-ST5.md new file mode 100644 index 000000000..fdbc6d763 --- /dev/null +++ b/docs/dev/design_issues/ITL-1198-ST5.md @@ -0,0 +1,23 @@ +## `src/orcapod/core/streams/arrow_table_stream.py` — tag order + +### ST5 — `keys()` lists tags in `tag_columns` order, `output_schema()` in physical order +**Status:** open +**Severity:** low +**Issue:** ITL-1198 + +`ArrowTableStream.keys()` gives the tag columns in the order of its `tag_columns` argument, +while `output_schema()` gives them in the table's physical column order (`_tag_schema` is the +table schema filtered to the tags). The data keys and the data schema both follow the physical +order and agree. A plain source shows the difference: over a table with columns `b, a, v` and +`tag_columns=["a", "b"]`, `keys()` lists `a, b` and `output_schema()` lists `b, a`. + +`TagBy` (ITL-1175) meets it: it keeps the input's layout, so the data order stays the input's +(the predicted one), and its predicted tag order is the requested one, which `keys()` reports. +For names given in another order than the table's, `output_schema()` of the computed stream +lists the new tags in table order. No layout can give both orders, since one physical order +decides both schemas. + +No hash depends on either order (`Schema` equality and the pipeline and content hashes are +order-insensitive), and column order is not part of a stream's identity (ITL-1112). The fix +is to decide one order for both methods in `ArrowTableStream`, without moving any stored +table's layout. diff --git a/docs/dev/design_issues/ITL-1205-AB1.md b/docs/dev/design_issues/ITL-1205-AB1.md new file mode 100644 index 000000000..306669fa4 --- /dev/null +++ b/docs/dev/design_issues/ITL-1205-AB1.md @@ -0,0 +1,22 @@ +## `src/orcapod/core/operators/static_output_pod.py` — async barrier input + +### AB1 — The async barrier path rebuilds its input with data-tags first and in the default data context +**Status:** open +**Severity:** medium +**Issue:** ITL-1205 + +An operator without its own `async_execute` collects its input rows and rebuilds them as a +stream (`StaticOutputOperatorPod._materialize_input_stream`, through `_rows_to_table`), then +applies its sync rule. Two properties of that rebuild make the async result differ from the +sync one, for every operator on this path. + +1. Data order: `_rows_to_table` takes a data-tag (a tag that is also a data column, today a + referential tag) from the tags and lays it out with them, so it leads the data columns. + On the sync path a data-tag stays where the input table has it. For an input with data + `a, x, y` and `x` a data-tag, `PolarsFilter` (and `TagBy`) give data `a, x, y` on the sync + path and `x, a, y` on the async path. +2. Data context: the rebuilt input has neither the rows' data context nor a context column, + so it takes the default context. Only one data context is registered today, so this is + not observable yet. + +The fix must keep the row-built layout of stored tables (`_in_row_order`) unchanged. diff --git a/docs/dev/specs/ITL-1175-tag-by.md b/docs/dev/specs/ITL-1175-tag-by.md new file mode 100644 index 000000000..350d9fbfc --- /dev/null +++ b/docs/dev/specs/ITL-1175-tag-by.md @@ -0,0 +1,121 @@ +# ITL-1175: mark existing data columns as referential tags (`tag_by`) + +Status: approved by the coordinator on 2026-11-16, with two developer decisions (the +name `tag_by` / `TagBy`; a computed column may become a tag). Stacked on the referential +tags slice (ITL-1023, PR #342). The sections below are the design as sent; "Decisions" +records what changed. + +## Decisions + +- Name: `tag_by` (developer); the class follows the operator naming: `TagBy`. +- A computed column may become a tag and key a join or a group (developer); no lineage + check. +- Type rule: the slice has no type rule for a referential tag declared at a source (a + source accepts any type), so `TagBy` uses its own rule, in one function + (`referential_tags.require_taggable_type`) that the source path could adopt: `int`, + `float`, `bool`, `str`, `bytes`, `datetime`, `date`. `Decimal`, time of day and + duration are not in it: the type converter reads them as no Python type (`Any`), and + `Any` is refused. The source path is not changed here; the inconsistency is reported + in the PR. +- Removing a role afterwards (developer, November 16, 2026, ITL-1195): no new operators. + For a referential tag, dropping the tag leaves plain data, dropping the data leaves a + plain tag, dropping the column removes both roles. So `tag_by("x")` followed by + `drop_data_columns(["x"])` leaving a plain tag is intended behaviour. The docs + describe the three moves by role in one paragraph; final wording comes with ITL-1195. + +## Public API + +- Operator `orcapod.core.operators.TagBy(columns)`; stream method + `stream.tag_by(*columns, label=None)`. `s.tag_by("x", "y")` equals + `s.tag_by(["x", "y"])` (the convention of `select_columns`). +- The name is a choice for the developer. It lives in one place per surface: the class + name, the stream method, the serialization registry entry, the docs. Not + `reference_tags`: `ArrowTableStream.reference_tags` is already a property. + Alternatives: `as_tags` / `AsTags`, `mark_tags` / `MarkTags`. +- No `strict` flag: a name that cannot be marked is always an error. + +## Order rule and identity + +- The names are ordered: the output's user tags are the input's user tags, in their + order, then the named columns in the order given. So the order is part of the + operator, and its identity is `("TagBy", [names in the order given])`; its pipeline + identity is the same. A `set` / `frozenset` is refused (`TypeError`, unordered, as for + `record_id_columns`), as is an empty list and a repeated name (`ValueError`). + +## Refusals + +At construction (`ValueError` / `TypeError`, naming the column): no name; a non-string; +a repeated name; a set; the meta prefix (`require_names_not_meta`); a system column name +(`require_names_not_system`); a name holding `::` (`require_names_without_separator`). + +At validation (`InputValidationError`, naming the column, raised when the pipeline is +built, sync and async alike): + +- not a data column of the input (lists the input's data columns); +- already a tag: a referential tag ("already a referential tag"); a plain tag is never a + data column, so it is reported as a tag, not a data column; +- a type that cannot be a tag. Recommended rule: the column's Python type must be one of + `int`, `float`, `bool`, `str`, `bytes`, `datetime`, `date` (see Decisions). Refused: `T | None` (the error says to apply + `drop_nulls("x")` first, which narrows it to `T`), lists, sets, dicts, tuples, structs, + user classes and every logical type (`Path`, `UUID`, arrays...). Sources and function + outputs are non-nullable by default, so this costs nothing in the common case. + Conservative variant: also refuse `float` (equality on floats as a key). + +## What the output is + +- Tags: input user tags, then the new names; system tags unchanged (rule 1, + name-preserving: same column names, same values; record ids unchanged). +- Data: unchanged: same names, order, types, nullability, values; each column keeps its + own source-info token. No token is created or merged. +- Table layout: the column is stored once and stays where it is, as a source leaves a + referential tag at its place in the table: the data order is the input's (the + predicted one). (The first version moved it into the user tag group, which put the + new tags first among the data columns, unlike the prediction: CodeRabbit on PR #420.) +- Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with + their data types, after the user tags and before the system tags; the data schema is + unchanged. Verified against `static_process(...).output_schema(...)`. +- Sync: the input table re-wrapped as an `ArrowTableStream` with the extra tags in + `tag_columns` and `reference_tags`: no Polars round trip, no cast. +- Async: no own `async_execute`: the base barrier path collects the rows and applies the + sync rule, so the rows are identical by construction. Added to + `test_async_sync_equivalence.py`. A streaming version can come later. + +## Why "no data element comes from a tag value" holds + +The operator adds a tag role to a data column; it creates no value and moves none. Every +output data column is the input data column of that name, with the same values and +source info, and no input tag becomes data. The column stays one physical column, as a +source's referential tag is. The direction the rule forbids (a tag value offered as data) +stays with `DerivedSource(tags_as_data=...)`. + +## Downstream + +- `Join` with a stream where `x` is a plain tag or a referential tag: the slice's rules + (one referential tag, one data column, merged `join_v1` token when referential on + several inputs). With a stream where `x` is data only: the existing tag/data error (its + remedy text gains "or mark it with `tag_by`"). +- `GroupBy(["x"])`, `Batch`, `MapTags`, select/drop: the slice's rules for any + referential tag. +- Removing the data role afterwards (`tag_by("x").drop_data_columns("x")`) leaves a plain + tag holding the data value, with the `ReferenceMaterializationWarning`, as for any + referential tag. That is the data-to-tag direction (the rule holds), but it makes the + issue's out-of-scope "plain tag from a data column" reachable in two steps. + Recommendation: allow (existing slice behaviour, warned). Conservative variant: none + that is small; it would need tracking where a tag came from. +- Jobs: an operator node runs it in every cache mode; in `LOG` mode it stores its output + with the new tag marked (operator storage of referential tags came with #407, ITL-1019; + tested). A function node downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and + the class is registered in `pipeline/serialization.py`. + +## Open choice: may a computed column key a join? + +Recommendation: yes. It is the issue's own typical use (a function computes a session +label that later steps join, group or sort by), and nothing in hashing or record +identity depends on where a tag's value came from. Conservative variant: `TagBy` refuses +a column a function pod produced. That needs a lineage walk upstream (through renames and +joins) in one helper called from validation; it is not small, and I recommend against it. + +## Not in scope + +Golden pins (identities provisional in 0.4.0rc1); a plain-tag variant; the reverse +direction; `MergeJoin` (ITL-1194). diff --git a/src/orcapod/core/operators/__init__.py b/src/orcapod/core/operators/__init__.py index 032cd0a60..838f6eb7c 100644 --- a/src/orcapod/core/operators/__init__.py +++ b/src/orcapod/core/operators/__init__.py @@ -16,6 +16,7 @@ from .merge_join import MergeJoin from .pick import Pick from .semijoin import SemiJoin +from .tag_by import TagBy __all__ = [ "Join", @@ -36,4 +37,5 @@ "DropNulls", "Pick", "Index", + "TagBy", ] diff --git a/src/orcapod/core/operators/base.py b/src/orcapod/core/operators/base.py index eb65ae1ce..bd6851502 100644 --- a/src/orcapod/core/operators/base.py +++ b/src/orcapod/core/operators/base.py @@ -101,8 +101,8 @@ def _tag_data_error(name: str, tag_side: str, data_side: str) -> InputValidation return InputValidationError( f"column {name!r} is a tag in the {tag_side} input and a data column in the " + f"{data_side}, and a name cannot be both. Rename it in one input with " - + "map_tags or map_data, or declare it a tag at the source that has it as " - + "a data column." + + "map_tags or map_data, or make it a tag of the input that has it as a data column (declare it " + + "a tag at its source, or mark it with tag_by)." ) diff --git a/src/orcapod/core/operators/tag_by.py b/src/orcapod/core/operators/tag_by.py new file mode 100644 index 000000000..6072be8d6 --- /dev/null +++ b/src/orcapod/core/operators/tag_by.py @@ -0,0 +1,222 @@ +"""TagBy operator: give existing data columns the tag role, as referential tags (ITL-1175).""" + +from __future__ import annotations + +from collections.abc import Sequence, Set +from typing import Any, ClassVar + +from typing_extensions import override + +from orcapod.core.operators.base import UnaryOperator +from orcapod.core.referential_tags import require_taggable_type, stream_references +from orcapod.core.streams import ArrowTableStream +from orcapod.errors import InputValidationError +from orcapod.protocols.core_protocols import StreamProtocol +from orcapod.system_constants import constants +from orcapod.types import ColumnConfig, Schema +from orcapod.utils import schema_utils + + +def _tag_by_names(columns: str | Sequence[str]) -> tuple[str, ...]: + """Return the names ``TagBy`` marks, in the order given. + + Args: + columns: One name, or a sequence of names. + + Returns: + The names, in the order given. + + Raises: + TypeError: If ``columns`` is a set (unordered: the order of the names is the + order of the new tags) or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + """ + if isinstance(columns, str): + return (columns,) + if isinstance(columns, Set): + raise TypeError( + "TagBy: give the column names as a list or a tuple, not a set: their " + + "order is the order of the new tags" + ) + names = tuple(columns) + if not names: + raise ValueError("TagBy: expected at least one column name") + if not_strings := [name for name in names if not isinstance(name, str)]: + raise TypeError(f"TagBy: column names must be strings, got {not_strings!r}") + if repeated := sorted({name for name in names if names.count(name) > 1}): + raise ValueError(f"TagBy: columns {repeated} are named more than once") + return names + + +class TagBy(UnaryOperator): + """Give existing data columns of a stream the tag role, as referential tags. + + Each named data column becomes a referential tag (ITL-1023): one column in + both roles, a tag and the data column it already was. Nothing else changes: + the data columns keep their names, order, types, values and source info, + the system tags keep their names and values (rule 1, name-preserving), and + no row is added, removed or reordered. The new tags follow the input's user + tags, in the order given, and the column is stored once, where it was. + + The operator creates no value and moves none: every output data column is the + input data column of that name, and no tag becomes data. A data column that a + function computed may become a tag (developer, November 16, 2026), and then + key a join or a group. + + Args: + columns: One data column name, or a sequence of them. Their order is the + order of the new tags, and part of the operator's identity. + + Raises: + TypeError: If ``columns`` is a set or holds a name that is not a string. + ValueError: If no name is given, or a name is repeated. + InputValidationError: If a name is reserved: the meta prefix, a system + column name, or one holding ``::``. + """ + + # Referential tags (ITL-1023): handled; this operator makes them. + REFERENTIAL_TAGS_ISSUE: ClassVar[str | None] = None + + def __init__(self, columns: str | Sequence[str], **kwargs: Any) -> None: + names = _tag_by_names(columns) + schema_utils.require_names_not_meta(names) + schema_utils.require_names_not_system(names) + schema_utils.require_names_without_separator(names) + self.columns: tuple[str, ...] = names + super().__init__(**kwargs) + + @override + def identity_structure(self) -> Any: + """Return the operator's identity: its class name and the names, in order. + + Returns: + ``("TagBy", [names in the order given])``. + """ + return (self.__class__.__name__, list(self.columns)) + + @override + def to_config(self) -> dict[str, Any]: + """Serialize this operator to a config dict. + + Returns: + A dict with ``class_name``, ``module_path`` and ``config`` keys, where + ``config`` holds ``columns``, the names in order. + """ + config = super().to_config() + config["config"] = {"columns": list(self.columns)} + return config + + @override + def __repr__(self) -> str: + return f"TagBy({list(self.columns)!r})" + + @override + def validate_unary_input(self, stream: StreamProtocol) -> None: + """Check that every name is a data column of ``stream`` that can be a tag. + + Args: + stream: The input stream. + + Raises: + InputValidationError: Naming the column, if it is already a tag + (referential or plain), is not a data column of ``stream``, or has a + type that cannot be a tag (``referential_tags.require_taggable_type``). + """ + tag_columns, data_columns = stream.keys() + references = stream_references(stream) + _, data_schema = stream.output_schema() + for name in self.columns: + if name in references: + raise InputValidationError( + f"TagBy: column {name!r} is already a referential tag" + ) + if name in tag_columns: + raise InputValidationError( + f"TagBy: column {name!r} is a plain tag, not a data column: only " + + "a data column can be given the tag role" + ) + user_data = [ + c for c in data_columns if not c.startswith(constants.META_PREFIX) + ] + if name not in user_data: + raise InputValidationError( + f"TagBy: column {name!r} is not a data column of the input; its " + + f"data columns are {user_data}" + ) + try: + require_taggable_type(data_schema[name]) + except InputValidationError as exc: + raise InputValidationError( + f"TagBy: column {name!r} cannot be a tag: {exc}" + ) from exc + + @override + def unary_output_schema( + self, + stream: StreamProtocol, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> tuple[Schema, Schema]: + """Return the input schemas with the named columns added to the tags. + + The tag schema is the input's user tags, then the named columns with their + data types, then the system tags (when ``columns`` includes them); the data + schema is the input's. + + Args: + stream: The input stream. + columns: Column inclusion config passed through to ``output_schema``. + all_info: Whether to include all info columns. + + Returns: + A ``(tag_schema, data_schema)`` tuple. + """ + tag_schema, data_schema = stream.output_schema( + columns=columns, all_info=all_info + ) + user_tags = { + name: python_type + for name, python_type in tag_schema.items() + if not name.startswith(constants.SYSTEM_TAG_PREFIX) + } + system_tags = { + name: python_type + for name, python_type in tag_schema.items() + if name.startswith(constants.SYSTEM_TAG_PREFIX) + } + new_tags = {name: data_schema[name] for name in self.columns} + return Schema({**user_tags, **new_tags, **system_tags}), data_schema + + @override + def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: + """Re-wrap the input's rows with the named columns among the tags. + + The input's table as it is laid out: no column moved, no cast, no value + changed. The data columns are taken in table order, so leaving each named + column where it is keeps the input's data order (the predicted one); the + tag order is the ``tag_columns`` given. The stream keeps the input's data + context, and each row its own data context key (the context column). + + Args: + stream: The input stream. + + Returns: + The stream with the named columns as referential tags. + """ + tag_columns, _ = stream.keys() + # The context column too, so each row keeps its own data context key. + table = stream.as_table( + columns={ + "source": True, + "system_tags": True, + "context": True, + "sort_by_tags": False, + } + ) + return ArrowTableStream( + table, + tag_columns=[*tag_columns, *self.columns], + reference_tags=[*stream_references(stream), *self.columns], + data_context=stream.data_context_key, + ) diff --git a/src/orcapod/core/referential_tags.py b/src/orcapod/core/referential_tags.py index 41a74daa1..52bc7d465 100644 --- a/src/orcapod/core/referential_tags.py +++ b/src/orcapod/core/referential_tags.py @@ -19,6 +19,8 @@ from __future__ import annotations +import datetime +import types import typing import warnings import weakref @@ -322,6 +324,58 @@ def without_reference_marks(table: "pa.Table") -> "pa.Table": return mark_references(table, ()) +# The Python types a column may have to become a referential tag by ``tag_by`` +# (ITL-1175): one value per row, of a type with plain equality and order. A column +# whose Arrow type the type converter reads as no Python type (``Any``: decimal, time +# of day, duration) is refused. Sources do not apply this rule yet: a source accepts +# any type for a declared tag. +TAGGABLE_TYPES: tuple[type, ...] = ( + int, + float, + bool, + str, + bytes, + datetime.datetime, + datetime.date, +) + + +def require_taggable_type(python_type: object) -> None: + """Refuse a column type that cannot become a referential tag (ITL-1175). + + The rule ``tag_by`` applies: the type must be one of ``TAGGABLE_TYPES``. A + nullable type (``T | None``) is refused, with the way out: ``drop_nulls`` narrows + the column to ``T``. Lists, sets, dicts, tuples, structs, user classes and every + logical type (``Path``, ``UUID``, arrays) are refused, and so is a column the type + converter gives no Python type (``Any``). + + Args: + python_type: The column's Python type, as ``output_schema()`` gives it. + + Raises: + InputValidationError: If the type cannot be a tag. The caller names the + column. + """ + if python_type in TAGGABLE_TYPES: + return + members = typing.get_args(python_type) + if typing.get_origin(python_type) in (typing.Union, types.UnionType) and ( + type(None) in members + ): + rest = [member for member in members if member is not type(None)] + if len(rest) == 1 and rest[0] in TAGGABLE_TYPES: + raise InputValidationError( + f"its type {python_type} is nullable, and a tag needs a value in " + + "every row: drop the rows with a null first with drop_nulls, which " + + f"narrows the column to {rest[0].__name__}" + ) + names = ", ".join(t.__name__ for t in TAGGABLE_TYPES) + raise InputValidationError( + f"its type {python_type} cannot be a tag: a tag is one value of one of " + + f"these types: {names}" + ) + + def not_supported_yet(operation: str, issue: str) -> NotImplementedError: """Return the error for an operation that does not handle referential tags yet. diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index a0016940d..a2f695b69 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -2,7 +2,14 @@ import logging from abc import abstractmethod -from collections.abc import AsyncIterator, Callable, Collection, Iterator, Mapping +from collections.abc import ( + AsyncIterator, + Callable, + Collection, + Iterator, + Mapping, + Sequence, +) from contextvars import ContextVar from datetime import datetime from typing import TYPE_CHECKING, Any, cast @@ -565,6 +572,41 @@ def drop_nulls( return DropNulls(columns)(self, label=label) + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> StreamBase: + """Give the named data columns the tag role, as referential tags. + + ``s.tag_by("x", "y")`` equals ``s.tag_by(["x", "y"])``. Each named data + column becomes a referential tag: one column, both a tag and the data column + it was, with its values and source info unchanged. The new tags follow this + stream's tags, in the order given. See ``orcapod.core.operators.TagBy``. + + Args: + *columns: One or more data column names, or one list or tuple of names + as the only positional argument (not a set: the order matters). + label: Optional node label for the pipeline graph. + + Returns: + The stream with the named columns as referential tags. + + Raises: + TypeError: When no name is given, the names are a set, or a name is not + a string. + ValueError: When a name is repeated. + InputValidationError: When a name is reserved, is not a data column of + this stream, is already a tag, or has a type that cannot be a tag. + """ + from orcapod.core.operators import TagBy + + if len(columns) == 1 and not isinstance(columns[0], str): + names: str | Sequence[str] = columns[0] + else: + names = cast("tuple[str, ...]", columns) + return TagBy(names)(self, label=label) + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/src/orcapod/pipeline/serialization.py b/src/orcapod/pipeline/serialization.py index ee71b74c3..af73be040 100644 --- a/src/orcapod/pipeline/serialization.py +++ b/src/orcapod/pipeline/serialization.py @@ -205,6 +205,7 @@ def _build_operator_registry() -> dict[str, type]: SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) return { @@ -224,6 +225,7 @@ def _build_operator_registry() -> dict[str, type]: "MapColumns": MapColumns, "PolarsFilter": PolarsFilter, "DropNulls": DropNulls, + "TagBy": TagBy, } diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index 584931565..a09af7c9a 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -423,6 +423,18 @@ def drop_nulls( """ ... + def tag_by( + self, + *columns: str | Sequence[str], + label: str | None = None, + ) -> "StreamProtocol": + """ + Give the named data columns the tag role, as referential tags: each stays + one column, a tag and the data column it was, with its values and source + info unchanged. The new tags follow the stream's tags, in the order given. + """ + ... + def select_tag_columns( self, tag_columns: str | Collection[str], diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index a015bd90f..ccb10ee0d 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -47,6 +47,7 @@ SelectDataColumns, SelectTagColumns, SemiJoin, + TagBy, ) from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod from orcapod.core.sources import ArrowTableSource @@ -591,6 +592,12 @@ class Case: lambda: DropTagColumns(["ta"]), lambda: (source_a(referential=True),), ), + # Not selecting a referential tag removes its tag role only (ITL-1195). + Case( + "ref_select_tag_columns", + lambda: SelectTagColumns(["k"]), + lambda: (source_a(referential=True),), + ), Case( "ref_drop_data_columns", lambda: DropDataColumns(["k"]), @@ -632,6 +639,11 @@ class Case: lambda: MapData({"ta": "tb"}), lambda: (source_a(referential=True),), ), + # TagBy gives data columns the tag role, as referential tags (ITL-1175). + Case("tag_by", lambda: TagBy(["i8", "bn"]), lambda: (source_a(),)), + Case("tag_by_reversed", lambda: TagBy(["bn", "i8"]), lambda: (source_a(),)), + Case("tag_by_b", lambda: TagBy(["u8", "ls"]), lambda: (source_b(),)), + Case("ref_tag_by", lambda: TagBy("i8"), lambda: (source_a(referential=True),)), ] CASE = pytest.mark.parametrize("case", CASES, ids=[c.label for c in CASES]) diff --git a/tests/test_core/referential_tags/test_roles.py b/tests/test_core/referential_tags/test_roles.py new file mode 100644 index 000000000..dab4bd0e5 --- /dev/null +++ b/tests/test_core/referential_tags/test_roles.py @@ -0,0 +1,191 @@ +"""The select and drop operators act on a referential tag by role (ITL-1195). + +Developer decision, November 16, 2026: for a referential tag ``x`` (one column in two +roles), dropping the tag removes the tag role only and leaves ``x`` plain data; dropping +the data removes the data role only and leaves ``x`` a plain tag, with a +``ReferenceMaterializationWarning``; dropping the column removes it from both roles. The select operators follow the same rule: selecting tags +or data drops the others by role. Plain tags and plain data columns behave as before. + +Every operator is driven through its public call and read through the stream's public +methods; a schema prediction is checked against ``static_process``. +""" + +from __future__ import annotations + +from collections.abc import Callable +import warnings +from typing import Any + +import pytest + +import orcapod as op +from orcapod.core.operators import ( + DropColumns, + DropDataColumns, + DropTagColumns, + SelectColumns, + SelectDataColumns, + SelectTagColumns, + TagBy, +) +from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod +from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, + stream_references, +) +from orcapod.errors import InputValidationError +from orcapod.system_constants import constants + +ROWS = [{"k": 1, "x": 10, "y": 0.5}, {"k": 2, "x": 20, "y": 1.5}] + + +def _source() -> op.sources.DictSource: + """Referential tags ``k`` and ``x``; data ``y``.""" + return op.sources.DictSource(ROWS, tag_columns=["k", "x"], source_id="R") + + +def _system_tags(stream: Any) -> list[dict[str, Any]]: # pyright: ignore[reportExplicitAny] # any stream + table = stream.as_table(columns={"system_tags": True}) + names = [c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX)] + return sorted(table.select(names).to_pylist(), key=repr) + + +def _source_info(stream: Any) -> dict[str, list[Any]]: # pyright: ignore[reportExplicitAny] # any stream + table = stream.as_table(columns={"source": True}) + return { + name.removeprefix(constants.SOURCE_PREFIX): table.column(name).to_pylist() + for name in table.column_names + if name.startswith(constants.SOURCE_PREFIX) + } + + +# (operator, expected keys, expected referential tags): ``x`` is the one moved. +ROLE_CASES: list[tuple[str, Callable[[], StaticOutputOperatorPod], Any, Any]] = [ + # Dropping or not selecting the tag: the tag role goes, x is plain data. + ("drop_tag", lambda: DropTagColumns(["x"]), (("k",), ("k", "x", "y")), ("k",)), + ("select_tag", lambda: SelectTagColumns(["k"]), (("k",), ("k", "x", "y")), ("k",)), + # Dropping or not selecting the data: the data role goes, x is a plain tag. + ("drop_data", lambda: DropDataColumns(["x"]), (("k", "x"), ("k", "y")), ("k",)), + ( + "select_data", + lambda: SelectDataColumns(["k", "y"]), + (("k", "x"), ("k", "y")), + ("k",), + ), + # Dropping or not selecting the column: both roles go. + ("drop_column", lambda: DropColumns(["x"]), (("k",), ("k", "y")), ("k",)), + ("select_column", lambda: SelectColumns(["k", "y"]), (("k",), ("k", "y")), ("k",)), +] +ROLE_IDS = [case[0] for case in ROLE_CASES] + + +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +def test_each_operator_acts_by_role( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # expected keys + references: tuple[str, ...], +) -> None: + source = _source() + result = make_op()(source) + assert result.keys() == keys + assert stream_references(result) == references + # No value changes: every remaining column holds the source's values. + for row in result.as_table().to_pylist(): + assert row == {name: _by_k(row["k"])[name] for name in row} + # System tags and record ids never change. + assert _system_tags(result) == _system_tags(source) + # Source info belongs to the data role. + info = _source_info(result) + _, data = keys + assert set(info) == set(data) + if "x" in data: + assert info["x"] == _source_info(source)["x"] + _ = label + + +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +def test_only_removing_the_data_role_warns( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # unused case field + references: Any, # pyright: ignore[reportExplicitAny] # unused case field +) -> None: + """Dropping or selecting away the data of ``x`` leaves a plain tag and warns + (``ReferenceMaterializationWarning``); removing the tag role or the whole column + is silent (developer, November 17, 2026: keep the warning for now).""" + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + _ = make_op()(_source()).keys() + warned = any(w.category is ReferenceMaterializationWarning for w in caught) + assert warned == (label in ("drop_data", "select_data")) + + +def _by_k(k: int) -> dict[str, Any]: # pyright: ignore[reportExplicitAny] # row values + return next(row for row in ROWS if row["k"] == k) + + +@pytest.mark.parametrize( + ("label", "make_op", "keys", "references"), ROLE_CASES, ids=ROLE_IDS +) +@pytest.mark.parametrize("all_info", [False, True]) +def test_the_schema_is_predicted_by_role( + label: str, + make_op: Callable[[], StaticOutputOperatorPod], + keys: Any, # pyright: ignore[reportExplicitAny] # expected keys + references: tuple[str, ...], + all_info: bool, +) -> None: + source = _source() + operator = make_op() + predicted = operator.output_schema(source, all_info=all_info) + assert predicted == operator.static_process(source).output_schema(all_info=all_info) + _ = (label, keys, references) + + +def test_dropping_the_data_role_of_the_last_data_column_is_refused() -> None: + only = op.sources.DictSource( + [{"k": 1, "x": 10}], + tag_columns=["k", "x"], + tag_only_columns=["k"], + source_id="O", + ) + assert only.keys() == (("k", "x"), ("x",)) + with pytest.raises(InputValidationError, match="no data column would remain"): + _ = DropDataColumns(["x"])(only) + + +def test_a_role_that_is_not_there_is_a_missing_name() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k", "x"], tag_only_columns=["x"], source_id="P" + ) + # A plain tag has no data role; a data-only column has no tag role. + with pytest.raises(InputValidationError, match="Missing data columns"): + _ = DropDataColumns(["x"])(plain) + with pytest.raises(InputValidationError, match="Missing tag columns"): + _ = DropTagColumns(["y"])(plain) + + +def test_plain_tags_and_data_are_dropped_as_before() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k", "x"], tag_only_columns=True, source_id="P" + ) + assert DropTagColumns(["x"])(plain).keys() == (("k",), ("y",)) + assert DropColumns(["x"])(plain).keys() == (("k",), ("y",)) + assert SelectTagColumns(["k"])(plain).keys() == (("k",), ("y",)) + + +def test_tag_by_then_drop_tag_gives_back_the_rows() -> None: + plain = op.sources.DictSource( + ROWS, tag_columns=["k"], tag_only_columns=True, source_id="P" + ) + back = DropTagColumns(["x"])(TagBy("x")(plain)) + assert back.keys() == plain.keys() + assert sorted(back.as_table().to_pylist(), key=repr) == sorted( + plain.as_table().to_pylist(), key=repr + ) + assert _source_info(back) == _source_info(plain) diff --git a/tests/test_core/referential_tags/test_tag_by.py b/tests/test_core/referential_tags/test_tag_by.py new file mode 100644 index 000000000..9e5faa6c3 --- /dev/null +++ b/tests/test_core/referential_tags/test_tag_by.py @@ -0,0 +1,451 @@ +"""``TagBy`` / ``stream.tag_by``: data columns become referential tags (ITL-1175). + +A named data column becomes a referential tag: one column, a tag and the data column it +was, with its values and source info unchanged. The new tags follow the input's tags in +the order given. Every operator is driven through its public call and read through the +stream's public methods; a schema prediction is checked against ``static_process``. +""" + +from __future__ import annotations + +import datetime +import decimal +import warnings +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.core.operators import TagBy +from orcapod.core.referential_tags import ( + ReferenceMaterializationWarning, + stored_references, + stream_references, +) +from orcapod.databases import InMemoryArrowDatabase +from orcapod.errors import InputValidationError +from orcapod.pipeline.serialization import resolve_operator_from_config +from orcapod.pipeline import ( + AsyncPipelineOrchestrator, + PipelineJob, + SyncPipelineOrchestrator, +) +from orcapod.system_constants import OPERATOR_DB_SCHEMA_VERSION, constants +from orcapod.types import CacheMode + +ORCHESTRATORS = pytest.mark.parametrize( + "orchestrator", + [SyncPipelineOrchestrator, AsyncPipelineOrchestrator], + ids=["sync", "async"], +) + +ROWS = [ + {"unit": 1, "trial": 10, "amp": 0.5, "label": "a"}, + {"unit": 2, "trial": 20, "amp": 1.5, "label": "b"}, +] + + +def _plain() -> op.sources.DictSource: + """``unit`` a plain tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource( + ROWS, tag_columns=["unit"], tag_only_columns=True, source_id="spikes" + ) + + +def _referential() -> op.sources.DictSource: + """``unit`` a referential tag; ``trial``, ``amp`` and ``label`` data.""" + return op.sources.DictSource(ROWS, tag_columns=["unit"], source_id="spikes") + + +@op.function_pod(output_keys=["session"]) +def session_of(trial: int) -> str: + """A computed value later steps key on.""" + return f"s{trial // 20}" + + +def _sorted_rows(table: pa.Table) -> list[dict[str, Any]]: + return sorted(table.to_pylist(), key=lambda row: repr(sorted(row.items()))) + + +# ── what the output is ────────────────────────────────────────────────────── + + +def test_named_data_columns_become_referential_tags_in_the_order_given() -> None: + tagged = _plain().tag_by("label", "trial") + assert tagged.keys() == ( + ("unit", "label", "trial"), + ("trial", "amp", "label"), + ) + assert stream_references(tagged) == ("label", "trial") + for tag, data in tagged.iter_data(): + assert tag["label"] == data["label"] and tag["trial"] == data["trial"] + + +def test_the_column_is_stored_once_among_the_tags() -> None: + table = _plain().tag_by("trial").as_table() + assert table.column_names == ["unit", "trial", "amp", "label"] + + +def test_values_system_tags_and_source_info_are_unchanged() -> None: + source = _plain() + tagged = source.tag_by("trial") + before = source.as_table(all_info=True) + after = tagged.as_table(all_info=True) + unchanged = [ + name + for name in before.column_names + if name.startswith((constants.SYSTEM_TAG_PREFIX, constants.SOURCE_PREFIX)) + or name in ("unit", "trial", "amp", "label") + ] + assert set(unchanged) <= set(after.column_names) + assert _sorted_rows(after.select(unchanged)) == _sorted_rows( + before.select(unchanged) + ) + # The data's types and nullability are the input's. + for name in ("trial", "amp", "label"): + assert after.schema.field(name) == before.schema.field(name) + + +def test_an_existing_reference_is_kept() -> None: + tagged = _referential().tag_by("trial") + assert stream_references(tagged) == ("unit", "trial") + assert tagged.keys() == (("unit", "trial"), ("unit", "trial", "amp", "label")) + + +def test_names_as_strings_or_one_sequence_are_one_operator() -> None: + source = _plain() + by_strings = source.tag_by("trial", "label") + by_list = source.tag_by(["trial", "label"]) + by_tuple = source.tag_by(("trial", "label")) + assert ( + by_strings.content_hash() == by_list.content_hash() == by_tuple.content_hash() + ) + assert by_strings.pipeline_hash() == by_list.pipeline_hash() + + +def test_the_order_of_the_names_is_part_of_the_identity() -> None: + source = _plain() + forward = source.tag_by("trial", "label") + backward = source.tag_by("label", "trial") + assert forward.keys()[0] == ("unit", "trial", "label") + assert backward.keys()[0] == ("unit", "label", "trial") + assert forward.content_hash() != backward.content_hash() + assert forward.pipeline_hash() != backward.pipeline_hash() + assert TagBy(["trial", "label"]).identity_structure() == ( + "TagBy", + ["trial", "label"], + ) + + +@pytest.mark.parametrize("all_info", [False, True]) +def test_the_schema_is_predicted(all_info: bool) -> None: + source = _referential() + tag_by = TagBy(["label", "trial"]) + predicted = tag_by.output_schema(source, all_info=all_info) + assert predicted == tag_by.static_process(source).output_schema(all_info=all_info) + tag_schema, data_schema = predicted + assert list(tag_schema)[:3] == ["unit", "label", "trial"] + assert data_schema == source.output_schema()[1] + + +def test_the_data_order_is_the_inputs_and_the_predicted_one() -> None: + """The named columns stay where they are: tagging reorders no data column. + + The order of ``keys()`` and of the data schema of the computed stream + (``static_process``) equals the input's and the predicted one, also when the + names are given in another order than the data (CodeRabbit on PR #420). + """ + source = _referential() + tag_by = TagBy(["label", "trial"]) + computed = tag_by.static_process(source) + _, input_data = source.keys() + assert input_data == ("unit", "trial", "amp", "label") + assert computed.keys() == (("unit", "label", "trial"), input_data) + assert list(computed.output_schema()[1]) == list(input_data) + assert list(tag_by.output_schema(source)[1]) == list(input_data) + assert source.tag_by("label", "trial").keys() == computed.keys() + + +def test_each_row_keeps_its_data_context_key() -> None: + """The context column passes through: no row takes the default key instead.""" + source = _plain() + before = source.as_table(columns={"context": True}) + after = TagBy(["trial"]).static_process(source).as_table(columns={"context": True}) + assert ( + after.column(constants.CONTEXT_KEY).to_pylist() + == before.column(constants.CONTEXT_KEY).to_pylist() + ) + + +def test_zero_rows_have_the_types_of_rows() -> None: + source = _plain() + tagged = source.tag_by("trial") + empty = source.polars_filter(unit=99).tag_by("trial") + assert empty.as_table().schema == tagged.as_table().schema + assert empty.as_table().num_rows == 0 + + +def test_the_config_round_trips_and_is_registered_for_loading() -> None: + tag_by = TagBy(["label", "trial"]) + config = tag_by.to_config() + assert config["class_name"] == "TagBy" + assert config["config"] == {"columns": ["label", "trial"]} + for loaded in (TagBy.from_config(config), resolve_operator_from_config(config)): + assert isinstance(loaded, TagBy) + assert loaded.columns == ("label", "trial") + assert loaded.content_hash() == tag_by.content_hash() + assert loaded.pipeline_hash() == tag_by.pipeline_hash() + + +# ── after a function pod: a computed column keys a join and a group ───────── + + +def test_a_computed_column_becomes_a_tag_and_keys_a_join() -> None: + computed = TagBy("session")(session_of.pod(_referential())) + assert computed.keys() == (("unit", "session"), ("unit", "session")) + sessions = op.sources.DictSource( + [{"session": "s0", "room": "A"}, {"session": "s1", "room": "B"}], + tag_columns=["session"], + source_id="sessions", + ) + joined = computed.join(sessions) + assert set(stream_references(joined)) == {"unit", "session"} + rows = sorted(joined.as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["room"]) for r in rows] == [ + (1, "s0", "A"), + (2, "s1", "B"), + ] + + +def test_a_computed_column_becomes_a_tag_and_keys_a_group() -> None: + grouped = TagBy("session")(session_of.pod(_referential())).group_by(["session"]) + assert grouped.keys() == (("session",), ("unit", "session")) + rows = sorted(grouped.as_table().to_pylist(), key=lambda r: r["session"]) + assert [(r["session"], r["unit"]) for r in rows] == [("s0", [1]), ("s1", [2])] + + +def test_a_join_with_a_data_only_column_of_that_name_is_refused() -> None: + other = op.sources.DictSource( + [{"k": 1, "label": "a"}], + tag_columns=["k"], + tag_only_columns=True, + source_id="o", + ) + with pytest.raises(InputValidationError, match="tag_by"): + _ = _plain().tag_by("label").join(other) + + +def test_removing_the_data_role_afterwards_leaves_a_plain_tag_with_the_warning() -> ( + None +): + tagged = _plain().tag_by("label") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + plain = tagged.drop_data_columns(["label"]) + assert any(w.category is ReferenceMaterializationWarning for w in caught) + assert plain.keys() == (("unit", "label"), ("trial", "amp")) + + +def test_a_nullable_column_is_tagged_after_drop_nulls() -> None: + table = pa.table( + {"unit": pa.array([1, 2]), "label": pa.array(["a", None], pa.large_string())} + ) + source = op.sources.ArrowTableSource( + table, tag_columns=["unit"], tag_only_columns=True, source_id="n" + ) + with pytest.raises(InputValidationError, match="drop_nulls"): + _ = source.tag_by("label") + tagged = source.drop_nulls("label").tag_by("label") + assert tagged.as_table().column("label").to_pylist() == ["a"] + + +# ── refusals ──────────────────────────────────────────────────────────────── + + +@pytest.mark.parametrize( + ("columns", "error"), + [ + ({"trial", "label"}, TypeError), + (frozenset({"trial"}), TypeError), + ([], ValueError), + (["trial", "trial"], ValueError), + ([1], TypeError), + (["__meta"], InputValidationError), + ([f"{constants.SYSTEM_TAG_PREFIX}x"], InputValidationError), + ([f"{constants.SOURCE_PREFIX}x"], InputValidationError), + (["a::b"], InputValidationError), + ], + ids=[ + "set", + "frozenset", + "empty", + "repeated", + "not_a_string", + "meta", + "system_tag", + "source_info", + "separator", + ], +) +def test_bad_names_are_refused_at_construction(columns: Any, error: type) -> None: + with pytest.raises(error): + _ = TagBy(columns) + + +def test_the_stream_method_refuses_no_name_and_a_set() -> None: + with pytest.raises(TypeError): + _ = _plain().tag_by({"trial"}) # pyright: ignore[reportArgumentType] # the refusal is the test + with pytest.raises(ValueError, match="at least one"): + _ = _plain().tag_by([]) + + +@pytest.mark.parametrize( + ("name", "match"), + [ + ("missing", "'missing' is not a data column"), + ("unit", "'unit' is a plain tag"), + ], +) +def test_a_name_that_is_not_a_data_column_is_refused(name: str, match: str) -> None: + with pytest.raises(InputValidationError, match=match): + _ = _plain().tag_by(name) + + +def test_an_existing_referential_tag_is_refused() -> None: + with pytest.raises(InputValidationError, match="'unit' is already a referential"): + _ = _referential().tag_by("unit") + + +def _typed_source(column: pa.Array[Any] | pa.ChunkedArray[Any]) -> Any: + """Tag ``unit``; data ``v`` of the column's type, non-nullable.""" + schema = pa.schema( + [ + pa.field("unit", pa.int64(), nullable=False), + pa.field("v", column.type, False), + ] + ) + return op.sources.ArrowTableSource( + pa.Table.from_arrays([pa.array([1, 2], pa.int64()), column], schema=schema), + tag_columns=["unit"], + tag_only_columns=True, + source_id="typed", + ) + + +@pytest.mark.parametrize( + "column", + [ + pa.array([[1], [2]], pa.large_list(pa.int64())), + pa.array([{"a": 1}, {"a": 2}], pa.struct([("a", pa.int64())])), + pa.array([decimal.Decimal("1.5"), decimal.Decimal("2.5")]), + pa.array([datetime.time(1), datetime.time(2)]), + pa.array([datetime.timedelta(1), datetime.timedelta(2)]), + ], + ids=["list", "struct", "decimal", "time", "duration"], +) +def test_a_nested_or_untyped_type_is_refused(column: pa.Array[Any]) -> None: + with pytest.raises(InputValidationError, match="'v' cannot be a tag"): + _ = _typed_source(column).tag_by("v") + + +def test_a_logical_type_is_refused() -> None: + source = op.sources.DictSource( + [{"unit": 1, "path": Path("/a")}, {"unit": 2, "path": Path("/b")}], + tag_columns=["unit"], + tag_only_columns=True, + source_id="paths", + ) + with pytest.raises(InputValidationError, match="'path' cannot be a tag"): + _ = source.tag_by("path") + + +@pytest.mark.parametrize( + "column", + [ + pa.array([1, 2], pa.int8()), + pa.array([0.5, 1.5], pa.float32()), + pa.array([True, False]), + pa.array(["a", "b"], pa.string()), + pa.array([b"a", b"b"], pa.binary()), + pa.array([datetime.datetime(2026, 11, 16), datetime.datetime(2026, 11, 17)]), + pa.array([datetime.date(2026, 11, 16), datetime.date(2026, 11, 17)]), + ], + ids=[ + "int8", + "float32", + "bool", + "string", + "binary", + "timestamp", + "date", + ], +) +def test_a_scalar_type_is_accepted(column: pa.Array[Any]) -> None: + source = _typed_source(column) + tagged = source.tag_by("v") + assert stream_references(tagged) == ("v",) + assert tagged.as_table().schema.field("v") == source.as_table().schema.field("v") + + +# ── in a job ──────────────────────────────────────────────────────────────── + + +@ORCHESTRATORS +def test_in_a_job_a_function_node_stores_the_new_tag_marked( + tmp_path: Path, + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(database=database) + with job: + computed = session_of.pod(_referential(), label="session") + tagged = TagBy("session")(computed, label="tag_by") + double.pod(tagged, label="double") + job.run(orchestrator=orchestrator()) + assert job.nodes["tag_by"].keys() == (("unit", "session"), ("unit", "session")) + assert job.nodes["double"].keys() == (("unit", "session"), ("unit", "session", "y")) + rows = sorted(job.nodes["double"].as_table().to_pylist(), key=lambda r: r["unit"]) + assert [(r["unit"], r["session"], r["y"]) for r in rows] == [ + (1, "s0", "s0s0"), + (2, "s1", "s1s1"), + ] + marked = { + stored_references(database.get_all_records(path).schema) # pyright: ignore[reportOptionalMemberAccess] # every listed table has records + for path in database.list_table_paths() + if "_result" not in path + } + assert ("unit", "session") in marked + + job.save(tmp_path / "job.json") + loaded = PipelineJob.load(tmp_path / "job.json") + assert loaded.nodes["tag_by"].keys() == job.nodes["tag_by"].keys() + assert loaded.nodes["tag_by"].content_hash() == job.nodes["tag_by"].content_hash() + + +@op.function_pod(output_keys=["y"]) +def double(session: str) -> str: + """Reads the new tag's data column; ``unit`` falls through.""" + return session * 2 + + +@ORCHESTRATORS +def test_a_logged_tag_by_node_stores_its_new_tag_marked( + orchestrator: type[SyncPipelineOrchestrator | AsyncPipelineOrchestrator], +) -> None: + database = InMemoryArrowDatabase() + job = PipelineJob(name="ops", database=database) + with job: + _ = TagBy("trial")(_referential(), label="tag_by") + job.nodes["tag_by"].attach_databases( + pipeline_database=database.at("ops"), cache_mode=CacheMode.LOG + ) + job.run(orchestrator=orchestrator()) + paths = [ + p for p in database.list_table_paths() if p[-1] == OPERATOR_DB_SCHEMA_VERSION + ] + assert len(paths) == 1 + stored = database.get_all_records(paths[0]) + assert stored is not None + assert stored_references(stored.schema) == ("unit", "trial")