Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions changelog.d/ITL-1175.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
### Added

- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as
referential tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each
named column becomes one column in both roles, with its values and source info
unchanged; the new tags follow the stream's tags in the order given, and that order is
part of the operator's identity. System tags, record ids and every data column are
unchanged. A column a function computed may become a tag and then key a join or a
group. Refused, naming the column: a name that is not a data column of the input, a
name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is
one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a
nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered
for saved jobs and pipelines.

### Changed

- The error for a name that is a tag in one join input and a data column in the other
also offers `tag_by` as a way out.
48 changes: 48 additions & 0 deletions docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,58 @@ A function output named like a tag is an error when the pipeline is built. A fun
that does not read a referenced column keeps its result identity: retagging a source
does not make it compute again.

## Making a data column a tag: `tag_by`

A source declares its tags, but a value a later step should join, group or sort by is
often computed on the way: a session label, a trial index, a phase. `tag_by` gives
existing data columns the tag role: each becomes a referential tag, one column that is
a tag and the data column it was, with its values and source info unchanged.

```python
@op.function_pod(output_keys=["phase"])
def phase_of(time: int) -> str:
return "early" if time < 15 else "late"


phased = phase_of.pod(spikes).tag_by("phase")
print(phased.keys())
# (('unit', 'time', 'phase'), ('unit', 'time', 'phase'))

by_phase = phased.group_by(["phase"])
print(by_phase.keys())
# (('phase',), ('unit', 'time', 'phase'))
```

- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's
tags in the order given, and that order is part of the operator (`TagBy`): the names
are a list, never a set.
- Nothing else changes: no row is added, removed or reordered, the data columns keep
their names, types and values, every data column keeps its own source-info token,
and the system tags keep their names and values.
- No data element comes from a tag value: the operator adds a tag role to a column
that is already data, and creates or moves no value. Offering a tag's value as data
stays with `DerivedSource(tags_as_data=...)`.
- A computed column may become a tag, and then key a join or a group, as `phase` does
above.

`tag_by` refuses, naming the column: a name that is not a data column of the stream; a
name that is already a tag (plain or referential); a reserved name (the `__` prefix, a
system column, a name holding `::`); and a column whose type cannot be a tag. A tag is
one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`.
A nullable column (`T | None`) is refused: drop the rows with a null first,
`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs,
your own classes and logical types (`Path`, `UUID`, arrays) are refused.

`tag_by` adds the tag role; the drop operators remove a role. For a referential tag `x`,
dropping the tag (`drop_tag_columns(["x"])`) leaves `x` plain data, dropping the data
(`drop_data_columns(["x"])`) leaves `x` a plain tag, and dropping the column
(`drop_columns(["x"])`) removes it from both roles.

## Operators

| Operator | Referential tag `x` |
|---|---|
| `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. |
| `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. |
| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tuple of the input's non-key tags (then record ids), the same order for every data list of the group, so all of them stay aligned. |
| `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. |
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/design_issues/ITL-1023-RT1.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
**Severity:** medium
**Issue:** ITL-1023

The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` and `TagBy` (`tag_by`, ITL-1175), which gives a data column the tag role. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
23 changes: 23 additions & 0 deletions docs/dev/design_issues/ITL-1198-ST5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## `src/orcapod/core/streams/arrow_table_stream.py` — tag order

### ST5 — `keys()` lists tags in `tag_columns` order, `output_schema()` in physical order
**Status:** open
**Severity:** low
**Issue:** ITL-1198

`ArrowTableStream.keys()` gives the tag columns in the order of its `tag_columns` argument,
while `output_schema()` gives them in the table's physical column order (`_tag_schema` is the
table schema filtered to the tags). The data keys and the data schema both follow the physical
order and agree. A plain source shows the difference: over a table with columns `b, a, v` and
`tag_columns=["a", "b"]`, `keys()` lists `a, b` and `output_schema()` lists `b, a`.

`TagBy` (ITL-1175) meets it: it keeps the input's layout, so the data order stays the input's
(the predicted one), and its predicted tag order is the requested one, which `keys()` reports.
For names given in another order than the table's, `output_schema()` of the computed stream
lists the new tags in table order. No layout can give both orders, since one physical order
decides both schemas.

No hash depends on either order (`Schema` equality and the pipeline and content hashes are
order-insensitive), and column order is not part of a stream's identity (ITL-1112). The fix
is to decide one order for both methods in `ArrowTableStream`, without moving any stored
table's layout.
121 changes: 121 additions & 0 deletions docs/dev/specs/ITL-1175-tag-by.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# ITL-1175: mark existing data columns as referential tags (`tag_by`)

Status: approved by the coordinator on 2026-11-16, with two developer decisions (the
name `tag_by` / `TagBy`; a computed column may become a tag). Stacked on the referential
tags slice (ITL-1023, PR #342). The sections below are the design as sent; "Decisions"
records what changed.

## Decisions

- Name: `tag_by` (developer); the class follows the operator naming: `TagBy`.
- A computed column may become a tag and key a join or a group (developer); no lineage
check.
- Type rule: the slice has no type rule for a referential tag declared at a source (a
source accepts any type), so `TagBy` uses its own rule, in one function
(`referential_tags.require_taggable_type`) that the source path could adopt: `int`,
`float`, `bool`, `str`, `bytes`, `datetime`, `date`. `Decimal`, time of day and
duration are not in it: the type converter reads them as no Python type (`Any`), and
`Any` is refused. The source path is not changed here; the inconsistency is reported
in the PR.
- Removing a role afterwards (developer, November 16, 2026, ITL-1195): no new operators.
For a referential tag, dropping the tag leaves plain data, dropping the data leaves a
plain tag, dropping the column removes both roles. So `tag_by("x")` followed by
`drop_data_columns(["x"])` leaving a plain tag is intended behaviour. The docs
describe the three moves by role in one paragraph; final wording comes with ITL-1195.

## Public API

- Operator `orcapod.core.operators.TagBy(columns)`; stream method
`stream.tag_by(*columns, label=None)`. `s.tag_by("x", "y")` equals
`s.tag_by(["x", "y"])` (the convention of `select_columns`).
- The name is a choice for the developer. It lives in one place per surface: the class
name, the stream method, the serialization registry entry, the docs. Not
`reference_tags`: `ArrowTableStream.reference_tags` is already a property.
Alternatives: `as_tags` / `AsTags`, `mark_tags` / `MarkTags`.
- No `strict` flag: a name that cannot be marked is always an error.

## Order rule and identity

- The names are ordered: the output's user tags are the input's user tags, in their
order, then the named columns in the order given. So the order is part of the
operator, and its identity is `("TagBy", [names in the order given])`; its pipeline
identity is the same. A `set` / `frozenset` is refused (`TypeError`, unordered, as for
`record_id_columns`), as is an empty list and a repeated name (`ValueError`).

## Refusals

At construction (`ValueError` / `TypeError`, naming the column): no name; a non-string;
a repeated name; a set; the meta prefix (`require_names_not_meta`); a system column name
(`require_names_not_system`); a name holding `::` (`require_names_without_separator`).

At validation (`InputValidationError`, naming the column, raised when the pipeline is
built, sync and async alike):

- not a data column of the input (lists the input's data columns);
- already a tag: a referential tag ("already a referential tag"); a plain tag is never a
data column, so it is reported as a tag, not a data column;
- a type that cannot be a tag. Recommended rule: the column's Python type must be one of
`int`, `float`, `bool`, `str`, `bytes`, `datetime`, `date` (see Decisions). Refused: `T | None` (the error says to apply
`drop_nulls("x")` first, which narrows it to `T`), lists, sets, dicts, tuples, structs,
user classes and every logical type (`Path`, `UUID`, arrays...). Sources and function
outputs are non-nullable by default, so this costs nothing in the common case.
Conservative variant: also refuse `float` (equality on floats as a key).

## What the output is

- Tags: input user tags, then the new names; system tags unchanged (rule 1,
name-preserving: same column names, same values; record ids unchanged).
- Data: unchanged: same names, order, types, nullability, values; each column keeps its
own source-info token. No token is created or merged.
- Table layout: the column is stored once and stays where it is, as a source leaves a
referential tag at its place in the table: the data order is the input's (the
predicted one). (The first version moved it into the user tag group, which put the
new tags first among the data columns, unlike the prediction: CodeRabbit on PR #420.)
- Schema prediction (`unary_output_schema`): the tag schema gains the named columns, with
their data types, after the user tags and before the system tags; the data schema is
unchanged. Verified against `static_process(...).output_schema(...)`.
- Sync: the input table re-wrapped as an `ArrowTableStream` with the extra tags in
`tag_columns` and `reference_tags`: no Polars round trip, no cast.
- Async: no own `async_execute`: the base barrier path collects the rows and applies the
sync rule, so the rows are identical by construction. Added to
`test_async_sync_equivalence.py`. A streaming version can come later.

## Why "no data element comes from a tag value" holds

The operator adds a tag role to a data column; it creates no value and moves none. Every
output data column is the input data column of that name, with the same values and
source info, and no input tag becomes data. The column stays one physical column, as a
source's referential tag is. The direction the rule forbids (a tag value offered as data)
stays with `DerivedSource(tags_as_data=...)`.

## Downstream

- `Join` with a stream where `x` is a plain tag or a referential tag: the slice's rules
(one referential tag, one data column, merged `join_v1` token when referential on
several inputs). With a stream where `x` is data only: the existing tag/data error (its
remedy text gains "or mark it with `tag_by`").
- `GroupBy(["x"])`, `Batch`, `MapTags`, select/drop: the slice's rules for any
referential tag.
- Removing the data role afterwards (`tag_by("x").drop_data_columns("x")`) leaves a plain
tag holding the data value, with the `ReferenceMaterializationWarning`, as for any
referential tag. That is the data-to-tag direction (the rule holds), but it makes the
issue's out-of-scope "plain tag from a data column" reachable in two steps.
Recommendation: allow (existing slice behaviour, warned). Conservative variant: none
that is small; it would need tracking where a tag came from.
- Jobs: an operator node runs it in every cache mode; in `LOG` mode it stores its output
with the new tag marked (operator storage of referential tags came with #407, ITL-1019;
tested). A function node downstream stores the marker (ITL-1019). Saved jobs: `to_config` / `from_config`, and
the class is registered in `pipeline/serialization.py`.

## Open choice: may a computed column key a join?

Recommendation: yes. It is the issue's own typical use (a function computes a session
label that later steps join, group or sort by), and nothing in hashing or record
identity depends on where a tag's value came from. Conservative variant: `TagBy` refuses
a column a function pod produced. That needs a lineage walk upstream (through renames and
joins) in one helper called from validation; it is not small, and I recommend against it.

## Not in scope

Golden pins (identities provisional in 0.4.0rc1); a plain-tag variant; the reverse
direction; `MergeJoin` (ITL-1194).
2 changes: 2 additions & 0 deletions src/orcapod/core/operators/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
from .merge_join import MergeJoin
from .pick import Pick
from .semijoin import SemiJoin
from .tag_by import TagBy

__all__ = [
"Join",
Expand All @@ -36,4 +37,5 @@
"DropNulls",
"Pick",
"Index",
"TagBy",
]
4 changes: 2 additions & 2 deletions src/orcapod/core/operators/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ def _tag_data_error(name: str, tag_side: str, data_side: str) -> InputValidation
return InputValidationError(
f"column {name!r} is a tag in the {tag_side} input and a data column in the "
+ f"{data_side}, and a name cannot be both. Rename it in one input with "
+ "map_tags or map_data, or declare it a tag at the source that has it as "
+ "a data column."
+ "map_tags or map_data, or make it a tag of the input that has it as a data column (declare it "
+ "a tag at its source, or mark it with tag_by)."
)


Expand Down
Loading
Loading