Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
8b73ba7
feat(operators): TagBy / stream.tag_by makes data columns referential…
Nov 17, 2026
6aa2314
docs(referential-tags): describe the three moves by role after tag_by…
Nov 17, 2026
e91d256
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
eb32ab1
test(tag_by): a logged TagBy operator node stores its new tag marked …
Nov 17, 2026
a6b4a17
docs(referential-tags): select and drop act on a referential tag by r…
Nov 17, 2026
3ab814e
refactor(referential_tags): remove ReferenceMaterializationWarning (I…
Nov 17, 2026
194254e
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
8cd398f
fix(tag_by): keep the input's column layout and data context (ITL-1175)
Nov 17, 2026
424f731
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
5292a60
revert(referential_tags): keep ReferenceMaterializationWarning (ITL-1…
Nov 17, 2026
83ff606
fix(tag_by): keep each row's data context key; correct the spec on LO…
Nov 17, 2026
35579f8
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
01c3b72
docs(design_issues): log ST5, the two tag orders of ArrowTableStream …
Nov 17, 2026
447597b
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
5dd0634
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
378eca2
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
05deee1
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
a373ca6
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
816d8c8
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
bd8f020
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
e43c609
docs(design_issues): log AB1, the async barrier input's column order …
Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions changelog.d/ITL-1175.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
### Added

- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as
referential tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each
named column becomes one column in both roles, with its values and source info
unchanged; the new tags follow the stream's tags in the order given, and that order is
part of the operator's identity. System tags, record ids and every data column are
unchanged. A column a function computed may become a tag and then key a join or a
group. Refused, naming the column: a name that is not a data column of the input, a
name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is
one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a
nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered
for saved jobs and pipelines.

### Changed

- The error for a name that is a tag in one join input and a data column in the other
also offers `tag_by` as a way out.
13 changes: 13 additions & 0 deletions changelog.d/ITL-1195.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
### Changed

- The select and drop operators act on a referential tag by role (developer decision,
November 16, 2026, ITL-1195), as documented in "Moving a column between roles": for a
referential tag `x`, dropping the tag (`drop_tag_columns`) removes the tag role only
and leaves `x` plain data with its source info; dropping the data (`drop_data_columns`)
removes the data role only and leaves `x` a plain tag with the same values; dropping
the column (`drop_columns`) removes it from both roles. `select_tag_columns`,
`select_data_columns` and `select_columns` follow the same rule for the names they do
not select. This was already the slice's behaviour; it is now the stated rule, with
tests by role. Removing the data role still emits `ReferenceMaterializationWarning`,
as on the slice; removing the tag role or the column is silent. No second name (such as
`untag`) is added.
93 changes: 91 additions & 2 deletions docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,103 @@ A function output named like a tag is an error when the pipeline is built. A fun
that does not read a referenced column keeps its result identity: retagging a source
does not make it compute again.

## Moving a column between roles

A referential tag is one column in two roles, a tag and a data column. Four steps move a
column between the roles, and none of them changes, copies or creates a value:

| Step | Data column `x` | Referential tag `x` | Plain tag `x` |
|---|---|---|---|
| `tag_by("x")` | becomes a referential tag | refused: already a tag | refused: not a data column |
| `drop_tag_columns(["x"])` | refused: not a tag | the tag role goes: `x` is plain data | removed |
| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag, with a `ReferenceMaterializationWarning` | refused: not a data column |
| `drop_columns(["x"])` | removed | removed from both roles | removed |

The select operators follow the same rule by role: `select_tag_columns` removes the tag
role of every tag it does not name, `select_data_columns` the data role of every data
column it does not name, and `select_columns` keeps exactly the columns it names, in
both roles. (The drop and select refusals are those of a missing name: `strict=False`
ignores the name instead.)

For a referential tag `x`, in plain words:

- Dropping or selecting away the **data** of `x` leaves a plain tag `x`, and issues a
`ReferenceMaterializationWarning`: the tag no longer refers to a data column, so its
values now live only in the tag (see below to turn the warning off).
- Dropping or selecting away the **tag** `x` leaves plain data `x`, silently.
- Dropping the **column** `x` removes it from both roles, silently.

```python
print(spikes.drop_tag_columns(["time"]).keys())
# (('unit',), ('unit', 'time', 'amp'))
print(spikes.drop_data_columns(["time"]).keys())
# (('unit', 'time'), ('unit', 'amp'))
print(spikes.drop_columns(["time"]).keys())
# (('unit',), ('unit', 'amp'))
```

- System tags and record ids never change.
- Source info belongs to the data role: dropping the tag keeps the column's source-info
token, dropping the data removes it (a plain tag carries none). The row's provenance
is still its system tags (its source id and record id).
- No data element comes from a tag value: dropping the tag leaves a data column that was
data before; dropping the data leaves a tag whose values came from a data column.
- A stream keeps at least one data column: dropping the data role of the last one is
refused.
- A stored table marks exactly the referential tags of what it stores, so a column moved
out of one role is stored unmarked.

### Adding the tag role: `tag_by`

A source declares its tags, but a value a later step should join, group or sort by is
often computed on the way: a session label, a trial index, a phase. `tag_by` gives
existing data columns the tag role: each becomes a referential tag, one column that is
a tag and the data column it was, with its values and source info unchanged.

```python
@op.function_pod(output_keys=["phase"])
def phase_of(time: int) -> str:
return "early" if time < 15 else "late"


phased = phase_of.pod(spikes).tag_by("phase")
print(phased.keys())
# (('unit', 'time', 'phase'), ('unit', 'time', 'phase'))

by_phase = phased.group_by(["phase"])
print(by_phase.keys())
# (('phase',), ('unit', 'time', 'phase'))
```

- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's
tags in the order given, and that order is part of the operator (`TagBy`): the names
are a list, never a set.
- Nothing else changes: no row is added, removed or reordered, the data columns keep
their names, types and values, every data column keeps its own source-info token,
and the system tags keep their names and values.
- No data element comes from a tag value: the operator adds a tag role to a column
that is already data, and creates or moves no value. Offering a tag's value as data
stays with `DerivedSource(tags_as_data=...)`.
- A computed column may become a tag, and then key a join or a group, as `phase` does
above.

`tag_by` refuses, naming the column: a name that is not a data column of the stream; a
name that is already a tag (plain or referential); a reserved name (the `__` prefix, a
system column, a name holding `::`); and a column whose type cannot be a tag. A tag is
one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`.
A nullable column (`T | None`) is refused: drop the rows with a null first,
`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs,
your own classes and logical types (`Path`, `UUID`, arrays) are refused.

## Operators

| Operator | Referential tag `x` |
|---|---|
| `TagBy` / `tag_by` | Makes the data column `x` a referential tag (above); everything else is unchanged. |
| `Join` | Plain with plain: as before. Plain with referential, or referential with referential: a referential tag and **one** data column `x`. When `x` is referential on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. |
| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tuple of the input's non-key tags (then record ids), the same order for every data list of the group, so all of them stay aligned. |
| `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. |
| `SelectTagColumns`, `DropTagColumns` | Removing the tag leaves `x` a data column. |
| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `ReferenceMaterializationWarning`. |
| `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. |
| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. |
| `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. |
| `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. |
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/design_issues/ITL-1023-RT1.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
**Severity:** medium
**Issue:** ITL-1023

The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `MergeJoin` (since ITL-1194), `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` and `TagBy` (`tag_by`, ITL-1175), which gives a data column the tag role. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
23 changes: 23 additions & 0 deletions docs/dev/design_issues/ITL-1198-ST5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## `src/orcapod/core/streams/arrow_table_stream.py` — tag order

### ST5 — `keys()` lists tags in `tag_columns` order, `output_schema()` in physical order
**Status:** open
**Severity:** low
**Issue:** ITL-1198

`ArrowTableStream.keys()` gives the tag columns in the order of its `tag_columns` argument,
while `output_schema()` gives them in the table's physical column order (`_tag_schema` is the
table schema filtered to the tags). The data keys and the data schema both follow the physical
order and agree. A plain source shows the difference: over a table with columns `b, a, v` and
`tag_columns=["a", "b"]`, `keys()` lists `a, b` and `output_schema()` lists `b, a`.

`TagBy` (ITL-1175) meets it: it keeps the input's layout, so the data order stays the input's
(the predicted one), and its predicted tag order is the requested one, which `keys()` reports.
For names given in another order than the table's, `output_schema()` of the computed stream
lists the new tags in table order. No layout can give both orders, since one physical order
decides both schemas.

No hash depends on either order (`Schema` equality and the pipeline and content hashes are
order-insensitive), and column order is not part of a stream's identity (ITL-1112). The fix
is to decide one order for both methods in `ArrowTableStream`, without moving any stored
table's layout.
22 changes: 22 additions & 0 deletions docs/dev/design_issues/ITL-1205-AB1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## `src/orcapod/core/operators/static_output_pod.py` — async barrier input

### AB1 — The async barrier path rebuilds its input with data-tags first and in the default data context
**Status:** open
**Severity:** medium
**Issue:** ITL-1205

An operator without its own `async_execute` collects its input rows and rebuilds them as a
stream (`StaticOutputOperatorPod._materialize_input_stream`, through `_rows_to_table`), then
applies its sync rule. Two properties of that rebuild make the async result differ from the
sync one, for every operator on this path.

1. Data order: `_rows_to_table` takes a data-tag (a tag that is also a data column, today a
referential tag) from the tags and lays it out with them, so it leads the data columns.
On the sync path a data-tag stays where the input table has it. For an input with data
`a, x, y` and `x` a data-tag, `PolarsFilter` (and `TagBy`) give data `a, x, y` on the sync
path and `x, a, y` on the async path.
2. Data context: the rebuilt input has neither the rows' data context nor a context column,
so it takes the default context. Only one data context is registered today, so this is
not observable yet.

The fix must keep the row-built layout of stored tables (`_in_row_order`) unchanged.
Loading
Loading