Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ src/orcapod/
batch.py — Batch (group rows by count, types become list[T])
group_by.py — GroupBy (many→one reduction keyed on tag values)
column_selection.py — Select/Drop Tag/Data columns; SelectColumns/DropColumns (tag or data)
mappers.py — MapTags, MapData (rename columns)
mappers.py — MapTags, MapData, MapColumns (rename columns; strict by default)
filters.py — PolarsFilter
drop_nulls.py — DropNulls (drop rows with a null in named or all columns, T | None → T)
hashing/
Expand Down Expand Up @@ -658,8 +658,10 @@ ITL-1080): on the sync and the async path, in rows and in the predicted output s
raw ArrowTableStream's `__` columns and for the meta a cached pod's or function node's rows
carry. One rule, core/operators/meta_columns.py. A carry-forward mode is ITL-867.
The meta prefix (`__`) is reserved: ArrowTableStream and the source builder refuse a tag named
with it, and MapTags / MapData (targets) and Pick / Index (out) refuse it as a new name
(schema_utils.require_names_not_meta, at construction).
with it, and MapTags / MapData / MapColumns (targets) and Pick / Index (out) refuse it as a new name
(schema_utils.require_names_not_meta, at construction). The three mappers also refuse a new name
that a stream reads as a system column (system-tag or source-info prefix, the context key column;
schema_utils.require_names_not_system, ITL-1160).

Operator or derived source? (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
Expand Down
8 changes: 5 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,7 @@ src/orcapod/
│ ├── batch.py # Batch (group rows by count, types become list[T])
│ ├── group_by.py # GroupBy (many→one reduction keyed on tag values)
│ ├── column_selection.py # Select/Drop Tag/Data columns; SelectColumns/DropColumns (tag or data)
│ ├── mappers.py # MapTags, MapData (rename columns)
│ ├── mappers.py # MapTags, MapData, MapColumns (rename columns; strict by default)
│ ├── filters.py # PolarsFilter
│ └── drop_nulls.py # DropNulls (drop rows with a null in named or all columns, T | None → T)
├── hashing/
Expand Down Expand Up @@ -678,8 +678,10 @@ raw `ArrowTableStream`'s `__` columns and for the meta a cached pod's or functio
carry. The rule is in one place, `core/operators/meta_columns.py` (sync results through the
operator base classes, async rows through `WithoutMetaWriter`). A carry-forward mode is ITL-867.
The meta prefix (`__`) is reserved: `ArrowTableStream` and the source builder refuse a tag named
with it, and `MapTags` / `MapData` (targets) and `Pick` / `Index` (`out`) refuse it as a new name
(`schema_utils.require_names_not_meta`, at construction).
with it, and `MapTags` / `MapData` / `MapColumns` (targets) and `Pick` / `Index` (`out`) refuse it as a new name
(`schema_utils.require_names_not_meta`, at construction). The three mappers also refuse a new name
that a stream reads as a system column (system-tag or source-info prefix, the context key column;
`schema_utils.require_names_not_system`, ITL-1160).

**Operator or derived source?** (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
Expand Down
14 changes: 14 additions & 0 deletions changelog.d/ITL-1160.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
### Breaking changes and migration

- `map_tags` / `MapTags` and `map_data` / `MapData` refuse, by default, a mapping key that is not a column of the kind they rename (ITL-1160). Before, such a key was accepted and renamed nothing, which hid mistakes such as `map_tags` of a column that had become data after `group_by`. The call now raises `InputValidationError` naming the missing names and the columns of that kind; a name that is a column of the other kind says so (`'age' is a data column: use map_data`). The check runs when the operator is called on a stream, before any row is processed, and on the async path against the first row. A raw stream's meta (`__`) column is not a data column to rename. Migration: fix the name, use the other method (or `map_columns`), or pass `strict=False` to keep the old behaviour.
- `strict` is part of the identity of `MapTags` and `MapData`, as it is of the column selection operators: an operator's identity is its full configuration (developer decision, ITL-1160). Their identity is now `(class name, name_map, drop_unmapped, strict)`, and it is saved in `to_config()`. This moves the content and pipeline hash of every `MapTags` and `MapData` step and of everything downstream of one, and with them the paths of the tables that are scoped by those hashes.
- The stream methods `map_tags`, `map_data` and `map_columns` take the renames dict as their only positional argument, and `drop_unmapped`, `strict` and `label` are keyword-only (ITL-1160, ITL-1171). `stream.map_tags(name_map, True)` and `stream.map_tags(name_map, False, "label")` raise `TypeError`: write `drop_unmapped=True`, `label="label"`. `stream.map_data(name_map={...})` no longer names the dict (it reads as a rename of a column called `name_map` and raises `TypeError`): pass the dict by position. The operator classes `MapTags`, `MapData` and `MapColumns` keep their arguments.

### Added

- `stream.map_columns(name_map)` and the operator `MapColumns` rename tag and data columns in one step (ITL-1160). A renamed tag stays a tag and a renamed data column stays data, with its source-info column; system tags pass through. `drop_unmapped=True` drops every tag and data column not in the mapping. As with `map_tags` and `map_data`, a name that is neither a tag nor a data column is refused unless `strict=False`, a new name taken by a column that is not renamed, or given to two columns, is refused, a `__` new name is refused when the operator is built, and a result with no data column is refused. Like `SelectColumns`, it carries no data meta (`__`) column of the input. Its identity is `("MapColumns", name_map, drop_unmapped, strict)`.
- `map_tags`, `map_data` and `map_columns` take the renames as keyword arguments too: `stream.map_data(age="age_weeks")` builds the same step, with the same hashes, as `stream.map_data({"age": "age_weeks"})` (ITL-1171). Give one dict or keyword arguments, not both (`TypeError`). `drop_unmapped`, `strict` and `label` are options, never renames: a column with one of those names, or a name that is not a Python identifier, is renamed through the dict. An old or new name that is not a string raises `TypeError` in both forms.

### Fixed

- `MapTags`, `MapData` and `MapColumns` refuse, when built, a new name that a stream reads as a system column: one that starts with the system-tag prefix (`_tag_`) or the source-info prefix (`_source_`), or the context key column (`_context_key`) (ITL-1160). Before, such a rename was accepted and the column then vanished from the stream or the stream failed on read with `No data columns found`.
27 changes: 27 additions & 0 deletions design_issues/ITL-1165-RN1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## Cross-cutting

### RN1 — Reserved `_` column names: sources silently drop them, `Pick` / `Index` accept them as new names
**Status:** open
**Severity:** medium
**Issue:** ITL-1165

Columns whose names a stream reads as system columns are handled inconsistently (found on
November 16, 2026 while fixing ITL-1160, probed on that branch):

* `ArrowTableSource` silently drops a data column named `_foo`, `_tag_y`, `_source_y`,
`_content_hash` or `_context_key`: the source builds without error and `keys()` lists only
the other data columns.
* `Index(..., out=...)` accepts `_tag_y`, `_source_y` and `_context_key`; `keys()` then lists
the column but `as_table()` does not have it. `Pick` has the same out-name check (only the
meta prefix `__` is refused, `schema_utils.require_names_not_meta`).
* ITL-1160 fixed the mappers only: `MapTags`, `MapData` and `MapColumns` refuse, when built, a
new name that starts with the system-tag prefix (`_tag_`) or the source-info prefix
(`_source_`), or is the context key column (`_context_key`)
(`schema_utils.require_names_not_system`).
* Open question: is a bare `_` name reserved at all? A source drops `_foo` and `_content_hash`,
while `MapData({"x": "_foo"})` and `Index(out="_foo")` give a working data column, and so
does renaming to `_content_hash`.

To decide: which names are reserved; then a source refuses such a column with an error naming
it instead of dropping it, and `Pick` / `Index` refuse a reserved `out` name through the same
helper as the mappers.
2 changes: 2 additions & 0 deletions docs/api/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ Operators perform structural transformations on streams without inspecting or sy

::: orcapod.core.operators.MapData

::: orcapod.core.operators.MapColumns

::: orcapod.core.operators.PolarsFilter

::: orcapod.core.operators.Pick
Expand Down
55 changes: 48 additions & 7 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ function pod's `__computed`, that describe how an input row was produced. An ope
output row is produced by the operator, so its `Data` holds none of them and its predicted
output schema lists none, on the sync and the async path alike. A way to carry them forward
across operators is an open design question (ITL-867). The `__` prefix is reserved for meta
columns: a stream and a source refuse a tag column named with it, and `MapTags` and `MapData`
refuse it as the new name of a tag or data column.
columns: a stream and a source refuse a tag column named with it, and `MapTags`, `MapData`
and `MapColumns` refuse it as the new name of a tag or data column.

## Operator categories

Expand Down Expand Up @@ -385,8 +385,11 @@ Selecting no tag column is allowed.

- **`MapTags(name_map={"old_name": "new_name"})`** -- rename tag columns
- **`MapData(name_map={"old_name": "new_name"})`** -- rename data columns
- **`MapColumns(name_map={"old_name": "new_name"})`** -- rename columns, tag or data, in
one step

Columns that are not in `name_map` are kept unless you pass `drop_unmapped=True`.
Columns that are not in `name_map` are kept unless you pass `drop_unmapped=True`. A renamed
data column's source-info column is renamed with it, and system tags are kept.

```python
from orcapod.operators import MapData
Expand All @@ -395,6 +398,43 @@ renamed = MapData(name_map={"age": "age_weeks"})(source)
print(renamed.keys()[1]) # ('age_weeks', 'weight')
```

Streams offer them as `map_tags`, `map_data` and `map_columns`. `map_columns` renames tag and
data columns in one call; a renamed tag stays a tag and a renamed data column stays data:

```python
renamed = sessions.map_columns({"session": "session_no", "age": "age_weeks"})
print(renamed.keys()) # (('subject_id', 'session_no'), ('age_weeks', 'weight'))
```

The three methods also take the renames as keyword arguments, `old="new"`. Both forms build
the same operator, with the same hashes:

```python
by_keyword = sessions.map_columns(session="session_no", age="age_weeks")
print(by_keyword.pipeline_hash() == renamed.pipeline_hash()) # True
```

Give the renames as one dict (the only positional argument) or as keyword arguments, not both.
`drop_unmapped`, `strict` and `label` are options, never renames: a column with one of those
names, or a name that is not a Python identifier (`"spike count"`), is renamed through the dict.

A name that is not a column the operator renames raises `InputValidationError` when the
operator is called: a tag name for `map_data`, a data name for `map_tags`, and for all three a
name that is no column at all. The message names the columns that exist and, for a column of
the other kind, the method that renames it. Pass `strict=False` to ignore such names instead.

```python
from orcapod.errors import InputValidationError

try:
sessions.map_tags({"age": "age_weeks"})
except InputValidationError as error:
print(error)
# MapTags: ['age'] not among the tag columns ['subject_id', 'session'] (use strict=False to ignore names that are not tag columns); 'age' is a data column: use map_data
```

A new name already taken by a column that is not renamed is refused too.

### PolarsFilter

Filter rows using Polars expressions:
Expand Down Expand Up @@ -703,8 +743,8 @@ operators or function pods.

Every stream also exposes the operators as methods, which reads well when chaining:
`join`, `semi_join`, `batch`, `group_by`, `select_tag_columns`, `select_data_columns`,
`drop_tag_columns`, `drop_data_columns`, `map_tags`, `map_data`, `polars_filter`,
`drop_nulls`, `pick` and `index`.
`drop_tag_columns`, `drop_data_columns`, `select_columns`, `drop_columns`, `map_tags`,
`map_data`, `map_columns`, `polars_filter`, `drop_nulls`, `pick` and `index`.

```python
import polars as pl
Expand All @@ -716,8 +756,9 @@ for tag, data in adults.iter_data():
```

!!! note
The fluent `map_tags` and `map_data` methods and the `MapTags` and `MapData` operator
classes all keep the columns they do not rename. Pass `drop_unmapped=True` to drop them.
The fluent `map_tags`, `map_data` and `map_columns` methods and the `MapTags`, `MapData`
and `MapColumns` operator classes all keep the columns they do not rename. Pass
`drop_unmapped=True` to drop them.

## How it connects to other concepts

Expand Down
1 change: 1 addition & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,7 @@ The table below shows what each class returns from `identity_structure()`.
| Operators (unary) | `(operator_class_name, upstream_stream)` | Stream reference resolved via `content_resolver` |
| Operators (binary/N-ary) | `(operator_class_name, argument_symmetry(streams))` | `frozenset` for commutative (Join, MergeJoin); `tuple` for ordered (SemiJoin) |
| `SelectTagColumns`, `SelectDataColumns`, `DropTagColumns`, `DropDataColumns`, `SelectColumns`, `DropColumns` | `(operator_class_name, sorted unique column names as a list, strict)` | The operators act on a set of names: a list, a tuple, a set, a single string, a repeated or a reordered name list give one hash (ITL-942) |
| `MapTags`, `MapData`, `MapColumns` | `(operator_class_name, name_map, drop_unmapped, strict)` | An operator's identity is its full configuration, `strict` included, as for the column selection operators (ITL-1160) |

**Inputs of a commutative operator are a set (kept by decision, ITL-942).** A pipeline hash
covers the structure of the pipeline, not only its schemas. `MergeJoin()(m1, m1)`, a stream
Expand Down
23 changes: 17 additions & 6 deletions docs/tutorial/03-operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,19 @@ print(renamed.keys())
# (('subject_id',), ('sex', 'age', 'genotype'))
```

Columns you do not rename are kept. Pass `drop_unmapped=True` to keep only the renamed
ones:
or the same renames as keyword arguments, which build the same step:

```python
print(subjects.map_data(age_weeks="age").keys())
# (('subject_id',), ('sex', 'age', 'genotype'))
```

A name that is not a Python identifier, such as `"spike count"`, needs the dictionary.

Columns you do not rename are kept. A name that is not a column of that kind is refused: a
`map_data` of a tag column raises `InputValidationError` and says to use `map_tags`.
`map_columns` renames tag and data columns in one call. Pass `drop_unmapped=True` to keep only
the renamed ones:

```python
print(subjects.map_data({"age_weeks": "age"}, drop_unmapped=True).keys())
Expand Down Expand Up @@ -261,10 +272,10 @@ side_by_side = imec0.map_data({"n_spikes": "n_imec0"}).join(
)
pair_tags = ["subject_id", "session", "unit"]
print(side_by_side.as_pandas_df().sort_values(pair_tags, ignore_index=True).head(3))
# subject_id session unit n_imec1 n_imec0
# 0 mouse_01 s1 0 215 149
# 1 mouse_01 s1 1 299 326
# 2 mouse_01 s1 2 135 662
# subject_id session unit n_imec0 n_imec1
# 0 mouse_01 s1 0 149 215
# 1 mouse_01 s1 1 326 299
# 2 mouse_01 s1 2 662 135
```

**`MergeJoin`**, when they are repeated measurements of the same quantity and you want them
Expand Down
4 changes: 2 additions & 2 deletions orcapod-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ Keep only specified tag or data columns. Optional `strict` mode raises on missin
### DropTagColumns / DropDataColumns
Remove specified tag or data columns. `DropDataColumns` also removes associated source-info columns.

### MapTags / MapData
Rename tag or data columns via a name mapping. `MapData` automatically renames associated source-info columns. Optional `drop_unmapped` mode removes columns not in the mapping.
### MapTags / MapData / MapColumns
Rename tag or data columns via a name mapping; `MapColumns` renames both in one step, and a tag stays a tag. A renamed data column's source-info column is renamed with it. Optional `drop_unmapped` mode removes columns not in the mapping. A name that is not a column of the mapped kind is refused by default (`strict=True`, part of the identity; ITL-1160).

### PolarsFilter
Applies Polars filtering predicates to rows. Output schema is unchanged from input.
Expand Down
4 changes: 2 additions & 2 deletions src/orcapod/contexts/data/v1.json
Original file line number Diff line number Diff line change
Expand Up @@ -235,13 +235,13 @@
"added": {
"arrow-01.json": "c66fa55619178f2631d04232555cd58059813e97e9ecd2b9de9e63699e60c964",
"datagram-01.json": "598d4a2851819d76ba674e189222e61733ca1861ca82a59e990a8ee5e1b0a7b3",
"pipeline-01.json": "f6fd38d591b20876725d4050ce4f11ad541460e9913262b05d61c1b398770711",
"pipeline-01.json": "ec0049a39e85645d59d9a276a17528bfa64a91f349aa0094052a6d55bc930be5",
"schema-01.json": "79aa2a068e0724891cf43ee528ce8f569e65f08864faa5e34773bfc69ac9576e",
"semantic-01.json": "8e5d9dfff461f9f1d6bdd49a3019e7bf6122e75d799d725b17531d31b080003c",
"torch-01.json": "af2e8b68d66dffaa37e1d0ad75acd42f595d65b9fae00ddcde81bf9e13a265c8"
},
"previous": null,
"digest": "2a2dcc4dd5e52a906f79f0ca3a26989ef0869b5531882831da004570f70c4a4d"
"digest": "0a789cf9e8d1df9fb762d6dc2fabffcd3383cd6a21ef147e8c6583f963280c2e"
}
]
},
Expand Down
3 changes: 2 additions & 1 deletion src/orcapod/core/operators/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from .group_by import GroupBy
from .index import Index
from .join import Join
from .mappers import MapData, MapTags
from .mappers import MapColumns, MapData, MapTags
from .merge_join import MergeJoin
from .pick import Pick
from .semijoin import SemiJoin
Expand All @@ -23,6 +23,7 @@
"SemiJoin",
"MapTags",
"MapData",
"MapColumns",
"Batch",
"GroupBy",
"SelectTagColumns",
Expand Down
Loading
Loading