Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -548,7 +548,10 @@ no input tag name may appear among the data columns.
Two operators reduce row count many→one. Batch groups consecutive rows by count; GroupBy
groups by tag values: it collapses the N rows sharing a tag tuple into one row with
list-valued members. Neither synthesizes new data values, and GroupBy keys only on tags,
never on data. Its non-key tags stay tags, list-valued, as Batch keeps them.
never on data. GroupBy keeps only its `by` tags and drops every other tag (developer,
November 13, 2026, ITL-1070): a list of the members' values is not a key, as SQL drops a
non-key column after GROUP BY. Batch has no key and keeps every tag, list-valued (developer,
same day): its lists are what tells a batch's members apart.

Filters are the operators that read data columns (ITL-943): PolarsFilter evaluates its
predicate on values, and DropNulls reads only whether a value is present (its validity),
Expand Down Expand Up @@ -657,10 +660,10 @@ generation of hashing; rows of the current data context are the current entries.
2. Name-extending — multi-input ops. System tag column name gets
::{pipeline_hash}:{canonical_position} appended. Commutative operators sort by
pipeline_hash and sort system tag values per row.
3. Reducing — many→one ops (Batch, GroupBy). User tag and data columns become list[T] and
stay in their own group (a tag stays a tag; only the scalar keys of a GroupBy are not
list-wrapped), and
source-info columns become list[str] with one element per member. System tag columns must
3. Reducing — many→one ops (Batch, GroupBy). Data columns become list[T]; Batch makes every
user tag list[T] and keeps it a tag, while GroupBy keeps its scalar keys and drops every
other user tag (ITL-1070). No tag ever moves to the data group. Source-info columns become
list[str] with one element per member. System tag columns must
stay scalar, because _build_record_id_preimage (core/nodes/function_node.py) hashes them
directly to derive record identity. They fold to a deterministic digest via
arrow_utils.fold_system_tag_values and their column name gains ::{pipeline_hash}.
Expand Down Expand Up @@ -793,10 +796,14 @@ data. The rules:
Batch — group before batching, not after). Members sort by non-key tag values with system
record_id as tiebreaker; groups emit in key order, so reordered input yields a
byte-identical table.
- GroupBy's output tag columns are its input's tag columns, in input order: keys scalar,
every other tag list[T] (ITL-955). A grouped stream can be grouped again on a coarser key;
its list-valued tags then nest (list[list[T]]) and still order the members, so the member
sort key (_null_last_key) is recursive and sorts nulls last at every depth.
- GroupBy's output tag columns are its `by` columns, scalar, in the input's tag order; every
other tag is dropped (ITL-1070; ITL-955 had kept them as list[T] tags, never released). The
dropped tags still order the members, so the data lists are as before. Their values are not
carried: a step that needs them makes the column data before grouping (at the source, or
DerivedSource(tags_as_data=...)). A list-valued tag of the input (a list-typed source tag)
can still order members, so the member sort key (_null_last_key) is
recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser
key orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- Data source-info values may be str, None, or list[...], nested to any depth; Arrow/Python
types are derived from the value, with None mapping to large_string. That rule has one
definition, arrow_utils.source_info_arrow_type, and applies only to tokens supplied as
Expand Down
26 changes: 16 additions & 10 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -567,8 +567,10 @@ no input tag name may appear among the data columns.
Two operators reduce row count many→one. `Batch` groups consecutive rows by count; `GroupBy`
groups by tag values: it collapses the N rows sharing a tag tuple into one row with
list-valued members. Neither synthesizes new data values — every emitted element came from an
input row — and `GroupBy` keys only on tags, never on data. Its non-key tags stay tags,
list-valued, as `Batch` keeps them.
input row — and `GroupBy` keys only on tags, never on data. `GroupBy` keeps only its `by`
tags and drops every other tag (developer, November 13, 2026, ITL-1070): a list of the members'
values is not a key, as SQL drops a non-key column after `GROUP BY`. `Batch` has no key and keeps
every tag, list-valued (developer, same day): its lists are what tells a batch's members apart.

Filters are the operators that read data columns (ITL-943): `PolarsFilter` evaluates its
predicate on values, and `DropNulls` reads only whether a value is present (its validity),
Expand Down Expand Up @@ -686,10 +688,10 @@ current data context are the current entries.
2. **Name-extending** — multi-input ops (join, merge join). Each input's system tag column
name gets `::{pipeline_hash}:{canonical_position}` appended. Commutative operators
canonically order inputs by `pipeline_hash` and sort system tag values per row.
3. **Reducing** — many→one ops (`Batch`, `GroupBy`). User tag and data columns become
`list[T]` and stay in their own group (a tag stays a tag; only the scalar keys of a
`GroupBy` are not list-wrapped), and source-info columns become `list[str]` with one element
per member. System
3. **Reducing** — many→one ops (`Batch`, `GroupBy`). Data columns become `list[T]`; `Batch`
makes every user tag `list[T]` and keeps it a tag, while `GroupBy` keeps its scalar keys and
drops every other user tag (ITL-1070). No tag ever moves to the data group. Source-info
columns become `list[str]` with one element per member. System
tag columns must stay **scalar**, because `_build_record_id_preimage`
(`core/nodes/function_node.py`) hashes them directly to derive record identity. They fold
to a deterministic digest via `arrow_utils.fold_system_tag_values` and their column name
Expand Down Expand Up @@ -839,10 +841,14 @@ data. The rules:
tag values with the system `record_id` as tiebreaker, so the hashed lists are stable across
runs. Groups themselves are emitted in key order, so a reordered input yields a
byte-identical table.
- `GroupBy`'s output tag columns are its input's tag columns, in input order: keys scalar,
every other tag `list[T]` (ITL-955). A grouped stream can be grouped again on a coarser key;
its list-valued tags then nest (`list[list[T]]`) and still order the members, so the member
sort key (`_null_last_key`) is recursive and sorts nulls last at every depth.
- `GroupBy`'s output tag columns are its `by` columns, scalar, in the input's tag order; every
other tag is dropped (ITL-1070; ITL-955 had kept them as `list[T]` tags, never released). The
dropped tags still order the members, so the data lists are as before. Their values are not
carried: a step that needs them makes the column data before grouping (at the source, or
`DerivedSource(tags_as_data=...)`). A list-valued tag of the input (a list-typed source tag)
can still order members, so the member sort key (`_null_last_key`) is
recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser key
orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- `Data` source-info values may be `str`, `None`, or `list[...]`, nested to any depth; the
Arrow and Python types are derived from the value, with `None` mapping to `large_string`.
That rule has one definition, `arrow_utils.source_info_arrow_type`, and applies only to
Expand Down
50 changes: 50 additions & 0 deletions changelog.d/ITL-1070.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
### Breaking changes and migration

#### `GroupBy` keeps only its `by` tags (ITL-955, ITL-1070)

`GroupBy` used to turn every tag that was not a grouping key into a list-valued **data** column,
with null source info, where a function pod could read it. That broke the rule that separates an
operator from a function pod: no data element may come from a tag value. A tag that is not in
`by` is now **dropped**: a list of the members' values is not a key for the group, just as SQL
drops a non-key, non-aggregated column after `GROUP BY`. The output's tag columns are the `by`
columns, scalar, in the input's tag order. `Batch` is unchanged: it has no key and keeps every
tag, list-valued.

| `GroupBy(by=["subject"])` on tags `subject, session, probe`, data `path` | v0.2.0 | Now |
|---|---|---|
| Tag columns | `subject` | `subject` |
| Data columns | `session: list[int]`, `probe: list[str]`, `path: list[str]` | `path: list[str]` |

**Member tag values are not carried.** If a later step needs to know which member each element
came from, make the column data *before* grouping: declare it as data at the source, or use
`DerivedSource(origin, tags_as_data={"probe": "probe_name"})` and group that.

- **What to change.** A function that took a non-key tag as a parameter
(`def f(probe: list[str], path: list[str])`) now fails input-schema validation; drop the
parameter, or offer the tag as data before grouping as above. A `select_data_columns` that was
only there to drop the moved tags can go.
- **Unchanged.** `by` must still name scalar tag columns. Members are still ordered by the
input's non-key tag values, then by record id, so every data list holds the same elements in
the same order; groups are emitted in key order. The Arrow types of the list columns, the
folded system tags and the data columns' list-valued source info are the same as before. The
all-null `_source_<tag>` columns of the moved tags are gone.
- **Grouping again.** A grouped stream can be grouped again on a coarser key; its data lists
nest (`list[list[T]]`). The second grouping orders its members by the first grouping's keys
that are not in the coarser key, then by record id; a tag the first grouping dropped no
longer takes part.
- **Hashes and stored results.** No pipeline hash, content hash or table path moves: the
`GroupBy` stream hashes its operator and its input, not its output columns. Against v0.2.0,
each grouped row's data hash changes where a tag was dropped, because the row's data no longer
includes it; a grouping on every tag is byte-identical. A function whose input was already
only the real data columns (`GroupBy` then `select_data_columns`) receives the same data and
stores the same columns as with v0.2.0, so it reuses its stored results without computing
again. A function whose signature has to change is a different function and computes once.
`node.as_source()` over a grouped node has a new `pipeline_hash`, because a source's pipeline
hash is its schema.
- **Stores written by a development build.** Between November 12 and this change, a
development build kept non-key tags as list-valued tags. A store it wrote that holds a
`GroupBy` over a non-key tag keeps that tag column in the operator cache and in the tables of
the function nodes after it, at the same paths. With this release, writing to such a table
fails with `StoreWriteError` (a non-null violation naming the dropped column), and reading it
(`CacheMode.REPLAY`, `get_all_records`) returns the old column. Delete those tables. No
released version wrote that shape.
32 changes: 0 additions & 32 deletions changelog.d/ITL-955.md

This file was deleted.

25 changes: 25 additions & 0 deletions design_issues/ITL-1070-GB2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
## `src/orcapod/core/operators/group_by.py` — tags entering the data group (ITL-955)

### GB2 — `GroupBy` kept non-key tags as list-valued tags
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1070

GB1 (ITL-955) stopped `GroupBy` from moving non-key tags into the data group by keeping them as
list-valued tags. A list of the members' values is not a key for the group, though, and a
grouped non-key tag that falls through a function pod is stored on every result row: in the
referential workshop pipeline it was the remaining cost (77 MB against 11 MB of store).

The rule (developer, November 13, 2026): `GroupBy` keeps only its `by` tags; a tag that is not
a key is dropped, as SQL drops a non-key, non-aggregated column after `GROUP BY`. Data columns
are unchanged. `Batch` keeps its list-valued tags: it has no key. An opt-in mode that keeps
non-key tags as lists is ITL-1071.

**Fix:** the output's tag columns are the `by` columns, scalar, in the input's tag order;
`unary_output_schema` predicts the same. The dropped tags still order the members, so every
data list, system-tag fold, row data hash and stream hash is unchanged; no golden value moves
(six `pipeline:group_by.row[...]` keys are renamed, because a row's key names its user tags).
Tables written by a development build between ITL-955 and this fix sit at the same paths with
the old tag column: a write to them fails and a read returns the old column; the changelog
tells users to delete them. Nothing checks a stored table's columns against the node's schema
on read; that gap is ITL-1072.
Loading
Loading