From ef24598ea4f0a12d73a016d9b18bb24596fb025c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 10:24:37 -0800 Subject: [PATCH 1/4] docs(plan): ITL-1070 plan and measurements (WIP) Co-Authored-By: Claude Opus 5.5 --- ...1-13-itl-1070-groupby-drops-non-by-tags.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md diff --git a/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md b/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md new file mode 100644 index 000000000..08b27e0d0 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md @@ -0,0 +1,76 @@ +# ITL-1070 — GroupBy drops the tags that are not in `by` + +Status: DRAFT, plan sent to coordinator for approval (hash-critical). No code before approval. + +Branch: `eywalker/itl-1070-groupby-drops-the-tags-that-are-not-in-by` (from origin/main 5c7a058f). +History: ITL-955 (#316) made non-key tags list-valued tags. Future opt-in mode: ITL-1071 (not ours). + +## Goal + +`GroupBy(by)` output tag columns are exactly `by`, in input tag order, scalar. Non-key tags are +dropped. Data columns, source info, system-tag folding, member order (still sorted by the +input's non-key tag values, record-id tiebreak) and group order are unchanged. + +## Code sites (src/orcapod/core/operators/group_by.py) + +- `unary_static_process`: keep sorting members on the input's non-key tags + record ids (rows + from `to_pylist` still have them); exclude the non-key tags from `member_columns`; build the + aggregated table from the input schema minus the dropped tags (`build_aggregated_table` + iterates the whole input schema, so they must be removed from it, else they come back as + null scalars); declare `tag_columns = [t for t in input tags if t in by]` (input order). +- `unary_output_schema`: drop non-key user tags; system tags keep the suffixed names. +- No `async_execute` of its own: the async path is the barrier path over `static_process`. +- System-tag naming/folding: untouched (fold of member values, `::{pipeline_hash}` suffix). +- `_null_last_key`: keep the recursion. It is no longer reached via re-grouping a GroupBy + output, but list-valued tags still reach GroupBy as non-key tags from a source with a + list-typed tag column or a Join with a `Batch` output; docstring updated. +- `identity_structure` / `to_config`: `("GroupBy", by)` only; no change. +- `streams/base.py` `group_by` docstring; class docstring. + +## Measurement + +Script: scratchpad `measure.py` (source -> GroupBy -> counting pod in a PipelineJob on Delta, +plus OperatorJobNode LOG cache, plus `as_source()`), run on main and on the change, and the +main store re-run with the change. + +## Tests (failing first) + +`tests/test_core/operators/test_group_by.py` (public `op(stream)`), async/sync equivalence, +chained aggregation (re-group coarser key), aggregation job (stored node), golden GroupBy entries. + +## Docs + +docs/concepts/operators.md, CLAUDE.md + .zed/rules (boundary paragraph, rule 3, GroupBy bullets), +changelog.d/ITL-1070.md, design_issues/ITL-1070-GB2.md. + +## Measurement results (main vs prototype, 2026-11-13) + +Prototype diff saved at scratchpad `proto.diff` (10 lines in group_by.py). + +- Stream with a non-key tag: content hash, pipeline hash, system-tag column names and values, + every row's data hash, data lists and their order, `_source_*` lists: IDENTICAL. Only the + tag keys/schema lose `probe`. GroupBy op identity `('GroupBy', by)` / config unchanged. + Downstream pod node: pipeline hash, content hash, table path unchanged; pdb table loses + `probe` column; result (rdb) table unchanged. odb (LOG) path unchanged; table loses `probe`. + `as_source()` DerivedSource: source_id and content hash unchanged; pipeline hash MOVES + (root = schema). +- Stream with no non-key tag: everything byte-identical, including stored tables. +- Golden: no value moves. Six `pipeline:group_by.row[...]` keys are renamed (row label is the + user-tag dict; `subject` list leaves it); values equal. +- Cross-version (store written on main, then the change): paths are the SAME, so old tables ARE + found: pod: no recompute (rdb reused), but pdb write fails (Delta "Non-nullable column + violation for probe") -> StoreWriteError; odb LOG write fails the same way; odb REPLAY serves + rows that still carry `probe` while keys() lacks it; counter get_all_records returns `probe`. + Only stores written by v0.3-dev after #316 (Nov 12) with a GroupBy over a non-key tag. + v0.2-shape stores (GroupBy -> SelectDataColumns(path) -> pod) are served exactly + (TestStoreWrittenBeforeTagsStayedTags second test passes except its keys assertion). +- Tests relying on list tags: 31 failures in targeted run (test_group_by TagsStayTags + + Ordering, async_sync_equivalence, chained_aggregation x3, empty_inputs[GroupBy], + aggregation_job legacy x2, spiraldb tags_as_data workshop case, tensor/array test_group_by, + golden x9). docs/concepts/sources.md sample uses `.drop_tag_columns("timestamp")` after + group_by (now gone). PR #346: `group_by_g` layout `all_but_g` -> `data_only`, checker + comment on list-valued tags, job `grouped` node loses `n`. + +## Notes for successor + +Plan sent to coordinator via ask (2026-11-13); waiting on decision about stale v0.3-dev tables. From 3babd2b877211aecde26698dd941ec6f8648139d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 10:29:41 -0800 Subject: [PATCH 2/4] fix(group_by): keep only the by tags, drop the non-key tags (ITL-1070) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/operators/group_by.py | 71 ++++--- tests/test_arrays/test_array_operators.py | 7 +- .../operators/test_async_sync_equivalence.py | 22 +- .../operators/test_chained_aggregation.py | 16 +- .../test_core/operators/test_empty_inputs.py | 3 +- tests/test_core/operators/test_group_by.py | 199 +++++++++--------- ...test_spiraldb_table_source_tags_as_data.py | 10 +- tests/test_hashing/golden/data/pipeline.json | 12 +- tests/test_pipeline/test_aggregation_job.py | 102 ++++++--- tests/test_torch/test_tensor_operators.py | 7 +- 10 files changed, 248 insertions(+), 201 deletions(-) diff --git a/src/orcapod/core/operators/group_by.py b/src/orcapod/core/operators/group_by.py index cbe21cbc7..6ba8e7e32 100644 --- a/src/orcapod/core/operators/group_by.py +++ b/src/orcapod/core/operators/group_by.py @@ -24,8 +24,9 @@ def _null_last_key(value: Any) -> tuple[Any, ...]: """Build an order key that sorts nulls last, at any list depth. The leading integer keeps a null comparable against a real value of any - type. A list is keyed element by element, so a list-valued tag that holds - a null (the non-key tag of an already grouped stream) still compares. + type. A list is keyed element by element, so a list-valued non-key tag + that holds a null (a list-typed tag column of a source, or the tags of a + ``Batch`` output joined in) still compares. Args: value: A tag or system-tag value: a scalar, ``None``, or a list of @@ -42,7 +43,7 @@ def _null_last_key(value: Any) -> tuple[Any, ...]: class GroupBy(UnaryOperator): - """Reduce rows sharing a tag tuple into one packet with list-valued members. + """Reduce rows sharing a key into one row with list-valued data members. This is the only many->one operator. Every other operator preserves one row per tag; ``GroupBy`` collapses N rows into one, which is what lets a @@ -52,10 +53,11 @@ class GroupBy(UnaryOperator): Given tags ``(subject, date, probe)`` and data ``(path)``, grouping by ``["subject", "date"]`` emits one row per distinct ``(subject, date)``: - * ``subject`` and ``date`` stay scalar tag columns - * ``probe`` stays a **tag** column and becomes list-valued, one element - per member, exactly as ``Batch`` keeps its tags. The tag columns of the - output are the tag columns of the input, in the same order. + * ``subject`` and ``date`` stay scalar tag columns, in the input's tag + order; they are the only tag columns of the output + * ``probe`` is **dropped** (ITL-1070): a list of the members' values is not + a key for the group, as SQL drops a non-key, non-aggregated column after + ``GROUP BY`` * ``path`` becomes list-valued * the ``_source_*`` column of each data column becomes list-valued, one element per member @@ -64,11 +66,12 @@ class GroupBy(UnaryOperator): No data element ever comes from a tag value: ``GroupBy`` uses tags to form groups, but the data columns of the output are exactly the data columns of - the input. A function pod downstream therefore receives only ``path``; if - it needs to know which member each element came from, that value has to - be a data column of the input. + the input. The members' values of a dropped tag are not carried anywhere. + If a downstream step needs to know which member each element came from, + make that value a data column before grouping: declare it as data at the + source, or use ``DerivedSource(origin, tags_as_data={...})``. - Members are sorted by their non-group-key tag values, with ``record_id`` + Members are sorted by the input's non-group-key tag values, with ``record_id`` appended as a final tiebreaker, so the emitted lists are stable across runs. This matters because orcapod hashes those lists to build the cache key -- an unsorted list would make an identical member set hash differently @@ -82,12 +85,15 @@ class GroupBy(UnaryOperator): is materialized -- so their order is stable for a given source table, but undefined if the source table itself is permuted. - A grouped stream can be grouped again on a coarser key: its list-valued - tags nest one level deeper (``list[list[T]]``), as source info does. A - list-valued tag can order members but cannot be a grouping key. + A grouped stream can be grouped again on a coarser key: its data lists nest + one level deeper (``list[list[T]]``), as source info does. The finer key + is gone by then, so the members of the second grouping are ordered by + ``record_id`` alone. A list-valued tag of the input can order members but + cannot be a grouping key. - Contrast with ``Batch``, which partitions by row count for throughput and - makes every tag column list-valued, having no scalar keys. + Contrast with ``Batch``, which partitions by row count for throughput. It + has no key, so it keeps every tag column, list-valued: those lists are + what tells a batch's members apart. Args: by: Tag column names to group on. Must be non-empty, free of @@ -175,8 +181,7 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: Returns: A stream with one row per distinct group-key tuple, whose tag - columns are the input's tag columns: group keys scalar, every - other tag list-valued. + columns are the group keys, scalar, in the input's tag order. """ table = stream.as_table(columns={"source": True, "system_tags": True}) tag_columns, _ = stream.keys() @@ -184,10 +189,15 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: system_tag_columns = tuple( c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ) + # A tag that is not a key is dropped from the output (ITL-1070), but it + # still orders the members below, so the data lists keep their order. + dropped_tags = tuple(c for c in tag_columns if c not in self.by) member_columns = tuple( c for c in table.column_names - if c not in self.by and c not in system_tag_columns + if c not in self.by + and c not in system_tag_columns + and c not in dropped_tags ) # Non-key user tags order the members of a group. The `record_id` # columns are appended as a final tiebreaker: tag tuples are supposed @@ -203,7 +213,7 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: # consulting only the first would tie on exactly the fan-out rows the # tiebreaker exists for. They are sorted by name so the key order is # itself independent of column order in the input table. - sort_columns = tuple(c for c in tag_columns if c not in self.by) + sort_columns = dropped_tags sort_columns += tuple( sorted( c @@ -240,9 +250,11 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: } ) + # The schema handed over must not name the dropped tags: the builder + # emits every column of it, and would bring them back as nulls. grouped_table = arrow_utils.build_aggregated_table( grouped_rows, - table.schema, + table.drop_columns(list(dropped_tags)).schema, member_columns, stream.data_context.type_converter, ) @@ -252,12 +264,10 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol: grouped_table, stream.pipeline_hash().to_hex(n_char) ) - # Every input tag stays a tag (ITL-955): the non-key ones are the - # list-valued member columns built above. Declaring only `self.by` - # here would let them fall into the data group. + # The keys, in the input's tag order rather than the order of `by`. return ArrowTableStream( grouped_table, - tag_columns=tag_columns, + tag_columns=[c for c in tag_columns if c in self.by], data_context=stream.data_context, ) @@ -281,9 +291,10 @@ def unary_output_schema( Returns: A ``(tag_schema, data_schema)`` tuple. The tag schema holds the - input's tags in input order: group keys scalar, every other user - tag ``list[T]``, system tags scalar under their suffixed names. - The data schema holds the input's data columns as ``list[T]``. + group keys, scalar, in the input's tag order, and the system tags, + scalar under their suffixed names; every other user tag is + dropped. The data schema holds the input's data columns as + ``list[T]``. """ tag_types, data_types = stream.output_schema(columns=columns, all_info=all_info) n_char = self.orcapod_config.hashing.system_tag_n_char @@ -295,9 +306,7 @@ def unary_output_schema( out_tag_types[f"{name}{constants.BLOCK_SEPARATOR}{suffix}"] = col_type elif name in self.by: out_tag_types[name] = col_type - else: - # A non-key tag stays a tag, list-valued (ITL-955). - out_tag_types[name] = list[col_type] + # Any other user tag is dropped (ITL-1070). out_data_types = {name: list[t] for name, t in data_types.items()} diff --git a/tests/test_arrays/test_array_operators.py b/tests/test_arrays/test_array_operators.py index f53962ddc..83a3b7966 100644 --- a/tests/test_arrays/test_array_operators.py +++ b/tests/test_arrays/test_array_operators.py @@ -145,9 +145,10 @@ def test_group_by() -> None: out = list(GroupBy(by=["g"])(arrays_source()).iter_data()) assert sorted(fields(t)["g"] for t, _ in out) == ["g0", "g1", "g2"] for tag, data in out: - # GroupBy keeps the non-key tag ``k`` as a list-valued tag. - names = list(fields(tag)["k"]) - assert sorted(names) == sorted(n for n in NAMES if GROUP[n] == fields(tag)["g"]) + # GroupBy drops the non-key tag ``k`` (ITL-1070); it still orders the + # members, so the group's names come in ascending order. + assert set(fields(tag)) == {"g"} + names = sorted(n for n in NAMES if GROUP[n] == fields(tag)["g"]) for name, x in zip(names, fields(data)["x"], strict=True): assert_arrays_identical(x, expected_x(name)) diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 420a5b72f..da3790e47 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -648,24 +648,28 @@ def test_drop_nulls_without_tag_columns() -> None: PROVENANCE_CASES = [pytest.param(c, id=c.label) for c in CASES] -def test_group_by_keeps_non_key_tags_as_tags_on_both_paths() -> None: - """Neither path moves a non-key tag into the data (ITL-955). +def test_group_by_keeps_only_key_tags_on_both_paths() -> None: + """Both paths drop the non-key tags and keep the data lists (ITL-1070). - ``source_a`` has tags ``k`` and ``ta``; grouping by ``ta`` leaves ``k`` a - list-valued tag, and the data of every row holds only the input's data - columns. + ``source_a`` has tags ``k`` and ``ta``; grouping by ``ta`` drops ``k``, and + the data of every row holds only the input's data columns, ordered by ``k``. """ _, data_keys = source_a().keys() + results = [] for rows in ( sync_rows(GroupBy(["ta"]), source_a()), async_rows(GroupBy(["ta"]), source_a()), ): - members = sorted(tuple(cast(list[int], tag.as_dict()["k"])) for tag, _ in rows) - assert members == [(1, 2), (3,)] for tag, data in rows: - assert set(tag.keys()) == {"k", "ta"} + assert set(tag.keys()) == {"ta"} assert set(data.keys()) == set(data_keys) - assert tag.as_table().schema.field("k").type == pa.large_list(pa.int16()) + results.append( + sorted( + (tag.as_dict()["ta"], data.as_table().to_pylist()[0]) + for tag, data in rows + ) + ) + assert results[0] == results[1] @pytest.mark.parametrize("case", PROVENANCE_CASES) diff --git a/tests/test_core/operators/test_chained_aggregation.py b/tests/test_core/operators/test_chained_aggregation.py index 544c511e4..006d4f713 100644 --- a/tests/test_core/operators/test_chained_aggregation.py +++ b/tests/test_core/operators/test_chained_aggregation.py @@ -131,6 +131,11 @@ def merge_join_zero() -> StreamProtocol: "lst": LL(LL(LL(pa.int8()))), "p": LIST_LIST_PATH, } +# ``GroupBy(["g"])`` drops the non-key tag ``id`` (ITL-1070), so a chain whose +# upstream is a ``GroupBy`` has no ``id`` column to aggregate again. +GROUP_BY_THEN_AGGREGATE = { + k: v for k, v in AGGREGATE_THEN_AGGREGATE.items() if k != "id" +} def _batch_members(upstream: Rows, row: int, size: int = 2) -> list[int]: @@ -150,7 +155,7 @@ def _group_members(upstream: Rows, key: Any) -> list[int]: "mergejoin_group_by": { "upstream": merge_join_ac, "op": lambda: GroupBy(["g"]), - "types": {"id": LL(pa.int64()), "g": pa.int64(), **MERGE_JOIN_THEN_AGGREGATE}, + "types": {"g": pa.int64(), **MERGE_JOIN_THEN_AGGREGATE}, "members": lambda up, r, tag: _group_members(up, tag["g"]), }, "mergejoin_batch": { @@ -172,13 +177,13 @@ def _group_members(upstream: Rows, key: Any) -> list[int]: "group_by_batch": { "upstream": lambda: GroupBy(["g"])(source_a()), "op": lambda: Batch(batch_size=2), - "types": {"g": LL(pa.int64()), **AGGREGATE_THEN_AGGREGATE}, + "types": {"g": LL(pa.int64()), **GROUP_BY_THEN_AGGREGATE}, "members": lambda up, r, tag: _batch_members(up, r), }, "group_by_group_by": { "upstream": lambda: GroupBy(["g"])(source_a()), "op": lambda: GroupBy(["g"]), - "types": {"g": pa.int64(), **AGGREGATE_THEN_AGGREGATE}, + "types": {"g": pa.int64(), **GROUP_BY_THEN_AGGREGATE}, "members": lambda up, r, tag: _group_members(up, tag["g"]), }, } @@ -293,9 +298,8 @@ def test_source_info_is_a_list_per_row_of_its_members_tokens( ) -> None: """Every row's provenance lists its own members' upstream tokens. - Only data columns carry provenance: a non-key tag that an upstream - ``GroupBy`` kept as a list-valued tag (``id``) has no source-info - column at all (ITL-955). + Only data columns carry provenance: a tag has no source-info column + (ITL-955). """ upstream = list(chain.upstream.iter_data()) rows = list(chain.result.iter_data()) diff --git a/tests/test_core/operators/test_empty_inputs.py b/tests/test_core/operators/test_empty_inputs.py index 0f69fd71e..99ba961b8 100644 --- a/tests/test_core/operators/test_empty_inputs.py +++ b/tests/test_core/operators/test_empty_inputs.py @@ -477,7 +477,8 @@ def _empty_list_stream() -> ArrowTableStream: "GroupBy", lambda: GroupBy(by=["probe"]), lambda: [_int_stream(["probe", "session"], ["n_units"], 0, nullable=False)], - {"probe", "session", "n_units"}, + # The non-key tag ``session`` is dropped (ITL-1070). + {"probe", "n_units"}, ), ( "SelectTagColumns", diff --git a/tests/test_core/operators/test_group_by.py b/tests/test_core/operators/test_group_by.py index 8f2151957..b0ce47a28 100644 --- a/tests/test_core/operators/test_group_by.py +++ b/tests/test_core/operators/test_group_by.py @@ -79,68 +79,63 @@ def test_system_tags_are_scalar_and_renamed(self, session_source): assert col.count(constants.BLOCK_SEPARATOR) >= 2 -class TestGroupByTagsStayTags: - """No data element may come from a tag value (ITL-955). +class TestGroupByKeepsOnlyKeyTags: + """``GroupBy`` keeps only its ``by`` tags (ITL-1070). - ``GroupBy`` uses tags to form groups, but tag content never becomes data: - a non-key tag stays in the tag group as a list-valued tag, as ``Batch`` - keeps it. + A tag that is not a key is dropped: a list of the members' values is not a + key for the group, as SQL drops a non-key, non-aggregated column after + ``GROUP BY``. Tag content never becomes data either (ITL-955). The dropped + tags still order the members inside each data list. """ - def test_non_key_tags_stay_list_valued_tags(self, session_source): - """A tag not named in ``by`` stays a tag and holds one value per member.""" + def test_output_tags_are_exactly_the_keys(self, session_source): + """A tag not named in ``by`` is not in the output, as tag or as data.""" out = GroupBy(by=["subject", "date"])(session_source) - tag_cols, data_cols = out.keys() - assert tag_cols == ("subject", "date", "probe") - assert data_cols == ("path",) - assert out.as_table().column("probe").to_pylist() == [[0, 1], [0, 1]] + assert out.keys() == (("subject", "date"), ("path",)) + assert "probe" not in out.as_table(all_info=True).column_names def test_tag_order_is_the_input_tag_order(self, session_source): - """A key named last in ``by`` does not reorder the tag columns.""" - out = GroupBy(by=["date"])(session_source) - assert out.keys()[0] == ("subject", "date", "probe") + """Keys named in another order keep the input's tag order.""" + out = GroupBy(by=["date", "subject"])(session_source) + assert out.keys()[0] == ("subject", "date") - def test_no_data_column_is_an_input_tag(self, session_source): - """The tag and data column names are the input's: no tag moves to data.""" - in_tags, in_data = session_source.keys() - out = GroupBy(by=["subject"])(session_source) - assert out.keys() == (in_tags, in_data) - - def test_non_key_tag_schema_is_a_list_tag(self, session_source): - """``output_schema()`` reports a non-key tag as a list in the tag schema.""" + def test_key_tags_stay_scalar(self, session_source): + """Every output tag keeps the input tag's type: no list-valued tag.""" out = GroupBy(by=["subject", "date"])(session_source) + in_tag_schema, _ = session_source.output_schema() tag_schema, data_schema = out.output_schema() + assert dict(tag_schema) == { + "subject": in_tag_schema["subject"], + "date": in_tag_schema["date"], + } + assert all(get_origin(t) is not list for t in tag_schema.values()) assert "probe" not in data_schema - assert get_origin(tag_schema["probe"]) is list - def test_non_key_tag_keeps_its_arrow_type(self): - """The list element is the input tag's type: ``int8`` is not widened.""" - table = pa.table( - { - "subject": pa.array(["G", "G"], pa.large_string()), - "probe": pa.array([1, 0], pa.int8()), - "path": pa.array(["b", "a"], pa.large_string()), - } - ) - source = ArrowTableSource(table, tag_columns=["subject", "probe"]) - out = GroupBy(by=["subject"])(source) - assert out.as_table().schema.field("probe").type == pa.large_list(pa.int8()) + def test_data_columns_are_the_input_data_columns(self, session_source): + """No input tag appears among the data columns.""" + _, in_data = session_source.keys() + out = GroupBy(by=["subject"])(session_source) + assert out.keys() == (("subject",), in_data) - def test_non_key_tags_get_no_source_info_column(self, session_source): - """Source info belongs to data columns; a tag has none.""" + def test_no_source_info_column_for_a_dropped_tag(self, session_source): + """Source info belongs to data columns; a dropped tag leaves none.""" out = GroupBy(by=["subject", "date"])(session_source) names = out.as_table(columns={"source": True}).column_names assert f"{constants.SOURCE_PREFIX}probe" not in names assert f"{constants.SOURCE_PREFIX}path" in names - def test_datagrams_carry_non_key_tags_on_the_tag(self, session_source): - """Each row's ``Tag`` holds the non-key tags; its ``Data`` holds only data.""" + def test_datagrams_hold_only_keys_and_data(self, session_source): + """Each row's ``Tag`` holds the keys; its ``Data`` holds only data.""" out = GroupBy(by=["subject", "date"])(session_source) rows = list(out.iter_data()) - assert len(rows) == 2 - for tag, data in rows: - assert tag.as_dict()["probe"] == [0, 1] - assert set(data.as_dict()) == {"path"} + assert [tag.as_dict() for tag, _ in rows] == [ + {"subject": "G", "date": "d1"}, + {"subject": "G", "date": "d2"}, + ] + assert [data.as_dict() for _, data in rows] == [ + {"path": ["a", "b"]}, + {"path": ["c", "d"]}, + ] def test_a_pod_receives_only_data_columns(self, session_source): """A function pod never sees tag content, so its signature is the data.""" @@ -148,65 +143,51 @@ def test_a_pod_receives_only_data_columns(self, session_source): PythonDataFunction(_count_paths, output_keys="n") # pyright: ignore[reportArgumentType] # protocol mismatch baselined elsewhere (ITL-844) ) out = pod(GroupBy(by=["subject", "date"])(session_source)) + assert out.keys() == (("subject", "date"), ("n",)) assert [d.as_dict()["n"] for _, d in out.iter_data()] == [2, 2] - def test_list_valued_non_key_tag_still_cannot_be_a_key(self, session_source): - """A list-valued tag of a grouped stream is refused as a grouping key.""" - grouped = GroupBy(by=["subject", "date"])(session_source) - with pytest.raises(InputValidationError, match="list-valued"): - GroupBy(by=["probe"])(grouped) + def test_grouping_on_every_tag_keeps_every_tag(self, session_source): + """With no non-key tag nothing is dropped: tags as given, data as lists.""" + out = GroupBy(by=["subject", "date", "probe"])(session_source) + assert out.keys() == session_source.keys() + assert out.as_table().to_pylist() == [ + {"subject": "G", "date": "d1", "probe": 0, "path": ["a"]}, + {"subject": "G", "date": "d1", "probe": 1, "path": ["b"]}, + {"subject": "G", "date": "d2", "probe": 0, "path": ["c"]}, + {"subject": "G", "date": "d2", "probe": 1, "path": ["d"]}, + ] - def test_regroup_nests_list_tags(self, session_source): - """Grouping a grouped stream on a coarser key nests the list tags.""" + def test_regroup_on_a_coarser_key(self, session_source): + """A grouped stream grouped again keeps the coarser key; data nests. + + The first grouping dropped ``date`` and ``probe``, so no tag is left to + order the members of the second grouping: they are ordered by their + folded ``record_id``, which is stable across runs. + """ out = GroupBy(by=["subject"])(GroupBy(by=["subject", "date"])(session_source)) - assert out.keys() == (("subject", "date", "probe"), ("path",)) - row = out.as_table().to_pylist()[0] - assert row["date"] == ["d1", "d2"] - assert row["probe"] == [[0, 1], [0, 1]] - assert row["path"] == [["a", "b"], ["c", "d"]] - - def test_regroup_orders_list_tags_containing_nulls(self): - """A list-valued non-key tag holding a null still orders the members.""" - table = pa.table( - { - "a": ["x", "x", "x", "x"], - "b": [1, 1, 2, 2], - "c": pa.array([None, 1, 2, 1], pa.int64()), - "v": [1, 2, 3, 4], - } - ) - source = ArrowTableSource( - table, tag_columns=["a", "b", "c"], infer_nullable=True - ) - first = GroupBy(by=["a", "b"])(source) - # Regrouping on ``a`` alone leaves ``b`` scalar and ``c`` list-valued: - # the two members tie on nothing, and are ordered by ``b`` first. - out = GroupBy(by=["a"])(first) - row = out.as_table().to_pylist()[0] - assert row["b"] == [1, 2] - assert row["c"] == [[1, None], [1, 2]] - assert row["v"] == [[2, 1], [4, 3]] - - def test_members_tied_on_scalar_tags_order_by_list_tag_with_null(self): - """``[1, None]`` and ``[1, 2]`` compare without a ``TypeError``.""" + assert out.keys() == (("subject",), ("path",)) + rows = out.as_table().to_pylist() + assert len(rows) == 1 + assert rows[0]["subject"] == "G" + assert sorted(rows[0]["path"]) == [["a", "b"], ["c", "d"]] + + def test_list_valued_non_key_tag_orders_members(self): + """A list-typed tag of the input still orders members, nulls last. + + ``[1, None]`` and ``[1, 2]`` compare without a ``TypeError``: the member + sort key is recursive. The tag itself is dropped. + """ table = pa.table( { - "a": ["x", "x", "x", "x"], - "b": ["p", "p", "q", "q"], - "c": pa.array([1, None, 1, 2], pa.int64()), - "v": [1, 2, 3, 4], + "a": pa.array(["x", "x", "x"], pa.large_string()), + "c": pa.array([[1, None], [1, 2], [0]], pa.large_list(pa.int64())), + "v": pa.array([1, 2, 3], pa.int64()), } ) - source = ArrowTableSource( - table, tag_columns=["a", "b", "c"], infer_nullable=True - ) - # Drop ``b`` after the first grouping so the two members differ only in - # the list-valued tag ``c``: [1, None] against [1, 2]. - first = GroupBy(by=["a", "b"])(source).drop_tag_columns(["b"]) - out = GroupBy(by=["a"])(first) - row = out.as_table().to_pylist()[0] - assert row["c"] == [[1, 2], [1, None]] - assert row["v"] == [[3, 4], [1, 2]] + source = ArrowTableSource(table, tag_columns=["a", "c"], infer_nullable=True) + out = GroupBy(by=["a"])(source) + assert out.keys() == (("a",), ("v",)) + assert out.as_table().to_pylist() == [{"a": "x", "v": [3, 2, 1]}] class TestGroupByOrdering: @@ -228,15 +209,19 @@ class TestGroupByOrdering: # determinism, and which sort key. Deferred per review on PR #250. def test_members_sorted_by_non_key_tags(self, session_source): - """probe=[1,0] on input must emit as [0,1].""" - out = GroupBy(by=["subject", "date"]).process(session_source) - assert out.as_table().column("probe").to_pylist()[0] == [0, 1] + """probe=[1,0] on input orders the members as probe 0, then 1. + + ``probe`` is dropped from the output (ITL-1070), but it still orders the + members: ``path`` b (probe 1) and a (probe 0) emit as ``["a", "b"]``. + """ + out = GroupBy(by=["subject", "date"])(session_source) + assert out.as_table().column("path").to_pylist() == [["a", "b"], ["c", "d"]] def test_scalar_non_key_tags_sort_ascending_with_nulls_last(self): """Pins the member order for scalar tags: ascending, nulls last. - The sort key is recursive so that list-valued tags compare (ITL-955); - for scalar tags it must give exactly the order it always gave. + The non-key tags ``n`` and ``s`` are dropped (ITL-1070); the order they + impose is read from the data column ``v``. """ table = pa.table( { @@ -249,10 +234,10 @@ def test_scalar_non_key_tags_sort_ascending_with_nulls_last(self): source = ArrowTableSource( table, tag_columns=["g", "n", "s"], infer_nullable=True ) - row = GroupBy(by=["g"])(source).as_table().to_pylist()[0] - assert row["n"] == [1, 1, 2, 3, None] - assert row["s"] == ["a", None, "c", "b", "a"] - assert row["v"] == [4, 2, 3, 0, 1] + # Members by (n, s): (1, "a"), (1, None), (2, "c"), (3, "b"), (None, "a"). + assert GroupBy(by=["g"])(source).as_table().to_pylist() == [ + {"g": "x", "v": [4, 2, 3, 0, 1]} + ] def test_row_order_does_not_affect_output(self, session_table): """Same rows, shuffled, must produce a byte-identical table.""" @@ -422,6 +407,14 @@ def diff(label, predicted, actual_schema): diff("tag", pred_tag, act_tag) diff("data", pred_data, act_data) + def test_predicted_user_tags_are_the_keys(self, session_source): + """The prediction names only the keys among the user tags (ITL-1070).""" + op = GroupBy(by=["subject", "date"]) + pred_tag, _ = op.unary_output_schema(session_source) + # Verifying a schema prediction: compared against the materialized stream. + act_tag, _ = op.unary_static_process(session_source).output_schema() + assert list(pred_tag) == list(act_tag) == ["subject", "date"] + class TestGroupByIdentity: def test_identity_structure_includes_by(self): diff --git a/tests/test_core/sources/test_spiraldb_table_source_tags_as_data.py b/tests/test_core/sources/test_spiraldb_table_source_tags_as_data.py index a137c3dbf..bad094b3c 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_tags_as_data.py +++ b/tests/test_core/sources/test_spiraldb_table_source_tags_as_data.py @@ -360,12 +360,14 @@ class TestTheWorkshopCase: """Grouping on the unit key gives each unit's spike times in time order.""" def test_group_by_the_unit_key_orders_spike_times_by_time(self, project): - """The only tag left to order members is the time, so the lists are in time order.""" + """The only tag left to order members is the time, so the lists are in time order. + + ``group_by`` drops the non-key tag ``timestamp`` (ITL-1070); its copy + ``spike_time_us`` is data and is kept. + """ spikes = _source(tags_as_data=_COPY) - per_unit = spikes.group_by(["subject_id", "unit_id"]).drop_tag_columns( - "timestamp" - ) + per_unit = spikes.group_by(["subject_id", "unit_id"]) assert per_unit.keys() == ( ("subject_id", "unit_id"), diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index 7ee2faca9..6e3ccda85 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -222,12 +222,12 @@ ], "group_by.content_hash": "semantic_v1:124f0ae9102b952b711bb8f17fa75fcf75667cefa9e4e047739325bdd726d552", "group_by.pipeline_hash": "semantic_v1:8bdfc3479f44ae3d482f31417cee7672b0b3098e2ee2f3719e74c17f12e1e029", - "group_by.row[{\"cohort\":\"a\",\"subject\":[\"s1\",\"s2\"]}]._tag_record_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "7d212ee5c358519c9e4729f4a0898aad", - "group_by.row[{\"cohort\":\"a\",\"subject\":[\"s1\",\"s2\"]}]._tag_source_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "11316416683632ce9125c3a3ab58c085c03b46bc8b8b14767b70ad33d90d621f", - "group_by.row[{\"cohort\":\"a\",\"subject\":[\"s1\",\"s2\"]}].data_hash": "arrow_v1:000002b50de37e741a154b01d9b0492bc3f3b6db1044edf8abee33e6e895037fdef81e", - "group_by.row[{\"cohort\":\"b\",\"subject\":[\"s3\"]}]._tag_record_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "b7f2f8f7412a54fdb03cd96b6b0b1643", - "group_by.row[{\"cohort\":\"b\",\"subject\":[\"s3\"]}]._tag_source_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "0e3b86a9fbf8566e5a1f4e53b501f566da56327a6d0ecf2e78f2288f76167dd7", - "group_by.row[{\"cohort\":\"b\",\"subject\":[\"s3\"]}].data_hash": "arrow_v1:0000026bc48a0e74f57c3f5968efabefb44d0a2364abcba5d4b2237bb1a27ffe993d6c", + "group_by.row[{\"cohort\":\"a\"}]._tag_record_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "7d212ee5c358519c9e4729f4a0898aad", + "group_by.row[{\"cohort\":\"a\"}]._tag_source_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "11316416683632ce9125c3a3ab58c085c03b46bc8b8b14767b70ad33d90d621f", + "group_by.row[{\"cohort\":\"a\"}].data_hash": "arrow_v1:000002b50de37e741a154b01d9b0492bc3f3b6db1044edf8abee33e6e895037fdef81e", + "group_by.row[{\"cohort\":\"b\"}]._tag_record_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "b7f2f8f7412a54fdb03cd96b6b0b1643", + "group_by.row[{\"cohort\":\"b\"}]._tag_source_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c": "0e3b86a9fbf8566e5a1f4e53b501f566da56327a6d0ecf2e78f2288f76167dd7", + "group_by.row[{\"cohort\":\"b\"}].data_hash": "arrow_v1:0000026bc48a0e74f57c3f5968efabefb44d0a2364abcba5d4b2237bb1a27ffe993d6c", "group_by.system_tag_columns": [ "_tag_source_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c", "_tag_record_id::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c::7d650c4855ec007531f6a110836de7a0ff2ac68b4bfff32e58b2c41d02f44f7c" diff --git a/tests/test_pipeline/test_aggregation_job.py b/tests/test_pipeline/test_aggregation_job.py index a802ed501..5764a25e2 100644 --- a/tests/test_pipeline/test_aggregation_job.py +++ b/tests/test_pipeline/test_aggregation_job.py @@ -42,8 +42,8 @@ def count_paths(path: list[str]) -> int: """Record the members' `path` values and return how many there are. - `Batch` and `GroupBy` both keep `probe` as a list-valued *tag*, so `path` - is the only data column reaching the pod. + `Batch` keeps `probe` as a list-valued *tag* and `GroupBy` drops it, so + `path` is the only data column reaching the pod. """ _CALLS["path"].append(list(path)) return len(path) @@ -106,6 +106,21 @@ def test_group_by_completes(self, store, session_source_factory): assert len(_CALLS["path"]) == 2 assert sorted(_CALLS["path"]) == [["a", "b"], ["c", "d"]] + def test_stored_records_hold_only_the_key_tags(self, store, session_source_factory): + """The pod after ``GroupBy`` stores the keys, not the dropped tag (ITL-1070).""" + job = _run( + store, session_source_factory(), GroupBy(by=["subject", "date"]), "gb" + ) + counter = job.nodes["counter"] + assert counter.keys() == (("subject", "date"), ("n",)) + records = counter.get_all_records() + assert records is not None + assert "probe" not in records.column_names + assert sorted(records.select(["date", "n"]).to_pylist(), key=str) == [ + {"date": "d1", "n": 2}, + {"date": "d2", "n": 2}, + ] + def test_batch_completes(self, store, session_source_factory): """The provenance fix is independent of grouping.""" _run( @@ -179,11 +194,11 @@ def test_changed_member_invalidates_only_its_group( class _LegacyGroupBy(GroupBy): - """``GroupBy`` with the output shape it had before ITL-955. + """``GroupBy`` with the output shape of v0.2.0, before ITL-955 and ITL-1070. - Non-key tags are declared as data columns, which is what stores written by - an earlier version hold. It keeps the identity of ``GroupBy`` so that it - occupies the same place in the pipeline. + Non-key tags are list-valued data columns, which is what stores written by + v0.2.0 hold. It keeps the identity of ``GroupBy`` so that it occupies the + same place in the pipeline. """ def identity_structure(self): @@ -191,10 +206,12 @@ def identity_structure(self): return ("GroupBy", self.by) def unary_static_process(self, stream): - """Group as ``GroupBy`` does, then declare only the ``by`` columns as tags. + """Group as ``GroupBy`` does, then add the non-key tags as data columns. - The non-key tags become data columns, the shape ``GroupBy`` produced before - ITL-955. + ``GroupBy`` drops the non-key tags (ITL-1070); v0.2.0 kept each as a + list-valued data column, in member order. The members of a group are + ordered by their non-key tag values, so each such list is its group's + values sorted (the fixture's tags hold no nulls or ties). Args: stream: The stream to group. @@ -203,14 +220,34 @@ def unary_static_process(self, stream): The grouped stream in the legacy shape. """ grouped = super().unary_static_process(stream) + table = grouped.as_table(columns={"source": True, "system_tags": True}) + tag_columns, _ = stream.keys() + non_key = [c for c in tag_columns if c not in self.by] + input_table = stream.as_table() + members: dict[tuple, list[dict]] = {} + for row in input_table.to_pylist(): + members.setdefault(tuple(row[c] for c in self.by), []).append(row) + keys = [ + tuple(r[c] for c in self.by) + for r in table.select(list(self.by)).to_pylist() + ] + # v0.2.0 placed the moved tags right after the keys, before the data. + for position, column in enumerate(non_key, start=len(self.by)): + values = [sorted(m[column] for m in members[key]) for key in keys] + list_type = pa.large_list(input_table.schema.field(column).type) + table = table.add_column( + position, + pa.field(column, list_type, nullable=False), + pa.array(values, type=list_type), + ) return ArrowTableStream( - grouped.as_table(columns={"source": True, "system_tags": True}), + table, tag_columns=self.by, data_context=grouped.data_context, # pyright: ignore[reportAttributeAccessIssue] # not on StreamProtocol, as in GroupBy itself (ITL-844) ) def unary_output_schema(self, stream, *, columns=None, all_info=False): - """Predict the legacy schema: non-key tags are moved to the data schema. + """Predict the legacy schema: non-key tags are list-valued data columns. Args: stream: The stream to group. @@ -223,22 +260,19 @@ def unary_output_schema(self, stream, *, columns=None, all_info=False): tag_schema, data_schema = super().unary_output_schema( stream, columns=columns, all_info=all_info ) - tags = { - k: v - for k, v in tag_schema.items() - if k in self.by or k.startswith(constants.SYSTEM_TAG_PREFIX) - } - moved = {k: v for k, v in tag_schema.items() if k not in tags} - return Schema(tags), Schema({**moved, **data_schema}) - - -class TestStoreWrittenBeforeTagsStayedTags: - """Results stored before ITL-955 are still found after it. - - Before the fix a pod could only follow ``GroupBy`` by selecting the real - data columns (or by taking the moved tags as parameters). The fix changes - neither a pipeline hash nor the pod's input data, so a store written by the - old shape serves the new one without recomputing. + in_tags, _ = stream.output_schema() + moved = {k: list[v] for k, v in in_tags.items() if k not in self.by} + return tag_schema, Schema({**moved, **data_schema}) + + +class TestStoreWrittenByV020: + """Results stored by v0.2.0 are still found (ITL-955, ITL-1070). + + In v0.2.0 a pod could only follow ``GroupBy`` by selecting the real data + columns (or by taking the moved tags as parameters). Neither later change + moves a pipeline hash or the pod's input data, and the dropped tags make + the stored shape after the select equal to today's, so a store written by + the old shape serves the new one without recomputing. """ @staticmethod @@ -286,15 +320,13 @@ def test_rerun_on_old_store_does_not_recompute(self, store, session_source_facto assert counter.pipeline_hash() == old_counter.pipeline_hash() assert _CALLS["path"] == [], "stored results must be reused, not recomputed" - assert after.nodes["agg"].keys() == (("subject", "date", "probe"), ("path",)) - assert counter.keys() == (("subject", "date", "probe"), ("n",)) - rows = sorted( - counter.as_table().select(["date", "probe", "n"]).to_pylist(), - key=lambda r: r["date"], - ) + assert after.nodes["agg"].keys() == (("subject", "date"), ("path",)) + # The stored shape of v0.2.0 is the shape of today (ITL-1070). + assert counter.keys() == old_counter.keys() == (("subject", "date"), ("n",)) + rows = sorted(counter.as_table().to_pylist(), key=lambda r: r["date"]) assert rows == [ - {"date": "d1", "probe": [0, 1], "n": 2}, - {"date": "d2", "probe": [0, 1], "n": 2}, + {"subject": "G", "date": "d1", "n": 2}, + {"subject": "G", "date": "d2", "n": 2}, ] diff --git a/tests/test_torch/test_tensor_operators.py b/tests/test_torch/test_tensor_operators.py index 36b4a5f13..55cd7154a 100644 --- a/tests/test_torch/test_tensor_operators.py +++ b/tests/test_torch/test_tensor_operators.py @@ -151,9 +151,10 @@ def test_group_by() -> None: out = list(GroupBy(by=["g"])(tensors_source()).iter_data()) assert sorted(fields(t)["g"] for t, _ in out) == ["g0", "g1", "g2"] for tag, data in out: - # GroupBy keeps the non-key tag ``k`` as a list-valued tag. - names = list(fields(tag)["k"]) - assert sorted(names) == sorted(n for n in NAMES if GROUP[n] == fields(tag)["g"]) + # GroupBy drops the non-key tag ``k`` (ITL-1070); it still orders the + # members, so the group's names come in ascending order. + assert set(fields(tag)) == {"g"} + names = sorted(n for n in NAMES if GROUP[n] == fields(tag)["g"]) for name, x in zip(names, fields(data)["x"], strict=True): assert_tensors_identical(x, expected_x(name)) From 7850774c9d8750a312981dc5f902497f4540a1db Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 10:34:44 -0800 Subject: [PATCH 3/4] docs(group_by): document that GroupBy drops non-key tags; changelog and design issue (ITL-1070) Co-Authored-By: Claude Opus 5.5 --- .zed/rules | 25 ++++--- CLAUDE.md | 26 ++++--- changelog.d/ITL-1070.md | 49 +++++++++++++ changelog.d/ITL-955.md | 32 --------- design_issues/ITL-1070-GB2.md | 25 +++++++ docs/concepts/operators.md | 69 ++++++++++++------- docs/concepts/sources.md | 5 +- src/orcapod/core/operators/group_by.py | 3 +- src/orcapod/core/streams/base.py | 10 +-- ...1-13-itl-1070-groupby-drops-non-by-tags.md | 4 +- tests/test_pipeline/test_aggregation_job.py | 4 +- 11 files changed, 164 insertions(+), 88 deletions(-) create mode 100644 changelog.d/ITL-1070.md delete mode 100644 changelog.d/ITL-955.md create mode 100644 design_issues/ITL-1070-GB2.md diff --git a/.zed/rules b/.zed/rules index 5b81b2de9..93f282be6 100644 --- a/.zed/rules +++ b/.zed/rules @@ -548,7 +548,10 @@ no input tag name may appear among the data columns. Two operators reduce row count many→one. Batch groups consecutive rows by count; GroupBy groups by tag values: it collapses the N rows sharing a tag tuple into one row with list-valued members. Neither synthesizes new data values, and GroupBy keys only on tags, -never on data. Its non-key tags stay tags, list-valued, as Batch keeps them. +never on data. GroupBy keeps only its `by` tags and drops every other tag (developer, +November 13, 2026, ITL-1070): a list of the members' values is not a key, as SQL drops a +non-key column after GROUP BY. Batch has no key and keeps every tag, list-valued (developer, +same day): its lists are what tells a batch's members apart. Filters are the operators that read data columns (ITL-943): PolarsFilter evaluates its predicate on values, and DropNulls reads only whether a value is present (its validity), @@ -657,10 +660,10 @@ generation of hashing; rows of the current data context are the current entries. 2. Name-extending — multi-input ops. System tag column name gets ::{pipeline_hash}:{canonical_position} appended. Commutative operators sort by pipeline_hash and sort system tag values per row. -3. Reducing — many→one ops (Batch, GroupBy). User tag and data columns become list[T] and - stay in their own group (a tag stays a tag; only the scalar keys of a GroupBy are not - list-wrapped), and - source-info columns become list[str] with one element per member. System tag columns must +3. Reducing — many→one ops (Batch, GroupBy). Data columns become list[T]; Batch makes every + user tag list[T] and keeps it a tag, while GroupBy keeps its scalar keys and drops every + other user tag (ITL-1070). No tag ever moves to the data group. Source-info columns become + list[str] with one element per member. System tag columns must stay scalar, because _build_record_id_preimage (core/nodes/function_node.py) hashes them directly to derive record identity. They fold to a deterministic digest via arrow_utils.fold_system_tag_values and their column name gains ::{pipeline_hash}. @@ -793,10 +796,14 @@ data. The rules: Batch — group before batching, not after). Members sort by non-key tag values with system record_id as tiebreaker; groups emit in key order, so reordered input yields a byte-identical table. -- GroupBy's output tag columns are its input's tag columns, in input order: keys scalar, - every other tag list[T] (ITL-955). A grouped stream can be grouped again on a coarser key; - its list-valued tags then nest (list[list[T]]) and still order the members, so the member - sort key (_null_last_key) is recursive and sorts nulls last at every depth. +- GroupBy's output tag columns are its `by` columns, scalar, in the input's tag order; every + other tag is dropped (ITL-1070; ITL-955 had kept them as list[T] tags, never released). The + dropped tags still order the members, so the data lists are as before. Their values are not + carried: a step that needs them makes the column data before grouping (at the source, or + DerivedSource(tags_as_data=...)). A list-valued tag of the input (a list-typed source tag) + can still order members, so the member sort key (_null_last_key) is + recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser + key orders its members by record id alone. An opt-in mode that keeps non-key tags is ITL-1071. - Data source-info values may be str, None, or list[...], nested to any depth; Arrow/Python types are derived from the value, with None mapping to large_string. That rule has one definition, arrow_utils.source_info_arrow_type, and applies only to tokens supplied as diff --git a/CLAUDE.md b/CLAUDE.md index 0bebc5ade..bcf885a49 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -567,8 +567,10 @@ no input tag name may appear among the data columns. Two operators reduce row count many→one. `Batch` groups consecutive rows by count; `GroupBy` groups by tag values: it collapses the N rows sharing a tag tuple into one row with list-valued members. Neither synthesizes new data values — every emitted element came from an -input row — and `GroupBy` keys only on tags, never on data. Its non-key tags stay tags, -list-valued, as `Batch` keeps them. +input row — and `GroupBy` keys only on tags, never on data. `GroupBy` keeps only its `by` +tags and drops every other tag (developer, November 13, 2026, ITL-1070): a list of the members' +values is not a key, as SQL drops a non-key column after `GROUP BY`. `Batch` has no key and keeps +every tag, list-valued (developer, same day): its lists are what tells a batch's members apart. Filters are the operators that read data columns (ITL-943): `PolarsFilter` evaluates its predicate on values, and `DropNulls` reads only whether a value is present (its validity), @@ -686,10 +688,10 @@ current data context are the current entries. 2. **Name-extending** — multi-input ops (join, merge join). Each input's system tag column name gets `::{pipeline_hash}:{canonical_position}` appended. Commutative operators canonically order inputs by `pipeline_hash` and sort system tag values per row. -3. **Reducing** — many→one ops (`Batch`, `GroupBy`). User tag and data columns become - `list[T]` and stay in their own group (a tag stays a tag; only the scalar keys of a - `GroupBy` are not list-wrapped), and source-info columns become `list[str]` with one element - per member. System +3. **Reducing** — many→one ops (`Batch`, `GroupBy`). Data columns become `list[T]`; `Batch` + makes every user tag `list[T]` and keeps it a tag, while `GroupBy` keeps its scalar keys and + drops every other user tag (ITL-1070). No tag ever moves to the data group. Source-info + columns become `list[str]` with one element per member. System tag columns must stay **scalar**, because `_build_record_id_preimage` (`core/nodes/function_node.py`) hashes them directly to derive record identity. They fold to a deterministic digest via `arrow_utils.fold_system_tag_values` and their column name @@ -839,10 +841,14 @@ data. The rules: tag values with the system `record_id` as tiebreaker, so the hashed lists are stable across runs. Groups themselves are emitted in key order, so a reordered input yields a byte-identical table. -- `GroupBy`'s output tag columns are its input's tag columns, in input order: keys scalar, - every other tag `list[T]` (ITL-955). A grouped stream can be grouped again on a coarser key; - its list-valued tags then nest (`list[list[T]]`) and still order the members, so the member - sort key (`_null_last_key`) is recursive and sorts nulls last at every depth. +- `GroupBy`'s output tag columns are its `by` columns, scalar, in the input's tag order; every + other tag is dropped (ITL-1070; ITL-955 had kept them as `list[T]` tags, never released). The + dropped tags still order the members, so the data lists are as before. Their values are not + carried: a step that needs them makes the column data before grouping (at the source, or + `DerivedSource(tags_as_data=...)`). A list-valued tag of the input (a list-typed source tag) + can still order members, so the member sort key (`_null_last_key`) is + recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser key + orders its members by record id alone. An opt-in mode that keeps non-key tags is ITL-1071. - `Data` source-info values may be `str`, `None`, or `list[...]`, nested to any depth; the Arrow and Python types are derived from the value, with `None` mapping to `large_string`. That rule has one definition, `arrow_utils.source_info_arrow_type`, and applies only to diff --git a/changelog.d/ITL-1070.md b/changelog.d/ITL-1070.md new file mode 100644 index 000000000..1a8b3877f --- /dev/null +++ b/changelog.d/ITL-1070.md @@ -0,0 +1,49 @@ +### Breaking changes and migration + +#### `GroupBy` keeps only its `by` tags (ITL-955, ITL-1070) + +`GroupBy` used to turn every tag that was not a grouping key into a list-valued **data** column, +with null source info, where a function pod could read it. That broke the rule that separates an +operator from a function pod: no data element may come from a tag value. A tag that is not in +`by` is now **dropped**: a list of the members' values is not a key for the group, just as SQL +drops a non-key, non-aggregated column after `GROUP BY`. The output's tag columns are the `by` +columns, scalar, in the input's tag order. `Batch` is unchanged: it has no key and keeps every +tag, list-valued. + +| `GroupBy(by=["subject"])` on tags `subject, session, probe`, data `path` | v0.2.0 | Now | +|---|---|---| +| Tag columns | `subject` | `subject` | +| Data columns | `session: list[int]`, `probe: list[str]`, `path: list[str]` | `path: list[str]` | + +**Member tag values are not carried.** If a later step needs to know which member each element +came from, make the column data *before* grouping: declare it as data at the source, or use +`DerivedSource(origin, tags_as_data={"probe": "probe_name"})` and group that. + +- **What to change.** A function that took a non-key tag as a parameter + (`def f(probe: list[str], path: list[str])`) now fails input-schema validation; drop the + parameter, or offer the tag as data before grouping as above. A `select_data_columns` that was + only there to drop the moved tags can go. +- **Unchanged.** `by` must still name scalar tag columns. Members are still ordered by the + input's non-key tag values, then by record id, so every data list holds the same elements in + the same order; groups are emitted in key order. The Arrow types of the list columns, the + folded system tags and the data columns' list-valued source info are the same as before. The + all-null `_source_` columns of the moved tags are gone. +- **Grouping again.** A grouped stream can be grouped again on a coarser key; its data lists + nest (`list[list[T]]`). The finer key is gone by then, so the second grouping orders its + members by record id alone. +- **Hashes and stored results.** No pipeline hash, content hash or table path moves: the + `GroupBy` stream hashes its operator and its input, not its output columns. Against v0.2.0, + each grouped row's data hash changes where a tag was dropped, because the row's data no longer + includes it; a grouping on every tag is byte-identical. A function whose input was already + only the real data columns (`GroupBy` then `select_data_columns`) receives the same data and + stores the same columns as with v0.2.0, so it reuses its stored results without computing + again. A function whose signature has to change is a different function and computes once. + `node.as_source()` over a grouped node has a new `pipeline_hash`, because a source's pipeline + hash is its schema. +- **Stores written by a development build.** Between November 12 and this change, a + development build kept non-key tags as list-valued tags. A store it wrote that holds a + `GroupBy` over a non-key tag keeps that tag column in the operator cache and in the tables of + the function nodes after it, at the same paths. With this release, writing to such a table + fails with `StoreWriteError` (a non-null violation naming the dropped column), and reading it + (`CacheMode.REPLAY`, `get_all_records`) returns the old column. Delete those tables. No + released version wrote that shape. diff --git a/changelog.d/ITL-955.md b/changelog.d/ITL-955.md deleted file mode 100644 index 2e807e570..000000000 --- a/changelog.d/ITL-955.md +++ /dev/null @@ -1,32 +0,0 @@ -### Breaking changes and migration - -#### `GroupBy` keeps non-key tags as tags (ITL-955) - -`GroupBy` used to turn every tag that was not a grouping key into a list-valued **data** column, -with null source info, where a function pod could read it. That broke the rule that separates an -operator from a function pod: no data element may come from a tag value. Non-key tags now stay -in the tag group as list-valued tags, in the input's tag order, the way `Batch` keeps its tags. - -| `GroupBy(by=["subject"])` on tags `subject, session, probe`, data `path` | Before | Now | -|---|---|---| -| Tag columns | `subject` | `subject`, `session: list[int]`, `probe: list[str]` | -| Data columns | `session: list[int]`, `probe: list[str]`, `path: list[str]` | `path: list[str]` | - -- **What to change.** A function that took a moved tag as a parameter - (`def f(probe: list[int], path: list[str])`) now fails input-schema validation; drop the - parameter. A function cannot read a tag: if it needs the value, make it a data column of the - source. A `select_data_columns` that was only there to drop the moved tags can go. To remove - the list-valued tags from the output, use `drop_tag_columns`. -- **Unchanged.** `by` must still name scalar tag columns; members are still ordered by their - non-key tag values, then by record id; groups are emitted in key order. The Arrow types of the - list columns, the folded system tags and the data columns' list-valued source info are the - same as before. The two all-null `_source_` columns are gone. -- **New.** A grouped stream can be grouped again on a coarser key; its list-valued tags nest - (`list[list[T]]`). -- **Hashes and stored results.** No pipeline hash and no table path moves: the `GroupBy` - stream's `pipeline_hash` and `content_hash`, and the `pipeline_hash` of everything downstream, - are the same. Each grouped row's data hash changes, because the row's data no longer includes - the moved tags. A function whose input was already only the real data columns - (`GroupBy` then `select_data_columns`) receives the same data as before and reuses its stored - results without computing again; its output now also carries the list-valued tags. A function - whose signature has to change is a different function and computes once. diff --git a/design_issues/ITL-1070-GB2.md b/design_issues/ITL-1070-GB2.md new file mode 100644 index 000000000..8d5aec91c --- /dev/null +++ b/design_issues/ITL-1070-GB2.md @@ -0,0 +1,25 @@ +## `src/orcapod/core/operators/group_by.py` — tags entering the data group (ITL-955) + +### GB2 — `GroupBy` kept non-key tags as list-valued tags +**Status:** resolved +**Severity:** medium +**Issue:** ITL-1070 + +GB1 (ITL-955) stopped `GroupBy` from moving non-key tags into the data group by keeping them as +list-valued tags. A list of the members' values is not a key for the group, though, and a +grouped non-key tag that falls through a function pod is stored on every result row: in the +referential workshop pipeline it was the remaining cost (77 MB against 11 MB of store). + +The rule (developer, November 13, 2026): `GroupBy` keeps only its `by` tags; a tag that is not +a key is dropped, as SQL drops a non-key, non-aggregated column after `GROUP BY`. Data columns +are unchanged. `Batch` keeps its list-valued tags: it has no key. An opt-in mode that keeps +non-key tags as lists is ITL-1071. + +**Fix:** the output's tag columns are the `by` columns, scalar, in the input's tag order; +`unary_output_schema` predicts the same. The dropped tags still order the members, so every +data list, system-tag fold, row data hash and stream hash is unchanged; no golden value moves +(six `pipeline:group_by.row[...]` keys are renamed, because a row's key names its user tags). +Tables written by a development build between ITL-955 and this fix sit at the same paths with +the old tag column: a write to them fails and a read returns the old column; the changelog +tells users to delete them. Nothing checks a stored table's columns against the node's schema +on read; that gap is ITL-1072. diff --git a/docs/concepts/operators.md b/docs/concepts/operators.md index 9ba88daf0..a113a8ec6 100644 --- a/docs/concepts/operators.md +++ b/docs/concepts/operators.md @@ -23,10 +23,11 @@ This boundary ensures that structural operations (joins, filters) and value comp (transformations, model inference) are cleanly separated, making pipelines easier to reason about and optimize. -`GroupBy` is the one operator that reduces row count many-to-one: it collapses N rows sharing -a tag tuple into a single row with list-valued members. It stays on the operator side of the -boundary because it synthesizes no new data values -- every element of every emitted list came -from an input row -- and it keys only on tags, never on data content. +`GroupBy` is the one operator that reduces row count many-to-one by key: it collapses N rows +sharing a key into a single row with list-valued data members, and keeps only its key tags. It +stays on the operator side of the boundary because it synthesizes no new data values -- every +element of every emitted list came from an input row -- and it keys only on tags, never on data +content. One rule holds for every operator: **no data element comes from a tag value**. An operator may use tags to join, filter, group or order rows, but it never turns a tag column into a data @@ -204,6 +205,10 @@ batch may be smaller; pass `drop_partial_batch=True` to discard it: batch = Batch(batch_size=10, drop_partial_batch=True) ``` +`Batch` keeps every tag, list-valued. It batches blindly, by count: it has no key, so the lists +of tag values are the only thing that tells a batch's members apart. `GroupBy` (below) has a +key and keeps only that. + ### GroupBy Collapses every set of rows that share a tag tuple into a single row -- the one many-to-one @@ -226,43 +231,57 @@ source = DictSource( grouped = GroupBy(by=["subject_id"])(source) print(grouped.keys()) -# (('subject_id', 'probe'), ('spike_count',)) +# (('subject_id',), ('spike_count',)) for tag, data in grouped.iter_data(): print("Tags:", tag.as_dict(), "Data:", data.as_dict()) - # Tags: {'subject_id': 'mouse_01', 'probe': ['imec0', 'imec1']} Data: {'spike_count': [1200, 980]} - # Tags: {'subject_id': 'mouse_02', 'probe': ['imec0']} Data: {'spike_count': [1450]} + # Tags: {'subject_id': 'mouse_01'} Data: {'spike_count': [1200, 980]} + # Tags: {'subject_id': 'mouse_02'} Data: {'spike_count': [1450]} ``` The output schema follows three rules: -- **Group keys stay scalar tag columns** (`subject_id: str`). -- **Non-key tag columns stay tag columns and become list-valued** (`probe: list[str]`), one - element per member, in the same order as the data lists. The output's tag columns are the - input's tag columns, in the same order. This is what `Batch` does with its tags too. -- **Data columns become `list[T]`** (`spike_count: list[int]`). +- **Group keys stay scalar tag columns** (`subject_id: str`), in the input's tag order. They are + the only tag columns of the output. +- **A tag that is not a key is dropped** (`probe`). A list of the members' values is not a key + for the group, just as SQL drops a non-key, non-aggregated column after `GROUP BY`. +- **Data columns become `list[T]`** (`spike_count: list[int]`), one element per member. -The data columns of the output are exactly the data columns of the input. A function pod that -follows `GroupBy` receives `spike_count` only; it cannot read `probe`, because a pod never sees -tags. If the function needs to know which member each element came from, make that value a data -column of the source. To remove the list-valued tags, use `drop_tag_columns`: +!!! warning "Member tag values are not carried" + `GroupBy` does not keep the members' values of a non-key tag, as a tag or as data. If a + later step needs to know which member each element came from, make the column data + *before* grouping: declare it as data at the source, or derive a source that offers the tag + as data with `DerivedSource(origin, tags_as_data={...})`. ```python -print(grouped.drop_tag_columns(["probe"]).keys()) -# (('subject_id',), ('spike_count',)) +from orcapod.sources import DerivedSource + +with_probe = DerivedSource(source, tags_as_data={"probe": "probe_name"}) +by_subject = GroupBy(by=["subject_id"])(with_probe) +print(by_subject.keys()) +# (('subject_id',), ('spike_count', 'probe_name')) +for tag, data in by_subject.iter_data(): + print(tag.as_dict(), data.as_dict()) + # {'subject_id': 'mouse_01'} {'spike_count': [1200, 980], 'probe_name': ['imec0', 'imec1']} + # {'subject_id': 'mouse_02'} {'spike_count': [1450], 'probe_name': ['imec0']} ``` -A grouped stream can be grouped again on a coarser key. Its list-valued tags then nest one level -deeper (`list[list[T]]`), like the data lists. +The data columns of the output are exactly the data columns of the input: a function pod that +follows `GroupBy` receives `spike_count` (and `probe_name`, when the source offers it), never a +tag. Ordering is deterministic by construction, because orcapod hashes the emitted lists to build -cache keys: members are sorted by their non-key tag values (with the internal record id as a -final tiebreaker), and groups are emitted in group-key order. A reordered input therefore -produces a byte-identical output table. +cache keys: members are sorted by the input's non-key tag values (with the internal record id +as a final tiebreaker), even though those tags are dropped, and groups are emitted in group-key +order. A reordered input therefore produces a byte-identical output table. + +A grouped stream can be grouped again on a coarser key; its data lists then nest one level +deeper (`list[list[T]]`). The finer key is gone by then, so the members of the second grouping +are ordered by record id alone. Every name in `by` must be a **scalar tag column** of the input. `GroupBy` raises `InputValidationError` for an unknown column, a data column, or a list-valued tag column -- -the last means the stream came from `Batch` or from an earlier `GroupBy` that did not key on -that column, so group *before* batching rather than after. +the last usually means the stream came from `Batch`, so group *before* batching rather than +after. Streams also expose this as a fluent method: diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index 75155e01a..83d4a8319 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -194,14 +194,15 @@ unit_key = [ "spike_extraction_hash", "subject_id", "session_start_datetime", "recording_index", "probe_index", "unit_id", ] -per_unit = spikes.group_by(unit_key).drop_tag_columns("timestamp") +per_unit = spikes.group_by(unit_key) print(per_unit.keys()) # (('spike_extraction_hash', 'subject_id', 'session_start_datetime', 'recording_index', 'probe_index', 'unit_id'), ('spike_amplitude_uv', 'spike_time_us')) ``` `per_unit` has one row per unit, and its `spike_time_us` list is in time order: `group_by` orders the members of a group by the tags outside the group key, and on the full unit key -the only such tag is `timestamp`. `group_by(["unit_id"])` alone sorts by +the only such tag is `timestamp`. The `timestamp` tag itself is dropped, as `group_by` drops +every tag outside the key; its copy `spike_time_us` is data and stays. `group_by(["unit_id"])` alone sorts by `spike_extraction_hash`, subject, session, recording, probe, then `timestamp`, so time order holds only within one session and probe; group on the full unit key. diff --git a/src/orcapod/core/operators/group_by.py b/src/orcapod/core/operators/group_by.py index 6ba8e7e32..43f9ee36c 100644 --- a/src/orcapod/core/operators/group_by.py +++ b/src/orcapod/core/operators/group_by.py @@ -25,8 +25,7 @@ def _null_last_key(value: Any) -> tuple[Any, ...]: The leading integer keeps a null comparable against a real value of any type. A list is keyed element by element, so a list-valued non-key tag - that holds a null (a list-typed tag column of a source, or the tags of a - ``Batch`` output joined in) still compares. + that holds a null (a list-typed tag column of a source) still compares. Args: value: A tag or system-tag value: a scalar, ``None``, or a list of diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 00788eb01..eb1728f65 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -188,11 +188,13 @@ def group_by( by: Collection[str], label: str | None = None, ) -> StreamBase: - """Reduce rows sharing a tag tuple into one packet per group. + """Reduce rows sharing a key into one row per group. - Group-key columns stay scalar tags; every other column becomes - list-valued and stays in its group: a non-key tag stays a tag, a data - column stays data. See ``orcapod.core.operators.GroupBy``. + Group-key columns stay scalar tags and are the only tags of the output; + every other tag is dropped, and every data column becomes list-valued. + A dropped tag's values are not carried: make the column data before + grouping if a later step needs it. See + ``orcapod.core.operators.GroupBy``. Args: by: Tag column names to group on. diff --git a/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md b/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md index 08b27e0d0..4c20e7922 100644 --- a/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md +++ b/superpowers/plans/2026-11-13-itl-1070-groupby-drops-non-by-tags.md @@ -1,6 +1,6 @@ # ITL-1070 — GroupBy drops the tags that are not in `by` -Status: DRAFT, plan sent to coordinator for approval (hash-critical). No code before approval. +Status: APPROVED by coordinator 2026-11-13 (option A: no identity change; delete changelog.d/ITL-955.md and make ITL-1070.md the net entry; keep _null_last_key recursion; stale-table gap filed as ITL-1072). Implemented. Branch: `eywalker/itl-1070-groupby-drops-the-tags-that-are-not-in-by` (from origin/main 5c7a058f). History: ITL-955 (#316) made non-key tags list-valued tags. Future opt-in mode: ITL-1071 (not ours). @@ -73,4 +73,4 @@ Prototype diff saved at scratchpad `proto.diff` (10 lines in group_by.py). ## Notes for successor -Plan sent to coordinator via ask (2026-11-13); waiting on decision about stale v0.3-dev tables. +Implemented, tests/docs/changelog/design issue done; ITL-1048 note posted. Next: PR, Copilot review, CI on 3.11/3.12/3.13. diff --git a/tests/test_pipeline/test_aggregation_job.py b/tests/test_pipeline/test_aggregation_job.py index 5764a25e2..9704b7c82 100644 --- a/tests/test_pipeline/test_aggregation_job.py +++ b/tests/test_pipeline/test_aggregation_job.py @@ -9,6 +9,7 @@ from __future__ import annotations from collections.abc import Iterator +from typing import Any from pathlib import Path import pyarrow as pa @@ -22,7 +23,6 @@ from orcapod.core.streams import ArrowTableStream from orcapod.databases import DeltaTableDatabase from orcapod.pipeline import PipelineJob -from orcapod.system_constants import constants from orcapod.types import Schema # --------------------------------------------------------------------------- @@ -224,7 +224,7 @@ def unary_static_process(self, stream): tag_columns, _ = stream.keys() non_key = [c for c in tag_columns if c not in self.by] input_table = stream.as_table() - members: dict[tuple, list[dict]] = {} + members: dict[tuple[Any, ...], list[dict[str, Any]]] = {} for row in input_table.to_pylist(): members.setdefault(tuple(row[c] for c in self.by), []).append(row) keys = [ From 753a3e586b520e867a30ca794cceffd2830c779d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 10:59:03 -0800 Subject: [PATCH 4/4] docs(group_by): a regrouped stream orders members by the first grouping's keys; exact regroup order in test (ITL-1070) Address CodeRabbit review on #365. Co-Authored-By: Claude Opus 5.5 --- .zed/rules | 2 +- CLAUDE.md | 2 +- changelog.d/ITL-1070.md | 5 +++-- docs/concepts/operators.md | 5 +++-- src/orcapod/core/operators/group_by.py | 10 ++++++---- tests/test_core/operators/test_group_by.py | 8 ++++---- 6 files changed, 18 insertions(+), 14 deletions(-) diff --git a/.zed/rules b/.zed/rules index 93f282be6..779681b3b 100644 --- a/.zed/rules +++ b/.zed/rules @@ -803,7 +803,7 @@ data. The rules: DerivedSource(tags_as_data=...)). A list-valued tag of the input (a list-typed source tag) can still order members, so the member sort key (_null_last_key) is recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser - key orders its members by record id alone. An opt-in mode that keeps non-key tags is ITL-1071. + key orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071. - Data source-info values may be str, None, or list[...], nested to any depth; Arrow/Python types are derived from the value, with None mapping to large_string. That rule has one definition, arrow_utils.source_info_arrow_type, and applies only to tokens supplied as diff --git a/CLAUDE.md b/CLAUDE.md index bcf885a49..b6592cc81 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -848,7 +848,7 @@ data. The rules: `DerivedSource(tags_as_data=...)`). A list-valued tag of the input (a list-typed source tag) can still order members, so the member sort key (`_null_last_key`) is recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser key - orders its members by record id alone. An opt-in mode that keeps non-key tags is ITL-1071. + orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071. - `Data` source-info values may be `str`, `None`, or `list[...]`, nested to any depth; the Arrow and Python types are derived from the value, with `None` mapping to `large_string`. That rule has one definition, `arrow_utils.source_info_arrow_type`, and applies only to diff --git a/changelog.d/ITL-1070.md b/changelog.d/ITL-1070.md index 1a8b3877f..92730bc05 100644 --- a/changelog.d/ITL-1070.md +++ b/changelog.d/ITL-1070.md @@ -29,8 +29,9 @@ came from, make the column data *before* grouping: declare it as data at the sou folded system tags and the data columns' list-valued source info are the same as before. The all-null `_source_` columns of the moved tags are gone. - **Grouping again.** A grouped stream can be grouped again on a coarser key; its data lists - nest (`list[list[T]]`). The finer key is gone by then, so the second grouping orders its - members by record id alone. + nest (`list[list[T]]`). The second grouping orders its members by the first grouping's keys + that are not in the coarser key, then by record id; a tag the first grouping dropped no + longer takes part. - **Hashes and stored results.** No pipeline hash, content hash or table path moves: the `GroupBy` stream hashes its operator and its input, not its output columns. Against v0.2.0, each grouped row's data hash changes where a tag was dropped, because the row's data no longer diff --git a/docs/concepts/operators.md b/docs/concepts/operators.md index a113a8ec6..60d6656e0 100644 --- a/docs/concepts/operators.md +++ b/docs/concepts/operators.md @@ -275,8 +275,9 @@ as a final tiebreaker), even though those tags are dropped, and groups are emitt order. A reordered input therefore produces a byte-identical output table. A grouped stream can be grouped again on a coarser key; its data lists then nest one level -deeper (`list[list[T]]`). The finer key is gone by then, so the members of the second grouping -are ordered by record id alone. +deeper (`list[list[T]]`). The members of the second grouping are ordered by the first +grouping's keys that are not in the coarser key, then by record id; a tag the first grouping +dropped no longer takes part. Every name in `by` must be a **scalar tag column** of the input. `GroupBy` raises `InputValidationError` for an unknown column, a data column, or a list-valued tag column -- diff --git a/src/orcapod/core/operators/group_by.py b/src/orcapod/core/operators/group_by.py index 43f9ee36c..7875e77d0 100644 --- a/src/orcapod/core/operators/group_by.py +++ b/src/orcapod/core/operators/group_by.py @@ -85,10 +85,12 @@ class GroupBy(UnaryOperator): undefined if the source table itself is permuted. A grouped stream can be grouped again on a coarser key: its data lists nest - one level deeper (``list[list[T]]``), as source info does. The finer key - is gone by then, so the members of the second grouping are ordered by - ``record_id`` alone. A list-valued tag of the input can order members but - cannot be a grouping key. + one level deeper (``list[list[T]]``), as source info does. The members of + the second grouping are ordered by the first grouping's keys that are not + in the coarser key (scalar tags of the grouped stream), then by + ``record_id``; a tag the first grouping dropped no longer takes part. A + list-valued tag of the input can order members but cannot be a grouping + key. Contrast with ``Batch``, which partitions by row count for throughput. It has no key, so it keeps every tag column, list-valued: those lists are diff --git a/tests/test_core/operators/test_group_by.py b/tests/test_core/operators/test_group_by.py index b0ce47a28..6e8c24f88 100644 --- a/tests/test_core/operators/test_group_by.py +++ b/tests/test_core/operators/test_group_by.py @@ -160,16 +160,16 @@ def test_grouping_on_every_tag_keeps_every_tag(self, session_source): def test_regroup_on_a_coarser_key(self, session_source): """A grouped stream grouped again keeps the coarser key; data nests. - The first grouping dropped ``date`` and ``probe``, so no tag is left to - order the members of the second grouping: they are ordered by their - folded ``record_id``, which is stable across runs. + The first grouping keeps ``date`` as a key and drops ``probe``; the + second drops ``date`` but still orders its members by it (``record_id`` + is only the tiebreaker), so the ``d1`` group comes first. """ out = GroupBy(by=["subject"])(GroupBy(by=["subject", "date"])(session_source)) assert out.keys() == (("subject",), ("path",)) rows = out.as_table().to_pylist() assert len(rows) == 1 assert rows[0]["subject"] == "G" - assert sorted(rows[0]["path"]) == [["a", "b"], ["c", "d"]] + assert rows[0]["path"] == [["a", "b"], ["c", "d"]] def test_list_valued_non_key_tag_orders_members(self): """A list-typed tag of the input still orders members, nulls last.