Repository navigation
fix(group_by): keep non-key tags as list-valued tags, never data (ITL-955) #316
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
5411cf6
6f77cbf
f773956
29b8f94
361298b
3f5271e
7f1e1eb
c3360a3
4bf2c6f
1bc26ba
c9977de
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| #### `GroupBy` keeps non-key tags as tags (ITL-955) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Put the migration text inside bullet items. This fragment places a subsection, prose, and a table directly under the category heading. Move them into bullet items. As per coding guidelines, each category heading must be “followed by its bullet lines.”
|
||
|
|
||
| `GroupBy` used to turn every tag that was not a grouping key into a list-valued **data** column, | ||
| with null source info, where a function pod could read it. That broke the rule that separates an | ||
| operator from a function pod: no data element may come from a tag value. Non-key tags now stay | ||
| in the tag group as list-valued tags, in the input's tag order, the way `Batch` keeps its tags. | ||
|
|
||
| | `GroupBy(by=["subject"])` on tags `subject, session, probe`, data `path` | Before | Now | | ||
| |---|---|---| | ||
| | Tag columns | `subject` | `subject`, `session: list[int]`, `probe: list[str]` | | ||
| | Data columns | `session: list[int]`, `probe: list[str]`, `path: list[str]` | `path: list[str]` | | ||
|
|
||
| - **What to change.** A function that took a moved tag as a parameter | ||
| (`def f(probe: list[int], path: list[str])`) now fails input-schema validation; drop the | ||
| parameter. A function cannot read a tag: if it needs the value, make it a data column of the | ||
| source. A `select_data_columns` that was only there to drop the moved tags can go. To remove | ||
| the list-valued tags from the output, use `drop_tag_columns`. | ||
| - **Unchanged.** `by` must still name scalar tag columns; members are still ordered by their | ||
| non-key tag values, then by record id; groups are emitted in key order. The Arrow types of the | ||
| list columns, the folded system tags and the data columns' list-valued source info are the | ||
| same as before. The two all-null `_source_<tag>` columns are gone. | ||
| - **New.** A grouped stream can be grouped again on a coarser key; its list-valued tags nest | ||
| (`list[list[T]]`). | ||
| - **Hashes and stored results.** No pipeline hash and no table path moves: the `GroupBy` | ||
| stream's `pipeline_hash` and `content_hash`, and the `pipeline_hash` of everything downstream, | ||
| are the same. Each grouped row's data hash changes, because the row's data no longer includes | ||
|
Comment on lines
+26
to
+28
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| the moved tags. A function whose input was already only the real data columns | ||
| (`GroupBy` then `select_data_columns`) receives the same data as before and reuses its stored | ||
| results without computing again; its output now also carries the list-valued tags. A function | ||
| whose signature has to change is a different function and computes once. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| ## `src/orcapod/core/operators/group_by.py` — tags entering the data group (ITL-955) | ||
|
|
||
| ### GB1 — `GroupBy` moved non-key tag values into data columns | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-955 | ||
|
|
||
| `GroupBy(by=[...])` built the right table (keys scalar, every other column list-valued) but | ||
| declared only the keys as the output's tag columns, so each non-key tag became a list-valued | ||
| **data** column with an all-null `_source_<tag>` column. A function pod could then read tag | ||
| content, which breaks the operator / function pod boundary. `Batch` kept its tags as tags. | ||
|
|
||
| The rule (developer, November 11, 2026): no data element may come from a tag value. An operator | ||
| may use tags, but tag content never becomes data. | ||
|
|
||
| **Fix:** the output's tag columns are the input's tag columns, in input order: keys scalar, | ||
| every other tag `list[T]`; `unary_output_schema` predicts the same. A grouped stream can now be | ||
| grouped again on a coarser key with list-valued non-key tags, so the member sort key | ||
| (`_null_last_key`) is recursive and sorts nulls last at every depth; for scalar tags the order is | ||
| unchanged. No pipeline hash or table path moves; each grouped row's data hash does (six pinned | ||
| `pipeline:group_by.row[...]` golden entries replaced, approved). Results stored by a pod that | ||
| followed `GroupBy` through `select_data_columns` are reused without recomputation | ||
| (`tests/test_pipeline/test_aggregation_job.py::TestStoreWrittenBeforeTagsStayedTags`). | ||
|
|
||
| Audit of the other operators against the rule (November 11, 2026): `Batch`, `Join`, `MergeJoin`, | ||
| `SemiJoin`, `SelectTagColumns`, `DropTagColumns`, `SelectDataColumns`, `DropDataColumns`, | ||
| `MapTags`, `MapData`, `PolarsFilter`, `Index` and `Pick` never put an input tag into the data | ||
| group. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct the claim that only
GroupByreduces row count.Batchalso combines multiple input rows into one output row when its batch size exceeds one. This instruction contradicts the reducing rule at Line 422 and can cause an operator audit to missBatch. DescribeGroupByas the operator that groups by a tag tuple. Correct the matching claim inCLAUDE.mdLine 399.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declined in this PR. The sentence is main's existing wording, unchanged by this PR (the diff against
maindoes not touch it). The point stands thatBatchalso reduces many rows to one; the wording will be fixed separately inCLAUDE.mdand.zed/rulestogether.Posted by a Claude Code agent on behalf of @eywalker.