Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
fc22de6
test(referential_tags): failing EDI-shaped end-to-end test (ITL-1023)
Nov 13, 2026
02b6235
feat(referential_tags): WIP representation, sources, select/drop, Joi…
Nov 13, 2026
c3c5e99
feat(referential_tags): row-preserving ops, renames, guards, async Jo…
Nov 13, 2026
c42eece
feat(referential_tags): renames per developer decision; EDI test on t…
Nov 13, 2026
880dcd4
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
853da82
test(referential_tags): existing tests pin plain tags where they asse…
Nov 13, 2026
1ec232e
feat(pipeline): saved job and pipeline format 0.3.0; v0.3 files load …
Nov 13, 2026
6120671
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
860045e
test(referential_tags): pin plain tags in main's torch golden case an…
Nov 13, 2026
6a6d72b
feat(function_pod): referential tags fall through a pod call (ITL-1023)
Nov 13, 2026
ed51400
fix(referential_tags): fall-through keeps exact types and returned ob…
Nov 13, 2026
5832772
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
3e12d62
fix(referential_tags): type-check clean; fall-through pod refuses to …
Nov 13, 2026
d8628af
test(pipeline): commit the saved-job fixture's CSV and Parquet files …
Nov 13, 2026
1fd2daa
test(objective): pin plain tags where the objective tests assume v0.3…
Nov 13, 2026
34fb2a0
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4473e2e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4c0c687
docs(plan): ITL-1023 status log
Nov 13, 2026
5607af1
docs(plan): ITL-1017 merged source-info token plan
Nov 13, 2026
5e8028e
fix(referential_tags): one merged source-info token for a value merge…
Nov 13, 2026
aaff7da
docs(referential_tags): merged source-info token, changelog, open poi…
Nov 13, 2026
8d7e3c9
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
c78ba80
docs(plan): ITL-1017 status log
Nov 13, 2026
8c7cfb6
Merge origin/main into the referential tags slice (ITL-1023, ITL-1017)
Nov 14, 2026
0bfd72a
Merge origin/main (ITL-1074) into the referential tags slice (ITL-1023)
Nov 14, 2026
634ad40
fix(group_by): a referential non-key tag keeps its data column as a l…
Nov 14, 2026
99b5737
Merge origin/main (4dd015a4) into the referential tags slice (ITL-1023)
Nov 16, 2026
404066e
fix(referential_tags): main's new paths meet referential tags (ITL-1023)
Nov 16, 2026
177d5ad
fix(referential_tags): GroupBy key order, meta stripping and join err…
Nov 16, 2026
61129ae
fix(referential_tags): zero-row heads keep references; pin main's new…
Nov 16, 2026
f359aa0
docs: main's new pages pin plain tags on the referential tags slice (…
Nov 16, 2026
3e0ae5a
feat(join): a referential key that differs only in nullability joins …
Nov 16, 2026
22e7662
test(referential_tags): type-clean key nullability assertion (ITL-1023)
Nov 16, 2026
157632b
test: pin plain tags in main's remaining v0.3-shape suites (ITL-1023)
Nov 16, 2026
84d1312
feat(referential_tags): stored tables and saved jobs keep tag binding…
Nov 16, 2026
199221c
feat(referential_tags): refuse a database that cannot keep the marker…
Nov 16, 2026
8f7d50a
ci(coderabbit): review pull requests that target any base branch, on …
eywalker Nov 16, 2026
7d6c580
chore: merge the slice branch (.coderabbit.yaml, PR #405) into ITL-1019
Nov 16, 2026
fc1db0a
fix(pipeline): a saved-node error of a loaded job names the job file …
Nov 16, 2026
00818bd
fix(pipeline): refuse a saved source config whose bindings contradict…
Nov 16, 2026
ec5984a
fix(pipeline): check an explicitly empty tag_columns against the slot…
Nov 16, 2026
26d981a
feat(referential_tags): operator nodes store and derived sources pres…
Nov 16, 2026
cb3dbc4
refactor(operator_node): the stored-table check returns nothing; one …
Nov 16, 2026
3c51256
test(referential_tags): reach the DerivedSource binding check; log IT…
Nov 16, 2026
cecbaf3
feat(databases): a database states keeps_field_metadata; an undeclare…
Nov 16, 2026
25ff7db
feat(batch): Batch on referential tags (ITL-1017) — into the referent…
eywalker Nov 16, 2026
67435e0
feat(referential_tags): SelectColumns/DropColumns on referential tags…
eywalker Nov 16, 2026
57aaa5c
Merge origin/main (37716c76, ITL-1154) into the referential tags slic…
Nov 16, 2026
c111ca4
feat(referential_tags): stored tables and saved jobs keep tag binding…
eywalker Nov 16, 2026
69776b0
chore: merge the slice branch (PR #402 squash-merged) into ITL-1019 W3
Nov 16, 2026
40c57da
Merge the slice head c111ca47 (#402, ITL-1019) into the ITL-1154 refr…
Nov 16, 2026
7137e72
fix(databases): a SpiralDB retry publishes metadata whose write faile…
Nov 17, 2026
1e9565a
fix(databases): a SpiralDB write that stores no row never replaces st…
Nov 17, 2026
5261e9a
test(referential_tags): type-clean field metadata in the SpiralDB sto…
Nov 17, 2026
ded5305
fix(databases): SpiralDB writes a table's Arrow metadata before its r…
Nov 17, 2026
f030aab
fix(databases): a failed first SpiralDB write removes the metadata it…
Nov 17, 2026
edc98fd
Merge origin/main (62899bda: #408, #409, #410, #411) into the referen…
Nov 17, 2026
4b49640
fix(databases): a failed SpiralDB write puts the metadata entry back …
Nov 17, 2026
a6cf552
fix(databases): read the SpiralDB metadata snapshot before the nullab…
Nov 17, 2026
ac68895
Merge origin/main (7c735b7a, v0.3.0 release commit) into the referent…
Nov 17, 2026
01f3db4
docs(changelog): concurrent SpiralDB writers are not coordinated; fol…
Nov 17, 2026
6274a3c
chore: merge the refreshed slice (main 7c735b7a, v0.3.0) into ITL-101…
Nov 17, 2026
a7f08ed
test(referential_tags): meet main's SpiralDB column refusal and hash-…
Nov 17, 2026
ce32310
fix(operator_node): replayed rows carry the node's data context in th…
Nov 17, 2026
8b73ba7
feat(operators): TagBy / stream.tag_by makes data columns referential…
Nov 17, 2026
6aa2314
docs(referential-tags): describe the three moves by role after tag_by…
Nov 17, 2026
e037e58
Merge pull request #407 from metamorphic-labs/eywalker/itl-1019-w3-de…
eywalker Nov 17, 2026
e91d256
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
43bb0ef
Merge origin/main (23fd8f2a: #414, #415, #416) into the referential t…
Nov 17, 2026
eb32ab1
test(tag_by): a logged TagBy operator node stores its new tag marked …
Nov 17, 2026
9c48452
chore(changelog): fold the slice's fragments into changelog.d/ITL-102…
Nov 17, 2026
a6b4a17
docs(referential-tags): select and drop act on a referential tag by r…
Nov 17, 2026
3ab814e
refactor(referential_tags): remove ReferenceMaterializationWarning (I…
Nov 17, 2026
194254e
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
8cd398f
fix(tag_by): keep the input's column layout and data context (ITL-1175)
Nov 17, 2026
424f731
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
5292a60
revert(referential_tags): keep ReferenceMaterializationWarning (ITL-1…
Nov 17, 2026
1e585a7
Merge origin/main (565201b2: #417) into the referential tags slice (I…
Nov 17, 2026
27fee5a
feat(merge_join): handle referential tags as Join does (ITL-1194)
Nov 17, 2026
83ff606
fix(tag_by): keep each row's data context key; correct the spec on LO…
Nov 17, 2026
35579f8
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
01c3b72
docs(design_issues): log ST5, the two tag orders of ArrowTableStream …
Nov 17, 2026
447597b
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
50b580b
fix(referential_tags): address the CodeRabbit review of the slice (IT…
Nov 17, 2026
016052a
docs(referential_tags): log the deferred review findings (ITL-1199 to…
Nov 17, 2026
fc205da
docs(referential-tags): sort the MergeJoin sample's rows before print…
Nov 17, 2026
5dd0634
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
378eca2
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
d7d1b34
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
38478e0
Merge origin/main (232ee360: #418) into the referential tags slice (I…
Nov 17, 2026
05deee1
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
a373ca6
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
09ffdf7
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
0937242
Merge pull request #423 from metamorphic-labs/eywalker/itl-1194-merge…
eywalker Nov 17, 2026
460ff01
chore(changelog): fold the ITL-1194 fragment (#423) into changelog.d/…
Nov 17, 2026
816d8c8
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
bd8f020
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
e43c609
docs(design_issues): log AB1, the async barrier input's column order …
Nov 17, 2026
c4cee17
Merge pull request #421 from metamorphic-labs/eywalker/itl-1195-opera…
eywalker Nov 17, 2026
b77cb01
refactor(data_tags): rename "referential tag" to "data-tag" (ITL-1023)
Nov 17, 2026
b011809
Merge origin/main (51be5e4e: #422 add_tags, #419) into the data-tags …
Nov 17, 2026
c2aafc4
fix(data_tags): data-tags come first among the data columns (ITL-1205)
Nov 17, 2026
20309a0
feat(add_tags): AddTags works on a stream with data-tags (ITL-1196, I…
Nov 17, 2026
0248fc0
fix(data_tags): async rows keep data-tags first; fall-through prevali…
Nov 17, 2026
8d6a1d9
docs(data_tags): answer the CodeRabbit doc findings on PR #342 (ITL-1…
Nov 17, 2026
3817334
Merge origin/main (7c8c5efb: #425 Starlight docs site) into the data-…
Nov 17, 2026
66d6ed7
fix(side_effects): pass a data-tag only to a parameter that takes a k…
Nov 17, 2026
99487ce
fix(data_tags): refuse an EmptyData hit of a fall-through node; selec…
Nov 17, 2026
ce427c4
Merge origin/main (ad3f2411: #424 pipeline diagram, #430 Graphviz rem…
Nov 17, 2026
b702ce4
fix(pick, index): only a data-tag is refused in place; a plain tag ge…
Nov 17, 2026
c06ecf0
feat(group_by, tag_by): member order free of the input's column arran…
Nov 17, 2026
b7dbbc0
feat(group_by)!: the order of `by` is not identity; groups come out b…
Nov 17, 2026
c7a8e73
test(golden): pin GroupBy over several tags and data columns, in thre…
Nov 17, 2026
e3eca07
docs(group_by, tag_by): member order, key order and arrangement rule;…
Nov 17, 2026
9246962
test(group_by): type the arrangement tests on StreamBase (ITL-1226)
Nov 17, 2026
f06d4ba
Merge origin/main into ITL-1226 (after #342)
Nov 18, 2026
e4a84f6
Merge origin/main into ITL-1226 (after #428)
Nov 18, 2026
bc46d75
docs(group_by): stored caches keep pre-upgrade rows after the member-…
Nov 18, 2026
8c633f7
docs(group_by): keep the stored-rows migration note in the changelog …
Nov 18, 2026
c89f2b4
Merge origin/main into ITL-1226 (ITL-1221 column_names, ITL-1224 docs)
Nov 18, 2026
969f9b0
fix(group_by): a bare-string member_order names the list form in its …
Nov 18, 2026
90c0b60
Merge origin/main into ITL-1226 (notebooks page, #433)
Nov 18, 2026
4129b70
docs(notebooks): show the last rows of an explicitly sorted view (ITL…
Nov 18, 2026
2f377fc
fix(group_by): a NaN tag has one place in the member order (ITL-1226)
Nov 18, 2026
6aacdf6
Merge origin/main into ITL-1226 (#446: data context v1 reopened)
Nov 18, 2026
64194a5
docs(group_by): the stored-table note names NaN-ordered members; data…
Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -1179,9 +1179,9 @@ data. The rules:
unchanged path; any other type difference, and every data column, behaves as before.
- GroupBy requires every column in `by` to be a scalar tag column; raises InputValidationError
for unknown, data, or list-valued columns (list-valued usually means the stream came from
Batch — group before batching, not after). Members sort by non-key tag values with system
record_id, then source id, as tiebreakers; groups emit in key order, so reordered input yields a
byte-identical table.
Batch — group before batching, not after). Members sort by the non-key tags in name order
(or as member_order gives) with system record_id, then source id, as tiebreakers; groups
emit sorted by their keys in name order, so reordered input yields a byte-identical table.
- GroupBy's output tag columns are its `by` columns, scalar, in the input's tag order; every
other tag is dropped (ITL-1070; ITL-955 had kept them as list[T] tags, never released). The
dropped tags still order the members, so the data lists are as before. Their values are not
Expand All @@ -1190,6 +1190,18 @@ data. The rules:
can still order members, so the member sort key (_null_last_key) is
recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser
key orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- GroupBy never depends on how its input's columns are arranged (developer, November 17,
2026, ITL-1226). Members are ordered by the non-key tags in name order (each ascending,
nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
only: a data column, a key, a repeat, a set or an empty list is refused; `"name"` is the
default; an ordering function is reserved, refused for now) names tags to order by first,
the rest follow in name order. A member is never ordered by a data value. The keys in `by`
are kept in name order: their order is not identity, and groups are emitted sorted by them.
`member_order` enters the identity and config only when given. Every arrangement of the
input (tag order, data order, table layout, the order of `by`) gives identical values,
record ids, folds, source-info tokens and hashes; only the output's column order follows
the input, as presentation. `TagBy`'s identity is its sorted names, and its new tags are
placed in name order.
- Data source-info values may be str, None, or list[...], nested to any depth; Arrow/Python
types are derived from the value, with None mapping to large_string. That rule has one
definition, arrow_utils.source_info_arrow_type, and applies only to tokens supplied as
Expand Down
21 changes: 17 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1223,10 +1223,11 @@ data. The rules:
unchanged path; any other type difference, and every data column, behaves as before.
- `GroupBy` requires every column in `by` to be a scalar tag column; it raises
`InputValidationError` for unknown, data, or list-valued columns (the last usually means the
stream came from `Batch` — group before batching, not after). Members are sorted by non-key
tag values with the system `record_id` and then the source id as tiebreakers (a record id
names a row within its source only), so the hashed lists are stable across runs. Groups themselves are emitted in key order, so a reordered input yields a
byte-identical table.
stream came from `Batch` — group before batching, not after). Members are sorted by the
non-key tags in name order (or as `member_order` gives), with the system `record_id` and
then the source id as tiebreakers (a record id names a row within its source only), so the
hashed lists are stable across runs. Groups themselves are emitted sorted by their keys in
name order, so a reordered input yields a byte-identical table.
- `GroupBy`'s output tag columns are its `by` columns, scalar, in the input's tag order; every
other tag is dropped (ITL-1070; ITL-955 had kept them as `list[T]` tags, never released). The
dropped tags still order the members, so the data lists are as before. Their values are not
Expand All @@ -1235,6 +1236,18 @@ data. The rules:
can still order members, so the member sort key (`_null_last_key`) is
recursive and sorts nulls last at every depth. A grouped stream grouped again on a coarser key
orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- GroupBy never depends on how its input's columns are arranged (developer, November 17,
2026, ITL-1226). Members are ordered by the non-key tags in name order (each ascending,
nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
only: a data column, a key, a repeat, a set or an empty list is refused; `"name"` is the
default; an ordering function is reserved, refused for now) names tags to order by first,
the rest follow in name order. A member is never ordered by a data value. The keys in `by`
are kept in name order: their order is not identity, and groups are emitted sorted by them.
`member_order` enters the identity and config only when given. Every arrangement of the
input (tag order, data order, table layout, the order of `by`) gives identical values,
record ids, folds, source-info tokens and hashes; only the output's column order follows
the input, as presentation. `TagBy`'s identity is its sorted names, and its new tags are
placed in name order.
- `Data` source-info values may be `str`, `None`, or `list[...]`, nested to any depth; the
Arrow and Python types are derived from the value, with `None` mapping to `large_string`.
That rule has one definition, `arrow_utils.source_info_arrow_type`, and applies only to
Expand Down
8 changes: 4 additions & 4 deletions changelog.d/ITL-1023.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@
plain with data-tag gives a data-tag. `GroupBy` keeps a data-tag key scalar
in both roles. A data-tag non-key loses its tag role, as every non-key tag does
(ITL-1070), but keeps its data column as a list with one element and one source-info
token per member; the members are ordered by the tuple of the input's non-key tags,
the same order for every data list of the group.
token per member; the members are ordered by the input's non-key tags in name order
(ITL-1226), the same order for every data list of the group.
- A source-info column has the shape of its data column. A value merged from several equal
elements (a data-tag `GroupBy` key; a `Join` key that is a data-tag on several inputs) has
one merged token instead of a list of every member's token (ITL-1017): the merging
Expand Down Expand Up @@ -100,8 +100,8 @@
- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as
data-tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each
named column becomes one column in both roles, with its values and source info
unchanged; the new tags follow the stream's tags in the order given, and that order is
part of the operator's identity. System tags, record ids and every data column are
unchanged; the new tags follow the stream's tags in name order, and the order the
names are given in is not part of the operator (ITL-1226). System tags, record ids and every data column are
unchanged. A column a function computed may become a tag and then key a join or a
group. Refused, naming the column: a name that is not a data column of the input, a
name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is
Expand Down
65 changes: 65 additions & 0 deletions changelog.d/ITL-1226.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
### Breaking changes and migration

- `GroupBy` no longer depends on how its input's columns are arranged (ITL-1226). Two
separate effects for pipelines written with 0.3.0:
- **Member order.** The members of a group are now ordered by the non-key tags in name
order, then by record id and source id as before; they were ordered by the non-key
tags in the input's tag order. Affected: a `GroupBy` whose input has two or more
non-key tags that are not in name order (for example `tag_columns=["subject", "unit",
"probe"]` grouped by `subject`: `unit` came before `probe`). For such a `GroupBy` the
order of every list in a group's row changes, and so do the group's folded record id
and source id (`group_v1:...`), the `group_v1` source-info token of a data-tag key, and
the data hash of every grouped row. Everything downstream that reads the lists
recomputes: a function over a grouped row gets a new input and is called again. The
hashes of the `GroupBy` itself and its stored paths do not change. A `GroupBy` with
one non-key tag, or whose non-key tags are already in name order, is not affected,
unless a non-key tag holds NaN (at any list depth): its members are ordered by the NaN
rule under Fixed, which can move them too. To keep the earlier order, name the tags:
`member_order=["unit", "probe"]`. Stored rows of every affected `GroupBy` are not
replaced. An operator node in `LOG` or `REPLAY` (the `GroupBy` and every
operator node after it) keeps its table: `REPLAY` returns the rows stored before the
upgrade, and a `LOG` run adds the new rows beside them, so `REPLAY` then returns both.
A function node after it keeps its pipeline table, so `node.as_source()` lists the
earlier records beside the new ones. Before running such a pipeline with this version,
delete those stored tables, or attach a new database; `clear_cache()` clears memory
only.
- **The order of `by`.** The keys are kept in name order: `GroupBy(by=["b", "a"])` is
now the same operator as `GroupBy(by=["a", "b"])`, and groups are emitted sorted by
the keys in name order (they were emitted in the order of `by`). Affected: a `GroupBy`
with two or more keys not in name order. Its content and pipeline hash change, and so
do its stored operator-cache path, the order of its rows, and the hashes and stored
pipeline-table paths of every node downstream. Values do not change: the lists, folded
ids and data hashes are as before, and a function downstream finds its stored results
(results are keyed by the function and the input data, neither of which changes), so
nothing is computed again; its records are written to a new pipeline table.
For `GroupBy`'s `by` and for `TagBy`, the order of a list or tuple no longer enters
the identity; it did before this change.

### Added

- `GroupBy(..., member_order=...)` and `stream.group_by(..., member_order=...)` (ITL-1226):
the tags that order the members of a group, in order; the other non-key tags follow in
name order. Only tags of the input may be named (plain tags or data-tags, not a key); a
member is never ordered by a data value, and a data column is refused with `tag_by` as
the way to make it a tag. `"name"` is the default order. An ordering function is
planned and refused for now. The option is part of the operator's identity only when
given, so a `GroupBy` without it keeps its hash.

### Changed

- `TagBy` and `tag_by` (ITL-1226): the order of the names is no longer part of the
operator. The identity is the sorted names, the new tags follow the stream's tags in name
order, and the names may be given in any iterable, a set included.

### Fixed

- `GroupBy` orders a member whose ordering tag holds NaN (ITL-1226): NaN sorts after
every number and before null, at any list depth, and two NaN tie (the record id breaks
the tie). Before, a NaN compared with nothing and the members of its group came out in
the input's row order, so their lists and hashes depended on it. A NaN group key is
unchanged: each row with a NaN key is a group of its own.

### Known limitations

- A raw `ArrowTableStream`'s content hash depends on the order of its `tag_columns`
(ITL-1227); its values and pipeline hash do not, and a source's hashes do not.
8 changes: 4 additions & 4 deletions docs/concepts/data-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,9 @@ print(by_phase.keys())
# (('phase',), ('phase', 'unit', 'time'))
```

- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's
tags in the order given, and that order is part of the operator (`TagBy`): the names
are a list, never a set.
- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`, `s.tag_by("y", "x")` and
`s.tag_by({"x", "y"})`. The new tags follow the stream's tags in name order, whatever
order they are named in. The order of the names is not part of the operator (`TagBy`).
- Nothing else changes: no row is added, removed or reordered, the data columns keep
their names, types and values, every data column keeps its own source-info token,
and the system tags keep their names and values.
Expand All @@ -173,7 +173,7 @@ your own classes and logical types (`Path`, `UUID`, arrays) are refused.
|---|---|
| `TagBy` / `tag_by` | Makes the data column `x` a data-tag (above); everything else is unchanged. |
| `Join` | Plain with plain: as before. Plain with data-tag, or data-tag with data-tag: a data-tag and **one** data column `x`. When `x` is a data-tag on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. |
| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tuple of the input's non-key tags (then record ids), the same order for every data list of the group, so all of them stay aligned. |
| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tags `member_order` names, in its order, then by the input's other non-key tags in name order, then by record ids and last by source ids. That order is the same for every data list of the group, so all of them stay aligned. |
| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `DataTagMaterializationWarning`. |
| `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. |
| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` a data-tag, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. |
Expand Down
4 changes: 2 additions & 2 deletions docs/concepts/notebooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,8 @@ means = job.nodes["session_means"]
view = means.run() # the node holds a current run: this computes nothing
print(view.header())
# FunctionJobNode[session_means] — all 8 rows
print(view.tail(3))
# FunctionJobNode[session_means] — last 3 of 8 rows
print(view.sort_by_tags().tail(3))
# FunctionJobNode[session_means] — last 3 of 8 rows, sorted by tags
# * = tag
# shape: (3, 3)
# ┌─────────────┬──────────┬──────────────┐
Expand Down
45 changes: 40 additions & 5 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,14 +305,49 @@ plain tag. A data-tag group key stays a data column too (scalar, in both roles),
following function can read its value (see [Data-tags](data-tags.md)).

Ordering is deterministic by construction, because orcapod hashes the emitted lists to build
cache keys: members are sorted by the input's non-key tag values (with the internal record id
as a final tiebreaker), even though those tags are dropped, and groups are emitted in group-key
order. A reordered input therefore produces a byte-identical output table.
cache keys:

- **Members** are sorted by the input's non-key tags, even though those tags are dropped.
The tags are taken in name order, each ascending with nulls last, and the internal record
id breaks ties. `member_order` names tags to sort by first, in its order; the other
non-key tags follow in name order. A member is never ordered by a data value: a name in
`member_order` must be a tag of the input (a data-tag counts), and a data column is
refused (make it a tag first with `tag_by`).
- **Groups** are emitted sorted by their keys, taken in name order. The order of the names
in `by` is not part of the operator: `GroupBy(by=["a", "b"])` and `GroupBy(by=["b", "a"])`
are one operator.
- **No arrangement of the input changes the result.** The order of its tags, the order of
its data columns and the layout of its table leave every value, record id and hash of the
output unchanged. A reordered input produces a byte-identical output table. The output
lists its columns as the input does (the keys in the input's tag order); that order is
presentation, and no hash depends on it.

```python
units = DictSource(
data=[
{"subject_id": "mouse_01", "unit": 0, "probe": "imec1", "spike_count": 980},
{"subject_id": "mouse_01", "unit": 1, "probe": "imec0", "spike_count": 1200},
{"subject_id": "mouse_01", "unit": 0, "probe": "imec0", "spike_count": 1100},
],
tag_columns=["subject_id", "unit", "probe"], tag_only_columns=True,
)
for _, data in units.group_by(["subject_id"]).iter_data():
print(data.as_dict())
# {'spike_count': [1100, 1200, 980]}
for _, data in units.group_by(["subject_id"], member_order=["unit"]).iter_data():
print(data.as_dict())
# {'spike_count': [1100, 980, 1200]}
```

By default the members are ordered by `probe`, then `unit` (name order), whatever the order
of `tag_columns`. With `member_order=["unit"]`, they are ordered by `unit`, then `probe`. The
option is part of the operator's identity only when it is given, so a `GroupBy` without it
keeps its hash.

A grouped stream can be grouped again on a coarser key; its data lists then nest one level
deeper (`list[list[T]]`). The members of the second grouping are ordered by the first
grouping's keys that are not in the coarser key, then by record id; a tag the first grouping
dropped no longer takes part.
grouping's keys that are not in the coarser key (in name order, or as `member_order` gives),
then by record id; a tag the first grouping dropped no longer takes part.

Every name in `by` must be a **scalar tag column** of the input. `GroupBy` raises
`InputValidationError` for an unknown column, a data column, or a list-valued tag column --
Expand Down
Loading
Loading