Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -1180,7 +1180,7 @@ data. The rules:
key orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- GroupBy never depends on how its input's columns are arranged (developer, November 17,
2026, ITL-1226). Members are ordered by the non-key tags in name order (each ascending,
nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
NaN after every number, nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
only: a data column, a key, a repeat, a set or an empty list is refused; `"name"` is the
default; an ordering function is reserved, refused for now) names tags to order by first,
the rest follow in name order. A member is never ordered by a data value. The keys in `by`
Expand All @@ -1190,6 +1190,22 @@ data. The rules:
record ids, folds, source-info tokens and hashes; only the output's column order follows
the input, as presentation. `TagBy`'s identity is its sorted names, and its new tags are
placed in name order.
- `GroupBy` groups keys as the joins match them, except null (developer, November 17, 2026,
ITL-1234): all rows with a null key form one group, and so do all rows with a NaN key (the
grouping key holds every NaN as one object, `_grouping_key`); `0.0` equals `-0.0`. A NaN key
sorts after every number and before null, as in the member order (`_null_last_key`, which
also sorts a NaN tag after every number). The hashes read a float's bits, so a group emits,
for each float key column, its members' value with the smallest bit pattern at the column's
own width, float16, float32 or float64 (`_emitted_key_row`: `0.0` over `-0.0`, a positive NaN
over a negative one; a signaling NaN has smaller bits than the canonical quiet NaN, so it
wins), taken from the input's Arrow column, never through Python (a float32 signaling NaN
comes back quiet from Python): always a value a member holds, never the first row's, never
chosen by `member_order`. Bits are read only for a group whose key is zero or NaN. A logical
type stored as a float is a float key, read through its storage (`_float_storage`). Every
other key is rebuilt from its Python value, as before: equal Python values that differ in
Arrow (`time64[ns]` within a microsecond) are one group with one rebuilt key; grouping by
the exact value is ITL-1250. The stance on `0.0`/`-0.0` and NaN for the whole library
(match by value, as Polars does; show the smallest bit pattern; rewrite nothing) is ITL-1251.
- Data source-info values may be str, None, or list[...], nested to any depth; Arrow/Python
types are derived from the value, with None mapping to large_string. That rule has one
definition, arrow_utils.source_info_arrow_type, and applies only to tokens supplied as
Expand Down
18 changes: 17 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1226,7 +1226,7 @@ data. The rules:
orders its members by the first grouping's keys outside the coarser key, then by record id. An opt-in mode that keeps non-key tags is ITL-1071.
- GroupBy never depends on how its input's columns are arranged (developer, November 17,
2026, ITL-1226). Members are ordered by the non-key tags in name order (each ascending,
nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
NaN after every number, nulls last), then the record-id and source-id columns; `member_order` (non-key TAG names
only: a data column, a key, a repeat, a set or an empty list is refused; `"name"` is the
default; an ordering function is reserved, refused for now) names tags to order by first,
the rest follow in name order. A member is never ordered by a data value. The keys in `by`
Expand All @@ -1236,6 +1236,22 @@ data. The rules:
record ids, folds, source-info tokens and hashes; only the output's column order follows
the input, as presentation. `TagBy`'s identity is its sorted names, and its new tags are
placed in name order.
- `GroupBy` groups keys as the joins match them, except null (developer, November 17, 2026,
ITL-1234): all rows with a null key form one group, and so do all rows with a NaN key (the
grouping key holds every NaN as one object, `_grouping_key`); `0.0` equals `-0.0`. A NaN key
sorts after every number and before null, as in the member order (`_null_last_key`, which
also sorts a NaN tag after every number). The hashes read a float's bits, so a group emits,
for each float key column, its members' value with the smallest bit pattern at the column's
own width, float16, float32 or float64 (`_emitted_key_row`: `0.0` over `-0.0`, a positive NaN
over a negative one; a signaling NaN has smaller bits than the canonical quiet NaN, so it
wins), taken from the input's Arrow column, never through Python (a float32 signaling NaN
comes back quiet from Python): always a value a member holds, never the first row's, never
chosen by `member_order`. Bits are read only for a group whose key is zero or NaN. A logical
type stored as a float is a float key, read through its storage (`_float_storage`). Every
other key is rebuilt from its Python value, as before: equal Python values that differ in
Arrow (`time64[ns]` within a microsecond) are one group with one rebuilt key; grouping by
the exact value is ITL-1250. The stance on `0.0`/`-0.0` and NaN for the whole library
(match by value, as Polars does; show the smallest bit pattern; rewrite nothing) is ITL-1251.
- `Data` source-info values may be `str`, `None`, or `list[...]`, nested to any depth; the
Arrow and Python types are derived from the value, with `None` mapping to `large_string`.
That rule has one definition, `arrow_utils.source_info_arrow_type`, and applies only to
Expand Down
22 changes: 22 additions & 0 deletions changelog.d/ITL-1234.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Changed

- `GroupBy` puts all rows whose key is NaN into one group (ITL-1234), as the joins match a
NaN key; with several keys, the key tuples compare element by element with that rule.
Before, each such row was a group of its own, and those groups came out in the input's
row order. A NaN key sorts after every number and before null. Only two kinds of input
change: one with a NaN key (its groups, member lists, folded ids and hashes), and one
with a group whose members mix `0.0` and `-0.0` (its key and hashes; see Fixed). Stored
rows of such a `GroupBy` and of the nodes after it are not replaced: delete those tables
or attach a new database before running it with this version (`clear_cache()` clears
memory only).

### Fixed

- `GroupBy` emits a float key whose members differ in bits as the one with the smallest
bit pattern at the column's own width (ITL-1234): `0.0` over `-0.0`, a positive NaN over
a negative one, and a signaling NaN (smaller bits) over the canonical quiet NaN. Before,
`0.0` and `-0.0` were already one group, but the group emitted the key of the first row
in input order, so its value and the row's hashes depended on the row order. The key is
now taken from the input's Arrow column: a `float32` signaling NaN key is emitted with
its bits (it came back quiet). A logical type stored as a float is chosen by the bits of
its storage. A group whose members agree emits their value as before.
15 changes: 13 additions & 2 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,14 +308,25 @@ Ordering is deterministic by construction, because orcapod hashes the emitted li
cache keys:

- **Members** are sorted by the input's non-key tags, even though those tags are dropped.
The tags are taken in name order, each ascending with nulls last, and the internal record
id breaks ties. `member_order` names tags to sort by first, in its order; the other
The tags are taken in name order, each ascending with NaN after every number and nulls
last, and the internal record id breaks ties. `member_order` names tags to sort by first, in its order; the other
non-key tags follow in name order. A member is never ordered by a data value: a name in
`member_order` must be a tag of the input (a data-tag counts), and a data column is
refused (make it a tag first with `tag_by`).
- **Groups** are emitted sorted by their keys, taken in name order. The order of the names
in `by` is not part of the operator: `GroupBy(by=["a", "b"])` and `GroupBy(by=["b", "a"])`
are one operator.
- **Keys** group as the joins match them, except for null: rows with equal keys form one
group, all rows with a null key form one group, and so do all rows with a NaN key (NaN
equals NaN here; `0.0` equals `-0.0`). A NaN key sorts after every number and before null.
When the members of a group hold a float key with different bits (`0.0` and `-0.0`, or
NaN with another sign or payload), the group emits the one with the smallest bit pattern
at the column's own width (`float16`, `float32` or `float64`; a logical type stored as a
float is read through its storage), for each key column: `0.0` over `-0.0`, and a
positive NaN over a negative one. A signaling NaN has smaller bits than the canonical
quiet NaN, so when a member holds one, it is the key emitted. The emitted key is always a
value a member holds, bit for bit, and neither the order of the rows nor `member_order`
decides it.
- **No arrangement of the input changes the result.** The order of its tags, the order of
its data columns and the layout of its table leave every value, record id and hash of the
output unchanged. A reordered input produces a byte-identical output table. The output
Expand Down
4 changes: 2 additions & 2 deletions src/orcapod/contexts/data/v1.json
Original file line number Diff line number Diff line change
Expand Up @@ -254,10 +254,10 @@
"date": "2026-11-17",
"release": null,
"added": {
"pipeline-02.json": "1ec3cbbe09bbb353366d1c8f2573881467d1c8e1ae53a20f8df4e7f43bbfa74e"
"pipeline-02.json": "7778b4c69ff706debbcbd9fc67dbe5b723df56252158be8253f8751326a5bd41"
},
"previous": "6cdd29cdc6fa98df322931e2ab0ff9df151fdb47d3381adecd9f58fd55d2fb22",
"digest": "fa4c504d4eb289138f85029217770c25c45ebea9e7dd8855c42c83b9c515d37c"
"digest": "838e8b0023619f2192415496893e3c4b35981980e388c6a617202d9cb1991d40"
}
]
},
Expand Down
151 changes: 144 additions & 7 deletions src/orcapod/core/operators/group_by.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

import math
from collections.abc import Collection, Sequence, Set
from typing import Any, ClassVar, TYPE_CHECKING, get_origin
from typing import Any, ClassVar, TYPE_CHECKING, cast, get_origin

from orcapod.core.operators.base import UnaryOperator
from orcapod.core.data_tags import stream_data_tags
Expand Down Expand Up @@ -50,6 +50,95 @@ def _null_last_key(value: Any) -> tuple[Any, ...]:
return (0, value)


# The grouping form of a NaN key value: one shared object, so every row whose key
# is NaN falls into one group (ITL-1234), as the joins match a NaN key
# (``match_keys``). ``0.0`` and ``-0.0`` already fall into one group: they are equal.
_NAN_KEY = object()


def _grouping_key(key: tuple[Any, ...]) -> tuple[Any, ...]:
"""Return a group key in the form that groups rows: every NaN as one value.

Args:
key: The values of the ``by`` columns of one row, in the order of ``by``.

Returns:
The key with each NaN replaced by one shared object.
"""
return tuple(
_NAN_KEY if isinstance(value, float) and math.isnan(value) else value
for value in key
)


def _float_storage(arrow_type: "pa.DataType") -> "pa.DataType | None":
"""Return the float type a column is stored as, or ``None`` when it is not a float.

A logical (extension) type over float storage, such as a temperature stored as
``float64``, is stored as that float.

Args:
arrow_type: The Arrow type of a key column.

Returns:
The ``float16``, ``float32`` or ``float64`` type of the column's storage, or
``None``.
"""
storage = (
arrow_type.storage_type
if isinstance(arrow_type, pa.ExtensionType)
else arrow_type
)
return storage if pa.types.is_floating(storage) else None


def _emitted_key_row(column: "pa.ChunkedArray[Any]", indices: Sequence[int]) -> int:
"""Return the input row whose value of one key column a group emits.

The members of a group hold equal key values, but equal floats may differ in
their bits: ``0.0`` and ``-0.0``, and NaN with another sign or payload (equal
floats of any other value have the same bits). The hashes of the emitted row
read the bits, so for a zero or NaN key the group emits the member value with
the smallest bit pattern at the column's own width (ITL-1234), chosen per key
column. That is ``0.0`` over ``-0.0``, a positive NaN over a negative one, and
the NaN with the smallest payload; a signaling NaN has smaller bits than the
canonical quiet NaN, so it is the one emitted when a member holds it. A group
whose members agree emits their value. Neither the row order nor
``member_order`` decides it, and the caller takes the value from the Arrow
column, so its bits are kept exactly.

Args:
column: A key column of the input table.
indices: The rows of one group, at least one, all with equal keys.

Returns:
The row, one of ``indices``, whose value the group emits for this column.
"""
first = indices[0]
storage_type = _float_storage(column.type)
if len(indices) == 1 or storage_type is None:
return first
# Only a zero or NaN key can differ in bits among equal values, so the first
# member's value decides whether the group is gathered at all.
scalar = column[first]
if isinstance(scalar, pa.ExtensionScalar):
# A null extension scalar has no storage scalar.
storage = scalar.value
value = None if storage is None else cast(float | None, storage.as_py())
else:
value = cast(float | None, scalar.as_py())
if value is None or (value != 0.0 and not math.isnan(value)):
return first
members = column.take(pa.array(indices, pa.int64())).combine_chunks()
if isinstance(members, pa.ExtensionArray):
members = members.storage
# The unsigned integer of the float's width, to read its bits; no value of the
# group is null, since a null key equals no float.
unsigned = {16: pa.uint16, 32: pa.uint32, 64: pa.uint64}[storage_type.bit_width]()
bits = cast(list[int], members.view(unsigned).to_pylist())
return indices[min(range(len(indices)), key=bits.__getitem__)]


GROUP_BY_FOLD_LABEL = "group_v1"
"""Label of the folds of ``GroupBy``: the folded source id and record id are
``"group_v1:<hex>"`` (ITL-1154), and so is the one source-info token of a data-tag
Expand Down Expand Up @@ -177,6 +266,16 @@ class GroupBy(UnaryOperator):
reordered input or a reordered ``by`` must
produce an identical output.

Rows with equal keys form one group, as the joins match keys, except that all
rows with a null key form one group too. So do all rows with a NaN key, and
``0.0`` equals ``-0.0`` (ITL-1234). A NaN key sorts after every number and
before null. A float key that the members hold with different bits is
emitted, per key column, as the one with the smallest bit pattern at the
column's own width (``0.0`` over ``-0.0``, a positive NaN over a negative
one, a signaling NaN over the canonical quiet NaN), taken from the input's
Arrow column, so the emitted key is a value a member holds, bit for bit, and
never depends on the row order or on ``member_order``.

Tag tuples are expected to be unique within a stream, but nothing enforces
that. If the input does contain duplicate tag tuples, the members sharing
a tuple are separated only by ``record_id``, which is fixed when the source
Expand Down Expand Up @@ -394,16 +493,36 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
sorted(c for c in system_tag_columns if c.startswith(prefix))
)

groups: dict[tuple[Any, ...], list[dict[str, Any]]] = {}
for row in table.to_pylist():
groups.setdefault(tuple(row[c] for c in self.by), []).append(row)
# Every NaN key is one value here (ITL-1234), so the rows whose key is NaN
# form one group, as the joins match them.
rows = table.to_pylist()
groups: dict[tuple[Any, ...], list[int]] = {}
for index, row in enumerate(rows):
key = _grouping_key(tuple(row[c] for c in self.by))
groups.setdefault(key, []).append(index)
# For each group, the row whose value each key column emits.
keyed_groups: list[tuple[tuple[int, ...], list[dict[str, Any]]]] = [
(
tuple(_emitted_key_row(table.column(c), indices) for c in self.by),
[rows[i] for i in indices],
)
for indices in groups.values()
]

grouped_rows: list[dict[str, Any]] = []
# Emit groups in key order rather than first-appearance order, so a
# reordered input produces a byte-identical output table.
for key, members in sorted(
groups.items(), key=lambda kv: tuple(_null_last_key(v) for v in kv[0])
# reordered input produces a byte-identical output table. The keys are in
# name order (``by`` is sorted); a NaN key sorts after every number and
# before null, as a member's tag does.
key_rows: list[tuple[int, ...]] = []
for group_key_rows, members in sorted(
keyed_groups,
key=lambda km: tuple(
_null_last_key(rows[r][c]) for r, c in zip(km[0], self.by)
),
):
key_rows.append(group_key_rows)
key = tuple(rows[r][c] for r, c in zip(group_key_rows, self.by))
if sort_columns:
members.sort(
key=lambda r: tuple(_null_last_key(r[c]) for c in sort_columns)
Expand Down Expand Up @@ -437,6 +556,24 @@ def unary_static_process(self, stream: StreamProtocol) -> StreamProtocol:
stream.data_context.type_converter,
)

# A float key column (a float, or a logical type stored as one) holds the
# chosen member's value as the input's Arrow column has it, never through
# Python: a conversion can change a float's bits (a float32 signaling NaN
# comes back quiet). Every other key keeps the value rebuilt from Python,
# as before: its members may differ in Arrow although equal in Python (two
# time64[ns] values within one microsecond read as one ``datetime.time``),
# and taking one member's would make the key depend on the row order.
# Grouping such keys by their exact value is ITL-1250.
for position, c in enumerate(self.by):
if _float_storage(table.column(c).type) is None:
continue
chosen = pa.array([r[position] for r in key_rows], pa.int64())
grouped_table = grouped_table.set_column(
grouped_table.schema.get_field_index(c),
grouped_table.schema.field(c),
table.column(c).take(chosen),
)

n_char = self.orcapod_config.hashing.system_tag_n_char
grouped_table = arrow_utils.append_to_system_tags(
grouped_table, stream.pipeline_hash().to_hex(n_char)
Expand Down
Loading
Loading