From 44cbd4c817c3250114608c0bbd1a57d31f012127 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 23:28:17 -0800 Subject: [PATCH 01/16] docs(plan): ITL-1053 head sorting plan --- .../plans/2026-11-13-itl-1053-head-sort.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 superpowers/plans/2026-11-13-itl-1053-head-sort.md diff --git a/superpowers/plans/2026-11-13-itl-1053-head-sort.md b/superpowers/plans/2026-11-13-itl-1053-head-sort.md new file mode 100644 index 000000000..08d27953b --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1053-head-sort.md @@ -0,0 +1,19 @@ +# ITL-1053: head() honours sort_by_tags like as_table + +Decision (developer, 2026-11-12): head takes as_table's column configuration, sorting included, +applies it to the rows it has, then takes the first n. Default config stays unsorted. + +## Findings (before coding) +- `head_column_config` forced sort off / raised. Replace by `ColumnConfig.handle_config` (no wrapper). +- `FunctionJobNode._head_of_pairs` and `FunctionPodStream.head` sliced BEFORE `_shape_table`; `_shape_table` + attaches the content hash before sorting (ITL-1029), so for a sorted config: shape all rows, then slice n. +- `ArrowTableStream.head` assembles `limit=n` then sorts: for sorted config assemble all, sort, slice n. +- OperatorJobNode / DerivedSource / SourceProxy / CachedSource / StaticOutputPod delegate to ArrowTableStream.head. +- SpiralDB `_head` reads <= n rows (bounded), `_whole_table` already sorts via as_table: sorts rows read only. +- No store read changes; content hashes for all rows computed only when sort + content_hash asked. + +## Steps +1. Failing tests (head vs as_table.slice for each ColumnConfig flag; reversed async completion, in-memory + Delta, + function and operator nodes; bounded source; default unsorted; sort no longer raises). +2. Implement; docs, docstrings; design_issues entry, changelog.d/ITL-1053.md. +3. Full suite 3.11/3.12/3.13, ruff, basedpyright; PR; CodeRabbit. From 66859c0e606f1c7a1ef40258d78c9936f051116a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 23:34:27 -0800 Subject: [PATCH 02/16] feat(streams)!: head honours sort_by_tags and all_info like as_table (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- changelog.d/ITL-1053.md | 16 ++ design_issues/ITL-1053-PV2.md | 24 ++ docs/concepts/sources.md | 4 +- docs/concepts/streams.md | 27 ++ src/orcapod/core/function_pod.py | 27 +- src/orcapod/core/nodes/function_node.py | 18 +- src/orcapod/core/nodes/operator_node.py | 5 +- src/orcapod/core/nodes/source_node.py | 5 +- .../core/operators/static_output_pod.py | 3 +- src/orcapod/core/sources/derived_source.py | 27 +- src/orcapod/core/sources/polling_source.py | 12 +- src/orcapod/core/sources/source_proxy.py | 5 +- .../core/sources/spiraldb_table_source.py | 4 +- .../core/streams/arrow_table_stream.py | 16 +- src/orcapod/core/streams/base.py | 49 ++-- src/orcapod/side_effects.py | 8 +- .../test_spiraldb_table_source_streaming.py | 14 ++ .../streams/test_head_and_preview.py | 16 +- tests/test_core/streams/test_head_sorting.py | 232 ++++++++++++++++++ tests/test_pipeline/test_node_head_sorting.py | 183 ++++++++++++++ 20 files changed, 607 insertions(+), 88 deletions(-) create mode 100644 changelog.d/ITL-1053.md create mode 100644 design_issues/ITL-1053-PV2.md create mode 100644 tests/test_core/streams/test_head_sorting.py create mode 100644 tests/test_pipeline/test_node_head_sorting.py diff --git a/changelog.d/ITL-1053.md b/changelog.d/ITL-1053.md new file mode 100644 index 000000000..1592700c1 --- /dev/null +++ b/changelog.d/ITL-1053.md @@ -0,0 +1,16 @@ +### Changed + +- `head()` takes the column configuration of `as_table`, sorting included, and applies it to + the rows it has before taking the first `n` (ITL-1053). `head(n, columns={"sort_by_tags": + True})` no longer raises, and `head(n, all_info=True)` now sorts by tag, as `as_table` + does; the default `head()` (and the notebook preview) stays unsorted. A stream that has all + its rows (a job node's stored rows, a computed stream, an in-memory source) gives exactly + `as_table(...).slice(0, n)` for every configuration, whatever order its rows were computed + in. A stream that has read or computed only part of its rows (a `SpiralDBTableSource` + head, a partly computed stream) sorts the rows it has: the rows read, in tag order. + +### Fixed + +- A node's `head(n, all_info=True)` no longer differs from `as_table(all_info=True)`'s first + rows when an async run completed rows out of tag order (the cause of an occasionally + failing test, ITL-1001). diff --git a/design_issues/ITL-1053-PV2.md b/design_issues/ITL-1053-PV2.md new file mode 100644 index 000000000..2433cd3c2 --- /dev/null +++ b/design_issues/ITL-1053-PV2.md @@ -0,0 +1,24 @@ +## `src/orcapod/core/streams/` — notebook preview and `head()` + +### PV2 — `head()` refused `sort_by_tags` and showed rows in completion order +**Status:** resolved +**Severity:** medium +**Issue:** ITL-1053 + +`head()` (ITL-991) was built as "the first rows in stream order": `head_column_config` switched +tag sorting off even under `all_info=True` and raised `ValueError` when `sort_by_tags` was +asked. After an async run a node's stored order is completion order, so its head showed rows +in an order that varied from run to run, and a test asserting that `head(5, all_info=True)` +equals the prefix of `as_table(all_info=True)` failed whenever rows finished out of tag order +(`test_head_keeps_the_tag_arrow_type_of_the_input[async-delta]`, seen on CI; ITL-1001). + +**Fix:** developer decision, November 12, 2026: "make it consistent with as_table". +`head_column_config` is removed; `head` takes `as_table`'s column configuration, sorting +included, and applies it to the rows it has, then takes the first `n`. The default stays +unsorted. A node, an `ArrowTableStream`, a computed `FunctionPodStream` and a `DerivedSource` +over a job node shape and sort every row they have (each row keeps its own content hash, +ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded read +(`SpiralDBTableSource`, a partly computed stream, a polled source's batches) sorts the rows it +has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads +more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the +rows read, sorted" unless the store returns rows in tag order. diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index dc7075718..ef5e50baa 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -154,7 +154,9 @@ with `max_rows=None`, but it cannot yet be run through a job on a laptop. **Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell (its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10 rows, in one scan to which the row cap does not apply, so a table of any size can be looked -at; `head(n)` reads `n`. The header line says what is shown, for +at; `head(n)` reads `n`. With `sort_by_tags=True` (or `all_info=True`) the head sorts only +the rows it read: the rows read, in tag order, not the first `n` of the table in tag order. +The header line says what is shown, for example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's recorded row count, read without scanning), `all 4 rows` when the source reads no more than that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index ebcee55ad..a7da9d4af 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -162,6 +162,33 @@ What each kind of object shows: | A pipeline blueprint node, an unbound input slot | none (`unavailable`) | | A side-effect stream | none: showing it never delivers its side effects | +`head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`, +sorting included, and applies it to the rows it has before taking the first `n`. The default +does not sort (so the notebook preview, which is the default `head()`, shows rows in the +order the stream holds them, as `as_table()` does). With `sort_by_tags=True`, or +`all_info=True`, rows are sorted by tag first: + +```python +from orcapod.sources import DictSource + +unordered = DictSource( + data=[{"subject_id": f"mouse_{i:02d}", "age": 8 + i} for i in (3, 1, 2)], + tag_columns=["subject_id"], + label="unordered", +) +print(unordered.head(2).as_table().column("subject_id").to_pylist()) +# ['mouse_03', 'mouse_01'] +print(unordered.head(2, all_info=True).as_table().column("subject_id").to_pylist()) +# ['mouse_01', 'mouse_02'] +``` + +A stream that has all of its rows (a job node's stored rows, a computed stream, an in-memory +source) then gives exactly the first `n` rows of `as_table` with the same configuration, +whatever order the rows were computed in. A stream that has only some of its rows sorts +those: a source that reads a bounded number of rows (such as a `SpiralDBTableSource`) shows +the rows it read, in tag order, and a stream computed only in part shows the rows computed so +far in tag order. It is not "the first `n` rows in tag order" of rows it never read. + A head of an operator is never the operator applied to the heads of its inputs: a group over the first ten rows has the wrong members. So an uncomputed stream shows its columns and "not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 07fbb3849..015e00098 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -16,7 +16,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, schema_requires_io, ) from orcapod.core.streams.head import ( @@ -1215,11 +1214,17 @@ def head( the first of the result. After an iteration that stopped part way they are the rows computed so far (``PARTIAL``). Before any computation, or once an input changed, there are none. A ``CachedFunctionPod``'s result store is not looked - up: that needs the input rows' hashes, i.e. reading the input. See - ``StreamBase.head``. + up: that needs the input rows' hashes, i.e. reading the input. With + ``sort_by_tags`` (or ``all_info``) the rows held are sorted by tag and the first + ``n`` of those returned: after a complete computation that is + ``as_table(...).slice(0, n)``; after a partial one, the rows computed so far in + tag order ("the rows it has", not the first ``n`` of the whole result in tag + order). See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) + # Sorting needs every row held, not only the first n: read them all (in memory). + wanted = len(self._cached_output_datas) if config.sort_by_tags else n has_results = bool(self._cached_output_datas) or ( self._cached_output_table is not None ) @@ -1247,7 +1252,7 @@ def head( # A complete computation: its table holds the rows; only the shown rows' # data are read, and only when their content hashes are asked for. def shown_content_hashes() -> pa.Array[Any]: - shown, _ = self._computed_prefix(n) + shown, _ = self._computed_prefix(wanted) return pa.array( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), @@ -1256,8 +1261,12 @@ def shown_content_hashes() -> pa.Array[Any]: return StreamHead.of_rows( self, self._shape_table( - self._cached_output_table.slice(0, n), config, shown_content_hashes - ), + self._cached_output_table + if config.sort_by_tags + else self._cached_output_table.slice(0, n), + config, + shown_content_hashes, + ).slice(0, n), tag_columns=tag_columns, total_rows=self._cached_output_table.num_rows, origin="computed", @@ -1265,7 +1274,7 @@ def shown_content_hashes() -> pa.Array[Any]: finished = not self._needs_iterator and self._cached_input_iterator is None if not finished and not self._cached_output_datas: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) - shown, total = self._computed_prefix(n) + shown, total = self._computed_prefix(wanted) status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL if not shown: # Nothing to show (an empty result, inputs that all gave no output so far, @@ -1292,7 +1301,7 @@ def shown_content_hashes() -> pa.Array[Any]: [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), ), - ), + ).slice(0, n), tag_columns=tag_columns, total_rows=total, status=status, diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 5d3660d5e..328f29988 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -33,7 +33,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( BLUEPRINT_NOTE, @@ -629,7 +628,7 @@ def head( """Return no rows: a node of this kind holds no data. See ``StreamBase.head``.""" check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note=BLUEPRINT_NOTE, ) @@ -651,7 +650,11 @@ def _head_of_pairs( (t, d) for t, d in pairs if d is not None and not isinstance(d, EmptyData) ] missing = sum(1 for _, d in pairs if isinstance(d, EmptyData)) - shown = rows[:n] + # With sorting, every row is shaped and sorted (each keeps its own content + # hash), then the first n are taken: the head is ``as_table(config)[:n]``. + # At least one row is shaped even for n == 0, so that the empty head has the + # columns of the rows' own table (the context column among them). + shown = rows if config.sort_by_tags else rows[: max(n, 1)] table = self._shape_table( self._table_from_pairs(shown), config, @@ -659,7 +662,7 @@ def _head_of_pairs( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), ), - ) + ).slice(0, n) note = None if missing: note = ( @@ -3017,7 +3020,10 @@ def head( After a run (or ``execute``) the rows are the node's results in memory, the first of what ``as_table`` returns. Otherwise they are the first of its stored rows: the node's whole stored slice is read and nothing is kept on the node. - With nothing stored, there are no rows. See ``StreamBase.head``. + With nothing stored, there are no rows. The node has all of its rows, so with + ``sort_by_tags`` (or ``all_info``) the head is exactly + ``as_table(...).slice(0, n)``, whatever order the rows were computed in. See + ``StreamBase.head``. Raises: RuntimeError: If the node was loaded and the attached store does not hold @@ -3026,7 +3032,7 @@ def head( from orcapod.pipeline.serialization import LoadStatus check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) status = self.load_status if status == LoadStatus.UNAVAILABLE: return self._head_without_rows( diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 9f22703ec..41b22c998 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -30,7 +30,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( STALE_NOTE, @@ -402,7 +401,7 @@ def head( """Return no rows: a blueprint node holds no data.""" check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note=BLUEPRINT_NOTE, ) @@ -1332,7 +1331,7 @@ def head( ``as_table`` does. A loaded node never raises here. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) if self._cached_output_stream is not None: if not self._loaded and self.is_stale: # run() would recompute; the kept rows are not the current result. diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index eaa888091..a4416831f 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -25,7 +25,6 @@ ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.errors import SourceSpecMismatchError, UnboundSourceError from orcapod.protocols.core_protocols import DataProtocol, TagProtocol @@ -316,7 +315,7 @@ def head( """ check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note="input slot; bind a source", ) @@ -637,7 +636,7 @@ def head( if self._bound_source is None: check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note="no source bound", ) diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 34ffa81db..081209815 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -15,7 +15,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( STALE_NOTE, @@ -536,7 +535,7 @@ def head( or after an input changed, there are no rows. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) had_result = self._cached_stream is not None self._update_cache_status() if self._cached_stream is not None and self._cached_time is not None: diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 26c25f02a..7c2dd81e3 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -14,7 +14,6 @@ ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, schema_requires_io, ) from orcapod.errors import InputValidationError @@ -477,10 +476,13 @@ def head( The records are the ones this source already read. Otherwise, an origin source gives its own ``head(n)`` (so a polling or caching origin is never polled or written), and an origin job node gives its stored rows, read now and not kept. - An origin job node that has not run gives no rows. + An origin job node that has not run gives no rows. With ``sort_by_tags`` (or + ``all_info``), the rows are sorted by tag as ``as_table`` sorts them: all the + stored rows of an origin job node, but only the rows an origin source reads for + its head (see ``StreamBase.head``). """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) origin_kind = "held" if isinstance(self._origin, RootSource) else "stored" if self._tags_as_data is not None and self._cached_stream is not None: return self._cached_stream.head(n, columns=config).for_stream( @@ -493,6 +495,18 @@ def head( return stream.head(n, columns=config).for_stream(self, origin=origin_kind) if isinstance(self._origin, RootSource): origin_head = self._origin.head(n) + if ( + config.sort_by_tags + and origin_head.status is HeadStatus.ROWS + and not origin_head.total_is_estimate + and origin_head.total_rows is not None + and origin_head.total_rows > n + ): + # The origin has all its rows (its total is exact): sorting needs them + # all, and the system tags this source adds follow the origin's row + # order, so the origin is not sorted first. A bounded origin (an + # estimated total) is sorted among the rows it read. + origin_head = self._origin.head(origin_head.total_rows) if origin_head.status is not HeadStatus.ROWS: if origin_head.table.num_columns == 0: # The origin's columns are not known without polling it. @@ -530,8 +544,11 @@ def head( tag_columns=stream.keys(columns=config)[0], note="nothing stored: its origin has not run", ) - # Only the shown rows are enriched; the total is the whole stored count. - head = self._stream_over(records.slice(0, n)).head(n, columns=config) + # Without sorting only the shown rows are enriched. With sorting every stored + # row is (they are all read already), so the head is as_table cut to n rows. + # The total is the whole stored count. + rows = records if config.sort_by_tags else records.slice(0, n) + head = self._stream_over(rows).head(n, columns=config) return dataclasses.replace( head.for_stream(self, origin="stored"), total_rows=records.num_rows ) diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index b45b0a335..98fe4d80c 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -26,7 +26,6 @@ ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.errors import ( CursorInvalidatedError, @@ -472,13 +471,18 @@ def head( or none when the source declared no schema (finding it would mean polling). """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) batches = list(self._batches) # snapshot: the list is append-only if batches: - # Combine only the batches that hold the first n rows; count the rest. + # Combine only the batches that hold the first n rows (every batch when + # sorting); count the rest. sizes = [batch.head(0).total_rows or 0 for batch in batches] needed, covered = 0, 0 - while needed < len(batches) and (covered < n or needed == 0): + # A sorted head orders every polled batch: the first n in tag order can lie + # in any of them. The batches are already in memory; nothing is polled. + while needed < len(batches) and ( + covered < n or needed == 0 or config.sort_by_tags + ): covered += sizes[needed] needed += 1 prefix = batches[0] diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index 6714072c9..803fe20a8 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -18,9 +18,9 @@ StreamHead, head_of, ) +from orcapod.types import ColumnConfig from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.protocols.core_protocols import SourceProtocol from orcapod.types import ContentHash, Schema @@ -31,7 +31,6 @@ import pyarrow as pa from orcapod.protocols.core_protocols import DataProtocol, TagProtocol - from orcapod.types import ColumnConfig class SourceProxy(RootSource): @@ -224,7 +223,7 @@ def head( self._delegate.head(n, columns=columns, all_info=all_info), self ) check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.UNAVAILABLE, diff --git a/src/orcapod/core/sources/spiraldb_table_source.py b/src/orcapod/core/sources/spiraldb_table_source.py index eaf1abf8d..3ea5db8c0 100644 --- a/src/orcapod/core/sources/spiraldb_table_source.py +++ b/src/orcapod/core/sources/spiraldb_table_source.py @@ -44,7 +44,7 @@ from orcapod.core.sources.derived_source import validate_tags_as_data from orcapod.core.sources.stream_builder import SourceStreamBuilder from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import check_head_size, head_column_config +from orcapod.core.streams.base import check_head_size from orcapod.core.streams.head import HeadStatus, StreamHead from orcapod.databases.spiraldb_connector import SpiralDBConnector from orcapod.errors import InputValidationError, SourceTooLargeError @@ -693,7 +693,7 @@ def head( preview is this head. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) head = self._head(n, columns=config) tag_columns = self.keys(columns=config)[0] if not head.rows_read: diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 721f59e36..e16a2debf 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -10,7 +10,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import PodProtocol, StreamProtocol, TagProtocol @@ -241,14 +240,21 @@ def head( ) -> StreamHead: """Return the first ``n`` rows of the table this stream holds. - Only those rows are assembled: content hashes, when asked for, are computed for - them alone. See ``StreamBase.head``. + Without sorting only those rows are assembled: content hashes, when asked for, + are computed for them alone. With ``sort_by_tags`` (or ``all_info``) the rows are + sorted first, as ``as_table`` sorts them, and the first ``n`` of those are + returned, so the head is always ``as_table(...)`` cut to ``n`` rows. See + ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) + if config.sort_by_tags: + table = self._assemble_table(config, limit=None).slice(0, n) + else: + table = self._assemble_table(config, limit=n) return StreamHead.of_rows( self, - self._assemble_table(config, limit=n), + table, tag_columns=self.keys(columns=config)[0], total_rows=self._table.num_rows, ) diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index bb036a19c..c5d0cabfe 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -1,6 +1,5 @@ from __future__ import annotations -import dataclasses import logging from abc import abstractmethod from collections.abc import AsyncIterator, Collection, Iterator, Mapping @@ -68,32 +67,6 @@ def check_head_size(n: int) -> None: raise ValueError(f"head() takes a non-negative number of rows n, got {n}") -def head_column_config( - columns: ColumnConfig | dict[str, Any] | None, all_info: bool -) -> ColumnConfig: - """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. - - The column groups mean what they mean for ``as_table``. ``head`` returns rows in - stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. - - Raises: - ValueError: If ``columns`` asks for ``sort_by_tags``. - """ - if isinstance(columns, ColumnConfig): - asked_to_sort = bool(columns.sort_by_tags) - elif isinstance(columns, Mapping): - asked_to_sort = bool(columns.get("sort_by_tags", False)) - else: - asked_to_sort = False - if asked_to_sort: - raise ValueError( - "head() returns the first rows in stream order and cannot sort them by " - "tags; sort the result of as_table() instead" - ) - config = ColumnConfig.handle_config(columns, all_info=all_info) - return dataclasses.replace(config, sort_by_tags=False) - - class StreamBase(TraceableBase): @property @abstractmethod @@ -514,8 +487,16 @@ def head( The rows are ones the stream already holds, has cached from an earlier computation, has stored, or can read with a bounded read of its table. ``head`` never calls a function, runs an operator, delivers a side effect, polls a source - or writes anything. When rows are shown, they are the first ``n`` rows that - ``as_table()`` would return. The notebook preview of a stream is its ``head()``. + or writes anything. ``head`` takes the column configuration of ``as_table``, + sorting included, and applies it to the rows the stream has, then takes the first + ``n``. The default configuration does not sort, as ``as_table()`` does not, so + the rows are the first ``n`` that ``as_table()`` would return. With + ``sort_by_tags`` (or ``all_info``) a stream that has all of its rows (a node's + stored slice, a computed stream, an in-memory source) returns the first ``n`` + of ``as_table`` with that configuration. A stream that has read or computed only + part of its rows (a source's bounded read, a partly computed stream) sorts those + rows: the rows read, in tag order, not the first ``n`` in tag order. The + notebook preview of a stream is its default (unsorted) ``head()``. This default shows no rows: a stream class that can serve rows without computing overrides it. The zero-row table then has the stream's columns, typed from its @@ -524,9 +505,9 @@ def head( Args: n: The largest number of rows to return. - columns: Column groups to include, as for ``as_table``. ``sort_by_tags`` is - not accepted: the rows come in stream order. - all_info: Include every column group, as for ``as_table`` (without sorting). + columns: Column groups to include and whether to sort by tags, exactly as + for ``as_table``. + all_info: Include every column group and sort by tags, as for ``as_table``. Returns: A ``StreamHead`` with the rows and a ``status`` that tells an empty result @@ -534,10 +515,10 @@ def head( Raises: TypeError: If ``n`` is not an ``int``. - ValueError: If ``n`` is negative or ``columns`` asks for ``sort_by_tags``. + ValueError: If ``n`` is negative. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) def _head_without_rows( diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 2388dfa67..452ca3431 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -20,7 +20,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( SIDE_EFFECTS_NOTE, @@ -48,6 +47,7 @@ OLD_TRACKING_DB_SCHEMA_VERSIONS, TRACKING_DB_SCHEMA_VERSION, ) +from orcapod.types import ColumnConfig from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -62,7 +62,7 @@ ) from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.protocols.observability_protocols import ExecutionObserverProtocol - from orcapod.types import ColumnConfig, ContentHash, Schema + from orcapod.types import ContentHash, Schema else: pa = LazyModule("pyarrow") @@ -166,7 +166,7 @@ def head( ) -> StreamHead: """Return no rows: showing this stream must never deliver its side effects.""" check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE ) @@ -762,7 +762,7 @@ def head( ) -> StreamHead: """Return no rows: showing this stream must never deliver its side effects.""" check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE ) diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 384ead58e..38640d111 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -629,6 +629,20 @@ def test_head_reads_one_bounded_scan(self, big): assert (head.total_rows, head.total_is_estimate) == (25, True) assert len(head) == 10 + def test_a_sorted_head_sorts_only_the_rows_it_read(self, big): + """``sort_by_tags`` sorts the one bounded read; it reads no more rows (ITL-1053).""" + source = SpiralDBTableSource(_PROJECT, "big", label="big") + tags = ["subject_id", "timestamp"] + + unsorted_head = source.head() + sorted_head = source.head(columns={"sort_by_tags": True}) + + assert self._scan_limits() == [10, 10] + assert len(sorted_head) == 10 + assert sorted_head.table.equals( + unsorted_head.table.sort_by([(name, "ascending") for name in tags]) + ) + def test_an_unlabelled_source_is_shown_by_its_table_name(self, big): """With no label, the header names the table; the label itself is unchanged.""" source = SpiralDBTableSource(_PROJECT, "big") diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 2dac4d642..3e082a04f 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -86,23 +86,25 @@ def test_head_of_a_short_source_says_all_rows() -> None: def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: - """With every column group, head is the prefix of as_table, in stream order.""" + """With every column group, head is the prefix of as_table (ITL-1053).""" source = _source() head = source.head(4, columns=ALL_COLUMNS) assert head.table.equals(source.as_table(columns=ALL_COLUMNS).slice(0, 4)) - # all_info means every column group; head returns rows in stream order. + # all_info sorts by tag, as it does for as_table. assert source.head(4, all_info=True).table.equals( - source.as_table(columns=ALL_COLUMNS).slice(0, 4) + source.as_table(all_info=True).slice(0, 4) ) -def test_head_refuses_negative_n_and_sorting() -> None: - """A negative n and sort_by_tags are refused.""" +def test_head_refuses_a_negative_n_and_accepts_sorting() -> None: + """A negative n is refused; sort_by_tags is a column option, as for as_table.""" source = _source() with pytest.raises(ValueError, match="n"): source.head(-1) - with pytest.raises(ValueError, match="sort"): - source.head(columns={"sort_by_tags": True}) + head = source.head(3, columns={"sort_by_tags": True}) + assert head.table.equals( + source.as_table(columns={"sort_by_tags": True}).slice(0, 3) + ) def test_head_of_zero_rows_keeps_the_columns() -> None: diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py new file mode 100644 index 000000000..324e7372d --- /dev/null +++ b/tests/test_core/streams/test_head_sorting.py @@ -0,0 +1,232 @@ +"""A stream's ``head()`` sorts the way ``as_table`` does, among the rows it has (ITL-1053). + +``head(n, cfg)`` takes the column configuration of ``as_table``, sorting included: with +``sort_by_tags`` (or ``all_info``) the rows the stream has are sorted by tag and the first +``n`` are returned, so ``head(n, cfg) == as_table(cfg).slice(0, n)``. The default +configuration does not sort, as ``as_table()`` does not. A stream that reads or holds only +part of its rows sorts those rows. Sorting never computes or reads anything more. +""" + +from __future__ import annotations + +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream +from orcapod.core.operators import PolarsFilter, SelectDataColumns +from orcapod.core.sources import ArrowTableSource, DerivedSource, DictSource +from orcapod.core.sources.polling_source import PollingSource +from orcapod.streams import HeadStatus +from orcapod.types import ColumnConfig, Cursor, Schema + +ROWS = 8 +# Not in tag order, with the tag order's first rows last. +ORDER = [5, 2, 7, 0, 6, 1, 4, 3] + +CONFIGS: dict[str, dict[str, Any]] = { + "default": {}, + "sort_by_tags": {"columns": {"sort_by_tags": True}}, + "sort_by_tags_config": {"columns": ColumnConfig(sort_by_tags=True)}, + "all_info": {"all_info": True}, + "meta": {"columns": {"meta": True}}, + "context": {"columns": {"context": True}}, + "source": {"columns": {"source": True}}, + "system_tags": {"columns": {"system_tags": True}}, + "content_hash": {"columns": {"content_hash": True}}, + "content_hash_sorted": {"columns": {"content_hash": True, "sort_by_tags": True}}, +} + +CALLS: list[int] = [] + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@pytest.fixture(autouse=True) +def _reset_calls() -> None: + CALLS.clear() + + +def _table() -> pa.Table: + return pa.table( + {"id": ORDER, "x": ORDER}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + + +def _source() -> ArrowTableSource: + return ArrowTableSource( + _table(), tag_columns=["id"], source_id="nums", label="nums" + ) + + +def _computed_pod_stream() -> FunctionPodStream: + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + stream.as_table() + CALLS.clear() + return stream + + +def _computed_operator_stream() -> Any: + stream = SelectDataColumns(["x"])(_source()) + stream.as_table() + return stream + + +def _derived_source() -> DerivedSource: + return DerivedSource(_source(), tags_as_data={"id": "id_copy"}) + + +STREAMS = { + "arrow_table_source": _source, + "computed_function_pod_stream": _computed_pod_stream, + "computed_operator_stream": _computed_operator_stream, + "derived_source": _derived_source, +} + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("n", [0, 3, ROWS, 50]) +@pytest.mark.parametrize("kind", list(STREAMS)) +def test_head_is_the_prefix_of_as_table_for_every_configuration( + kind: str, n: int, config: str +) -> None: + """A stream whose rows are all held: head(n, cfg) is as_table(cfg) cut to n rows.""" + stream = STREAMS[kind]() + head = stream.head(n, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(stream.as_table(**CONFIGS[config]).slice(0, n)) + assert CALLS == [] + + +@pytest.mark.parametrize("kind", list(STREAMS)) +def test_the_default_head_is_not_sorted_and_the_sorted_one_is(kind: str) -> None: + """Only sort_by_tags and all_info sort; the first n of the sorted rows are shown.""" + stream = STREAMS[kind]() + ids = stream.head(ROWS).table.column("id").to_pylist() + assert ids == ORDER + for config in ({"columns": {"sort_by_tags": True}}, {"all_info": True}): + assert stream.head(3, **config).table.column("id").to_pylist() == [0, 1, 2] + + +def test_asking_for_sorted_rows_does_not_raise() -> None: + """``sort_by_tags`` was refused by ``head`` before; now it is a column option.""" + head = _source().head(columns={"sort_by_tags": True}) + assert head.table.column("id").to_pylist() == sorted(ORDER) + + +def test_a_partly_computed_stream_sorts_the_rows_computed_so_far() -> None: + """After a partial iteration, the head is the computed rows, sorted; no more is computed.""" + stream = triple.pod(_source()) + rows = stream.iter_data() + for _ in range(4): # computes the first four inputs: ids 5, 2, 7, 0 + next(rows) + assert CALLS == [5, 2, 7, 0] + + head = stream.head(3, all_info=True) + unsorted_head = stream.head(3) + + assert CALLS == [5, 2, 7, 0] + assert head.status is HeadStatus.PARTIAL + # The rows computed so far in tag order, not the first 3 of the whole result in tag order. + assert head.table.column("id").to_pylist() == [0, 2, 5] + assert unsorted_head.table.column("id").to_pylist() == [5, 2, 7] + + +class _OneBatch: + """A ``DynamicSourceProtocol`` double that gives one batch, in ``ORDER``, and counts.""" + + def __init__(self) -> None: + self.polls = 0 + self.fetches = 0 + + def identity(self) -> Any: + return "one-batch" + + def to_config(self) -> dict[str, Any] | None: + return None + + @classmethod + def from_config(cls, config: dict[str, Any]) -> _OneBatch: + raise NotImplementedError + + def schema(self) -> Any: + return None + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + self.polls += 1 + return cursor is None + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + self.fetches += 1 + return Cursor(value=1), { + "id": pa.array(ORDER, type=pa.int64()), + "val": pa.array(ORDER, type=pa.int64()), + } + + async def close(self) -> None: + return None + + +def test_a_polling_source_sorts_the_rows_it_has_polled_and_polls_nothing() -> None: + """Before a poll there are no rows to sort; after one, the polled rows sort.""" + fake = _OneBatch() + source = PollingSource(fake, tag_columns="id") + assert source.head(3, all_info=True).status is HeadStatus.NOT_COMPUTED + assert (fake.polls, fake.fetches) == (0, 0) + + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + head = source.head(3, all_info=True) + + assert (fake.polls, fake.fetches) == seen + assert head.table.column("id").to_pylist() == [0, 1, 2] + assert source.head(3).table.column("id").to_pylist() == ORDER[:3] + + +def test_a_filter_stream_after_a_run_sorts_like_as_table() -> None: + """An operator stream with dropped rows: head(n, cfg) is as_table(cfg) cut to n.""" + stream = PolarsFilter(constraints={"x": 5})(_source()) + stream.as_table() + head = stream.head(5, all_info=True) + assert head.table.equals(stream.as_table(all_info=True).slice(0, 5)) + + +def test_a_dict_source_head_sorts() -> None: + """Every source that wraps an ArrowTableSource sorts the same way.""" + source = DictSource( + [{"id": i, "x": i} for i in ORDER], tag_columns=["id"], source_id="d" + ) + assert source.head(3, all_info=True).table.equals( + source.as_table(all_info=True).slice(0, 3) + ) + + +def test_an_unbound_source_proxy_head_accepts_a_sorted_configuration() -> None: + """A proxy with no live source shows no rows, whatever the column configuration.""" + from orcapod.core.sources.source_proxy import SourceProxy + + live = _source() + proxy = SourceProxy( + source_id="nums", + expected_class_name="ArrowTableSource", + tag_schema=Schema({"id": int}), + data_schema=Schema({"x": int}), + content_hash_str=live.content_hash().to_string(), + pipeline_hash_str=live.pipeline_hash().to_string(), + ) + head = proxy.head(3, all_info=True) + assert head.status is HeadStatus.UNAVAILABLE diff --git a/tests/test_pipeline/test_node_head_sorting.py b/tests/test_pipeline/test_node_head_sorting.py new file mode 100644 index 000000000..a3996ac95 --- /dev/null +++ b/tests/test_pipeline/test_node_head_sorting.py @@ -0,0 +1,183 @@ +"""A job node's ``head()`` sorts the way ``as_table`` does, among the rows it has (ITL-1053). + +``head(n, cfg)`` is ``as_table(cfg).slice(0, n)`` for every column configuration, whatever +order the rows were computed in: input order on the sync path, and on the async path an +order forced here (events, no sleeps) to be the reverse of tag order. Operator nodes are +tested after a sync run, or from a store a sync run filled: an operator node's head after +an async run says NOT_COMPUTED until ITL-1042 is fixed. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.nodes import OperatorJobNode +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.streams import HeadStatus +from orcapod.types import CacheMode + +N_ROWS = 6 + +CONFIGS: dict[str, dict[str, Any]] = { + "default": {}, + "sort_by_tags": {"columns": {"sort_by_tags": True}}, + "all_info": {"all_info": True}, + "meta": {"columns": {"meta": True}}, + "context": {"columns": {"context": True}}, + "source": {"columns": {"source": True}}, + "system_tags": {"columns": {"system_tags": True}}, + "content_hash": {"columns": {"content_hash": True}}, + "content_hash_sorted": {"columns": {"content_hash": True, "sort_by_tags": True}}, +} + +CALLS: list[int] = [] + +# Rows are released in reverse tag order: row i waits for row i + 1. Bound to the running +# loop on first use, so each async run builds its own. +gates: list[asyncio.Event] = [] + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@function_pod(output_keys="y") +async def triple_last_row_first(x: int) -> int: + """Triple ``x``; row ``i`` finishes only after row ``i + 1``, so rows complete in + reverse tag order. The wait is bounded so a failure is a missing row, not a hang.""" + CALLS.append(x) + if x + 1 < N_ROWS: + await asyncio.wait_for(gates[x + 1].wait(), timeout=60) + gates[x].set() + return 3 * x + + +def _source(order: list[int]) -> ArrowTableSource: + """A source of rows ``id`` and ``x`` in the given order (non-nullable ``int64``).""" + table = pa.table( + {"id": order, "x": order}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="nums", label="nums") + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +def _build(pod: Any, store: ArrowDatabaseProtocol, order: list[int]) -> PipelineJob: + job = PipelineJob(name="itl1053", store=store) + with job: + pod.pod(_source(order), label="node") + return job + + +def _run(kind: str, store: ArrowDatabaseProtocol) -> PipelineJob: + """Run a job so that its rows are stored out of tag order; return the job. + + ``sync``: the input is in reverse tag order. ``async``: the input is in tag order and + the rows complete in reverse tag order. + """ + gates.clear() + if kind == "sync": + job = _build(triple, store, list(reversed(range(N_ROWS)))) + job.run(orchestrator=SyncPipelineOrchestrator()) + return job + gates.extend(asyncio.Event() for _ in range(N_ROWS)) + job = _build(triple_last_row_first, store, list(range(N_ROWS))) + job.run(orchestrator=AsyncPipelineOrchestrator()) + return job + + +def _held_order(node: Any) -> list[int]: + """The tag order the node holds its rows in (unsorted ``as_table``).""" + return node.as_table().column("id").to_pylist() + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("n", [0, 3, N_ROWS, 50]) +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("run_kind", ["sync", "async"]) +def test_a_function_nodes_head_is_the_prefix_of_as_table_in_any_completion_order( + run_kind: str, store_kind: str, n: int, config: str, tmp_path: Path +) -> None: + """After a run, and from the store in a new job, head(n, cfg) == as_table(cfg)[:n].""" + store = _store(store_kind, tmp_path) + job = _run(run_kind, store) + assert _held_order(job.nodes["node"]) != sorted(_held_order(job.nodes["node"])) + CALLS.clear() + fresh = _build( + triple if run_kind == "sync" else triple_last_row_first, + store, + list(range(N_ROWS)), + ) + for node in (job.nodes["node"], fresh.nodes["node"]): + head = node.head(n, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(node.as_table(**CONFIGS[config]).slice(0, n)) + assert CALLS == [] + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("run_kind", ["sync", "async"]) +def test_head_sorts_only_when_asked_to( + run_kind: str, store_kind: str, tmp_path: Path +) -> None: + """The default head keeps the node's order; sort_by_tags and all_info sort by tag.""" + job = _run(run_kind, _store(store_kind, tmp_path)) + node = job.nodes["node"] + held = _held_order(node) + assert held != sorted(held) + assert node.head(N_ROWS).table.column("id").to_pylist() == held + for config in ({"columns": {"sort_by_tags": True}}, {"all_info": True}): + ids = node.head(N_ROWS, **config).table.column("id").to_pylist() + assert ids == sorted(held) + # The first n of the sorted rows, not the sorted first n of the held rows. + assert node.head(2, all_info=True).table.column("id").to_pylist() == [0, 1] + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_an_operator_nodes_head_is_the_prefix_of_as_table( + store_kind: str, config: str, tmp_path: Path +) -> None: + """An operator node's stored rows, out of tag order, sort as ``as_table`` sorts.""" + store = _store(store_kind, tmp_path) + order = list(reversed(range(N_ROWS))) + + def node(mode: CacheMode) -> OperatorJobNode: + return OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source(order)], + pipeline_database=store, + cache_mode=mode, + ) + + ran = node(CacheMode.LOG) + ran.execute(_source(order)) + for each in (ran, node(CacheMode.REPLAY)): + head = each.head(3, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(each.as_table(**CONFIGS[config]).slice(0, 3)) + sorted_ids = node(CacheMode.REPLAY).head(3, all_info=True).table.column("id") + assert sorted_ids.to_pylist() == [0, 1, 2] From a4db4aedf5775e2609180f109b6ddc91d37d5bdc Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 23:46:43 -0800 Subject: [PATCH 03/16] test(streams): type the head sorting cases for basedpyright (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_core/streams/test_head_sorting.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 324e7372d..95afbc22a 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -118,8 +118,10 @@ def test_the_default_head_is_not_sorted_and_the_sorted_one_is(kind: str) -> None stream = STREAMS[kind]() ids = stream.head(ROWS).table.column("id").to_pylist() assert ids == ORDER - for config in ({"columns": {"sort_by_tags": True}}, {"all_info": True}): - assert stream.head(3, **config).table.column("id").to_pylist() == [0, 1, 2] + by_columns = stream.head(3, columns={"sort_by_tags": True}) + by_all_info = stream.head(3, all_info=True) + assert by_columns.table.column("id").to_pylist() == [0, 1, 2] + assert by_all_info.table.column("id").to_pylist() == [0, 1, 2] def test_asking_for_sorted_rows_does_not_raise() -> None: From 156f6aca43994a36911b254052865b637302313f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 00:01:24 -0800 Subject: [PATCH 04/16] feat(streams): one deterministic tag sort and a tag-sorted notebook preview (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- changelog.d/ITL-1053.md | 11 +- design_issues/ITL-1053-PV2.md | 7 ++ docs/concepts/sources.md | 17 +-- docs/concepts/streams.md | 16 +-- src/orcapod/core/function_pod.py | 48 +++------ src/orcapod/core/nodes/function_node.py | 14 +-- src/orcapod/core/nodes/source_node.py | 10 +- .../core/sources/spiraldb_table_source.py | 22 +++- .../core/streams/arrow_table_stream.py | 12 +-- src/orcapod/core/streams/base.py | 12 +-- src/orcapod/core/streams/head.py | 40 ++++++- src/orcapod/utils/arrow_utils.py | 44 +++++--- .../test_spiraldb_table_source_streaming.py | 9 +- .../streams/test_head_and_preview.py | 13 ++- tests/test_core/streams/test_head_sorting.py | 53 ++++++++++ tests/test_core/streams/test_streams.py | 8 +- tests/test_pipeline/test_node_head_sorting.py | 100 ++++++++++++++++++ 17 files changed, 320 insertions(+), 116 deletions(-) diff --git a/changelog.d/ITL-1053.md b/changelog.d/ITL-1053.md index 1592700c1..7560182e6 100644 --- a/changelog.d/ITL-1053.md +++ b/changelog.d/ITL-1053.md @@ -3,7 +3,10 @@ - `head()` takes the column configuration of `as_table`, sorting included, and applies it to the rows it has before taking the first `n` (ITL-1053). `head(n, columns={"sort_by_tags": True})` no longer raises, and `head(n, all_info=True)` now sorts by tag, as `as_table` - does; the default `head()` (and the notebook preview) stays unsorted. A stream that has all + does; the default `head()` stays unsorted, as `as_table()` is. The notebook preview asks for the + sorted head, so it shows rows in tag order whatever order they were computed in, and its + header line says "sorted by tags" (for a bounded read, "10 rows read, sorted by tags, of + about N"). A stream that has all its rows (a job node's stored rows, a computed stream, an in-memory source) gives exactly `as_table(...).slice(0, n)` for every configuration, whatever order its rows were computed in. A stream that has read or computed only part of its rows (a `SpiralDBTableSource` @@ -14,3 +17,9 @@ - A node's `head(n, all_info=True)` no longer differs from `as_table(all_info=True)`'s first rows when an async run completed rows out of tag order (the cause of an occasionally failing test, ITL-1001). +- Every tag sort is now one deterministic sort (`arrow_utils.sort_by_tag_columns`): by the + tag columns, nulls last, rows with equal tags ordered by their system tag columns. A job + node's and a function pod stream's sorted `as_table` used an unstable Polars sort that put + nulls first and left the order of rows with equal tags unspecified; it now orders them as + an `ArrowTableStream` does. Only the order of rows within equal or null tags changes: no + values, record ids or hashes. diff --git a/design_issues/ITL-1053-PV2.md b/design_issues/ITL-1053-PV2.md index 2433cd3c2..027b76c69 100644 --- a/design_issues/ITL-1053-PV2.md +++ b/design_issues/ITL-1053-PV2.md @@ -22,3 +22,10 @@ ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded r has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the rows read, sorted" unless the store returns rows in tag order. + +The notebook preview asks for the tag-sorted head (developer, November 12, 2026: "preview should +sort by tags"; `preview_html`), and its header says "sorted by tags", or for a bounded read +"N rows read, sorted by tags, of about T". Every tag sort goes through +`arrow_utils.sort_by_tag_columns`: tags ascending, nulls last, ties by the system tag columns, +then input order (the node and function pod stream sorts were an unstable Polars sort with nulls +first). diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index ef5e50baa..75155e01a 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -154,14 +154,15 @@ with `max_rows=None`, but it cannot yet be run through a job on a laptop. **Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell (its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10 rows, in one scan to which the row cap does not apply, so a table of any size can be looked -at; `head(n)` reads `n`. With `sort_by_tags=True` (or `all_info=True`) the head sorts only -the rows it read: the rows read, in tag order, not the first `n` of the table in tag order. -The header line says what is shown, for -example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's -recorded row count, read without scanning), `all 4 rows` when the source reads no more than -that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with -`where=`. Tag columns come first, and the line above the table names them. To see more rows, use `as_df()` with a -`limit=` or `where=` source. +at; `head(n)` reads `n`. The preview shows the rows it read in tag order (it asks for +`sort_by_tags`, as does `head(columns={"sort_by_tags": True})` or `all_info=True`): the rows +read, sorted, not the first `n` of the table in tag order. The header line says what is +shown, for example `SpiralDBTableSource[spike_event] — 10 rows read, sorted by tags, of +about 63,349` (the table's recorded row count, read without scanning), `all 4 rows, sorted +by tags` when the source reads no more than that, or `10 matching rows read, sorted by tags +(where unit_id=106); the table has about 63,349 rows` with `where=`. The unsorted `head()` +says `first 10 rows of about 63,349`. Tag columns come first, and the line above the table +names them. To see more rows, use `as_df()` with a `limit=` or `where=` source. **Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the same table, so two sources on one table have the same `source_id` and `content_hash` diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index a7da9d4af..3f389304c 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -103,8 +103,9 @@ Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_l ## Looking at data In a notebook, a stream, source or job node shown at the end of a cell renders a short -table: its first rows, with a line saying what they are. That preview is `head()` rendered, -and `head(n)` gives the same rows to work with. Neither ever computes anything: they show +table: its first rows in tag order, with a line saying what they are. That preview is +`head()` asked for `sort_by_tags` and rendered, and `head(n, columns={"sort_by_tags": True})` +gives the same rows to work with. Neither ever computes anything: they show only what is readily available, and never call a function, run an operator, deliver a side effect, poll a source or write to a database. @@ -164,9 +165,10 @@ What each kind of object shows: `head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`, sorting included, and applies it to the rows it has before taking the first `n`. The default -does not sort (so the notebook preview, which is the default `head()`, shows rows in the -order the stream holds them, as `as_table()` does). With `sort_by_tags=True`, or -`all_info=True`, rows are sorted by tag first: +does not sort, as `as_table()` does not, so `head()` shows rows in the order the stream +holds them. The notebook preview asks for the sorted head, so it shows the same rows +whatever order they were computed in. With `sort_by_tags=True`, or `all_info=True`, rows are +sorted by tag first (nulls last; rows with equal tags are ordered by their system tags): ```python from orcapod.sources import DictSource @@ -187,7 +189,9 @@ source) then gives exactly the first `n` rows of `as_table` with the same config whatever order the rows were computed in. A stream that has only some of its rows sorts those: a source that reads a bounded number of rows (such as a `SpiralDBTableSource`) shows the rows it read, in tag order, and a stream computed only in part shows the rows computed so -far in tag order. It is not "the first `n` rows in tag order" of rows it never read. +far in tag order. It is not "the first `n` rows in tag order" of rows it never read, and the +preview's header line says so ("10 rows read, sorted by tags, of about 63,349"). A stream +with no tag columns previews its rows in stream order. A head of an operator is never the operator applied to the heads of its inputs: a group over the first ten rows has the wrong members. So an uncomputed stream shows its columns and diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 015e00098..45e0992ff 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1349,48 +1349,28 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``; called only when ``column_config`` asks for them. """ - drop_columns = [] - if not column_config.system_tags: - # TODO: get system tags more effiicently - drop_columns.extend( - [ - c - for c in table.column_names - if c.startswith(constants.SYSTEM_TAG_PREFIX) - ] - ) - if not column_config.source: - drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) - if not column_config.context: - drop_columns.append(constants.CONTEXT_KEY) - - output_table = table.drop([c for c in drop_columns if c in table.column_names]) - if column_config.content_hash: hash_column_name = ( "_content_hash" if column_config.content_hash is True else column_config.content_hash ) - output_table = output_table.append_column( - hash_column_name, content_hashes() - ) - + table = table.append_column(hash_column_name, content_hashes()) if column_config.sort_by_tags: - # TODO: reimplement using polars natively - output_table_schema = output_table.schema - output_table = ( - pl.DataFrame(output_table) - .sort(by=self.keys()[0], descending=False) - .to_arrow() - ) - output_table = arrow_utils.restore_schema_nullability( - output_table, output_table_schema + # Before the system tags are dropped: they break ties between equal tags. + table = arrow_utils.sort_by_tag_columns(table, self.keys()[0]) + drop_columns = [] + if not column_config.system_tags: + drop_columns.extend( + c + for c in table.column_names + if c.startswith(constants.SYSTEM_TAG_PREFIX) ) - # output_table = output_table.sort_by( - # [(column, "ascending") for column in self.keys()[0]] - # ) - return output_table + if not column_config.source: + drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) + if not column_config.context: + drop_columns.append(constants.CONTEXT_KEY) + return table.drop([c for c in drop_columns if c in table.column_names]) class CallableWithPodProtocol(Protocol): diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 328f29988..5c5ad42c5 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -18,7 +18,6 @@ import asyncio import contextlib -import dataclasses import logging import uuid from abc import abstractmethod @@ -601,22 +600,15 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``, in the order of ``table``; called only when ``column_config`` asks for them. """ - tag_keys = self.keys()[0] - output_table = arrow_utils.apply_column_config( - table, dataclasses.replace(column_config, sort_by_tags=False), tag_keys - ) if column_config.content_hash: hash_column_name = ( "_content_hash" if column_config.content_hash is True else column_config.content_hash ) - output_table = output_table.append_column( - hash_column_name, content_hashes() - ) - if column_config.sort_by_tags: - output_table = arrow_utils.sort_by_tag_columns(output_table, tag_keys) - return output_table + table = table.append_column(hash_column_name, content_hashes()) + # The sort sees every column, the system tags included (they break ties). + return arrow_utils.apply_column_config(table, column_config, self.keys()[0]) def head( self, diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index a4416831f..e5546f46b 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -21,7 +21,7 @@ HeadStatus, StreamHead, head_of, - render_error_html, + preview_html, ) from orcapod.core.streams.base import ( check_head_size, @@ -333,12 +333,8 @@ def _head_without_rows( ) def _repr_html_(self) -> str: - """Render ``head()`` for a notebook; never computes.""" - try: - head = self.head() - except Exception as error: # noqa: BLE001 - a display hook must not raise - return render_error_html(self, error) - return head._repr_html_() + """Render ``head()`` in tag order for a notebook; never computes.""" + return preview_html(self) async def async_iter_data(self): """Asynchronous iterator over (tag, data) pairs. diff --git a/src/orcapod/core/sources/spiraldb_table_source.py b/src/orcapod/core/sources/spiraldb_table_source.py index 3ea5db8c0..0027745c3 100644 --- a/src/orcapod/core/sources/spiraldb_table_source.py +++ b/src/orcapod/core/sources/spiraldb_table_source.py @@ -662,19 +662,31 @@ def _head( recorded = min(self._limit, recorded) return _Head(rows, True, recorded, True) - def _head_note(self, head: "_Head") -> str: - """Return the words a notebook display puts after the header: the one place.""" + def _head_note(self, head: "_Head", sorted_by_tags: bool = False) -> str: + """Return the words a notebook display puts after the header: the one place. + + With ``sorted_by_tags`` the rows read are in tag order, which is not the first + rows of the table in tag order, and the words say so. + """ shown = head.rows.num_rows if not head.rows_read: return f"rows not read; the table has about {head.total_rows:,} rows" if not head.total_is_estimate: - return f"all {shown:,} row{'' if shown == 1 else 's'}" + all_rows = f"all {shown:,} row{'' if shown == 1 else 's'}" + return f"{all_rows}, sorted by tags" if sorted_by_tags else all_rows if self._where: condition = ", ".join(f"{c}={v!r}" for c, v in self._where.items()) + read = ( + f"{shown:,} matching rows read, sorted by tags" + if sorted_by_tags + else f"first {shown:,} matching rows" + ) return ( - f"first {shown:,} matching rows (where {condition}); the table has " + f"{read} (where {condition}); the table has " f"about {head.total_rows:,} rows" ) + if sorted_by_tags: + return f"{shown:,} rows read, sorted by tags, of about {head.total_rows:,}" return f"first {shown:,} rows of about {head.total_rows:,}" def head( @@ -713,7 +725,7 @@ def head( total_rows=head.total_rows, total_is_estimate=head.total_is_estimate, origin="read", - description=self._head_note(head), + description=self._head_note(head, config.sort_by_tags), ) def _display_name(self) -> str: diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index e16a2debf..698a6ebbe 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -301,17 +301,7 @@ def _assemble_table( table = arrow_utils.hstack_tables(*table_stack) if columns_config.sort_by_tags: - # TODO: cleanup the sorting tag selection logic - try: - target_tags = ( - self._all_tag_columns - if columns_config.system_tags - else self._tag_columns - ) - return table.sort_by([(column, "ascending") for column in target_tags]) - except pa.ArrowTypeError: - # If sorting fails, fall back to unsorted table - return table + return arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) return table diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index c5d0cabfe..6ca28e901 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -12,7 +12,7 @@ HeadStatus, StreamHead, default_note, - render_error_html, + preview_html, ) from orcapod.protocols.core_protocols import ( DataProtocol, @@ -496,7 +496,7 @@ def head( of ``as_table`` with that configuration. A stream that has read or computed only part of its rows (a source's bounded read, a partly computed stream) sorts those rows: the rows read, in tag order, not the first ``n`` in tag order. The - notebook preview of a stream is its default (unsorted) ``head()``. + notebook preview of a stream is its ``head()`` asked for ``sort_by_tags``. This default shows no rows: a stream class that can serve rows without computing overrides it. The zero-row table then has the stream's columns, typed from its @@ -562,9 +562,5 @@ def _schema_requires_io(self, config: ColumnConfig) -> bool: return any(schema_requires_io(upstream, config) for upstream in self.upstreams) def _repr_html_(self) -> str: - """Render ``head()`` for a notebook; never computes.""" - try: - head = self.head() - except Exception as error: # noqa: BLE001 - a display hook must not raise - return render_error_html(self, error) - return head._repr_html_() + """Render ``head()`` in tag order for a notebook; never computes.""" + return preview_html(self) diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index a794fcc1a..14f7f018f 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -111,6 +111,10 @@ class StreamHead: description: When given, the whole text after the name in the header, written by a stream that words its own rows (a table-backed source naming its ``where``); otherwise the header is built from the fields above. + sorted_by_tags: ``True`` when the rows were asked for in tag order (the notebook + preview): the header then says so, and for a bounded read or a partial + computation says they are the rows read or computed so far, in tag order. + A ``description`` words its own sorting. """ table: pa.Table @@ -123,6 +127,7 @@ class StreamHead: origin: str = "held" note: str | None = None description: str | None = None + sorted_by_tags: bool = False @classmethod def of_rows( @@ -216,10 +221,11 @@ def _describe(self) -> str: text = self._describe_rows(shown) elif self.status is HeadStatus.PARTIAL: total = self.total_rows if self.total_rows is not None else shown + suffix = ", sorted by tags" if self.sorted_by_tags else "" if shown == total: - text = f"{total:,} {_rows(total)} computed so far" + text = f"{total:,} {_rows(total)} computed so far{suffix}" else: - text = f"first {shown:,} of {total:,} rows computed so far" + text = f"first {shown:,} of {total:,} rows computed so far{suffix}" text += " (computation incomplete)" else: return self.note or _DEFAULT_NOTES[self.status] @@ -233,13 +239,19 @@ def _describe_rows(self, shown: int) -> str: noun = _ORIGIN_NOUNS.get(self.origin, "rows") if shown == 0 and (total is None or total == 0): return "computed: no rows" if self.origin == "computed" else f"no {noun}" + suffix = ", sorted by tags" if self.sorted_by_tags else "" if total is None: - return f"first {shown:,} {_rows(shown)}" + return f"first {shown:,} {_rows(shown)}{suffix}" if self.total_is_estimate: + if self.sorted_by_tags: + # A bounded read: the rows read, in tag order, not the first in tag order. + return ( + f"{shown:,} {_rows(shown)} read, sorted by tags, of about {total:,}" + ) return f"first {shown:,} {_rows(shown)} of about {total:,}" if shown == total: - return f"all {total:,} {noun if total != 1 else noun[:-1]}" - return f"first {shown:,} of {total:,} {noun}" + return f"all {total:,} {noun if total != 1 else noun[:-1]}{suffix}" + return f"first {shown:,} of {total:,} {noun}{suffix}" def _display_frame(self) -> pl.DataFrame: """Return a Polars frame of the rows for display: tag columns first, large cells @@ -314,6 +326,24 @@ def display_name(stream: Any) -> str: return name +def preview_html(stream: Any) -> str: + """Return the notebook preview of ``stream``: its head with the rows in tag order. + + The preview asks ``head`` for ``sort_by_tags`` (developer, November 12, 2026), so the + rows shown do not depend on the order they were computed in. For a bounded read or a + partial computation they are the rows read or computed so far, in tag order. A stream + with no tag column shows its rows in stream order. It never computes anything more + than ``head`` does. + """ + try: + head = stream.head(columns={"sort_by_tags": True}) + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(stream, error) + if head.tag_columns and head.table.num_rows: + head = dataclasses.replace(head, sorted_by_tags=True) + return head._repr_html_() + + def head_of( head: Any, stream: Any, *, origin: str | None = None, note: str | None = None ) -> StreamHead: diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index d85e9a75c..b5ed6c9a2 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1831,17 +1831,24 @@ def apply_column_config( if c.startswith(constants.META_PREFIX) and not any(c.startswith(p) for p in normalized_meta_prefixes) ) - output_table = table.drop([c for c in drop_columns if c in table.column_names]) if column_config.sort_by_tags: - output_table = sort_by_tag_columns(output_table, tag_keys) - return output_table + # Before the system tags are dropped: they break ties between equal tags. + table = sort_by_tag_columns(table, tag_keys) + return table.drop([c for c in drop_columns if c in table.column_names]) def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Table": - """Return ``table`` with its rows sorted ascending by the tag columns ``tag_keys``. + """Return ``table`` with its rows in tag order: the one sort every tag sort uses. - Whole rows move together: a column added to ``table`` before the sort stays with - its row. Column types and nullability are those of ``table``. + The rows are ordered ascending by the tag columns ``tag_keys``, nulls last. Rows with + equal tags are then ordered by the system tag columns ``table`` has (they hold the + record ids), in column order, and rows still equal keep their input order, so the + result does not depend on the order the rows arrived in whenever the table has system + tags. A table with no tag column is returned as it is. + + Whole rows move together: a column added to ``table`` before the sort (a content hash) + stays with its row. Column types and nullability are those of ``table``. Sort before + dropping the system tag columns, or the tiebreak is lost. Args: table: The table to sort. @@ -1850,12 +1857,25 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab Returns: A new table with the same columns, rows in tag order. """ - import polars as pl - - sorted_table = ( - pl.DataFrame(table).sort(by=list(tag_keys), descending=False).to_arrow() - ) - return restore_schema_nullability(sorted_table, table.schema) + system_tags = [ + c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) + ] + keys = [*tag_keys, *(c for c in system_tags if c not in tag_keys)] + if not keys: + return table + try: + return table.sort_by([(key, "ascending") for key in keys]) + except (pa.ArrowNotImplementedError, pa.ArrowTypeError): + # Arrow cannot order this type (a list tag): order by Polars, nulls last, and + # keep the input order of equal rows. + import polars as pl + + sorted_table = ( + pl.DataFrame(table) + .sort(by=keys, descending=False, nulls_last=True, maintain_order=True) + .to_arrow() + ) + return restore_schema_nullability(sorted_table, table.schema) if __name__ == "__main__": diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 38640d111..1884c446f 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -614,7 +614,10 @@ def test_display_reads_one_head_of_ten_rows(self, big): html = source._repr_html_() assert self._scan_limits() == [10] - assert "SpiralDBTableSource[big] — first 10 rows of about 25" in html + assert ( + "SpiralDBTableSource[big] — 10 rows read, sorted by tags, of about 25" + in html + ) assert "shape: (10, 3)" in html assert "tags: subject_id, timestamp" in html @@ -660,7 +663,7 @@ def test_a_table_above_the_cap_previews_without_raising(self, big): html = source._repr_html_() assert self._scan_limits() == [10] - assert "first 10 rows of about 25" in html + assert "10 rows read, sorted by tags, of about 25" in html def test_a_small_table_shows_all_its_rows(self, project): """A table of fewer than 10 rows is shown whole, saying so.""" @@ -704,7 +707,7 @@ def test_the_header_is_escaped_in_the_display(self, project): html = source._repr_html_() assert ( - "first 10 matching rows (where subject_id='<b>'); " + "10 matching rows read, sorted by tags (where subject_id='<b>'); " "the table has about 25 rows" ) in html assert "'" not in html diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 3e082a04f..c4193b937 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -137,11 +137,18 @@ def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: def test_preview_is_the_rendered_head() -> None: - """The notebook preview is head() rendered.""" + """The notebook preview is head() rendered, with the rows in tag order (ITL-1053).""" source = _source() html = source._repr_html_() - assert "ArrowTableSource[readings] — first 10 of 25 rows" in html - assert html == source.head()._repr_html_() + assert "ArrowTableSource[readings] — first 10 of 25 rows, sorted by tags" in html + assert ( + html.split("")[1:] + == ( + source.head(columns={"sort_by_tags": True}) + ._repr_html_() + .split("")[1:] + ) + ) # --------------------------------------------------------------------------- diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 95afbc22a..927096520 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -232,3 +232,56 @@ def test_an_unbound_source_proxy_head_accepts_a_sorted_configuration() -> None: ) head = proxy.head(3, all_info=True) assert head.status is HeadStatus.UNAVAILABLE + + +# --------------------------------------------------------------------------- +# The notebook preview sorts by tags (developer, November 12, 2026) +# --------------------------------------------------------------------------- + + +def _preview_ids(html: str) -> list[int]: + """The ``id`` cells of a preview, in the order they are shown.""" + import re + + return [int(v) for v in re.findall(r"(\d+)", html)][::2] + + +def test_the_preview_of_a_source_shows_rows_in_tag_order_and_says_so() -> None: + """Rows not in tag order are previewed sorted; the default head is still unsorted.""" + source = _source() + html = source._repr_html_() + assert "ArrowTableSource[nums] — all 8 rows, sorted by tags" in html + assert _preview_ids(html) == sorted(ORDER) + assert source.head().table.column("id").to_pylist() == ORDER + + +def test_the_preview_of_a_partly_computed_stream_is_the_rows_computed_so_far() -> None: + """A partial computation previews the computed rows, in tag order; nothing more runs.""" + stream = triple.pod(_source()) + rows = stream.iter_data() + for _ in range(4): + next(rows) + html = stream._repr_html_() + assert CALLS == [5, 2, 7, 0] + assert "first 4 of 4 rows computed so far" not in html + assert "4 rows computed so far, sorted by tags (computation incomplete)" in html + assert _preview_ids(html) == [0, 2, 5, 7] + + +def test_the_preview_of_a_stream_with_no_computed_rows_has_no_sorting_caption() -> None: + """Only shown rows are described as sorted.""" + html = triple.pod(_source())._repr_html_() + assert "sorted by tags" not in html + assert CALLS == [] + + +def test_the_preview_of_a_stream_without_tag_columns_keeps_stream_order() -> None: + """With no tag column there is nothing to sort by; the rows keep their order.""" + source = ArrowTableSource( + _table().select(["x"]), tag_columns=[], source_id="untagged", label="untagged" + ) + html = source._repr_html_() + assert "sorted by tags" not in html + assert source.head(columns={"sort_by_tags": True}).table.column( + "x" + ).to_pylist() == (ORDER) diff --git a/tests/test_core/streams/test_streams.py b/tests/test_core/streams/test_streams.py index 3fcb254c1..e7dd6e427 100644 --- a/tests/test_core/streams/test_streams.py +++ b/tests/test_core/streams/test_streams.py @@ -543,8 +543,12 @@ def test_repr_html_is_the_rendered_head(self): html = stream._repr_html_() - assert html == stream.head()._repr_html_() - assert "ArrowTableStream[numbers] — first 10 of 25 rows" in html + sorted_head = stream.head(columns={"sort_by_tags": True}) + assert ( + html.split("")[1:] + == sorted_head._repr_html_().split("")[1:] + ) + assert "ArrowTableStream[numbers] — first 10 of 25 rows, sorted by tags" in html assert "tags: id" in html assert "shape: (10, 2)" in html diff --git a/tests/test_pipeline/test_node_head_sorting.py b/tests/test_pipeline/test_node_head_sorting.py index a3996ac95..a39223f7f 100644 --- a/tests/test_pipeline/test_node_head_sorting.py +++ b/tests/test_pipeline/test_node_head_sorting.py @@ -181,3 +181,103 @@ def node(mode: CacheMode) -> OperatorJobNode: assert head.table.equals(each.as_table(**CONFIGS[config]).slice(0, 3)) sorted_ids = node(CacheMode.REPLAY).head(3, all_info=True).table.column("id") assert sorted_ids.to_pylist() == [0, 1, 2] + + +# --------------------------------------------------------------------------- +# Equal and null tags: one deterministic sort (ITL-1053) +# --------------------------------------------------------------------------- + +TIE_KEYS: list[int | None] = [2, 1, None, 2, 1, None, 0, 2] + + +def _tie_source() -> ArrowTableSource: + """Rows with duplicate and null tags ``k``, each with its own ``x``.""" + table = pa.table( + {"k": pa.array(TIE_KEYS, pa.int64()), "x": pa.array(range(N_ROWS + 2))}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=True), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["k"], source_id="ties", label="ties") + + +def _tie_run(kind: str) -> Any: + """Run ``_tie_source`` through a function node; ``async`` completes rows in reverse.""" + gates.clear() + job = PipelineJob(name="ties", store=InMemoryArrowDatabase()) + if kind == "sync": + with job: + triple.pod(_tie_source(), label="node") + job.run(orchestrator=SyncPipelineOrchestrator()) + else: + gates.extend(asyncio.Event() for _ in range(len(TIE_KEYS))) + with job: + tie_last_row_first.pod(_tie_source(), label="node") + job.run(orchestrator=AsyncPipelineOrchestrator()) + return job.nodes["node"] + + +@function_pod(output_keys="y") +async def tie_last_row_first(x: int) -> int: + """Triple ``x``; row ``i`` finishes after row ``i + 1`` (reverse completion).""" + if x + 1 < len(TIE_KEYS): + await asyncio.wait_for(gates[x + 1].wait(), timeout=60) + gates[x].set() + return 3 * x + + +@pytest.mark.parametrize("config", ["sort_by_tags", "all_info", "content_hash_sorted"]) +def test_a_sorted_table_with_equal_and_null_tags_does_not_depend_on_completion_order( + config: str, +) -> None: + """Rows with equal tags are ordered by their system tags, nulls last, in any order.""" + sync_node, async_node = _tie_run("sync"), _tie_run("async") + assert _held_order_k(sync_node) != _held_order_k(async_node) + tables = { + name: node.as_table(**CONFIGS[config]) + for name, node in (("sync", sync_node), ("async", async_node)) + } + + def without_source_info(table: pa.Table) -> pa.Table: + """The table without ``_source_`` columns: their tokens name each run's rows.""" + return table.drop_columns( + [c for c in table.column_names if c.startswith("_source_")] + ) + + assert without_source_info(tables["sync"]).equals( + without_source_info(tables["async"]) + ) + assert tables["sync"].column("k").to_pylist() == [0, 1, 1, 2, 2, 2, None, None] + for name, node in (("sync", sync_node), ("async", async_node)): + for n in (0, 3, 5, 20): + head = node.head(n, **CONFIGS[config]) + assert head.table.equals(tables[name].slice(0, n)) + + +def _held_order_k(node: Any) -> list[int | None]: + return node.as_table().column("k").to_pylist() + + +# --------------------------------------------------------------------------- +# The notebook preview sorts by tags (developer, November 12, 2026) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_a_nodes_preview_is_the_same_after_any_completion_order( + store_kind: str, tmp_path: Path +) -> None: + """The preview of a node shows tag order: reversed completion looks like a sync run.""" + # A sync run over input in tag order holds its rows in tag order. + ascending = _build(triple, _store(store_kind, tmp_path / "a"), list(range(N_ROWS))) + ascending.run(orchestrator=SyncPipelineOrchestrator()) + sync_node = ascending.nodes["node"] + async_node = _run("async", _store(store_kind, tmp_path / "b")).nodes["node"] + assert _held_order(sync_node) != _held_order(async_node) + assert sync_node._repr_html_() == async_node._repr_html_() + html = async_node._repr_html_() + assert "all 6 rows, sorted by tags" in html + assert html.index("0") < html.index("5") From baede991e26bbd7aa18f5b30b3d1e17f420b4285 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 00:04:49 -0800 Subject: [PATCH 05/16] test(streams): type the preview cases (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_core/streams/test_head_sorting.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 927096520..c4c966648 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -258,6 +258,7 @@ def test_the_preview_of_a_source_shows_rows_in_tag_order_and_says_so() -> None: def test_the_preview_of_a_partly_computed_stream_is_the_rows_computed_so_far() -> None: """A partial computation previews the computed rows, in tag order; nothing more runs.""" stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) rows = stream.iter_data() for _ in range(4): next(rows) @@ -270,7 +271,9 @@ def test_the_preview_of_a_partly_computed_stream_is_the_rows_computed_so_far() - def test_the_preview_of_a_stream_with_no_computed_rows_has_no_sorting_caption() -> None: """Only shown rows are described as sorted.""" - html = triple.pod(_source())._repr_html_() + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + html = stream._repr_html_() assert "sorted by tags" not in html assert CALLS == [] From f01fcfa48191632f0d179e8b1d4ce40a15d1b7cf Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 00:51:37 -0800 Subject: [PATCH 06/16] fix(streams): keep system tags through the sort, custom hash column and tagless order (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- src/orcapod/core/nodes/function_node.py | 20 +++++++++++++-- .../core/sources/spiraldb_table_source.py | 4 ++- .../core/streams/arrow_table_stream.py | 9 +++---- src/orcapod/utils/arrow_utils.py | 6 ++--- .../test_spiraldb_table_source_streaming.py | 25 +++++++++++++------ tests/test_core/streams/test_head_sorting.py | 25 +++++++++++++++++++ 6 files changed, 70 insertions(+), 19 deletions(-) diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 5c5ad42c5..77947dfd5 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -18,6 +18,7 @@ import asyncio import contextlib +import dataclasses import logging import uuid from abc import abstractmethod @@ -600,6 +601,8 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``, in the order of ``table``; called only when ``column_config`` asks for them. """ + tag_keys = self.keys()[0] + hash_column_name = None if column_config.content_hash: hash_column_name = ( "_content_hash" @@ -607,8 +610,21 @@ def _shape_table( else column_config.content_hash ) table = table.append_column(hash_column_name, content_hashes()) - # The sort sees every column, the system tags included (they break ties). - return arrow_utils.apply_column_config(table, column_config, self.keys()[0]) + if column_config.sort_by_tags: + # The sort sees the hash column (it stays with its row) and the system tags + # (they break ties between equal tags). + table = arrow_utils.sort_by_tag_columns(table, tag_keys) + hashes = None + if hash_column_name is not None: + # Set the hash column aside: a custom name could look like a meta column. + hashes = table.column(hash_column_name) + table = table.drop_columns([hash_column_name]) + shaped = arrow_utils.apply_column_config( + table, dataclasses.replace(column_config, sort_by_tags=False), tag_keys + ) + if hash_column_name is not None and hashes is not None: + shaped = shaped.append_column(hash_column_name, hashes) + return shaped def head( self, diff --git a/src/orcapod/core/sources/spiraldb_table_source.py b/src/orcapod/core/sources/spiraldb_table_source.py index 0027745c3..a25f84863 100644 --- a/src/orcapod/core/sources/spiraldb_table_source.py +++ b/src/orcapod/core/sources/spiraldb_table_source.py @@ -725,7 +725,9 @@ def head( total_rows=head.total_rows, total_is_estimate=head.total_is_estimate, origin="read", - description=self._head_note(head, config.sort_by_tags), + description=self._head_note( + head, config.sort_by_tags and bool(self._tag_columns) + ), ) def _display_name(self) -> str: diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 698a6ebbe..e777af1eb 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -289,9 +289,6 @@ def _assemble_table( pa.field(hash_column_name, pa.large_string(), nullable=False), pa.array(content_hashes, type=pa.large_string()), ) - if not columns_config.system_tags: - # Check in original implementation - output_table = output_table.drop_columns(list(self._system_tag_columns)) table_stack = (output_table,) if columns_config.context: table_stack += (data_context_table,) @@ -301,8 +298,10 @@ def _assemble_table( table = arrow_utils.hstack_tables(*table_stack) if columns_config.sort_by_tags: - return arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) - + # With the system tags still in the table: they break ties between equal tags. + table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) + if not columns_config.system_tags: + table = table.drop_columns(list(self._system_tag_columns)) return table def clear_cache(self) -> None: diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index b5ed6c9a2..62f5be9ca 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1844,7 +1844,7 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab equal tags are then ordered by the system tag columns ``table`` has (they hold the record ids), in column order, and rows still equal keep their input order, so the result does not depend on the order the rows arrived in whenever the table has system - tags. A table with no tag column is returned as it is. + tags. A table with no regular tag column is returned as it is. Whole rows move together: a column added to ``table`` before the sort (a content hash) stays with its row. Column types and nullability are those of ``table``. Sort before @@ -1857,12 +1857,12 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab Returns: A new table with the same columns, rows in tag order. """ + if not tag_keys: + return table system_tags = [ c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ] keys = [*tag_keys, *(c for c in system_tags if c not in tag_keys)] - if not keys: - return table try: return table.sort_by([(key, "ascending") for key in keys]) except (pa.ArrowNotImplementedError, pa.ArrowTypeError): diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 1884c446f..1319a7bb8 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -632,19 +632,28 @@ def test_head_reads_one_bounded_scan(self, big): assert (head.total_rows, head.total_is_estimate) == (25, True) assert len(head) == 10 - def test_a_sorted_head_sorts_only_the_rows_it_read(self, big): - """``sort_by_tags`` sorts the one bounded read; it reads no more rows (ITL-1053).""" - source = SpiralDBTableSource(_PROJECT, "big", label="big") - tags = ["subject_id", "timestamp"] + def test_a_sorted_head_sorts_only_the_rows_it_read(self, project): + """``sort_by_tags`` reorders the one bounded read and reads no more (ITL-1053). + + The table is read in key order (subject, then timestamp) but tagged by + ``timestamp``, so the first ten rows read are not in tag order. + """ + table = project.create_table("default.mixed", key_schema=_KEYS) + table.write(_rows([10, 20, 30, 40, 50], subject="m1")) + table.write(_rows([1, 2, 3, 4, 5, 6], subject="m2")) + fake_pyspiral.scan_log.clear() + source = SpiralDBTableSource( + _PROJECT, "mixed", tag_columns=["timestamp"], label="mixed" + ) unsorted_head = source.head() sorted_head = source.head(columns={"sort_by_tags": True}) assert self._scan_limits() == [10, 10] - assert len(sorted_head) == 10 - assert sorted_head.table.equals( - unsorted_head.table.sort_by([(name, "ascending") for name in tags]) - ) + read = unsorted_head.table.column("timestamp").to_pylist() + assert read == [10, 20, 30, 40, 50, 1, 2, 3, 4, 5] + assert sorted_head.table.column("timestamp").to_pylist() == sorted(read) + assert "10 rows read, sorted by tags" in sorted_head.header() def test_an_unlabelled_source_is_shown_by_its_table_name(self, big): """With no label, the header names the table; the label itself is unchanged.""" diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index c4c966648..ee876f0e0 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -288,3 +288,28 @@ def test_the_preview_of_a_stream_without_tag_columns_keeps_stream_order() -> Non assert source.head(columns={"sort_by_tags": True}).table.column( "x" ).to_pylist() == (ORDER) + + +def test_a_custom_content_hash_column_is_kept_by_a_nodes_head() -> None: + """A custom hash column name that looks like a meta column is still shown.""" + from orcapod import PipelineJob + from orcapod.databases import InMemoryArrowDatabase + + job = PipelineJob(name="digest", store=InMemoryArrowDatabase()) + with job: + triple.pod(_source(), label="node") + job.run() + node = job.nodes["node"] + config = {"content_hash": "__digest", "sort_by_tags": True} + assert "__digest" in node.as_table(columns=config).column_names + head = node.head(3, columns=config) + assert head.table.equals(node.as_table(columns=config).slice(0, 3)) + + +def test_a_tagless_stream_keeps_its_order_when_sorted_with_system_tags() -> None: + """With no regular tag column the rows keep their order, system tags or not.""" + source = ArrowTableSource( + _table().select(["x"]), tag_columns=[], source_id="untagged", label="untagged" + ) + table = source.head(8, all_info=True).table + assert table.column("x").to_pylist() == ORDER From 93d6d7e1d92918f999af589338890608eac09743 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 01:07:48 -0800 Subject: [PATCH 07/16] test(spiraldb): type the read order (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_core/sources/test_spiraldb_table_source_streaming.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 1319a7bb8..bb06fbc13 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -650,7 +650,7 @@ def test_a_sorted_head_sorts_only_the_rows_it_read(self, project): sorted_head = source.head(columns={"sort_by_tags": True}) assert self._scan_limits() == [10, 10] - read = unsorted_head.table.column("timestamp").to_pylist() + read: list[int] = unsorted_head.table.column("timestamp").to_pylist() assert read == [10, 20, 30, 40, 50, 1, 2, 3, 4, 5] assert sorted_head.table.column("timestamp").to_pylist() == sorted(read) assert "10 rows read, sorted by tags" in sorted_head.header() From eb34ac07b2460ebcbafebe8bb54def583da9af4c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 01:15:10 -0800 Subject: [PATCH 08/16] test(spiraldb): cast the read order for the type checker (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_core/sources/test_spiraldb_table_source_streaming.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index bb06fbc13..fee141852 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -650,7 +650,7 @@ def test_a_sorted_head_sorts_only_the_rows_it_read(self, project): sorted_head = source.head(columns={"sort_by_tags": True}) assert self._scan_limits() == [10, 10] - read: list[int] = unsorted_head.table.column("timestamp").to_pylist() + read = cast(list[int], unsorted_head.table.column("timestamp").to_pylist()) assert read == [10, 20, 30, 40, 50, 1, 2, 3, 4, 5] assert sorted_head.table.column("timestamp").to_pylist() == sorted(read) assert "10 rows read, sorted by tags" in sorted_head.header() From fa39b0de317b432c9b226afd0f5dbb7ba6d42f24 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:05:21 -0800 Subject: [PATCH 09/16] fix(streams): keep a tagless stream's identity order, sorted heads with failed inputs, custom hash names (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- src/orcapod/core/cached_function_pod.py | 4 +- src/orcapod/core/function_pod.py | 37 ++++++++---- src/orcapod/core/nodes/function_node.py | 2 +- .../core/streams/arrow_table_stream.py | 7 ++- src/orcapod/utils/arrow_utils.py | 9 +-- tests/test_core/streams/test_head_sorting.py | 60 +++++++++++++++++-- 6 files changed, 95 insertions(+), 24 deletions(-) diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 6f088d98c..7ebad6c68 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -459,9 +459,9 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: yield tag, _without_stored_meta(data) def _computed_prefix( - self, limit: int + self, limit: int, contiguous: bool = True ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: - shown, total = super()._computed_prefix(limit) + shown, total = super()._computed_prefix(limit, contiguous) return [(tag, _without_stored_meta(data)) for tag, data in shown], total def head( diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 13dba76f0..7b6e6d2d3 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1410,7 +1410,9 @@ def head( # A complete computation: its table holds the rows; only the shown rows' # data are read, and only when their content hashes are asked for. def shown_content_hashes() -> pa.Array[Any]: - shown, _ = self._computed_prefix(wanted) + shown, _ = self._computed_prefix( + wanted, contiguous=not config.sort_by_tags + ) return pa.array( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), @@ -1436,7 +1438,7 @@ def shown_content_hashes() -> pa.Array[Any]: ) if not finished and not self._cached_output_datas: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) - shown, total = self._computed_prefix(wanted) + shown, total = self._computed_prefix(wanted, contiguous=not config.sort_by_tags) status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL if not shown: # Nothing to show (an empty result, inputs that all gave no output so far, @@ -1471,30 +1473,39 @@ def shown_content_hashes() -> pa.Array[Any]: ) def _computed_prefix( - self, limit: int + self, limit: int, contiguous: bool = True ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: """Return the first ``limit`` output rows computed so far, and how many there are. - The rows are the contiguous run of computed inputs from the first, in the order - iteration yields them; an input whose output is ``None`` (filtered out) is - skipped, as iteration skips it. Only the first ``limit`` rows are collected. + By default the rows are the contiguous run of computed inputs from the first, in + the order iteration yields them; an input whose output is ``None`` (filtered out) + is skipped, as iteration skips it. Only the first ``limit`` rows are collected. + With ``contiguous=False`` the rows are every computed output, in input order, + also those after an input that has no output (a failure): a sorted head sorts the + rows it has. Args: limit: The largest number of rows to return. + contiguous: Stop at the first input with no cached output. Returns: - The rows, and the number of output rows in the whole computed run. + The rows, and the number of output rows counted. """ shown: list[tuple[TagProtocol, DataProtocol]] = [] total = 0 - index = 0 - while index in self._cached_output_datas: + indexes = ( + iter(range(len(self._cached_output_datas) + 1)) + if contiguous + else iter(sorted(self._cached_output_datas)) + ) + for index in indexes: + if index not in self._cached_output_datas: + break tag, data = self._cached_output_datas[index] if data is not None: if total < limit: shown.append((tag, data)) total += 1 - index += 1 return shown, total def _shape_table( @@ -1511,6 +1522,7 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``; called only when ``column_config`` asks for them. """ + hash_column_name = None if column_config.content_hash: hash_column_name = ( "_content_hash" @@ -1518,15 +1530,16 @@ def _shape_table( else column_config.content_hash ) table = table.append_column(hash_column_name, content_hashes()) - if column_config.sort_by_tags: + if column_config.sort_by_tags and self.keys()[0]: # Before the system tags are dropped: they break ties between equal tags. table = arrow_utils.sort_by_tag_columns(table, self.keys()[0]) drop_columns = [] if not column_config.system_tags: + # The requested hash column stays, whatever its name starts with. drop_columns.extend( c for c in table.column_names - if c.startswith(constants.SYSTEM_TAG_PREFIX) + if c.startswith(constants.SYSTEM_TAG_PREFIX) and c != hash_column_name ) if not column_config.source: drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 7b5dfd81c..df517a236 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -632,7 +632,7 @@ def _shape_table( else column_config.content_hash ) table = table.append_column(hash_column_name, content_hashes()) - if column_config.sort_by_tags: + if column_config.sort_by_tags and tag_keys: # The sort sees the hash column (it stays with its row) and the system tags # (they break ties between equal tags). table = arrow_utils.sort_by_tag_columns(table, tag_keys) diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index e777af1eb..4a0d274a0 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -297,7 +297,12 @@ def _assemble_table( table = arrow_utils.hstack_tables(*table_stack) - if columns_config.sort_by_tags: + if columns_config.sort_by_tags and ( + self._tag_columns or columns_config.system_tags + ): + # A stream with no tag column is sorted only when the system tags are asked + # for (its identity, ``identity_structure``, depends on that order); the + # system tags stay in the table through the sort. # With the system tags still in the table: they break ties between equal tags. table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) if not columns_config.system_tags: diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index 62f5be9ca..2907021b6 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1831,7 +1831,7 @@ def apply_column_config( if c.startswith(constants.META_PREFIX) and not any(c.startswith(p) for p in normalized_meta_prefixes) ) - if column_config.sort_by_tags: + if column_config.sort_by_tags and tag_keys: # Before the system tags are dropped: they break ties between equal tags. table = sort_by_tag_columns(table, tag_keys) return table.drop([c for c in drop_columns if c in table.column_names]) @@ -1844,7 +1844,8 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab equal tags are then ordered by the system tag columns ``table`` has (they hold the record ids), in column order, and rows still equal keep their input order, so the result does not depend on the order the rows arrived in whenever the table has system - tags. A table with no regular tag column is returned as it is. + tags. A table with neither is returned as it is. A caller that must keep the order of + a stream with no regular tag column does not call this for it. Whole rows move together: a column added to ``table`` before the sort (a content hash) stays with its row. Column types and nullability are those of ``table``. Sort before @@ -1857,12 +1858,12 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab Returns: A new table with the same columns, rows in tag order. """ - if not tag_keys: - return table system_tags = [ c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ] keys = [*tag_keys, *(c for c in system_tags if c not in tag_keys)] + if not keys: + return table try: return table.sort_by([(key, "ascending") for key in keys]) except (pa.ArrowNotImplementedError, pa.ArrowTypeError): diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index ee876f0e0..7cd4e19c7 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -306,10 +306,62 @@ def test_a_custom_content_hash_column_is_kept_by_a_nodes_head() -> None: assert head.table.equals(node.as_table(columns=config).slice(0, 3)) -def test_a_tagless_stream_keeps_its_order_when_sorted_with_system_tags() -> None: - """With no regular tag column the rows keep their order, system tags or not.""" +def test_a_tagless_source_head_follows_as_table_with_and_without_system_tags() -> None: + """With no regular tag column the rows keep their order unless system tags are asked. + + Asked for, they are sorted by the system tags, as ``as_table`` sorts them (the + stream's identity depends on that); the head is that table cut to ``n`` rows. + """ source = ArrowTableSource( _table().select(["x"]), tag_columns=[], source_id="untagged", label="untagged" ) - table = source.head(8, all_info=True).table - assert table.column("x").to_pylist() == ORDER + plain = source.head(8, columns={"sort_by_tags": True}).table + assert plain.column("x").to_pylist() == ORDER + asked = source.head(8, all_info=True).table + assert asked.equals(source.as_table(all_info=True).slice(0, 8)) + + +def test_a_tagless_stream_with_system_tags_keeps_its_canonical_identity_order() -> None: + """A stream with no tag column is sorted by its system tags when they are asked for. + + ``identity_structure`` hashes ``as_table(system_tags, sort_by_tags)``: that order, + system tags only, is part of the content hash and must not change (ITL-1053). + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table({"record_id": ["b", "c", "a"], "x": [1, 2, 3]}) + stream = ArrowTableStream(table, tag_columns=[], system_tag_columns=["record_id"]) + ordered = stream.as_table(columns={"system_tags": True, "sort_by_tags": True}) + shown = next(c for c in ordered.column_names if c.endswith("record_id")) + assert ordered.column(shown).to_pylist() == ["a", "b", "c"] + plain = stream.as_table(columns={"sort_by_tags": True}) + assert plain.column("x").to_pylist() == [1, 2, 3] + + +def test_a_sorted_head_includes_rows_cached_after_a_missing_input() -> None: + """Sorting uses every output the stream holds, also those after an input with none.""" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + rows = stream.iter_data() + for _ in range(4): + next(rows) + # A concurrent failure: the first input has no cached output, the others do. + # Private state on purpose: this test is about how head reads that gap. + del stream._cached_output_datas[0] + unsorted_head = stream.head(10) + sorted_head = stream.head(10, columns={"sort_by_tags": True}) + assert len(unsorted_head) == 0 + assert sorted_head.table.column("id").to_pylist() == [0, 2, 7] + + +def test_a_custom_hash_column_named_like_a_system_tag_is_kept() -> None: + """A function pod stream keeps a requested hash column whatever its name starts with.""" + from orcapod.system_constants import constants + + name = f"{constants.SYSTEM_TAG_PREFIX}digest" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + config = {"content_hash": name, "sort_by_tags": True} + table = stream.as_table(columns=config) + assert name in table.column_names + assert stream.head(3, columns=config).table.equals(table.slice(0, 3)) From 95a4f522999a8621d57b9a4766ae71d6406edb4d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:09:54 -0800 Subject: [PATCH 10/16] test(hashing): pin the identity hash of a tagless producerless stream (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_hashing/golden/cases.py | 28 ++++++++++++++++++++ tests/test_hashing/golden/data/pipeline.json | 4 ++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index 14cb31181..515507845 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -1884,3 +1884,31 @@ def torch_datagram() -> dict[str, Any]: return stream_summary( DictSource(data=rows, tag_columns=["k"], source_id="golden_torch_tensor") ) + + +@case("pipeline", "tagless_producerless_stream") +def pipeline_tagless_producerless_stream() -> dict[str, Any]: + """Pin the content hash of a stream with no tag column and no producer (ITL-1053). + + Its identity hashes the rows sorted by their system tags, so the hash is the same + for rows given in any order. + + Returns: + The content hash for two orders of the same rows. + """ + from orcapod.core.streams import ArrowTableStream + + rows = {"record_id": ["b", "c", "a"], "x": [1, 2, 3]} + reordered = {"record_id": ["c", "a", "b"], "x": [2, 3, 1]} + return { + "as_given": ArrowTableStream( + pa.table(rows), tag_columns=[], system_tag_columns=["record_id"] + ) + .content_hash() + .to_string(), + "reordered": ArrowTableStream( + pa.table(reordered), tag_columns=[], system_tag_columns=["record_id"] + ) + .content_hash() + .to_string(), + } diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index b08876711..eb5b9a832 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -591,5 +591,7 @@ "sources.subjects.system_tag_columns": [ "_tag_source_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5", "_tag_record_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5" - ] + ], + "tagless_producerless_stream.as_given": "semantic_v1:8910e1f5088e3a86be550c356c8127e4dcd302c99378942f30bc34a982b1a108", + "tagless_producerless_stream.reordered": "semantic_v1:8910e1f5088e3a86be550c356c8127e4dcd302c99378942f30bc34a982b1a108" } From e1f3508da0c0ea2c3d743125fa5f697e433250ce Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:10:56 -0800 Subject: [PATCH 11/16] test(hashing): type the tagless stream golden case (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_hashing/golden/cases.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index 515507845..dec331810 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -1898,16 +1898,22 @@ def pipeline_tagless_producerless_stream() -> dict[str, Any]: """ from orcapod.core.streams import ArrowTableStream - rows = {"record_id": ["b", "c", "a"], "x": [1, 2, 3]} - reordered = {"record_id": ["c", "a", "b"], "x": [2, 3, 1]} + def table(record_ids: list[str], xs: list[int]) -> pa.Table: + return pa.table( + [pa.array(record_ids, pa.string()), pa.array(xs, pa.int64())], + names=["record_id", "x"], + ) + + rows = table(["b", "c", "a"], [1, 2, 3]) + reordered = table(["c", "a", "b"], [2, 3, 1]) return { "as_given": ArrowTableStream( - pa.table(rows), tag_columns=[], system_tag_columns=["record_id"] + rows, tag_columns=[], system_tag_columns=["record_id"] ) .content_hash() .to_string(), "reordered": ArrowTableStream( - pa.table(reordered), tag_columns=[], system_tag_columns=["record_id"] + reordered, tag_columns=[], system_tag_columns=["record_id"] ) .content_hash() .to_string(), From 6710b2f1f96f650845a0def827e7ddd5d487aa4f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:22:44 -0800 Subject: [PATCH 12/16] fix(streams): system tag priority follows the stream's order; hash carried under a temporary name (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- src/orcapod/core/function_pod.py | 32 ++++++++++----- src/orcapod/core/nodes/function_node.py | 36 ++++++++-------- .../core/streams/arrow_table_stream.py | 4 +- src/orcapod/utils/arrow_utils.py | 34 +++++++++++---- tests/test_core/streams/test_head_sorting.py | 41 +++++++++++++++++++ 5 files changed, 111 insertions(+), 36 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 7b6e6d2d3..588c2b4bd 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1522,30 +1522,40 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``; called only when ``column_config`` asks for them. """ - hash_column_name = None - if column_config.content_hash: - hash_column_name = ( - "_content_hash" - if column_config.content_hash is True - else column_config.content_hash - ) - table = table.append_column(hash_column_name, content_hashes()) + hashes = None if column_config.sort_by_tags and self.keys()[0]: + # The hash travels with its row under a temporary name that no column has: + # the requested name may be one of the columns the shaping drops or keeps. + carrier = arrow_utils.unused_column_name(table, "__row_content_hash") + if column_config.content_hash: + table = table.append_column(carrier, content_hashes()) # Before the system tags are dropped: they break ties between equal tags. table = arrow_utils.sort_by_tag_columns(table, self.keys()[0]) + if column_config.content_hash: + hashes = table.column(carrier) + table = table.drop_columns([carrier]) + elif column_config.content_hash: + hashes = content_hashes() drop_columns = [] if not column_config.system_tags: - # The requested hash column stays, whatever its name starts with. drop_columns.extend( c for c in table.column_names - if c.startswith(constants.SYSTEM_TAG_PREFIX) and c != hash_column_name + if c.startswith(constants.SYSTEM_TAG_PREFIX) ) if not column_config.source: drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) if not column_config.context: drop_columns.append(constants.CONTEXT_KEY) - return table.drop([c for c in drop_columns if c in table.column_names]) + shaped = table.drop([c for c in drop_columns if c in table.column_names]) + if hashes is not None: + name = ( + column_config.content_hash + if isinstance(column_config.content_hash, str) + else "_content_hash" + ) + shaped = shaped.append_column(name, hashes) + return shaped class CallableWithPodProtocol(Protocol): diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index df517a236..8a8ec6ae4 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -624,28 +624,30 @@ def _shape_table( order of ``table``; called only when ``column_config`` asks for them. """ tag_keys = self.keys()[0] - hash_column_name = None - if column_config.content_hash: - hash_column_name = ( - "_content_hash" - if column_config.content_hash is True - else column_config.content_hash - ) - table = table.append_column(hash_column_name, content_hashes()) + hashes = None if column_config.sort_by_tags and tag_keys: - # The sort sees the hash column (it stays with its row) and the system tags - # (they break ties between equal tags). + # The hash travels with its row under a temporary name that no column has: + # the requested name may be one of the columns the shaping drops or keeps. + carrier = arrow_utils.unused_column_name(table, "__row_content_hash") + if column_config.content_hash: + table = table.append_column(carrier, content_hashes()) + # The sort sees the hash column and the system tags (they break ties). table = arrow_utils.sort_by_tag_columns(table, tag_keys) - hashes = None - if hash_column_name is not None: - # Set the hash column aside: a custom name could look like a meta column. - hashes = table.column(hash_column_name) - table = table.drop_columns([hash_column_name]) + if column_config.content_hash: + hashes = table.column(carrier) + table = table.drop_columns([carrier]) + elif column_config.content_hash: + hashes = content_hashes() shaped = arrow_utils.apply_column_config( table, dataclasses.replace(column_config, sort_by_tags=False), tag_keys ) - if hash_column_name is not None and hashes is not None: - shaped = shaped.append_column(hash_column_name, hashes) + if hashes is not None: + name = ( + column_config.content_hash + if isinstance(column_config.content_hash, str) + else "_content_hash" + ) + shaped = shaped.append_column(name, hashes) return shaped def head( diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 4a0d274a0..3ed7d183c 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -304,7 +304,9 @@ def _assemble_table( # for (its identity, ``identity_structure``, depends on that order); the # system tags stay in the table through the sort. # With the system tags still in the table: they break ties between equal tags. - table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) + table = arrow_utils.sort_by_tag_columns( + table, tuple(self._tag_columns), self._system_tag_columns + ) if not columns_config.system_tags: table = table.drop_columns(list(self._system_tag_columns)) return table diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index 2907021b6..f58b82418 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1837,12 +1837,25 @@ def apply_column_config( return table.drop([c for c in drop_columns if c in table.column_names]) -def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Table": +def unused_column_name(table: "pa.Table", base: str) -> str: + """Return ``base``, with underscores added until no column of ``table`` has that name.""" + name = base + while name in table.column_names: + name += "_" + return name + + +def sort_by_tag_columns( + table: "pa.Table", + tag_keys: tuple[str, ...], + system_tag_keys: tuple[str, ...] | None = None, +) -> "pa.Table": """Return ``table`` with its rows in tag order: the one sort every tag sort uses. The rows are ordered ascending by the tag columns ``tag_keys``, nulls last. Rows with - equal tags are then ordered by the system tag columns ``table`` has (they hold the - record ids), in column order, and rows still equal keep their input order, so the + equal tags are then ordered by the system tag columns ``system_tag_keys`` (default: + those ``table`` has, in column order; they hold the record ids), and rows still equal + keep their input order, so the result does not depend on the order the rows arrived in whenever the table has system tags. A table with neither is returned as it is. A caller that must keep the order of a stream with no regular tag column does not call this for it. @@ -1854,14 +1867,21 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab Args: table: The table to sort. tag_keys: Names of the regular (non-system) tag columns, in sort priority. + system_tag_keys: Names of the system tag columns to break ties with, in priority; + names the table does not have are ignored. ``None`` means every system tag + column of ``table`` in column order. Returns: A new table with the same columns, rows in tag order. """ - system_tags = [ - c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) - ] - keys = [*tag_keys, *(c for c in system_tags if c not in tag_keys)] + if system_tag_keys is None: + system_tag_keys = tuple( + c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) + ) + present = set(table.column_names) + keys = list( + dict.fromkeys([*tag_keys, *(c for c in system_tag_keys if c in present)]) + ) if not keys: return table try: diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 7cd4e19c7..add61441f 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -365,3 +365,44 @@ def test_a_custom_hash_column_named_like_a_system_tag_is_kept() -> None: table = stream.as_table(columns=config) assert name in table.column_names assert stream.head(3, columns=config).table.equals(table.slice(0, 3)) + + +def test_system_tag_priority_follows_the_streams_own_order() -> None: + """Ties between equal tags use the system tags in the stream's own column order. + + ``system_tag_columns`` given as ``["s2", "s1"]`` for a table that stores ``s1`` first: + the identity order of a tagless stream sorts by ``s2`` first, as before (ITL-1053). + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table( + [ + pa.array(["a", "z"], pa.string()), + pa.array(["z", "a"], pa.string()), + pa.array([1, 2], pa.int64()), + ], + names=["s1", "s2", "x"], + ) + stream = ArrowTableStream(table, tag_columns=[], system_tag_columns=["s2", "s1"]) + ordered = stream.as_table(columns={"system_tags": True, "sort_by_tags": True}) + assert ordered.column("x").to_pylist() == [2, 1] + + +@pytest.mark.parametrize("name", ["_context_key", "_source_y", "_tag_digest"]) +def test_a_hash_column_named_like_another_column_group_is_kept(name: str) -> None: + """A requested hash name equal to a column the shaping drops or keeps still works.""" + from orcapod import PipelineJob + from orcapod.databases import InMemoryArrowDatabase + + job = PipelineJob(name="names", store=InMemoryArrowDatabase()) + with job: + pod_stream = triple.pod(_source(), label="node") + assert isinstance(pod_stream, FunctionPodStream) + job.run() + for target in (pod_stream, job.nodes["node"]): + for sort in (False, True): + config = {"content_hash": name, "sort_by_tags": sort} + table = target.as_table(columns=config) + assert table.column_names.count(name) == 1 + head = target.head(3, columns=config) + assert head.table.equals(table.slice(0, 3)) From 6c7128ff9375c9b94dc0b7c6126f597cd98e70d2 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:34:17 -0800 Subject: [PATCH 13/16] fix(streams): an ArrowTableStream keeps the order of tags Arrow cannot sort, as before (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- .../core/streams/arrow_table_stream.py | 7 +++++- src/orcapod/utils/arrow_utils.py | 6 +++++ tests/test_core/streams/test_head_sorting.py | 16 ++++++++++++ tests/test_hashing/golden/cases.py | 25 +++++++++++++++++++ tests/test_hashing/golden/data/pipeline.json | 2 ++ 5 files changed, 55 insertions(+), 1 deletion(-) diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 3ed7d183c..bb7b6c34a 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -304,8 +304,13 @@ def _assemble_table( # for (its identity, ``identity_structure``, depends on that order); the # system tags stay in the table through the sort. # With the system tags still in the table: they break ties between equal tags. + # A tag type Arrow cannot order (a list) keeps the stream's order: the + # identity of the stream hashes this table and must not move. table = arrow_utils.sort_by_tag_columns( - table, tuple(self._tag_columns), self._system_tag_columns + table, + tuple(self._tag_columns), + self._system_tag_columns, + order_unsortable=False, ) if not columns_config.system_tags: table = table.drop_columns(list(self._system_tag_columns)) diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index f58b82418..ad8890d03 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1849,6 +1849,7 @@ def sort_by_tag_columns( table: "pa.Table", tag_keys: tuple[str, ...], system_tag_keys: tuple[str, ...] | None = None, + order_unsortable: bool = True, ) -> "pa.Table": """Return ``table`` with its rows in tag order: the one sort every tag sort uses. @@ -1870,6 +1871,9 @@ def sort_by_tag_columns( system_tag_keys: Names of the system tag columns to break ties with, in priority; names the table does not have are ignored. ``None`` means every system tag column of ``table`` in column order. + order_unsortable: What to do with a type Arrow cannot order (a list tag): + ``True`` orders it with Polars (nulls last, input order among equals); + ``False`` returns ``table`` in its input order. Returns: A new table with the same columns, rows in tag order. @@ -1887,6 +1891,8 @@ def sort_by_tag_columns( try: return table.sort_by([(key, "ascending") for key in keys]) except (pa.ArrowNotImplementedError, pa.ArrowTypeError): + if not order_unsortable: + return table # Arrow cannot order this type (a list tag): order by Polars, nulls last, and # keep the input order of equal rows. import polars as pl diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index add61441f..597a2a317 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -406,3 +406,19 @@ def test_a_hash_column_named_like_another_column_group_is_kept(name: str) -> Non assert table.column_names.count(name) == 1 head = target.head(3, columns=config) assert head.table.equals(table.slice(0, 3)) + + +def test_a_stream_with_list_tags_keeps_its_order_and_its_identity() -> None: + """Arrow cannot order a list tag: an ``ArrowTableStream`` keeps its rows' order. + + Its identity hashes that table, so ordering it would move the content hash (ITL-1053). + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table( + [pa.array([[2], [1]], pa.large_list(pa.int64())), pa.array([20, 10])], + names=["k", "x"], + ) + stream = ArrowTableStream(table, tag_columns=["k"]) + for config in ({"sort_by_tags": True}, {"system_tags": True, "sort_by_tags": True}): + assert stream.as_table(columns=config).column("x").to_pylist() == [20, 10] diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index dec331810..c2a0c3129 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -1918,3 +1918,28 @@ def table(record_ids: list[str], xs: list[int]) -> pa.Table: .content_hash() .to_string(), } + + +@case("pipeline", "list_tag_producerless_stream") +def pipeline_list_tag_producerless_stream() -> dict[str, Any]: + """Pin the content hash of a stream with a list-valued tag and no producer (ITL-1053). + + Arrow cannot order a list column, so the stream's identity hashes its rows in the + order given: the hash of the same rows in two orders differs. + + Returns: + The content hash for two orders of the same rows. + """ + from orcapod.core.streams import ArrowTableStream + + def stream(ks: list[list[int]], xs: list[int]) -> ArrowTableStream: + table = pa.table( + [pa.array(ks, pa.large_list(pa.int64())), pa.array(xs, pa.int64())], + names=["k", "x"], + ) + return ArrowTableStream(table, tag_columns=["k"]) + + return { + "as_given": stream([[2], [1]], [20, 10]).content_hash().to_string(), + "reversed": stream([[1], [2]], [10, 20]).content_hash().to_string(), + } diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index eb5b9a832..7ee2faca9 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -367,6 +367,8 @@ "_tag_source_id::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c:1", "_tag_record_id::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c:1" ], + "list_tag_producerless_stream.as_given": "semantic_v1:13bcfc176f286f93f6e726f100d484889b970f27ab9faa68e815d126e33f9ad9", + "list_tag_producerless_stream.reversed": "semantic_v1:9bb1d6d7a8958dfb76562fa48854acff9b95b72a735631dce0b2cf583aae53a3", "map_data.content_hash": "semantic_v1:7ed964c988b138e57d33b6db445eb9322babdd11dd9d16ef09b7a618efc16b4b", "map_data.pipeline_hash": "semantic_v1:e7b348a99246e0f719402540be0b77865bf2c0fe8de25e3355b707f1aa3fe2d7", "map_data.row[{\"subject\":\"s1\"}]._tag_record_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5": "e8e02084627f5e1aad83839e421538e8", From 4a97fe0d8a06a70c476fa2c9cf7296958521bd13 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:42:09 -0800 Subject: [PATCH 14/16] refactor(streams): a stream's identity keeps main's row order in its own method (ITL-1053) The new deterministic tag sort is a presentation sort only. identity_structure hashes ArrowTableStream._identity_table, the ordering code origin/main had; a regression test holds a literal copy of it. Co-Authored-By: Claude Sonnet 5.5 --- .../core/streams/arrow_table_stream.py | 47 ++++---- src/orcapod/utils/arrow_utils.py | 41 +++---- tests/test_core/streams/test_head_sorting.py | 38 +++---- .../streams/test_identity_order_regression.py | 100 ++++++++++++++++++ 4 files changed, 152 insertions(+), 74 deletions(-) create mode 100644 tests/test_core/streams/test_identity_order_regression.py diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index bb7b6c34a..f4da783a9 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -150,18 +150,32 @@ def identity_structure(self) -> Any: # hash label, which must never enter a hash input. return ( self.__class__.__name__, - self.as_table( - columns=ColumnConfig( - meta=True, - context=True, - source=True, - system_tags=True, - sort_by_tags=True, - ) - ), + self._identity_table(), self._tag_columns, ) + def _identity_table(self) -> "pa.Table": + """Return the rows in the order and columns that ``identity_structure`` hashes. + + THIS ORDER IS A HASH INPUT AND MUST NOT CHANGE: it is the ordering that every + stream's content hash has been built from since before ITL-1053. It is kept apart + from ``as_table``'s sort on purpose; the sort in the body is the code + ``as_table(sort_by_tags=True)`` had on origin/main, unchanged: the tags and + system tags in this stream's own priority, a type Arrow cannot order leaving the + rows in stream order, and a stream with nothing to sort by raising. + """ + table = self._assemble_table( + ColumnConfig(meta=True, context=True, source=True, system_tags=True), + limit=None, + ) + try: + return table.sort_by( + [(column, "ascending") for column in self._all_tag_columns] + ) + except pa.ArrowTypeError: + # If sorting fails, fall back to unsorted table + return table + def pipeline_identity_structure(self) -> Any: if self._producer is None or not isinstance( self._producer, PipelineElementProtocol @@ -300,18 +314,11 @@ def _assemble_table( if columns_config.sort_by_tags and ( self._tag_columns or columns_config.system_tags ): + # The presentation sort (not the identity order, see ``_identity_table``). # A stream with no tag column is sorted only when the system tags are asked - # for (its identity, ``identity_structure``, depends on that order); the - # system tags stay in the table through the sort. - # With the system tags still in the table: they break ties between equal tags. - # A tag type Arrow cannot order (a list) keeps the stream's order: the - # identity of the stream hashes this table and must not move. - table = arrow_utils.sort_by_tag_columns( - table, - tuple(self._tag_columns), - self._system_tag_columns, - order_unsortable=False, - ) + # for. The system tags stay in the table through the sort: they break ties + # between equal tags. + table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) if not columns_config.system_tags: table = table.drop_columns(list(self._system_tag_columns)) return table diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index ad8890d03..903328622 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -1845,21 +1845,20 @@ def unused_column_name(table: "pa.Table", base: str) -> str: return name -def sort_by_tag_columns( - table: "pa.Table", - tag_keys: tuple[str, ...], - system_tag_keys: tuple[str, ...] | None = None, - order_unsortable: bool = True, -) -> "pa.Table": - """Return ``table`` with its rows in tag order: the one sort every tag sort uses. +def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Table": + """Return ``table`` with its rows in tag order: the sort every presentation uses. + + This is the order of ``as_table(sort_by_tags=True)``, ``head`` and the notebook + preview. It is not a hash input: a stream's identity keeps its own ordering + (``ArrowTableStream._identity_table``). The rows are ordered ascending by the tag columns ``tag_keys``, nulls last. Rows with - equal tags are then ordered by the system tag columns ``system_tag_keys`` (default: - those ``table`` has, in column order; they hold the record ids), and rows still equal - keep their input order, so the + equal tags are then ordered by the system tag columns ``table`` has (they hold the + record ids), in column order, and rows still equal keep their input order, so the result does not depend on the order the rows arrived in whenever the table has system tags. A table with neither is returned as it is. A caller that must keep the order of - a stream with no regular tag column does not call this for it. + a stream with no regular tag column does not call this for it. A tag type Arrow cannot + order (a list) is ordered with Polars, nulls last. Whole rows move together: a column added to ``table`` before the sort (a content hash) stays with its row. Column types and nullability are those of ``table``. Sort before @@ -1868,33 +1867,19 @@ def sort_by_tag_columns( Args: table: The table to sort. tag_keys: Names of the regular (non-system) tag columns, in sort priority. - system_tag_keys: Names of the system tag columns to break ties with, in priority; - names the table does not have are ignored. ``None`` means every system tag - column of ``table`` in column order. - order_unsortable: What to do with a type Arrow cannot order (a list tag): - ``True`` orders it with Polars (nulls last, input order among equals); - ``False`` returns ``table`` in its input order. Returns: A new table with the same columns, rows in tag order. """ - if system_tag_keys is None: - system_tag_keys = tuple( - c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) - ) - present = set(table.column_names) - keys = list( - dict.fromkeys([*tag_keys, *(c for c in system_tag_keys if c in present)]) + system_tags = ( + c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ) + keys = list(dict.fromkeys([*tag_keys, *system_tags])) if not keys: return table try: return table.sort_by([(key, "ascending") for key in keys]) except (pa.ArrowNotImplementedError, pa.ArrowTypeError): - if not order_unsortable: - return table - # Arrow cannot order this type (a list tag): order by Polars, nulls last, and - # keep the input order of equal rows. import polars as pl sorted_table = ( diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 597a2a317..960a0a539 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -367,27 +367,6 @@ def test_a_custom_hash_column_named_like_a_system_tag_is_kept() -> None: assert stream.head(3, columns=config).table.equals(table.slice(0, 3)) -def test_system_tag_priority_follows_the_streams_own_order() -> None: - """Ties between equal tags use the system tags in the stream's own column order. - - ``system_tag_columns`` given as ``["s2", "s1"]`` for a table that stores ``s1`` first: - the identity order of a tagless stream sorts by ``s2`` first, as before (ITL-1053). - """ - from orcapod.core.streams import ArrowTableStream - - table = pa.table( - [ - pa.array(["a", "z"], pa.string()), - pa.array(["z", "a"], pa.string()), - pa.array([1, 2], pa.int64()), - ], - names=["s1", "s2", "x"], - ) - stream = ArrowTableStream(table, tag_columns=[], system_tag_columns=["s2", "s1"]) - ordered = stream.as_table(columns={"system_tags": True, "sort_by_tags": True}) - assert ordered.column("x").to_pylist() == [2, 1] - - @pytest.mark.parametrize("name", ["_context_key", "_source_y", "_tag_digest"]) def test_a_hash_column_named_like_another_column_group_is_kept(name: str) -> None: """A requested hash name equal to a column the shaping drops or keeps still works.""" @@ -408,10 +387,13 @@ def test_a_hash_column_named_like_another_column_group_is_kept(name: str) -> Non assert head.table.equals(table.slice(0, 3)) -def test_a_stream_with_list_tags_keeps_its_order_and_its_identity() -> None: - """Arrow cannot order a list tag: an ``ArrowTableStream`` keeps its rows' order. +def test_the_presentation_sort_orders_a_list_tag_while_identity_keeps_stream_order() -> ( + None +): + """``as_table(sort_by_tags)`` orders list tags (Polars); the identity table does not. - Its identity hashes that table, so ordering it would move the content hash (ITL-1053). + The identity of a stream hashes its own table, in the order main always gave it + (``ArrowTableStream._identity_table``); only presentations use the new sort. """ from orcapod.core.streams import ArrowTableStream @@ -420,5 +402,9 @@ def test_a_stream_with_list_tags_keeps_its_order_and_its_identity() -> None: names=["k", "x"], ) stream = ArrowTableStream(table, tag_columns=["k"]) - for config in ({"sort_by_tags": True}, {"system_tags": True, "sort_by_tags": True}): - assert stream.as_table(columns=config).column("x").to_pylist() == [20, 10] + assert stream.as_table(columns={"sort_by_tags": True}).column("x").to_pylist() == [ + 10, + 20, + ] + identity = stream.identity_structure()[1] + assert identity.column("x").to_pylist() == [20, 10] diff --git a/tests/test_core/streams/test_identity_order_regression.py b/tests/test_core/streams/test_identity_order_regression.py new file mode 100644 index 000000000..6f4b5b3a5 --- /dev/null +++ b/tests/test_core/streams/test_identity_order_regression.py @@ -0,0 +1,100 @@ +"""The row order an ``ArrowTableStream`` hashes for its identity never changes (ITL-1053). + +``identity_structure`` of a stream with no producer hashes its table, so the order of its +rows is a hash input. ITL-1053 gave ``as_table``, ``head`` and the preview a new +deterministic sort; the identity keeps the ordering code it had before. This test holds a +literal copy of that code and compares it with the stream's identity table over a battery +of shapes: tag and system tag columns, system tags given in either order, nulls, tied +tags and list tags. +""" + +from __future__ import annotations + +import itertools +import random +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod.core.streams import ArrowTableStream +from orcapod.types import ColumnConfig + + +def main_ordering(stream: ArrowTableStream, table: pa.Table) -> pa.Table: + """The sort block of ``ArrowTableStream._assemble_table`` on origin/main. + + Copied literally from origin/main at commit 89160c47abf149c8160496de2cdfebe7d651e448 + (``src/orcapod/core/streams/arrow_table_stream.py``, last changed in f51ce67c), with + ``columns_config.system_tags`` set to ``True``, which is what ``identity_structure`` + asked for, and ``columns_config.sort_by_tags`` true. Do not edit it. + """ + # --- literal copy below; ``self`` is ``stream``, ``columns_config.system_tags`` is True + self = stream + try: + target_tags = self._all_tag_columns # pyright: ignore[reportPrivateUsage] + return table.sort_by([(column, "ascending") for column in target_tags]) + except pa.ArrowTypeError: + # If sorting fails, fall back to unsorted table + return table + + +def shapes() -> list[tuple[Any, ...]]: + """Every shape of the battery: tag columns, system tag order, nulls, ties, list tags.""" + return list( + itertools.product( + [(), ("k",), ("k", "j")], + [("s1", "s2"), ("s2", "s1"), ("s1",), ()], + [False, True], + [False, True], + [False, True], + ) + ) + + +def build(shape: tuple[Any, ...]) -> ArrowTableStream: + """A six-row stream of the given shape; random but seeded values.""" + tags, system, nulls, ties, list_tag = shape + rng = random.Random(7) + n = 6 + k: list[Any] = [rng.choice([0, 1, 2]) if ties else i for i in range(n)] + j: list[Any] = [rng.choice(["a", "b"]) for _ in range(n)] + if nulls: + k[1] = None + j[3] = None + columns: dict[str, pa.Array[Any]] = { + "k": pa.array( + [None if v is None else [v] for v in k], pa.large_list(pa.int64()) + ) + if list_tag + else pa.array(k, pa.int64()), + "j": pa.array(j, pa.string()), + "s1": pa.array([rng.choice(["p", "q", "r"]) for _ in range(n)], pa.string()), + "s2": pa.array([rng.choice(["u", "v"]) for _ in range(n)], pa.string()), + "x": pa.array(range(n), pa.int64()), + } + names = [c for c in columns if c in tags or c in system or c == "x"] + table = pa.table([columns[c] for c in names], names=names) + return ArrowTableStream(table, tag_columns=tags, system_tag_columns=system) + + +@pytest.mark.parametrize("shape", shapes()) +def test_the_identity_table_has_the_row_order_main_gave_it( + shape: tuple[Any, ...], +) -> None: + """``identity_structure``'s table is main's ordering of the same columns, or both raise.""" + stream = build(shape) + unsorted = stream.as_table( + columns=ColumnConfig(meta=True, context=True, source=True, system_tags=True) + ) + try: + expected = main_ordering(stream, unsorted) + except pa.ArrowInvalid: + # Main raises when there is nothing to sort by: so does the identity. + with pytest.raises(pa.ArrowInvalid): + stream.identity_structure() + return + name, table, tag_columns = stream.identity_structure() + assert name == "ArrowTableStream" + assert table.equals(expected) + assert tag_columns == tuple(c for c in shape[0]) From 4a72158d2ae009c9fcf5906cf2c422d9ceaeca7c Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:51:11 -0800 Subject: [PATCH 15/16] fix(streams): a requested hash column name never acts as a system tag in the sort (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- .../core/streams/arrow_table_stream.py | 18 +++++++++++++---- tests/test_core/streams/test_head_sorting.py | 20 +++++++++++++++++++ 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index f4da783a9..ecdd13d6b 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -289,6 +289,7 @@ def _assemble_table( output_table = output_table.slice(0, limit) data_context_table = data_context_table.slice(0, limit) source_info_table = source_info_table.slice(0, limit) + carrier = hash_column_name = None if columns_config.content_hash: hash_column_name = ( "_content_hash" @@ -299,8 +300,12 @@ def _assemble_table( if limit is not None: rows = itertools.islice(rows, limit) content_hashes = [str(data.content_hash()) for _, data in rows] + # The hash travels with its row under a temporary name that is no column + # group's (a requested name such as ``_tag_digest`` must not be taken for a + # system tag by the sort); it gets its requested name after the sort. + carrier = arrow_utils.unused_column_name(output_table, "__row_content_hash") output_table = output_table.append_column( - pa.field(hash_column_name, pa.large_string(), nullable=False), + pa.field(carrier, pa.large_string(), nullable=False), pa.array(content_hashes, type=pa.large_string()), ) table_stack = (output_table,) @@ -310,10 +315,11 @@ def _assemble_table( table_stack += (source_info_table,) table = arrow_utils.hstack_tables(*table_stack) - - if columns_config.sort_by_tags and ( + sort_this = columns_config.sort_by_tags and ( self._tag_columns or columns_config.system_tags - ): + ) + + if sort_this: # The presentation sort (not the identity order, see ``_identity_table``). # A stream with no tag column is sorted only when the system tags are asked # for. The system tags stay in the table through the sort: they break ties @@ -321,6 +327,10 @@ def _assemble_table( table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) if not columns_config.system_tags: table = table.drop_columns(list(self._system_tag_columns)) + if carrier is not None and hash_column_name is not None: + table = table.rename_columns( + [hash_column_name if c == carrier else c for c in table.column_names] + ) return table def clear_cache(self) -> None: diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py index 960a0a539..cb14a71b6 100644 --- a/tests/test_core/streams/test_head_sorting.py +++ b/tests/test_core/streams/test_head_sorting.py @@ -408,3 +408,23 @@ def test_the_presentation_sort_orders_a_list_tag_while_identity_keeps_stream_ord ] identity = stream.identity_structure()[1] assert identity.column("x").to_pylist() == [20, 10] + + +def test_a_hash_column_named_like_a_system_tag_does_not_reorder_tied_rows() -> None: + """On an ``ArrowTableStream`` a requested ``_tag_digest`` is not a sort key. + + Rows with equal tags keep their input order whatever the hash column is called. + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table( + [pa.array([1, 1, 1], pa.int64()), pa.array([3, 1, 2], pa.int64())], + names=["k", "x"], + ) + stream = ArrowTableStream(table, tag_columns=["k"]) + for name in (True, "_tag_digest", "digest"): + config = {"content_hash": name, "sort_by_tags": True} + shown = stream.as_table(columns=config) + assert shown.column("x").to_pylist() == [3, 1, 2] + assert shown.column_names[-1] == ("_content_hash" if name is True else name) + assert stream.head(2, columns=config).table.equals(shown.slice(0, 2)) From 282a0e2bc7f8f6e7e512e6b19542a36d6770f60a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Fri, 13 Nov 2026 03:55:05 -0800 Subject: [PATCH 16/16] refactor(streams): one helper carries a requested content-hash column through sorting and filtering (ITL-1053) Co-Authored-By: Claude Sonnet 5.5 --- src/orcapod/core/function_pod.py | 57 +++++++--------- src/orcapod/core/nodes/function_node.py | 33 +++------ .../core/streams/arrow_table_stream.py | 68 ++++++++----------- src/orcapod/utils/arrow_utils.py | 46 ++++++++++++- 4 files changed, 106 insertions(+), 98 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 588c2b4bd..63e5da4f7 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1522,40 +1522,29 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``; called only when ``column_config`` asks for them. """ - hashes = None - if column_config.sort_by_tags and self.keys()[0]: - # The hash travels with its row under a temporary name that no column has: - # the requested name may be one of the columns the shaping drops or keeps. - carrier = arrow_utils.unused_column_name(table, "__row_content_hash") - if column_config.content_hash: - table = table.append_column(carrier, content_hashes()) - # Before the system tags are dropped: they break ties between equal tags. - table = arrow_utils.sort_by_tag_columns(table, self.keys()[0]) - if column_config.content_hash: - hashes = table.column(carrier) - table = table.drop_columns([carrier]) - elif column_config.content_hash: - hashes = content_hashes() - drop_columns = [] - if not column_config.system_tags: - drop_columns.extend( - c - for c in table.column_names - if c.startswith(constants.SYSTEM_TAG_PREFIX) - ) - if not column_config.source: - drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) - if not column_config.context: - drop_columns.append(constants.CONTEXT_KEY) - shaped = table.drop([c for c in drop_columns if c in table.column_names]) - if hashes is not None: - name = ( - column_config.content_hash - if isinstance(column_config.content_hash, str) - else "_content_hash" - ) - shaped = shaped.append_column(name, hashes) - return shaped + + def shape(carried: pa.Table) -> pa.Table: + if column_config.sort_by_tags and self.keys()[0]: + # Before the system tags are dropped: they break ties between equal tags. + carried = arrow_utils.sort_by_tag_columns(carried, self.keys()[0]) + drop_columns = [] + if not column_config.system_tags: + drop_columns.extend( + c + for c in carried.column_names + if c.startswith(constants.SYSTEM_TAG_PREFIX) + ) + if not column_config.source: + drop_columns.extend( + f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1] + ) + if not column_config.context: + drop_columns.append(constants.CONTEXT_KEY) + return carried.drop([c for c in drop_columns if c in carried.column_names]) + + return arrow_utils.with_content_hash_column( + table, column_config.content_hash, content_hashes, shape + ) class CallableWithPodProtocol(Protocol): diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 8a8ec6ae4..0a576e1e1 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -18,7 +18,6 @@ import asyncio import contextlib -import dataclasses import logging import uuid from abc import abstractmethod @@ -624,31 +623,15 @@ def _shape_table( order of ``table``; called only when ``column_config`` asks for them. """ tag_keys = self.keys()[0] - hashes = None - if column_config.sort_by_tags and tag_keys: - # The hash travels with its row under a temporary name that no column has: - # the requested name may be one of the columns the shaping drops or keeps. - carrier = arrow_utils.unused_column_name(table, "__row_content_hash") - if column_config.content_hash: - table = table.append_column(carrier, content_hashes()) - # The sort sees the hash column and the system tags (they break ties). - table = arrow_utils.sort_by_tag_columns(table, tag_keys) - if column_config.content_hash: - hashes = table.column(carrier) - table = table.drop_columns([carrier]) - elif column_config.content_hash: - hashes = content_hashes() - shaped = arrow_utils.apply_column_config( - table, dataclasses.replace(column_config, sort_by_tags=False), tag_keys + # The sort sees the hash column and the system tags (they break ties). + return arrow_utils.with_content_hash_column( + table, + column_config.content_hash, + content_hashes, + lambda carried: arrow_utils.apply_column_config( + carried, column_config, tag_keys + ), ) - if hashes is not None: - name = ( - column_config.content_hash - if isinstance(column_config.content_hash, str) - else "_content_hash" - ) - shaped = shaped.append_column(name, hashes) - return shaped def head( self, diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index ecdd13d6b..e881a88c2 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -289,49 +289,41 @@ def _assemble_table( output_table = output_table.slice(0, limit) data_context_table = data_context_table.slice(0, limit) source_info_table = source_info_table.slice(0, limit) - carrier = hash_column_name = None - if columns_config.content_hash: - hash_column_name = ( - "_content_hash" - if columns_config.content_hash is True - else columns_config.content_hash - ) + + def hashes() -> pa.Array[Any]: rows = self.iter_data() if limit is not None: rows = itertools.islice(rows, limit) - content_hashes = [str(data.content_hash()) for _, data in rows] - # The hash travels with its row under a temporary name that is no column - # group's (a requested name such as ``_tag_digest`` must not be taken for a - # system tag by the sort); it gets its requested name after the sort. - carrier = arrow_utils.unused_column_name(output_table, "__row_content_hash") - output_table = output_table.append_column( - pa.field(carrier, pa.large_string(), nullable=False), - pa.array(content_hashes, type=pa.large_string()), + return pa.array( + [str(data.content_hash()) for _, data in rows], type=pa.large_string() ) - table_stack = (output_table,) - if columns_config.context: - table_stack += (data_context_table,) - if columns_config.source: - table_stack += (source_info_table,) - - table = arrow_utils.hstack_tables(*table_stack) - sort_this = columns_config.sort_by_tags and ( - self._tag_columns or columns_config.system_tags - ) - if sort_this: - # The presentation sort (not the identity order, see ``_identity_table``). - # A stream with no tag column is sorted only when the system tags are asked - # for. The system tags stay in the table through the sort: they break ties - # between equal tags. - table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) - if not columns_config.system_tags: - table = table.drop_columns(list(self._system_tag_columns)) - if carrier is not None and hash_column_name is not None: - table = table.rename_columns( - [hash_column_name if c == carrier else c for c in table.column_names] - ) - return table + def shape(carried: "pa.Table") -> "pa.Table": + table_stack = (carried,) + if columns_config.context: + table_stack += (data_context_table,) + if columns_config.source: + table_stack += (source_info_table,) + table = arrow_utils.hstack_tables(*table_stack) + if columns_config.sort_by_tags and ( + self._tag_columns or columns_config.system_tags + ): + # The presentation sort (not the identity order, see + # ``_identity_table``). A stream with no tag column is sorted only when + # the system tags are asked for. The system tags stay in the table + # through the sort: they break ties between equal tags. + table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) + if not columns_config.system_tags: + table = table.drop_columns(list(self._system_tag_columns)) + return table + + return arrow_utils.with_content_hash_column( + output_table, + columns_config.content_hash, + hashes, + shape, + nullable=False, + ) def clear_cache(self) -> None: """ diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index 903328622..5630eec1a 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -4,7 +4,7 @@ import json import uuid from collections import defaultdict -from collections.abc import Mapping, Collection, Sequence +from collections.abc import Callable, Mapping, Collection, Sequence from typing import Any, TYPE_CHECKING, cast from orcapod.system_constants import constants @@ -1845,6 +1845,50 @@ def unused_column_name(table: "pa.Table", base: str) -> str: return name +def with_content_hash_column( + table: "pa.Table", + content_hash: bool | str, + hashes: "Callable[[], pa.Array[Any]]", + shape: "Callable[[pa.Table], pa.Table]", + nullable: bool = True, +) -> "pa.Table": + """Return ``shape(table)`` with the requested content-hash column attached. + + The ONE mechanism by which a requested content-hash column is added while rows are + sorted or columns are filtered (ITL-1029, ITL-1053). The hash column is attached to + ``table`` before ``shape`` runs, so a sort moves every hash with its own row, and it + travels under a reserved temporary name that belongs to no column group, so that + neither the sort (a name starting with the system tag prefix is not a tie-break key) + nor the filtering (a name equal to ``_context_key`` or ``_source_y``) can mistake or + drop it. It gets its requested name last. + + Args: + table: The rows, with every column ``shape`` needs. + content_hash: The column configuration's ``content_hash``: ``False`` for none, + ``True`` for ``_content_hash``, or the name of the column. + hashes: Returns the content hash of each row of ``table``, in its order, as a + ``large_string`` array. Called only when a content hash is requested. + shape: Sorts and filters the table (it must keep every column it does not drop + by its own rules, and must not read the hash column). + nullable: Whether the hash column's field is nullable. + + Returns: + The shaped table, the content-hash column last under its requested name. + """ + if not content_hash: + return shape(table) + name = content_hash if isinstance(content_hash, str) else "_content_hash" + carrier = unused_column_name(table, "row content hash") + carried = table.append_column( + pa.field(carrier, pa.large_string(), nullable=nullable), + hashes(), + ) + shaped = shape(carried) + return shaped.rename_columns( + [name if column == carrier else column for column in shaped.column_names] + ) + + def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Table": """Return ``table`` with its rows in tag order: the sort every presentation uses.