diff --git a/changelog.d/ITL-1053.md b/changelog.d/ITL-1053.md new file mode 100644 index 000000000..7560182e6 --- /dev/null +++ b/changelog.d/ITL-1053.md @@ -0,0 +1,25 @@ +### Changed + +- `head()` takes the column configuration of `as_table`, sorting included, and applies it to + the rows it has before taking the first `n` (ITL-1053). `head(n, columns={"sort_by_tags": + True})` no longer raises, and `head(n, all_info=True)` now sorts by tag, as `as_table` + does; the default `head()` stays unsorted, as `as_table()` is. The notebook preview asks for the + sorted head, so it shows rows in tag order whatever order they were computed in, and its + header line says "sorted by tags" (for a bounded read, "10 rows read, sorted by tags, of + about N"). A stream that has all + its rows (a job node's stored rows, a computed stream, an in-memory source) gives exactly + `as_table(...).slice(0, n)` for every configuration, whatever order its rows were computed + in. A stream that has read or computed only part of its rows (a `SpiralDBTableSource` + head, a partly computed stream) sorts the rows it has: the rows read, in tag order. + +### Fixed + +- A node's `head(n, all_info=True)` no longer differs from `as_table(all_info=True)`'s first + rows when an async run completed rows out of tag order (the cause of an occasionally + failing test, ITL-1001). +- Every tag sort is now one deterministic sort (`arrow_utils.sort_by_tag_columns`): by the + tag columns, nulls last, rows with equal tags ordered by their system tag columns. A job + node's and a function pod stream's sorted `as_table` used an unstable Polars sort that put + nulls first and left the order of rows with equal tags unspecified; it now orders them as + an `ArrowTableStream` does. Only the order of rows within equal or null tags changes: no + values, record ids or hashes. diff --git a/design_issues/ITL-1053-PV2.md b/design_issues/ITL-1053-PV2.md new file mode 100644 index 000000000..027b76c69 --- /dev/null +++ b/design_issues/ITL-1053-PV2.md @@ -0,0 +1,31 @@ +## `src/orcapod/core/streams/` — notebook preview and `head()` + +### PV2 — `head()` refused `sort_by_tags` and showed rows in completion order +**Status:** resolved +**Severity:** medium +**Issue:** ITL-1053 + +`head()` (ITL-991) was built as "the first rows in stream order": `head_column_config` switched +tag sorting off even under `all_info=True` and raised `ValueError` when `sort_by_tags` was +asked. After an async run a node's stored order is completion order, so its head showed rows +in an order that varied from run to run, and a test asserting that `head(5, all_info=True)` +equals the prefix of `as_table(all_info=True)` failed whenever rows finished out of tag order +(`test_head_keeps_the_tag_arrow_type_of_the_input[async-delta]`, seen on CI; ITL-1001). + +**Fix:** developer decision, November 12, 2026: "make it consistent with as_table". +`head_column_config` is removed; `head` takes `as_table`'s column configuration, sorting +included, and applies it to the rows it has, then takes the first `n`. The default stays +unsorted. A node, an `ArrowTableStream`, a computed `FunctionPodStream` and a `DerivedSource` +over a job node shape and sort every row they have (each row keeps its own content hash, +ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded read +(`SpiralDBTableSource`, a partly computed stream, a polled source's batches) sorts the rows it +has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads +more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the +rows read, sorted" unless the store returns rows in tag order. + +The notebook preview asks for the tag-sorted head (developer, November 12, 2026: "preview should +sort by tags"; `preview_html`), and its header says "sorted by tags", or for a bounded read +"N rows read, sorted by tags, of about T". Every tag sort goes through +`arrow_utils.sort_by_tag_columns`: tags ascending, nulls last, ties by the system tag columns, +then input order (the node and function pod stream sorts were an unstable Polars sort with nulls +first). diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index dc7075718..75155e01a 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -154,12 +154,15 @@ with `max_rows=None`, but it cannot yet be run through a job on a laptop. **Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell (its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10 rows, in one scan to which the row cap does not apply, so a table of any size can be looked -at; `head(n)` reads `n`. The header line says what is shown, for -example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's -recorded row count, read without scanning), `all 4 rows` when the source reads no more than -that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with -`where=`. Tag columns come first, and the line above the table names them. To see more rows, use `as_df()` with a -`limit=` or `where=` source. +at; `head(n)` reads `n`. The preview shows the rows it read in tag order (it asks for +`sort_by_tags`, as does `head(columns={"sort_by_tags": True})` or `all_info=True`): the rows +read, sorted, not the first `n` of the table in tag order. The header line says what is +shown, for example `SpiralDBTableSource[spike_event] — 10 rows read, sorted by tags, of +about 63,349` (the table's recorded row count, read without scanning), `all 4 rows, sorted +by tags` when the source reads no more than that, or `10 matching rows read, sorted by tags +(where unit_id=106); the table has about 63,349 rows` with `where=`. The unsorted `head()` +says `first 10 rows of about 63,349`. Tag columns come first, and the line above the table +names them. To see more rows, use `as_df()` with a `limit=` or `where=` source. **Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the same table, so two sources on one table have the same `source_id` and `content_hash` diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index e26973cae..a6a5dc547 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -103,8 +103,9 @@ Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_l ## Looking at data In a notebook, a stream, source or job node shown at the end of a cell renders a short -table: its first rows, with a line saying what they are. That preview is `head()` rendered, -and `head(n)` gives the same rows to work with. Neither ever computes anything: they show +table: its first rows in tag order, with a line saying what they are. That preview is +`head()` asked for `sort_by_tags` and rendered, and `head(n, columns={"sort_by_tags": True})` +gives the same rows to work with. Neither ever computes anything: they show only what is readily available, and never call a function, run an operator, deliver a side effect, poll a source or write to a database. @@ -162,6 +163,36 @@ What each kind of object shows: | A pipeline blueprint node, an unbound input slot | none (`unavailable`) | | A side-effect stream | none: showing it never delivers its side effects | +`head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`, +sorting included, and applies it to the rows it has before taking the first `n`. The default +does not sort, as `as_table()` does not, so `head()` shows rows in the order the stream +holds them. The notebook preview asks for the sorted head, so it shows the same rows +whatever order they were computed in. With `sort_by_tags=True`, or `all_info=True`, rows are +sorted by tag first (nulls last; rows with equal tags are ordered by their system tags): + +```python +from orcapod.sources import DictSource + +unordered = DictSource( + data=[{"subject_id": f"mouse_{i:02d}", "age": 8 + i} for i in (3, 1, 2)], + tag_columns=["subject_id"], + label="unordered", +) +print(unordered.head(2).as_table().column("subject_id").to_pylist()) +# ['mouse_03', 'mouse_01'] +print(unordered.head(2, all_info=True).as_table().column("subject_id").to_pylist()) +# ['mouse_01', 'mouse_02'] +``` + +A stream that has all of its rows (a job node's stored rows, a computed stream, an in-memory +source) then gives exactly the first `n` rows of `as_table` with the same configuration, +whatever order the rows were computed in. A stream that has only some of its rows sorts +those: a source that reads a bounded number of rows (such as a `SpiralDBTableSource`) shows +the rows it read, in tag order, and a stream computed only in part shows the rows computed so +far in tag order. It is not "the first `n` rows in tag order" of rows it never read, and the +preview's header line says so ("10 rows read, sorted by tags, of about 63,349"). A stream +with no tag columns previews its rows in stream order. + A head of an operator is never the operator applied to the heads of its inputs: a group over the first ten rows has the wrong members. So an uncomputed stream shows its columns and "not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 6f088d98c..7ebad6c68 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -459,9 +459,9 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: yield tag, _without_stored_meta(data) def _computed_prefix( - self, limit: int + self, limit: int, contiguous: bool = True ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: - shown, total = super()._computed_prefix(limit) + shown, total = super()._computed_prefix(limit, contiguous) return [(tag, _without_stored_meta(data)) for tag, data in shown], total def head( diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 9e170ad26..63e5da4f7 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -17,7 +17,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, schema_requires_io, ) from orcapod.core.streams.head import ( @@ -1369,11 +1368,17 @@ def head( the first of the result. After an iteration that stopped part way they are the rows computed so far (``PARTIAL``). Before any computation, or once an input changed, there are none. A ``CachedFunctionPod``'s result store is not looked - up: that needs the input rows' hashes, i.e. reading the input. See - ``StreamBase.head``. + up: that needs the input rows' hashes, i.e. reading the input. With + ``sort_by_tags`` (or ``all_info``) the rows held are sorted by tag and the first + ``n`` of those returned: after a complete computation that is + ``as_table(...).slice(0, n)``; after a partial one, the rows computed so far in + tag order ("the rows it has", not the first ``n`` of the whole result in tag + order). See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) + # Sorting needs every row held, not only the first n: read them all (in memory). + wanted = len(self._cached_output_datas) if config.sort_by_tags else n has_results = bool(self._cached_output_datas) or ( self._cached_output_table is not None ) @@ -1405,7 +1410,9 @@ def head( # A complete computation: its table holds the rows; only the shown rows' # data are read, and only when their content hashes are asked for. def shown_content_hashes() -> pa.Array[Any]: - shown, _ = self._computed_prefix(n) + shown, _ = self._computed_prefix( + wanted, contiguous=not config.sort_by_tags + ) return pa.array( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), @@ -1414,8 +1421,12 @@ def shown_content_hashes() -> pa.Array[Any]: return StreamHead.of_rows( self, self._shape_table( - self._cached_output_table.slice(0, n), config, shown_content_hashes - ), + self._cached_output_table + if config.sort_by_tags + else self._cached_output_table.slice(0, n), + config, + shown_content_hashes, + ).slice(0, n), tag_columns=tag_columns, total_rows=self._cached_output_table.num_rows, origin="computed", @@ -1427,7 +1438,7 @@ def shown_content_hashes() -> pa.Array[Any]: ) if not finished and not self._cached_output_datas: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) - shown, total = self._computed_prefix(n) + shown, total = self._computed_prefix(wanted, contiguous=not config.sort_by_tags) status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL if not shown: # Nothing to show (an empty result, inputs that all gave no output so far, @@ -1454,7 +1465,7 @@ def shown_content_hashes() -> pa.Array[Any]: [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), ), - ), + ).slice(0, n), tag_columns=tag_columns, total_rows=total, status=status, @@ -1462,30 +1473,39 @@ def shown_content_hashes() -> pa.Array[Any]: ) def _computed_prefix( - self, limit: int + self, limit: int, contiguous: bool = True ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: """Return the first ``limit`` output rows computed so far, and how many there are. - The rows are the contiguous run of computed inputs from the first, in the order - iteration yields them; an input whose output is ``None`` (filtered out) is - skipped, as iteration skips it. Only the first ``limit`` rows are collected. + By default the rows are the contiguous run of computed inputs from the first, in + the order iteration yields them; an input whose output is ``None`` (filtered out) + is skipped, as iteration skips it. Only the first ``limit`` rows are collected. + With ``contiguous=False`` the rows are every computed output, in input order, + also those after an input that has no output (a failure): a sorted head sorts the + rows it has. Args: limit: The largest number of rows to return. + contiguous: Stop at the first input with no cached output. Returns: - The rows, and the number of output rows in the whole computed run. + The rows, and the number of output rows counted. """ shown: list[tuple[TagProtocol, DataProtocol]] = [] total = 0 - index = 0 - while index in self._cached_output_datas: + indexes = ( + iter(range(len(self._cached_output_datas) + 1)) + if contiguous + else iter(sorted(self._cached_output_datas)) + ) + for index in indexes: + if index not in self._cached_output_datas: + break tag, data = self._cached_output_datas[index] if data is not None: if total < limit: shown.append((tag, data)) total += 1 - index += 1 return shown, total def _shape_table( @@ -1502,48 +1522,29 @@ def _shape_table( content_hashes: Returns the content hash of each row of ``table``; called only when ``column_config`` asks for them. """ - drop_columns = [] - if not column_config.system_tags: - # TODO: get system tags more effiicently - drop_columns.extend( - [ + + def shape(carried: pa.Table) -> pa.Table: + if column_config.sort_by_tags and self.keys()[0]: + # Before the system tags are dropped: they break ties between equal tags. + carried = arrow_utils.sort_by_tag_columns(carried, self.keys()[0]) + drop_columns = [] + if not column_config.system_tags: + drop_columns.extend( c - for c in table.column_names + for c in carried.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) - ] - ) - if not column_config.source: - drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]) - if not column_config.context: - drop_columns.append(constants.CONTEXT_KEY) - - output_table = table.drop([c for c in drop_columns if c in table.column_names]) - - if column_config.content_hash: - hash_column_name = ( - "_content_hash" - if column_config.content_hash is True - else column_config.content_hash - ) - output_table = output_table.append_column( - hash_column_name, content_hashes() - ) + ) + if not column_config.source: + drop_columns.extend( + f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1] + ) + if not column_config.context: + drop_columns.append(constants.CONTEXT_KEY) + return carried.drop([c for c in drop_columns if c in carried.column_names]) - if column_config.sort_by_tags: - # TODO: reimplement using polars natively - output_table_schema = output_table.schema - output_table = ( - pl.DataFrame(output_table) - .sort(by=self.keys()[0], descending=False) - .to_arrow() - ) - output_table = arrow_utils.restore_schema_nullability( - output_table, output_table_schema - ) - # output_table = output_table.sort_by( - # [(column, "ascending") for column in self.keys()[0]] - # ) - return output_table + return arrow_utils.with_content_hash_column( + table, column_config.content_hash, content_hashes, shape + ) class CallableWithPodProtocol(Protocol): diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index dca13d7d4..0a576e1e1 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -18,7 +18,6 @@ import asyncio import contextlib -import dataclasses import logging import uuid from abc import abstractmethod @@ -33,7 +32,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( BLUEPRINT_NOTE, @@ -625,21 +623,15 @@ def _shape_table( order of ``table``; called only when ``column_config`` asks for them. """ tag_keys = self.keys()[0] - output_table = arrow_utils.apply_column_config( - table, dataclasses.replace(column_config, sort_by_tags=False), tag_keys + # The sort sees the hash column and the system tags (they break ties). + return arrow_utils.with_content_hash_column( + table, + column_config.content_hash, + content_hashes, + lambda carried: arrow_utils.apply_column_config( + carried, column_config, tag_keys + ), ) - if column_config.content_hash: - hash_column_name = ( - "_content_hash" - if column_config.content_hash is True - else column_config.content_hash - ) - output_table = output_table.append_column( - hash_column_name, content_hashes() - ) - if column_config.sort_by_tags: - output_table = arrow_utils.sort_by_tag_columns(output_table, tag_keys) - return output_table def head( self, @@ -651,7 +643,7 @@ def head( """Return no rows: a node of this kind holds no data. See ``StreamBase.head``.""" check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note=BLUEPRINT_NOTE, ) @@ -673,7 +665,11 @@ def _head_of_pairs( (t, d) for t, d in pairs if d is not None and not isinstance(d, EmptyData) ] missing = sum(1 for _, d in pairs if isinstance(d, EmptyData)) - shown = rows[:n] + # With sorting, every row is shaped and sorted (each keeps its own content + # hash), then the first n are taken: the head is ``as_table(config)[:n]``. + # At least one row is shaped even for n == 0, so that the empty head has the + # columns of the rows' own table (the context column among them). + shown = rows if config.sort_by_tags else rows[: max(n, 1)] table = self._shape_table( self._table_from_pairs(shown), config, @@ -681,7 +677,7 @@ def _head_of_pairs( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), ), - ) + ).slice(0, n) note = None if missing: note = ( @@ -3058,7 +3054,10 @@ def head( After a run (or ``execute``) the rows are the node's results in memory, the first of what ``as_table`` returns. Otherwise they are the first of its stored rows: the node's whole stored slice is read and nothing is kept on the node. - With nothing stored, there are no rows. See ``StreamBase.head``. + With nothing stored, there are no rows. The node has all of its rows, so with + ``sort_by_tags`` (or ``all_info``) the head is exactly + ``as_table(...).slice(0, n)``, whatever order the rows were computed in. See + ``StreamBase.head``. Warns: StoreContentWarning: As for ``as_table``. @@ -3066,7 +3065,7 @@ def head( from orcapod.pipeline.serialization import LoadStatus check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) status = self.load_status if status == LoadStatus.UNAVAILABLE: return self._head_without_rows( diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 624be3b4f..c7c357f3d 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -30,7 +30,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( STALE_NOTE, @@ -437,7 +436,7 @@ def head( """Return no rows: a blueprint node holds no data.""" check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note=BLUEPRINT_NOTE, ) @@ -1489,7 +1488,7 @@ def head( ``as_table`` does. A loaded node never raises here. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) if self._cached_output_stream is not None: if not self._loaded and self.is_stale: # run() would recompute; the kept rows are not the current result. diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index eaa888091..e5546f46b 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -21,11 +21,10 @@ HeadStatus, StreamHead, head_of, - render_error_html, + preview_html, ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.errors import SourceSpecMismatchError, UnboundSourceError from orcapod.protocols.core_protocols import DataProtocol, TagProtocol @@ -316,7 +315,7 @@ def head( """ check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note="input slot; bind a source", ) @@ -334,12 +333,8 @@ def _head_without_rows( ) def _repr_html_(self) -> str: - """Render ``head()`` for a notebook; never computes.""" - try: - head = self.head() - except Exception as error: # noqa: BLE001 - a display hook must not raise - return render_error_html(self, error) - return head._repr_html_() + """Render ``head()`` in tag order for a notebook; never computes.""" + return preview_html(self) async def async_iter_data(self): """Asynchronous iterator over (tag, data) pairs. @@ -637,7 +632,7 @@ def head( if self._bound_source is None: check_head_size(n) return self._head_without_rows( - head_column_config(columns, all_info), + ColumnConfig.handle_config(columns, all_info=all_info), HeadStatus.UNAVAILABLE, note="no source bound", ) diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 34ffa81db..081209815 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -15,7 +15,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( STALE_NOTE, @@ -536,7 +535,7 @@ def head( or after an input changed, there are no rows. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) had_result = self._cached_stream is not None self._update_cache_status() if self._cached_stream is not None and self._cached_time is not None: diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 26c25f02a..7c2dd81e3 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -14,7 +14,6 @@ ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, schema_requires_io, ) from orcapod.errors import InputValidationError @@ -477,10 +476,13 @@ def head( The records are the ones this source already read. Otherwise, an origin source gives its own ``head(n)`` (so a polling or caching origin is never polled or written), and an origin job node gives its stored rows, read now and not kept. - An origin job node that has not run gives no rows. + An origin job node that has not run gives no rows. With ``sort_by_tags`` (or + ``all_info``), the rows are sorted by tag as ``as_table`` sorts them: all the + stored rows of an origin job node, but only the rows an origin source reads for + its head (see ``StreamBase.head``). """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) origin_kind = "held" if isinstance(self._origin, RootSource) else "stored" if self._tags_as_data is not None and self._cached_stream is not None: return self._cached_stream.head(n, columns=config).for_stream( @@ -493,6 +495,18 @@ def head( return stream.head(n, columns=config).for_stream(self, origin=origin_kind) if isinstance(self._origin, RootSource): origin_head = self._origin.head(n) + if ( + config.sort_by_tags + and origin_head.status is HeadStatus.ROWS + and not origin_head.total_is_estimate + and origin_head.total_rows is not None + and origin_head.total_rows > n + ): + # The origin has all its rows (its total is exact): sorting needs them + # all, and the system tags this source adds follow the origin's row + # order, so the origin is not sorted first. A bounded origin (an + # estimated total) is sorted among the rows it read. + origin_head = self._origin.head(origin_head.total_rows) if origin_head.status is not HeadStatus.ROWS: if origin_head.table.num_columns == 0: # The origin's columns are not known without polling it. @@ -530,8 +544,11 @@ def head( tag_columns=stream.keys(columns=config)[0], note="nothing stored: its origin has not run", ) - # Only the shown rows are enriched; the total is the whole stored count. - head = self._stream_over(records.slice(0, n)).head(n, columns=config) + # Without sorting only the shown rows are enriched. With sorting every stored + # row is (they are all read already), so the head is as_table cut to n rows. + # The total is the whole stored count. + rows = records if config.sort_by_tags else records.slice(0, n) + head = self._stream_over(rows).head(n, columns=config) return dataclasses.replace( head.for_stream(self, origin="stored"), total_rows=records.num_rows ) diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index b45b0a335..98fe4d80c 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -26,7 +26,6 @@ ) from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.errors import ( CursorInvalidatedError, @@ -472,13 +471,18 @@ def head( or none when the source declared no schema (finding it would mean polling). """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) batches = list(self._batches) # snapshot: the list is append-only if batches: - # Combine only the batches that hold the first n rows; count the rest. + # Combine only the batches that hold the first n rows (every batch when + # sorting); count the rest. sizes = [batch.head(0).total_rows or 0 for batch in batches] needed, covered = 0, 0 - while needed < len(batches) and (covered < n or needed == 0): + # A sorted head orders every polled batch: the first n in tag order can lie + # in any of them. The batches are already in memory; nothing is polled. + while needed < len(batches) and ( + covered < n or needed == 0 or config.sort_by_tags + ): covered += sizes[needed] needed += 1 prefix = batches[0] diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index 6714072c9..803fe20a8 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -18,9 +18,9 @@ StreamHead, head_of, ) +from orcapod.types import ColumnConfig from orcapod.core.streams.base import ( check_head_size, - head_column_config, ) from orcapod.protocols.core_protocols import SourceProtocol from orcapod.types import ContentHash, Schema @@ -31,7 +31,6 @@ import pyarrow as pa from orcapod.protocols.core_protocols import DataProtocol, TagProtocol - from orcapod.types import ColumnConfig class SourceProxy(RootSource): @@ -224,7 +223,7 @@ def head( self._delegate.head(n, columns=columns, all_info=all_info), self ) check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.UNAVAILABLE, diff --git a/src/orcapod/core/sources/spiraldb_table_source.py b/src/orcapod/core/sources/spiraldb_table_source.py index eaf1abf8d..a25f84863 100644 --- a/src/orcapod/core/sources/spiraldb_table_source.py +++ b/src/orcapod/core/sources/spiraldb_table_source.py @@ -44,7 +44,7 @@ from orcapod.core.sources.derived_source import validate_tags_as_data from orcapod.core.sources.stream_builder import SourceStreamBuilder from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import check_head_size, head_column_config +from orcapod.core.streams.base import check_head_size from orcapod.core.streams.head import HeadStatus, StreamHead from orcapod.databases.spiraldb_connector import SpiralDBConnector from orcapod.errors import InputValidationError, SourceTooLargeError @@ -662,19 +662,31 @@ def _head( recorded = min(self._limit, recorded) return _Head(rows, True, recorded, True) - def _head_note(self, head: "_Head") -> str: - """Return the words a notebook display puts after the header: the one place.""" + def _head_note(self, head: "_Head", sorted_by_tags: bool = False) -> str: + """Return the words a notebook display puts after the header: the one place. + + With ``sorted_by_tags`` the rows read are in tag order, which is not the first + rows of the table in tag order, and the words say so. + """ shown = head.rows.num_rows if not head.rows_read: return f"rows not read; the table has about {head.total_rows:,} rows" if not head.total_is_estimate: - return f"all {shown:,} row{'' if shown == 1 else 's'}" + all_rows = f"all {shown:,} row{'' if shown == 1 else 's'}" + return f"{all_rows}, sorted by tags" if sorted_by_tags else all_rows if self._where: condition = ", ".join(f"{c}={v!r}" for c, v in self._where.items()) + read = ( + f"{shown:,} matching rows read, sorted by tags" + if sorted_by_tags + else f"first {shown:,} matching rows" + ) return ( - f"first {shown:,} matching rows (where {condition}); the table has " + f"{read} (where {condition}); the table has " f"about {head.total_rows:,} rows" ) + if sorted_by_tags: + return f"{shown:,} rows read, sorted by tags, of about {head.total_rows:,}" return f"first {shown:,} rows of about {head.total_rows:,}" def head( @@ -693,7 +705,7 @@ def head( preview is this head. See ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) head = self._head(n, columns=config) tag_columns = self.keys(columns=config)[0] if not head.rows_read: @@ -713,7 +725,9 @@ def head( total_rows=head.total_rows, total_is_estimate=head.total_is_estimate, origin="read", - description=self._head_note(head), + description=self._head_note( + head, config.sort_by_tags and bool(self._tag_columns) + ), ) def _display_name(self) -> str: diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 721f59e36..e881a88c2 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -10,7 +10,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import PodProtocol, StreamProtocol, TagProtocol @@ -151,18 +150,32 @@ def identity_structure(self) -> Any: # hash label, which must never enter a hash input. return ( self.__class__.__name__, - self.as_table( - columns=ColumnConfig( - meta=True, - context=True, - source=True, - system_tags=True, - sort_by_tags=True, - ) - ), + self._identity_table(), self._tag_columns, ) + def _identity_table(self) -> "pa.Table": + """Return the rows in the order and columns that ``identity_structure`` hashes. + + THIS ORDER IS A HASH INPUT AND MUST NOT CHANGE: it is the ordering that every + stream's content hash has been built from since before ITL-1053. It is kept apart + from ``as_table``'s sort on purpose; the sort in the body is the code + ``as_table(sort_by_tags=True)`` had on origin/main, unchanged: the tags and + system tags in this stream's own priority, a type Arrow cannot order leaving the + rows in stream order, and a stream with nothing to sort by raising. + """ + table = self._assemble_table( + ColumnConfig(meta=True, context=True, source=True, system_tags=True), + limit=None, + ) + try: + return table.sort_by( + [(column, "ascending") for column in self._all_tag_columns] + ) + except pa.ArrowTypeError: + # If sorting fails, fall back to unsorted table + return table + def pipeline_identity_structure(self) -> Any: if self._producer is None or not isinstance( self._producer, PipelineElementProtocol @@ -241,14 +254,21 @@ def head( ) -> StreamHead: """Return the first ``n`` rows of the table this stream holds. - Only those rows are assembled: content hashes, when asked for, are computed for - them alone. See ``StreamBase.head``. + Without sorting only those rows are assembled: content hashes, when asked for, + are computed for them alone. With ``sort_by_tags`` (or ``all_info``) the rows are + sorted first, as ``as_table`` sorts them, and the first ``n`` of those are + returned, so the head is always ``as_table(...)`` cut to ``n`` rows. See + ``StreamBase.head``. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) + if config.sort_by_tags: + table = self._assemble_table(config, limit=None).slice(0, n) + else: + table = self._assemble_table(config, limit=n) return StreamHead.of_rows( self, - self._assemble_table(config, limit=n), + table, tag_columns=self.keys(columns=config)[0], total_rows=self._table.num_rows, ) @@ -269,45 +289,41 @@ def _assemble_table( output_table = output_table.slice(0, limit) data_context_table = data_context_table.slice(0, limit) source_info_table = source_info_table.slice(0, limit) - if columns_config.content_hash: - hash_column_name = ( - "_content_hash" - if columns_config.content_hash is True - else columns_config.content_hash - ) + + def hashes() -> pa.Array[Any]: rows = self.iter_data() if limit is not None: rows = itertools.islice(rows, limit) - content_hashes = [str(data.content_hash()) for _, data in rows] - output_table = output_table.append_column( - pa.field(hash_column_name, pa.large_string(), nullable=False), - pa.array(content_hashes, type=pa.large_string()), + return pa.array( + [str(data.content_hash()) for _, data in rows], type=pa.large_string() ) - if not columns_config.system_tags: - # Check in original implementation - output_table = output_table.drop_columns(list(self._system_tag_columns)) - table_stack = (output_table,) - if columns_config.context: - table_stack += (data_context_table,) - if columns_config.source: - table_stack += (source_info_table,) - - table = arrow_utils.hstack_tables(*table_stack) - - if columns_config.sort_by_tags: - # TODO: cleanup the sorting tag selection logic - try: - target_tags = ( - self._all_tag_columns - if columns_config.system_tags - else self._tag_columns - ) - return table.sort_by([(column, "ascending") for column in target_tags]) - except pa.ArrowTypeError: - # If sorting fails, fall back to unsorted table - return table - return table + def shape(carried: "pa.Table") -> "pa.Table": + table_stack = (carried,) + if columns_config.context: + table_stack += (data_context_table,) + if columns_config.source: + table_stack += (source_info_table,) + table = arrow_utils.hstack_tables(*table_stack) + if columns_config.sort_by_tags and ( + self._tag_columns or columns_config.system_tags + ): + # The presentation sort (not the identity order, see + # ``_identity_table``). A stream with no tag column is sorted only when + # the system tags are asked for. The system tags stay in the table + # through the sort: they break ties between equal tags. + table = arrow_utils.sort_by_tag_columns(table, tuple(self._tag_columns)) + if not columns_config.system_tags: + table = table.drop_columns(list(self._system_tag_columns)) + return table + + return arrow_utils.with_content_hash_column( + output_table, + columns_config.content_hash, + hashes, + shape, + nullable=False, + ) def clear_cache(self) -> None: """ diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index a5176d318..00788eb01 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -1,6 +1,5 @@ from __future__ import annotations -import dataclasses import logging from abc import abstractmethod from collections.abc import AsyncIterator, Collection, Iterator, Mapping @@ -13,7 +12,7 @@ HeadStatus, StreamHead, default_note, - render_error_html, + preview_html, ) from orcapod.protocols.core_protocols import ( DataProtocol, @@ -68,32 +67,6 @@ def check_head_size(n: int) -> None: raise ValueError(f"head() takes a non-negative number of rows n, got {n}") -def head_column_config( - columns: ColumnConfig | dict[str, Any] | None, all_info: bool -) -> ColumnConfig: - """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. - - The column groups mean what they mean for ``as_table``. ``head`` returns rows in - stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. - - Raises: - ValueError: If ``columns`` asks for ``sort_by_tags``. - """ - if isinstance(columns, ColumnConfig): - asked_to_sort = bool(columns.sort_by_tags) - elif isinstance(columns, Mapping): - asked_to_sort = bool(columns.get("sort_by_tags", False)) - else: - asked_to_sort = False - if asked_to_sort: - raise ValueError( - "head() returns the first rows in stream order and cannot sort them by " - "tags; sort the result of as_table() instead" - ) - config = ColumnConfig.handle_config(columns, all_info=all_info) - return dataclasses.replace(config, sort_by_tags=False) - - class StreamBase(TraceableBase): @property @abstractmethod @@ -514,8 +487,16 @@ def head( The rows are ones the stream already holds, has cached from an earlier computation, has stored, or can read with a bounded read of its table. ``head`` never calls a function, runs an operator, delivers a side effect, polls a source - or writes anything. When rows are shown, they are the first ``n`` rows that - ``as_table()`` would return. The notebook preview of a stream is its ``head()``. + or writes anything. ``head`` takes the column configuration of ``as_table``, + sorting included, and applies it to the rows the stream has, then takes the first + ``n``. The default configuration does not sort, as ``as_table()`` does not, so + the rows are the first ``n`` that ``as_table()`` would return. With + ``sort_by_tags`` (or ``all_info``) a stream that has all of its rows (a node's + stored slice, a computed stream, an in-memory source) returns the first ``n`` + of ``as_table`` with that configuration. A stream that has read or computed only + part of its rows (a source's bounded read, a partly computed stream) sorts those + rows: the rows read, in tag order, not the first ``n`` in tag order. The + notebook preview of a stream is its ``head()`` asked for ``sort_by_tags``. This default shows no rows: a stream class that can serve rows without computing overrides it. The zero-row table then has the stream's columns, typed from its @@ -527,9 +508,9 @@ def head( Args: n: The largest number of rows to return. - columns: Column groups to include, as for ``as_table``. ``sort_by_tags`` is - not accepted: the rows come in stream order. - all_info: Include every column group, as for ``as_table`` (without sorting). + columns: Column groups to include and whether to sort by tags, exactly as + for ``as_table``. + all_info: Include every column group and sort by tags, as for ``as_table``. Returns: A ``StreamHead`` with the rows and a ``status`` that tells an empty result @@ -537,10 +518,10 @@ def head( Raises: TypeError: If ``n`` is not an ``int``. - ValueError: If ``n`` is negative or ``columns`` asks for ``sort_by_tags``. + ValueError: If ``n`` is negative. """ check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) def _head_without_rows( @@ -584,9 +565,5 @@ def _schema_requires_io(self, config: ColumnConfig) -> bool: return any(schema_requires_io(upstream, config) for upstream in self.upstreams) def _repr_html_(self) -> str: - """Render ``head()`` for a notebook; never computes.""" - try: - head = self.head() - except Exception as error: # noqa: BLE001 - a display hook must not raise - return render_error_html(self, error) - return head._repr_html_() + """Render ``head()`` in tag order for a notebook; never computes.""" + return preview_html(self) diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index a794fcc1a..14f7f018f 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -111,6 +111,10 @@ class StreamHead: description: When given, the whole text after the name in the header, written by a stream that words its own rows (a table-backed source naming its ``where``); otherwise the header is built from the fields above. + sorted_by_tags: ``True`` when the rows were asked for in tag order (the notebook + preview): the header then says so, and for a bounded read or a partial + computation says they are the rows read or computed so far, in tag order. + A ``description`` words its own sorting. """ table: pa.Table @@ -123,6 +127,7 @@ class StreamHead: origin: str = "held" note: str | None = None description: str | None = None + sorted_by_tags: bool = False @classmethod def of_rows( @@ -216,10 +221,11 @@ def _describe(self) -> str: text = self._describe_rows(shown) elif self.status is HeadStatus.PARTIAL: total = self.total_rows if self.total_rows is not None else shown + suffix = ", sorted by tags" if self.sorted_by_tags else "" if shown == total: - text = f"{total:,} {_rows(total)} computed so far" + text = f"{total:,} {_rows(total)} computed so far{suffix}" else: - text = f"first {shown:,} of {total:,} rows computed so far" + text = f"first {shown:,} of {total:,} rows computed so far{suffix}" text += " (computation incomplete)" else: return self.note or _DEFAULT_NOTES[self.status] @@ -233,13 +239,19 @@ def _describe_rows(self, shown: int) -> str: noun = _ORIGIN_NOUNS.get(self.origin, "rows") if shown == 0 and (total is None or total == 0): return "computed: no rows" if self.origin == "computed" else f"no {noun}" + suffix = ", sorted by tags" if self.sorted_by_tags else "" if total is None: - return f"first {shown:,} {_rows(shown)}" + return f"first {shown:,} {_rows(shown)}{suffix}" if self.total_is_estimate: + if self.sorted_by_tags: + # A bounded read: the rows read, in tag order, not the first in tag order. + return ( + f"{shown:,} {_rows(shown)} read, sorted by tags, of about {total:,}" + ) return f"first {shown:,} {_rows(shown)} of about {total:,}" if shown == total: - return f"all {total:,} {noun if total != 1 else noun[:-1]}" - return f"first {shown:,} of {total:,} {noun}" + return f"all {total:,} {noun if total != 1 else noun[:-1]}{suffix}" + return f"first {shown:,} of {total:,} {noun}{suffix}" def _display_frame(self) -> pl.DataFrame: """Return a Polars frame of the rows for display: tag columns first, large cells @@ -314,6 +326,24 @@ def display_name(stream: Any) -> str: return name +def preview_html(stream: Any) -> str: + """Return the notebook preview of ``stream``: its head with the rows in tag order. + + The preview asks ``head`` for ``sort_by_tags`` (developer, November 12, 2026), so the + rows shown do not depend on the order they were computed in. For a bounded read or a + partial computation they are the rows read or computed so far, in tag order. A stream + with no tag column shows its rows in stream order. It never computes anything more + than ``head`` does. + """ + try: + head = stream.head(columns={"sort_by_tags": True}) + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(stream, error) + if head.tag_columns and head.table.num_rows: + head = dataclasses.replace(head, sorted_by_tags=True) + return head._repr_html_() + + def head_of( head: Any, stream: Any, *, origin: str | None = None, note: str | None = None ) -> StreamHead: diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 2388dfa67..452ca3431 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -20,7 +20,6 @@ from orcapod.core.streams.base import ( StreamBase, check_head_size, - head_column_config, ) from orcapod.core.streams.head import ( SIDE_EFFECTS_NOTE, @@ -48,6 +47,7 @@ OLD_TRACKING_DB_SCHEMA_VERSIONS, TRACKING_DB_SCHEMA_VERSION, ) +from orcapod.types import ColumnConfig from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -62,7 +62,7 @@ ) from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.protocols.observability_protocols import ExecutionObserverProtocol - from orcapod.types import ColumnConfig, ContentHash, Schema + from orcapod.types import ContentHash, Schema else: pa = LazyModule("pyarrow") @@ -166,7 +166,7 @@ def head( ) -> StreamHead: """Return no rows: showing this stream must never deliver its side effects.""" check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE ) @@ -762,7 +762,7 @@ def head( ) -> StreamHead: """Return no rows: showing this stream must never deliver its side effects.""" check_head_size(n) - config = head_column_config(columns, all_info) + config = ColumnConfig.handle_config(columns, all_info=all_info) return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE ) diff --git a/src/orcapod/utils/arrow_utils.py b/src/orcapod/utils/arrow_utils.py index d85e9a75c..5630eec1a 100644 --- a/src/orcapod/utils/arrow_utils.py +++ b/src/orcapod/utils/arrow_utils.py @@ -4,7 +4,7 @@ import json import uuid from collections import defaultdict -from collections.abc import Mapping, Collection, Sequence +from collections.abc import Callable, Mapping, Collection, Sequence from typing import Any, TYPE_CHECKING, cast from orcapod.system_constants import constants @@ -1831,17 +1831,82 @@ def apply_column_config( if c.startswith(constants.META_PREFIX) and not any(c.startswith(p) for p in normalized_meta_prefixes) ) - output_table = table.drop([c for c in drop_columns if c in table.column_names]) - if column_config.sort_by_tags: - output_table = sort_by_tag_columns(output_table, tag_keys) - return output_table + if column_config.sort_by_tags and tag_keys: + # Before the system tags are dropped: they break ties between equal tags. + table = sort_by_tag_columns(table, tag_keys) + return table.drop([c for c in drop_columns if c in table.column_names]) + + +def unused_column_name(table: "pa.Table", base: str) -> str: + """Return ``base``, with underscores added until no column of ``table`` has that name.""" + name = base + while name in table.column_names: + name += "_" + return name + + +def with_content_hash_column( + table: "pa.Table", + content_hash: bool | str, + hashes: "Callable[[], pa.Array[Any]]", + shape: "Callable[[pa.Table], pa.Table]", + nullable: bool = True, +) -> "pa.Table": + """Return ``shape(table)`` with the requested content-hash column attached. + + The ONE mechanism by which a requested content-hash column is added while rows are + sorted or columns are filtered (ITL-1029, ITL-1053). The hash column is attached to + ``table`` before ``shape`` runs, so a sort moves every hash with its own row, and it + travels under a reserved temporary name that belongs to no column group, so that + neither the sort (a name starting with the system tag prefix is not a tie-break key) + nor the filtering (a name equal to ``_context_key`` or ``_source_y``) can mistake or + drop it. It gets its requested name last. + + Args: + table: The rows, with every column ``shape`` needs. + content_hash: The column configuration's ``content_hash``: ``False`` for none, + ``True`` for ``_content_hash``, or the name of the column. + hashes: Returns the content hash of each row of ``table``, in its order, as a + ``large_string`` array. Called only when a content hash is requested. + shape: Sorts and filters the table (it must keep every column it does not drop + by its own rules, and must not read the hash column). + nullable: Whether the hash column's field is nullable. + + Returns: + The shaped table, the content-hash column last under its requested name. + """ + if not content_hash: + return shape(table) + name = content_hash if isinstance(content_hash, str) else "_content_hash" + carrier = unused_column_name(table, "row content hash") + carried = table.append_column( + pa.field(carrier, pa.large_string(), nullable=nullable), + hashes(), + ) + shaped = shape(carried) + return shaped.rename_columns( + [name if column == carrier else column for column in shaped.column_names] + ) def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Table": - """Return ``table`` with its rows sorted ascending by the tag columns ``tag_keys``. + """Return ``table`` with its rows in tag order: the sort every presentation uses. + + This is the order of ``as_table(sort_by_tags=True)``, ``head`` and the notebook + preview. It is not a hash input: a stream's identity keeps its own ordering + (``ArrowTableStream._identity_table``). + + The rows are ordered ascending by the tag columns ``tag_keys``, nulls last. Rows with + equal tags are then ordered by the system tag columns ``table`` has (they hold the + record ids), in column order, and rows still equal keep their input order, so the + result does not depend on the order the rows arrived in whenever the table has system + tags. A table with neither is returned as it is. A caller that must keep the order of + a stream with no regular tag column does not call this for it. A tag type Arrow cannot + order (a list) is ordered with Polars, nulls last. - Whole rows move together: a column added to ``table`` before the sort stays with - its row. Column types and nullability are those of ``table``. + Whole rows move together: a column added to ``table`` before the sort (a content hash) + stays with its row. Column types and nullability are those of ``table``. Sort before + dropping the system tag columns, or the tiebreak is lost. Args: table: The table to sort. @@ -1850,12 +1915,23 @@ def sort_by_tag_columns(table: "pa.Table", tag_keys: tuple[str, ...]) -> "pa.Tab Returns: A new table with the same columns, rows in tag order. """ - import polars as pl - - sorted_table = ( - pl.DataFrame(table).sort(by=list(tag_keys), descending=False).to_arrow() + system_tags = ( + c for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ) - return restore_schema_nullability(sorted_table, table.schema) + keys = list(dict.fromkeys([*tag_keys, *system_tags])) + if not keys: + return table + try: + return table.sort_by([(key, "ascending") for key in keys]) + except (pa.ArrowNotImplementedError, pa.ArrowTypeError): + import polars as pl + + sorted_table = ( + pl.DataFrame(table) + .sort(by=keys, descending=False, nulls_last=True, maintain_order=True) + .to_arrow() + ) + return restore_schema_nullability(sorted_table, table.schema) if __name__ == "__main__": diff --git a/superpowers/plans/2026-11-13-itl-1053-head-sort.md b/superpowers/plans/2026-11-13-itl-1053-head-sort.md new file mode 100644 index 000000000..08d27953b --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1053-head-sort.md @@ -0,0 +1,19 @@ +# ITL-1053: head() honours sort_by_tags like as_table + +Decision (developer, 2026-11-12): head takes as_table's column configuration, sorting included, +applies it to the rows it has, then takes the first n. Default config stays unsorted. + +## Findings (before coding) +- `head_column_config` forced sort off / raised. Replace by `ColumnConfig.handle_config` (no wrapper). +- `FunctionJobNode._head_of_pairs` and `FunctionPodStream.head` sliced BEFORE `_shape_table`; `_shape_table` + attaches the content hash before sorting (ITL-1029), so for a sorted config: shape all rows, then slice n. +- `ArrowTableStream.head` assembles `limit=n` then sorts: for sorted config assemble all, sort, slice n. +- OperatorJobNode / DerivedSource / SourceProxy / CachedSource / StaticOutputPod delegate to ArrowTableStream.head. +- SpiralDB `_head` reads <= n rows (bounded), `_whole_table` already sorts via as_table: sorts rows read only. +- No store read changes; content hashes for all rows computed only when sort + content_hash asked. + +## Steps +1. Failing tests (head vs as_table.slice for each ColumnConfig flag; reversed async completion, in-memory + Delta, + function and operator nodes; bounded source; default unsorted; sort no longer raises). +2. Implement; docs, docstrings; design_issues entry, changelog.d/ITL-1053.md. +3. Full suite 3.11/3.12/3.13, ruff, basedpyright; PR; CodeRabbit. diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 384ead58e..fee141852 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -614,7 +614,10 @@ def test_display_reads_one_head_of_ten_rows(self, big): html = source._repr_html_() assert self._scan_limits() == [10] - assert "SpiralDBTableSource[big] — first 10 rows of about 25" in html + assert ( + "SpiralDBTableSource[big] — 10 rows read, sorted by tags, of about 25" + in html + ) assert "shape: (10, 3)" in html assert "tags: subject_id, timestamp" in html @@ -629,6 +632,29 @@ def test_head_reads_one_bounded_scan(self, big): assert (head.total_rows, head.total_is_estimate) == (25, True) assert len(head) == 10 + def test_a_sorted_head_sorts_only_the_rows_it_read(self, project): + """``sort_by_tags`` reorders the one bounded read and reads no more (ITL-1053). + + The table is read in key order (subject, then timestamp) but tagged by + ``timestamp``, so the first ten rows read are not in tag order. + """ + table = project.create_table("default.mixed", key_schema=_KEYS) + table.write(_rows([10, 20, 30, 40, 50], subject="m1")) + table.write(_rows([1, 2, 3, 4, 5, 6], subject="m2")) + fake_pyspiral.scan_log.clear() + source = SpiralDBTableSource( + _PROJECT, "mixed", tag_columns=["timestamp"], label="mixed" + ) + + unsorted_head = source.head() + sorted_head = source.head(columns={"sort_by_tags": True}) + + assert self._scan_limits() == [10, 10] + read = cast(list[int], unsorted_head.table.column("timestamp").to_pylist()) + assert read == [10, 20, 30, 40, 50, 1, 2, 3, 4, 5] + assert sorted_head.table.column("timestamp").to_pylist() == sorted(read) + assert "10 rows read, sorted by tags" in sorted_head.header() + def test_an_unlabelled_source_is_shown_by_its_table_name(self, big): """With no label, the header names the table; the label itself is unchanged.""" source = SpiralDBTableSource(_PROJECT, "big") @@ -646,7 +672,7 @@ def test_a_table_above_the_cap_previews_without_raising(self, big): html = source._repr_html_() assert self._scan_limits() == [10] - assert "first 10 rows of about 25" in html + assert "10 rows read, sorted by tags, of about 25" in html def test_a_small_table_shows_all_its_rows(self, project): """A table of fewer than 10 rows is shown whole, saying so.""" @@ -690,7 +716,7 @@ def test_the_header_is_escaped_in_the_display(self, project): html = source._repr_html_() assert ( - "first 10 matching rows (where subject_id='<b>'); " + "10 matching rows read, sorted by tags (where subject_id='<b>'); " "the table has about 25 rows" ) in html assert "'" not in html diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 2dac4d642..c4193b937 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -86,23 +86,25 @@ def test_head_of_a_short_source_says_all_rows() -> None: def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: - """With every column group, head is the prefix of as_table, in stream order.""" + """With every column group, head is the prefix of as_table (ITL-1053).""" source = _source() head = source.head(4, columns=ALL_COLUMNS) assert head.table.equals(source.as_table(columns=ALL_COLUMNS).slice(0, 4)) - # all_info means every column group; head returns rows in stream order. + # all_info sorts by tag, as it does for as_table. assert source.head(4, all_info=True).table.equals( - source.as_table(columns=ALL_COLUMNS).slice(0, 4) + source.as_table(all_info=True).slice(0, 4) ) -def test_head_refuses_negative_n_and_sorting() -> None: - """A negative n and sort_by_tags are refused.""" +def test_head_refuses_a_negative_n_and_accepts_sorting() -> None: + """A negative n is refused; sort_by_tags is a column option, as for as_table.""" source = _source() with pytest.raises(ValueError, match="n"): source.head(-1) - with pytest.raises(ValueError, match="sort"): - source.head(columns={"sort_by_tags": True}) + head = source.head(3, columns={"sort_by_tags": True}) + assert head.table.equals( + source.as_table(columns={"sort_by_tags": True}).slice(0, 3) + ) def test_head_of_zero_rows_keeps_the_columns() -> None: @@ -135,11 +137,18 @@ def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: def test_preview_is_the_rendered_head() -> None: - """The notebook preview is head() rendered.""" + """The notebook preview is head() rendered, with the rows in tag order (ITL-1053).""" source = _source() html = source._repr_html_() - assert "ArrowTableSource[readings] — first 10 of 25 rows" in html - assert html == source.head()._repr_html_() + assert "ArrowTableSource[readings] — first 10 of 25 rows, sorted by tags" in html + assert ( + html.split("")[1:] + == ( + source.head(columns={"sort_by_tags": True}) + ._repr_html_() + .split("")[1:] + ) + ) # --------------------------------------------------------------------------- diff --git a/tests/test_core/streams/test_head_sorting.py b/tests/test_core/streams/test_head_sorting.py new file mode 100644 index 000000000..cb14a71b6 --- /dev/null +++ b/tests/test_core/streams/test_head_sorting.py @@ -0,0 +1,430 @@ +"""A stream's ``head()`` sorts the way ``as_table`` does, among the rows it has (ITL-1053). + +``head(n, cfg)`` takes the column configuration of ``as_table``, sorting included: with +``sort_by_tags`` (or ``all_info``) the rows the stream has are sorted by tag and the first +``n`` are returned, so ``head(n, cfg) == as_table(cfg).slice(0, n)``. The default +configuration does not sort, as ``as_table()`` does not. A stream that reads or holds only +part of its rows sorts those rows. Sorting never computes or reads anything more. +""" + +from __future__ import annotations + +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream +from orcapod.core.operators import PolarsFilter, SelectDataColumns +from orcapod.core.sources import ArrowTableSource, DerivedSource, DictSource +from orcapod.core.sources.polling_source import PollingSource +from orcapod.streams import HeadStatus +from orcapod.types import ColumnConfig, Cursor, Schema + +ROWS = 8 +# Not in tag order, with the tag order's first rows last. +ORDER = [5, 2, 7, 0, 6, 1, 4, 3] + +CONFIGS: dict[str, dict[str, Any]] = { + "default": {}, + "sort_by_tags": {"columns": {"sort_by_tags": True}}, + "sort_by_tags_config": {"columns": ColumnConfig(sort_by_tags=True)}, + "all_info": {"all_info": True}, + "meta": {"columns": {"meta": True}}, + "context": {"columns": {"context": True}}, + "source": {"columns": {"source": True}}, + "system_tags": {"columns": {"system_tags": True}}, + "content_hash": {"columns": {"content_hash": True}}, + "content_hash_sorted": {"columns": {"content_hash": True, "sort_by_tags": True}}, +} + +CALLS: list[int] = [] + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@pytest.fixture(autouse=True) +def _reset_calls() -> None: + CALLS.clear() + + +def _table() -> pa.Table: + return pa.table( + {"id": ORDER, "x": ORDER}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + + +def _source() -> ArrowTableSource: + return ArrowTableSource( + _table(), tag_columns=["id"], source_id="nums", label="nums" + ) + + +def _computed_pod_stream() -> FunctionPodStream: + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + stream.as_table() + CALLS.clear() + return stream + + +def _computed_operator_stream() -> Any: + stream = SelectDataColumns(["x"])(_source()) + stream.as_table() + return stream + + +def _derived_source() -> DerivedSource: + return DerivedSource(_source(), tags_as_data={"id": "id_copy"}) + + +STREAMS = { + "arrow_table_source": _source, + "computed_function_pod_stream": _computed_pod_stream, + "computed_operator_stream": _computed_operator_stream, + "derived_source": _derived_source, +} + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("n", [0, 3, ROWS, 50]) +@pytest.mark.parametrize("kind", list(STREAMS)) +def test_head_is_the_prefix_of_as_table_for_every_configuration( + kind: str, n: int, config: str +) -> None: + """A stream whose rows are all held: head(n, cfg) is as_table(cfg) cut to n rows.""" + stream = STREAMS[kind]() + head = stream.head(n, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(stream.as_table(**CONFIGS[config]).slice(0, n)) + assert CALLS == [] + + +@pytest.mark.parametrize("kind", list(STREAMS)) +def test_the_default_head_is_not_sorted_and_the_sorted_one_is(kind: str) -> None: + """Only sort_by_tags and all_info sort; the first n of the sorted rows are shown.""" + stream = STREAMS[kind]() + ids = stream.head(ROWS).table.column("id").to_pylist() + assert ids == ORDER + by_columns = stream.head(3, columns={"sort_by_tags": True}) + by_all_info = stream.head(3, all_info=True) + assert by_columns.table.column("id").to_pylist() == [0, 1, 2] + assert by_all_info.table.column("id").to_pylist() == [0, 1, 2] + + +def test_asking_for_sorted_rows_does_not_raise() -> None: + """``sort_by_tags`` was refused by ``head`` before; now it is a column option.""" + head = _source().head(columns={"sort_by_tags": True}) + assert head.table.column("id").to_pylist() == sorted(ORDER) + + +def test_a_partly_computed_stream_sorts_the_rows_computed_so_far() -> None: + """After a partial iteration, the head is the computed rows, sorted; no more is computed.""" + stream = triple.pod(_source()) + rows = stream.iter_data() + for _ in range(4): # computes the first four inputs: ids 5, 2, 7, 0 + next(rows) + assert CALLS == [5, 2, 7, 0] + + head = stream.head(3, all_info=True) + unsorted_head = stream.head(3) + + assert CALLS == [5, 2, 7, 0] + assert head.status is HeadStatus.PARTIAL + # The rows computed so far in tag order, not the first 3 of the whole result in tag order. + assert head.table.column("id").to_pylist() == [0, 2, 5] + assert unsorted_head.table.column("id").to_pylist() == [5, 2, 7] + + +class _OneBatch: + """A ``DynamicSourceProtocol`` double that gives one batch, in ``ORDER``, and counts.""" + + def __init__(self) -> None: + self.polls = 0 + self.fetches = 0 + + def identity(self) -> Any: + return "one-batch" + + def to_config(self) -> dict[str, Any] | None: + return None + + @classmethod + def from_config(cls, config: dict[str, Any]) -> _OneBatch: + raise NotImplementedError + + def schema(self) -> Any: + return None + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + self.polls += 1 + return cursor is None + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + self.fetches += 1 + return Cursor(value=1), { + "id": pa.array(ORDER, type=pa.int64()), + "val": pa.array(ORDER, type=pa.int64()), + } + + async def close(self) -> None: + return None + + +def test_a_polling_source_sorts_the_rows_it_has_polled_and_polls_nothing() -> None: + """Before a poll there are no rows to sort; after one, the polled rows sort.""" + fake = _OneBatch() + source = PollingSource(fake, tag_columns="id") + assert source.head(3, all_info=True).status is HeadStatus.NOT_COMPUTED + assert (fake.polls, fake.fetches) == (0, 0) + + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + head = source.head(3, all_info=True) + + assert (fake.polls, fake.fetches) == seen + assert head.table.column("id").to_pylist() == [0, 1, 2] + assert source.head(3).table.column("id").to_pylist() == ORDER[:3] + + +def test_a_filter_stream_after_a_run_sorts_like_as_table() -> None: + """An operator stream with dropped rows: head(n, cfg) is as_table(cfg) cut to n.""" + stream = PolarsFilter(constraints={"x": 5})(_source()) + stream.as_table() + head = stream.head(5, all_info=True) + assert head.table.equals(stream.as_table(all_info=True).slice(0, 5)) + + +def test_a_dict_source_head_sorts() -> None: + """Every source that wraps an ArrowTableSource sorts the same way.""" + source = DictSource( + [{"id": i, "x": i} for i in ORDER], tag_columns=["id"], source_id="d" + ) + assert source.head(3, all_info=True).table.equals( + source.as_table(all_info=True).slice(0, 3) + ) + + +def test_an_unbound_source_proxy_head_accepts_a_sorted_configuration() -> None: + """A proxy with no live source shows no rows, whatever the column configuration.""" + from orcapod.core.sources.source_proxy import SourceProxy + + live = _source() + proxy = SourceProxy( + source_id="nums", + expected_class_name="ArrowTableSource", + tag_schema=Schema({"id": int}), + data_schema=Schema({"x": int}), + content_hash_str=live.content_hash().to_string(), + pipeline_hash_str=live.pipeline_hash().to_string(), + ) + head = proxy.head(3, all_info=True) + assert head.status is HeadStatus.UNAVAILABLE + + +# --------------------------------------------------------------------------- +# The notebook preview sorts by tags (developer, November 12, 2026) +# --------------------------------------------------------------------------- + + +def _preview_ids(html: str) -> list[int]: + """The ``id`` cells of a preview, in the order they are shown.""" + import re + + return [int(v) for v in re.findall(r"(\d+)", html)][::2] + + +def test_the_preview_of_a_source_shows_rows_in_tag_order_and_says_so() -> None: + """Rows not in tag order are previewed sorted; the default head is still unsorted.""" + source = _source() + html = source._repr_html_() + assert "ArrowTableSource[nums] — all 8 rows, sorted by tags" in html + assert _preview_ids(html) == sorted(ORDER) + assert source.head().table.column("id").to_pylist() == ORDER + + +def test_the_preview_of_a_partly_computed_stream_is_the_rows_computed_so_far() -> None: + """A partial computation previews the computed rows, in tag order; nothing more runs.""" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + rows = stream.iter_data() + for _ in range(4): + next(rows) + html = stream._repr_html_() + assert CALLS == [5, 2, 7, 0] + assert "first 4 of 4 rows computed so far" not in html + assert "4 rows computed so far, sorted by tags (computation incomplete)" in html + assert _preview_ids(html) == [0, 2, 5, 7] + + +def test_the_preview_of_a_stream_with_no_computed_rows_has_no_sorting_caption() -> None: + """Only shown rows are described as sorted.""" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + html = stream._repr_html_() + assert "sorted by tags" not in html + assert CALLS == [] + + +def test_the_preview_of_a_stream_without_tag_columns_keeps_stream_order() -> None: + """With no tag column there is nothing to sort by; the rows keep their order.""" + source = ArrowTableSource( + _table().select(["x"]), tag_columns=[], source_id="untagged", label="untagged" + ) + html = source._repr_html_() + assert "sorted by tags" not in html + assert source.head(columns={"sort_by_tags": True}).table.column( + "x" + ).to_pylist() == (ORDER) + + +def test_a_custom_content_hash_column_is_kept_by_a_nodes_head() -> None: + """A custom hash column name that looks like a meta column is still shown.""" + from orcapod import PipelineJob + from orcapod.databases import InMemoryArrowDatabase + + job = PipelineJob(name="digest", store=InMemoryArrowDatabase()) + with job: + triple.pod(_source(), label="node") + job.run() + node = job.nodes["node"] + config = {"content_hash": "__digest", "sort_by_tags": True} + assert "__digest" in node.as_table(columns=config).column_names + head = node.head(3, columns=config) + assert head.table.equals(node.as_table(columns=config).slice(0, 3)) + + +def test_a_tagless_source_head_follows_as_table_with_and_without_system_tags() -> None: + """With no regular tag column the rows keep their order unless system tags are asked. + + Asked for, they are sorted by the system tags, as ``as_table`` sorts them (the + stream's identity depends on that); the head is that table cut to ``n`` rows. + """ + source = ArrowTableSource( + _table().select(["x"]), tag_columns=[], source_id="untagged", label="untagged" + ) + plain = source.head(8, columns={"sort_by_tags": True}).table + assert plain.column("x").to_pylist() == ORDER + asked = source.head(8, all_info=True).table + assert asked.equals(source.as_table(all_info=True).slice(0, 8)) + + +def test_a_tagless_stream_with_system_tags_keeps_its_canonical_identity_order() -> None: + """A stream with no tag column is sorted by its system tags when they are asked for. + + ``identity_structure`` hashes ``as_table(system_tags, sort_by_tags)``: that order, + system tags only, is part of the content hash and must not change (ITL-1053). + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table({"record_id": ["b", "c", "a"], "x": [1, 2, 3]}) + stream = ArrowTableStream(table, tag_columns=[], system_tag_columns=["record_id"]) + ordered = stream.as_table(columns={"system_tags": True, "sort_by_tags": True}) + shown = next(c for c in ordered.column_names if c.endswith("record_id")) + assert ordered.column(shown).to_pylist() == ["a", "b", "c"] + plain = stream.as_table(columns={"sort_by_tags": True}) + assert plain.column("x").to_pylist() == [1, 2, 3] + + +def test_a_sorted_head_includes_rows_cached_after_a_missing_input() -> None: + """Sorting uses every output the stream holds, also those after an input with none.""" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + rows = stream.iter_data() + for _ in range(4): + next(rows) + # A concurrent failure: the first input has no cached output, the others do. + # Private state on purpose: this test is about how head reads that gap. + del stream._cached_output_datas[0] + unsorted_head = stream.head(10) + sorted_head = stream.head(10, columns={"sort_by_tags": True}) + assert len(unsorted_head) == 0 + assert sorted_head.table.column("id").to_pylist() == [0, 2, 7] + + +def test_a_custom_hash_column_named_like_a_system_tag_is_kept() -> None: + """A function pod stream keeps a requested hash column whatever its name starts with.""" + from orcapod.system_constants import constants + + name = f"{constants.SYSTEM_TAG_PREFIX}digest" + stream = triple.pod(_source()) + assert isinstance(stream, FunctionPodStream) + config = {"content_hash": name, "sort_by_tags": True} + table = stream.as_table(columns=config) + assert name in table.column_names + assert stream.head(3, columns=config).table.equals(table.slice(0, 3)) + + +@pytest.mark.parametrize("name", ["_context_key", "_source_y", "_tag_digest"]) +def test_a_hash_column_named_like_another_column_group_is_kept(name: str) -> None: + """A requested hash name equal to a column the shaping drops or keeps still works.""" + from orcapod import PipelineJob + from orcapod.databases import InMemoryArrowDatabase + + job = PipelineJob(name="names", store=InMemoryArrowDatabase()) + with job: + pod_stream = triple.pod(_source(), label="node") + assert isinstance(pod_stream, FunctionPodStream) + job.run() + for target in (pod_stream, job.nodes["node"]): + for sort in (False, True): + config = {"content_hash": name, "sort_by_tags": sort} + table = target.as_table(columns=config) + assert table.column_names.count(name) == 1 + head = target.head(3, columns=config) + assert head.table.equals(table.slice(0, 3)) + + +def test_the_presentation_sort_orders_a_list_tag_while_identity_keeps_stream_order() -> ( + None +): + """``as_table(sort_by_tags)`` orders list tags (Polars); the identity table does not. + + The identity of a stream hashes its own table, in the order main always gave it + (``ArrowTableStream._identity_table``); only presentations use the new sort. + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table( + [pa.array([[2], [1]], pa.large_list(pa.int64())), pa.array([20, 10])], + names=["k", "x"], + ) + stream = ArrowTableStream(table, tag_columns=["k"]) + assert stream.as_table(columns={"sort_by_tags": True}).column("x").to_pylist() == [ + 10, + 20, + ] + identity = stream.identity_structure()[1] + assert identity.column("x").to_pylist() == [20, 10] + + +def test_a_hash_column_named_like_a_system_tag_does_not_reorder_tied_rows() -> None: + """On an ``ArrowTableStream`` a requested ``_tag_digest`` is not a sort key. + + Rows with equal tags keep their input order whatever the hash column is called. + """ + from orcapod.core.streams import ArrowTableStream + + table = pa.table( + [pa.array([1, 1, 1], pa.int64()), pa.array([3, 1, 2], pa.int64())], + names=["k", "x"], + ) + stream = ArrowTableStream(table, tag_columns=["k"]) + for name in (True, "_tag_digest", "digest"): + config = {"content_hash": name, "sort_by_tags": True} + shown = stream.as_table(columns=config) + assert shown.column("x").to_pylist() == [3, 1, 2] + assert shown.column_names[-1] == ("_content_hash" if name is True else name) + assert stream.head(2, columns=config).table.equals(shown.slice(0, 2)) diff --git a/tests/test_core/streams/test_identity_order_regression.py b/tests/test_core/streams/test_identity_order_regression.py new file mode 100644 index 000000000..6f4b5b3a5 --- /dev/null +++ b/tests/test_core/streams/test_identity_order_regression.py @@ -0,0 +1,100 @@ +"""The row order an ``ArrowTableStream`` hashes for its identity never changes (ITL-1053). + +``identity_structure`` of a stream with no producer hashes its table, so the order of its +rows is a hash input. ITL-1053 gave ``as_table``, ``head`` and the preview a new +deterministic sort; the identity keeps the ordering code it had before. This test holds a +literal copy of that code and compares it with the stream's identity table over a battery +of shapes: tag and system tag columns, system tags given in either order, nulls, tied +tags and list tags. +""" + +from __future__ import annotations + +import itertools +import random +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod.core.streams import ArrowTableStream +from orcapod.types import ColumnConfig + + +def main_ordering(stream: ArrowTableStream, table: pa.Table) -> pa.Table: + """The sort block of ``ArrowTableStream._assemble_table`` on origin/main. + + Copied literally from origin/main at commit 89160c47abf149c8160496de2cdfebe7d651e448 + (``src/orcapod/core/streams/arrow_table_stream.py``, last changed in f51ce67c), with + ``columns_config.system_tags`` set to ``True``, which is what ``identity_structure`` + asked for, and ``columns_config.sort_by_tags`` true. Do not edit it. + """ + # --- literal copy below; ``self`` is ``stream``, ``columns_config.system_tags`` is True + self = stream + try: + target_tags = self._all_tag_columns # pyright: ignore[reportPrivateUsage] + return table.sort_by([(column, "ascending") for column in target_tags]) + except pa.ArrowTypeError: + # If sorting fails, fall back to unsorted table + return table + + +def shapes() -> list[tuple[Any, ...]]: + """Every shape of the battery: tag columns, system tag order, nulls, ties, list tags.""" + return list( + itertools.product( + [(), ("k",), ("k", "j")], + [("s1", "s2"), ("s2", "s1"), ("s1",), ()], + [False, True], + [False, True], + [False, True], + ) + ) + + +def build(shape: tuple[Any, ...]) -> ArrowTableStream: + """A six-row stream of the given shape; random but seeded values.""" + tags, system, nulls, ties, list_tag = shape + rng = random.Random(7) + n = 6 + k: list[Any] = [rng.choice([0, 1, 2]) if ties else i for i in range(n)] + j: list[Any] = [rng.choice(["a", "b"]) for _ in range(n)] + if nulls: + k[1] = None + j[3] = None + columns: dict[str, pa.Array[Any]] = { + "k": pa.array( + [None if v is None else [v] for v in k], pa.large_list(pa.int64()) + ) + if list_tag + else pa.array(k, pa.int64()), + "j": pa.array(j, pa.string()), + "s1": pa.array([rng.choice(["p", "q", "r"]) for _ in range(n)], pa.string()), + "s2": pa.array([rng.choice(["u", "v"]) for _ in range(n)], pa.string()), + "x": pa.array(range(n), pa.int64()), + } + names = [c for c in columns if c in tags or c in system or c == "x"] + table = pa.table([columns[c] for c in names], names=names) + return ArrowTableStream(table, tag_columns=tags, system_tag_columns=system) + + +@pytest.mark.parametrize("shape", shapes()) +def test_the_identity_table_has_the_row_order_main_gave_it( + shape: tuple[Any, ...], +) -> None: + """``identity_structure``'s table is main's ordering of the same columns, or both raise.""" + stream = build(shape) + unsorted = stream.as_table( + columns=ColumnConfig(meta=True, context=True, source=True, system_tags=True) + ) + try: + expected = main_ordering(stream, unsorted) + except pa.ArrowInvalid: + # Main raises when there is nothing to sort by: so does the identity. + with pytest.raises(pa.ArrowInvalid): + stream.identity_structure() + return + name, table, tag_columns = stream.identity_structure() + assert name == "ArrowTableStream" + assert table.equals(expected) + assert tag_columns == tuple(c for c in shape[0]) diff --git a/tests/test_core/streams/test_streams.py b/tests/test_core/streams/test_streams.py index 3fcb254c1..e7dd6e427 100644 --- a/tests/test_core/streams/test_streams.py +++ b/tests/test_core/streams/test_streams.py @@ -543,8 +543,12 @@ def test_repr_html_is_the_rendered_head(self): html = stream._repr_html_() - assert html == stream.head()._repr_html_() - assert "ArrowTableStream[numbers] — first 10 of 25 rows" in html + sorted_head = stream.head(columns={"sort_by_tags": True}) + assert ( + html.split("")[1:] + == sorted_head._repr_html_().split("")[1:] + ) + assert "ArrowTableStream[numbers] — first 10 of 25 rows, sorted by tags" in html assert "tags: id" in html assert "shape: (10, 2)" in html diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index 14cb31181..c2a0c3129 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -1884,3 +1884,62 @@ def torch_datagram() -> dict[str, Any]: return stream_summary( DictSource(data=rows, tag_columns=["k"], source_id="golden_torch_tensor") ) + + +@case("pipeline", "tagless_producerless_stream") +def pipeline_tagless_producerless_stream() -> dict[str, Any]: + """Pin the content hash of a stream with no tag column and no producer (ITL-1053). + + Its identity hashes the rows sorted by their system tags, so the hash is the same + for rows given in any order. + + Returns: + The content hash for two orders of the same rows. + """ + from orcapod.core.streams import ArrowTableStream + + def table(record_ids: list[str], xs: list[int]) -> pa.Table: + return pa.table( + [pa.array(record_ids, pa.string()), pa.array(xs, pa.int64())], + names=["record_id", "x"], + ) + + rows = table(["b", "c", "a"], [1, 2, 3]) + reordered = table(["c", "a", "b"], [2, 3, 1]) + return { + "as_given": ArrowTableStream( + rows, tag_columns=[], system_tag_columns=["record_id"] + ) + .content_hash() + .to_string(), + "reordered": ArrowTableStream( + reordered, tag_columns=[], system_tag_columns=["record_id"] + ) + .content_hash() + .to_string(), + } + + +@case("pipeline", "list_tag_producerless_stream") +def pipeline_list_tag_producerless_stream() -> dict[str, Any]: + """Pin the content hash of a stream with a list-valued tag and no producer (ITL-1053). + + Arrow cannot order a list column, so the stream's identity hashes its rows in the + order given: the hash of the same rows in two orders differs. + + Returns: + The content hash for two orders of the same rows. + """ + from orcapod.core.streams import ArrowTableStream + + def stream(ks: list[list[int]], xs: list[int]) -> ArrowTableStream: + table = pa.table( + [pa.array(ks, pa.large_list(pa.int64())), pa.array(xs, pa.int64())], + names=["k", "x"], + ) + return ArrowTableStream(table, tag_columns=["k"]) + + return { + "as_given": stream([[2], [1]], [20, 10]).content_hash().to_string(), + "reversed": stream([[1], [2]], [10, 20]).content_hash().to_string(), + } diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index b08876711..7ee2faca9 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -367,6 +367,8 @@ "_tag_source_id::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c:1", "_tag_record_id::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c::ddc871a1c97cfa92d4eeb46d3ad5a45900283dab98580b108a93c386f802982c:1" ], + "list_tag_producerless_stream.as_given": "semantic_v1:13bcfc176f286f93f6e726f100d484889b970f27ab9faa68e815d126e33f9ad9", + "list_tag_producerless_stream.reversed": "semantic_v1:9bb1d6d7a8958dfb76562fa48854acff9b95b72a735631dce0b2cf583aae53a3", "map_data.content_hash": "semantic_v1:7ed964c988b138e57d33b6db445eb9322babdd11dd9d16ef09b7a618efc16b4b", "map_data.pipeline_hash": "semantic_v1:e7b348a99246e0f719402540be0b77865bf2c0fe8de25e3355b707f1aa3fe2d7", "map_data.row[{\"subject\":\"s1\"}]._tag_record_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5": "e8e02084627f5e1aad83839e421538e8", @@ -591,5 +593,7 @@ "sources.subjects.system_tag_columns": [ "_tag_source_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5", "_tag_record_id::d22029fcc2b0458566a83389c23c1a850dc69499f2456ae83bc6345284f861a5" - ] + ], + "tagless_producerless_stream.as_given": "semantic_v1:8910e1f5088e3a86be550c356c8127e4dcd302c99378942f30bc34a982b1a108", + "tagless_producerless_stream.reordered": "semantic_v1:8910e1f5088e3a86be550c356c8127e4dcd302c99378942f30bc34a982b1a108" } diff --git a/tests/test_pipeline/test_node_head_sorting.py b/tests/test_pipeline/test_node_head_sorting.py new file mode 100644 index 000000000..a39223f7f --- /dev/null +++ b/tests/test_pipeline/test_node_head_sorting.py @@ -0,0 +1,283 @@ +"""A job node's ``head()`` sorts the way ``as_table`` does, among the rows it has (ITL-1053). + +``head(n, cfg)`` is ``as_table(cfg).slice(0, n)`` for every column configuration, whatever +order the rows were computed in: input order on the sync path, and on the async path an +order forced here (events, no sleeps) to be the reverse of tag order. Operator nodes are +tested after a sync run, or from a store a sync run filled: an operator node's head after +an async run says NOT_COMPUTED until ITL-1042 is fixed. +""" + +from __future__ import annotations + +import asyncio +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.nodes import OperatorJobNode +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.streams import HeadStatus +from orcapod.types import CacheMode + +N_ROWS = 6 + +CONFIGS: dict[str, dict[str, Any]] = { + "default": {}, + "sort_by_tags": {"columns": {"sort_by_tags": True}}, + "all_info": {"all_info": True}, + "meta": {"columns": {"meta": True}}, + "context": {"columns": {"context": True}}, + "source": {"columns": {"source": True}}, + "system_tags": {"columns": {"system_tags": True}}, + "content_hash": {"columns": {"content_hash": True}}, + "content_hash_sorted": {"columns": {"content_hash": True, "sort_by_tags": True}}, +} + +CALLS: list[int] = [] + +# Rows are released in reverse tag order: row i waits for row i + 1. Bound to the running +# loop on first use, so each async run builds its own. +gates: list[asyncio.Event] = [] + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@function_pod(output_keys="y") +async def triple_last_row_first(x: int) -> int: + """Triple ``x``; row ``i`` finishes only after row ``i + 1``, so rows complete in + reverse tag order. The wait is bounded so a failure is a missing row, not a hang.""" + CALLS.append(x) + if x + 1 < N_ROWS: + await asyncio.wait_for(gates[x + 1].wait(), timeout=60) + gates[x].set() + return 3 * x + + +def _source(order: list[int]) -> ArrowTableSource: + """A source of rows ``id`` and ``x`` in the given order (non-nullable ``int64``).""" + table = pa.table( + {"id": order, "x": order}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="nums", label="nums") + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +def _build(pod: Any, store: ArrowDatabaseProtocol, order: list[int]) -> PipelineJob: + job = PipelineJob(name="itl1053", store=store) + with job: + pod.pod(_source(order), label="node") + return job + + +def _run(kind: str, store: ArrowDatabaseProtocol) -> PipelineJob: + """Run a job so that its rows are stored out of tag order; return the job. + + ``sync``: the input is in reverse tag order. ``async``: the input is in tag order and + the rows complete in reverse tag order. + """ + gates.clear() + if kind == "sync": + job = _build(triple, store, list(reversed(range(N_ROWS)))) + job.run(orchestrator=SyncPipelineOrchestrator()) + return job + gates.extend(asyncio.Event() for _ in range(N_ROWS)) + job = _build(triple_last_row_first, store, list(range(N_ROWS))) + job.run(orchestrator=AsyncPipelineOrchestrator()) + return job + + +def _held_order(node: Any) -> list[int]: + """The tag order the node holds its rows in (unsorted ``as_table``).""" + return node.as_table().column("id").to_pylist() + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("n", [0, 3, N_ROWS, 50]) +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("run_kind", ["sync", "async"]) +def test_a_function_nodes_head_is_the_prefix_of_as_table_in_any_completion_order( + run_kind: str, store_kind: str, n: int, config: str, tmp_path: Path +) -> None: + """After a run, and from the store in a new job, head(n, cfg) == as_table(cfg)[:n].""" + store = _store(store_kind, tmp_path) + job = _run(run_kind, store) + assert _held_order(job.nodes["node"]) != sorted(_held_order(job.nodes["node"])) + CALLS.clear() + fresh = _build( + triple if run_kind == "sync" else triple_last_row_first, + store, + list(range(N_ROWS)), + ) + for node in (job.nodes["node"], fresh.nodes["node"]): + head = node.head(n, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(node.as_table(**CONFIGS[config]).slice(0, n)) + assert CALLS == [] + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("run_kind", ["sync", "async"]) +def test_head_sorts_only_when_asked_to( + run_kind: str, store_kind: str, tmp_path: Path +) -> None: + """The default head keeps the node's order; sort_by_tags and all_info sort by tag.""" + job = _run(run_kind, _store(store_kind, tmp_path)) + node = job.nodes["node"] + held = _held_order(node) + assert held != sorted(held) + assert node.head(N_ROWS).table.column("id").to_pylist() == held + for config in ({"columns": {"sort_by_tags": True}}, {"all_info": True}): + ids = node.head(N_ROWS, **config).table.column("id").to_pylist() + assert ids == sorted(held) + # The first n of the sorted rows, not the sorted first n of the held rows. + assert node.head(2, all_info=True).table.column("id").to_pylist() == [0, 1] + + +@pytest.mark.parametrize("config", list(CONFIGS)) +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_an_operator_nodes_head_is_the_prefix_of_as_table( + store_kind: str, config: str, tmp_path: Path +) -> None: + """An operator node's stored rows, out of tag order, sort as ``as_table`` sorts.""" + store = _store(store_kind, tmp_path) + order = list(reversed(range(N_ROWS))) + + def node(mode: CacheMode) -> OperatorJobNode: + return OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source(order)], + pipeline_database=store, + cache_mode=mode, + ) + + ran = node(CacheMode.LOG) + ran.execute(_source(order)) + for each in (ran, node(CacheMode.REPLAY)): + head = each.head(3, **CONFIGS[config]) + assert head.status is HeadStatus.ROWS + assert head.table.equals(each.as_table(**CONFIGS[config]).slice(0, 3)) + sorted_ids = node(CacheMode.REPLAY).head(3, all_info=True).table.column("id") + assert sorted_ids.to_pylist() == [0, 1, 2] + + +# --------------------------------------------------------------------------- +# Equal and null tags: one deterministic sort (ITL-1053) +# --------------------------------------------------------------------------- + +TIE_KEYS: list[int | None] = [2, 1, None, 2, 1, None, 0, 2] + + +def _tie_source() -> ArrowTableSource: + """Rows with duplicate and null tags ``k``, each with its own ``x``.""" + table = pa.table( + {"k": pa.array(TIE_KEYS, pa.int64()), "x": pa.array(range(N_ROWS + 2))}, + schema=pa.schema( + [ + pa.field("k", pa.int64(), nullable=True), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["k"], source_id="ties", label="ties") + + +def _tie_run(kind: str) -> Any: + """Run ``_tie_source`` through a function node; ``async`` completes rows in reverse.""" + gates.clear() + job = PipelineJob(name="ties", store=InMemoryArrowDatabase()) + if kind == "sync": + with job: + triple.pod(_tie_source(), label="node") + job.run(orchestrator=SyncPipelineOrchestrator()) + else: + gates.extend(asyncio.Event() for _ in range(len(TIE_KEYS))) + with job: + tie_last_row_first.pod(_tie_source(), label="node") + job.run(orchestrator=AsyncPipelineOrchestrator()) + return job.nodes["node"] + + +@function_pod(output_keys="y") +async def tie_last_row_first(x: int) -> int: + """Triple ``x``; row ``i`` finishes after row ``i + 1`` (reverse completion).""" + if x + 1 < len(TIE_KEYS): + await asyncio.wait_for(gates[x + 1].wait(), timeout=60) + gates[x].set() + return 3 * x + + +@pytest.mark.parametrize("config", ["sort_by_tags", "all_info", "content_hash_sorted"]) +def test_a_sorted_table_with_equal_and_null_tags_does_not_depend_on_completion_order( + config: str, +) -> None: + """Rows with equal tags are ordered by their system tags, nulls last, in any order.""" + sync_node, async_node = _tie_run("sync"), _tie_run("async") + assert _held_order_k(sync_node) != _held_order_k(async_node) + tables = { + name: node.as_table(**CONFIGS[config]) + for name, node in (("sync", sync_node), ("async", async_node)) + } + + def without_source_info(table: pa.Table) -> pa.Table: + """The table without ``_source_`` columns: their tokens name each run's rows.""" + return table.drop_columns( + [c for c in table.column_names if c.startswith("_source_")] + ) + + assert without_source_info(tables["sync"]).equals( + without_source_info(tables["async"]) + ) + assert tables["sync"].column("k").to_pylist() == [0, 1, 1, 2, 2, 2, None, None] + for name, node in (("sync", sync_node), ("async", async_node)): + for n in (0, 3, 5, 20): + head = node.head(n, **CONFIGS[config]) + assert head.table.equals(tables[name].slice(0, n)) + + +def _held_order_k(node: Any) -> list[int | None]: + return node.as_table().column("k").to_pylist() + + +# --------------------------------------------------------------------------- +# The notebook preview sorts by tags (developer, November 12, 2026) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_a_nodes_preview_is_the_same_after_any_completion_order( + store_kind: str, tmp_path: Path +) -> None: + """The preview of a node shows tag order: reversed completion looks like a sync run.""" + # A sync run over input in tag order holds its rows in tag order. + ascending = _build(triple, _store(store_kind, tmp_path / "a"), list(range(N_ROWS))) + ascending.run(orchestrator=SyncPipelineOrchestrator()) + sync_node = ascending.nodes["node"] + async_node = _run("async", _store(store_kind, tmp_path / "b")).nodes["node"] + assert _held_order(sync_node) != _held_order(async_node) + assert sync_node._repr_html_() == async_node._repr_html_() + html = async_node._repr_html_() + assert "all 6 rows, sorted by tags" in html + assert html.index("0") < html.index("5")