diff --git a/changelog.d/ITL-186.md b/changelog.d/ITL-186.md index de6b334da..99b35203a 100644 --- a/changelog.d/ITL-186.md +++ b/changelog.d/ITL-186.md @@ -39,10 +39,9 @@ What changes for existing code: not part of the source's identity (ITL-186). - `SpiralDBTableSource(..., max_rows=, batch_rows=)`, the `row_count_estimate` and `table_id` properties, and `orcapod.errors.SourceTooLargeError` (ITL-186). -- A `SpiralDBTableSource` shown in a notebook (its HTML display, or `view()`) reads only its - first 10 rows, in one scan to which `max_rows` does not apply, and its header says how - many rows the table has, for example `first 10 rows of about 63,349`. Other streams - display as before (ITL-186). +- A `SpiralDBTableSource` shown in a notebook (its `head()`) reads only its first 10 rows, + in one scan to which `max_rows` does not apply, and its header says how many rows the + table has, for example `first 10 rows of about 63,349` (ITL-186). - `SpiralDBConnector.scan_batches(table, where=, limit=, asof=)`, `get_arrow_schema`, `get_spiral_table_id`, `get_table_version` and `get_row_count_estimate`: a restricted scan and metadata-only reads (ITL-186). diff --git a/changelog.d/ITL-991.md b/changelog.d/ITL-991.md new file mode 100644 index 000000000..21830317a --- /dev/null +++ b/changelog.d/ITL-991.md @@ -0,0 +1,30 @@ +### Breaking changes and migration + +#### Showing a stream never computes it (ITL-991) + +Displaying a stream, source or node in a notebook no longer computes anything. The preview +shows the first 10 rows that are already available: rows a source holds, rows cached from an +earlier computation, rows a job node holds after a run or has stored, or a bounded read of a +source's table. Otherwise it shows the columns and a note such as "not computed" or "nothing +stored; run the job". Call `as_table()` or run the job to compute. Before, displaying a +function pod's output called the function on every row, displaying an operator's output ran +the operator, displaying a side-effect stream delivered every side effect, and displaying a +`CachedSource` wrote its cache table; none of these happen now. + +`StreamBase.view()` and `StreamView` are removed: use `head(n)` or `as_polars_df()`. + +### Added + +- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a + `StreamHead` (`orcapod.streams`): at most `n` rows as a PyArrow table with their exact + types, the row count when it is known without computing, and a `status` (`HeadStatus`) that + tells an empty result from one that is not computed. It never calls a function, runs an + operator, delivers a side effect, polls a source or writes. The notebook preview is + `head()` rendered, with large cells (arrays, bytes, long text, long lists) summarised. A job + node with nothing in memory reads its whole stored slice to show its first rows. (ITL-991) + +### Fixed + +- An operator's output stream computed again after one of its inputs changed now does so: + before, it served the result kept from before the change (`DynamicPodStream` cleared a + misspelled attribute instead of its kept result). (ITL-991) diff --git a/design_issues/ITL-991-PV1.md b/design_issues/ITL-991-PV1.md new file mode 100644 index 000000000..692a842de --- /dev/null +++ b/design_issues/ITL-991-PV1.md @@ -0,0 +1,22 @@ +## `src/orcapod/core/streams/` — notebook preview and `head()` + +### PV1 — Showing a stream computed it, delivered side effects and wrote caches +**Status:** resolved +**Severity:** high +**Issue:** ITL-991 + +`StreamBase._repr_html_` and `StreamBase.view()` called `as_polars_df()` on the whole +stream. Displaying a function pod's output in a notebook called the function on every row, +an operator's output ran the operator, a `SideEffectPodStream` or `SideEffectNode` delivered +every side effect, a `PollingSource` polled, a `CachedSource` wrote its cache table, and a +blueprint node raised inside the formatter. The developer ruled (November 12, 2026) that a +preview shows only what is readily available and never triggers computation, and that a +node's preview is a head of its storage. + +**Fix:** every stream, source and node has `head(n=10, columns=..., all_info=...)` +returning a `StreamHead` (rows, `total_rows`, `HeadStatus`), and `_repr_html_` renders +`head()`. The default shows the columns and `NOT_COMPUTED`; only classes that can serve rows +without computing override it. `view()` and `StreamView` are removed. A job node with nothing +in memory reads its whole stored slice; a bounded database read is a follow-up. The same +work fixed `DynamicPodStream` serving a stale result after an input changed (misspelled +`_cached_results` in `_update_cache_status`). diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index d96e54b1d..dc7075718 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -152,12 +152,13 @@ node, so 100,000 rows cost about 0.5 GB per node. A table of millions of rows ca with `max_rows=None`, but it cannot yet be run through a job on a laptop. **Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell -(or through `view()`) reads only its first 10 rows, in one scan to which the row cap does not -apply, so a table of any size can be looked at. The header line says what is shown, for +(its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10 +rows, in one scan to which the row cap does not apply, so a table of any size can be looked +at; `head(n)` reads `n`. The header line says what is shown, for example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's recorded row count, read without scanning), `all 4 rows` when the source reads no more than that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with -`where=`. Tag columns come first, marked with `*`. To see more rows, use `as_df()` with a +`where=`. Tag columns come first, and the line above the table names them. To see more rows, use `as_df()` with a `limit=` or `where=` source. **Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index feb7ec058..ebcee55ad 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -100,6 +100,77 @@ df = table.to_pandas() Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_lazy_frame()`. +## Looking at data + +In a notebook, a stream, source or job node shown at the end of a cell renders a short +table: its first rows, with a line saying what they are. That preview is `head()` rendered, +and `head(n)` gives the same rows to work with. Neither ever computes anything: they show +only what is readily available, and never call a function, run an operator, deliver a side +effect, poll a source or write to a database. + +```python +from orcapod import function_pod +from orcapod.sources import DictSource + +subjects = DictSource( + data=[ + {"subject_id": f"mouse_{i:02d}", "age": 8 + i, "weight": 20.0 + i} + for i in range(1, 26) + ], + tag_columns=["subject_id"], + label="subjects", +) + +head = subjects.head() +print(head.header()) +# DictSource[subjects] — first 10 of 25 rows +print(head.status.value, len(head)) +# rows 10 +print(subjects.head(3).as_table().column("age").to_pylist()) +# [9, 10, 11] +``` + +`head()` returns a `StreamHead`: the rows as a PyArrow table (`.table`, `.as_table()`, +`.as_polars_df()`), the row count when it is known without computing (`.total_rows`), and a +`.status` that says where the rows came from. A stream downstream of a computation shows rows +only once they are computed: + +```python +@function_pod(output_keys="age_months") +def to_months(age: int) -> int: + return age * 12 + + +months = to_months.pod(subjects, label="months") +print(months.head().header()) +# FunctionPodStream[months] — not computed; as_table(), iter_data() or a job run computes it +print(months.head().status.value, months.head().table.column_names) +# not_computed ['subject_id', 'age_months'] + +table = months.as_table() # computes every row +print(months.head(3).header()) +# FunctionPodStream[months] — first 3 of 25 computed rows +``` + +What each kind of object shows: + +| Object | `head()` shows | +|---|---| +| A source | its first rows (a bounded read for a source that reads its table lazily) | +| An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` | +| A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) | +| A pipeline blueprint node, an unbound input slot | none (`unavailable`) | +| A side-effect stream | none: showing it never delivers its side effects | + +A head of an operator is never the operator applied to the heads of its inputs: a group over +the first ten rows has the wrong members. So an uncomputed stream shows its columns and +"not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result +shows status `rows` with zero rows, so it is never mistaken for "not computed". + +In the preview, large cells are summarised (an array as `ndarray float64 (2, 3)`, bytes as +`bytes (300 B)`, long text and long lists shortened); `head().table` keeps the full values. +In a terminal, `print(stream.head())` prints the same preview as text. + ## Controlling column visibility with `ColumnConfig` By default, streams only expose user-facing tag and data columns. Orcapod also maintains diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index f40a67cf5..b1b27c562 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -2,6 +2,7 @@ from __future__ import annotations +import dataclasses import logging from datetime import datetime, timezone from collections.abc import Iterator @@ -9,6 +10,7 @@ from orcapod.core.datagrams import Datagram from orcapod.core.function_pod import FunctionPodStream, WrappedFunctionPod +from orcapod.core.streams.head import StreamHead from orcapod.system_constants import constants from orcapod.types import ColumnConfig from orcapod.contexts.stored import StaleTables @@ -451,12 +453,35 @@ class _CachedFunctionPodStream(FunctionPodStream): """ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: - keep = CachedFunctionPod.RESULT_COMPUTED_FLAG for tag, data in super().iter_data(): - stored = [name for name in data.meta_columns if name != keep] - if stored: - data = data.drop_meta_columns(*stored) - yield tag, data + yield tag, _without_stored_meta(data) + + def _computed_prefix( + self, limit: int + ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: + shown, total = super()._computed_prefix(limit) + return [(tag, _without_stored_meta(data)) for tag, data in shown], total + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first computed rows, with the columns ``as_table`` gives. + + The pod's result store is not looked up: that needs the input rows' hashes, + i.e. reading the input. See ``FunctionPodStream.head``. + """ + head = super().head(n, columns=columns, all_info=all_info) + column_config = ColumnConfig.handle_config(columns, all_info=all_info) + if column_config.meta or column_config.all_info: + return head + table = head.table.drop_columns( + [n for n in head.table.column_names if n.startswith(constants.META_PREFIX)] + ) + return dataclasses.replace(head, table=table) def as_table( self, @@ -471,3 +496,10 @@ def as_table( return table.drop_columns( [n for n in table.column_names if n.startswith(constants.META_PREFIX)] ) + + +def _without_stored_meta(data: DataProtocol) -> DataProtocol: + """Return ``data`` with only the ``__computed`` meta column of its stored ones.""" + keep = CachedFunctionPod.RESULT_COMPUTED_FLAG + stored = [name for name in data.meta_columns if name != keep] + return data.drop_meta_columns(*stored) if stored else data diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 4337be2b1..2768b1e53 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import dataclasses import logging from abc import abstractmethod from collections.abc import Callable, Collection, Iterator, Sequence @@ -12,7 +13,18 @@ from orcapod.config import OrcapodConfig from orcapod.core.base import TraceableBase from orcapod.core.data_function import CachedDataFunction, PythonDataFunction -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, + schema_requires_io, +) +from orcapod.core.streams.head import ( + SCHEMA_NEEDS_POLL_NOTE, + STALE_NOTE, + HeadStatus, + StreamHead, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import StoreError, TypeIdentityError from orcapod.utils.store_errors import keep_store_error, note_callback_failure @@ -1099,7 +1111,16 @@ async def _gather() -> list[tuple[TagProtocol, DataProtocol | None]]: def _table_from_pairs( self, pairs: list[tuple[TagProtocol, DataProtocol]] ) -> pa.Table: - """Return the output rows ``pairs`` as one table with every column group.""" + """Return the output rows ``pairs`` as one table with every column group. + + With no rows the table has the declared columns (tags, system tags and data), + typed from the declared schema, as the rest of a head without rows is. + """ + if not pairs: + tag_schema, data_schema = self.output_schema(all_info=True) + return arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) tag_tables: list[pa.Table] = [] all_data = [] data_schema = None @@ -1124,11 +1145,7 @@ def _table_from_pairs( struct_data = converter.python_dicts_to_struct_dicts( all_data, python_schema=python_schema ) - if tag_tables: - all_tags_as_tables = arrow_utils.concat_tables_by_name(tag_tables) - else: - # An empty stream has no row to take a tag table from. - all_tags_as_tables = pa.Table.from_pylist([]) + all_tags_as_tables = arrow_utils.concat_tables_by_name(tag_tables) all_data_as_tables: pa.Table = pa.Table.from_pylist( struct_data, schema=data_schema ) @@ -1148,13 +1165,164 @@ def as_table( column_config = ColumnConfig.handle_config(columns, all_info=all_info) + def all_content_hashes() -> pa.Array[Any]: + if self._cached_content_hash_column is None: + # TODO: verify that order will be preserved + self._cached_content_hash_column = pa.array( + [data.content_hash().to_string() for _, data in self.iter_data()], + type=pa.large_string(), + ) + return self._cached_content_hash_column + + return self._shape_table( + self._cached_output_table, column_config, all_content_hashes + ) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows already computed; never calls the function. + + After a complete computation (``as_table``, or iteration to the end) the rows are + the first of the result. After an iteration that stopped part way they are the + rows computed so far (``PARTIAL``). Before any computation, or once an input + changed, there are none. A ``CachedFunctionPod``'s result store is not looked + up: that needs the input rows' hashes, i.e. reading the input. See + ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + has_results = bool(self._cached_output_datas) or ( + self._cached_output_table is not None + ) + if has_results and self.is_stale: + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE + ) + if has_results and schema_requires_io(self, config): + # The input gave no rows, so its columns are still known only by polling + # it: show the result as computed, without columns. + finished = not self._needs_iterator and self._cached_input_iterator is None + return dataclasses.replace( + StreamHead.without_rows( + self, + HeadStatus.ROWS if finished else HeadStatus.PARTIAL, + pa.table({}), + tag_columns=(), + note=SCHEMA_NEEDS_POLL_NOTE, + ), + total_rows=0, + origin="computed", + ) + tag_columns = self.keys(columns=config)[0] if has_results else () + if self._cached_output_table is not None: + # A complete computation: its table holds the rows; only the shown rows' + # data are read, and only when their content hashes are asked for. + def shown_content_hashes() -> pa.Array[Any]: + shown, _ = self._computed_prefix(n) + return pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ) + + return StreamHead.of_rows( + self, + self._shape_table( + self._cached_output_table.slice(0, n), config, shown_content_hashes + ), + tag_columns=tag_columns, + total_rows=self._cached_output_table.num_rows, + origin="computed", + ) + finished = not self._needs_iterator and self._cached_input_iterator is None + if not finished and not self._cached_output_datas: + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + shown, total = self._computed_prefix(n) + status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL + if not shown: + # Nothing to show (an empty result, inputs that all gave no output so far, + # or n == 0): the declared columns. + table = self._shape_table( + self._table_from_pairs([]), + config, + lambda: pa.array([], type=pa.large_string()), + ) + return StreamHead.of_rows( + self, + table, + tag_columns=tag_columns, + total_rows=total, + status=status, + origin="computed", + ) + return StreamHead.of_rows( + self, + self._shape_table( + self._table_from_pairs(shown), + config, + lambda: pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ), + ), + tag_columns=tag_columns, + total_rows=total, + status=status, + origin="computed", + ) + + def _computed_prefix( + self, limit: int + ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: + """Return the first ``limit`` output rows computed so far, and how many there are. + + The rows are the contiguous run of computed inputs from the first, in the order + iteration yields them; an input whose output is ``None`` (filtered out) is + skipped, as iteration skips it. Only the first ``limit`` rows are collected. + + Args: + limit: The largest number of rows to return. + + Returns: + The rows, and the number of output rows in the whole computed run. + """ + shown: list[tuple[TagProtocol, DataProtocol]] = [] + total = 0 + index = 0 + while index in self._cached_output_datas: + tag, data = self._cached_output_datas[index] + if data is not None: + if total < limit: + shown.append((tag, data)) + total += 1 + index += 1 + return shown, total + + def _shape_table( + self, + table: pa.Table, + column_config: ColumnConfig, + content_hashes: Callable[[], pa.Array[Any]], + ) -> pa.Table: + """Return ``table`` with the column groups of ``column_config``. + + Args: + table: Output rows with every column group (``_table_from_pairs``). + column_config: The column groups to keep, and whether to sort by tags. + content_hashes: Returns the content hash of each row of ``table``; called + only when ``column_config`` asks for them. + """ drop_columns = [] if not column_config.system_tags: # TODO: get system tags more effiicently drop_columns.extend( [ c - for c in self._cached_output_table.column_names + for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ] ) @@ -1163,30 +1331,16 @@ def as_table( if not column_config.context: drop_columns.append(constants.CONTEXT_KEY) - output_table = self._cached_output_table.drop( - [c for c in drop_columns if c in self._cached_output_table.column_names] - ) + output_table = table.drop([c for c in drop_columns if c in table.column_names]) - # lazily prepare content hash column if requested if column_config.content_hash: - if self._cached_content_hash_column is None: - content_hashes = [] - # TODO: verify that order will be preserved - for tag, data in self.iter_data(): - content_hashes.append(data.content_hash().to_string()) - self._cached_content_hash_column = pa.array( - content_hashes, type=pa.large_string() - ) - assert self._cached_content_hash_column is not None, ( - "_cached_content_hash_column should not be None here." - ) hash_column_name = ( "_content_hash" if column_config.content_hash is True else column_config.content_hash ) output_table = output_table.append_column( - hash_column_name, self._cached_content_hash_column + hash_column_name, content_hashes() ) if column_config.sort_by_tags: diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index d38f79d08..6148314b4 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -21,7 +21,7 @@ import logging import uuid from abc import abstractmethod -from collections.abc import Iterator, Mapping +from collections.abc import Callable, Iterator, Mapping, Sequence from typing import TYPE_CHECKING, Any, Literal, NamedTuple, cast from orcapod import contexts @@ -29,7 +29,16 @@ from orcapod.config import OrcapodConfig from orcapod.core.cached_function_pod import CachedFunctionPod from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + BLUEPRINT_NOTE, + HeadStatus, + StreamHead, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.core.datagrams.tag_data import EmptyData, Tag from orcapod.errors import ( @@ -225,6 +234,11 @@ class FunctionNodeBase(StreamBase): shared by both the blueprint and the execution variant. """ + # Set by each subclass's ``__init__``: the output table and content hashes that + # ``as_table`` builds once. + _cached_output_table: "pa.Table | None" + _cached_content_hash_column: "pa.Array[Any] | None" + def __init__( self, function_pod: FunctionPodProtocol, @@ -510,29 +524,18 @@ def as_table( self._cached_output_table = self._table_from_pairs(list(self.iter_data())) column_config = ColumnConfig.handle_config(columns, all_info=all_info) - output_table = arrow_utils.apply_column_config( - self._cached_output_table, column_config, self.keys()[0] - ) - if column_config.content_hash: + def all_content_hashes() -> "pa.Array[Any]": if self._cached_content_hash_column is None: - content_hashes = [] - for tag, data in self.iter_data(): - content_hashes.append(data.content_hash().to_string()) self._cached_content_hash_column = pa.array( - content_hashes, type=pa.large_string() + [data.content_hash().to_string() for _, data in self.iter_data()], + type=pa.large_string(), ) - assert self._cached_content_hash_column is not None - hash_column_name = ( - "_content_hash" - if column_config.content_hash is True - else column_config.content_hash - ) - output_table = output_table.append_column( - hash_column_name, self._cached_content_hash_column - ) + return self._cached_content_hash_column - return output_table + return self._shape_table( + self._cached_output_table, column_config, all_content_hashes + ) def _table_from_pairs( self, pairs: "list[tuple[TagProtocol, DataProtocol]]" @@ -576,6 +579,90 @@ def _table_from_pairs( raise InputValidationError(f"Function node {self.label!r}: {exc}") from exc return arrow_utils.hstack_tables(tag_table, data_table) + def _shape_table( + self, + table: "pa.Table", + column_config: ColumnConfig, + content_hashes: "Callable[[], pa.Array[Any]]", + ) -> "pa.Table": + """Return ``table`` with the column groups of ``column_config``. + + Args: + table: Output rows with every column group (``_table_from_pairs``). + column_config: The column groups to keep. + content_hashes: Returns the content hash of each row of ``table``; called + only when ``column_config`` asks for them. + """ + output_table = arrow_utils.apply_column_config( + table, column_config, self.keys()[0] + ) + if column_config.content_hash: + hash_column_name = ( + "_content_hash" + if column_config.content_hash is True + else column_config.content_hash + ) + output_table = output_table.append_column( + hash_column_name, content_hashes() + ) + return output_table + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: a node of this kind holds no data. See ``StreamBase.head``.""" + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note=BLUEPRINT_NOTE, + ) + + def _head_of_pairs( + self, + pairs: "Sequence[tuple[TagProtocol, DataProtocol | None]]", + n: int, + config: ColumnConfig, + origin: str, + ) -> StreamHead: + """Return the first ``n`` of ``pairs`` as this node's head. + + A ``None`` output (a filtered input) is not a row. An ``EmptyData`` placeholder + (a result kept in an ephemeral store that is gone) has no values to show: it is + left out and counted in the note. + """ + rows = [ + (t, d) for t, d in pairs if d is not None and not isinstance(d, EmptyData) + ] + missing = sum(1 for _, d in pairs if isinstance(d, EmptyData)) + shown = rows[:n] + table = self._shape_table( + self._table_from_pairs(shown), + config, + lambda: pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ), + ) + note = None + if missing: + note = ( + f"{missing:,} {'result' if missing == 1 else 'results'} not shown: " + "kept in an ephemeral store that is gone" + ) + return StreamHead.of_rows( + self, + table, + tag_columns=self.keys(columns=config)[0], + total_rows=len(rows), + origin=origin, + note=note, + ) + def __repr__(self) -> str: return ( f"{type(self).__name__}(data_function={self._data_function!r}, " @@ -2902,6 +2989,81 @@ async def _async_execute_cache_only( finally: await output.close() + # ------------------------------------------------------------------ + # Head + # ------------------------------------------------------------------ + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows the node holds or has stored; never runs the function. + + After a run (or ``execute``) the rows are the node's results in memory, the + first of what ``as_table`` returns. Otherwise they are the first of its stored + rows: the node's whole stored slice is read and nothing is kept on the node. + With nothing stored, there are no rows. See ``StreamBase.head``. + + Raises: + RuntimeError: If the node was loaded and the attached store does not hold + the records its saved run stored (the wrong store). + """ + from orcapod.pipeline.serialization import LoadStatus + + check_head_size(n) + config = head_column_config(columns, all_info) + status = self.load_status + if status == LoadStatus.UNAVAILABLE: + return self._head_without_rows( + config, HeadStatus.UNAVAILABLE, note="no store attached" + ) + current = status == LoadStatus.CACHE_ONLY or not self.is_stale + if current and self._cached_output_datas: + return self._head_of_pairs( + list(self._cached_output_datas.values()), n, config, origin="held" + ) + return self._head_from_store(n, config) + + def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: + """Return the first ``n`` of this node's stored rows. + + Reads the node's whole stored slice (pipeline table joined with the result + table; no bounded read exists yet) and keeps nothing on the node. + """ + if self._function_pod is None: + # A loaded node: its table is built from the stored Arrow records. + self._require_expected_records() + fetched = self._fetch_joined_records() + stored = None if fetched is None else self._stored_rows_stream(fetched) + stored_ids = set(stored[0]) if stored is not None else set() + missing = 0 + if fetched is not None: + missing = sum( + 1 for k in fetched.empty_data_tokens if k not in stored_ids + ) + note = ( + f"{missing:,} {'result' if missing == 1 else 'results'} not shown: kept " + "in an ephemeral store, which is not saved with the job" + if missing + else None + ) + if stored is not None: + return ( + stored[1] + .head(n, columns=config) + .for_stream(self, origin="stored", note=note) + ) + return self._head_without_rows(config, HeadStatus.NOTHING_STORED, note=note) + if self._cached_function_pod is None: + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + entries = list(self._load_cached_entries().values()) + if not any(data is not None for _, data in entries): + return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + return self._head_of_pairs(entries, n, config, origin="stored") + # ------------------------------------------------------------------ # Iteration # ------------------------------------------------------------------ diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 36dcc906e..9f22703ec 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -27,7 +27,18 @@ from orcapod.config import OrcapodConfig from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + STALE_NOTE, + BLUEPRINT_NOTE, + HeadStatus, + StreamHead, + head_of, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( PipelineJobRequiredError, @@ -381,6 +392,21 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: return # pragma: no cover yield # pragma: no cover + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: a blueprint node holds no data.""" + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note=BLUEPRINT_NOTE, + ) + def as_table( self, *, @@ -1291,6 +1317,66 @@ def as_table( empty_stream = ArrowTableStream(empty_records, tag_columns=tag_keys) return empty_stream.as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows the node holds or has stored; never runs the operator. + + After a run (or ``execute``) the rows are the node's output in memory. Before + that, a node in ``CacheMode.REPLAY`` shows the first of its stored rows, read + from its whole stored table; in ``LOG`` and ``OFF`` it shows none, as + ``as_table`` does. A loaded node never raises here. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + if self._cached_output_stream is not None: + if not self._loaded and self.is_stale: + # run() would recompute; the kept rows are not the current result. + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE + ) + return head_of(self._cached_output_stream.head(n, columns=config), self) + if self._cache_mode == CacheMode.REPLAY and self._pipeline_database is not None: + return self._head_from_store(n, config) + if self._loaded: + return self._head_without_rows( + config, + HeadStatus.UNAVAILABLE, + note=( + "loaded node whose results are not stored (cache mode OFF)" + if self._cache_mode == CacheMode.OFF + else "loaded node whose stored results are not replayed " + f"(cache mode {self._cache_mode.value.upper()})" + ), + ) + if self._cache_mode == CacheMode.LOG and self._pipeline_database is not None: + note = ( + "not run in this session; rows of earlier runs are stored but not " + "replayed (cache mode LOG)" + ) + else: + note = "not run; cache mode OFF stores nothing" + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED, note=note) + + def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: + """Return the first ``n`` of this node's stored rows. + + Reads the node's whole stored slice (no bounded read exists yet) through the + same path as ``as_table`` and keeps nothing on the node. + + Raises: + RuntimeError: If the node was loaded and its cache table is absent from + the attached store (the wrong store). + """ + stream = self._load_cached_stream_from_db() + if stream is None or stream.as_table().num_rows == 0: + return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + return stream.head(n, columns=config).for_stream(self, origin="stored") + # ------------------------------------------------------------------ # DB retrieval # ------------------------------------------------------------------ @@ -1483,12 +1569,20 @@ async def forward() -> None: # does, and store it in LOG mode (sync DB write, post-hoc). A run # with no rows replaces an earlier run's output too: the node then # reads as a zero-row table. - self._cached_output_stream = None if collected: stream = StaticOutputOperatorPod._materialize_to_stream(collected) self._cached_output_stream = stream if should_store: self._store_output_stream(stream) + else: + # A completed run with no rows keeps a typed zero-row result, as the + # sync path does, so the node reads as computed and empty (not as + # never run). Nothing is stored for it. + self._cached_output_stream = ArrowTableStream( + self._make_empty_table(), + tag_columns=self.keys()[0], + data_context=self.data_context, + ) self._update_modified_time() diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index 4b42b81b4..eaa888091 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -16,9 +16,21 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.base import TraceableBase +from orcapod.core.streams.base import schema_requires_io +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + head_of, + render_error_html, +) +from orcapod.core.streams.base import ( + check_head_size, + head_column_config, +) from orcapod.errors import SourceSpecMismatchError, UnboundSourceError from orcapod.protocols.core_protocols import DataProtocol, TagProtocol from orcapod.types import ColumnConfig, Schema +from orcapod.utils import arrow_utils from orcapod.utils.arrow_utils import system_tag_column_names from orcapod.utils.schema_utils import compute_source_schema_hash @@ -291,6 +303,44 @@ def as_table( "Use PipelineJob.bind() to attach data before calling as_table()." ) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: an input slot holds no data until a source is bound. + + See ``StreamBase.head``. + """ + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note="input slot; bind a source", + ) + + def _head_without_rows( + self, config: ColumnConfig, status: HeadStatus, note: str | None = None + ) -> StreamHead: + """Return a head with no rows: the slot's columns from its declared schema.""" + tag_schema, data_schema = self.output_schema(columns=config) + table = arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) + return StreamHead.without_rows( + self, status, table, tag_columns=tuple(tag_schema.keys()), note=note + ) + + def _repr_html_(self) -> str: + """Render ``head()`` for a notebook; never computes.""" + try: + head = self.head() + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(self, error) + return head._repr_html_() + async def async_iter_data(self): """Asynchronous iterator over (tag, data) pairs. @@ -570,6 +620,31 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: ) return self._bound_source.iter_data() + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the bound source.""" + return self._bound_source is not None and schema_requires_io( + self._bound_source, config + ) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the bound source's head, or no rows when no source is bound.""" + if self._bound_source is None: + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note="no source bound", + ) + return head_of( + self._bound_source.head(n, columns=columns, all_info=all_info), self + ) + async def async_iter_data( self, ) -> AsyncIterator[tuple[TagProtocol, DataProtocol]]: diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 748f41381..34ffa81db 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -12,7 +12,17 @@ from orcapod.contexts import DataContext from orcapod.core.base import TraceableBase from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + STALE_NOTE, + HeadStatus, + StreamHead, + head_of, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( ArgumentGroup, @@ -469,7 +479,7 @@ def _update_cache_status(self) -> None: upstream_times.append(self._pod.last_modified) if any(t is None for t in upstream_times): - self._cached_results = None + self._cached_stream = None self._cached_time = None return @@ -478,7 +488,7 @@ def _update_cache_status(self) -> None: # Invalidate cache if upstream is newer and update the cache time if max_upstream_time > self._cached_time: - self._cached_results = None + self._cached_stream = None self._cached_time = max_upstream_time def run(self, *args: Any, **kwargs: Any) -> None: @@ -512,5 +522,32 @@ def iter_data( ) return self._cached_stream.iter_data() + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows of the operator's kept result; never runs the operator. + + The head of an operator is not the operator over the heads of its inputs (a + group over ten rows has the wrong members), so before the result is computed, + or after an input changed, there are no rows. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + had_result = self._cached_stream is not None + self._update_cache_status() + if self._cached_stream is not None and self._cached_time is not None: + return head_of( + self._cached_stream.head(n, columns=config), self, origin="computed" + ) + return self._head_without_rows( + config, + HeadStatus.NOT_COMPUTED, + note=STALE_NOTE if had_result else None, + ) + def __repr__(self) -> str: return f"{self.__class__.__name__}(kernel={self.producer}, upstreams={self.upstreams})" diff --git a/src/orcapod/core/sources/base.py b/src/orcapod/core/sources/base.py index e40fb2730..195473241 100644 --- a/src/orcapod/core/sources/base.py +++ b/src/orcapod/core/sources/base.py @@ -5,6 +5,7 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import StreamProtocol from orcapod.types import ColumnConfig, Schema @@ -186,6 +187,32 @@ def as_table( """Delegate to the underlying stream's as_table.""" return self._stream.as_table(columns=columns, all_info=all_info) + def _display_name(self) -> str: + """The name a preview header shows: the label when one was given, else the + source id (a source id is the source's name), e.g. a table's name. The label + itself is not changed.""" + if self.has_assigned_label or self._source_id is None: + return self.label + return self._source_id + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first ``n`` rows of the table the source holds, without computing. + + A source that reads its table when it is built holds it in memory, so this is a + slice of it. A source that reads its table later overrides this with a bounded + read. See ``StreamBase.head``. + """ + # _stream is set by each concrete subclass (see the class docstring), as for + # the other delegating methods here. + stream = self._stream # pyright: ignore[reportAttributeAccessIssue] + return stream.head(n, columns=columns, all_info=all_info).for_stream(self) + # ------------------------------------------------------------------------- # Convenience — caching # ------------------------------------------------------------------------- diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index e3a1079f7..4e3847172 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -1,5 +1,6 @@ from __future__ import annotations + import logging from collections.abc import Iterator from typing import TYPE_CHECKING, Any @@ -8,6 +9,12 @@ from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.base import check_head_size, schema_requires_io +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + head_of, +) from orcapod.contexts.stored import ( keep_rows_of_data_context, require_same_data_context, @@ -405,6 +412,43 @@ def as_table( assert self._cached_stream is not None return self._cached_stream.as_table(columns=columns, all_info=all_info) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the wrapped source.""" + return schema_requires_io(self._source, config) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows without reading or writing the cache. + + Once the source has been used, its merged stream is in memory and this is a slice + of it. Before that its rows are not known: they are the cache's earlier rows + merged with the wrapped source's, and reading them writes the cache. The head + then has the wrapped source's columns, no rows and ``NOT_COMPUTED`` (or the + wrapped source's own reason when it has none to give). + """ + check_head_size(n) + if self._cached_stream is not None and not self.is_stale: + return head_of( + self._cached_stream.head(n, columns=columns, all_info=all_info), self + ) + wrapped = head_of( + self._source.head(0, columns=columns, all_info=all_info), self + ) + if wrapped.status is not HeadStatus.ROWS: + return wrapped + return StreamHead.without_rows( + self, + HeadStatus.NOT_COMPUTED, + wrapped.table, + tag_columns=wrapped.tag_columns, + note="not read yet; as_table() or iter_data() reads and updates the cache", + ) + def get_all_records( self, columns: ColumnConfig | dict[str, Any] | None = None, diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index f9ad67ab1..26c25f02a 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -1,5 +1,6 @@ from __future__ import annotations +import dataclasses from collections.abc import Mapping from typing import TYPE_CHECKING, Any, ClassVar @@ -7,6 +8,15 @@ from orcapod.core.sources.source_ids import hash_source_id from orcapod.core.sources.stream_builder import SourceStreamBuilder from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, +) +from orcapod.core.streams.base import ( + check_head_size, + head_column_config, + schema_requires_io, +) from orcapod.errors import InputValidationError from orcapod.system_constants import constants from orcapod.types import ColumnConfig, ContentHash, Schema @@ -405,6 +415,17 @@ def _build_derived_stream( has_rows = table is not None if table is None: table = self._empty_origin_table() + return self._derive_from(table, tags_as_data), has_rows + + def _derive_from( + self, table: "pa.Table", tags_as_data: Mapping[str, str] + ) -> ArrowTableStream: + """Append the tag copies to the origin rows ``table`` and enrich it as a source. + + Args: + table: Rows of the origin. + tags_as_data: Maps a tag column of the origin to the name of its copy. + """ for tag, name in tags_as_data.items(): # The copy is the tag's own Arrow column under a renamed field, so its # type, nullability and field metadata are the tag's exactly. @@ -418,7 +439,7 @@ def _build_derived_stream( source_id=self.source_id, record_id_column=self._record_id_column, ) - return result.stream, has_rows + return result.stream def _get_stream(self) -> ArrowTableStream: if self._tags_as_data is not None: @@ -440,6 +461,87 @@ def _get_stream(self) -> ArrowTableStream: tag_keys = self._origin.keys()[0] return ArrowTableStream(self._cached_table, tag_columns=tag_keys) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the origin.""" + return schema_requires_io(self._origin, config) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows of the origin's records, without computing. + + The records are the ones this source already read. Otherwise, an origin source + gives its own ``head(n)`` (so a polling or caching origin is never polled or + written), and an origin job node gives its stored rows, read now and not kept. + An origin job node that has not run gives no rows. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + origin_kind = "held" if isinstance(self._origin, RootSource) else "stored" + if self._tags_as_data is not None and self._cached_stream is not None: + return self._cached_stream.head(n, columns=config).for_stream( + self, origin=origin_kind + ) + if self._tags_as_data is None and self._cached_table is not None: + stream = ArrowTableStream( + self._cached_table, tag_columns=self._origin.keys()[0] + ) + return stream.head(n, columns=config).for_stream(self, origin=origin_kind) + if isinstance(self._origin, RootSource): + origin_head = self._origin.head(n) + if origin_head.status is not HeadStatus.ROWS: + if origin_head.table.num_columns == 0: + # The origin's columns are not known without polling it. + return origin_head.for_stream(self) + # No rows: this source's own columns (tag copies included), with the + # origin's status and explanation. + empty = self._stream_over(origin_head.table.slice(0, 0)) + return dataclasses.replace( + StreamHead.without_rows( + self, + origin_head.status, + empty.as_table(columns=config), + tag_columns=empty.keys(columns=config)[0], + note=origin_head.note, + ), + total_rows=origin_head.total_rows, + total_is_estimate=origin_head.total_is_estimate, + description=origin_head.description, + ) + stream = self._stream_over(origin_head.table) + return dataclasses.replace( + stream.head(n, columns=config).for_stream(self), + total_rows=origin_head.total_rows, + total_is_estimate=origin_head.total_is_estimate, + origin=origin_head.origin, + note=origin_head.note, + ) + records = self._origin_records() + if records is None or records.num_rows == 0: + stream = self._stream_over(self._empty_origin_table()) + return StreamHead.without_rows( + self, + HeadStatus.NOTHING_STORED, + stream.as_table(columns=config), + tag_columns=stream.keys(columns=config)[0], + note="nothing stored: its origin has not run", + ) + # Only the shown rows are enriched; the total is the whole stored count. + head = self._stream_over(records.slice(0, n)).head(n, columns=config) + return dataclasses.replace( + head.for_stream(self, origin="stored"), total_rows=records.num_rows + ) + + def _stream_over(self, table: "pa.Table") -> ArrowTableStream: + """Return this source's stream over the origin rows ``table``.""" + if self._tags_as_data is not None: + return self._derive_from(table, self._tags_as_data) + return ArrowTableStream(table, tag_columns=self._origin.keys()[0]) + def iter_data(self): return self._get_stream().iter_data() diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index 1de88b4a2..b45b0a335 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -20,6 +20,14 @@ from orcapod.core.sources.base import RootSource from orcapod.core.sources.stream_builder import SourceStreamBuilder +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, +) +from orcapod.core.streams.base import ( + check_head_size, + head_column_config, +) from orcapod.errors import ( CursorInvalidatedError, InputValidationError, @@ -436,6 +444,71 @@ def keys( self._sync_poll_and_commit() return self._get_combined_stream().keys(columns=columns, all_info=all_info) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll. + + True while nothing was polled and either no schema was declared, or ``config`` + asks for a column group known only from rows (source info, meta, context, + content hashes): only a poll tells those columns then. + """ + if self._batches: + return False + if self._tag_schema is None or self._data_schema is None: + return True + return bool( + config.meta or config.source or config.context or config.content_hash + ) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows already polled; never polls. + + Before the first poll there are no rows. The columns are the declared schema's, + or none when the source declared no schema (finding it would mean polling). + """ + check_head_size(n) + config = head_column_config(columns, all_info) + batches = list(self._batches) # snapshot: the list is append-only + if batches: + # Combine only the batches that hold the first n rows; count the rest. + sizes = [batch.head(0).total_rows or 0 for batch in batches] + needed, covered = 0, 0 + while needed < len(batches) and (covered < n or needed == 0): + covered += sizes[needed] + needed += 1 + prefix = batches[0] + for batch in batches[1:needed]: + prefix = self._combine(prefix, batch) + return dataclasses.replace( + prefix.head(n, columns=config).for_stream(self), + total_rows=sum(sizes), + ) + if self._tag_schema is not None and self._data_schema is not None: + # The declared schema answers without a fetch, system tags included; the + # column groups that need data (source info, context, ...) are left out. + tag_schema, data_schema = self.output_schema( + columns=ColumnConfig(system_tags=config.system_tags) + ) + table = arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) + tag_columns = tuple(tag_schema.keys()) + else: + table = pa.table({}) + tag_columns = () + return StreamHead.without_rows( + self, + HeadStatus.NOT_COMPUTED, + table, + tag_columns=tag_columns, + note="nothing polled yet; iterating it or a job run polls it", + ) + def iter_data(self): """Iterate over (tag, data) pairs from the current snapshot.""" self._sync_poll_and_commit() diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index 3612f1b29..6714072c9 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -13,6 +13,15 @@ from typing import TYPE_CHECKING, Any from orcapod.core.sources.base import RootSource +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + head_of, +) +from orcapod.core.streams.base import ( + check_head_size, + head_column_config, +) from orcapod.protocols.core_protocols import SourceProtocol from orcapod.types import ContentHash, Schema @@ -202,6 +211,26 @@ def as_table( ) -> pa.Table: return self._require_delegate().as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the bound source's head, or no rows when no live source is bound.""" + if self._delegate is not None: + return head_of( + self._delegate.head(n, columns=columns, all_info=all_info), self + ) + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, + HeadStatus.UNAVAILABLE, + note="source not available; bind a live source", + ) + def resolve_field(self, record_id: str, field_name: str) -> Any: """Delegate to bound source, or raise.""" return self._require_delegate().resolve_field(record_id, field_name) diff --git a/src/orcapod/core/sources/spiraldb_table_source.py b/src/orcapod/core/sources/spiraldb_table_source.py index 054601174..eaf1abf8d 100644 --- a/src/orcapod/core/sources/spiraldb_table_source.py +++ b/src/orcapod/core/sources/spiraldb_table_source.py @@ -34,6 +34,7 @@ from __future__ import annotations import asyncio +import dataclasses import datetime from collections.abc import AsyncIterator, Collection, Generator, Iterator, Mapping from contextlib import contextmanager @@ -43,6 +44,8 @@ from orcapod.core.sources.derived_source import validate_tags_as_data from orcapod.core.sources.stream_builder import SourceStreamBuilder from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.base import check_head_size, head_column_config +from orcapod.core.streams.head import HeadStatus, StreamHead from orcapod.databases.spiraldb_connector import SpiralDBConnector from orcapod.errors import InputValidationError, SourceTooLargeError from orcapod.types import ColumnConfig @@ -162,9 +165,9 @@ class SpiralDBTableSource(RootSource): ``as_table`` and the dataframe methods hold every row they return. Nothing is cached: every read scans again. - Display. Shown in a notebook (``_repr_html_``, ``view``), the source reads only its - first ``PREVIEW_ROWS`` rows (10), in one scan to which ``max_rows`` does not apply, - and the header says how many rows the table has. + Display. ``head(n)`` and the notebook display (its ``head()``) read only the first + ``n`` rows (``PREVIEW_ROWS``, 10, by default), in one scan to which ``max_rows`` does + not apply, and the header says how many rows the table has. Identity. The source is identified by where the table is, never by its rows. The default ``source_id`` is ``/.@``, where @@ -674,15 +677,49 @@ def _head_note(self, head: "_Head") -> str: ) return f"first {shown:,} rows of about {head.total_rows:,}" - def _preview( + def head( self, + n: int = PREVIEW_ROWS, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, - ) -> "tuple[pl.DataFrame, str | None]": - """Return the first ``PREVIEW_ROWS`` rows for a notebook display, and the note.""" - head = self._head(PREVIEW_ROWS, columns=columns, all_info=all_info) - return pl.DataFrame(head.rows), self._head_note(head) + ) -> StreamHead: + """Return the source's first ``n`` rows, read in one bounded scan. + + At most ``n`` rows are read (fewer when the source's ``limit`` is smaller), with + ``where`` and the pinned version applied and ``max_rows`` not, so a table of any + size can be looked at. The total is exact when these are all the rows the + source reads, else the table's recorded row count (an estimate). The notebook + preview is this head. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + head = self._head(n, columns=config) + tag_columns = self.keys(columns=config)[0] + if not head.rows_read: + empty = StreamHead.without_rows( + self, HeadStatus.NOT_COMPUTED, head.rows, tag_columns=tag_columns + ) + return dataclasses.replace( + empty, + total_rows=head.total_rows, + total_is_estimate=True, + description=self._head_note(head), + ) + return StreamHead.of_rows( + self, + head.rows, + tag_columns=tag_columns, + total_rows=head.total_rows, + total_is_estimate=head.total_is_estimate, + origin="read", + description=self._head_note(head), + ) + + def _display_name(self) -> str: + """The name a preview header shows: the label when one was given, else the + table's name (the default source id also carries the table's id).""" + return self.label if self.has_assigned_label else self._table_name # ------------------------------------------------------------------------- # Serialization diff --git a/src/orcapod/core/streams/__init__.py b/src/orcapod/core/streams/__init__.py index 752c876bf..36f9302a0 100644 --- a/src/orcapod/core/streams/__init__.py +++ b/src/orcapod/core/streams/__init__.py @@ -1,7 +1,10 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import HeadStatus, StreamHead __all__ = [ "ArrowTableStream", + "HeadStatus", "StreamBase", + "StreamHead", ] diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 77afe8944..721f59e36 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -1,12 +1,18 @@ from __future__ import annotations +import itertools import logging from collections.abc import Collection, Iterator from typing import TYPE_CHECKING, Any from orcapod import contexts from orcapod.core.datagrams import Data, Tag -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import PodProtocol, StreamProtocol, TagProtocol from orcapod.protocols.hashing_protocols import PipelineElementProtocol from orcapod.system_constants import constants @@ -224,14 +230,55 @@ def as_table( This is useful for converting the stream to a table format. """ columns_config = ColumnConfig.handle_config(columns, all_info=all_info) + return self._assemble_table(columns_config, limit=None) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first ``n`` rows of the table this stream holds. + + Only those rows are assembled: content hashes, when asked for, are computed for + them alone. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + return StreamHead.of_rows( + self, + self._assemble_table(config, limit=n), + tag_columns=self.keys(columns=config)[0], + total_rows=self._table.num_rows, + ) + + def _assemble_table( + self, columns_config: ColumnConfig, limit: int | None + ) -> "pa.Table": + """Return the stream's rows with the column groups of ``columns_config``. + + Args: + columns_config: The column groups to include, and whether to sort by tags. + limit: Assemble only the first ``limit`` rows; ``None`` for all of them. + """ output_table = self._table + data_context_table = self._data_context_table + source_info_table = self._source_info_table + if limit is not None: + output_table = output_table.slice(0, limit) + data_context_table = data_context_table.slice(0, limit) + source_info_table = source_info_table.slice(0, limit) if columns_config.content_hash: hash_column_name = ( "_content_hash" if columns_config.content_hash is True else columns_config.content_hash ) - content_hashes = [str(data.content_hash()) for _, data in self.iter_data()] + rows = self.iter_data() + if limit is not None: + rows = itertools.islice(rows, limit) + content_hashes = [str(data.content_hash()) for _, data in rows] output_table = output_table.append_column( pa.field(hash_column_name, pa.large_string(), nullable=False), pa.array(content_hashes, type=pa.large_string()), @@ -241,9 +288,9 @@ def as_table( output_table = output_table.drop_columns(list(self._system_tag_columns)) table_stack = (output_table,) if columns_config.context: - table_stack += (self._data_context_table,) + table_stack += (data_context_table,) if columns_config.source: - table_stack += (self._source_info_table,) + table_stack += (source_info_table,) table = arrow_utils.hstack_tables(*table_stack) diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 9805cf35f..af13d5540 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -1,6 +1,6 @@ from __future__ import annotations -import html +import dataclasses import logging from abc import abstractmethod from collections.abc import AsyncIterator, Collection, Iterator, Mapping @@ -8,6 +8,13 @@ from typing import TYPE_CHECKING, Any from orcapod.core.base import TraceableBase +from orcapod.core.streams.head import ( + SCHEMA_NEEDS_POLL_NOTE, + HeadStatus, + StreamHead, + default_note, + render_error_html, +) from orcapod.protocols.core_protocols import ( DataProtocol, PodProtocol, @@ -15,6 +22,7 @@ TagProtocol, ) from orcapod.types import ColumnConfig, Schema +from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -33,6 +41,59 @@ logger = logging.getLogger(__name__) +def schema_requires_io(stream: object, config: ColumnConfig) -> bool: + """Whether reading ``stream``'s schema with ``config`` would read or poll a source. + + A stream that does not say (``_schema_requires_io``) is taken not to. + + Args: + stream: The stream whose schema would be read. + config: The column groups the schema would be read with: source info, meta, + context and content hashes may be known only from rows. + """ + check = getattr(stream, "_schema_requires_io", None) + return bool(check(config)) if callable(check) else False + + +def check_head_size(n: int) -> None: + """Check the row count given to ``head``. + + Raises: + TypeError: If ``n`` is not an ``int``. + ValueError: If ``n`` is negative. + """ + if isinstance(n, bool) or not isinstance(n, int): + raise TypeError(f"head() takes an int number of rows, got {type(n).__name__}") + if n < 0: + raise ValueError(f"head() takes a non-negative number of rows n, got {n}") + + +def head_column_config( + columns: ColumnConfig | dict[str, Any] | None, all_info: bool +) -> ColumnConfig: + """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. + + The column groups mean what they mean for ``as_table``. ``head`` returns rows in + stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. + + Raises: + ValueError: If ``columns`` asks for ``sort_by_tags``. + """ + if isinstance(columns, ColumnConfig): + asked_to_sort = bool(columns.sort_by_tags) + elif isinstance(columns, Mapping): + asked_to_sort = bool(columns.get("sort_by_tags", False)) + else: + asked_to_sort = False + if asked_to_sort: + raise ValueError( + "head() returns the first rows in stream order and cannot sort them by " + "tags; sort the result of as_table() instead" + ) + config = ColumnConfig.handle_config(columns, all_info=all_info) + return dataclasses.replace(config, sort_by_tags=False) + + class StreamBase(TraceableBase): @property @abstractmethod @@ -441,66 +502,88 @@ def flow( """ return [e for e in self.iter_data()] - def _preview( + def head( self, + n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, - ) -> "tuple[pl.DataFrame, str | None]": - """Return the rows a notebook display of the stream shows, and a note on them. + ) -> StreamHead: + """Return at most ``n`` rows that are readily available, without computing. + + The rows are ones the stream already holds, has cached from an earlier + computation, has stored, or can read with a bounded read of its table. ``head`` + never calls a function, runs an operator, delivers a side effect, polls a source + or writes anything. When rows are shown, they are the first ``n`` rows that + ``as_table()`` would return. The notebook preview of a stream is its ``head()``. - ``_repr_html_`` and ``view`` show what this returns. The default is every row of - the stream and no note. A stream whose rows are costly to read overrides it to - show fewer, and says so in the note. + This default shows no rows: a stream class that can serve rows without computing + overrides it. The zero-row table then has the stream's columns, typed from its + declared (Python) schema, so a column's Arrow width can differ from the one the + computed stream will have (``int`` shows as ``int64``). Args: - columns: Column configuration, as for ``as_table``. - all_info: Include all column groups, as for ``as_table``. + n: The largest number of rows to return. + columns: Column groups to include, as for ``as_table``. ``sort_by_tags`` is + not accepted: the rows come in stream order. + all_info: Include every column group, as for ``as_table`` (without sorting). Returns: - The rows to show and a note for the header line, or ``None`` for no note. + A ``StreamHead`` with the rows and a ``status`` that tells an empty result + from one that is not computed. + + Raises: + TypeError: If ``n`` is not an ``int``. + ValueError: If ``n`` is negative or ``columns`` asks for ``sort_by_tags``. """ - return self.as_polars_df(columns=columns, all_info=all_info), None + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) - def _preview_header(self, note: str | None) -> str: - """Return the header line a notebook display puts above the rows, HTML-escaped.""" - header = f"{self.__class__.__name__}[{self.label}]" - return html.escape( - header if note is None else f"{header} — {note}", quote=False + def _head_without_rows( + self, + config: ColumnConfig, + status: HeadStatus, + note: str | None = None, + ) -> StreamHead: + """Return a head with no rows: the stream's columns from its declared schema. + + When reading the schema would poll a source (``schema_requires_io``), the head + has no columns either, and the note says so. + """ + if schema_requires_io(self, config): + return StreamHead.without_rows( + self, + status, + pa.table({}), + tag_columns=(), + note=f"{note or default_note(status)}; {SCHEMA_NEEDS_POLL_NOTE}", + ) + tag_schema, data_schema = self.output_schema(columns=config) + table = arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) + return StreamHead.without_rows( + self, status, table, tag_columns=tuple(tag_schema.keys()), note=note ) - def _repr_html_(self) -> str: - df, note = self._preview() - # reorder columns - new_column_order = [c for c in df.columns if c in self.keys()[0]] + [ - c for c in df.columns if c not in self.keys()[0] - ] - df = df[new_column_order] - tag_map = {t: f"*{t}" for t in self.keys()[0]} - # TODO: construct repr html better - df = df.rename(tag_map) - return self._preview_header(note) + "\n" + df._repr_html_() - - def view( - self, - *, - columns: ColumnConfig | dict[str, Any] | None = None, - all_info: bool = False, - ) -> "StreamView": - df, note = self._preview(columns=columns, all_info=all_info) - tag_map = {t: f"*{t}" for t in self.keys()[0]} - # TODO: construct repr html better - df = df.rename(tag_map) - return StreamView(self, df, self._preview_header(note)) - - -class StreamView: - def __init__( - self, stream: StreamBase, view_df: "pl.DataFrame", header: str - ) -> None: - self._stream = stream - self._view_df = view_df - self._header = header + def _display_name(self) -> str: + """The name a preview header shows: the label. Never used as an identity.""" + return self.label + + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading this stream's schema with ``config`` would read or poll a source. + + True when an upstream's would: a ``PollingSource`` that has not polled yet + answers its schema only by polling when it declared no schema, or when + ``config`` asks for a column group known only from rows. + """ + return any(schema_requires_io(upstream, config) for upstream in self.upstreams) def _repr_html_(self) -> str: - return self._header + "\n" + self._view_df._repr_html_() + """Render ``head()`` for a notebook; never computes.""" + try: + head = self.head() + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(self, error) + return head._repr_html_() diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py new file mode 100644 index 000000000..a794fcc1a --- /dev/null +++ b/src/orcapod/core/streams/head.py @@ -0,0 +1,547 @@ +"""The first rows of a stream that are readily available, and their notebook preview (ITL-991). + +``head(n)`` on a stream, source or node returns a ``StreamHead``: at most ``n`` rows the +object already holds, has cached from an earlier computation, has stored, or can read from +its table with a bounded read. It never computes: no user function is called, no operator +is run, no side effect is delivered, no source is polled and nothing is written. The +notebook preview of a stream is its ``head()`` rendered by ``StreamHead._repr_html_``. +""" + +from __future__ import annotations + +import ast +import dataclasses +import html +import importlib +from collections.abc import Sequence +from dataclasses import dataclass +from enum import Enum +from typing import TYPE_CHECKING, Any, cast + +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import polars as pl + import pyarrow as pa + import pyarrow.compute as pc +else: + pa = LazyModule("pyarrow") + pc = LazyModule("pyarrow.compute") + pl = LazyModule("polars") + + +class HeadStatus(str, Enum): + """Where the rows of a ``StreamHead`` came from, or why it has none. + + Attributes: + ROWS: The rows are the first rows of the stream's complete content. Zero rows + means the content is empty. + PARTIAL: A computation stopped part way; the rows are those computed so far. + NOT_COMPUTED: Nothing is computed, or what was computed is out of date. + NOTHING_STORED: A job node whose store holds no rows for it yet. + UNAVAILABLE: No rows can exist here: a pipeline blueprint, an input slot with no + source bound, or a source that is not available. + """ + + ROWS = "rows" + PARTIAL = "partial" + NOT_COMPUTED = "not_computed" + NOTHING_STORED = "nothing_stored" + UNAVAILABLE = "unavailable" + + +NOT_COMPUTED_NOTE = "not computed; as_table(), iter_data() or a job run computes it" +STALE_NOTE = "not computed: inputs changed since the last computation" +NOTHING_STORED_NOTE = "nothing stored; run the job" +BLUEPRINT_NOTE = "pipeline blueprint; run it in a PipelineJob" +SIDE_EFFECTS_NOTE = ( + "not run; iterating it or a job run delivers its side effects; " + "displaying never delivers side effects" +) + +_DEFAULT_NOTES = { + HeadStatus.NOT_COMPUTED: NOT_COMPUTED_NOTE, + HeadStatus.NOTHING_STORED: NOTHING_STORED_NOTE, + HeadStatus.UNAVAILABLE: "no rows available", +} + +SCHEMA_NEEDS_POLL_NOTE = "columns not known until its source is polled" + + +def default_note(status: HeadStatus) -> str: + """Return the note shown for a head with ``status`` and no rows of its own.""" + return _DEFAULT_NOTES[status] + + +# Words naming the rows of a ``ROWS`` head, by where they came from. +_ORIGIN_NOUNS = { + "held": "rows", + "read": "rows", + "computed": "computed rows", + "stored": "stored rows", +} + +# Display limits of the preview. +_MAX_TEXT = 80 +_TEXT_START = 56 +_TEXT_END = 20 +_MAX_LIST_ITEMS = 5 + + +@dataclass(frozen=True) +class StreamHead: + """At most ``n`` readily available rows of a stream, and how they were obtained. + + Attributes: + table: The rows, with the exact Arrow types the stream gives them. When the + stream has no rows to show, a zero-row table with its columns. + status: Where the rows came from, or why there are none. + kind: Class name of the stream, source or node shown. + label: The name shown for it: its label, or for a source given no label its + source id (``display_name``). Never used as an identity. + tag_columns: The columns of ``table`` that are tags. + total_rows: The stream's row count when it is known without computing, else + ``None``. For ``PARTIAL``, the number of rows computed so far. + total_is_estimate: ``True`` when ``total_rows`` is a recorded estimate. + origin: What the rows are: ``"held"`` (in memory), ``"read"`` (a bounded read + of a table), ``"computed"`` (cached from an earlier computation) or + ``"stored"`` (a job node's store). + note: For a head without rows, the explanation shown instead of the default; + for a head with rows, a remark shown after the row count. + description: When given, the whole text after the name in the header, written + by a stream that words its own rows (a table-backed source naming its + ``where``); otherwise the header is built from the fields above. + """ + + table: pa.Table + status: HeadStatus + kind: str + label: str + tag_columns: tuple[str, ...] = () + total_rows: int | None = None + total_is_estimate: bool = False + origin: str = "held" + note: str | None = None + description: str | None = None + + @classmethod + def of_rows( + cls, + stream: Any, + table: pa.Table, + *, + tag_columns: tuple[str, ...], + total_rows: int | None, + status: HeadStatus = HeadStatus.ROWS, + total_is_estimate: bool = False, + origin: str = "held", + note: str | None = None, + description: str | None = None, + ) -> StreamHead: + """Return a head with rows of ``stream``.""" + return cls( + table=table, + status=status, + kind=type(stream).__name__, + label=display_name(stream), + tag_columns=tuple(c for c in tag_columns if c in table.column_names), + total_rows=total_rows, + total_is_estimate=total_is_estimate, + origin=origin, + note=note, + description=description, + ) + + @classmethod + def without_rows( + cls, + stream: Any, + status: HeadStatus, + table: pa.Table, + *, + tag_columns: tuple[str, ...], + note: str | None = None, + ) -> StreamHead: + """Return a head of ``stream`` with no rows: ``table`` holds its columns only.""" + return cls( + table=table.slice(0, 0), + status=status, + kind=type(stream).__name__, + label=display_name(stream), + tag_columns=tuple(c for c in tag_columns if c in table.column_names), + note=note, + ) + + def for_stream( + self, stream: Any, *, origin: str | None = None, note: str | None = None + ) -> StreamHead: + """Return the same rows shown as ``stream`` (a source over its inner stream, a node + over its cached output), optionally with another origin or note.""" + return dataclasses.replace( + self, + kind=type(stream).__name__, + label=display_name(stream), + origin=self.origin if origin is None else origin, + note=self.note if note is None else note, + ) + + # ------------------------------------------------------------------ + # Data access + # ------------------------------------------------------------------ + + def as_table(self) -> pa.Table: + """Return the rows as a PyArrow table.""" + return self.table + + def as_polars_df(self) -> pl.DataFrame: + """Return the rows as a Polars DataFrame.""" + return pl.DataFrame(self.table) + + def __len__(self) -> int: + """Return the number of rows in ``table``.""" + return self.table.num_rows + + # ------------------------------------------------------------------ + # Rendering + # ------------------------------------------------------------------ + + def header(self) -> str: + """Return the one-line description shown above the rows.""" + return f"{self.kind}[{self.label}] — {self.description or self._describe()}" + + def _describe(self) -> str: + """Return the header text after the name, for this status.""" + shown = self.table.num_rows + if self.status is HeadStatus.ROWS: + text = self._describe_rows(shown) + elif self.status is HeadStatus.PARTIAL: + total = self.total_rows if self.total_rows is not None else shown + if shown == total: + text = f"{total:,} {_rows(total)} computed so far" + else: + text = f"first {shown:,} of {total:,} rows computed so far" + text += " (computation incomplete)" + else: + return self.note or _DEFAULT_NOTES[self.status] + if self.note: + text += f"; {self.note}" + return text + + def _describe_rows(self, shown: int) -> str: + """Return the header text for a ``ROWS`` head showing ``shown`` rows.""" + total = self.total_rows + noun = _ORIGIN_NOUNS.get(self.origin, "rows") + if shown == 0 and (total is None or total == 0): + return "computed: no rows" if self.origin == "computed" else f"no {noun}" + if total is None: + return f"first {shown:,} {_rows(shown)}" + if self.total_is_estimate: + return f"first {shown:,} {_rows(shown)} of about {total:,}" + if shown == total: + return f"all {total:,} {noun if total != 1 else noun[:-1]}" + return f"first {shown:,} of {total:,} {noun}" + + def _display_frame(self) -> pl.DataFrame: + """Return a Polars frame of the rows for display: tag columns first, large cells + summarised. ``table`` itself is never changed.""" + order = [c for c in self.tag_columns if c in self.table.column_names] + [ + c for c in self.table.column_names if c not in self.tag_columns + ] + table = self.table.select(order) + columns = [] + for name in table.column_names: + column = table.column(name) + summary = _summarise(column) + if summary is None: + columns.append(column) + else: + columns.append(pa.chunked_array([summary], type=pa.large_string())) + return pl.DataFrame(pa.table(columns, names=table.column_names)) + + def _tags_line(self) -> str: + """Return the line naming the tag columns, or an empty string when there are none.""" + if not self.tag_columns: + return "" + return "tags: " + ", ".join(self.tag_columns) + + def _repr_html_(self) -> str: + """Return the notebook preview: header, tag line and the rows (large cells summarised).""" + parts = [ + f"
{html.escape(self.header(), quote=False)}
" + ] + tags = self._tags_line() + if tags: + parts.append(f"
{html.escape(tags, quote=False)}
") + if self.table.num_columns: + # Polars shortens strings to its own default length; show up to _MAX_TEXT. + with pl.Config(fmt_str_lengths=_MAX_TEXT, tbl_rows=-1): + parts.append(self._display_frame()._repr_html_()) + return "\n".join(parts) + + def __str__(self) -> str: + """Return the preview as text: header, tag line and the rows.""" + lines = [self.header()] + tags = self._tags_line() + if tags: + lines.append(tags) + if self.table.num_columns: + with pl.Config(fmt_str_lengths=_MAX_TEXT, tbl_rows=-1): + lines.append(str(self._display_frame())) + return "\n".join(lines) + + def __repr__(self) -> str: + """Return a short description: kind, label, status and row count.""" + return ( + f"StreamHead({self.kind}[{self.label}], status={self.status.value}, " + f"rows={self.table.num_rows})" + ) + + +_MAX_NAME = 48 + + +def display_name(stream: Any) -> str: + """Return the name the header shows for ``stream``: never its label changed. + + A stream that says (``_display_name``) gives its own, e.g. a source with no label + of its own is shown by its source id; otherwise its label. A long name keeps its + start and end around an ellipsis. + """ + name_of = getattr(stream, "_display_name", None) + name = str(name_of() if callable(name_of) else stream.label) + if len(name) > _MAX_NAME: + name = f"{name[:30]}…{name[-12:]}" + return name + + +def head_of( + head: Any, stream: Any, *, origin: str | None = None, note: str | None = None +) -> StreamHead: + """Return ``head`` (another stream's) shown as ``stream``. + + A ``StreamHead`` keeps every field. Any other result of ``head`` (a + ``StreamHeadProtocol``) is adapted from what that protocol gives: its rows, status + and total. + + Args: + head: What another stream's ``head`` returned. + stream: The stream, source or node that shows those rows as its own. + origin: Another origin for the rows, if given. + note: Another note, if given. + """ + if not isinstance(head, StreamHead): + head = StreamHead( + table=head.table, + status=HeadStatus(head.status), + kind=type(stream).__name__, + label=display_name(stream), + total_rows=head.total_rows, + ) + return head.for_stream(stream, origin=origin, note=note) + + +def render_error_html(stream: Any, error: BaseException) -> str: + """Return the preview of a stream whose ``head`` raised: its name and the error.""" + title = f"{type(stream).__name__}[{display_name(stream)}] — could not read: {error}" + return f"
{html.escape(title, quote=False)}
" + + +def _rows(count: int) -> str: + """Return ``row`` or ``rows`` for ``count``.""" + return "row" if count == 1 else "rows" + + +def _size(count: int) -> str: + """Return a byte count as text: ``300 B``, ``1.2 KB``, ``3.4 MB``.""" + value = float(count) + for unit in ("B", "KB", "MB", "GB"): + if value < 1000 or unit == "GB": + return f"{count} B" if unit == "B" else f"{value:.1f} {unit}" + value /= 1000 + return f"{count} B" # pragma: no cover + + +def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: + """Return display strings for a column whose cells are too large to show as they are, + or ``None`` to show the column as it is.""" + arrow_type = column.type + if isinstance(arrow_type, pa.ExtensionType): + return _summarise_extension(column, arrow_type) + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + return _byte_summaries([column], "bytes") + if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): + # Lengths come from Arrow; only the kept parts become Python strings. A long + # value keeps its start and its end (hash keys differ at the end), with an + # ellipsis between. + lengths: list[int | None] = pc.utf8_length(column).to_pylist() + if all(n is None or n <= _MAX_TEXT for n in lengths): + return None + shorts: list[str | None] = pc.utf8_slice_codeunits( + column, 0, _MAX_TEXT + ).to_pylist() + starts: list[str | None] = pc.utf8_slice_codeunits( + column, 0, _TEXT_START + ).to_pylist() + ends = [ + None if n is None or n <= _MAX_TEXT else _text_end(column[i]) + for i, n in enumerate(lengths) + ] + return [ + None if n is None else f"{a}…{b}" if n > _MAX_TEXT else short + for n, short, a, b in zip(lengths, shorts, starts, ends) + ] + if ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ): + return _summarise_list(column, arrow_type.value_type) + return None + + +def _summarise_extension( + column: pa.ChunkedArray[Any], arrow_type: pa.ExtensionType +) -> list[str | None] | None: + """Return display strings for a logical-type column, or ``None`` to let Polars show it. + + An array column is recognised by its logical type's extension name, never by the + bytes of a cell: it shows as ``ndarray float64 (2, 3)``, read from its stored form + without decoding the data. Another logical type over binary storage shows its size, + over a struct its name; any other storage is shown by Polars. + """ + # The array logical type's module is read when a column is shown, not imported: + # it sits above this module in the import graph (types, protocols, side effects), + # and it loads numpy, which only an array column needs. + numpy_type = importlib.import_module("orcapod.logical_types.numpy_type") + array_extension_name: str = numpy_type.NDARRAY_EXTENSION_NAME + + storage_type = arrow_type.storage_type + short_name = arrow_type.extension_name.rsplit(".", 1)[-1] + storages = [ + cast("pa.ExtensionArray[Any]", array).storage for array in column.chunks + ] + is_binary = pa.types.is_binary(storage_type) or pa.types.is_large_binary( + storage_type + ) + if arrow_type.extension_name == array_extension_name: + if pa.types.is_struct(storage_type): + return [ + None if v is None else f"ndarray {v['dtype']} {_shape_text(v['shape'])}" + for storage in storages + for v in _struct_fields( + cast(pa.StructArray, storage), ["dtype", "shape"] + ) + ] + if is_binary: + return [ + _npy_summary(storage[i]) + for storage in storages + for i in range(len(storage)) + ] + if is_binary: + return _byte_summaries(storages, short_name) + if pa.types.is_struct(storage_type): + return [ + f"<{short_name}>" if valid else None + for storage in storages + for valid in storage.is_valid().to_pylist() + ] + return None + + +def _npy_summary(cell: Any) -> str | None: + """Return ``ndarray `` for one ``.npy`` cell, reading its header only. + + The ``.npy`` header is a magic string, a version, the header length and a Python + dict literal with ``descr``, ``fortran_order`` and ``shape``. The buffer is sliced + without copying, and the array data is never read. + """ + if not cell.is_valid: + return None + buffer = cell.as_buffer() + size = f"ndarray ({_size(buffer.size)})" + try: + prefix = buffer[:12].to_pybytes() + if not prefix.startswith(b"\x93NUMPY"): + return size + if prefix[6] == 1: + start, length = 10, int.from_bytes(prefix[8:10], "little") + else: + start, length = 12, int.from_bytes(prefix[8:12], "little") + text = buffer[start : start + length].to_pybytes().decode("utf-8") + header = ast.literal_eval(text) + import numpy as np + + dtype = np.dtype(header["descr"]) + return f"ndarray {dtype.name} {_shape_text(header['shape'])}" + except (ValueError, SyntaxError, KeyError, TypeError, IndexError): + return size + + +def _byte_summaries(arrays: Sequence[Any], name: str) -> list[str | None]: + """Return ``name (size)`` for each binary cell of ``arrays``, in order. + + Sizes come from Arrow (``binary_length``); the bytes are never read. + """ + return [ + None if n is None else f"{name} ({_size(n)})" + for array in arrays + for n in pc.binary_length(array).to_pylist() + ] + + +def _struct_fields( + array: pa.StructArray, names: list[str] +) -> list[dict[str, Any] | None]: + """Return each row of ``array`` as a dict of the named fields (``None`` for a null row).""" + fields = {name: array.field(name).to_pylist() for name in names} + valid = array.is_valid().to_pylist() + return [ + {name: fields[name][i] for name in names} if valid[i] else None + for i in range(len(array)) + ] + + +def _shape_text(shape: Any) -> str: + """Return an array shape as Python writes a tuple: ``(40,)``, ``(2, 3)``.""" + dims = list(shape or []) + if len(dims) == 1: + return f"({dims[0]},)" + return "(" + ", ".join(str(d) for d in dims) + ")" + + +def _text_end(cell: Any) -> str: + """Return the last ``_TEXT_END`` characters of a string cell, reading only its end. + + A character is at most 4 bytes in UTF-8, so the last ``4 * _TEXT_END`` bytes hold + them; a character cut at the start of that window is dropped when decoding. + """ + buffer = cell.as_buffer() + window = buffer[max(0, buffer.size - 4 * _TEXT_END) :].to_pybytes() + return window.decode("utf-8", errors="ignore")[-_TEXT_END:] + + +def _summarise_list( + column: pa.ChunkedArray[Any], value_type: pa.DataType +) -> list[str | None] | None: + """Return display strings for a list column: short lists of scalars as written, others by length.""" + simple_items = not ( + isinstance(value_type, pa.ExtensionType) + or pa.types.is_nested(value_type) + or pa.types.is_binary(value_type) + or pa.types.is_large_binary(value_type) + ) + lengths: list[int | None] = pc.list_value_length(column).to_pylist() + if simple_items and all(n is None or n <= _MAX_LIST_ITEMS for n in lengths): + return None + item = "list" if pa.types.is_nested(value_type) else None + out: list[str | None] = [] + for index, length in enumerate(lengths): + if length is None: + out.append(None) + elif simple_items and length <= _MAX_LIST_ITEMS: + # Only a short list is converted to Python, to show its items. + out.append(repr(column[index].as_py())) + else: + out.append(f"list (len {length})" + (f" of {item}" if item else "")) + return out diff --git a/src/orcapod/protocols/core_protocols/__init__.py b/src/orcapod/protocols/core_protocols/__init__.py index ed6b2acbf..9c27952bd 100644 --- a/src/orcapod/protocols/core_protocols/__init__.py +++ b/src/orcapod/protocols/core_protocols/__init__.py @@ -8,7 +8,7 @@ from .data_function import DataFunctionProtocol from .pod import ArgumentGroup, PodProtocol from .sources import DynamicSourceProtocol, SourceProtocol -from .streams import StreamProtocol +from .streams import StreamHeadProtocol, StreamProtocol from .side_effect_pod import SideEffectPodProtocol from .trackers import TrackerProtocol, TrackerManagerProtocol @@ -19,6 +19,7 @@ "DataProtocol", "DynamicSourceProtocol", "SourceProtocol", + "StreamHeadProtocol", "StreamProtocol", "PodProtocol", "ArgumentGroup", diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index 894e437a3..bb751c957 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -16,6 +16,44 @@ from orcapod.protocols.core_protocols.pod import PodProtocol +class StreamHeadProtocol(Protocol): + """The readily available first rows of a stream, as ``head`` returns them. + + ``orcapod.core.streams.head.StreamHead`` is the implementation. + """ + + @property + def table(self) -> "pa.Table": + """The rows, with the exact Arrow types the stream gives them.""" + ... + + @property + def status(self) -> str: + """Where the rows came from, or why there are none (a ``HeadStatus``).""" + ... + + @property + def total_rows(self) -> int | None: + """The stream's row count when it is known without computing.""" + ... + + def header(self) -> str: + """The one-line description shown above the rows.""" + ... + + def as_table(self) -> "pa.Table": + """The rows as a PyArrow table.""" + ... + + def as_polars_df(self) -> "pl.DataFrame": + """The rows as a Polars DataFrame.""" + ... + + def _repr_html_(self) -> str: + """The notebook preview of the rows.""" + ... + + @runtime_checkable class StreamProtocol(TraceableProtocol, PipelineElementProtocol, Protocol): """ @@ -157,6 +195,25 @@ def as_table( """ ... + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> "StreamHeadProtocol": + """Return at most ``n`` readily available rows, without computing. + + The rows are ones the stream already holds, has cached from an earlier + computation, has stored, or can read with a bounded read. It never calls a + function, runs an operator, delivers a side effect, polls or writes. + + Returns: + StreamHead: The rows, and a status that tells an empty result from one + that is not computed. + """ + ... + class StreamWithOperationsProtocol(StreamProtocol, Protocol): def as_df( diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 1893a1dc7..2388dfa67 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -17,7 +17,16 @@ from orcapod.core.base import TraceableBase from orcapod.core.nodes.function_node import _build_record_id_preimage -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + SIDE_EFFECTS_NOTE, + HeadStatus, + StreamHead, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( StoreError, @@ -148,6 +157,20 @@ def keys( ) -> tuple[tuple[str, ...], tuple[str, ...]]: return self._input_stream.keys(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: showing this stream must never deliver its side effects.""" + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE + ) + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: for tag, data in self._input_stream.iter_data(): result = _execute_side_effect_row( @@ -730,6 +753,20 @@ def keys( ) -> tuple[tuple[str, ...], tuple[str, ...]]: return self._input_stream.keys(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: showing this stream must never deliver its side effects.""" + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE + ) + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Standalone iteration — executes delivery with no DB logging.""" for tag, data in self._input_stream.iter_data(): diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md new file mode 100644 index 000000000..ea827f3c2 --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -0,0 +1,221 @@ +# ITL-991 `head()` and a preview that never computes: implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this +> plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** every stream, source and node gets `head(n=10, *, columns, all_info) -> StreamHead`, +which never computes, and `_repr_html_` renders `self.head()`. + +**Architecture:** one new module `core/streams/head.py` holds `HeadStatus`, `StreamHead` +(result + renderer) and the shared helpers. `StreamBase.head` is the safe default (schema +only, `NOT_COMPUTED`); each class that can serve rows without computing overrides `head`. +Spec: `superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md`. + +**Tech stack:** PyArrow (the head table), Polars (HTML rendering only). + +## Status of decisions (coordinator, November 12, 2026) + +- D3 approved: OperatorJobNode in LOG mode with nothing in memory shows no rows + a note. +- D5 approved: fix F1 (DynamicPodStream `_cached_results` misspelling) in this PR, its own + commit, with a regression test and its own `### Fixed` changelog bullet. +- D1 (StreamHead), D2 (whole stored slice for nodes; bounded read is a follow-up), D4 + (remove `view()`/`StreamView`) are with the developer. Work proceeds on the + recommendations, each behind ONE seam: + - D1 seam: `StreamBase.head` returns `StreamHead`; a `pa.Table` answer would mean + `head()` returns `self._head_result(...).table` and `_repr_html_` calls `_head_result`. + - D2 seam: every node's stored read goes through one method per node class + (`_head_from_store`), so a bounded read replaces only that method. + - D4 seam: the removal of `view`/`StreamView` is one commit. +- **Do not open the PR until the coordinator confirms D1, D2, D4.** +- Coordinator requirements: (1) a node header that reads its whole stored slice must not + claim a bounded read, and a test shows the read performs no computation; (2) the workshop + path (a source preview; a job node right after `job.run()` in the same process) is cheap: + a slice of what is in memory. + +## Files + +- Create `src/orcapod/core/streams/head.py`: `HeadStatus`, `StreamHead`, `head_column_config`, + cell summaries and HTML/text rendering. +- Modify `src/orcapod/core/streams/base.py`: `head` default, `_repr_html_`, remove + `view`/`StreamView`, `_schema_only_table`. +- Modify `src/orcapod/core/streams/__init__.py`, `src/orcapod/__init__.py`: export + `StreamHead`, `HeadStatus`. +- Modify `src/orcapod/core/streams/arrow_table_stream.py`: `head`. +- Modify `src/orcapod/core/sources/base.py` (RootSource delegation), `cached_source.py`, + `derived_source.py`, `polling_source.py`, `source_proxy.py`. +- Modify `src/orcapod/core/operators/static_output_pod.py`: F1 fix; `DynamicPodStream.head`. +- Modify `src/orcapod/core/function_pod.py`: `FunctionPodStream.head` (+ factor the table + build of `as_table` into helpers shared with head). +- Modify `src/orcapod/side_effects.py`: side-effect note. +- Modify `src/orcapod/core/nodes/{function_node,operator_node,source_node}.py`. +- Tests: `tests/test_core/streams/test_head_and_preview.py` (streams, sources, operators, + function pods, rendering), `tests/test_core/streams/test_dynamic_pod_stream_staleness.py` + (F1), `tests/test_pipeline/test_node_head.py` (job nodes, sync/async, in-memory/Delta, + loaded job). +- Docs: `docs/concepts/` "Looking at data" section; `changelog.d/ITL-991.md`; + `design_issues/ITL-991-PV1.md`. + +## StreamHead contract (Task 1) + +```python +class HeadStatus(StrEnum): + ROWS = "rows" # the first rows of the stream's complete content + PARTIAL = "partial" # computation stopped part way; rows computed so far + NOT_COMPUTED = "not_computed" + NOTHING_STORED = "nothing_stored" + UNAVAILABLE = "unavailable" + +@dataclass(frozen=True) +class StreamHead: + table: pa.Table + status: HeadStatus + kind: str # class name of the stream shown + label: str + tag_columns: tuple[str, ...] + total_rows: int | None = None + total_is_estimate: bool = False + origin: str = "held" # "held" | "computed" | "stored" | "read" + detail: str | None = None # extra note ("inputs changed ...", side effects) + def as_table(self) -> pa.Table + def as_polars_df(self) -> pl.DataFrame + def __len__(self) -> int + def header(self) -> str # the wording table of the spec + def for_stream(self, stream) -> StreamHead # same rows, shown as `stream` + def _repr_html_(self) -> str + def __str__(self) -> str +``` + +Header wording (exact; #333 depends on the estimate line): + +| case | text after `Kind[label] — ` | +|---|---| +| ROWS, rows == total | `all {k} rows` (`all 1 row`) | +| ROWS, total known | `first {k} of {total:,} rows` / `... stored rows` / `... computed rows` | +| ROWS, estimate | `first {k} rows of about {total:,}` | +| ROWS, total unknown | `first {k} rows` | +| ROWS, 0 rows | `no rows` (`computed: no rows` for origin computed) | +| PARTIAL | `first {k} of {total:,} rows computed so far (computation incomplete)` | +| NOT_COMPUTED | `not computed; as_table(), iter_data() or a job run computes it` | +| NOTHING_STORED | `nothing stored; run the job` | +| UNAVAILABLE | detail only | +| detail present | appended after `; ` (or the whole note for UNAVAILABLE / a NOT_COMPUTED with a reason) | + +## Tasks + +### Task 1: F1 regression test and fix (own commit) +- [ ] Test `tests/test_core/streams/test_dynamic_pod_stream_staleness.py`: an operator over a + `FunctionPodStream`; `as_table()`; `fn_stream.clear_cache()`; `as_table()` again calls + `static_process` twice (spy with `mock.patch.object(SelectDataColumns, "static_process", + autospec=True, side_effect=original)`; comment: the test is about re-running it). +- [ ] Run: fails (1 call). +- [ ] Fix `static_output_pod.py:383,392`: `self._cached_results = None` → `self._cached_stream = None`. +- [ ] Run test + `tests/test_core/operators` → pass. +- [ ] Commit `fix(operators): an operator stream recomputes after an upstream changes (ITL-991)`. + +### Task 2: failing tests for streams, sources, rendering +- [ ] Write `tests/test_core/streams/test_head_and_preview.py` (cases in spec g1–g5, g7). +- [ ] Run: fails (no `head`). Commit `test(streams): head() and the preview never compute (ITL-991)`. + +### Task 3: `head.py`, `StreamBase.head`, `_repr_html_`, remove `view`/`StreamView` +- [ ] Implement `head.py` per the contract; `StreamBase.head` default + (`NOT_COMPUTED`, `_schema_only_table(config)` = `make_empty_table` of + `output_schema(columns=config)`; any exception → zero-column table). +- [ ] `_repr_html_`: `try: return self.head()._repr_html_() except Exception as e:` header + + `could not read: {e}`. +- [ ] Remove `view`, `StreamView` (D4 seam: separate commit). + +### Task 4: ArrowTableStream and sources +- [ ] `ArrowTableStream.head`: `as_table(columns=config without content_hash).slice(0, n)`; + content hashes for `islice(iter_data(), n)` only; total = `num_rows`. +- [ ] `RootSource.head`: `self._stream.head(...).for_stream(self)`. +- [ ] CachedSource: current `_cached_stream` → its head; else `_source.head` with detail + `rows of the wrapped source; the cache is not read until first use`; never `_ensure_stream`. +- [ ] DerivedSource: factor `_peek_stream()` (no assignment) out of `_get_stream`; head uses + the cached stream/table if any, else `_peek_stream()`; origin `stored`; no origin rows → + NOTHING_STORED. +- [ ] PollingSource: `_batches` → combined stream head; else NOT_COMPUTED `nothing polled yet` + with the declared schema, or a zero-column table when none (never `keys()`). +- [ ] SourceProxy: UNAVAILABLE `source not available; bind a live source`. + +### Task 5: operator and function streams +- [ ] `DynamicPodStream.head`: `had = self._cached_stream is not None`; + `self._update_cache_status()`; cached → `self._cached_stream.head(...)` with origin + `computed`; `had` and now gone → NOT_COMPUTED `inputs changed since the last computation`. +- [ ] `FunctionPodStream`: factor `_table_from_pairs(pairs)` and `_shape_table(table, config, + hashes)` out of `as_table`; head: stale with cache → NOT_COMPUTED "inputs changed"; + `_cached_output_table` → shape + slice; else contiguous prefix of `_cached_output_datas` + (skip `None` outputs) → ROWS (iteration finished) or PARTIAL; empty → NOT_COMPUTED. +- [ ] Side-effect streams/nodes: base default with detail `displaying never delivers side effects`. + +### Task 6: nodes (`tests/test_pipeline/test_node_head.py` first) +- [ ] Blueprints `FunctionNode`, `OperatorNode`: UNAVAILABLE `pipeline blueprint; run it in a PipelineJob`. +- [ ] `SourceNodeBase.head` + `_repr_html_`; `SourceNode` UNAVAILABLE `input slot; bind a + source`; `SourceJobNode` → bound source head, else UNAVAILABLE `no source bound`. +- [ ] `OperatorJobNode.head`: in-memory → inner head; REPLAY + store → `_head_from_store` + (`_read_stored_records`, internal columns dropped, ArrowTableStream head, origin stored); + none → NOTHING_STORED; LOG → NOT_COMPUTED `not run in this session; rows of earlier runs + are stored but not replayed (cache mode LOG)`; OFF → NOT_COMPUTED `not run; cache mode OFF + stores nothing`. Never raises the loaded-node errors. +- [ ] `FunctionJobNode.head`: UNAVAILABLE load status → UNAVAILABLE; `_cached_output_table` + → slice; `_cached_output_datas` → first n via the node's conversion; else + `_head_from_store` (loaded: `_stored_rows_stream(_fetch_joined_records())[1].head`; + live: `_load_cached_entries()` values, not kept) → ROWS origin stored; empty → NOTHING_STORED. +- [ ] Tests: sync and async orchestrators × in-memory and Delta; function counter unchanged + by head; before run NOTHING_STORED; loaded job; OFF/LOG/REPLAY operator nodes. + +### Task 7: docs, changelog, design issue, full suite +- [ ] "Looking at data" section in `docs/concepts/streams.md` (or the closest page), executed + samples with `# ...` output comments. +- [ ] `changelog.d/ITL-991.md` (spec h text + F1 `### Fixed` bullet); + `uv run python scripts/assemble_changelog.py --check`. +- [ ] `design_issues/ITL-991-PV1.md` (resolved); `uv run python scripts/design_issues.py --check`. +- [ ] ruff check/format, basedpyright (exit code), `uv run pytest tests/`. +- [ ] Merge origin/main, rerun, push; wait for D1/D2/D4 confirmation before the PR. + +## Progress notes (for a successor) + +- 2026-11-12: spec 70639010 pushed; design ask answered (D3, D5 approved; D1/D2/D4 pending + with developer). Plan saved. +- 2026-11-12 (later): D1, D2, D4 approved by the developer. Tasks 1-7 implemented and + pushed: F1 fix, StreamHead, sources, operator/function streams, cached pod stream, nodes, + docs ("Looking at data" in docs/concepts/streams.md), changelog fragment, design issue + ITL-991-PV1. Next: full suite, merge main, PR, CodeRabbit. + +## Merge of origin/main 859106f7 (#335 ITL-928, #328 ITL-981, #336) — November 12, 2026 + +Resolution agreed in advance by the #334 and #335 workers (coordinator brief): + +- [x] `git merge origin/main` (merge commit, no rebase). Conflict expected in + `core/nodes/function_node.py`; check `core/function_pod.py` for a duplicated + `_table_from_pairs` left by the auto-merge. +- [x] Keep #334's structure: `head`, `_shape_table`, `_computed_prefix`, `as_table` = + `_table_from_pairs(list(self.iter_data()))` then `_shape_table`. +- [x] Take main's (#335) `_table_from_pairs` bodies as the non-empty build in both classes + (per-row `tag.as_table` / `data.as_table` concatenation, `_concat_row_tables`, + mixed Arrow types raise `InputValidationError`). +- [x] Empty pairs: `FunctionPodStream._table_from_pairs` keeps #334's declared-columns branch + (`make_empty_table`); `FunctionNodeBase._table_from_pairs` keeps main's empty branch. +- [x] (not needed: the resolution was mechanical) If the resolution is not mechanical, ask the coordinator before pushing. +- [x] Tests: #335's (tag types kept on both stores, mixed rows error, join after function + node), #334's (no computation on display, prefix invariant in every ROWS state), + #328's array tests + ndarray `.npy` preview summary; full suite on 3.12, 3.11, 3.13; + ruff, format, basedpyright; golden unchanged. +- [ ] One push; CodeRabbit reviews the new head; answer every thread; worker_done with sha. + +Result: function_pod.py auto-merged into two `_table_from_pairs`; the #334 one was removed +and its empty branch moved into main's (the now-dead "no tag tables" fallback dropped). +function_node.py: #334's `_table_from_pairs` removed, main's kept in place. New test +`test_head_keeps_the_tag_arrow_type_of_the_input` (int8 tag, both orchestrators and stores). +Suite 10712 passed on 3.11, 3.12, 3.13; ruff, format, basedpyright exit 0; golden unchanged. + +## CodeRabbit round on 19ad76ce (coordinator approved a 2nd push: F3, F4 only) + +- [x] F3: rowless head downstream of an unpolled declared-schema `PollingSource` fetched + when `source`/`meta`/`context`/`content_hash`/`all_info` was asked for. The private + schema-IO check takes the head's `ColumnConfig`; regression test first (zero polls and + fetches, `FunctionPodStream` and `SideEffectPodStream`). +- [x] F4: docstring on the nested `build` helper in `test_node_head.py`. +- [ ] Decline F1 (ITL-1002), F2 (follow-up), linked-issues check (D2) in the threads. + +Also merged origin/main #337, #338 and #286 (no conflicts; golden and contexts identical to +main). Suite 11029 passed on 3.11, 3.12, 3.13; basedpyright exit 0. diff --git a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md new file mode 100644 index 000000000..209df2dc6 --- /dev/null +++ b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md @@ -0,0 +1,409 @@ +# ITL-991: `head()` and a notebook preview that never computes + +Status: approved (developer, November 12, 2026: "agreed with all 3 recs"; D3 and D5 approved by the coordinator). +Implementation notes at the end record where the code differs from the text below. +Issue: ITL-991. Related: ITL-186 / PR #333 (Spiral source), ITL-990 (workshop notebook). + +## Rulings this design implements (developer, November 12, 2026) + +1. "Stream's preview should only show a short preview of what it can readily access but + without causing materialization unnecessary and definitely not trigger new computation." +2. "For nodes, they are associated with their storage and preview on nodes should be just a + head preview of the storage content WITHOUT triggering computation." +3. "Yes bounded 10 row read from Spiral should be considered readily accessible." +4. "Yes they should get explicit head style method and preview should just be triggering + this. For stream that's downstream of computation, head should NOT trigger the + computation but rather show the content of cached stream contents if present." + +## Today (verified on main 25650c42) + +`StreamBase._repr_html_` and `StreamBase.view()` (`core/streams/base.py:443-468`) call +`as_polars_df()` on the whole stream. So displaying: + +- a `FunctionPodStream` calls the user's function on every row; +- a `DynamicPodStream` (any operator result) runs `static_process` on the whole input; +- a `SideEffectPodStream` or `SideEffectNode` **delivers the side effects** (sends the + emails, writes the files) for every row; +- a `SpiralDBTableSource` after #333, a `PollingSource` (polls), a `CachedSource` (reads its + source and **writes** the cache table), a `DerivedSource` (reads the whole origin store); +- a blueprint `FunctionNode` / `OperatorNode` / unbound `SourceJobNode` raises inside the + formatter. + +`__repr__` never computes anywhere (checked: `TraceableBase`, `ArrowTableStream`, +`DynamicPodStream`, the nodes). `SourceNodeBase` is not a `StreamBase`, so source nodes have +no preview at all. No test or doc uses `view()`, `StreamView` or `_repr_html_`. + +## a. The method + +```python +def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, +) -> StreamHead: ... +``` + +- On `StreamBase` (every stream, source and stream-like node) and on `SourceNodeBase`. +- `columns` / `all_info` mean exactly what they mean for `as_table()`. `sort_by_tags=True` + raises `ValueError` ("head returns the first rows in stream order; sort as_table() + instead"): sorting needs every row, so it cannot be a bounded read. +- `n` is a non-negative `int`; anything else raises `ValueError`/`TypeError`. +- **Never computes**: never calls a user function, never runs an operator's + `process`/`static_process`, never delivers a side effect, never polls, never writes to a + database, never keeps what it read on the object (no hot-loading into in-memory caches). +- The base implementation is the safe default: no rows, the declared schema, status + `NOT_COMPUTED`. A stream class serves rows only by overriding `head`, so a new stream class + can never compute by accident. +- `_repr_html_` is `self.head()._repr_html_()` and nothing else: one code path. It catches an + exception from `head` and renders the schema with "could not read: " instead of + letting the notebook formatter print a traceback. + +**Return type: `StreamHead`** (recommended), a small frozen result object in +`orcapod.core.streams` (exported from `orcapod`): + +| Member | Meaning | +|---|---| +| `table: pa.Table` | at most `n` rows, exact Arrow types (logical types, widths, nullability as the stream has them) | +| `status: HeadStatus` | where the rows came from, or why there are none (below) | +| `total_rows: int \| None` | the stream's row count when known without computing | +| `total_is_estimate: bool` | `True` for a recorded estimate (Spiral manifests) | +| `tag_columns: tuple[str, ...]` | which columns of `table` are tags | +| `label`, `kind` | the stream's label and class name, for the header | +| `as_table()`, `as_polars_df()`, `__len__` | convenience, same names as the stream's | +| `_repr_html_()` | the preview (section e) | + +`HeadStatus` (a `str` enum): `ROWS` (the first rows of the stream's complete content), +`PARTIAL` (a computation stopped part way; these are the rows computed so far), +`NOT_COMPUTED`, `NOTHING_STORED` (a job node with no stored rows), `UNAVAILABLE` (no data can +exist here: a blueprint node, an unbound source slot, a `SourceProxy`). + +Why not the alternatives: + +- A plain `pa.Table` (matches `as_table`) cannot tell "empty result" from "not computed" + (point d) without a second method, and `stream.head(50)` in a notebook would print + pyarrow's text repr instead of the preview. +- A `pl.DataFrame` has the same two problems, and Polars is the convenience layer here + (`as_polars_df` is built from `as_table`), not the source of truth for types. +- A small `ArrowTableStream` would be a new root stream with its own identity; feeding it to + a pod silently builds a pipeline on 10 rows, and it cannot carry the status either. + +With `StreamHead`, `stream.head(50)` renders the same preview with 50 rows, the data is one +attribute away (`.table`, or `.as_polars_df()`), and `.status` answers point d. + +Header line (rendered by `StreamHead`, one wording table so #333 and every class agree): + +| Status / case | Header | +|---|---| +| ROWS, total known, total > n | `ArrowTableSource[readings] — first 10 of 1,234 rows` | +| ROWS, total ≤ n | `ArrowTableSource[readings] — all 7 rows` | +| ROWS, estimate | `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` | +| ROWS, total unknown | `… — first 10 rows` | +| ROWS from a node's store | `FunctionJobNode[score] — first 10 of 1,234 stored rows` | +| ROWS from a stream's cache | `FunctionPodStream[score] — first 10 of 500 computed rows` | +| ROWS, zero rows | `DynamicPodStream[filter] — computed: no rows` | +| PARTIAL | `FunctionPodStream[score] — first 10 of 37 rows computed so far (computation incomplete)` | +| NOT_COMPUTED | `DynamicPodStream[join] — not computed; as_table(), iter_data() or a job run computes it` | +| NOT_COMPUTED (stale cache) | `… — not computed: inputs changed since the last computation` | +| NOTHING_STORED | `FunctionJobNode[score] — nothing stored; run the job` | +| UNAVAILABLE | `FunctionNode[score] — pipeline blueprint; run it in a PipelineJob` / `SourceJobNode[raw] — no source bound` / `SourceProxy[raw] — source not available` | +| side effects | adds `; displaying never delivers side effects` | + +Below the header: a one-line schema summary (`tags: id: int, session: str · data: x: float, +frame: ndarray`), then the table (tag columns first). When there are no rows the table is +the column header row only, so the schema is always visible. + +## b. Per kind of object + +"Reads" is the I/O head performs; "knows" is how the code decides without computing. +Every row-serving case returns the first `n` rows of what `as_table()` would return when +`as_table()` would not compute (this is the invariant the tests check: +`head(n).table == as_table().slice(0, n)` in every `ROWS` state). + +| Object | Readily available | head returns | Reads / cost | How it knows | +|---|---|---|---|---| +| `ArrowTableStream` | its table | ROWS, first n, exact total | slices of the held tables; source info / context / content hash built for n rows only | always holds its table | +| In-memory and file sources: `ArrowTableSource`, `DictSource`, `ListSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource`, `DBTableSource` / `SQLiteTableSource` / `PostgreSQLTableSource` (all read their table at construction today) | the held `_stream` | delegate to `_stream.head` (RootSource default) | none beyond the slice | `_stream` is built in `__init__` | +| `SpiralDBTableSource` on main (reads at construction) | held `_stream` | as above | none | as above | +| `SpiralDBTableSource` after #333 (streams) | a bounded read | ROWS, ≤ n rows read with `limit=min(n, limit)` and the source's `where`/`asof`; total = `row_count_estimate` (estimate) when no `where`, else unknown | one scan of at most n rows + a manifest count | ruling 3 | +| `PollingSource` | batches already committed | ROWS of committed rows (exact count) or NOT_COMPUTED "nothing polled yet"; **never polls** | none | `self._batches` non-empty. Without a declared schema and before a poll its `keys()` itself polls; head then shows "nothing polled yet; schema unknown" instead of calling `keys()` | +| `CachedSource` | the built merged stream if current; else its wrapped source's head | ROWS; **never writes** the cache table | slice, or the wrapped source's head | `_cached_stream is not None and not is_stale` | +| `DerivedSource` | its cached table/stream; else the origin node's stored rows | ROWS / NOTHING_STORED (origin not run) | v0.3: the origin's stored read (see "bounded node reads" below), not cached on the object | `_cached_table` / `_cached_stream`; the origin node's head | +| `SourceProxy` | nothing | UNAVAILABLE + declared schema | none | it has no delegate | +| `DynamicPodStream` (every operator result) | its cached result, if current | ROWS of the cached result, or NOT_COMPUTED (+ "inputs changed" when stale) | slice of `_cached_stream` | `_cached_stream is not None` and a side-effect-free freshness check (see finding F1) | +| `FunctionPodStream` (`FunctionPod`) | computed rows in `_cached_output_datas` / `_cached_output_table` | ROWS (iteration finished: `_cached_input_iterator is None` and cache non-empty), PARTIAL (iteration stopped: the contiguous prefix of computed indices, skipping filtered `None` outputs), or NOT_COMPUTED | converts ≤ n cached `(tag, data)` pairs to Arrow with the same typed conversion `as_table` uses (factored into one helper) | the cache dict and the iterator state; `is_stale` → NOT_COMPUTED "inputs changed" (head does not clear the cache) | +| `FunctionPodStream` of a `CachedFunctionPod` | same as above: the in-memory cache | same | same | same. Its result store is NOT consulted: looking a row up needs the input row's hash, i.e. reading and hashing the upstream. The note says "results may be in the result store; as_table() reads them without calling the function". | +| `SideEffectPodStream` | nothing (it keeps no rows) | NOT_COMPUTED, header adds "displaying never delivers side effects" | none | always | +| `FunctionNode`, `OperatorNode` (blueprints) | nothing | UNAVAILABLE "pipeline blueprint" + schema | none | the class | +| `SourceNode` (input slot) | nothing | UNAVAILABLE "input slot; bind a source" + declared schema | none | the class | +| `SourceJobNode` | the bound source | delegates to `bound_source.head(...)`; unbound → UNAVAILABLE "no source bound" | that source's head | `bound_source` | +| `OperatorJobNode` | its in-memory output; else its stored table | delegate to `_cached_output_stream.head` (which may itself be an un-run `DynamicPodStream` → NOT_COMPUTED); else stored rows (REPLAY) | stored read in REPLAY only, see below | `_cached_output_stream`; `_cache_mode`, `_pipeline_database` | +| `OperatorJobNode`, cache mode OFF | in-memory output only | ROWS after a run; else NOT_COMPUTED "uncached operator; nothing is stored" | none | `cache_mode == OFF` | +| `OperatorJobNode`, cache mode LOG | in memory; the stored table holds rows of earlier runs | in memory → ROWS; else NOTHING_STORED-style note "not run in this session; N rows stored by earlier runs (LOG mode, not replayed)" with no rows — **decision D3** | — | — | +| `FunctionJobNode` | `_cached_output_datas` after a run; else the stored rows (pipeline table ⋈ result table) | ROWS (in memory, or "stored rows"), NOTHING_STORED | in memory: ≤ n conversions. Stored: see "bounded node reads" | dict non-empty; `_fetch_joined_records` non-empty | +| `SideEffectNode` / `SideEffectJobNode` | nothing it can show without delivering | NOT_COMPUTED, "displaying never delivers side effects" | none | the class | +| Loaded job nodes (`PipelineJob.load`) | the store only | function: stored rows / NOTHING_STORED; operator: REPLAY stored rows, else UNAVAILABLE "not stored (cache mode OFF)"; source: bound source or `SourceProxy` UNAVAILABLE. Never raises the loaded-node errors | stored read | `_function_pod is None`, `_loaded` | + +### Bounded node reads + +`ArrowDatabaseProtocol` has no limited read: `get_all_records` reads the whole table, and a +node's rows are filtered from a shared table (`__data_context`, node content hash), and for +a function node joined to the result table and de-duplicated by recomputation index (the +newest record of a base entry may lie anywhere in the table). + +- **v0.3 (recommended):** a node with nothing in memory reads its stored rows with its + existing read helpers (`_read_stored_records`, `_fetch_joined_records` / + `_stored_rows_stream`), returns the first n, and keeps nothing. No computation, exact + total ("first 10 of 1,234 stored rows"), but the I/O is the node's whole stored slice. + After `job.run()` in the same process (the workshop's path) the rows are in memory and + nothing is read at all. +- **Follow-up (new issue):** `ArrowDatabaseProtocol.get_first_records(record_path, n, *, + column_values=None, record_id_column=None)` implemented by every database class (Delta: + `dataset.head(n, filter=...)` plus pending rows; in-memory: filter + slice; connector: + `WHERE … LIMIT n`; no-op; extension-aware wrapper), used by operator and source caches + directly and by function nodes for the pipeline rows plus `get_records_by_ids` for their + results. The recomputation-index de-duplication makes the function-node case need care + (read rows for the chosen base entries only). About one more day; public protocol change. + +## c. Operators + +Confirmed: head never runs an operator, even when its inputs are readily available, because +the operator of the heads is not the head of the operator: + +- `GroupBy(by=["subject"])` over the first 10 rows of a source gives **wrong values**, not + just fewer rows: a group's list holds only the members among those 10 rows. +- `Join` / `SemiJoin` of two heads can be empty while the full join is not (the matching + keys are beyond row 10 of one side). +- `PolarsFilter` / `DropNulls` of a head can be empty while the full result is not. +- `MergeJoin` lists and `Batch(drop_partial_batch=True)` change content too. + +So an operator stream shows its schema and NOT_COMPUTED until it is materialized, then its +rows. Which stream classes keep their materialized rows today: + +| Class | Keeps rows | Where | +|---|---|---| +| `ArrowTableStream` | always | `_table` | +| `DynamicPodStream` | yes, after `as_table`/`iter_data`/`run` | `_cached_stream` (the `ArrowTableStream` from `static_process`) | +| `FunctionPodStream` | yes, incrementally during iteration; the table after `as_table` | `_cached_output_datas`, `_cached_output_table` | +| `SideEffectPodStream`, `SideEffectNode` | no | — | +| `CachedSource` | yes, after first use | `_cached_stream` | +| `DerivedSource` | yes (tags_as_data: only once the origin has rows) | `_cached_table` / `_cached_stream` | +| `PollingSource` | committed batches | `_batches` | +| `OperatorJobNode` | after `run`/`execute` (lazy `DynamicPodStream` after `run` with OFF and no store) | `_cached_output_stream` | +| `FunctionJobNode` | after `run`/`execute`/first `iter_data` | `_cached_output_datas` | + +## d. Nothing available + +`head` returns a zero-row table with the full schema and a status that is not `ROWS`: + +- Classes that hold an Arrow schema (sources, `ArrowTableStream`, stored tables) slice it to + zero rows: exact types. +- Uncomputed operator / function streams know only their Python schema (`output_schema()` + exposes Python types). Their empty table is + `arrow_utils.make_empty_table({**tag_schema, **data_schema}, type_converter)`: correct + names, nullability and logical types, but widths are the converter's defaults (`int` → + `int64`) and may differ from the eventual computed column (an `int8` input passed through + an operator). This is documented on `head`; the status tells the caller the table is a + schema, not a result. +- Empty result vs not computed: `status == ROWS and len(table) == 0` is an empty result; + `NOT_COMPUTED` / `NOTHING_STORED` / `UNAVAILABLE` are not. + +## e. Large cells (preview only) + +`StreamHead._repr_html_` renders a display copy; `StreamHead.table` is never changed. + +| Arrow type of the column | Shown as | +|---|---| +| `orcapod.ndarray` extension (and any extension whose storage is a struct with `dtype` and `shape`) | `ndarray float64 (2, 3)`, read from the `dtype`/`shape` fields without decoding `data` | +| other logical (extension) types | Polars' own rendering if the storage is a string or number (`Path` shows the path); otherwise `` | +| `large_binary` / `binary` | `bytes (300 B)`, sizes humanised (`1.2 MB`) | +| `large_string` longer than 80 characters | first 77 characters + `…` (length in a tooltip) | +| `large_list` | `[1, 2, 3]` when ≤ 5 scalar items, else `list (len 12)`; nested lists `list (len 3) of list` | +| `struct` | Polars' rendering, with these rules applied to its fields | +| null | Polars' `null` | + +Tag columns come first and are marked in the header row (a `tag` group label above them). +The renderer builds a Polars frame of display strings for the summarised columns and uses +Polars' HTML for the table, so the look stays what users see today. + +## f. `view()` and `__repr__` + +- **Remove `view()` and `StreamView`.** `view()` is today's "whole stream as a frame" and + would either compute (forbidden by the ruling) or duplicate `head`. Nothing in the repo + uses it; greenfield rule, no shim. `as_polars_df()` remains the way to get everything. +- **`__repr__`**: unchanged; it already never computes. `str(stream.head())` gives a text + preview (header + Polars' text table) for terminal use. + +## g. Tests (public interface; failing first) + +New file `tests/test_core/streams/test_head_and_preview.py`, plus node cases in +`tests/test_core/nodes/`: + +1. Function pod: a counting function. `stream.head()` and `stream._repr_html_()` before + compute → count 0, `NOT_COMPUTED`, the schema's columns in the empty table. After three + `next()` on `iter_data()` → `PARTIAL`, 3 rows, count still 3. After `as_table()` → `ROWS`, + `head(n).table == as_table().slice(0, n)`, count unchanged by head. After the upstream is + modified → `NOT_COMPUTED` (stale). +2. Same with a concurrent executor; and a `CachedFunctionPod` stream whose result store is + full → still `NOT_COMPUTED` and the function is not called. +3. Operators (`Join`, `GroupBy`, `PolarsFilter`): `unittest.mock.patch.object(, + "static_process", wraps=...)` as a spy (non-public, commented: the test is about it not + running) → zero calls from `head`/`_repr_html_`; after `as_table()` → rows equal to the + table's prefix; empty filter result → `ROWS` with 0 rows, not `NOT_COMPUTED`. +4. Side effects: a `SideEffectPodStream` and a job's side-effect node → displaying delivers + nothing (counter 0). +5. Sources: every in-memory source → `ROWS`, exact total, prefix invariant; `PollingSource` + → no poll (counter on the impl); `CachedSource` → no table written (store unchanged); + `SourceProxy` → `UNAVAILABLE`. Spiral's bounded read is tested in #333 (fake pyspiral + records the `limit` it was given) once the two meet. +6. Nodes, in-memory store and Delta store, after `job.run()` sync and async: function, + operator (OFF, LOG, REPLAY) and source nodes → expected status and prefix rows; the + function counter unchanged. Before a run → `NOTHING_STORED` / `NOT_COMPUTED`. Loaded + job (`PipelineJob.load`) in a fresh object → stored rows, no raise; blueprint nodes → + `UNAVAILABLE`. With the follow-up bounded read, a spy database asserts `get_first_records` + is called with `n` and `get_all_records` is not. +7. Preview: header wording table above; ndarray / bytes / long string / list summaries; + `head().table` still holds the full values; `head(n=-1)` and `sort_by_tags=True` raise. +8. Golden hash tests untouched and passing. + +## h. What changes for users + +- Displaying a stream, source or node in a notebook no longer computes: an operator result + or function pod output that has not been computed shows its schema and "not computed" + until `as_table()`, `iter_data()` or a job run computes it. Displaying a side-effect + stream no longer delivers its side effects. Displaying a `CachedSource` no longer writes + its cache. +- New `head(n=10)` on every stream, source and node, returning a `StreamHead`. +- `view()` and `StreamView` are removed. +- No hash, golden value, stored table, path or stored schema moves: head only reads, and no + `identity_structure` changes. + +Changelog fragment `changelog.d/ITL-991.md`: + +```markdown +### Breaking changes and migration + +- Displaying a stream, source or node in a notebook no longer computes anything. The preview + shows the first 10 rows that are already available (held in memory, cached from an + earlier computation, stored by a job run, or a bounded read of a source's table) and + otherwise the schema with a note such as "not computed". Call `as_table()` or run the job + to compute. Displaying a side-effect stream no longer delivers its side effects. +- `StreamBase.view()` and `StreamView` are removed: use `head(n)` or `as_polars_df()`. + +### Added + +- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a + `StreamHead`: at most `n` rows as an Arrow table with their exact types, the row count + when it is known without computing, and a `status` that tells an empty result from "not + computed". It never calls a function, runs an operator or polls a source. +``` + +## i. Size, commits, schedule + +About 600–800 lines of code and 400–600 of tests. + +1. `test:` failing tests for streams and sources (g1–g5, g7). +2. `feat(streams):` `StreamHead`, `HeadStatus`, `StreamBase.head` default, renderer with cell + summaries, `_repr_html_` → `head()`; remove `view`/`StreamView`. +3. `feat(sources):` `ArrowTableStream.head`, RootSource delegation, Polling / Cached / + Derived / Proxy. +4. `feat(streams):` `DynamicPodStream.head`, `FunctionPodStream.head` (shared typed + conversion helper), side-effect streams. +5. `test:` + `feat(nodes):` node heads (g6), `SourceNodeBase.head`. +6. `docs:` "Looking at data" section (preview, `head`, `as_table`, `as_polars_df`, + `as_pandas_df`); changelog fragment; design-issue file. + +Commits 1–6 with the v0.3 node read (full stored slice): Nov 13–14, PR and CodeRabbit by +Nov 15. The bounded database read (`get_first_records`) as a separate follow-up after the +workshop unless the developer wants it in v0.3 (then +1 day, still before Nov 16 but with +no slack). + +## #333 interplay (whichever merges second makes the mechanical change) + +#333 adds `StreamBase._preview(columns, all_info) -> (pl.DataFrame, note)` (default: +today's behaviour) and a Spiral override that reads ≤ 10 rows with Spiral's `limit` and +renders `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349`. + +- `StreamBase._preview` is removed; `head` is the override point. +- The Spiral override becomes: + + ```python + def head(self, n=10, *, columns=None, all_info=False) -> StreamHead: + config = head_column_config(columns, all_info) # shared helper; rejects sort_by_tags + limit = n if self._limit is None else min(n, self._limit) + rows = -> pa.Table + return StreamHead.of_rows( + self, rows, config, + total_rows=None if self._where else self.row_count_estimate, + total_is_estimate=True, + ) + ``` + + i.e. the same read, returning the Arrow table instead of `pl.DataFrame(...)`, and the note + string replaced by `total_rows` + `total_is_estimate`. The header wording above produces + #333's exact text, so #333's rendering assertions keep passing. +- If #333 merges first: this PR deletes `_preview` from `StreamBase` and rewrites the Spiral + override as above. If this PR merges first: #333 drops its `StreamBase._preview` and + implements `head` as above. Suggest the coordinator forwards this section to #333's worker. + +## Findings (out of scope unless decided otherwise) + +- **F1 `DynamicPodStream` never invalidates its cache.** `_update_cache_status` + (`core/operators/static_output_pod.py:383,392`) sets `self._cached_results = None` (a + misspelling; the attribute is `_cached_stream`) and then moves `_cached_time` up to the + upstream time, so after an upstream changes, `as_table()` returns the old result, and any + read of `last_modified` makes it look fresh. head needs a correct freshness check. + Proposal: fix it here (one line plus a regression test), since head depends on it. +- **F2** Displaying a `SideEffectPodStream` / `SideEffectNode` today delivers every side + effect (fixed by this design). +- **F3** `PollingSource.keys()` polls when no schema was declared and nothing was polled. + +## Decisions requested + +- **D1** Return type `StreamHead` (recommended) vs `pa.Table` vs `pl.DataFrame`. +- **D2** Node stored reads for v0.3: whole stored slice, no computation (recommended), with + bounded `get_first_records` as a follow-up; or bounded in v0.3 (+1 day). +- **D3** `OperatorJobNode` in LOG mode with nothing in memory: no rows, note that rows of + earlier runs are stored but not replayed (recommended, matches `as_table`), or show them. +- **D4** Remove `view()` / `StreamView` (recommended) or keep `view()` as an alias of `head`. +- **D5** Fix F1 in this PR (recommended) or file it separately. + +## Implementation notes (November 12, 2026) + +- Tag columns come first; the line above the table names them (`tags: id, grp`). Polars' + HTML shows each column's dtype, so there is no separate schema line. +- Struct cells are left to Polars' rendering (the per-field rules are not applied). +- A loaded `FunctionJobNode` whose attached store lacks the records its saved run stored (the + wrong store) still raises from `head()`; the preview shows that error as "could not read". + Every other loaded-node state returns a head without raising. +- `check_head_size` and `head_column_config` live in `core/streams/base.py`, and `StreamHead` + / `HeadStatus` in `core/streams/head.py`, which imports nothing from `orcapod.types`: that + keeps `StreamProtocol.head` free of a new import cycle. +- `head()` is part of `StreamProtocol`. +- `_CachedFunctionPodStream` (added on main by ITL-988) drops stored meta columns in head as + in `as_table`. +- `StreamProtocol.head` returns `StreamHeadProtocol` (in `protocols/core_protocols/streams.py`), + which `StreamHead` implements; that keeps `head.py` out of the protocol import chain. + Delegating heads go through `head_of(head, stream)`. +- An array column is recognised by the array logical type's extension name + (`numpy_type.NDARRAY_EXTENSION_NAME`, read at call time through `importlib` because that + module sits above `head.py` in the import graph), never by sniffing bytes. Both stored + forms are shown as `ndarray `: the `{dtype, shape, data}` struct on main + today, and one `.npy` file per cell after PR #328 (ITL-981), whose header alone is read + (zero-copy buffer slice). Other binary cells show `bytes (N B)`. +- Long strings keep their first 56 and last 20 characters around an ellipsis, so ~140 + character hash keys stay distinguishable; Polars' own string shortening is raised to 80 + characters while rendering (coordinator input from the ITL-990 workshop rundown). +- A rowless head does not read its schema when that would poll a source upstream + (`schema_requires_io`); it then has no columns and says so (CodeRabbit round 1). +- #333 merged first (main eeeb993e). Its `StreamBase._preview` / `_preview_header`, `view()` and + `StreamView` are gone; `SpiralDBTableSource.head()` wraps its bounded `_head` read and uses + `_head_note` (still the one place for its wording) as the header text, through + `StreamHead.description`. An unlabelled Spiral source is shown by its table name. #333's + display tests now go through `head()` / `header()`; their scan assertions are unchanged. +- CodeRabbit rounds 3-4: a `CachedSource` not yet used shows its columns and `NOT_COMPUTED` + (its rows merge earlier cached rows with the wrapped source's, unknown until read); an + operator node whose input changed shows `NOT_COMPUTED` ("inputs changed") rather than its + kept rows; an async operator run with no rows keeps a typed empty result; `head_of` adapts + any `StreamHeadProtocol` result; the preview shows all `n` rows (Polars `tbl_rows=-1`). diff --git a/tests/test_core/sources/test_spiraldb_table_source_streaming.py b/tests/test_core/sources/test_spiraldb_table_source_streaming.py index 6cd8a92d6..384ead58e 100644 --- a/tests/test_core/sources/test_spiraldb_table_source_streaming.py +++ b/tests/test_core/sources/test_spiraldb_table_source_streaming.py @@ -608,15 +608,33 @@ def _scan_limits() -> list[object]: return [scan.kwargs.get("limit") for scan in fake_pyspiral.scan_log] def test_display_reads_one_head_of_ten_rows(self, big): - """``_repr_html_`` reads one 10-row head, headed with the table's size.""" + """The notebook display is ``head()``: one 10-row scan, headed with the size.""" source = SpiralDBTableSource(_PROJECT, "big", label="big") html = source._repr_html_() assert self._scan_limits() == [10] - assert html.startswith("SpiralDBTableSource[big] — first 10 rows of about 25\n") + assert "SpiralDBTableSource[big] — first 10 rows of about 25" in html assert "shape: (10, 3)" in html - assert "*subject_id" in html and "*timestamp" in html + assert "tags: subject_id, timestamp" in html + + def test_head_reads_one_bounded_scan(self, big): + """``head()`` reads one scan of at most 10 rows and says the table's size.""" + source = SpiralDBTableSource(_PROJECT, "big", label="big") + + head = source.head() + + assert self._scan_limits() == [10] + assert head.header() == "SpiralDBTableSource[big] — first 10 rows of about 25" + assert (head.total_rows, head.total_is_estimate) == (25, True) + assert len(head) == 10 + + def test_an_unlabelled_source_is_shown_by_its_table_name(self, big): + """With no label, the header names the table; the label itself is unchanged.""" + source = SpiralDBTableSource(_PROJECT, "big") + + assert source.head().header().startswith("SpiralDBTableSource[big] — ") + assert source.label == "SpiralDBTableSource" def test_a_table_above_the_cap_previews_without_raising(self, big): """A source refused by its cap still previews a 10-row head.""" @@ -632,68 +650,57 @@ def test_a_table_above_the_cap_previews_without_raising(self, big): def test_a_small_table_shows_all_its_rows(self, project): """A table of fewer than 10 rows is shown whole, saying so.""" - html = SpiralDBTableSource(_PROJECT, "event")._repr_html_() + head = SpiralDBTableSource(_PROJECT, "event").head() assert self._scan_limits() == [10] - assert " — all 5 rows\n" in html - assert "shape: (5, 3)" in html + assert head.header().endswith(" — all 5 rows") + assert len(head) == 5 def test_the_sources_own_limit_bounds_the_head(self, big): """A ``limit`` below 10 bounds the head, shown as all rows.""" - html = SpiralDBTableSource(_PROJECT, "big", limit=3)._repr_html_() + head = SpiralDBTableSource(_PROJECT, "big", limit=3).head() assert self._scan_limits() == [3] - assert " — all 3 rows\n" in html + assert head.header().endswith(" — all 3 rows") def test_a_limit_above_the_head_is_the_row_count_shown(self, big): """A ``limit`` above 10 reads 10 rows and is shown as the row count.""" - html = SpiralDBTableSource(_PROJECT, "big", limit=15)._repr_html_() + head = SpiralDBTableSource(_PROJECT, "big", limit=15).head() assert self._scan_limits() == [10] - assert " — first 10 rows of about 15\n" in html + assert head.header().endswith(" — first 10 rows of about 15") def test_where_is_applied_and_named(self, big): """A ``where`` restricts the head and is named in the header.""" - html = SpiralDBTableSource( - _PROJECT, "big", where={"subject_id": "m1"} - )._repr_html_() + head = SpiralDBTableSource(_PROJECT, "big", where={"subject_id": "m1"}).head() assert self._scan_limits() == [10] - assert ( + assert head.header().endswith( " — first 10 matching rows (where subject_id='m1'); " - "the table has about 25 rows\n" - ) in html + "the table has about 25 rows" + ) - def test_the_header_is_escaped_in_both_displays(self, project): + def test_the_header_is_escaped_in_the_display(self, project): """A ``where`` value with markup characters is shown as text, not markup.""" project.create_table("default.marked", key_schema=_KEYS).write( _rows(list(range(1, 26)), subject="") ) source = SpiralDBTableSource(_PROJECT, "marked", where={"subject_id": ""}) - for shown in (source._repr_html_(), source.view()._repr_html_()): - header = shown.splitlines()[0] - assert header.endswith( - "first 10 matching rows (where subject_id='<b>'); " - "the table has about 25 rows" - ) - - def test_view_reads_the_same_head(self, big): - """``view()`` reads the same 10-row head as the notebook display.""" - source = SpiralDBTableSource(_PROJECT, "big", label="big") - - html = source.view()._repr_html_() + html = source._repr_html_() - assert self._scan_limits() == [10] - assert html.startswith("SpiralDBTableSource[big] — first 10 rows of about 25\n") - assert "shape: (10, 3)" in html + assert ( + "first 10 matching rows (where subject_id='<b>'); " + "the table has about 25 rows" + ) in html + assert "'" not in html - def test_view_with_all_info_shows_system_columns(self, big): - """``view(all_info=True)`` shows the system tag columns.""" - html = SpiralDBTableSource(_PROJECT, "big").view(all_info=True)._repr_html_() + def test_head_with_all_info_shows_system_columns(self, big): + """``head(all_info=True)`` shows the system tag columns, from the same scan.""" + head = SpiralDBTableSource(_PROJECT, "big").head(all_info=True) assert self._scan_limits() == [10] - assert "_tag_record_id::" in html + assert any(c.startswith("_tag_record_id::") for c in head.table.column_names) def test_repr_reads_nothing(self, big): """``repr`` is the class name and scans nothing.""" @@ -706,17 +713,17 @@ def test_switched_off_preview_shows_the_schema_without_a_scan( self, big, monkeypatch ): # About the non-public switch: the fallback kept for the ITL-991 ruling. - """With the preview off, both displays show the empty schema, unscanned.""" + """With the preview off, ``head()`` shows the empty schema, unscanned.""" monkeypatch.setattr(SpiralDBTableSource, "_preview_reads_rows", False) source = SpiralDBTableSource(_PROJECT, "big", label="big") + head = source.head() html = source._repr_html_() - view = source.view()._repr_html_() assert fake_pyspiral.scan_log == [] - for shown in (html, view): - assert shown.startswith( - "SpiralDBTableSource[big] — rows not read; the table has about 25 rows\n" - ) - assert "shape: (0, 3)" in shown - assert "*subject_id" in html + assert head.header() == ( + "SpiralDBTableSource[big] — rows not read; the table has about 25 rows" + ) + assert head.table.num_rows == 0 + assert head.table.num_columns == 3 + assert "tags: subject_id, timestamp" in html diff --git a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py new file mode 100644 index 000000000..d682d3a05 --- /dev/null +++ b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py @@ -0,0 +1,53 @@ +"""An operator stream recomputes after an upstream changes (ITL-991, finding F1). + +``DynamicPodStream`` keeps the result of its operator. When an upstream is modified after +that, the kept result is stale and the next read must run the operator again. +""" + +from __future__ import annotations + +from unittest import mock + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource + + +@function_pod(output_keys="y") +def double(x: int) -> int: + """Double ``x``.""" + return 2 * x + + +def _source() -> ArrowTableSource: + table = pa.table( + {"id": [1, 2, 3], "x": [1, 2, 3]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="s") + + +def test_operator_stream_reruns_after_upstream_is_modified() -> None: + """A kept operator result is recomputed once an upstream changed.""" + fn_stream = double.pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) + out = SelectDataColumns(["y"])(fn_stream) + original = SelectDataColumns.static_process + # A spy on the non-public static_process: the test is about whether it runs again. + with mock.patch.object( + SelectDataColumns, "static_process", autospec=True, side_effect=original + ) as spy: + first = out.as_table() + assert spy.call_count == 1 + fn_stream.clear_cache() # the upstream is modified after the result was kept + second = out.as_table() + assert spy.call_count == 2 + assert first.equals(second) diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py new file mode 100644 index 000000000..2dac4d642 --- /dev/null +++ b/tests/test_core/streams/test_head_and_preview.py @@ -0,0 +1,731 @@ +"""``head()`` and the notebook preview show what is readily available and never compute (ITL-991). + +A stream's ``head(n)`` returns at most ``n`` rows it already holds, has cached from an earlier +computation, or can read from its table, and the preview (``_repr_html_``) renders exactly +that. Neither calls a function, runs an operator, delivers a side effect, polls a source or +writes a cache. +""" + +from __future__ import annotations + +import io +from typing import Any +from unittest import mock + +import numpy as np +import numpy.typing as npt +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream +from orcapod.core.operators import GroupBy, Join, PolarsFilter, SelectDataColumns +from orcapod.core.sources import ( + ArrowTableSource, + CachedSource, + DerivedSource, + DictSource, + ListSource, +) +from orcapod.core.sources.polling_source import PollingSource +from orcapod.databases import InMemoryArrowDatabase +from orcapod.streams import HeadStatus, StreamHead +from orcapod.side_effects import SideEffectPod +from orcapod.types import ColumnConfig, Cursor, Schema + +ALL_COLUMNS = ColumnConfig( + meta=True, context=True, source=True, system_tags=True, content_hash=True +) + + +def _table(rows: int, offset: int = 0) -> pa.Table: + """``id`` and ``x``, both non-nullable ``int64``.""" + return pa.table( + { + "id": list(range(offset, offset + rows)), + "x": [10 * i for i in range(offset, offset + rows)], + }, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + + +def _source(rows: int = 25, label: str = "readings") -> ArrowTableSource: + """An ``ArrowTableSource`` over ``_table(rows)``, labelled ``label``.""" + return ArrowTableSource( + _table(rows), tag_columns=["id"], source_id="readings", label=label + ) + + +# --------------------------------------------------------------------------- +# Sources and in-memory streams +# --------------------------------------------------------------------------- + + +def test_source_head_is_the_first_rows_with_the_exact_total() -> None: + """A source's head is the first rows of its table, with the exact row count.""" + source = _source() + head = source.head() + assert isinstance(head, StreamHead) + assert head.status is HeadStatus.ROWS + assert len(head) == 10 + assert head.total_rows == 25 + assert head.table.equals(source.as_table().slice(0, 10)) + assert head.header() == "ArrowTableSource[readings] — first 10 of 25 rows" + + +def test_head_of_a_short_source_says_all_rows() -> None: + """A source no longer than n says it shows all of its rows.""" + head = _source(rows=3).head() + assert len(head) == 3 + assert head.header() == "ArrowTableSource[readings] — all 3 rows" + + +def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: + """With every column group, head is the prefix of as_table, in stream order.""" + source = _source() + head = source.head(4, columns=ALL_COLUMNS) + assert head.table.equals(source.as_table(columns=ALL_COLUMNS).slice(0, 4)) + # all_info means every column group; head returns rows in stream order. + assert source.head(4, all_info=True).table.equals( + source.as_table(columns=ALL_COLUMNS).slice(0, 4) + ) + + +def test_head_refuses_negative_n_and_sorting() -> None: + """A negative n and sort_by_tags are refused.""" + source = _source() + with pytest.raises(ValueError, match="n"): + source.head(-1) + with pytest.raises(ValueError, match="sort"): + source.head(columns={"sort_by_tags": True}) + + +def test_head_of_zero_rows_keeps_the_columns() -> None: + """head(0) has no rows and keeps the columns.""" + head = _source().head(0) + assert head.status is HeadStatus.ROWS + assert head.table.column_names == ["id", "x"] + assert head.table.num_rows == 0 + + +@pytest.mark.parametrize( + "make", + [ + lambda: DictSource( + [{"id": i, "x": i} for i in range(12)], tag_columns=["id"], label="d" + ), + lambda: ListSource( + name="v", data=list(range(12)), tag_function=lambda e, i: {"i": i} + ), + ], + ids=["dict", "list"], +) +def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: + """Dict and list sources serve the first rows they hold.""" + source = make() + head = source.head(5) + assert head.status is HeadStatus.ROWS + assert head.total_rows == 12 + assert head.table.equals(source.as_table().slice(0, 5)) + + +def test_preview_is_the_rendered_head() -> None: + """The notebook preview is head() rendered.""" + source = _source() + html = source._repr_html_() + assert "ArrowTableSource[readings] — first 10 of 25 rows" in html + assert html == source.head()._repr_html_() + + +# --------------------------------------------------------------------------- +# Function pods +# --------------------------------------------------------------------------- + + +def _counting_pod(calls: list[int]) -> Any: + """A function pod tripling ``x`` that appends each input to ``calls``.""" + + @function_pod(output_keys="y") + def triple(x: int) -> int: + """Triple ``x``, recording the call.""" + calls.append(x) + return 3 * x + + return triple + + +def test_function_pod_stream_is_not_computed_by_head_or_preview() -> None: + """Neither head nor the preview calls the function of an uncomputed stream.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source(), label="triple") + head = stream.head() + html = stream._repr_html_() + assert calls == [] + assert head.status is HeadStatus.NOT_COMPUTED + assert len(head) == 0 + assert head.table.column_names == ["id", "y"] + assert "not computed" in html + assert "FunctionPodStream[triple]" in html + + +def test_function_pod_stream_shows_rows_computed_so_far() -> None: + """After an incomplete iteration, head shows the rows computed so far.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + rows = stream.iter_data() + for _ in range(3): + next(rows) + head = stream.head() + assert calls == [0, 10, 20] + assert head.status is HeadStatus.PARTIAL + assert head.table.column("y").to_pylist() == [0, 30, 60] + assert "computed so far" in head.header() + + +def test_function_pod_stream_shows_its_computed_rows() -> None: + """After as_table, head shows the computed rows without calling the function.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + table = stream.as_table() + assert len(calls) == 25 + head = stream.head(7) + assert len(calls) == 25 + assert head.status is HeadStatus.ROWS + assert head.total_rows == 25 + assert head.table.equals(table.slice(0, 7)) + assert head.header().endswith("first 7 of 25 computed rows") + + +def test_function_pod_stream_rows_from_iteration_only() -> None: + """After a full iteration, head is the prefix of as_table with every column group.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + _ = list(stream.iter_data()) + head = stream.head(4, columns=ALL_COLUMNS) + assert len(calls) == 25 + assert head.status is HeadStatus.ROWS + assert head.table.equals(stream.as_table(columns=ALL_COLUMNS).slice(0, 4)) + assert len(calls) == 25 + + +# --------------------------------------------------------------------------- +# Operators +# --------------------------------------------------------------------------- + + +def _other_source() -> ArrowTableSource: + """A source sharing ``id`` with ``_source``, with a ``z`` column for joins.""" + table = pa.table( + {"id": list(range(25)), "z": [i * i for i in range(25)]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("z", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="other") + + +def _grouped_source() -> ArrowTableSource: + """A source with an ``id`` and a ``grp`` tag, for grouping.""" + table = pa.table( + { + "id": list(range(25)), + "grp": [f"g{i % 3}" for i in range(25)], + "x": list(range(25)), + }, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="grouped") + + +OPERATOR_CASES = { + "join": (Join, lambda: Join()(_source(), _other_source())), + "group_by": (GroupBy, lambda: GroupBy(by=["grp"])(_grouped_source())), + "filter": ( + PolarsFilter, + lambda: PolarsFilter(constraints={"x": 20})(_source()), + ), +} + + +@pytest.mark.parametrize("case", list(OPERATOR_CASES), ids=list(OPERATOR_CASES)) +def test_operator_stream_is_not_run_by_head_or_preview(case: str) -> None: + """Neither head nor the preview runs the operator; its kept result is shown.""" + operator_class, build = OPERATOR_CASES[case] + stream = build() + original = operator_class.static_process + # A spy on the non-public static_process: the test is about whether it runs. + with mock.patch.object( + operator_class, "static_process", autospec=True, side_effect=original + ) as spy: + head = stream.head() + _ = stream._repr_html_() + assert spy.call_count == 0 + assert head.status is HeadStatus.NOT_COMPUTED + assert len(head) == 0 + table = stream.as_table() + assert spy.call_count == 1 + after = stream.head(2) + assert spy.call_count == 1 + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 2)) + + +def test_an_empty_operator_result_is_not_not_computed() -> None: + """A computed empty result is ROWS with no rows, not NOT_COMPUTED.""" + stream = PolarsFilter(constraints={"x": -1})(_source()) + _ = stream.as_table() + head = stream.head() + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert head.header().endswith("computed: no rows") + + +def test_operator_stream_with_changed_inputs_is_not_computed() -> None: + """A kept result whose input changed is not shown.""" + calls: list[int] = [] + fn_stream = _counting_pod(calls).pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) + out = SelectDataColumns(["y"])(fn_stream) + _ = out.as_table() + fn_stream.clear_cache() + head = out.head() + assert head.status is HeadStatus.NOT_COMPUTED + assert "inputs changed" in head.header() + + +# --------------------------------------------------------------------------- +# Side effects, polling and caching sources +# --------------------------------------------------------------------------- + + +def test_side_effect_stream_preview_never_delivers() -> None: + """Showing a side-effect stream delivers nothing.""" + delivered: list[int] = [] + + def deliver(x: int) -> None: + """The side effect; recording or refusing a delivery.""" + delivered.append(x) + + stream = SideEffectPod(deliver)(_source()) + head = stream.head() + html = stream._repr_html_() + assert delivered == [] + assert head.status is HeadStatus.NOT_COMPUTED + assert "displaying never delivers side effects" in html + + +class _FakeDynamicSource: + """A ``DynamicSourceProtocol`` double that records every poll and fetch.""" + + def __init__(self, declared: Schema | None = None) -> None: + """Set up the double.""" + self.polls = 0 + self.fetches = 0 + self.declared = declared + + def identity(self) -> Any: + """Return the double's identity.""" + return "fake" + + def to_config(self) -> dict[str, Any] | None: + """Return no config: the double is not serializable.""" + return None + + @classmethod + def from_config(cls, config: dict[str, Any]) -> _FakeDynamicSource: + """Refuse: the double is not serializable.""" + raise NotImplementedError + + def schema(self) -> Any: + """Return the declared schema, if any.""" + return self.declared + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + """Count the poll and say whether new data is there.""" + self.polls += 1 + return False + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + """Count the fetch and return the next batch.""" + self.fetches += 1 + return Cursor(value=1), { + "id": pa.array([1], type=pa.int64()), + "val": pa.array([7], type=pa.int64()), + } + + async def close(self) -> None: + """Release nothing.""" + pass + + +def test_polling_source_head_never_polls() -> None: + """A polling source's head shows polled rows only and never polls.""" + fake = _FakeDynamicSource() + source = PollingSource(fake, tag_columns="id") + head = source.head() + _ = source._repr_html_() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert "nothing polled yet" in head.header() + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + after = source.head() + assert (fake.polls, fake.fetches) == seen + assert after.status is HeadStatus.ROWS + assert after.table.column("val").to_pylist() == [7] + + +def test_polling_source_head_shows_declared_system_tags_without_polling() -> None: + """A declared schema answers head's columns, system tags included, without a poll.""" + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + source = PollingSource(fake, tag_columns="id") + head = source.head(columns={"system_tags": True}) + assert (fake.polls, fake.fetches) == (0, 0) + tag_schema, data_schema = source.output_schema(columns={"system_tags": True}) + assert head.table.column_names == [*tag_schema, *data_schema] + assert any(name.startswith("_tag") for name in head.table.column_names) + + +def test_derived_source_over_a_polling_source_never_polls() -> None: + """A derived source's head asks its origin for its head, so nothing is polled.""" + fake = _FakeDynamicSource() + origin = PollingSource(fake, tag_columns="id") + # Building the derived source reads the origin's columns, which polls it once. + derived = DerivedSource(origin, tags_as_data={"id": "id_value"}) + seen = (fake.polls, fake.fetches) + head = derived.head() + _ = derived._repr_html_() + assert (fake.polls, fake.fetches) == seen + assert head.status is HeadStatus.ROWS + assert head.table.column("id_value").to_pylist() == [1] + + +def test_downstream_of_an_unpolled_source_never_polls() -> None: + """A stream over a source whose columns are known only by polling shows none.""" + fake = _FakeDynamicSource() + source = PollingSource(fake, tag_columns="id") + + def deliver(val: int) -> None: + """The side effect; recording or refusing a delivery.""" + raise AssertionError("delivered") + + stream = SideEffectPod(deliver)(source) + head = stream.head() + _ = stream._repr_html_() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.num_columns == 0 + assert "columns not known until its source is polled" in head.header() + + +@pytest.mark.parametrize( + ("columns", "all_info"), + [ + (ColumnConfig(source=True), False), + (ColumnConfig(meta=True), False), + (ColumnConfig(context=True), False), + (ColumnConfig(content_hash=True), False), + (None, True), + ], + ids=["source", "meta", "context", "content_hash", "all_info"], +) +@pytest.mark.parametrize("kind", ["function_pod", "side_effect"]) +def test_downstream_head_never_polls_for_data_columns( + kind: str, columns: ColumnConfig | None, all_info: bool +) -> None: + """Asking for data-dependent columns downstream of an unpolled source never polls. + + A declared schema gives the user columns, but source info, meta, context and + content hashes are known only from polled rows: the head then shows no columns. + """ + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + source = PollingSource(fake, tag_columns="id") + + def deliver(val: int) -> None: + """The side effect; refusing a delivery.""" + raise AssertionError(f"delivered {val}") + + @function_pod(output_keys="out") + def copy(val: int) -> int: + """Return ``val``; refusing a call.""" + raise AssertionError(f"called with {val}") + + stream = ( + copy.pod(source) if kind == "function_pod" else SideEffectPod(deliver)(source) + ) + head = stream.head(columns=columns, all_info=all_info) + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.num_columns == 0 + assert "columns not known until its source is polled" in head.header() + + +def test_cached_source_head_writes_nothing() -> None: + """A cached source's head writes nothing to its cache store.""" + store = InMemoryArrowDatabase() + source = CachedSource(_source(), cache_database=store) + head = source.head(3) + _ = source._repr_html_() + assert store.list_table_paths() == [] + # Before first use its rows (earlier cached rows merged with the wrapped source's) + # are not known: the columns are shown, with no rows. + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.column_names == ["id", "x"] + table = source.as_table() + after = source.head(3) + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 3)) + + +# --------------------------------------------------------------------------- +# Rendering of large cells +# --------------------------------------------------------------------------- + + +def test_preview_summarises_large_cells_and_head_keeps_them() -> None: + """The preview summarises large cells; head's table keeps the full values.""" + source = DictSource( + [ + { + "id": 1, + "arr": np.arange(6.0).reshape(2, 3), + "blob": b"xyz" * 100, + "text": "a" * 200, + "items": list(range(12)), + } + ], + tag_columns=["id"], + label="cells", + ) + head = source.head() + html = head._repr_html_() + assert "ndarray float64 (2, 3)" in html + assert "bytes (300 B)" in html + assert "a" * 200 not in html + assert "…" in html + assert "list (len 12)" in html + assert head.table.equals(source.as_table()) + + +def test_text_form_has_the_header() -> None: + """The text form of a head starts with its header.""" + text = str(_source().head(2)) + assert text.startswith("ArrowTableSource[readings] — first 2 of 25 rows") + + +def test_cached_function_pod_stream_does_not_read_its_result_store() -> None: + """A cached pod's stream shows only computed rows and never calls the function.""" + from orcapod.core.cached_function_pod import CachedFunctionPod + + calls: list[int] = [] + store = InMemoryArrowDatabase() + pod = CachedFunctionPod(_counting_pod(calls).pod, result_database=store) + _ = pod.process(_source()).as_table() # fills the result store + assert len(calls) == 25 + + stream = pod.process(_source()) + head = stream.head() + assert head.status is HeadStatus.NOT_COMPUTED + table = stream.as_table() # served from the result store + assert len(calls) == 25 + after = stream.head(5) + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 5)) + assert len(calls) == 25 + + +class _NpyArrayType(pa.ExtensionType): + """The array logical type stored as one ``.npy`` file per cell (the ITL-981 form).""" + + def __init__(self) -> None: + """Set up the double.""" + from orcapod.logical_types.numpy_type import NDARRAY_EXTENSION_NAME + + super().__init__(pa.large_binary(), NDARRAY_EXTENSION_NAME) + + def __arrow_ext_serialize__(self) -> bytes: + """Return no extension metadata.""" + return b"" + + @classmethod + def __arrow_ext_deserialize__( + cls, storage_type: pa.DataType, serialized: bytes + ) -> _NpyArrayType: + """Rebuild the type.""" + return cls() + + +def _npy(array: npt.NDArray[Any]) -> bytes: + """Return ``array`` as the bytes of a ``.npy`` file (format 3.0).""" + buffer = io.BytesIO() + np.lib.format.write_array(buffer, array, version=(3, 0)) + return buffer.getvalue() + + +def test_preview_reads_an_npy_cell_header_only() -> None: + """An array cell stored as ``.npy`` shows its dtype and shape from the header.""" + storage = pa.array( + [_npy(np.zeros(40, dtype=np.float32)), _npy(np.ones((2, 3))), None], + type=pa.large_binary(), + ) + table = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int64()), + "frame": pa.ExtensionArray.from_storage(_NpyArrayType(), storage), + "jpeg": pa.array([b"\xff\xd8\xff\xe0" * 10, b"", None], pa.large_binary()), + } + ) + head = StreamHead( + table=table, + status=HeadStatus.ROWS, + kind="ArrowTableStream", + label="frames", + tag_columns=("id",), + total_rows=3, + ) + html = head._repr_html_() + assert "ndarray float32 (40,)" in html + assert "ndarray float64 (2, 3)" in html + assert "bytes (40 B)" in html + assert "NUMPY" not in html + + +def test_preview_keeps_both_ends_of_a_long_key() -> None: + """A long hash key keeps its start and its distinguishing end in the preview.""" + key = "derived:arrow_v1:" + "ab" * 61 + "XYZ" + source = DictSource([{"key": key, "x": 1}], tag_columns=["key"], label="keys") + html = source._repr_html_() + assert "derived:arrow_v1:abab" in html + assert "XYZ" in html + assert key not in html + assert source.head().table.column("key").to_pylist() == [key] + + +def test_an_unlabelled_source_is_shown_by_its_source_id() -> None: + """A source given no label is named by its source id in the header; its label is kept.""" + source = DictSource( + [{"id": 1, "x": 2}], tag_columns=["id"], source_id="spike_event" + ) + assert source.head().header() == "DictSource[spike_event] — all 1 row" + assert source.label == "DictSource" + + +class _BatchedDynamicSource(_FakeDynamicSource): + """Serves three batches of two rows, one per poll that reports new data.""" + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + """Count the poll and say whether new data is there.""" + self.polls += 1 + return (cursor.value if cursor is not None else 0) < 3 + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + """Count the fetch and return the next batch.""" + self.fetches += 1 + index = cursor.value if cursor is not None else 0 + return Cursor(value=index + 1), { + "id": pa.array([2 * index, 2 * index + 1], type=pa.int64()), + "val": pa.array([10 * index, 10 * index + 1], type=pa.int64()), + } + + +def test_polling_source_head_over_several_batches_is_the_prefix() -> None: + """With several polled batches, head is the prefix of all rows, with the full count.""" + fake = _BatchedDynamicSource() + source = PollingSource(fake, tag_columns="id") + for _ in range(3): + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + head = source.head(3) + assert (fake.polls, fake.fetches) == seen + assert head.total_rows == 6 + assert head.table.equals(source.as_table().slice(0, 3)) + + +def test_derived_source_rowless_head_has_its_own_columns() -> None: + """Before its origin polls, a derived source shows its columns, tag copies included.""" + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + origin = PollingSource(fake, tag_columns="id") + derived = DerivedSource(origin, tags_as_data={"id": "id_value"}) + head = derived.head() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert "id_value" in head.table.column_names + assert "nothing polled yet" in head.header() + + +def test_cached_source_keeps_the_wrapped_explanation_when_it_has_no_rows() -> None: + """A cached source over an unpolled source says why there are no rows.""" + source = CachedSource( + PollingSource(_FakeDynamicSource(), tag_columns="id"), + cache_database=InMemoryArrowDatabase(), + ) + head = source.head() + assert "nothing polled yet" in head.header() + + +def test_head_of_adapts_any_head_the_protocol_allows() -> None: + """A head that is not a ``StreamHead`` but follows the protocol is adapted.""" + from orcapod.core.streams.head import head_of + + class _PlainHead: + """A minimal ``StreamHeadProtocol`` result.""" + + table = _table(2) + status = "rows" + total_rows = 2 + + def header(self) -> str: + """Return a fixed header.""" + return "plain" + + def as_table(self) -> pa.Table: + """Return the rows.""" + return self.table + + def as_polars_df(self) -> Any: + """Return nothing: not used by the test.""" + return None + + def _repr_html_(self) -> str: + """Return an empty preview.""" + return "" + + source = _source() + head = head_of(_PlainHead(), source) + assert head.status is HeadStatus.ROWS + assert head.header() == "ArrowTableSource[readings] — all 2 rows" + + +def test_head_of_n_rows_renders_all_n_rows() -> None: + """The preview of ``head(n)`` shows all ``n`` rows, not Polars' default of 10.""" + text = str(_source().head(25)) + assert "shape: (25, 2)" in text + assert "…" not in text.split("\n", 2)[2] + + +def test_an_empty_computed_result_has_the_requested_columns() -> None: + """A computed empty result has the columns ``as_table`` gives, hashes included.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(PolarsFilter(constraints={"x": -1})(_source())) + _ = list(stream.iter_data()) + head = stream.head(columns={"content_hash": True}) + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert ( + head.table.column_names + == stream.as_table(columns={"content_hash": True}).column_names + ) diff --git a/tests/test_core/streams/test_streams.py b/tests/test_core/streams/test_streams.py index e0e8e5f96..3fcb254c1 100644 --- a/tests/test_core/streams/test_streams.py +++ b/tests/test_core/streams/test_streams.py @@ -534,23 +534,22 @@ def test_infer_schema_nullable_before_build_produces_plain_type(self): class TestNotebookDisplay: - """A stream shown in a notebook renders every row, tag columns first and marked.""" + """A stream shown in a notebook renders its ``head()``: the first rows, tags first.""" - def test_repr_html_is_the_header_and_every_row(self): + def test_repr_html_is_the_rendered_head(self): + """The display is ``head()`` rendered: header, tag names, the first 10 rows.""" stream = make_table_stream(n_rows=25) stream.label = "numbers" html = stream._repr_html_() - frame = stream.as_polars_df() - expected = frame.select(["id", "value"]).rename({"id": "*id"}) - assert html == "ArrowTableStream[numbers]\n" + expected._repr_html_() + assert html == stream.head()._repr_html_() + assert "ArrowTableStream[numbers] — first 10 of 25 rows" in html + assert "tags: id" in html + assert "shape: (10, 2)" in html - def test_view_is_the_header_and_every_row(self): + def test_head_is_the_first_rows_of_the_table(self): + """``head(n)`` holds the first ``n`` rows of ``as_table()``.""" stream = make_table_stream(n_rows=25) - stream.label = "numbers" - - html = stream.view()._repr_html_() - expected = stream.as_polars_df().rename({"id": "*id"}) - assert html == "ArrowTableStream[numbers]\n" + expected._repr_html_() + assert stream.head(5).table.equals(stream.as_table().slice(0, 5)) diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py new file mode 100644 index 000000000..028881ab0 --- /dev/null +++ b/tests/test_pipeline/test_node_head.py @@ -0,0 +1,300 @@ +"""A job node's ``head()`` shows its in-memory or stored rows and never runs it (ITL-991). + +A node is associated with its storage (developer, November 12, 2026): its preview is the +first rows of what it holds after a run, or of its stored table, and showing it never +computes. These tests run jobs on both orchestrators and both stores, and load saved jobs. +""" + +from __future__ import annotations + +from pathlib import Path +from unittest import mock + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream +from orcapod.core.nodes import OperatorJobNode +from orcapod.core.nodes.source_node import SourceJobNode, SourceNode +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.streams import HeadStatus +from orcapod.types import CacheMode, Schema + +CALLS: list[int] = [] + +ORCHESTRATORS = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, +} + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@pytest.fixture(autouse=True) +def _reset_calls() -> None: + """Clear the record of calls before each test.""" + CALLS.clear() + + +def _source(rows: int = 25) -> ArrowTableSource: + """A source of ``rows`` rows with ``id`` and ``x``, both non-nullable ``int64``.""" + table = pa.table( + {"id": list(range(rows)), "x": list(range(rows))}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="nums", label="nums") + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + """A fresh store of the given kind (``in_memory`` or ``delta``).""" + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +def _job(store: ArrowDatabaseProtocol) -> PipelineJob: + """A job of ``triple`` over the source, then a select of ``y``.""" + job = PipelineJob(name="itl991", store=store) + with job: + tripled = triple.pod(_source(), label="triple") + _ = SelectDataColumns(["y"])(tripled, label="sel") + return job + + +def test_nodes_before_a_run_show_no_rows_and_compute_nothing() -> None: + """Before a run, nodes show no rows and run neither the function nor the operator.""" + job = _job(InMemoryArrowDatabase()) + # The operator must not run: either entry point raises if it is called. + with ( + mock.patch.object( + SelectDataColumns, "process", side_effect=AssertionError("process ran") + ), + mock.patch.object( + SelectDataColumns, + "static_process", + side_effect=AssertionError("static_process ran"), + ), + ): + fn_head = job.nodes["triple"].head() + op_head = job.nodes["sel"].head() + _ = job.nodes["triple"]._repr_html_() + op_html = job.nodes["sel"]._repr_html_() + assert "could not read" not in op_html + assert CALLS == [] + assert fn_head.status is HeadStatus.NOTHING_STORED + assert fn_head.table.column_names == ["id", "y"] + assert op_head.status is HeadStatus.NOT_COMPUTED + assert "cache mode OFF" in op_head.header() + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_nodes_after_a_run_show_their_first_rows( + orchestrator: str, store_kind: str, tmp_path: Path +) -> None: + """Right after a run, nodes show their first rows from memory.""" + store = _store(store_kind, tmp_path) + job = _job(store) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert len(CALLS) == 25 + for label in ("triple", "sel"): + node = job.nodes[label] + # Right after a run the rows are in memory: the store is not read at all. + with mock.patch.object( + type(store), "get_all_records", side_effect=AssertionError("store read") + ): + head = node.head() + assert head.status is HeadStatus.ROWS, label + assert head.total_rows == 25 + assert head.table.equals(node.as_table().slice(0, 10)), label + assert "first 10 of 25" in node._repr_html_() + assert len(CALLS) == 25 + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing( + store_kind: str, tmp_path: Path +) -> None: + """A new job over a filled store shows stored rows without computing.""" + store = _store(store_kind, tmp_path) + _ = _job(store).run() + CALLS.clear() + + node = _job(store).nodes["triple"] + head = node.head(4) + assert CALLS == [] + assert head.status is HeadStatus.ROWS + # The whole stored slice is read, so the header states the exact count and + # makes no claim of a bounded read. + assert head.header() == "FunctionJobNode[triple] — first 4 of 25 stored rows" + assert head.table.equals(node.as_table().slice(0, 4)) + + +def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None: + """A loaded job's nodes show stored rows, or say why there are none.""" + store = DeltaTableDatabase(base_path=tmp_path / "store") + job = _job(store) + _ = job.run() + saved = tmp_path / "job.json" + job.save(saved) + CALLS.clear() + + loaded = PipelineJob.load(str(saved), store=store) + fn_head = loaded.nodes["triple"].head() + op_head = loaded.nodes["sel"].head() + assert CALLS == [] + assert fn_head.status is HeadStatus.ROWS + assert fn_head.header().endswith("first 10 of 25 stored rows") + assert op_head.status is HeadStatus.UNAVAILABLE + assert "cache mode OFF" in op_head.header() + + +def test_blueprint_nodes_are_unavailable() -> None: + """Blueprint nodes have no rows to show.""" + job = _job(InMemoryArrowDatabase()) + for label in ("triple", "sel"): + blueprint = job.nodes[label].as_node() + head = blueprint.head() + assert head.status is HeadStatus.UNAVAILABLE + assert "pipeline blueprint" in blueprint._repr_html_() + + +def test_source_nodes() -> None: + """Input slots and unbound source nodes show no rows; bound ones delegate.""" + slot = SourceNode( + name="nums", tag_schema=Schema({"id": int}), data_schema=Schema({"x": int}) + ) + assert slot.head().status is HeadStatus.UNAVAILABLE + assert "input slot" in slot._repr_html_() + unbound = SourceJobNode( + name="nums", tag_schema=Schema({"id": int}), data_schema=Schema({"x": int}) + ) + assert unbound.head().status is HeadStatus.UNAVAILABLE + assert "no source bound" in unbound.head().header() + source = _source() + bound = SourceJobNode(name="nums", bound_source=source) + assert bound.head().table.equals(source.head().table) + assert bound.head().header() == "SourceJobNode[nums] — first 10 of 25 rows" + + +def _operator_node( + cache_mode: CacheMode, store: InMemoryArrowDatabase +) -> OperatorJobNode: + """An operator node selecting ``x`` from the source, on ``store``.""" + return OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source()], + pipeline_database=store, + cache_mode=cache_mode, + ) + + +def test_operator_node_cache_modes() -> None: + """Operator nodes show stored rows in REPLAY and none in LOG before a run.""" + store = InMemoryArrowDatabase() + assert _operator_node(CacheMode.REPLAY, store).head().status is ( + HeadStatus.NOTHING_STORED + ) + seed = _operator_node(CacheMode.LOG, store) + _ = seed.execute(_source()) + + replay = _operator_node(CacheMode.REPLAY, store) + head = replay.head(3) + assert head.status is HeadStatus.ROWS + assert ( + head.header() + == "OperatorJobNode[SelectDataColumns] — first 3 of 25 stored rows" + ) + assert head.table.equals(replay.as_table().slice(0, 3)) + + log = _operator_node(CacheMode.LOG, store) + log_head = log.head() + assert log_head.status is HeadStatus.NOT_COMPUTED + assert "not replayed" in log_head.header() + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_an_operator_node_with_an_empty_result_reads_as_computed( + orchestrator: str, +) -> None: + """After a run that produced no rows, an operator node is computed and empty.""" + from orcapod.core.operators import PolarsFilter + + job = PipelineJob(name="itl991_empty", store=InMemoryArrowDatabase()) + with job: + _ = PolarsFilter(constraints={"x": -1})(_source(), label="none") + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + head = job.nodes["none"].head() + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert head.table.column_names == ["id", "x"] + + +def test_an_operator_node_with_changed_inputs_shows_no_stale_rows() -> None: + """After an input changed, an operator node's kept rows are not shown.""" + fn_stream = triple.pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) + node = OperatorJobNode(operator=SelectDataColumns(["y"]), input_streams=[fn_stream]) + _ = node.execute(fn_stream) + assert node.head().status is HeadStatus.ROWS + fn_stream.clear_cache() # the input is modified after the run + head = node.head() + assert head.status is HeadStatus.NOT_COMPUTED + assert "inputs changed" in head.header() + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_head_keeps_the_tag_arrow_type_of_the_input( + orchestrator: str, store_kind: str, tmp_path: Path +) -> None: + """A node's head keeps an ``int8`` tag ``int8`` and is the prefix of ``as_table``. + + ``head`` and ``as_table`` share one table build (ITL-928 typing), in memory after a + run and from the store in a new job. + """ + table = pa.table( + {"id": pa.array(range(12), pa.int8()), "x": pa.array(range(12), pa.int64())}, + schema=pa.schema( + [ + pa.field("id", pa.int8(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + + def build(store: ArrowDatabaseProtocol) -> PipelineJob: + """Build a job of ``triple`` over the narrow-tag source on ``store``.""" + job = PipelineJob(name="itl991_types", store=store) + with job: + _ = triple.pod( + ArrowTableSource(table, tag_columns=["id"], source_id="narrow"), + label="triple", + ) + return job + + store = _store(store_kind, tmp_path) + job = build(store) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + for node in (job.nodes["triple"], build(store).nodes["triple"]): + head = node.head(5, all_info=True) + assert head.status is HeadStatus.ROWS + assert head.table.schema.field("id").type == pa.int8() + assert head.table.equals(node.as_table(all_info=True).slice(0, 5))