Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
7063901
docs(specs): ITL-991 head() and a preview that never computes (design)
Nov 12, 2026
fc3b11c
fix(operators): an operator stream recomputes after an upstream chang…
Nov 12, 2026
cf25515
test(streams): head() and the preview never compute (ITL-991)
Nov 12, 2026
0b30e5f
feat(streams): head() and a preview that never computes, for streams …
Nov 12, 2026
45fda03
feat(streams): operator, function pod and side-effect streams show on…
Nov 12, 2026
d36bd49
Merge remote-tracking branch 'origin/main' into eywalker/itl-991-a-st…
Nov 12, 2026
184b549
feat(streams): a cached function pod's stream shows its computed rows…
Nov 12, 2026
1e227da
test(nodes): a node's head shows its in-memory or stored rows and nev…
Nov 12, 2026
7e91225
feat(nodes): job nodes show their in-memory or stored rows and never …
Nov 12, 2026
6542dd4
docs: looking at data with head() and the preview; changelog and desi…
Nov 12, 2026
c392b99
test(nodes): a node's head right after a run reads nothing from the s…
Nov 12, 2026
5fe6d96
fix(streams): address CodeRabbit round 1 on head() (ITL-991)
Nov 12, 2026
142db4a
feat(streams): preview shows array cells from either stored form, lon…
Nov 12, 2026
b16f159
fix(sources): address CodeRabbit round 2 on head() (ITL-991)
Nov 13, 2026
1658eb7
Merge origin/main (#333): head() replaces the private _preview hook (…
Nov 13, 2026
ff279bc
fix(nodes): address CodeRabbit round 3 on head() (ITL-991)
Nov 13, 2026
7ce91ec
Merge origin/main (#332) into ITL-991 head()
Nov 13, 2026
b5499db
fix(streams): address CodeRabbit review of the #333 merge (ITL-991)
Nov 13, 2026
53b2f93
fix(streams): address CodeRabbit round 5 on head() (ITL-991)
Nov 13, 2026
e3c4c6c
Merge origin/main (#335, #328, #336) into ITL-991 head()
Nov 13, 2026
19ad76c
docs(plan): record the merge of origin/main into ITL-991
Nov 13, 2026
36a8cc8
fix(streams): a rowless head never polls for data-dependent columns (…
Nov 13, 2026
a75ba1a
Merge origin/main (#337, #338) into ITL-991 head()
Nov 13, 2026
4609de2
Merge origin/main (#286) into ITL-991 head()
Nov 13, 2026
a291e00
docs(plan): record the CodeRabbit round on 19ad76ce
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions changelog.d/ITL-186.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,9 @@ What changes for existing code:
not part of the source's identity (ITL-186).
- `SpiralDBTableSource(..., max_rows=, batch_rows=)`, the `row_count_estimate` and `table_id`
properties, and `orcapod.errors.SourceTooLargeError` (ITL-186).
- A `SpiralDBTableSource` shown in a notebook (its HTML display, or `view()`) reads only its
first 10 rows, in one scan to which `max_rows` does not apply, and its header says how
many rows the table has, for example `first 10 rows of about 63,349`. Other streams
display as before (ITL-186).
- A `SpiralDBTableSource` shown in a notebook (its `head()`) reads only its first 10 rows,
in one scan to which `max_rows` does not apply, and its header says how many rows the
table has, for example `first 10 rows of about 63,349` (ITL-186).
- `SpiralDBConnector.scan_batches(table, where=, limit=, asof=)`, `get_arrow_schema`,
`get_spiral_table_id`, `get_table_version` and `get_row_count_estimate`: a restricted scan
and metadata-only reads (ITL-186).
Expand Down
30 changes: 30 additions & 0 deletions changelog.d/ITL-991.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
### Breaking changes and migration

#### Showing a stream never computes it (ITL-991)

Displaying a stream, source or node in a notebook no longer computes anything. The preview
shows the first 10 rows that are already available: rows a source holds, rows cached from an
earlier computation, rows a job node holds after a run or has stored, or a bounded read of a
source's table. Otherwise it shows the columns and a note such as "not computed" or "nothing
stored; run the job". Call `as_table()` or run the job to compute. Before, displaying a
function pod's output called the function on every row, displaying an operator's output ran
the operator, displaying a side-effect stream delivered every side effect, and displaying a
`CachedSource` wrote its cache table; none of these happen now.

`StreamBase.view()` and `StreamView` are removed: use `head(n)` or `as_polars_df()`.

### Added

- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a
`StreamHead` (`orcapod.streams`): at most `n` rows as a PyArrow table with their exact
types, the row count when it is known without computing, and a `status` (`HeadStatus`) that
tells an empty result from one that is not computed. It never calls a function, runs an
operator, delivers a side effect, polls a source or writes. The notebook preview is
`head()` rendered, with large cells (arrays, bytes, long text, long lists) summarised. A job
node with nothing in memory reads its whole stored slice to show its first rows. (ITL-991)

### Fixed

- An operator's output stream computed again after one of its inputs changed now does so:
before, it served the result kept from before the change (`DynamicPodStream` cleared a
misspelled attribute instead of its kept result). (ITL-991)
22 changes: 22 additions & 0 deletions design_issues/ITL-991-PV1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## `src/orcapod/core/streams/` — notebook preview and `head()`

### PV1 — Showing a stream computed it, delivered side effects and wrote caches
**Status:** resolved
**Severity:** high
**Issue:** ITL-991

`StreamBase._repr_html_` and `StreamBase.view()` called `as_polars_df()` on the whole
stream. Displaying a function pod's output in a notebook called the function on every row,
an operator's output ran the operator, a `SideEffectPodStream` or `SideEffectNode` delivered
every side effect, a `PollingSource` polled, a `CachedSource` wrote its cache table, and a
blueprint node raised inside the formatter. The developer ruled (November 12, 2026) that a
preview shows only what is readily available and never triggers computation, and that a
node's preview is a head of its storage.

**Fix:** every stream, source and node has `head(n=10, columns=..., all_info=...)`
returning a `StreamHead` (rows, `total_rows`, `HeadStatus`), and `_repr_html_` renders
`head()`. The default shows the columns and `NOT_COMPUTED`; only classes that can serve rows
without computing override it. `view()` and `StreamView` are removed. A job node with nothing
in memory reads its whole stored slice; a bounded database read is a follow-up. The same
work fixed `DynamicPodStream` serving a stale result after an input changed (misspelled
`_cached_results` in `_update_cache_status`).
7 changes: 4 additions & 3 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,12 +152,13 @@ node, so 100,000 rows cost about 0.5 GB per node. A table of millions of rows ca
with `max_rows=None`, but it cannot yet be run through a job on a laptop.

**Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell
(or through `view()`) reads only its first 10 rows, in one scan to which the row cap does not
apply, so a table of any size can be looked at. The header line says what is shown, for
(its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10
rows, in one scan to which the row cap does not apply, so a table of any size can be looked
at; `head(n)` reads `n`. The header line says what is shown, for
example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's
recorded row count, read without scanning), `all 4 rows` when the source reads no more than
that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with
`where=`. Tag columns come first, marked with `*`. To see more rows, use `as_df()` with a
`where=`. Tag columns come first, and the line above the table names them. To see more rows, use `as_df()` with a
`limit=` or `where=` source.

**Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the
Expand Down
71 changes: 71 additions & 0 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,77 @@ df = table.to_pandas()

Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_lazy_frame()`.

## Looking at data

In a notebook, a stream, source or job node shown at the end of a cell renders a short
table: its first rows, with a line saying what they are. That preview is `head()` rendered,
and `head(n)` gives the same rows to work with. Neither ever computes anything: they show
only what is readily available, and never call a function, run an operator, deliver a side
effect, poll a source or write to a database.

```python
from orcapod import function_pod
from orcapod.sources import DictSource

subjects = DictSource(
data=[
{"subject_id": f"mouse_{i:02d}", "age": 8 + i, "weight": 20.0 + i}
for i in range(1, 26)
],
tag_columns=["subject_id"],
label="subjects",
)

head = subjects.head()
print(head.header())
# DictSource[subjects] — first 10 of 25 rows
print(head.status.value, len(head))
# rows 10
print(subjects.head(3).as_table().column("age").to_pylist())
# [9, 10, 11]
```

`head()` returns a `StreamHead`: the rows as a PyArrow table (`.table`, `.as_table()`,
`.as_polars_df()`), the row count when it is known without computing (`.total_rows`), and a
`.status` that says where the rows came from. A stream downstream of a computation shows rows
only once they are computed:

```python
@function_pod(output_keys="age_months")
def to_months(age: int) -> int:
return age * 12


months = to_months.pod(subjects, label="months")
print(months.head().header())
# FunctionPodStream[months] — not computed; as_table(), iter_data() or a job run computes it
print(months.head().status.value, months.head().table.column_names)
# not_computed ['subject_id', 'age_months']

table = months.as_table() # computes every row
print(months.head(3).header())
# FunctionPodStream[months] — first 3 of 25 computed rows
```

What each kind of object shows:

| Object | `head()` shows |
|---|---|
| A source | its first rows (a bounded read for a source that reads its table lazily) |
| An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` |
| A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) |
| A pipeline blueprint node, an unbound input slot | none (`unavailable`) |
| A side-effect stream | none: showing it never delivers its side effects |

A head of an operator is never the operator applied to the heads of its inputs: a group over
the first ten rows has the wrong members. So an uncomputed stream shows its columns and
"not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result
shows status `rows` with zero rows, so it is never mistaken for "not computed".

In the preview, large cells are summarised (an array as `ndarray float64 (2, 3)`, bytes as
`bytes (300 B)`, long text and long lists shortened); `head().table` keeps the full values.
In a terminal, `print(stream.head())` prints the same preview as text.

## Controlling column visibility with `ColumnConfig`

By default, streams only expose user-facing tag and data columns. Orcapod also maintains
Expand Down
42 changes: 37 additions & 5 deletions src/orcapod/core/cached_function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@

from __future__ import annotations

import dataclasses
import logging
from datetime import datetime, timezone
from collections.abc import Iterator
from typing import TYPE_CHECKING, Any

from orcapod.core.datagrams import Datagram
from orcapod.core.function_pod import FunctionPodStream, WrappedFunctionPod
from orcapod.core.streams.head import StreamHead
from orcapod.system_constants import constants
from orcapod.types import ColumnConfig
from orcapod.contexts.stored import StaleTables
Expand Down Expand Up @@ -451,12 +453,35 @@ class _CachedFunctionPodStream(FunctionPodStream):
"""

def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]:
keep = CachedFunctionPod.RESULT_COMPUTED_FLAG
for tag, data in super().iter_data():
stored = [name for name in data.meta_columns if name != keep]
if stored:
data = data.drop_meta_columns(*stored)
yield tag, data
yield tag, _without_stored_meta(data)

def _computed_prefix(
self, limit: int
) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]:
shown, total = super()._computed_prefix(limit)
return [(tag, _without_stored_meta(data)) for tag, data in shown], total

def head(
self,
n: int = 10,
*,
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> StreamHead:
"""Return the first computed rows, with the columns ``as_table`` gives.

The pod's result store is not looked up: that needs the input rows' hashes,
i.e. reading the input. See ``FunctionPodStream.head``.
"""
head = super().head(n, columns=columns, all_info=all_info)
column_config = ColumnConfig.handle_config(columns, all_info=all_info)
if column_config.meta or column_config.all_info:
return head
table = head.table.drop_columns(
[n for n in head.table.column_names if n.startswith(constants.META_PREFIX)]
)
return dataclasses.replace(head, table=table)

def as_table(
self,
Expand All @@ -471,3 +496,10 @@ def as_table(
return table.drop_columns(
[n for n in table.column_names if n.startswith(constants.META_PREFIX)]
)


def _without_stored_meta(data: DataProtocol) -> DataProtocol:
"""Return ``data`` with only the ``__computed`` meta column of its stored ones."""
keep = CachedFunctionPod.RESULT_COMPUTED_FLAG
stored = [name for name in data.meta_columns if name != keep]
return data.drop_meta_columns(*stored) if stored else data
Loading
Loading