Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
44cbd4c
docs(plan): ITL-1053 head sorting plan
Nov 13, 2026
66859c0
feat(streams)!: head honours sort_by_tags and all_info like as_table …
Nov 13, 2026
a4db4ae
test(streams): type the head sorting cases for basedpyright (ITL-1053)
Nov 13, 2026
156f6ac
feat(streams): one deterministic tag sort and a tag-sorted notebook p…
Nov 13, 2026
baede99
test(streams): type the preview cases (ITL-1053)
Nov 13, 2026
9b1bc31
Merge remote-tracking branch 'origin/main' into eywalker/itl-1053-hea…
Nov 13, 2026
a65d7ea
Merge remote-tracking branch 'origin/main' into eywalker/itl-1053-hea…
Nov 13, 2026
982a2de
Merge remote-tracking branch 'origin/main' into eywalker/itl-1053-hea…
Nov 13, 2026
f01fcfa
fix(streams): keep system tags through the sort, custom hash column a…
Nov 13, 2026
93d6d7e
test(spiraldb): type the read order (ITL-1053)
Nov 13, 2026
eb34ac0
test(spiraldb): cast the read order for the type checker (ITL-1053)
Nov 13, 2026
e0fd5ad
Merge remote-tracking branch 'origin/main' into eywalker/itl-1053-hea…
Nov 13, 2026
fa39b0d
fix(streams): keep a tagless stream's identity order, sorted heads wi…
Nov 13, 2026
95a4f52
test(hashing): pin the identity hash of a tagless producerless stream…
Nov 13, 2026
e1f3508
test(hashing): type the tagless stream golden case (ITL-1053)
Nov 13, 2026
6710b2f
fix(streams): system tag priority follows the stream's order; hash ca…
Nov 13, 2026
6c7128f
fix(streams): an ArrowTableStream keeps the order of tags Arrow canno…
Nov 13, 2026
4a97fe0
refactor(streams): a stream's identity keeps main's row order in its …
Nov 13, 2026
4a72158
fix(streams): a requested hash column name never acts as a system tag…
Nov 13, 2026
282a0e2
refactor(streams): one helper carries a requested content-hash column…
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions changelog.d/ITL-1053.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
### Changed

- `head()` takes the column configuration of `as_table`, sorting included, and applies it to
the rows it has before taking the first `n` (ITL-1053). `head(n, columns={"sort_by_tags":
True})` no longer raises, and `head(n, all_info=True)` now sorts by tag, as `as_table`
does; the default `head()` stays unsorted, as `as_table()` is. The notebook preview asks for the
sorted head, so it shows rows in tag order whatever order they were computed in, and its
header line says "sorted by tags" (for a bounded read, "10 rows read, sorted by tags, of
about N"). A stream that has all
its rows (a job node's stored rows, a computed stream, an in-memory source) gives exactly
`as_table(...).slice(0, n)` for every configuration, whatever order its rows were computed
in. A stream that has read or computed only part of its rows (a `SpiralDBTableSource`
head, a partly computed stream) sorts the rows it has: the rows read, in tag order.

### Fixed

- A node's `head(n, all_info=True)` no longer differs from `as_table(all_info=True)`'s first
rows when an async run completed rows out of tag order (the cause of an occasionally
failing test, ITL-1001).
- Every tag sort is now one deterministic sort (`arrow_utils.sort_by_tag_columns`): by the
tag columns, nulls last, rows with equal tags ordered by their system tag columns. A job
node's and a function pod stream's sorted `as_table` used an unstable Polars sort that put
nulls first and left the order of rows with equal tags unspecified; it now orders them as
an `ArrowTableStream` does. Only the order of rows within equal or null tags changes: no
values, record ids or hashes.
31 changes: 31 additions & 0 deletions design_issues/ITL-1053-PV2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
## `src/orcapod/core/streams/` — notebook preview and `head()`

### PV2 — `head()` refused `sort_by_tags` and showed rows in completion order
**Status:** resolved
**Severity:** medium
**Issue:** ITL-1053

`head()` (ITL-991) was built as "the first rows in stream order": `head_column_config` switched
tag sorting off even under `all_info=True` and raised `ValueError` when `sort_by_tags` was
asked. After an async run a node's stored order is completion order, so its head showed rows
in an order that varied from run to run, and a test asserting that `head(5, all_info=True)`
equals the prefix of `as_table(all_info=True)` failed whenever rows finished out of tag order
(`test_head_keeps_the_tag_arrow_type_of_the_input[async-delta]`, seen on CI; ITL-1001).

**Fix:** developer decision, November 12, 2026: "make it consistent with as_table".
`head_column_config` is removed; `head` takes `as_table`'s column configuration, sorting
included, and applies it to the rows it has, then takes the first `n`. The default stays
unsorted. A node, an `ArrowTableStream`, a computed `FunctionPodStream` and a `DerivedSource`
over a job node shape and sort every row they have (each row keeps its own content hash,
ITL-1029) and slice, so `head(n, cfg) == as_table(cfg).slice(0, n)`. A bounded read
(`SpiralDBTableSource`, a partly computed stream, a polled source's batches) sorts the rows it
has: "the rows read, in tag order", not "the first `n` in tag order". Nothing computes or reads
more than before. When ITL-1002 bounds a node's stored read, a node's sorted head becomes "the
rows read, sorted" unless the store returns rows in tag order.

The notebook preview asks for the tag-sorted head (developer, November 12, 2026: "preview should
sort by tags"; `preview_html`), and its header says "sorted by tags", or for a bounded read
"N rows read, sorted by tags, of about T". Every tag sort goes through
`arrow_utils.sort_by_tag_columns`: tags ascending, nulls last, ties by the system tag columns,
then input order (the node and function pod stream sorts were an unstable Polars sort with nulls
first).
15 changes: 9 additions & 6 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,12 +154,15 @@ with `max_rows=None`, but it cannot yet be run through a job on a laptop.
**Looking at a table in a notebook.** A Spiral source shown at the end of a notebook cell
(its `head()`, see [Looking at data](streams.md#looking-at-data)) reads only its first 10
rows, in one scan to which the row cap does not apply, so a table of any size can be looked
at; `head(n)` reads `n`. The header line says what is shown, for
example `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` (the table's
recorded row count, read without scanning), `all 4 rows` when the source reads no more than
that, or `first 10 matching rows (where unit_id=106); the table has about 63,349 rows` with
`where=`. Tag columns come first, and the line above the table names them. To see more rows, use `as_df()` with a
`limit=` or `where=` source.
at; `head(n)` reads `n`. The preview shows the rows it read in tag order (it asks for
`sort_by_tags`, as does `head(columns={"sort_by_tags": True})` or `all_info=True`): the rows
read, sorted, not the first `n` of the table in tag order. The header line says what is
shown, for example `SpiralDBTableSource[spike_event] — 10 rows read, sorted by tags, of
about 63,349` (the table's recorded row count, read without scanning), `all 4 rows, sorted
by tags` when the source reads no more than that, or `10 matching rows read, sorted by tags
(where unit_id=106); the table has about 63,349 rows` with `where=`. The unsorted `head()`
says `first 10 rows of about 63,349`. Tag columns come first, and the line above the table
names them. To see more rows, use `as_df()` with a `limit=` or `where=` source.

**Restrictions are not part of identity.** `where`, `limit` and `asof` choose rows of the
same table, so two sources on one table have the same `source_id` and `content_hash`
Expand Down
35 changes: 33 additions & 2 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,9 @@ Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_l
## Looking at data

In a notebook, a stream, source or job node shown at the end of a cell renders a short
table: its first rows, with a line saying what they are. That preview is `head()` rendered,
and `head(n)` gives the same rows to work with. Neither ever computes anything: they show
table: its first rows in tag order, with a line saying what they are. That preview is
`head()` asked for `sort_by_tags` and rendered, and `head(n, columns={"sort_by_tags": True})`
gives the same rows to work with. Neither ever computes anything: they show
only what is readily available, and never call a function, run an operator, deliver a side
effect, poll a source or write to a database.

Expand Down Expand Up @@ -162,6 +163,36 @@ What each kind of object shows:
| A pipeline blueprint node, an unbound input slot | none (`unavailable`) |
| A side-effect stream | none: showing it never delivers its side effects |

`head(n, columns=..., all_info=...)` takes the same column configuration as `as_table`,
sorting included, and applies it to the rows it has before taking the first `n`. The default
does not sort, as `as_table()` does not, so `head()` shows rows in the order the stream
holds them. The notebook preview asks for the sorted head, so it shows the same rows
whatever order they were computed in. With `sort_by_tags=True`, or `all_info=True`, rows are
sorted by tag first (nulls last; rows with equal tags are ordered by their system tags):

```python
from orcapod.sources import DictSource

unordered = DictSource(
data=[{"subject_id": f"mouse_{i:02d}", "age": 8 + i} for i in (3, 1, 2)],
tag_columns=["subject_id"],
label="unordered",
)
print(unordered.head(2).as_table().column("subject_id").to_pylist())
# ['mouse_03', 'mouse_01']
print(unordered.head(2, all_info=True).as_table().column("subject_id").to_pylist())
# ['mouse_01', 'mouse_02']
```

A stream that has all of its rows (a job node's stored rows, a computed stream, an in-memory
source) then gives exactly the first `n` rows of `as_table` with the same configuration,
whatever order the rows were computed in. A stream that has only some of its rows sorts
those: a source that reads a bounded number of rows (such as a `SpiralDBTableSource`) shows
the rows it read, in tag order, and a stream computed only in part shows the rows computed so
far in tag order. It is not "the first `n` rows in tag order" of rows it never read, and the
preview's header line says so ("10 rows read, sorted by tags, of about 63,349"). A stream
with no tag columns previews its rows in stream order.

A head of an operator is never the operator applied to the heads of its inputs: a group over
the first ten rows has the wrong members. So an uncomputed stream shows its columns and
"not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result
Expand Down
4 changes: 2 additions & 2 deletions src/orcapod/core/cached_function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -459,9 +459,9 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]:
yield tag, _without_stored_meta(data)

def _computed_prefix(
self, limit: int
self, limit: int, contiguous: bool = True
) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]:
shown, total = super()._computed_prefix(limit)
shown, total = super()._computed_prefix(limit, contiguous)
return [(tag, _without_stored_meta(data)) for tag, data in shown], total

def head(
Expand Down
113 changes: 57 additions & 56 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
from orcapod.core.streams.base import (
StreamBase,
check_head_size,
head_column_config,
schema_requires_io,
)
from orcapod.core.streams.head import (
Expand Down Expand Up @@ -1369,11 +1368,17 @@ def head(
the first of the result. After an iteration that stopped part way they are the
rows computed so far (``PARTIAL``). Before any computation, or once an input
changed, there are none. A ``CachedFunctionPod``'s result store is not looked
up: that needs the input rows' hashes, i.e. reading the input. See
``StreamBase.head``.
up: that needs the input rows' hashes, i.e. reading the input. With
``sort_by_tags`` (or ``all_info``) the rows held are sorted by tag and the first
``n`` of those returned: after a complete computation that is
``as_table(...).slice(0, n)``; after a partial one, the rows computed so far in
tag order ("the rows it has", not the first ``n`` of the whole result in tag
order). See ``StreamBase.head``.
"""
check_head_size(n)
config = head_column_config(columns, all_info)
config = ColumnConfig.handle_config(columns, all_info=all_info)
# Sorting needs every row held, not only the first n: read them all (in memory).
wanted = len(self._cached_output_datas) if config.sort_by_tags else n
has_results = bool(self._cached_output_datas) or (
self._cached_output_table is not None
)
Expand Down Expand Up @@ -1405,7 +1410,9 @@ def head(
# A complete computation: its table holds the rows; only the shown rows'
# data are read, and only when their content hashes are asked for.
def shown_content_hashes() -> pa.Array[Any]:
shown, _ = self._computed_prefix(n)
shown, _ = self._computed_prefix(
wanted, contiguous=not config.sort_by_tags
)
return pa.array(
[data.content_hash().to_string() for _, data in shown],
type=pa.large_string(),
Expand All @@ -1414,8 +1421,12 @@ def shown_content_hashes() -> pa.Array[Any]:
return StreamHead.of_rows(
self,
self._shape_table(
self._cached_output_table.slice(0, n), config, shown_content_hashes
),
self._cached_output_table
if config.sort_by_tags
else self._cached_output_table.slice(0, n),
config,
shown_content_hashes,
).slice(0, n),
tag_columns=tag_columns,
total_rows=self._cached_output_table.num_rows,
origin="computed",
Expand All @@ -1427,7 +1438,7 @@ def shown_content_hashes() -> pa.Array[Any]:
)
if not finished and not self._cached_output_datas:
return self._head_without_rows(config, HeadStatus.NOT_COMPUTED)
shown, total = self._computed_prefix(n)
shown, total = self._computed_prefix(wanted, contiguous=not config.sort_by_tags)
status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL
if not shown:
# Nothing to show (an empty result, inputs that all gave no output so far,
Expand All @@ -1454,38 +1465,47 @@ def shown_content_hashes() -> pa.Array[Any]:
[data.content_hash().to_string() for _, data in shown],
type=pa.large_string(),
),
),
).slice(0, n),
tag_columns=tag_columns,
total_rows=total,
status=status,
origin="computed",
)

def _computed_prefix(
self, limit: int
self, limit: int, contiguous: bool = True
) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]:
"""Return the first ``limit`` output rows computed so far, and how many there are.

The rows are the contiguous run of computed inputs from the first, in the order
iteration yields them; an input whose output is ``None`` (filtered out) is
skipped, as iteration skips it. Only the first ``limit`` rows are collected.
By default the rows are the contiguous run of computed inputs from the first, in
the order iteration yields them; an input whose output is ``None`` (filtered out)
is skipped, as iteration skips it. Only the first ``limit`` rows are collected.
With ``contiguous=False`` the rows are every computed output, in input order,
also those after an input that has no output (a failure): a sorted head sorts the
rows it has.

Args:
limit: The largest number of rows to return.
contiguous: Stop at the first input with no cached output.

Returns:
The rows, and the number of output rows in the whole computed run.
The rows, and the number of output rows counted.
"""
shown: list[tuple[TagProtocol, DataProtocol]] = []
total = 0
index = 0
while index in self._cached_output_datas:
indexes = (
iter(range(len(self._cached_output_datas) + 1))
if contiguous
else iter(sorted(self._cached_output_datas))
)
for index in indexes:
if index not in self._cached_output_datas:
break
tag, data = self._cached_output_datas[index]
if data is not None:
if total < limit:
shown.append((tag, data))
total += 1
index += 1
return shown, total

def _shape_table(
Expand All @@ -1502,48 +1522,29 @@ def _shape_table(
content_hashes: Returns the content hash of each row of ``table``; called
only when ``column_config`` asks for them.
"""
drop_columns = []
if not column_config.system_tags:
# TODO: get system tags more effiicently
drop_columns.extend(
[

def shape(carried: pa.Table) -> pa.Table:
if column_config.sort_by_tags and self.keys()[0]:
# Before the system tags are dropped: they break ties between equal tags.
carried = arrow_utils.sort_by_tag_columns(carried, self.keys()[0])
drop_columns = []
if not column_config.system_tags:
drop_columns.extend(
c
for c in table.column_names
for c in carried.column_names
if c.startswith(constants.SYSTEM_TAG_PREFIX)
]
)
if not column_config.source:
drop_columns.extend(f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1])
if not column_config.context:
drop_columns.append(constants.CONTEXT_KEY)

output_table = table.drop([c for c in drop_columns if c in table.column_names])

if column_config.content_hash:
hash_column_name = (
"_content_hash"
if column_config.content_hash is True
else column_config.content_hash
)
output_table = output_table.append_column(
hash_column_name, content_hashes()
)
)
if not column_config.source:
drop_columns.extend(
f"{constants.SOURCE_PREFIX}{c}" for c in self.keys()[1]
)
if not column_config.context:
drop_columns.append(constants.CONTEXT_KEY)
return carried.drop([c for c in drop_columns if c in carried.column_names])

if column_config.sort_by_tags:
# TODO: reimplement using polars natively
output_table_schema = output_table.schema
output_table = (
pl.DataFrame(output_table)
.sort(by=self.keys()[0], descending=False)
.to_arrow()
)
output_table = arrow_utils.restore_schema_nullability(
output_table, output_table_schema
)
# output_table = output_table.sort_by(
# [(column, "ascending") for column in self.keys()[0]]
# )
return output_table
return arrow_utils.with_content_hash_column(
table, column_config.content_hash, content_hashes, shape
)


class CallableWithPodProtocol(Protocol):
Expand Down
Loading
Loading