From 70639010cab4be51bfa58613c3401f79f3739518 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:51:51 -0800 Subject: [PATCH 01/19] docs(specs): ITL-991 head() and a preview that never computes (design) Co-Authored-By: Claude Opus 5.5 --- ...-itl-991-stream-head-and-preview-design.md | 369 ++++++++++++++++++ 1 file changed, 369 insertions(+) create mode 100644 superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md diff --git a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md new file mode 100644 index 000000000..05ec1ae28 --- /dev/null +++ b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md @@ -0,0 +1,369 @@ +# ITL-991: `head()` and a notebook preview that never computes + +Status: design, awaiting developer approval (November 12, 2026). +Issue: ITL-991. Related: ITL-186 / PR #333 (Spiral source), ITL-990 (workshop notebook). + +## Rulings this design implements (developer, November 12, 2026) + +1. "Stream's preview should only show a short preview of what it can readily access but + without causing materialization unnecessary and definitely not trigger new computation." +2. "For nodes, they are associated with their storage and preview on nodes should be just a + head preview of the storage content WITHOUT triggering computation." +3. "Yes bounded 10 row read from Spiral should be considered readily accessible." +4. "Yes they should get explicit head style method and preview should just be triggering + this. For stream that's downstream of computation, head should NOT trigger the + computation but rather show the content of cached stream contents if present." + +## Today (verified on main 25650c42) + +`StreamBase._repr_html_` and `StreamBase.view()` (`core/streams/base.py:443-468`) call +`as_polars_df()` on the whole stream. So displaying: + +- a `FunctionPodStream` calls the user's function on every row; +- a `DynamicPodStream` (any operator result) runs `static_process` on the whole input; +- a `SideEffectPodStream` or `SideEffectNode` **delivers the side effects** (sends the + emails, writes the files) for every row; +- a `SpiralDBTableSource` after #333, a `PollingSource` (polls), a `CachedSource` (reads its + source and **writes** the cache table), a `DerivedSource` (reads the whole origin store); +- a blueprint `FunctionNode` / `OperatorNode` / unbound `SourceJobNode` raises inside the + formatter. + +`__repr__` never computes anywhere (checked: `TraceableBase`, `ArrowTableStream`, +`DynamicPodStream`, the nodes). `SourceNodeBase` is not a `StreamBase`, so source nodes have +no preview at all. No test or doc uses `view()`, `StreamView` or `_repr_html_`. + +## a. The method + +```python +def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, +) -> StreamHead: ... +``` + +- On `StreamBase` (every stream, source and stream-like node) and on `SourceNodeBase`. +- `columns` / `all_info` mean exactly what they mean for `as_table()`. `sort_by_tags=True` + raises `ValueError` ("head returns the first rows in stream order; sort as_table() + instead"): sorting needs every row, so it cannot be a bounded read. +- `n` is a non-negative `int`; anything else raises `ValueError`/`TypeError`. +- **Never computes**: never calls a user function, never runs an operator's + `process`/`static_process`, never delivers a side effect, never polls, never writes to a + database, never keeps what it read on the object (no hot-loading into in-memory caches). +- The base implementation is the safe default: no rows, the declared schema, status + `NOT_COMPUTED`. A stream class serves rows only by overriding `head`, so a new stream class + can never compute by accident. +- `_repr_html_` is `self.head()._repr_html_()` and nothing else: one code path. It catches an + exception from `head` and renders the schema with "could not read: " instead of + letting the notebook formatter print a traceback. + +**Return type: `StreamHead`** (recommended), a small frozen result object in +`orcapod.core.streams` (exported from `orcapod`): + +| Member | Meaning | +|---|---| +| `table: pa.Table` | at most `n` rows, exact Arrow types (logical types, widths, nullability as the stream has them) | +| `status: HeadStatus` | where the rows came from, or why there are none (below) | +| `total_rows: int \| None` | the stream's row count when known without computing | +| `total_is_estimate: bool` | `True` for a recorded estimate (Spiral manifests) | +| `tag_columns: tuple[str, ...]` | which columns of `table` are tags | +| `label`, `kind` | the stream's label and class name, for the header | +| `as_table()`, `as_polars_df()`, `__len__` | convenience, same names as the stream's | +| `_repr_html_()` | the preview (section e) | + +`HeadStatus` (a `str` enum): `ROWS` (the first rows of the stream's complete content), +`PARTIAL` (a computation stopped part way; these are the rows computed so far), +`NOT_COMPUTED`, `NOTHING_STORED` (a job node with no stored rows), `UNAVAILABLE` (no data can +exist here: a blueprint node, an unbound source slot, a `SourceProxy`). + +Why not the alternatives: + +- A plain `pa.Table` (matches `as_table`) cannot tell "empty result" from "not computed" + (point d) without a second method, and `stream.head(50)` in a notebook would print + pyarrow's text repr instead of the preview. +- A `pl.DataFrame` has the same two problems, and Polars is the convenience layer here + (`as_polars_df` is built from `as_table`), not the source of truth for types. +- A small `ArrowTableStream` would be a new root stream with its own identity; feeding it to + a pod silently builds a pipeline on 10 rows, and it cannot carry the status either. + +With `StreamHead`, `stream.head(50)` renders the same preview with 50 rows, the data is one +attribute away (`.table`, or `.as_polars_df()`), and `.status` answers point d. + +Header line (rendered by `StreamHead`, one wording table so #333 and every class agree): + +| Status / case | Header | +|---|---| +| ROWS, total known, total > n | `ArrowTableSource[readings] — first 10 of 1,234 rows` | +| ROWS, total ≤ n | `ArrowTableSource[readings] — all 7 rows` | +| ROWS, estimate | `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349` | +| ROWS, total unknown | `… — first 10 rows` | +| ROWS from a node's store | `FunctionJobNode[score] — first 10 of 1,234 stored rows` | +| ROWS from a stream's cache | `FunctionPodStream[score] — first 10 of 500 computed rows` | +| ROWS, zero rows | `DynamicPodStream[filter] — computed: no rows` | +| PARTIAL | `FunctionPodStream[score] — first 10 of 37 rows computed so far (computation incomplete)` | +| NOT_COMPUTED | `DynamicPodStream[join] — not computed; as_table(), iter_data() or a job run computes it` | +| NOT_COMPUTED (stale cache) | `… — not computed: inputs changed since the last computation` | +| NOTHING_STORED | `FunctionJobNode[score] — nothing stored; run the job` | +| UNAVAILABLE | `FunctionNode[score] — pipeline blueprint; run it in a PipelineJob` / `SourceJobNode[raw] — no source bound` / `SourceProxy[raw] — source not available` | +| side effects | adds `; displaying never delivers side effects` | + +Below the header: a one-line schema summary (`tags: id: int, session: str · data: x: float, +frame: ndarray`), then the table (tag columns first). When there are no rows the table is +the column header row only, so the schema is always visible. + +## b. Per kind of object + +"Reads" is the I/O head performs; "knows" is how the code decides without computing. +Every row-serving case returns the first `n` rows of what `as_table()` would return when +`as_table()` would not compute (this is the invariant the tests check: +`head(n).table == as_table().slice(0, n)` in every `ROWS` state). + +| Object | Readily available | head returns | Reads / cost | How it knows | +|---|---|---|---|---| +| `ArrowTableStream` | its table | ROWS, first n, exact total | slices of the held tables; source info / context / content hash built for n rows only | always holds its table | +| In-memory and file sources: `ArrowTableSource`, `DictSource`, `ListSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource`, `DBTableSource` / `SQLiteTableSource` / `PostgreSQLTableSource` (all read their table at construction today) | the held `_stream` | delegate to `_stream.head` (RootSource default) | none beyond the slice | `_stream` is built in `__init__` | +| `SpiralDBTableSource` on main (reads at construction) | held `_stream` | as above | none | as above | +| `SpiralDBTableSource` after #333 (streams) | a bounded read | ROWS, ≤ n rows read with `limit=min(n, limit)` and the source's `where`/`asof`; total = `row_count_estimate` (estimate) when no `where`, else unknown | one scan of at most n rows + a manifest count | ruling 3 | +| `PollingSource` | batches already committed | ROWS of committed rows (exact count) or NOT_COMPUTED "nothing polled yet"; **never polls** | none | `self._batches` non-empty. Without a declared schema and before a poll its `keys()` itself polls; head then shows "nothing polled yet; schema unknown" instead of calling `keys()` | +| `CachedSource` | the built merged stream if current; else its wrapped source's head | ROWS; **never writes** the cache table | slice, or the wrapped source's head | `_cached_stream is not None and not is_stale` | +| `DerivedSource` | its cached table/stream; else the origin node's stored rows | ROWS / NOTHING_STORED (origin not run) | v0.3: the origin's stored read (see "bounded node reads" below), not cached on the object | `_cached_table` / `_cached_stream`; the origin node's head | +| `SourceProxy` | nothing | UNAVAILABLE + declared schema | none | it has no delegate | +| `DynamicPodStream` (every operator result) | its cached result, if current | ROWS of the cached result, or NOT_COMPUTED (+ "inputs changed" when stale) | slice of `_cached_stream` | `_cached_stream is not None` and a side-effect-free freshness check (see finding F1) | +| `FunctionPodStream` (`FunctionPod`) | computed rows in `_cached_output_datas` / `_cached_output_table` | ROWS (iteration finished: `_cached_input_iterator is None` and cache non-empty), PARTIAL (iteration stopped: the contiguous prefix of computed indices, skipping filtered `None` outputs), or NOT_COMPUTED | converts ≤ n cached `(tag, data)` pairs to Arrow with the same typed conversion `as_table` uses (factored into one helper) | the cache dict and the iterator state; `is_stale` → NOT_COMPUTED "inputs changed" (head does not clear the cache) | +| `FunctionPodStream` of a `CachedFunctionPod` | same as above: the in-memory cache | same | same | same. Its result store is NOT consulted: looking a row up needs the input row's hash, i.e. reading and hashing the upstream. The note says "results may be in the result store; as_table() reads them without calling the function". | +| `SideEffectPodStream` | nothing (it keeps no rows) | NOT_COMPUTED, header adds "displaying never delivers side effects" | none | always | +| `FunctionNode`, `OperatorNode` (blueprints) | nothing | UNAVAILABLE "pipeline blueprint" + schema | none | the class | +| `SourceNode` (input slot) | nothing | UNAVAILABLE "input slot; bind a source" + declared schema | none | the class | +| `SourceJobNode` | the bound source | delegates to `bound_source.head(...)`; unbound → UNAVAILABLE "no source bound" | that source's head | `bound_source` | +| `OperatorJobNode` | its in-memory output; else its stored table | delegate to `_cached_output_stream.head` (which may itself be an un-run `DynamicPodStream` → NOT_COMPUTED); else stored rows (REPLAY) | stored read in REPLAY only, see below | `_cached_output_stream`; `_cache_mode`, `_pipeline_database` | +| `OperatorJobNode`, cache mode OFF | in-memory output only | ROWS after a run; else NOT_COMPUTED "uncached operator; nothing is stored" | none | `cache_mode == OFF` | +| `OperatorJobNode`, cache mode LOG | in memory; the stored table holds rows of earlier runs | in memory → ROWS; else NOTHING_STORED-style note "not run in this session; N rows stored by earlier runs (LOG mode, not replayed)" with no rows — **decision D3** | — | — | +| `FunctionJobNode` | `_cached_output_datas` after a run; else the stored rows (pipeline table ⋈ result table) | ROWS (in memory, or "stored rows"), NOTHING_STORED | in memory: ≤ n conversions. Stored: see "bounded node reads" | dict non-empty; `_fetch_joined_records` non-empty | +| `SideEffectNode` / `SideEffectJobNode` | nothing it can show without delivering | NOT_COMPUTED, "displaying never delivers side effects" | none | the class | +| Loaded job nodes (`PipelineJob.load`) | the store only | function: stored rows / NOTHING_STORED; operator: REPLAY stored rows, else UNAVAILABLE "not stored (cache mode OFF)"; source: bound source or `SourceProxy` UNAVAILABLE. Never raises the loaded-node errors | stored read | `_function_pod is None`, `_loaded` | + +### Bounded node reads + +`ArrowDatabaseProtocol` has no limited read: `get_all_records` reads the whole table, and a +node's rows are filtered from a shared table (`__data_context`, node content hash), and for +a function node joined to the result table and de-duplicated by recomputation index (the +newest record of a base entry may lie anywhere in the table). + +- **v0.3 (recommended):** a node with nothing in memory reads its stored rows with its + existing read helpers (`_read_stored_records`, `_fetch_joined_records` / + `_stored_rows_stream`), returns the first n, and keeps nothing. No computation, exact + total ("first 10 of 1,234 stored rows"), but the I/O is the node's whole stored slice. + After `job.run()` in the same process (the workshop's path) the rows are in memory and + nothing is read at all. +- **Follow-up (new issue):** `ArrowDatabaseProtocol.get_first_records(record_path, n, *, + column_values=None, record_id_column=None)` implemented by every database class (Delta: + `dataset.head(n, filter=...)` plus pending rows; in-memory: filter + slice; connector: + `WHERE … LIMIT n`; no-op; extension-aware wrapper), used by operator and source caches + directly and by function nodes for the pipeline rows plus `get_records_by_ids` for their + results. The recomputation-index de-duplication makes the function-node case need care + (read rows for the chosen base entries only). About one more day; public protocol change. + +## c. Operators + +Confirmed: head never runs an operator, even when its inputs are readily available, because +the operator of the heads is not the head of the operator: + +- `GroupBy(by=["subject"])` over the first 10 rows of a source gives **wrong values**, not + just fewer rows: a group's list holds only the members among those 10 rows. +- `Join` / `SemiJoin` of two heads can be empty while the full join is not (the matching + keys are beyond row 10 of one side). +- `PolarsFilter` / `DropNulls` of a head can be empty while the full result is not. +- `MergeJoin` lists and `Batch(drop_partial_batch=True)` change content too. + +So an operator stream shows its schema and NOT_COMPUTED until it is materialized, then its +rows. Which stream classes keep their materialized rows today: + +| Class | Keeps rows | Where | +|---|---|---| +| `ArrowTableStream` | always | `_table` | +| `DynamicPodStream` | yes, after `as_table`/`iter_data`/`run` | `_cached_stream` (the `ArrowTableStream` from `static_process`) | +| `FunctionPodStream` | yes, incrementally during iteration; the table after `as_table` | `_cached_output_datas`, `_cached_output_table` | +| `SideEffectPodStream`, `SideEffectNode` | no | — | +| `CachedSource` | yes, after first use | `_cached_stream` | +| `DerivedSource` | yes (tags_as_data: only once the origin has rows) | `_cached_table` / `_cached_stream` | +| `PollingSource` | committed batches | `_batches` | +| `OperatorJobNode` | after `run`/`execute` (lazy `DynamicPodStream` after `run` with OFF and no store) | `_cached_output_stream` | +| `FunctionJobNode` | after `run`/`execute`/first `iter_data` | `_cached_output_datas` | + +## d. Nothing available + +`head` returns a zero-row table with the full schema and a status that is not `ROWS`: + +- Classes that hold an Arrow schema (sources, `ArrowTableStream`, stored tables) slice it to + zero rows: exact types. +- Uncomputed operator / function streams know only their Python schema (`output_schema()` + exposes Python types). Their empty table is + `arrow_utils.make_empty_table({**tag_schema, **data_schema}, type_converter)`: correct + names, nullability and logical types, but widths are the converter's defaults (`int` → + `int64`) and may differ from the eventual computed column (an `int8` input passed through + an operator). This is documented on `head`; the status tells the caller the table is a + schema, not a result. +- Empty result vs not computed: `status == ROWS and len(table) == 0` is an empty result; + `NOT_COMPUTED` / `NOTHING_STORED` / `UNAVAILABLE` are not. + +## e. Large cells (preview only) + +`StreamHead._repr_html_` renders a display copy; `StreamHead.table` is never changed. + +| Arrow type of the column | Shown as | +|---|---| +| `orcapod.ndarray` extension (and any extension whose storage is a struct with `dtype` and `shape`) | `ndarray float64 (2, 3)`, read from the `dtype`/`shape` fields without decoding `data` | +| other logical (extension) types | Polars' own rendering if the storage is a string or number (`Path` shows the path); otherwise `` | +| `large_binary` / `binary` | `bytes (300 B)`, sizes humanised (`1.2 MB`) | +| `large_string` longer than 80 characters | first 77 characters + `…` (length in a tooltip) | +| `large_list` | `[1, 2, 3]` when ≤ 5 scalar items, else `list (len 12)`; nested lists `list (len 3) of list` | +| `struct` | Polars' rendering, with these rules applied to its fields | +| null | Polars' `null` | + +Tag columns come first and are marked in the header row (a `tag` group label above them). +The renderer builds a Polars frame of display strings for the summarised columns and uses +Polars' HTML for the table, so the look stays what users see today. + +## f. `view()` and `__repr__` + +- **Remove `view()` and `StreamView`.** `view()` is today's "whole stream as a frame" and + would either compute (forbidden by the ruling) or duplicate `head`. Nothing in the repo + uses it; greenfield rule, no shim. `as_polars_df()` remains the way to get everything. +- **`__repr__`**: unchanged; it already never computes. `str(stream.head())` gives a text + preview (header + Polars' text table) for terminal use. + +## g. Tests (public interface; failing first) + +New file `tests/test_core/streams/test_head_and_preview.py`, plus node cases in +`tests/test_core/nodes/`: + +1. Function pod: a counting function. `stream.head()` and `stream._repr_html_()` before + compute → count 0, `NOT_COMPUTED`, the schema's columns in the empty table. After three + `next()` on `iter_data()` → `PARTIAL`, 3 rows, count still 3. After `as_table()` → `ROWS`, + `head(n).table == as_table().slice(0, n)`, count unchanged by head. After the upstream is + modified → `NOT_COMPUTED` (stale). +2. Same with a concurrent executor; and a `CachedFunctionPod` stream whose result store is + full → still `NOT_COMPUTED` and the function is not called. +3. Operators (`Join`, `GroupBy`, `PolarsFilter`): `unittest.mock.patch.object(, + "static_process", wraps=...)` as a spy (non-public, commented: the test is about it not + running) → zero calls from `head`/`_repr_html_`; after `as_table()` → rows equal to the + table's prefix; empty filter result → `ROWS` with 0 rows, not `NOT_COMPUTED`. +4. Side effects: a `SideEffectPodStream` and a job's side-effect node → displaying delivers + nothing (counter 0). +5. Sources: every in-memory source → `ROWS`, exact total, prefix invariant; `PollingSource` + → no poll (counter on the impl); `CachedSource` → no table written (store unchanged); + `SourceProxy` → `UNAVAILABLE`. Spiral's bounded read is tested in #333 (fake pyspiral + records the `limit` it was given) once the two meet. +6. Nodes, in-memory store and Delta store, after `job.run()` sync and async: function, + operator (OFF, LOG, REPLAY) and source nodes → expected status and prefix rows; the + function counter unchanged. Before a run → `NOTHING_STORED` / `NOT_COMPUTED`. Loaded + job (`PipelineJob.load`) in a fresh object → stored rows, no raise; blueprint nodes → + `UNAVAILABLE`. With the follow-up bounded read, a spy database asserts `get_first_records` + is called with `n` and `get_all_records` is not. +7. Preview: header wording table above; ndarray / bytes / long string / list summaries; + `head().table` still holds the full values; `head(n=-1)` and `sort_by_tags=True` raise. +8. Golden hash tests untouched and passing. + +## h. What changes for users + +- Displaying a stream, source or node in a notebook no longer computes: an operator result + or function pod output that has not been computed shows its schema and "not computed" + until `as_table()`, `iter_data()` or a job run computes it. Displaying a side-effect + stream no longer delivers its side effects. Displaying a `CachedSource` no longer writes + its cache. +- New `head(n=10)` on every stream, source and node, returning a `StreamHead`. +- `view()` and `StreamView` are removed. +- No hash, golden value, stored table, path or stored schema moves: head only reads, and no + `identity_structure` changes. + +Changelog fragment `changelog.d/ITL-991.md`: + +```markdown +### Breaking changes and migration + +- Displaying a stream, source or node in a notebook no longer computes anything. The preview + shows the first 10 rows that are already available (held in memory, cached from an + earlier computation, stored by a job run, or a bounded read of a source's table) and + otherwise the schema with a note such as "not computed". Call `as_table()` or run the job + to compute. Displaying a side-effect stream no longer delivers its side effects. +- `StreamBase.view()` and `StreamView` are removed: use `head(n)` or `as_polars_df()`. + +### Added + +- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a + `StreamHead`: at most `n` rows as an Arrow table with their exact types, the row count + when it is known without computing, and a `status` that tells an empty result from "not + computed". It never calls a function, runs an operator or polls a source. +``` + +## i. Size, commits, schedule + +About 600–800 lines of code and 400–600 of tests. + +1. `test:` failing tests for streams and sources (g1–g5, g7). +2. `feat(streams):` `StreamHead`, `HeadStatus`, `StreamBase.head` default, renderer with cell + summaries, `_repr_html_` → `head()`; remove `view`/`StreamView`. +3. `feat(sources):` `ArrowTableStream.head`, RootSource delegation, Polling / Cached / + Derived / Proxy. +4. `feat(streams):` `DynamicPodStream.head`, `FunctionPodStream.head` (shared typed + conversion helper), side-effect streams. +5. `test:` + `feat(nodes):` node heads (g6), `SourceNodeBase.head`. +6. `docs:` "Looking at data" section (preview, `head`, `as_table`, `as_polars_df`, + `as_pandas_df`); changelog fragment; design-issue file. + +Commits 1–6 with the v0.3 node read (full stored slice): Nov 13–14, PR and CodeRabbit by +Nov 15. The bounded database read (`get_first_records`) as a separate follow-up after the +workshop unless the developer wants it in v0.3 (then +1 day, still before Nov 16 but with +no slack). + +## #333 interplay (whichever merges second makes the mechanical change) + +#333 adds `StreamBase._preview(columns, all_info) -> (pl.DataFrame, note)` (default: +today's behaviour) and a Spiral override that reads ≤ 10 rows with Spiral's `limit` and +renders `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349`. + +- `StreamBase._preview` is removed; `head` is the override point. +- The Spiral override becomes: + + ```python + def head(self, n=10, *, columns=None, all_info=False) -> StreamHead: + config = head_column_config(columns, all_info) # shared helper; rejects sort_by_tags + limit = n if self._limit is None else min(n, self._limit) + rows = -> pa.Table + return StreamHead.of_rows( + self, rows, config, + total_rows=None if self._where else self.row_count_estimate, + total_is_estimate=True, + ) + ``` + + i.e. the same read, returning the Arrow table instead of `pl.DataFrame(...)`, and the note + string replaced by `total_rows` + `total_is_estimate`. The header wording above produces + #333's exact text, so #333's rendering assertions keep passing. +- If #333 merges first: this PR deletes `_preview` from `StreamBase` and rewrites the Spiral + override as above. If this PR merges first: #333 drops its `StreamBase._preview` and + implements `head` as above. Suggest the coordinator forwards this section to #333's worker. + +## Findings (out of scope unless decided otherwise) + +- **F1 `DynamicPodStream` never invalidates its cache.** `_update_cache_status` + (`core/operators/static_output_pod.py:383,392`) sets `self._cached_results = None` (a + misspelling; the attribute is `_cached_stream`) and then moves `_cached_time` up to the + upstream time, so after an upstream changes, `as_table()` returns the old result, and any + read of `last_modified` makes it look fresh. head needs a correct freshness check. + Proposal: fix it here (one line plus a regression test), since head depends on it. +- **F2** Displaying a `SideEffectPodStream` / `SideEffectNode` today delivers every side + effect (fixed by this design). +- **F3** `PollingSource.keys()` polls when no schema was declared and nothing was polled. + +## Decisions requested + +- **D1** Return type `StreamHead` (recommended) vs `pa.Table` vs `pl.DataFrame`. +- **D2** Node stored reads for v0.3: whole stored slice, no computation (recommended), with + bounded `get_first_records` as a follow-up; or bounded in v0.3 (+1 day). +- **D3** `OperatorJobNode` in LOG mode with nothing in memory: no rows, note that rows of + earlier runs are stored but not replayed (recommended, matches `as_table`), or show them. +- **D4** Remove `view()` / `StreamView` (recommended) or keep `view()` as an alias of `head`. +- **D5** Fix F1 in this PR (recommended) or file it separately. From fc3b11cfdd9341123da95840ee94a0dcb763d1ab Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 14:59:04 -0800 Subject: [PATCH 02/19] fix(operators): an operator stream recomputes after an upstream changes (ITL-991) DynamicPodStream._update_cache_status cleared a misspelled attribute (_cached_results) instead of _cached_stream, so a result kept before an upstream changed was served again without running the operator. Co-Authored-By: Claude Opus 5.5 --- .../core/operators/static_output_pod.py | 4 +- .../plans/2026-11-12-itl-991-stream-head.md | 178 ++++++++++++++++++ .../test_dynamic_pod_stream_staleness.py | 50 +++++ 3 files changed, 230 insertions(+), 2 deletions(-) create mode 100644 superpowers/plans/2026-11-12-itl-991-stream-head.md create mode 100644 tests/test_core/streams/test_dynamic_pod_stream_staleness.py diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 5ad11c5ad..9ff23d1ea 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -380,7 +380,7 @@ def _update_cache_status(self) -> None: upstream_times.append(self._pod.last_modified) if any(t is None for t in upstream_times): - self._cached_results = None + self._cached_stream = None self._cached_time = None return @@ -389,7 +389,7 @@ def _update_cache_status(self) -> None: # Invalidate cache if upstream is newer and update the cache time if max_upstream_time > self._cached_time: - self._cached_results = None + self._cached_stream = None self._cached_time = max_upstream_time def run(self, *args: Any, **kwargs: Any) -> None: diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md new file mode 100644 index 000000000..c2f485781 --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -0,0 +1,178 @@ +# ITL-991 `head()` and a preview that never computes: implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this +> plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** every stream, source and node gets `head(n=10, *, columns, all_info) -> StreamHead`, +which never computes, and `_repr_html_` renders `self.head()`. + +**Architecture:** one new module `core/streams/head.py` holds `HeadStatus`, `StreamHead` +(result + renderer) and the shared helpers. `StreamBase.head` is the safe default (schema +only, `NOT_COMPUTED`); each class that can serve rows without computing overrides `head`. +Spec: `superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md`. + +**Tech stack:** PyArrow (the head table), Polars (HTML rendering only). + +## Status of decisions (coordinator, November 12, 2026) + +- D3 approved: OperatorJobNode in LOG mode with nothing in memory shows no rows + a note. +- D5 approved: fix F1 (DynamicPodStream `_cached_results` misspelling) in this PR, its own + commit, with a regression test and its own `### Fixed` changelog bullet. +- D1 (StreamHead), D2 (whole stored slice for nodes; bounded read is a follow-up), D4 + (remove `view()`/`StreamView`) are with the developer. Work proceeds on the + recommendations, each behind ONE seam: + - D1 seam: `StreamBase.head` returns `StreamHead`; a `pa.Table` answer would mean + `head()` returns `self._head_result(...).table` and `_repr_html_` calls `_head_result`. + - D2 seam: every node's stored read goes through one method per node class + (`_head_from_store`), so a bounded read replaces only that method. + - D4 seam: the removal of `view`/`StreamView` is one commit. +- **Do not open the PR until the coordinator confirms D1, D2, D4.** +- Coordinator requirements: (1) a node header that reads its whole stored slice must not + claim a bounded read, and a test shows the read performs no computation; (2) the workshop + path (a source preview; a job node right after `job.run()` in the same process) is cheap: + a slice of what is in memory. + +## Files + +- Create `src/orcapod/core/streams/head.py`: `HeadStatus`, `StreamHead`, `head_column_config`, + cell summaries and HTML/text rendering. +- Modify `src/orcapod/core/streams/base.py`: `head` default, `_repr_html_`, remove + `view`/`StreamView`, `_schema_only_table`. +- Modify `src/orcapod/core/streams/__init__.py`, `src/orcapod/__init__.py`: export + `StreamHead`, `HeadStatus`. +- Modify `src/orcapod/core/streams/arrow_table_stream.py`: `head`. +- Modify `src/orcapod/core/sources/base.py` (RootSource delegation), `cached_source.py`, + `derived_source.py`, `polling_source.py`, `source_proxy.py`. +- Modify `src/orcapod/core/operators/static_output_pod.py`: F1 fix; `DynamicPodStream.head`. +- Modify `src/orcapod/core/function_pod.py`: `FunctionPodStream.head` (+ factor the table + build of `as_table` into helpers shared with head). +- Modify `src/orcapod/side_effects.py`: side-effect note. +- Modify `src/orcapod/core/nodes/{function_node,operator_node,source_node}.py`. +- Tests: `tests/test_core/streams/test_head_and_preview.py` (streams, sources, operators, + function pods, rendering), `tests/test_core/streams/test_dynamic_pod_stream_staleness.py` + (F1), `tests/test_pipeline/test_node_head.py` (job nodes, sync/async, in-memory/Delta, + loaded job). +- Docs: `docs/concepts/` "Looking at data" section; `changelog.d/ITL-991.md`; + `design_issues/ITL-991-PV1.md`. + +## StreamHead contract (Task 1) + +```python +class HeadStatus(StrEnum): + ROWS = "rows" # the first rows of the stream's complete content + PARTIAL = "partial" # computation stopped part way; rows computed so far + NOT_COMPUTED = "not_computed" + NOTHING_STORED = "nothing_stored" + UNAVAILABLE = "unavailable" + +@dataclass(frozen=True) +class StreamHead: + table: pa.Table + status: HeadStatus + kind: str # class name of the stream shown + label: str + tag_columns: tuple[str, ...] + total_rows: int | None = None + total_is_estimate: bool = False + origin: str = "held" # "held" | "computed" | "stored" | "read" + detail: str | None = None # extra note ("inputs changed ...", side effects) + def as_table(self) -> pa.Table + def as_polars_df(self) -> pl.DataFrame + def __len__(self) -> int + def header(self) -> str # the wording table of the spec + def for_stream(self, stream) -> StreamHead # same rows, shown as `stream` + def _repr_html_(self) -> str + def __str__(self) -> str +``` + +Header wording (exact; #333 depends on the estimate line): + +| case | text after `Kind[label] — ` | +|---|---| +| ROWS, rows == total | `all {k} rows` (`all 1 row`) | +| ROWS, total known | `first {k} of {total:,} rows` / `... stored rows` / `... computed rows` | +| ROWS, estimate | `first {k} rows of about {total:,}` | +| ROWS, total unknown | `first {k} rows` | +| ROWS, 0 rows | `no rows` (`computed: no rows` for origin computed) | +| PARTIAL | `first {k} of {total:,} rows computed so far (computation incomplete)` | +| NOT_COMPUTED | `not computed; as_table(), iter_data() or a job run computes it` | +| NOTHING_STORED | `nothing stored; run the job` | +| UNAVAILABLE | detail only | +| detail present | appended after `; ` (or the whole note for UNAVAILABLE / a NOT_COMPUTED with a reason) | + +## Tasks + +### Task 1: F1 regression test and fix (own commit) +- [ ] Test `tests/test_core/streams/test_dynamic_pod_stream_staleness.py`: an operator over a + `FunctionPodStream`; `as_table()`; `fn_stream.clear_cache()`; `as_table()` again calls + `static_process` twice (spy with `mock.patch.object(SelectDataColumns, "static_process", + autospec=True, side_effect=original)`; comment: the test is about re-running it). +- [ ] Run: fails (1 call). +- [ ] Fix `static_output_pod.py:383,392`: `self._cached_results = None` → `self._cached_stream = None`. +- [ ] Run test + `tests/test_core/operators` → pass. +- [ ] Commit `fix(operators): an operator stream recomputes after an upstream changes (ITL-991)`. + +### Task 2: failing tests for streams, sources, rendering +- [ ] Write `tests/test_core/streams/test_head_and_preview.py` (cases in spec g1–g5, g7). +- [ ] Run: fails (no `head`). Commit `test(streams): head() and the preview never compute (ITL-991)`. + +### Task 3: `head.py`, `StreamBase.head`, `_repr_html_`, remove `view`/`StreamView` +- [ ] Implement `head.py` per the contract; `StreamBase.head` default + (`NOT_COMPUTED`, `_schema_only_table(config)` = `make_empty_table` of + `output_schema(columns=config)`; any exception → zero-column table). +- [ ] `_repr_html_`: `try: return self.head()._repr_html_() except Exception as e:` header + + `could not read: {e}`. +- [ ] Remove `view`, `StreamView` (D4 seam: separate commit). + +### Task 4: ArrowTableStream and sources +- [ ] `ArrowTableStream.head`: `as_table(columns=config without content_hash).slice(0, n)`; + content hashes for `islice(iter_data(), n)` only; total = `num_rows`. +- [ ] `RootSource.head`: `self._stream.head(...).for_stream(self)`. +- [ ] CachedSource: current `_cached_stream` → its head; else `_source.head` with detail + `rows of the wrapped source; the cache is not read until first use`; never `_ensure_stream`. +- [ ] DerivedSource: factor `_peek_stream()` (no assignment) out of `_get_stream`; head uses + the cached stream/table if any, else `_peek_stream()`; origin `stored`; no origin rows → + NOTHING_STORED. +- [ ] PollingSource: `_batches` → combined stream head; else NOT_COMPUTED `nothing polled yet` + with the declared schema, or a zero-column table when none (never `keys()`). +- [ ] SourceProxy: UNAVAILABLE `source not available; bind a live source`. + +### Task 5: operator and function streams +- [ ] `DynamicPodStream.head`: `had = self._cached_stream is not None`; + `self._update_cache_status()`; cached → `self._cached_stream.head(...)` with origin + `computed`; `had` and now gone → NOT_COMPUTED `inputs changed since the last computation`. +- [ ] `FunctionPodStream`: factor `_table_from_pairs(pairs)` and `_shape_table(table, config, + hashes)` out of `as_table`; head: stale with cache → NOT_COMPUTED "inputs changed"; + `_cached_output_table` → shape + slice; else contiguous prefix of `_cached_output_datas` + (skip `None` outputs) → ROWS (iteration finished) or PARTIAL; empty → NOT_COMPUTED. +- [ ] Side-effect streams/nodes: base default with detail `displaying never delivers side effects`. + +### Task 6: nodes (`tests/test_pipeline/test_node_head.py` first) +- [ ] Blueprints `FunctionNode`, `OperatorNode`: UNAVAILABLE `pipeline blueprint; run it in a PipelineJob`. +- [ ] `SourceNodeBase.head` + `_repr_html_`; `SourceNode` UNAVAILABLE `input slot; bind a + source`; `SourceJobNode` → bound source head, else UNAVAILABLE `no source bound`. +- [ ] `OperatorJobNode.head`: in-memory → inner head; REPLAY + store → `_head_from_store` + (`_read_stored_records`, internal columns dropped, ArrowTableStream head, origin stored); + none → NOTHING_STORED; LOG → NOT_COMPUTED `not run in this session; rows of earlier runs + are stored but not replayed (cache mode LOG)`; OFF → NOT_COMPUTED `not run; cache mode OFF + stores nothing`. Never raises the loaded-node errors. +- [ ] `FunctionJobNode.head`: UNAVAILABLE load status → UNAVAILABLE; `_cached_output_table` + → slice; `_cached_output_datas` → first n via the node's conversion; else + `_head_from_store` (loaded: `_stored_rows_stream(_fetch_joined_records())[1].head`; + live: `_load_cached_entries()` values, not kept) → ROWS origin stored; empty → NOTHING_STORED. +- [ ] Tests: sync and async orchestrators × in-memory and Delta; function counter unchanged + by head; before run NOTHING_STORED; loaded job; OFF/LOG/REPLAY operator nodes. + +### Task 7: docs, changelog, design issue, full suite +- [ ] "Looking at data" section in `docs/concepts/streams.md` (or the closest page), executed + samples with `# ...` output comments. +- [ ] `changelog.d/ITL-991.md` (spec h text + F1 `### Fixed` bullet); + `uv run python scripts/assemble_changelog.py --check`. +- [ ] `design_issues/ITL-991-PV1.md` (resolved); `uv run python scripts/design_issues.py --check`. +- [ ] ruff check/format, basedpyright (exit code), `uv run pytest tests/`. +- [ ] Merge origin/main, rerun, push; wait for D1/D2/D4 confirmation before the PR. + +## Progress notes (for a successor) + +- 2026-11-12: spec 70639010 pushed; design ask answered (D3, D5 approved; D1/D2/D4 pending + with developer). Plan saved. diff --git a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py new file mode 100644 index 000000000..b4f1d97d9 --- /dev/null +++ b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py @@ -0,0 +1,50 @@ +"""An operator stream recomputes after an upstream changes (ITL-991, finding F1). + +``DynamicPodStream`` keeps the result of its operator. When an upstream is modified after +that, the kept result is stale and the next read must run the operator again. +""" + +from __future__ import annotations + +from unittest import mock + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource + + +@function_pod(output_keys="y") +def double(x: int) -> int: + """Double ``x``.""" + return 2 * x + + +def _source() -> ArrowTableSource: + table = pa.table( + {"id": [1, 2, 3], "x": [1, 2, 3]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="s") + + +def test_operator_stream_reruns_after_upstream_is_modified() -> None: + fn_stream = double.pod(_source()) + out = SelectDataColumns(["y"])(fn_stream) + original = SelectDataColumns.static_process + # A spy on the non-public static_process: the test is about whether it runs again. + with mock.patch.object( + SelectDataColumns, "static_process", autospec=True, side_effect=original + ) as spy: + first = out.as_table() + assert spy.call_count == 1 + fn_stream.clear_cache() # the upstream is modified after the result was kept + second = out.as_table() + assert spy.call_count == 2 + assert first.equals(second) From cf25515b7812f288872952089ce491dcea3e56aa Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:00:26 -0800 Subject: [PATCH 03/19] test(streams): head() and the preview never compute (ITL-991) Co-Authored-By: Claude Opus 5.5 --- .../streams/test_head_and_preview.py | 390 ++++++++++++++++++ 1 file changed, 390 insertions(+) create mode 100644 tests/test_core/streams/test_head_and_preview.py diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py new file mode 100644 index 000000000..4505a777b --- /dev/null +++ b/tests/test_core/streams/test_head_and_preview.py @@ -0,0 +1,390 @@ +"""``head()`` and the notebook preview show what is readily available and never compute (ITL-991). + +A stream's ``head(n)`` returns at most ``n`` rows it already holds, has cached from an earlier +computation, or can read from its table, and the preview (``_repr_html_``) renders exactly +that. Neither calls a function, runs an operator, delivers a side effect, polls a source or +writes a cache. +""" + +from __future__ import annotations + +from typing import Any +from unittest import mock + +import numpy as np +import pyarrow as pa +import pytest + +from orcapod import function_pod, side_effect_pod +from orcapod.core.operators import GroupBy, Join, PolarsFilter, SelectDataColumns +from orcapod.core.sources import ArrowTableSource, CachedSource, DictSource, ListSource +from orcapod.core.sources.polling_source import PollingSource +from orcapod.databases import InMemoryArrowDatabase +from orcapod.streams import HeadStatus, StreamHead +from orcapod.types import ColumnConfig, Cursor + +ALL_COLUMNS = ColumnConfig( + meta=True, context=True, source=True, system_tags=True, content_hash=True +) + + +def _table(rows: int, offset: int = 0) -> pa.Table: + """``id`` and ``x``, both non-nullable ``int64``.""" + return pa.table( + { + "id": list(range(offset, offset + rows)), + "x": [10 * i for i in range(offset, offset + rows)], + }, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + + +def _source(rows: int = 25, label: str = "readings") -> ArrowTableSource: + return ArrowTableSource( + _table(rows), tag_columns=["id"], source_id="readings", label=label + ) + + +# --------------------------------------------------------------------------- +# Sources and in-memory streams +# --------------------------------------------------------------------------- + + +def test_source_head_is_the_first_rows_with_the_exact_total() -> None: + source = _source() + head = source.head() + assert isinstance(head, StreamHead) + assert head.status is HeadStatus.ROWS + assert len(head) == 10 + assert head.total_rows == 25 + assert head.table.equals(source.as_table().slice(0, 10)) + assert head.header() == "ArrowTableSource[readings] — first 10 of 25 rows" + + +def test_head_of_a_short_source_says_all_rows() -> None: + head = _source(rows=3).head() + assert len(head) == 3 + assert head.header() == "ArrowTableSource[readings] — all 3 rows" + + +def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: + source = _source() + head = source.head(4, columns=ALL_COLUMNS) + assert head.table.equals(source.as_table(columns=ALL_COLUMNS).slice(0, 4)) + # all_info means every column group; head returns rows in stream order. + assert source.head(4, all_info=True).table.equals( + source.as_table(columns=ALL_COLUMNS).slice(0, 4) + ) + + +def test_head_refuses_negative_n_and_sorting() -> None: + source = _source() + with pytest.raises(ValueError, match="n"): + source.head(-1) + with pytest.raises(ValueError, match="sort"): + source.head(columns={"sort_by_tags": True}) + + +def test_head_of_zero_rows_keeps_the_columns() -> None: + head = _source().head(0) + assert head.status is HeadStatus.ROWS + assert head.table.column_names == ["id", "x"] + assert head.table.num_rows == 0 + + +@pytest.mark.parametrize( + "make", + [ + lambda: DictSource( + [{"id": i, "x": i} for i in range(12)], tag_columns=["id"], label="d" + ), + lambda: ListSource( + name="v", data=list(range(12)), tag_function=lambda e, i: {"i": i} + ), + ], + ids=["dict", "list"], +) +def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: + source = make() + head = source.head(5) + assert head.status is HeadStatus.ROWS + assert head.total_rows == 12 + assert head.table.equals(source.as_table().slice(0, 5)) + + +def test_preview_is_the_rendered_head() -> None: + source = _source() + html = source._repr_html_() + assert "ArrowTableSource[readings] — first 10 of 25 rows" in html + assert html == source.head()._repr_html_() + + +# --------------------------------------------------------------------------- +# Function pods +# --------------------------------------------------------------------------- + + +def _counting_pod(calls: list[int]) -> Any: + @function_pod(output_keys="y") + def triple(x: int) -> int: + calls.append(x) + return 3 * x + + return triple + + +def test_function_pod_stream_is_not_computed_by_head_or_preview() -> None: + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source(), label="triple") + head = stream.head() + html = stream._repr_html_() + assert calls == [] + assert head.status is HeadStatus.NOT_COMPUTED + assert len(head) == 0 + assert head.table.column_names == ["id", "y"] + assert "not computed" in html + assert "FunctionPodStream[triple]" in html + + +def test_function_pod_stream_shows_rows_computed_so_far() -> None: + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + rows = stream.iter_data() + for _ in range(3): + next(rows) + head = stream.head() + assert calls == [0, 10, 20] + assert head.status is HeadStatus.PARTIAL + assert head.table.column("y").to_pylist() == [0, 30, 60] + assert "computed so far" in head.header() + + +def test_function_pod_stream_shows_its_computed_rows() -> None: + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + table = stream.as_table() + assert len(calls) == 25 + head = stream.head(7) + assert len(calls) == 25 + assert head.status is HeadStatus.ROWS + assert head.total_rows == 25 + assert head.table.equals(table.slice(0, 7)) + assert head.header().endswith("first 7 of 25 computed rows") + + +def test_function_pod_stream_rows_from_iteration_only() -> None: + calls: list[int] = [] + stream = _counting_pod(calls).pod(_source()) + _ = list(stream.iter_data()) + head = stream.head(4, columns=ALL_COLUMNS) + assert len(calls) == 25 + assert head.status is HeadStatus.ROWS + assert head.table.equals(stream.as_table(columns=ALL_COLUMNS).slice(0, 4)) + assert len(calls) == 25 + + +# --------------------------------------------------------------------------- +# Operators +# --------------------------------------------------------------------------- + + +def _other_source() -> ArrowTableSource: + table = pa.table( + {"id": list(range(25)), "z": [i * i for i in range(25)]}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("z", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="other") + + +def _grouped_source() -> ArrowTableSource: + table = pa.table( + { + "id": list(range(25)), + "grp": [f"g{i % 3}" for i in range(25)], + "x": list(range(25)), + }, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("grp", pa.large_string(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id", "grp"], source_id="grouped") + + +OPERATOR_CASES = { + "join": (Join, lambda: Join()(_source(), _other_source())), + "group_by": (GroupBy, lambda: GroupBy(by=["grp"])(_grouped_source())), + "filter": ( + PolarsFilter, + lambda: PolarsFilter(constraints={"x": 20})(_source()), + ), +} + + +@pytest.mark.parametrize("case", list(OPERATOR_CASES), ids=list(OPERATOR_CASES)) +def test_operator_stream_is_not_run_by_head_or_preview(case: str) -> None: + operator_class, build = OPERATOR_CASES[case] + stream = build() + original = operator_class.static_process + # A spy on the non-public static_process: the test is about whether it runs. + with mock.patch.object( + operator_class, "static_process", autospec=True, side_effect=original + ) as spy: + head = stream.head() + _ = stream._repr_html_() + assert spy.call_count == 0 + assert head.status is HeadStatus.NOT_COMPUTED + assert len(head) == 0 + table = stream.as_table() + assert spy.call_count == 1 + after = stream.head(2) + assert spy.call_count == 1 + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 2)) + + +def test_an_empty_operator_result_is_not_not_computed() -> None: + stream = PolarsFilter(constraints={"x": -1})(_source()) + _ = stream.as_table() + head = stream.head() + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert head.header().endswith("computed: no rows") + + +def test_operator_stream_with_changed_inputs_is_not_computed() -> None: + calls: list[int] = [] + fn_stream = _counting_pod(calls).pod(_source()) + out = SelectDataColumns(["y"])(fn_stream) + _ = out.as_table() + fn_stream.clear_cache() + head = out.head() + assert head.status is HeadStatus.NOT_COMPUTED + assert "inputs changed" in head.header() + + +# --------------------------------------------------------------------------- +# Side effects, polling and caching sources +# --------------------------------------------------------------------------- + + +def test_side_effect_stream_preview_never_delivers() -> None: + delivered: list[int] = [] + + @side_effect_pod + def deliver(x: int) -> None: + delivered.append(x) + + stream = deliver(_source()) + head = stream.head() + html = stream._repr_html_() + assert delivered == [] + assert head.status is HeadStatus.NOT_COMPUTED + assert "displaying never delivers side effects" in html + + +class _FakeDynamicSource: + """A ``DynamicSourceProtocol`` double that records every poll and fetch.""" + + def __init__(self) -> None: + self.polls = 0 + self.fetches = 0 + + def identity(self) -> Any: + return "fake" + + def to_config(self) -> dict[str, Any] | None: + return None + + @classmethod + def from_config(cls, config: dict[str, Any]) -> _FakeDynamicSource: + raise NotImplementedError + + def schema(self) -> Any: + return None + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + self.polls += 1 + return False + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + self.fetches += 1 + return Cursor(value=1), { + "id": pa.array([1], type=pa.int64()), + "val": pa.array([7], type=pa.int64()), + } + + async def close(self) -> None: + pass + + +def test_polling_source_head_never_polls() -> None: + fake = _FakeDynamicSource() + source = PollingSource(fake, tag_columns="id") + head = source.head() + _ = source._repr_html_() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert "nothing polled yet" in head.header() + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + after = source.head() + assert (fake.polls, fake.fetches) == seen + assert after.status is HeadStatus.ROWS + assert after.table.column("val").to_pylist() == [7] + + +def test_cached_source_head_writes_nothing() -> None: + store = InMemoryArrowDatabase() + source = CachedSource(_source(), cache_database=store) + head = source.head(3) + _ = source._repr_html_() + assert store.list_table_paths() == [] + assert head.table.column("x").to_pylist() == [0, 10, 20] + + +# --------------------------------------------------------------------------- +# Rendering of large cells +# --------------------------------------------------------------------------- + + +def test_preview_summarises_large_cells_and_head_keeps_them() -> None: + source = DictSource( + [ + { + "id": 1, + "arr": np.arange(6.0).reshape(2, 3), + "blob": b"xyz" * 100, + "text": "a" * 200, + "items": list(range(12)), + } + ], + tag_columns=["id"], + label="cells", + ) + head = source.head() + html = head._repr_html_() + assert "ndarray float64 (2, 3)" in html + assert "bytes (300 B)" in html + assert "a" * 200 not in html + assert "…" in html + assert "list (len 12)" in html + assert head.table.equals(source.as_table()) + + +def test_text_form_has_the_header() -> None: + text = str(_source().head(2)) + assert text.startswith("ArrowTableSource[readings] — first 2 of 25 rows") From 0b30e5fe1302d4153bca199fb57358b0fa9b3a06 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:03:52 -0800 Subject: [PATCH 04/19] feat(streams): head() and a preview that never computes, for streams and sources (ITL-991) StreamHead and HeadStatus; StreamBase.head shows the schema and NOT_COMPUTED by default; _repr_html_ renders head(); view() and StreamView are removed. ArrowTableStream and every source serve the rows they hold; PollingSource never polls, CachedSource never writes. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/sources/base.py | 16 + src/orcapod/core/sources/cached_source.py | 23 + src/orcapod/core/sources/derived_source.py | 45 ++ src/orcapod/core/sources/polling_source.py | 39 ++ src/orcapod/core/sources/source_proxy.py | 26 ++ src/orcapod/core/streams/__init__.py | 3 + .../core/streams/arrow_table_stream.py | 49 +- src/orcapod/core/streams/base.py | 90 ++-- src/orcapod/core/streams/head.py | 424 ++++++++++++++++++ .../protocols/core_protocols/streams.py | 20 + 10 files changed, 702 insertions(+), 33 deletions(-) create mode 100644 src/orcapod/core/streams/head.py diff --git a/src/orcapod/core/sources/base.py b/src/orcapod/core/sources/base.py index e40fb2730..278c89825 100644 --- a/src/orcapod/core/sources/base.py +++ b/src/orcapod/core/sources/base.py @@ -5,6 +5,7 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import StreamProtocol from orcapod.types import ColumnConfig, Schema @@ -186,6 +187,21 @@ def as_table( """Delegate to the underlying stream's as_table.""" return self._stream.as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first ``n`` rows of the table the source holds, without computing. + + A source that reads its table when it is built holds it in memory, so this is a + slice of it. A source that reads its table later overrides this with a bounded + read. See ``StreamBase.head``. + """ + return self._stream.head(n, columns=columns, all_info=all_info).for_stream(self) + # ------------------------------------------------------------------------- # Convenience — caching # ------------------------------------------------------------------------- diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index e3a1079f7..1fe741da2 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -8,6 +8,7 @@ from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.head import StreamHead from orcapod.contexts.stored import ( keep_rows_of_data_context, require_same_data_context, @@ -405,6 +406,28 @@ def as_table( assert self._cached_stream is not None return self._cached_stream.as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows without reading or writing the cache. + + Once the source has been used, its merged stream is in memory and this is a slice + of it. Before that, the rows are the wrapped source's head: the cache table is + read and written only on first use (``iter_data``, ``as_table``). + """ + if self._cached_stream is not None and not self.is_stale: + return self._cached_stream.head( + n, columns=columns, all_info=all_info + ).for_stream(self) + return self._source.head(n, columns=columns, all_info=all_info).for_stream( + self, + note="rows of the wrapped source; the cache is read and written on first use", + ) + def get_all_records( self, columns: ColumnConfig | dict[str, Any] | None = None, diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 8ec06e038..8bb64e2bf 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -7,6 +7,12 @@ from orcapod.core.sources.source_ids import hash_source_id from orcapod.core.sources.stream_builder import SourceStreamBuilder from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.errors import InputValidationError from orcapod.system_constants import constants from orcapod.types import ColumnConfig, ContentHash, Schema @@ -422,6 +428,45 @@ def _get_stream(self) -> ArrowTableStream: tag_keys = self._origin.keys()[0] return ArrowTableStream(self._cached_table, tag_columns=tag_keys) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows of the origin's records, without computing. + + The records are the ones this source already read, or else the origin's + records read now (a job node's stored rows) and not kept. An origin job node + that has not run gives no rows. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + if self._tags_as_data is not None: + if self._cached_stream is not None: + stream, has_rows = self._cached_stream, True + else: + stream, has_rows = self._build_derived_stream(self._tags_as_data) + else: + table = self._cached_table + if table is None: + table = self._origin_records() + has_rows = table is not None and table.num_rows > 0 + if table is None: + table = self._empty_origin_table() + stream = ArrowTableStream(table, tag_columns=self._origin.keys()[0]) + origin = "held" if isinstance(self._origin, RootSource) else "stored" + if not has_rows and origin == "stored": + return StreamHead.without_rows( + self, + HeadStatus.NOTHING_STORED, + stream.as_table(columns=config), + tag_columns=stream.keys(columns=config)[0], + note="nothing stored: its origin has not run", + ) + return stream.head(n, columns=config).for_stream(self, origin=origin) + def iter_data(self): return self._get_stream().iter_data() diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index 1de88b4a2..cf9f57b2a 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -20,6 +20,12 @@ from orcapod.core.sources.base import RootSource from orcapod.core.sources.stream_builder import SourceStreamBuilder +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.errors import ( CursorInvalidatedError, InputValidationError, @@ -436,6 +442,39 @@ def keys( self._sync_poll_and_commit() return self._get_combined_stream().keys(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows already polled; never polls. + + Before the first poll there are no rows. The columns are the declared schema's, + or none when the source declared no schema (finding it would mean polling). + """ + check_head_size(n) + config = head_column_config(columns, all_info) + if self._batches: + return self._get_combined_stream().head(n, columns=config).for_stream(self) + if self._tag_schema is not None and self._data_schema is not None: + table = arrow_utils.make_empty_table( + {**self._tag_schema, **self._data_schema}, + self.data_context.type_converter, + ) + tag_columns = tuple(self._tag_schema.keys()) + else: + table = pa.table({}) + tag_columns = () + return StreamHead.without_rows( + self, + HeadStatus.NOT_COMPUTED, + table, + tag_columns=tag_columns, + note="nothing polled yet; iterating it or a job run polls it", + ) + def iter_data(self): """Iterate over (tag, data) pairs from the current snapshot.""" self._sync_poll_and_commit() diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index 3612f1b29..5dc501a8d 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -13,6 +13,12 @@ from typing import TYPE_CHECKING, Any from orcapod.core.sources.base import RootSource +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.protocols.core_protocols import SourceProtocol from orcapod.types import ContentHash, Schema @@ -202,6 +208,26 @@ def as_table( ) -> pa.Table: return self._require_delegate().as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the bound source's head, or no rows when no live source is bound.""" + if self._delegate is not None: + return self._delegate.head( + n, columns=columns, all_info=all_info + ).for_stream(self) + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, + HeadStatus.UNAVAILABLE, + note="source not available; bind a live source", + ) + def resolve_field(self, record_id: str, field_name: str) -> Any: """Delegate to bound source, or raise.""" return self._require_delegate().resolve_field(record_id, field_name) diff --git a/src/orcapod/core/streams/__init__.py b/src/orcapod/core/streams/__init__.py index 752c876bf..36f9302a0 100644 --- a/src/orcapod/core/streams/__init__.py +++ b/src/orcapod/core/streams/__init__.py @@ -1,7 +1,10 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import HeadStatus, StreamHead __all__ = [ "ArrowTableStream", + "HeadStatus", "StreamBase", + "StreamHead", ] diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index 77afe8944..d7606e524 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -1,5 +1,6 @@ from __future__ import annotations +import itertools import logging from collections.abc import Collection, Iterator from typing import TYPE_CHECKING, Any @@ -7,6 +8,7 @@ from orcapod import contexts from orcapod.core.datagrams import Data, Tag from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import StreamHead, check_head_size, head_column_config from orcapod.protocols.core_protocols import PodProtocol, StreamProtocol, TagProtocol from orcapod.protocols.hashing_protocols import PipelineElementProtocol from orcapod.system_constants import constants @@ -224,14 +226,55 @@ def as_table( This is useful for converting the stream to a table format. """ columns_config = ColumnConfig.handle_config(columns, all_info=all_info) + return self._assemble_table(columns_config, limit=None) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first ``n`` rows of the table this stream holds. + + Only those rows are assembled: content hashes, when asked for, are computed for + them alone. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + return StreamHead.of_rows( + self, + self._assemble_table(config, limit=n), + tag_columns=self.keys(columns=config)[0], + total_rows=self._table.num_rows, + ) + + def _assemble_table( + self, columns_config: ColumnConfig, limit: int | None + ) -> "pa.Table": + """Return the stream's rows with the column groups of ``columns_config``. + + Args: + columns_config: The column groups to include, and whether to sort by tags. + limit: Assemble only the first ``limit`` rows; ``None`` for all of them. + """ output_table = self._table + data_context_table = self._data_context_table + source_info_table = self._source_info_table + if limit is not None: + output_table = output_table.slice(0, limit) + data_context_table = data_context_table.slice(0, limit) + source_info_table = source_info_table.slice(0, limit) if columns_config.content_hash: hash_column_name = ( "_content_hash" if columns_config.content_hash is True else columns_config.content_hash ) - content_hashes = [str(data.content_hash()) for _, data in self.iter_data()] + rows = self.iter_data() + if limit is not None: + rows = itertools.islice(rows, limit) + content_hashes = [str(data.content_hash()) for _, data in rows] output_table = output_table.append_column( pa.field(hash_column_name, pa.large_string(), nullable=False), pa.array(content_hashes, type=pa.large_string()), @@ -241,9 +284,9 @@ def as_table( output_table = output_table.drop_columns(list(self._system_tag_columns)) table_stack = (output_table,) if columns_config.context: - table_stack += (self._data_context_table,) + table_stack += (data_context_table,) if columns_config.source: - table_stack += (self._source_info_table,) + table_stack += (source_info_table,) table = arrow_utils.hstack_tables(*table_stack) diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index e6d05cebf..6eb4e974f 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -7,6 +7,13 @@ from typing import TYPE_CHECKING, Any from orcapod.core.base import TraceableBase +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + check_head_size, + head_column_config, + render_error_html, +) from orcapod.protocols.core_protocols import ( DataProtocol, PodProtocol, @@ -14,6 +21,7 @@ TagProtocol, ) from orcapod.types import ColumnConfig, Schema +from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -440,41 +448,63 @@ def flow( """ return [e for e in self.iter_data()] - def _repr_html_(self) -> str: - df = self.as_polars_df() - # reorder columns - new_column_order = [c for c in df.columns if c in self.keys()[0]] + [ - c for c in df.columns if c not in self.keys()[0] - ] - df = df[new_column_order] - tag_map = {t: f"*{t}" for t in self.keys()[0]} - # TODO: construct repr html better - df = df.rename(tag_map) - return f"{self.__class__.__name__}[{self.label}]\n" + df._repr_html_() - - def view( + def head( self, + n: int = 10, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, - ) -> "StreamView": - df = self.as_polars_df( - columns=columns, - all_info=all_info, - ) - tag_map = {t: f"*{t}" for t in self.keys()[0]} - # TODO: construct repr html better - df = df.rename(tag_map) - return StreamView(self, df) + ) -> StreamHead: + """Return at most ``n`` rows that are readily available, without computing. + The rows are ones the stream already holds, has cached from an earlier + computation, has stored, or can read with a bounded read of its table. ``head`` + never calls a function, runs an operator, delivers a side effect, polls a source + or writes anything. When rows are shown, they are the first ``n`` rows that + ``as_table()`` would return. The notebook preview of a stream is its ``head()``. -class StreamView: - def __init__(self, stream: StreamBase, view_df: "pl.DataFrame") -> None: - self._stream = stream - self._view_df = view_df + This default shows no rows: a stream class that can serve rows without computing + overrides it. The zero-row table then has the stream's columns, typed from its + declared (Python) schema, so a column's Arrow width can differ from the one the + computed stream will have (``int`` shows as ``int64``). - def _repr_html_(self) -> str: - return ( - f"{self._stream.__class__.__name__}[{self._stream.label}]\n" - + self._view_df._repr_html_() + Args: + n: The largest number of rows to return. + columns: Column groups to include, as for ``as_table``. ``sort_by_tags`` is + not accepted: the rows come in stream order. + all_info: Include every column group, as for ``as_table`` (without sorting). + + Returns: + A ``StreamHead`` with the rows and a ``status`` that tells an empty result + from one that is not computed. + + Raises: + TypeError: If ``n`` is not an ``int``. + ValueError: If ``n`` is negative or ``columns`` asks for ``sort_by_tags``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + + def _head_without_rows( + self, + config: ColumnConfig, + status: HeadStatus, + note: str | None = None, + ) -> StreamHead: + """Return a head with no rows: the stream's columns from its declared schema.""" + tag_schema, data_schema = self.output_schema(columns=config) + table = arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) + return StreamHead.without_rows( + self, status, table, tag_columns=tuple(tag_schema.keys()), note=note ) + + def _repr_html_(self) -> str: + """Render ``head()`` for a notebook; never computes.""" + try: + head = self.head() + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(self, error) + return head._repr_html_() diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py new file mode 100644 index 000000000..28bd28a56 --- /dev/null +++ b/src/orcapod/core/streams/head.py @@ -0,0 +1,424 @@ +"""The first rows of a stream that are readily available, and their notebook preview (ITL-991). + +``head(n)`` on a stream, source or node returns a ``StreamHead``: at most ``n`` rows the +object already holds, has cached from an earlier computation, has stored, or can read from +its table with a bounded read. It never computes: no user function is called, no operator +is run, no side effect is delivered, no source is polled and nothing is written. The +notebook preview of a stream is its ``head()`` rendered by ``StreamHead._repr_html_``. +""" + +from __future__ import annotations + +import dataclasses +import html +from collections.abc import Mapping +from dataclasses import dataclass +from enum import Enum +from typing import TYPE_CHECKING, Any + +from orcapod.types import ColumnConfig +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import polars as pl + import pyarrow as pa + import pyarrow.compute as pc +else: + pa = LazyModule("pyarrow") + pc = LazyModule("pyarrow.compute") + pl = LazyModule("polars") + + +class HeadStatus(str, Enum): + """Where the rows of a ``StreamHead`` came from, or why it has none. + + Attributes: + ROWS: The rows are the first rows of the stream's complete content. Zero rows + means the content is empty. + PARTIAL: A computation stopped part way; the rows are those computed so far. + NOT_COMPUTED: Nothing is computed, or what was computed is out of date. + NOTHING_STORED: A job node whose store holds no rows for it yet. + UNAVAILABLE: No rows can exist here: a pipeline blueprint, an input slot with no + source bound, or a source that is not available. + """ + + ROWS = "rows" + PARTIAL = "partial" + NOT_COMPUTED = "not_computed" + NOTHING_STORED = "nothing_stored" + UNAVAILABLE = "unavailable" + + +NOT_COMPUTED_NOTE = "not computed; as_table(), iter_data() or a job run computes it" +STALE_NOTE = "not computed: inputs changed since the last computation" +NOTHING_STORED_NOTE = "nothing stored; run the job" +SIDE_EFFECTS_NOTE = ( + "not run; iterating it or a job run delivers its side effects; " + "displaying never delivers side effects" +) + +_DEFAULT_NOTES = { + HeadStatus.NOT_COMPUTED: NOT_COMPUTED_NOTE, + HeadStatus.NOTHING_STORED: NOTHING_STORED_NOTE, + HeadStatus.UNAVAILABLE: "no rows available", +} + +# Words naming the rows of a ``ROWS`` head, by where they came from. +_ORIGIN_NOUNS = { + "held": "rows", + "read": "rows", + "computed": "computed rows", + "stored": "stored rows", +} + +# Display limits of the preview. +_MAX_TEXT = 80 +_MAX_LIST_ITEMS = 5 + + +def check_head_size(n: int) -> None: + """Check the row count given to ``head``. + + Raises: + TypeError: If ``n`` is not an ``int``. + ValueError: If ``n`` is negative. + """ + if isinstance(n, bool) or not isinstance(n, int): + raise TypeError(f"head() takes an int number of rows, got {type(n).__name__}") + if n < 0: + raise ValueError(f"head() takes a non-negative number of rows n, got {n}") + + +def head_column_config( + columns: ColumnConfig | dict[str, Any] | None, all_info: bool +) -> ColumnConfig: + """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. + + The column groups mean what they mean for ``as_table``. ``head`` returns rows in + stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. + + Raises: + ValueError: If ``columns`` asks for ``sort_by_tags``. + """ + if isinstance(columns, ColumnConfig): + asked_to_sort = bool(columns.sort_by_tags) + elif isinstance(columns, Mapping): + asked_to_sort = bool(columns.get("sort_by_tags", False)) + else: + asked_to_sort = False + if asked_to_sort: + raise ValueError( + "head() returns the first rows in stream order and cannot sort them by " + "tags; sort the result of as_table() instead" + ) + config = ColumnConfig.handle_config(columns, all_info=all_info) + return dataclasses.replace(config, sort_by_tags=False) + + +@dataclass(frozen=True) +class StreamHead: + """At most ``n`` readily available rows of a stream, and how they were obtained. + + Attributes: + table: The rows, with the exact Arrow types the stream gives them. When the + stream has no rows to show, a zero-row table with its columns. + status: Where the rows came from, or why there are none. + kind: Class name of the stream, source or node shown. + label: Its label. + tag_columns: The columns of ``table`` that are tags. + total_rows: The stream's row count when it is known without computing, else + ``None``. For ``PARTIAL``, the number of rows computed so far. + total_is_estimate: ``True`` when ``total_rows`` is a recorded estimate. + origin: What the rows are: ``"held"`` (in memory), ``"read"`` (a bounded read + of a table), ``"computed"`` (cached from an earlier computation) or + ``"stored"`` (a job node's store). + note: For a head without rows, the explanation shown instead of the default; + for a head with rows, a remark shown after the row count. + """ + + table: pa.Table + status: HeadStatus + kind: str + label: str + tag_columns: tuple[str, ...] = () + total_rows: int | None = None + total_is_estimate: bool = False + origin: str = "held" + note: str | None = None + + @classmethod + def of_rows( + cls, + stream: Any, + table: pa.Table, + *, + tag_columns: tuple[str, ...], + total_rows: int | None, + status: HeadStatus = HeadStatus.ROWS, + total_is_estimate: bool = False, + origin: str = "held", + note: str | None = None, + ) -> StreamHead: + """Return a head with rows of ``stream``.""" + return cls( + table=table, + status=status, + kind=type(stream).__name__, + label=str(stream.label), + tag_columns=tuple(c for c in tag_columns if c in table.column_names), + total_rows=total_rows, + total_is_estimate=total_is_estimate, + origin=origin, + note=note, + ) + + @classmethod + def without_rows( + cls, + stream: Any, + status: HeadStatus, + table: pa.Table, + *, + tag_columns: tuple[str, ...], + note: str | None = None, + ) -> StreamHead: + """Return a head of ``stream`` with no rows: ``table`` holds its columns only.""" + return cls( + table=table.slice(0, 0), + status=status, + kind=type(stream).__name__, + label=str(stream.label), + tag_columns=tuple(c for c in tag_columns if c in table.column_names), + note=note, + ) + + def for_stream( + self, stream: Any, *, origin: str | None = None, note: str | None = None + ) -> StreamHead: + """Return the same rows shown as ``stream`` (a source over its inner stream, a node + over its cached output), optionally with another origin or note.""" + return dataclasses.replace( + self, + kind=type(stream).__name__, + label=str(stream.label), + origin=self.origin if origin is None else origin, + note=self.note if note is None else note, + ) + + # ------------------------------------------------------------------ + # Data access + # ------------------------------------------------------------------ + + def as_table(self) -> pa.Table: + """Return the rows as a PyArrow table.""" + return self.table + + def as_polars_df(self) -> pl.DataFrame: + """Return the rows as a Polars DataFrame.""" + return pl.DataFrame(self.table) + + def __len__(self) -> int: + return self.table.num_rows + + # ------------------------------------------------------------------ + # Rendering + # ------------------------------------------------------------------ + + def header(self) -> str: + """Return the one-line description shown above the rows.""" + return f"{self.kind}[{self.label}] — {self._describe()}" + + def _describe(self) -> str: + shown = self.table.num_rows + if self.status is HeadStatus.ROWS: + text = self._describe_rows(shown) + elif self.status is HeadStatus.PARTIAL: + total = self.total_rows if self.total_rows is not None else shown + if shown == total: + text = f"{total:,} {_rows(total)} computed so far" + else: + text = f"first {shown:,} of {total:,} rows computed so far" + text += " (computation incomplete)" + else: + return self.note or _DEFAULT_NOTES[self.status] + if self.note: + text += f"; {self.note}" + return text + + def _describe_rows(self, shown: int) -> str: + total = self.total_rows + noun = _ORIGIN_NOUNS.get(self.origin, "rows") + if shown == 0 and (total is None or total == 0): + return "computed: no rows" if self.origin == "computed" else f"no {noun}" + if total is None: + return f"first {shown:,} {_rows(shown)}" + if self.total_is_estimate: + return f"first {shown:,} {_rows(shown)} of about {total:,}" + if shown == total: + return f"all {total:,} {noun if total != 1 else noun[:-1]}" + return f"first {shown:,} of {total:,} {noun}" + + def _display_frame(self) -> pl.DataFrame: + """Return a Polars frame of the rows for display: tag columns first, large cells + summarised. ``table`` itself is never changed.""" + order = [c for c in self.tag_columns if c in self.table.column_names] + [ + c for c in self.table.column_names if c not in self.tag_columns + ] + table = self.table.select(order) + columns = [] + for name in table.column_names: + column = table.column(name) + summary = _summarise(column) + if summary is None: + columns.append(column) + else: + columns.append(pa.chunked_array([summary], type=pa.large_string())) + return pl.DataFrame(pa.table(columns, names=table.column_names)) + + def _tags_line(self) -> str: + if not self.tag_columns: + return "" + return "tags: " + ", ".join(self.tag_columns) + + def _repr_html_(self) -> str: + parts = [f"
{html.escape(self.header())}
"] + tags = self._tags_line() + if tags: + parts.append(f"
{html.escape(tags)}
") + if self.table.num_columns: + parts.append(self._display_frame()._repr_html_()) + return "\n".join(parts) + + def __str__(self) -> str: + lines = [self.header()] + tags = self._tags_line() + if tags: + lines.append(tags) + if self.table.num_columns: + lines.append(str(self._display_frame())) + return "\n".join(lines) + + def __repr__(self) -> str: + return ( + f"StreamHead({self.kind}[{self.label}], status={self.status.value}, " + f"rows={self.table.num_rows})" + ) + + +def render_error_html(stream: Any, error: BaseException) -> str: + """Return the preview of a stream whose ``head`` raised: its name and the error.""" + title = f"{type(stream).__name__}[{stream.label}] — could not read: {error}" + return f"
{html.escape(title)}
" + + +def _rows(count: int) -> str: + return "row" if count == 1 else "rows" + + +def _size(count: int) -> str: + """Return a byte count as text: ``300 B``, ``1.2 KB``, ``3.4 MB``.""" + value = float(count) + for unit in ("B", "KB", "MB", "GB"): + if value < 1000 or unit == "GB": + return f"{count} B" if unit == "B" else f"{value:.1f} {unit}" + value /= 1000 + return f"{count} B" # pragma: no cover + + +def _summarise(column: pa.ChunkedArray) -> list[str | None] | None: + """Return display strings for a column whose cells are too large to show as they are, + or ``None`` to show the column as it is.""" + arrow_type = column.type + if isinstance(arrow_type, pa.ExtensionType): + return _summarise_extension(column, arrow_type) + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + return [ + None if v is None else f"bytes ({_size(len(v))})" + for v in column.to_pylist() + ] + if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): + values = column.to_pylist() + if all(v is None or len(v) <= _MAX_TEXT for v in values): + return None + return [ + v if v is None or len(v) <= _MAX_TEXT else v[: _MAX_TEXT - 3] + "…" + for v in values + ] + if ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ): + return _summarise_list(column, arrow_type.value_type) + return None + + +def _summarise_extension( + column: pa.ChunkedArray, arrow_type: pa.ExtensionType +) -> list[str | None] | None: + storage_type = arrow_type.storage_type + short_name = arrow_type.extension_name.rsplit(".", 1)[-1] + if pa.types.is_struct(storage_type): + names = {storage_type.field(i).name for i in range(storage_type.num_fields)} + # Only the small fields are read; an ndarray's ``data`` is never decoded. + wanted = [n for n in ("dtype", "shape") if n in names] + values = [ + v for array in column.chunks for v in _struct_fields(array.storage, wanted) + ] + if {"dtype", "shape"} <= names: + return [ + None + if v is None + else f"{short_name} {v['dtype']} {_shape_text(v['shape'])}" + for v in values + ] + return [None if v is None else f"<{short_name}>" for v in values] + if pa.types.is_binary(storage_type) or pa.types.is_large_binary(storage_type): + return [ + None if v is None else f"{short_name} ({_size(len(v))})" + for array in column.chunks + for v in array.storage.to_pylist() + ] + return None + + +def _struct_fields(array: pa.StructArray, names: list[str]) -> list[dict | None]: + """Return each row of ``array`` as a dict of the named fields (``None`` for a null row).""" + fields = {name: array.field(name).to_pylist() for name in names} + valid = array.is_valid().to_pylist() + return [ + {name: fields[name][i] for name in names} if valid[i] else None + for i in range(len(array)) + ] + + +def _shape_text(shape: Any) -> str: + dims = list(shape or []) + if len(dims) == 1: + return f"({dims[0]},)" + return "(" + ", ".join(str(d) for d in dims) + ")" + + +def _summarise_list( + column: pa.ChunkedArray, value_type: pa.DataType +) -> list[str | None] | None: + simple_items = not ( + isinstance(value_type, pa.ExtensionType) + or pa.types.is_nested(value_type) + or pa.types.is_binary(value_type) + or pa.types.is_large_binary(value_type) + ) + lengths: list[int | None] = pc.list_value_length(column).to_pylist() + if simple_items and all(n is None or n <= _MAX_LIST_ITEMS for n in lengths): + return None + values = column.to_pylist() if simple_items else [None] * len(lengths) + item = "list" if pa.types.is_nested(value_type) else None + out: list[str | None] = [] + for value, length in zip(values, lengths): + if length is None: + out.append(None) + elif simple_items and length <= _MAX_LIST_ITEMS: + out.append(repr(value)) + else: + out.append(f"list (len {length})" + (f" of {item}" if item else "")) + return out diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index 894e437a3..200d8b261 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -13,6 +13,7 @@ import polars as pl import pyarrow as pa + from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols.pod import PodProtocol @@ -157,6 +158,25 @@ def as_table( """ ... + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> "StreamHead": + """Return at most ``n`` readily available rows, without computing. + + The rows are ones the stream already holds, has cached from an earlier + computation, has stored, or can read with a bounded read. It never calls a + function, runs an operator, delivers a side effect, polls or writes. + + Returns: + StreamHead: The rows, and a status that tells an empty result from one + that is not computed. + """ + ... + class StreamWithOperationsProtocol(StreamProtocol, Protocol): def as_df( From 45fda036a64fe1e25e1e2de0a4b6d8729c979857 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:06:54 -0800 Subject: [PATCH 05/19] feat(streams): operator, function pod and side-effect streams show only computed rows (ITL-991) DynamicPodStream.head shows its kept result, or NOT_COMPUTED; the operator is never run, and an input that changed shows 'inputs changed'. FunctionPodStream.head shows its computed rows (PARTIAL after an incomplete iteration) and never calls the function; its as_table is split into _table_from_pairs and _shape_table, shared with head. Side-effect streams and nodes show no rows and never deliver. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 207 +++++++++++++----- .../core/operators/static_output_pod.py | 34 +++ src/orcapod/side_effects.py | 35 +++ 3 files changed, 222 insertions(+), 54 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 1580962fa..594cb9434 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import dataclasses import logging from abc import abstractmethod from collections.abc import Callable, Collection, Iterator, Sequence @@ -13,6 +14,13 @@ from orcapod.core.base import TraceableBase from orcapod.core.data_function import CachedDataFunction, PythonDataFunction from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import ( + STALE_NOTE, + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import StoreError, TypeIdentityError from orcapod.utils.store_errors import keep_store_error, note_callback_failure @@ -1035,57 +1043,162 @@ def as_table( all_info: bool = False, ) -> pa.Table: if self._cached_output_table is None: - all_tags = [] - all_data = [] - tag_schema, data_schema = None, None - for tag, data in self.iter_data(): - if tag_schema is None: - tag_schema = tag.arrow_schema(all_info=True) - if data_schema is None: - data_schema = data.arrow_schema(all_info=True) - # TODO: make use of arrow_compat dict - all_tags.append(tag.as_dict(all_info=True)) - all_data.append(data.as_dict(all_info=True)) - - # TODO: re-verify the implemetation of this conversion - converter = self.data_context.type_converter - - # The data columns have declared types: convert each value as its column's - # type, never as the type the value happens to have. A subclass instance - # returned for a base-class column is then refused instead of being stored - # as the base without its own fields (ITL-972). Only the remaining columns - # (meta, source info) are typed from their values. - python_schema = dict(infer_python_schema_from_pylist_data(all_data)) - python_schema.update(self.output_schema()[1]) - struct_data = converter.python_dicts_to_struct_dicts( - all_data, python_schema=python_schema - ) - all_tags_as_tables: pa.Table = pa.Table.from_pylist( - all_tags, schema=tag_schema + self._cached_output_table = self._table_from_pairs(list(self.iter_data())) + assert self._cached_output_table is not None, ( + "_cached_output_table should not be None here." + ) + + column_config = ColumnConfig.handle_config(columns, all_info=all_info) + + def all_content_hashes() -> pa.Array: + if self._cached_content_hash_column is None: + # TODO: verify that order will be preserved + self._cached_content_hash_column = pa.array( + [data.content_hash().to_string() for _, data in self.iter_data()], + type=pa.large_string(), + ) + return self._cached_content_hash_column + + return self._shape_table( + self._cached_output_table, column_config, all_content_hashes + ) + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows already computed; never calls the function. + + After a complete computation (``as_table``, or iteration to the end) the rows are + the first of the result. After an iteration that stopped part way they are the + rows computed so far (``PARTIAL``). Before any computation, or once an input + changed, there are none. A ``CachedFunctionPod``'s result store is not looked + up: that needs the input rows' hashes, i.e. reading the input. See + ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + has_results = bool(self._cached_output_datas) or ( + self._cached_output_table is not None + ) + if has_results and self.is_stale: + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE ) - # drop context key column from tags table (guard: column absent on empty stream) - if constants.CONTEXT_KEY in all_tags_as_tables.column_names: - all_tags_as_tables = all_tags_as_tables.drop([constants.CONTEXT_KEY]) - all_data_as_tables: pa.Table = pa.Table.from_pylist( - struct_data, schema=data_schema + tag_columns = self.keys(columns=config)[0] + + # The contiguous run of computed inputs from the first; an input whose output + # is None (filtered out) is skipped, as iteration skips it. + computed: list[tuple[TagProtocol, DataProtocol]] = [] + index = 0 + while index in self._cached_output_datas: + tag, data = self._cached_output_datas[index] + if data is not None: + computed.append((tag, data)) + index += 1 + shown = computed[:n] + + def shown_content_hashes() -> pa.Array: + return pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), ) - self._cached_output_table = arrow_utils.hstack_tables( - all_tags_as_tables, all_data_as_tables + if self._cached_output_table is not None: + return StreamHead.of_rows( + self, + self._shape_table( + self._cached_output_table.slice(0, n), config, shown_content_hashes + ), + tag_columns=tag_columns, + total_rows=self._cached_output_table.num_rows, + origin="computed", ) - assert self._cached_output_table is not None, ( - "_cached_output_table should not be None here." + finished = not self._needs_iterator and self._cached_input_iterator is None + if not finished and not computed: + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + if not shown: + # Nothing to show (an empty result, or n == 0): the declared columns. + empty = self._head_without_rows(config, HeadStatus.ROWS) + return dataclasses.replace( + empty, + status=HeadStatus.ROWS if finished else HeadStatus.PARTIAL, + total_rows=len(computed), + origin="computed", + ) + return StreamHead.of_rows( + self, + self._shape_table( + self._table_from_pairs(shown), config, shown_content_hashes + ), + tag_columns=tag_columns, + total_rows=len(computed), + status=HeadStatus.ROWS if finished else HeadStatus.PARTIAL, + origin="computed", ) - column_config = ColumnConfig.handle_config(columns, all_info=all_info) + def _table_from_pairs( + self, pairs: list[tuple[TagProtocol, DataProtocol]] + ) -> pa.Table: + """Return the output rows ``pairs`` as one table with every column group.""" + all_tags = [] + all_data = [] + tag_schema, data_schema = None, None + for tag, data in pairs: + if tag_schema is None: + tag_schema = tag.arrow_schema(all_info=True) + if data_schema is None: + data_schema = data.arrow_schema(all_info=True) + # TODO: make use of arrow_compat dict + all_tags.append(tag.as_dict(all_info=True)) + all_data.append(data.as_dict(all_info=True)) + + # TODO: re-verify the implemetation of this conversion + converter = self.data_context.type_converter + + # The data columns have declared types: convert each value as its column's + # type, never as the type the value happens to have. A subclass instance + # returned for a base-class column is then refused instead of being stored + # as the base without its own fields (ITL-972). Only the remaining columns + # (meta, source info) are typed from their values. + python_schema = dict(infer_python_schema_from_pylist_data(all_data)) + python_schema.update(self.output_schema()[1]) + struct_data = converter.python_dicts_to_struct_dicts( + all_data, python_schema=python_schema + ) + all_tags_as_tables: pa.Table = pa.Table.from_pylist(all_tags, schema=tag_schema) + # drop context key column from tags table (guard: column absent on empty stream) + if constants.CONTEXT_KEY in all_tags_as_tables.column_names: + all_tags_as_tables = all_tags_as_tables.drop([constants.CONTEXT_KEY]) + all_data_as_tables: pa.Table = pa.Table.from_pylist( + struct_data, schema=data_schema + ) + return arrow_utils.hstack_tables(all_tags_as_tables, all_data_as_tables) + + def _shape_table( + self, + table: pa.Table, + column_config: ColumnConfig, + content_hashes: Callable[[], pa.Array], + ) -> pa.Table: + """Return ``table`` with the column groups of ``column_config``. + Args: + table: Output rows with every column group (``_table_from_pairs``). + column_config: The column groups to keep, and whether to sort by tags. + content_hashes: Returns the content hash of each row of ``table``; called + only when ``column_config`` asks for them. + """ drop_columns = [] if not column_config.system_tags: # TODO: get system tags more effiicently drop_columns.extend( [ c - for c in self._cached_output_table.column_names + for c in table.column_names if c.startswith(constants.SYSTEM_TAG_PREFIX) ] ) @@ -1094,30 +1207,16 @@ def as_table( if not column_config.context: drop_columns.append(constants.CONTEXT_KEY) - output_table = self._cached_output_table.drop( - [c for c in drop_columns if c in self._cached_output_table.column_names] - ) + output_table = table.drop([c for c in drop_columns if c in table.column_names]) - # lazily prepare content hash column if requested if column_config.content_hash: - if self._cached_content_hash_column is None: - content_hashes = [] - # TODO: verify that order will be preserved - for tag, data in self.iter_data(): - content_hashes.append(data.content_hash().to_string()) - self._cached_content_hash_column = pa.array( - content_hashes, type=pa.large_string() - ) - assert self._cached_content_hash_column is not None, ( - "_cached_content_hash_column should not be None here." - ) hash_column_name = ( "_content_hash" if column_config.content_hash is True else column_config.content_hash ) output_table = output_table.append_column( - hash_column_name, self._cached_content_hash_column + hash_column_name, content_hashes() ) if column_config.sort_by_tags: diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 9ff23d1ea..4aa565d8b 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -12,6 +12,13 @@ from orcapod.contexts import DataContext from orcapod.core.base import TraceableBase from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import ( + STALE_NOTE, + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( ArgumentGroup, @@ -423,5 +430,32 @@ def iter_data( ) return self._cached_stream.iter_data() + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows of the operator's kept result; never runs the operator. + + The head of an operator is not the operator over the heads of its inputs (a + group over ten rows has the wrong members), so before the result is computed, + or after an input changed, there are no rows. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + had_result = self._cached_stream is not None + self._update_cache_status() + if self._cached_stream is not None and self._cached_time is not None: + return self._cached_stream.head(n, columns=config).for_stream( + self, origin="computed" + ) + return self._head_without_rows( + config, + HeadStatus.NOT_COMPUTED, + note=STALE_NOTE if had_result else None, + ) + def __repr__(self) -> str: return f"{self.__class__.__name__}(kernel={self.producer}, upstreams={self.upstreams})" diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index 1893a1dc7..b64a1cd43 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -18,6 +18,13 @@ from orcapod.core.base import TraceableBase from orcapod.core.nodes.function_node import _build_record_id_preimage from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.head import ( + SIDE_EFFECTS_NOTE, + HeadStatus, + StreamHead, + check_head_size, + head_column_config, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( StoreError, @@ -148,6 +155,20 @@ def keys( ) -> tuple[tuple[str, ...], tuple[str, ...]]: return self._input_stream.keys(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: showing this stream must never deliver its side effects.""" + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE + ) + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: for tag, data in self._input_stream.iter_data(): result = _execute_side_effect_row( @@ -730,6 +751,20 @@ def keys( ) -> tuple[tuple[str, ...], tuple[str, ...]]: return self._input_stream.keys(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: showing this stream must never deliver its side effects.""" + check_head_size(n) + config = head_column_config(columns, all_info) + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=SIDE_EFFECTS_NOTE + ) + def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: """Standalone iteration — executes delivery with no DB logging.""" for tag, data in self._input_stream.iter_data(): From 184b54906b854f8c2727f6ff549d90a0ff4ad931 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:08:38 -0800 Subject: [PATCH 06/19] feat(streams): a cached function pod's stream shows its computed rows as as_table does (ITL-991) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/cached_function_pod.py | 42 ++++++++++++++++--- src/orcapod/core/function_pod.py | 25 +++++++---- .../streams/test_head_and_preview.py | 20 +++++++++ 3 files changed, 73 insertions(+), 14 deletions(-) diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index f40a67cf5..25202b7b4 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -2,6 +2,7 @@ from __future__ import annotations +import dataclasses import logging from datetime import datetime, timezone from collections.abc import Iterator @@ -9,6 +10,7 @@ from orcapod.core.datagrams import Datagram from orcapod.core.function_pod import FunctionPodStream, WrappedFunctionPod +from orcapod.core.streams.head import StreamHead from orcapod.system_constants import constants from orcapod.types import ColumnConfig from orcapod.contexts.stored import StaleTables @@ -451,12 +453,35 @@ class _CachedFunctionPodStream(FunctionPodStream): """ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: - keep = CachedFunctionPod.RESULT_COMPUTED_FLAG for tag, data in super().iter_data(): - stored = [name for name in data.meta_columns if name != keep] - if stored: - data = data.drop_meta_columns(*stored) - yield tag, data + yield tag, _without_stored_meta(data) + + def _computed_prefix(self) -> list[tuple[TagProtocol, DataProtocol]]: + return [ + (tag, _without_stored_meta(data)) + for tag, data in super()._computed_prefix() + ] + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first computed rows, with the columns ``as_table`` gives. + + The pod's result store is not looked up: that needs the input rows' hashes, + i.e. reading the input. See ``FunctionPodStream.head``. + """ + head = super().head(n, columns=columns, all_info=all_info) + column_config = ColumnConfig.handle_config(columns, all_info=all_info) + if column_config.meta or column_config.all_info: + return head + table = head.table.drop_columns( + [n for n in head.table.column_names if n.startswith(constants.META_PREFIX)] + ) + return dataclasses.replace(head, table=table) def as_table( self, @@ -471,3 +496,10 @@ def as_table( return table.drop_columns( [n for n in table.column_names if n.startswith(constants.META_PREFIX)] ) + + +def _without_stored_meta(data: DataProtocol) -> DataProtocol: + """Return ``data`` with only the ``__computed`` meta column of its stored ones.""" + keep = CachedFunctionPod.RESULT_COMPUTED_FLAG + stored = [name for name in data.meta_columns if name != keep] + return data.drop_meta_columns(*stored) if stored else data diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 5408303bc..87cd7d761 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1158,15 +1158,7 @@ def head( ) tag_columns = self.keys(columns=config)[0] - # The contiguous run of computed inputs from the first; an input whose output - # is None (filtered out) is skipped, as iteration skips it. - computed: list[tuple[TagProtocol, DataProtocol]] = [] - index = 0 - while index in self._cached_output_datas: - tag, data = self._cached_output_datas[index] - if data is not None: - computed.append((tag, data)) - index += 1 + computed = self._computed_prefix() shown = computed[:n] def shown_content_hashes() -> pa.Array: @@ -1208,6 +1200,21 @@ def shown_content_hashes() -> pa.Array: origin="computed", ) + def _computed_prefix(self) -> list[tuple[TagProtocol, DataProtocol]]: + """Return the output rows computed so far, in order, as iteration yields them. + + They are the contiguous run of computed inputs from the first; an input whose + output is ``None`` (filtered out) is skipped, as iteration skips it. + """ + computed: list[tuple[TagProtocol, DataProtocol]] = [] + index = 0 + while index in self._cached_output_datas: + tag, data = self._cached_output_datas[index] + if data is not None: + computed.append((tag, data)) + index += 1 + return computed + def _table_from_pairs( self, pairs: list[tuple[TagProtocol, DataProtocol]] ) -> pa.Table: diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 4505a777b..8da1bd180 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -388,3 +388,23 @@ def test_preview_summarises_large_cells_and_head_keeps_them() -> None: def test_text_form_has_the_header() -> None: text = str(_source().head(2)) assert text.startswith("ArrowTableSource[readings] — first 2 of 25 rows") + + +def test_cached_function_pod_stream_does_not_read_its_result_store() -> None: + from orcapod.core.cached_function_pod import CachedFunctionPod + + calls: list[int] = [] + store = InMemoryArrowDatabase() + pod = CachedFunctionPod(_counting_pod(calls).pod, result_database=store) + _ = pod.process(_source()).as_table() # fills the result store + assert len(calls) == 25 + + stream = pod.process(_source()) + head = stream.head() + assert head.status is HeadStatus.NOT_COMPUTED + table = stream.as_table() # served from the result store + assert len(calls) == 25 + after = stream.head(5) + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 5)) + assert len(calls) == 25 From 1e227da3d211f04c364f302a03a43844f64c0df1 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:09:34 -0800 Subject: [PATCH 07/19] test(nodes): a node's head shows its in-memory or stored rows and never runs it (ITL-991) Co-Authored-By: Claude Opus 5.5 --- tests/test_pipeline/test_node_head.py | 195 ++++++++++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 tests/test_pipeline/test_node_head.py diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py new file mode 100644 index 000000000..4d1f7bd40 --- /dev/null +++ b/tests/test_pipeline/test_node_head.py @@ -0,0 +1,195 @@ +"""A job node's ``head()`` shows its in-memory or stored rows and never runs it (ITL-991). + +A node is associated with its storage (developer, November 12, 2026): its preview is the +first rows of what it holds after a run, or of its stored table, and showing it never +computes. These tests run jobs on both orchestrators and both stores, and load saved jobs. +""" + +from __future__ import annotations + +from pathlib import Path + +import pyarrow as pa +import pytest + +from orcapod import function_pod +from orcapod.core.nodes import OperatorJobNode +from orcapod.core.nodes.source_node import SourceJobNode, SourceNode +from orcapod.core.operators import SelectDataColumns +from orcapod.core.sources import ArrowTableSource +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline.async_orchestrator import AsyncPipelineOrchestrator +from orcapod.pipeline.job import PipelineJob +from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator +from orcapod.protocols.database_protocols import ArrowDatabaseProtocol +from orcapod.streams import HeadStatus +from orcapod.types import CacheMode + +CALLS: list[int] = [] + +ORCHESTRATORS = { + "sync": SyncPipelineOrchestrator, + "async": AsyncPipelineOrchestrator, +} + + +@function_pod(output_keys="y") +def triple(x: int) -> int: + """Triple ``x``, counting every call.""" + CALLS.append(x) + return 3 * x + + +@pytest.fixture(autouse=True) +def _reset_calls() -> None: + CALLS.clear() + + +def _source(rows: int = 25) -> ArrowTableSource: + table = pa.table( + {"id": list(range(rows)), "x": list(range(rows))}, + schema=pa.schema( + [ + pa.field("id", pa.int64(), nullable=False), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + return ArrowTableSource(table, tag_columns=["id"], source_id="nums", label="nums") + + +def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + if kind == "in_memory": + return InMemoryArrowDatabase() + return DeltaTableDatabase(base_path=tmp_path / "store") + + +def _job(store: ArrowDatabaseProtocol) -> PipelineJob: + job = PipelineJob(name="itl991", store=store) + with job: + tripled = triple.pod(_source(), label="triple") + _ = SelectDataColumns(["y"])(tripled, label="sel") + return job + + +def test_nodes_before_a_run_show_no_rows_and_compute_nothing() -> None: + job = _job(InMemoryArrowDatabase()) + fn_head = job.nodes["triple"].head() + op_head = job.nodes["sel"].head() + _ = job.nodes["triple"]._repr_html_() + _ = job.nodes["sel"]._repr_html_() + assert CALLS == [] + assert fn_head.status is HeadStatus.NOTHING_STORED + assert fn_head.table.column_names == ["id", "y"] + assert op_head.status is HeadStatus.NOT_COMPUTED + assert "cache mode OFF" in op_head.header() + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_nodes_after_a_run_show_their_first_rows( + orchestrator: str, store_kind: str, tmp_path: Path +) -> None: + job = _job(_store(store_kind, tmp_path)) + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + assert len(CALLS) == 25 + for label in ("triple", "sel"): + node = job.nodes[label] + head = node.head() + assert head.status is HeadStatus.ROWS, label + assert head.total_rows == 25 + assert head.table.equals(node.as_table().slice(0, 10)), label + assert "first 10 of 25" in node._repr_html_() + assert len(CALLS) == 25 + + +@pytest.mark.parametrize("store_kind", ["in_memory", "delta"]) +def test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing( + store_kind: str, tmp_path: Path +) -> None: + store = _store(store_kind, tmp_path) + _ = _job(store).run() + CALLS.clear() + + node = _job(store).nodes["triple"] + head = node.head(4) + assert CALLS == [] + assert head.status is HeadStatus.ROWS + # The whole stored slice is read, so the header states the exact count and + # makes no claim of a bounded read. + assert head.header() == "FunctionJobNode[triple] — first 4 of 25 stored rows" + assert head.table.equals(node.as_table().slice(0, 4)) + + +def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None: + store = DeltaTableDatabase(base_path=tmp_path / "store") + job = _job(store) + _ = job.run() + saved = tmp_path / "job.json" + job.save(saved) + CALLS.clear() + + loaded = PipelineJob.load(str(saved), store=store) + fn_head = loaded.nodes["triple"].head() + op_head = loaded.nodes["sel"].head() + assert CALLS == [] + assert fn_head.status is HeadStatus.ROWS + assert fn_head.header().endswith("first 10 of 25 stored rows") + assert op_head.status is HeadStatus.UNAVAILABLE + assert "cache mode OFF" in op_head.header() + + +def test_blueprint_nodes_are_unavailable() -> None: + job = _job(InMemoryArrowDatabase()) + for label in ("triple", "sel"): + blueprint = job.nodes[label].as_node() + head = blueprint.head() + assert head.status is HeadStatus.UNAVAILABLE + assert "pipeline blueprint" in blueprint._repr_html_() + + +def test_source_nodes() -> None: + slot = SourceNode(name="nums", tag_schema={"id": int}, data_schema={"x": int}) + assert slot.head().status is HeadStatus.UNAVAILABLE + assert "input slot" in slot._repr_html_() + unbound = SourceJobNode(name="nums", tag_schema={"id": int}, data_schema={"x": int}) + assert unbound.head().status is HeadStatus.UNAVAILABLE + assert "no source bound" in unbound.head().header() + source = _source() + bound = SourceJobNode(name="nums", bound_source=source) + assert bound.head().table.equals(source.head().table) + assert bound.head().header() == "SourceJobNode[nums] — first 10 of 25 rows" + + +def _operator_node( + cache_mode: CacheMode, store: InMemoryArrowDatabase +) -> OperatorJobNode: + return OperatorJobNode( + operator=SelectDataColumns(["x"]), + input_streams=[_source()], + pipeline_database=store, + cache_mode=cache_mode, + ) + + +def test_operator_node_cache_modes() -> None: + store = InMemoryArrowDatabase() + assert _operator_node(CacheMode.REPLAY, store).head().status is ( + HeadStatus.NOTHING_STORED + ) + seed = _operator_node(CacheMode.LOG, store) + _ = seed.execute(_source()) + + replay = _operator_node(CacheMode.REPLAY, store) + head = replay.head(3) + assert head.status is HeadStatus.ROWS + assert ( + head.header() + == "OperatorJobNode[SelectDataColumns] — first 3 of 25 stored rows" + ) + assert head.table.equals(replay.as_table().slice(0, 3)) + + log = _operator_node(CacheMode.LOG, store) + log_head = log.head() + assert log_head.status is HeadStatus.NOT_COMPUTED + assert "not replayed" in log_head.header() From 7e91225d5d1bb810756005260ff3b0a87c747e9f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:14:08 -0800 Subject: [PATCH 08/19] feat(nodes): job nodes show their in-memory or stored rows and never run (ITL-991) FunctionJobNode and OperatorJobNode heads show the rows held after a run, or else the first of their stored rows (the node's whole stored slice is read; nothing is kept). Blueprints, input slots and unbound source nodes show UNAVAILABLE. Source nodes get head() and a preview. head() joins StreamProtocol; the argument checks live in core/streams/base.py to keep the import graph acyclic. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 14 +- src/orcapod/core/nodes/function_node.py | 253 ++++++++++++++---- src/orcapod/core/nodes/operator_node.py | 87 +++++- src/orcapod/core/nodes/source_node.py | 67 +++++ .../core/operators/static_output_pod.py | 8 +- src/orcapod/core/sources/base.py | 5 +- src/orcapod/core/sources/derived_source.py | 2 + src/orcapod/core/sources/polling_source.py | 2 + src/orcapod/core/sources/source_proxy.py | 2 + .../core/streams/arrow_table_stream.py | 8 +- src/orcapod/core/streams/base.py | 42 ++- src/orcapod/core/streams/head.py | 63 ++--- src/orcapod/side_effects.py | 8 +- .../test_dynamic_pod_stream_staleness.py | 2 + .../streams/test_head_and_preview.py | 8 +- tests/test_pipeline/test_node_head.py | 10 +- 16 files changed, 450 insertions(+), 131 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 87cd7d761..7e3915ff3 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -13,13 +13,15 @@ from orcapod.config import OrcapodConfig from orcapod.core.base import TraceableBase from orcapod.core.data_function import CachedDataFunction, PythonDataFunction -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) from orcapod.core.streams.head import ( STALE_NOTE, HeadStatus, StreamHead, - check_head_size, - head_column_config, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import StoreError, TypeIdentityError @@ -1118,7 +1120,7 @@ def as_table( column_config = ColumnConfig.handle_config(columns, all_info=all_info) - def all_content_hashes() -> pa.Array: + def all_content_hashes() -> pa.Array[Any]: if self._cached_content_hash_column is None: # TODO: verify that order will be preserved self._cached_content_hash_column = pa.array( @@ -1161,7 +1163,7 @@ def head( computed = self._computed_prefix() shown = computed[:n] - def shown_content_hashes() -> pa.Array: + def shown_content_hashes() -> pa.Array[Any]: return pa.array( [data.content_hash().to_string() for _, data in shown], type=pa.large_string(), @@ -1257,7 +1259,7 @@ def _shape_table( self, table: pa.Table, column_config: ColumnConfig, - content_hashes: Callable[[], pa.Array], + content_hashes: Callable[[], pa.Array[Any]], ) -> pa.Table: """Return ``table`` with the column groups of ``column_config``. diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 888d0b0df..ebc0d908a 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -21,7 +21,7 @@ import logging import uuid from abc import abstractmethod -from collections.abc import Iterator, Mapping +from collections.abc import Callable, Iterator, Mapping from typing import TYPE_CHECKING, Any, Literal, NamedTuple, cast from orcapod import contexts @@ -29,7 +29,16 @@ from orcapod.config import OrcapodConfig from orcapod.core.cached_function_pod import CachedFunctionPod from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + BLUEPRINT_NOTE, + HeadStatus, + StreamHead, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.core.datagrams.tag_data import EmptyData, Tag from orcapod.errors import ( @@ -173,6 +182,11 @@ class FunctionNodeBase(StreamBase): shared by both the blueprint and the execution variant. """ + # Set by each subclass's ``__init__``: the output table and content hashes that + # ``as_table`` builds once. + _cached_output_table: "pa.Table | None" + _cached_content_hash_column: "pa.Array[Any] | None" + def __init__( self, function_pod: FunctionPodProtocol, @@ -455,78 +469,133 @@ def as_table( all_info: bool = False, ) -> "pa.Table": if self._cached_output_table is None: - all_tags: list[dict] = [] - all_data: list[dict] = [] - # Derive Arrow schemas from the declared output_schema upfront so that - # the cached table always has a consistent, well-formed schema regardless - # of whether any rows were produced. - tag_python_schema, data_python_schema = self.output_schema(all_info=True) - for tag, data in self.iter_data(): - all_tags.append(tag.as_dict(all_info=True)) - all_data.append(data.as_dict(all_info=True)) - - converter = self.data_context.type_converter - tag_arrow_schema = converter.python_schema_to_arrow_schema( - tag_python_schema - ) - data_arrow_schema = converter.python_schema_to_arrow_schema( - data_python_schema - ) - - if not all_tags: - # No data was produced (node has not been run yet or produced zero - # outputs). Build a zero-row table whose schema is derived from the - # pod's declared output so that callers always receive a well-formed - # schema regardless of whether the node has actually executed. - self._cached_output_table = pa.Table.from_pylist( - [], - schema=converter.python_schema_to_arrow_schema( - tag_python_schema + data_python_schema - ), - ) - else: - struct_data = converter.python_dicts_to_struct_dicts( - all_data, python_schema=data_python_schema - ) - all_tags_as_tables: pa.Table = pa.Table.from_pylist( - all_tags, schema=tag_arrow_schema - ) - if constants.CONTEXT_KEY in all_tags_as_tables.column_names: - all_tags_as_tables = all_tags_as_tables.drop( - [constants.CONTEXT_KEY] - ) - all_data_as_tables: pa.Table = pa.Table.from_pylist( - struct_data, schema=data_arrow_schema - ) - self._cached_output_table = arrow_utils.hstack_tables( - all_tags_as_tables, all_data_as_tables - ) + self._cached_output_table = self._table_from_pairs(list(self.iter_data())) column_config = ColumnConfig.handle_config(columns, all_info=all_info) - output_table = arrow_utils.apply_column_config( - self._cached_output_table, column_config, self.keys()[0] - ) - if column_config.content_hash: + def all_content_hashes() -> "pa.Array[Any]": if self._cached_content_hash_column is None: - content_hashes = [] - for tag, data in self.iter_data(): - content_hashes.append(data.content_hash().to_string()) self._cached_content_hash_column = pa.array( - content_hashes, type=pa.large_string() + [data.content_hash().to_string() for _, data in self.iter_data()], + type=pa.large_string(), ) - assert self._cached_content_hash_column is not None + return self._cached_content_hash_column + + return self._shape_table( + self._cached_output_table, column_config, all_content_hashes + ) + + def _table_from_pairs( + self, pairs: list[tuple[TagProtocol, DataProtocol]] + ) -> "pa.Table": + """Return the output rows ``pairs`` as one table with every column group. + + The Arrow schemas come from the declared output schema, so the table is + well formed whether or not there are rows. + """ + all_tags: list[dict[str, Any]] = [] + all_data: list[dict[str, Any]] = [] + tag_python_schema, data_python_schema = self.output_schema(all_info=True) + for tag, data in pairs: + all_tags.append(tag.as_dict(all_info=True)) + all_data.append(data.as_dict(all_info=True)) + + converter = self.data_context.type_converter + tag_arrow_schema = converter.python_schema_to_arrow_schema(tag_python_schema) + data_arrow_schema = converter.python_schema_to_arrow_schema(data_python_schema) + + if not all_tags: + # No data was produced (node has not been run yet or produced zero + # outputs). Build a zero-row table whose schema is derived from the + # pod's declared output so that callers always receive a well-formed + # schema regardless of whether the node has actually executed. + return pa.Table.from_pylist( + [], + schema=converter.python_schema_to_arrow_schema( + tag_python_schema + data_python_schema + ), + ) + struct_data = converter.python_dicts_to_struct_dicts( + all_data, python_schema=data_python_schema + ) + all_tags_as_tables: pa.Table = pa.Table.from_pylist( + all_tags, schema=tag_arrow_schema + ) + if constants.CONTEXT_KEY in all_tags_as_tables.column_names: + all_tags_as_tables = all_tags_as_tables.drop([constants.CONTEXT_KEY]) + all_data_as_tables: pa.Table = pa.Table.from_pylist( + struct_data, schema=data_arrow_schema + ) + return arrow_utils.hstack_tables(all_tags_as_tables, all_data_as_tables) + + def _shape_table( + self, + table: "pa.Table", + column_config: ColumnConfig, + content_hashes: "Callable[[], pa.Array[Any]]", + ) -> "pa.Table": + """Return ``table`` with the column groups of ``column_config``. + + Args: + table: Output rows with every column group (``_table_from_pairs``). + column_config: The column groups to keep. + content_hashes: Returns the content hash of each row of ``table``; called + only when ``column_config`` asks for them. + """ + output_table = arrow_utils.apply_column_config( + table, column_config, self.keys()[0] + ) + if column_config.content_hash: hash_column_name = ( "_content_hash" if column_config.content_hash is True else column_config.content_hash ) output_table = output_table.append_column( - hash_column_name, self._cached_content_hash_column + hash_column_name, content_hashes() ) - return output_table + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: a node of this kind holds no data. See ``StreamBase.head``.""" + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note=BLUEPRINT_NOTE, + ) + + def _head_of_pairs( + self, + pairs: list[tuple[TagProtocol, DataProtocol]], + n: int, + config: ColumnConfig, + origin: str, + ) -> StreamHead: + """Return the first ``n`` of ``pairs`` as this node's head.""" + shown = pairs[:n] + table = self._shape_table( + self._table_from_pairs(shown), + config, + lambda: pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ), + ) + return StreamHead.of_rows( + self, + table, + tag_columns=self.keys(columns=config)[0], + total_rows=len(pairs), + origin=origin, + ) + def __repr__(self) -> str: return ( f"{type(self).__name__}(data_function={self._data_function!r}, " @@ -2851,6 +2920,72 @@ async def _async_execute_cache_only( finally: await output.close() + # ------------------------------------------------------------------ + # Head + # ------------------------------------------------------------------ + + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows the node holds or has stored; never runs the function. + + After a run (or ``execute``) the rows are the node's results in memory, the + first of what ``as_table`` returns. Otherwise they are the first of its stored + rows: the node's whole stored slice is read and nothing is kept on the node. + With nothing stored, there are no rows. See ``StreamBase.head``. + + Raises: + RuntimeError: If the node was loaded and the attached store does not hold + the records its saved run stored (the wrong store). + """ + from orcapod.pipeline.serialization import LoadStatus + + check_head_size(n) + config = head_column_config(columns, all_info) + status = self.load_status + if status == LoadStatus.UNAVAILABLE: + return self._head_without_rows( + config, HeadStatus.UNAVAILABLE, note="no store attached" + ) + current = status == LoadStatus.CACHE_ONLY or not self.is_stale + if current and self._cached_output_datas: + pairs = [ + (t, d) for t, d in self._cached_output_datas.values() if d is not None + ] + return self._head_of_pairs(pairs, n, config, origin="held") + return self._head_from_store(n, config) + + def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: + """Return the first ``n`` of this node's stored rows. + + Reads the node's whole stored slice (pipeline table joined with the result + table; no bounded read exists yet) and keeps nothing on the node. + """ + if self._function_pod is None: + # A loaded node: its table is built from the stored Arrow records. + self._require_expected_records() + fetched = self._fetch_joined_records() + stored = None if fetched is None else self._stored_rows_stream(fetched) + if stored is not None: + return ( + stored[1].head(n, columns=config).for_stream(self, origin="stored") + ) + return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + if self._cached_function_pod is None: + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + pairs = [ + (tag, data) + for tag, data in self._load_cached_entries().values() + if data is not None + ] + if not pairs: + return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + return self._head_of_pairs(pairs, n, config, origin="stored") + # ------------------------------------------------------------------ # Iteration # ------------------------------------------------------------------ diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 36dcc906e..df01553a5 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -27,7 +27,16 @@ from orcapod.config import OrcapodConfig from orcapod.core.operators.static_output_pod import StaticOutputOperatorPod from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import ( + BLUEPRINT_NOTE, + HeadStatus, + StreamHead, +) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( PipelineJobRequiredError, @@ -381,6 +390,21 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: return # pragma: no cover yield # pragma: no cover + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: a blueprint node holds no data.""" + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note=BLUEPRINT_NOTE, + ) + def as_table( self, *, @@ -1291,6 +1315,67 @@ def as_table( empty_stream = ArrowTableStream(empty_records, tag_columns=tag_keys) return empty_stream.as_table(columns=columns, all_info=all_info) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the first rows the node holds or has stored; never runs the operator. + + After a run (or ``execute``) the rows are the node's output in memory. Before + that, a node in ``CacheMode.REPLAY`` shows the first of its stored rows, read + from its whole stored table; in ``LOG`` and ``OFF`` it shows none, as + ``as_table`` does. A loaded node never raises here. See ``StreamBase.head``. + """ + check_head_size(n) + config = head_column_config(columns, all_info) + if self._cached_output_stream is not None: + return self._cached_output_stream.head(n, columns=config).for_stream(self) + if self._cache_mode == CacheMode.REPLAY and self._pipeline_database is not None: + return self._head_from_store(n, config) + if self._loaded: + return self._head_without_rows( + config, + HeadStatus.UNAVAILABLE, + note=( + "loaded node whose results are not stored (cache mode OFF)" + if self._cache_mode == CacheMode.OFF + else "loaded node whose stored results are not replayed " + f"(cache mode {self._cache_mode.value.upper()})" + ), + ) + if self._cache_mode == CacheMode.LOG and self._pipeline_database is not None: + note = ( + "not run in this session; rows of earlier runs are stored but not " + "replayed (cache mode LOG)" + ) + else: + note = "not run; cache mode OFF stores nothing" + return self._head_without_rows(config, HeadStatus.NOT_COMPUTED, note=note) + + def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: + """Return the first ``n`` of this node's stored rows. + + Reads the node's whole stored slice (no bounded read exists yet) and keeps + nothing on the node. + """ + records = self._read_stored_records() + if records is None or records.num_rows == 0: + return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + records = records.drop( + [ + c + for c in (constants.NODE_CONTENT_HASH_COL, self.HASH_COLUMN_NAME) + if c in records.column_names + ] + ) + stream = ArrowTableStream( + records, tag_columns=self.keys()[0], data_context=self.data_context + ) + return stream.head(n, columns=config).for_stream(self, origin="stored") + # ------------------------------------------------------------------ # DB retrieval # ------------------------------------------------------------------ diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index 4b42b81b4..d15727919 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -16,9 +16,19 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.base import TraceableBase +from orcapod.core.streams.head import ( + HeadStatus, + StreamHead, + render_error_html, +) +from orcapod.core.streams.base import ( + check_head_size, + head_column_config, +) from orcapod.errors import SourceSpecMismatchError, UnboundSourceError from orcapod.protocols.core_protocols import DataProtocol, TagProtocol from orcapod.types import ColumnConfig, Schema +from orcapod.utils import arrow_utils from orcapod.utils.arrow_utils import system_tag_column_names from orcapod.utils.schema_utils import compute_source_schema_hash @@ -291,6 +301,44 @@ def as_table( "Use PipelineJob.bind() to attach data before calling as_table()." ) + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return no rows: an input slot holds no data until a source is bound. + + See ``StreamBase.head``. + """ + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note="input slot; bind a source", + ) + + def _head_without_rows( + self, config: ColumnConfig, status: HeadStatus, note: str | None = None + ) -> StreamHead: + """Return a head with no rows: the slot's columns from its declared schema.""" + tag_schema, data_schema = self.output_schema(columns=config) + table = arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) + return StreamHead.without_rows( + self, status, table, tag_columns=tuple(tag_schema.keys()), note=note + ) + + def _repr_html_(self) -> str: + """Render ``head()`` for a notebook; never computes.""" + try: + head = self.head() + except Exception as error: # noqa: BLE001 - a display hook must not raise + return render_error_html(self, error) + return head._repr_html_() + async def async_iter_data(self): """Asynchronous iterator over (tag, data) pairs. @@ -570,6 +618,25 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: ) return self._bound_source.iter_data() + def head( + self, + n: int = 10, + *, + columns: ColumnConfig | dict[str, Any] | None = None, + all_info: bool = False, + ) -> StreamHead: + """Return the bound source's head, or no rows when no source is bound.""" + if self._bound_source is None: + check_head_size(n) + return self._head_without_rows( + head_column_config(columns, all_info), + HeadStatus.UNAVAILABLE, + note="no source bound", + ) + return self._bound_source.head( + n, columns=columns, all_info=all_info + ).for_stream(self) + async def async_iter_data( self, ) -> AsyncIterator[tuple[TagProtocol, DataProtocol]]: diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 4aa565d8b..83f40613b 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -11,13 +11,15 @@ from orcapod.config import OrcapodConfig from orcapod.contexts import DataContext from orcapod.core.base import TraceableBase -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) from orcapod.core.streams.head import ( STALE_NOTE, HeadStatus, StreamHead, - check_head_size, - head_column_config, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( diff --git a/src/orcapod/core/sources/base.py b/src/orcapod/core/sources/base.py index 278c89825..2464fe774 100644 --- a/src/orcapod/core/sources/base.py +++ b/src/orcapod/core/sources/base.py @@ -200,7 +200,10 @@ def head( slice of it. A source that reads its table later overrides this with a bounded read. See ``StreamBase.head``. """ - return self._stream.head(n, columns=columns, all_info=all_info).for_stream(self) + # _stream is set by each concrete subclass (see the class docstring), as for + # the other delegating methods here. + stream = self._stream # pyright: ignore[reportAttributeAccessIssue] + return stream.head(n, columns=columns, all_info=all_info).for_stream(self) # ------------------------------------------------------------------------- # Convenience — caching diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 8bb64e2bf..87182cdf6 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -10,6 +10,8 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, +) +from orcapod.core.streams.base import ( check_head_size, head_column_config, ) diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index cf9f57b2a..ada1875f0 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -23,6 +23,8 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, +) +from orcapod.core.streams.base import ( check_head_size, head_column_config, ) diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index 5dc501a8d..ed1c677e3 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -16,6 +16,8 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, +) +from orcapod.core.streams.base import ( check_head_size, head_column_config, ) diff --git a/src/orcapod/core/streams/arrow_table_stream.py b/src/orcapod/core/streams/arrow_table_stream.py index d7606e524..721f59e36 100644 --- a/src/orcapod/core/streams/arrow_table_stream.py +++ b/src/orcapod/core/streams/arrow_table_stream.py @@ -7,8 +7,12 @@ from orcapod import contexts from orcapod.core.datagrams import Data, Tag -from orcapod.core.streams.base import StreamBase -from orcapod.core.streams.head import StreamHead, check_head_size, head_column_config +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) +from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols import PodProtocol, StreamProtocol, TagProtocol from orcapod.protocols.hashing_protocols import PipelineElementProtocol from orcapod.system_constants import constants diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 6eb4e974f..b5803eaf0 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -1,5 +1,6 @@ from __future__ import annotations +import dataclasses import logging from abc import abstractmethod from collections.abc import AsyncIterator, Collection, Iterator, Mapping @@ -10,8 +11,6 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, - check_head_size, - head_column_config, render_error_html, ) from orcapod.protocols.core_protocols import ( @@ -40,6 +39,45 @@ logger = logging.getLogger(__name__) +def check_head_size(n: int) -> None: + """Check the row count given to ``head``. + + Raises: + TypeError: If ``n`` is not an ``int``. + ValueError: If ``n`` is negative. + """ + if isinstance(n, bool) or not isinstance(n, int): + raise TypeError(f"head() takes an int number of rows, got {type(n).__name__}") + if n < 0: + raise ValueError(f"head() takes a non-negative number of rows n, got {n}") + + +def head_column_config( + columns: ColumnConfig | dict[str, Any] | None, all_info: bool +) -> ColumnConfig: + """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. + + The column groups mean what they mean for ``as_table``. ``head`` returns rows in + stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. + + Raises: + ValueError: If ``columns`` asks for ``sort_by_tags``. + """ + if isinstance(columns, ColumnConfig): + asked_to_sort = bool(columns.sort_by_tags) + elif isinstance(columns, Mapping): + asked_to_sort = bool(columns.get("sort_by_tags", False)) + else: + asked_to_sort = False + if asked_to_sort: + raise ValueError( + "head() returns the first rows in stream order and cannot sort them by " + "tags; sort the result of as_table() instead" + ) + config = ColumnConfig.handle_config(columns, all_info=all_info) + return dataclasses.replace(config, sort_by_tags=False) + + class StreamBase(TraceableBase): @property @abstractmethod diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index 28bd28a56..ee6ec7e92 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -11,12 +11,10 @@ import dataclasses import html -from collections.abc import Mapping from dataclasses import dataclass from enum import Enum -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, cast -from orcapod.types import ColumnConfig from orcapod.utils.lazy_module import LazyModule if TYPE_CHECKING: @@ -52,6 +50,7 @@ class HeadStatus(str, Enum): NOT_COMPUTED_NOTE = "not computed; as_table(), iter_data() or a job run computes it" STALE_NOTE = "not computed: inputs changed since the last computation" NOTHING_STORED_NOTE = "nothing stored; run the job" +BLUEPRINT_NOTE = "pipeline blueprint; run it in a PipelineJob" SIDE_EFFECTS_NOTE = ( "not run; iterating it or a job run delivers its side effects; " "displaying never delivers side effects" @@ -76,45 +75,6 @@ class HeadStatus(str, Enum): _MAX_LIST_ITEMS = 5 -def check_head_size(n: int) -> None: - """Check the row count given to ``head``. - - Raises: - TypeError: If ``n`` is not an ``int``. - ValueError: If ``n`` is negative. - """ - if isinstance(n, bool) or not isinstance(n, int): - raise TypeError(f"head() takes an int number of rows, got {type(n).__name__}") - if n < 0: - raise ValueError(f"head() takes a non-negative number of rows n, got {n}") - - -def head_column_config( - columns: ColumnConfig | dict[str, Any] | None, all_info: bool -) -> ColumnConfig: - """Return the column configuration ``head`` uses for ``columns`` and ``all_info``. - - The column groups mean what they mean for ``as_table``. ``head`` returns rows in - stream order, so ``all_info`` does not sort, and asking for ``sort_by_tags`` raises. - - Raises: - ValueError: If ``columns`` asks for ``sort_by_tags``. - """ - if isinstance(columns, ColumnConfig): - asked_to_sort = bool(columns.sort_by_tags) - elif isinstance(columns, Mapping): - asked_to_sort = bool(columns.get("sort_by_tags", False)) - else: - asked_to_sort = False - if asked_to_sort: - raise ValueError( - "head() returns the first rows in stream order and cannot sort them by " - "tags; sort the result of as_table() instead" - ) - config = ColumnConfig.handle_config(columns, all_info=all_info) - return dataclasses.replace(config, sort_by_tags=False) - - @dataclass(frozen=True) class StreamHead: """At most ``n`` readily available rows of a stream, and how they were obtained. @@ -325,7 +285,7 @@ def _size(count: int) -> str: return f"{count} B" # pragma: no cover -def _summarise(column: pa.ChunkedArray) -> list[str | None] | None: +def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: """Return display strings for a column whose cells are too large to show as they are, or ``None`` to show the column as it is.""" arrow_type = column.type @@ -354,7 +314,7 @@ def _summarise(column: pa.ChunkedArray) -> list[str | None] | None: def _summarise_extension( - column: pa.ChunkedArray, arrow_type: pa.ExtensionType + column: pa.ChunkedArray[Any], arrow_type: pa.ExtensionType ) -> list[str | None] | None: storage_type = arrow_type.storage_type short_name = arrow_type.extension_name.rsplit(".", 1)[-1] @@ -363,7 +323,12 @@ def _summarise_extension( # Only the small fields are read; an ndarray's ``data`` is never decoded. wanted = [n for n in ("dtype", "shape") if n in names] values = [ - v for array in column.chunks for v in _struct_fields(array.storage, wanted) + v + for array in column.chunks + for v in _struct_fields( + cast(pa.StructArray, cast("pa.ExtensionArray[Any]", array).storage), + wanted, + ) ] if {"dtype", "shape"} <= names: return [ @@ -377,12 +342,14 @@ def _summarise_extension( return [ None if v is None else f"{short_name} ({_size(len(v))})" for array in column.chunks - for v in array.storage.to_pylist() + for v in cast("pa.ExtensionArray[Any]", array).storage.to_pylist() ] return None -def _struct_fields(array: pa.StructArray, names: list[str]) -> list[dict | None]: +def _struct_fields( + array: pa.StructArray, names: list[str] +) -> list[dict[str, Any] | None]: """Return each row of ``array`` as a dict of the named fields (``None`` for a null row).""" fields = {name: array.field(name).to_pylist() for name in names} valid = array.is_valid().to_pylist() @@ -400,7 +367,7 @@ def _shape_text(shape: Any) -> str: def _summarise_list( - column: pa.ChunkedArray, value_type: pa.DataType + column: pa.ChunkedArray[Any], value_type: pa.DataType ) -> list[str | None] | None: simple_items = not ( isinstance(value_type, pa.ExtensionType) diff --git a/src/orcapod/side_effects.py b/src/orcapod/side_effects.py index b64a1cd43..2388dfa67 100644 --- a/src/orcapod/side_effects.py +++ b/src/orcapod/side_effects.py @@ -17,13 +17,15 @@ from orcapod.core.base import TraceableBase from orcapod.core.nodes.function_node import _build_record_id_preimage -from orcapod.core.streams.base import StreamBase +from orcapod.core.streams.base import ( + StreamBase, + check_head_size, + head_column_config, +) from orcapod.core.streams.head import ( SIDE_EFFECTS_NOTE, HeadStatus, StreamHead, - check_head_size, - head_column_config, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( diff --git a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py index b4f1d97d9..caff63dcc 100644 --- a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py +++ b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py @@ -11,6 +11,7 @@ import pyarrow as pa from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream from orcapod.core.operators import SelectDataColumns from orcapod.core.sources import ArrowTableSource @@ -36,6 +37,7 @@ def _source() -> ArrowTableSource: def test_operator_stream_reruns_after_upstream_is_modified() -> None: fn_stream = double.pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) out = SelectDataColumns(["y"])(fn_stream) original = SelectDataColumns.static_process # A spy on the non-public static_process: the test is about whether it runs again. diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 8da1bd180..7ab6a172e 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -15,12 +15,14 @@ import pyarrow as pa import pytest -from orcapod import function_pod, side_effect_pod +from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream from orcapod.core.operators import GroupBy, Join, PolarsFilter, SelectDataColumns from orcapod.core.sources import ArrowTableSource, CachedSource, DictSource, ListSource from orcapod.core.sources.polling_source import PollingSource from orcapod.databases import InMemoryArrowDatabase from orcapod.streams import HeadStatus, StreamHead +from orcapod.side_effects import SideEffectPod from orcapod.types import ColumnConfig, Cursor ALL_COLUMNS = ColumnConfig( @@ -268,6 +270,7 @@ def test_an_empty_operator_result_is_not_not_computed() -> None: def test_operator_stream_with_changed_inputs_is_not_computed() -> None: calls: list[int] = [] fn_stream = _counting_pod(calls).pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) out = SelectDataColumns(["y"])(fn_stream) _ = out.as_table() fn_stream.clear_cache() @@ -284,11 +287,10 @@ def test_operator_stream_with_changed_inputs_is_not_computed() -> None: def test_side_effect_stream_preview_never_delivers() -> None: delivered: list[int] = [] - @side_effect_pod def deliver(x: int) -> None: delivered.append(x) - stream = deliver(_source()) + stream = SideEffectPod(deliver)(_source()) head = stream.head() html = stream._repr_html_() assert delivered == [] diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 4d1f7bd40..3f573fb58 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -23,7 +23,7 @@ from orcapod.pipeline.sync_orchestrator import SyncPipelineOrchestrator from orcapod.protocols.database_protocols import ArrowDatabaseProtocol from orcapod.streams import HeadStatus -from orcapod.types import CacheMode +from orcapod.types import CacheMode, Schema CALLS: list[int] = [] @@ -149,10 +149,14 @@ def test_blueprint_nodes_are_unavailable() -> None: def test_source_nodes() -> None: - slot = SourceNode(name="nums", tag_schema={"id": int}, data_schema={"x": int}) + slot = SourceNode( + name="nums", tag_schema=Schema({"id": int}), data_schema=Schema({"x": int}) + ) assert slot.head().status is HeadStatus.UNAVAILABLE assert "input slot" in slot._repr_html_() - unbound = SourceJobNode(name="nums", tag_schema={"id": int}, data_schema={"x": int}) + unbound = SourceJobNode( + name="nums", tag_schema=Schema({"id": int}), data_schema=Schema({"x": int}) + ) assert unbound.head().status is HeadStatus.UNAVAILABLE assert "no source bound" in unbound.head().header() source = _source() From 6542dd43f022956fbaa9b48952eb9b36a0aa53e7 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:15:09 -0800 Subject: [PATCH 09/19] docs: looking at data with head() and the preview; changelog and design issue (ITL-991) Co-Authored-By: Claude Opus 5.5 --- changelog.d/ITL-991.md | 30 ++++++++ design_issues/ITL-991-PV1.md | 22 ++++++ docs/concepts/streams.md | 71 +++++++++++++++++++ .../plans/2026-11-12-itl-991-stream-head.md | 4 ++ ...-itl-991-stream-head-and-preview-design.md | 18 ++++- 5 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 changelog.d/ITL-991.md create mode 100644 design_issues/ITL-991-PV1.md diff --git a/changelog.d/ITL-991.md b/changelog.d/ITL-991.md new file mode 100644 index 000000000..21830317a --- /dev/null +++ b/changelog.d/ITL-991.md @@ -0,0 +1,30 @@ +### Breaking changes and migration + +#### Showing a stream never computes it (ITL-991) + +Displaying a stream, source or node in a notebook no longer computes anything. The preview +shows the first 10 rows that are already available: rows a source holds, rows cached from an +earlier computation, rows a job node holds after a run or has stored, or a bounded read of a +source's table. Otherwise it shows the columns and a note such as "not computed" or "nothing +stored; run the job". Call `as_table()` or run the job to compute. Before, displaying a +function pod's output called the function on every row, displaying an operator's output ran +the operator, displaying a side-effect stream delivered every side effect, and displaying a +`CachedSource` wrote its cache table; none of these happen now. + +`StreamBase.view()` and `StreamView` are removed: use `head(n)` or `as_polars_df()`. + +### Added + +- `head(n=10, columns=..., all_info=...)` on every stream, source and node returns a + `StreamHead` (`orcapod.streams`): at most `n` rows as a PyArrow table with their exact + types, the row count when it is known without computing, and a `status` (`HeadStatus`) that + tells an empty result from one that is not computed. It never calls a function, runs an + operator, delivers a side effect, polls a source or writes. The notebook preview is + `head()` rendered, with large cells (arrays, bytes, long text, long lists) summarised. A job + node with nothing in memory reads its whole stored slice to show its first rows. (ITL-991) + +### Fixed + +- An operator's output stream computed again after one of its inputs changed now does so: + before, it served the result kept from before the change (`DynamicPodStream` cleared a + misspelled attribute instead of its kept result). (ITL-991) diff --git a/design_issues/ITL-991-PV1.md b/design_issues/ITL-991-PV1.md new file mode 100644 index 000000000..692a842de --- /dev/null +++ b/design_issues/ITL-991-PV1.md @@ -0,0 +1,22 @@ +## `src/orcapod/core/streams/` — notebook preview and `head()` + +### PV1 — Showing a stream computed it, delivered side effects and wrote caches +**Status:** resolved +**Severity:** high +**Issue:** ITL-991 + +`StreamBase._repr_html_` and `StreamBase.view()` called `as_polars_df()` on the whole +stream. Displaying a function pod's output in a notebook called the function on every row, +an operator's output ran the operator, a `SideEffectPodStream` or `SideEffectNode` delivered +every side effect, a `PollingSource` polled, a `CachedSource` wrote its cache table, and a +blueprint node raised inside the formatter. The developer ruled (November 12, 2026) that a +preview shows only what is readily available and never triggers computation, and that a +node's preview is a head of its storage. + +**Fix:** every stream, source and node has `head(n=10, columns=..., all_info=...)` +returning a `StreamHead` (rows, `total_rows`, `HeadStatus`), and `_repr_html_` renders +`head()`. The default shows the columns and `NOT_COMPUTED`; only classes that can serve rows +without computing override it. `view()` and `StreamView` are removed. A job node with nothing +in memory reads its whole stored slice; a bounded database read is a follow-up. The same +work fixed `DynamicPodStream` serving a stale result after an input changed (misspelled +`_cached_results` in `_update_cache_status`). diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index feb7ec058..ebcee55ad 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -100,6 +100,77 @@ df = table.to_pandas() Streams also have direct shortcuts: `as_pandas_df()`, `as_polars_df()` and `as_lazy_frame()`. +## Looking at data + +In a notebook, a stream, source or job node shown at the end of a cell renders a short +table: its first rows, with a line saying what they are. That preview is `head()` rendered, +and `head(n)` gives the same rows to work with. Neither ever computes anything: they show +only what is readily available, and never call a function, run an operator, deliver a side +effect, poll a source or write to a database. + +```python +from orcapod import function_pod +from orcapod.sources import DictSource + +subjects = DictSource( + data=[ + {"subject_id": f"mouse_{i:02d}", "age": 8 + i, "weight": 20.0 + i} + for i in range(1, 26) + ], + tag_columns=["subject_id"], + label="subjects", +) + +head = subjects.head() +print(head.header()) +# DictSource[subjects] — first 10 of 25 rows +print(head.status.value, len(head)) +# rows 10 +print(subjects.head(3).as_table().column("age").to_pylist()) +# [9, 10, 11] +``` + +`head()` returns a `StreamHead`: the rows as a PyArrow table (`.table`, `.as_table()`, +`.as_polars_df()`), the row count when it is known without computing (`.total_rows`), and a +`.status` that says where the rows came from. A stream downstream of a computation shows rows +only once they are computed: + +```python +@function_pod(output_keys="age_months") +def to_months(age: int) -> int: + return age * 12 + + +months = to_months.pod(subjects, label="months") +print(months.head().header()) +# FunctionPodStream[months] — not computed; as_table(), iter_data() or a job run computes it +print(months.head().status.value, months.head().table.column_names) +# not_computed ['subject_id', 'age_months'] + +table = months.as_table() # computes every row +print(months.head(3).header()) +# FunctionPodStream[months] — first 3 of 25 computed rows +``` + +What each kind of object shows: + +| Object | `head()` shows | +|---|---| +| A source | its first rows (a bounded read for a source that reads its table lazily) | +| An operator's or function pod's output stream | the rows already computed, or none with status `not_computed` | +| A job node | the rows it holds after `job.run()`, else the first of its stored rows, else none (`nothing_stored`) | +| A pipeline blueprint node, an unbound input slot | none (`unavailable`) | +| A side-effect stream | none: showing it never delivers its side effects | + +A head of an operator is never the operator applied to the heads of its inputs: a group over +the first ten rows has the wrong members. So an uncomputed stream shows its columns and +"not computed" until `as_table()`, `iter_data()` or a job run computes it. An empty result +shows status `rows` with zero rows, so it is never mistaken for "not computed". + +In the preview, large cells are summarised (an array as `ndarray float64 (2, 3)`, bytes as +`bytes (300 B)`, long text and long lists shortened); `head().table` keeps the full values. +In a terminal, `print(stream.head())` prints the same preview as text. + ## Controlling column visibility with `ColumnConfig` By default, streams only expose user-facing tag and data columns. Orcapod also maintains diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md index c2f485781..c8085ea39 100644 --- a/superpowers/plans/2026-11-12-itl-991-stream-head.md +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -176,3 +176,7 @@ Header wording (exact; #333 depends on the estimate line): - 2026-11-12: spec 70639010 pushed; design ask answered (D3, D5 approved; D1/D2/D4 pending with developer). Plan saved. +- 2026-11-12 (later): D1, D2, D4 approved by the developer. Tasks 1-7 implemented and + pushed: F1 fix, StreamHead, sources, operator/function streams, cached pod stream, nodes, + docs ("Looking at data" in docs/concepts/streams.md), changelog fragment, design issue + ITL-991-PV1. Next: full suite, merge main, PR, CodeRabbit. diff --git a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md index 05ec1ae28..30b0e7466 100644 --- a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md +++ b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md @@ -1,6 +1,7 @@ # ITL-991: `head()` and a notebook preview that never computes -Status: design, awaiting developer approval (November 12, 2026). +Status: approved (developer, November 12, 2026: "agreed with all 3 recs"; D3 and D5 approved by the coordinator). +Implementation notes at the end record where the code differs from the text below. Issue: ITL-991. Related: ITL-186 / PR #333 (Spiral source), ITL-990 (workshop notebook). ## Rulings this design implements (developer, November 12, 2026) @@ -367,3 +368,18 @@ renders `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349`. earlier runs are stored but not replayed (recommended, matches `as_table`), or show them. - **D4** Remove `view()` / `StreamView` (recommended) or keep `view()` as an alias of `head`. - **D5** Fix F1 in this PR (recommended) or file it separately. + +## Implementation notes (November 12, 2026) + +- Tag columns come first; the line above the table names them (`tags: id, grp`). Polars' + HTML shows each column's dtype, so there is no separate schema line. +- Struct cells are left to Polars' rendering (the per-field rules are not applied). +- A loaded `FunctionJobNode` whose attached store lacks the records its saved run stored (the + wrong store) still raises from `head()`; the preview shows that error as "could not read". + Every other loaded-node state returns a head without raising. +- `check_head_size` and `head_column_config` live in `core/streams/base.py`, and `StreamHead` + / `HeadStatus` in `core/streams/head.py`, which imports nothing from `orcapod.types`: that + keeps `StreamProtocol.head` free of a new import cycle. +- `head()` is part of `StreamProtocol`. +- `_CachedFunctionPodStream` (added on main by ITL-988) drops stored meta columns in head as + in `as_table`. From c392b995723987b15c41bb8c6de38e3ac367c441 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:15:32 -0800 Subject: [PATCH 10/19] test(nodes): a node's head right after a run reads nothing from the store (ITL-991) Co-Authored-By: Claude Opus 5.5 --- tests/test_pipeline/test_node_head.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 3f573fb58..80d34786e 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -8,6 +8,7 @@ from __future__ import annotations from pathlib import Path +from unittest import mock import pyarrow as pa import pytest @@ -90,12 +91,17 @@ def test_nodes_before_a_run_show_no_rows_and_compute_nothing() -> None: def test_nodes_after_a_run_show_their_first_rows( orchestrator: str, store_kind: str, tmp_path: Path ) -> None: - job = _job(_store(store_kind, tmp_path)) + store = _store(store_kind, tmp_path) + job = _job(store) _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) assert len(CALLS) == 25 for label in ("triple", "sel"): node = job.nodes[label] - head = node.head() + # Right after a run the rows are in memory: the store is not read at all. + with mock.patch.object( + type(store), "get_all_records", side_effect=AssertionError("store read") + ): + head = node.head() assert head.status is HeadStatus.ROWS, label assert head.total_rows == 25 assert head.table.equals(node.as_table().slice(0, 10)), label From 5fe6d961bbc100f6f1d04888f1fc7ef47941f9c6 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:44:33 -0800 Subject: [PATCH 11/19] fix(streams): address CodeRabbit round 1 on head() (ITL-991) - FunctionPodStream.head collects only the shown rows of its cache and reports PARTIAL when inputs were processed but gave no rows yet. - FunctionJobNode.head leaves out EmptyData placeholders (results of a lost ephemeral store) and says how many. - OperatorJobNode.head reads its store through the as_table path, so a loaded REPLAY node on the wrong store reports that error. - DerivedSource.head asks a source origin for its head instead of reading the whole origin (which could poll or write a cache). - PollingSource.head includes declared system tags when asked. - Cell summaries take lengths from Arrow; only short values become Python objects. - A rowless head no longer reads its schema when that would poll a source upstream (schema_requires_io). - Tests: no operator entry point runs before a job run; docstrings. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/cached_function_pod.py | 10 +-- src/orcapod/core/function_pod.py | 71 ++++++++++------- src/orcapod/core/nodes/function_node.py | 43 ++++++---- src/orcapod/core/nodes/operator_node.py | 22 ++---- src/orcapod/core/nodes/source_node.py | 5 ++ src/orcapod/core/sources/cached_source.py | 5 ++ src/orcapod/core/sources/derived_source.py | 78 ++++++++++++++----- src/orcapod/core/sources/polling_source.py | 16 +++- src/orcapod/core/streams/base.py | 33 +++++++- src/orcapod/core/streams/head.py | 47 +++++++---- .../test_dynamic_pod_stream_staleness.py | 1 + .../streams/test_head_and_preview.py | 77 +++++++++++++++++- tests/test_pipeline/test_node_head.py | 27 ++++++- 13 files changed, 325 insertions(+), 110 deletions(-) diff --git a/src/orcapod/core/cached_function_pod.py b/src/orcapod/core/cached_function_pod.py index 25202b7b4..b1b27c562 100644 --- a/src/orcapod/core/cached_function_pod.py +++ b/src/orcapod/core/cached_function_pod.py @@ -456,11 +456,11 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: for tag, data in super().iter_data(): yield tag, _without_stored_meta(data) - def _computed_prefix(self) -> list[tuple[TagProtocol, DataProtocol]]: - return [ - (tag, _without_stored_meta(data)) - for tag, data in super()._computed_prefix() - ] + def _computed_prefix( + self, limit: int + ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: + shown, total = super()._computed_prefix(limit) + return [(tag, _without_stored_meta(data)) for tag, data in shown], total def head( self, diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 7e3915ff3..910f1c97a 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1158,18 +1158,18 @@ def head( return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE ) - tag_columns = self.keys(columns=config)[0] - - computed = self._computed_prefix() - shown = computed[:n] - - def shown_content_hashes() -> pa.Array[Any]: - return pa.array( - [data.content_hash().to_string() for _, data in shown], - type=pa.large_string(), - ) - + # Rows exist only once the input was read, so the schema is known without I/O. + tag_columns = self.keys(columns=config)[0] if has_results else () if self._cached_output_table is not None: + # A complete computation: its table holds the rows; only the shown rows' + # data are read, and only when their content hashes are asked for. + def shown_content_hashes() -> pa.Array[Any]: + shown, _ = self._computed_prefix(n) + return pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ) + return StreamHead.of_rows( self, self._shape_table( @@ -1180,42 +1180,59 @@ def shown_content_hashes() -> pa.Array[Any]: origin="computed", ) finished = not self._needs_iterator and self._cached_input_iterator is None - if not finished and not computed: + if not finished and not self._cached_output_datas: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) + shown, total = self._computed_prefix(n) + status = HeadStatus.ROWS if finished else HeadStatus.PARTIAL if not shown: - # Nothing to show (an empty result, or n == 0): the declared columns. + # Nothing to show (an empty result, inputs that all gave no output so far, + # or n == 0): the declared columns. empty = self._head_without_rows(config, HeadStatus.ROWS) return dataclasses.replace( - empty, - status=HeadStatus.ROWS if finished else HeadStatus.PARTIAL, - total_rows=len(computed), - origin="computed", + empty, status=status, total_rows=total, origin="computed" ) return StreamHead.of_rows( self, self._shape_table( - self._table_from_pairs(shown), config, shown_content_hashes + self._table_from_pairs(shown), + config, + lambda: pa.array( + [data.content_hash().to_string() for _, data in shown], + type=pa.large_string(), + ), ), tag_columns=tag_columns, - total_rows=len(computed), - status=HeadStatus.ROWS if finished else HeadStatus.PARTIAL, + total_rows=total, + status=status, origin="computed", ) - def _computed_prefix(self) -> list[tuple[TagProtocol, DataProtocol]]: - """Return the output rows computed so far, in order, as iteration yields them. + def _computed_prefix( + self, limit: int + ) -> tuple[list[tuple[TagProtocol, DataProtocol]], int]: + """Return the first ``limit`` output rows computed so far, and how many there are. + + The rows are the contiguous run of computed inputs from the first, in the order + iteration yields them; an input whose output is ``None`` (filtered out) is + skipped, as iteration skips it. Only the first ``limit`` rows are collected. - They are the contiguous run of computed inputs from the first; an input whose - output is ``None`` (filtered out) is skipped, as iteration skips it. + Args: + limit: The largest number of rows to return. + + Returns: + The rows, and the number of output rows in the whole computed run. """ - computed: list[tuple[TagProtocol, DataProtocol]] = [] + shown: list[tuple[TagProtocol, DataProtocol]] = [] + total = 0 index = 0 while index in self._cached_output_datas: tag, data = self._cached_output_datas[index] if data is not None: - computed.append((tag, data)) + if total < limit: + shown.append((tag, data)) + total += 1 index += 1 - return computed + return shown, total def _table_from_pairs( self, pairs: list[tuple[TagProtocol, DataProtocol]] diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index ebc0d908a..6f47e95f3 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -21,7 +21,7 @@ import logging import uuid from abc import abstractmethod -from collections.abc import Callable, Iterator, Mapping +from collections.abc import Callable, Iterator, Mapping, Sequence from typing import TYPE_CHECKING, Any, Literal, NamedTuple, cast from orcapod import contexts @@ -573,13 +573,22 @@ def head( def _head_of_pairs( self, - pairs: list[tuple[TagProtocol, DataProtocol]], + pairs: "Sequence[tuple[TagProtocol, DataProtocol | None]]", n: int, config: ColumnConfig, origin: str, ) -> StreamHead: - """Return the first ``n`` of ``pairs`` as this node's head.""" - shown = pairs[:n] + """Return the first ``n`` of ``pairs`` as this node's head. + + A ``None`` output (a filtered input) is not a row. An ``EmptyData`` placeholder + (a result kept in an ephemeral store that is gone) has no values to show: it is + left out and counted in the note. + """ + rows = [ + (t, d) for t, d in pairs if d is not None and not isinstance(d, EmptyData) + ] + missing = sum(1 for _, d in pairs if isinstance(d, EmptyData)) + shown = rows[:n] table = self._shape_table( self._table_from_pairs(shown), config, @@ -588,12 +597,19 @@ def _head_of_pairs( type=pa.large_string(), ), ) + note = None + if missing: + note = ( + f"{missing:,} {'result' if missing == 1 else 'results'} not shown: " + "kept in an ephemeral store that is gone" + ) return StreamHead.of_rows( self, table, tag_columns=self.keys(columns=config)[0], - total_rows=len(pairs), + total_rows=len(rows), origin=origin, + note=note, ) def __repr__(self) -> str: @@ -2953,10 +2969,9 @@ def head( ) current = status == LoadStatus.CACHE_ONLY or not self.is_stale if current and self._cached_output_datas: - pairs = [ - (t, d) for t, d in self._cached_output_datas.values() if d is not None - ] - return self._head_of_pairs(pairs, n, config, origin="held") + return self._head_of_pairs( + list(self._cached_output_datas.values()), n, config, origin="held" + ) return self._head_from_store(n, config) def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: @@ -2977,14 +2992,10 @@ def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: return self._head_without_rows(config, HeadStatus.NOTHING_STORED) if self._cached_function_pod is None: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) - pairs = [ - (tag, data) - for tag, data in self._load_cached_entries().values() - if data is not None - ] - if not pairs: + entries = list(self._load_cached_entries().values()) + if not any(data is not None for _, data in entries): return self._head_without_rows(config, HeadStatus.NOTHING_STORED) - return self._head_of_pairs(pairs, n, config, origin="stored") + return self._head_of_pairs(entries, n, config, origin="stored") # ------------------------------------------------------------------ # Iteration diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index df01553a5..28b39643c 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1358,22 +1358,16 @@ def head( def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: """Return the first ``n`` of this node's stored rows. - Reads the node's whole stored slice (no bounded read exists yet) and keeps - nothing on the node. + Reads the node's whole stored slice (no bounded read exists yet) through the + same path as ``as_table`` and keeps nothing on the node. + + Raises: + RuntimeError: If the node was loaded and its cache table is absent from + the attached store (the wrong store). """ - records = self._read_stored_records() - if records is None or records.num_rows == 0: + stream = self._load_cached_stream_from_db() + if stream is None or stream.as_table().num_rows == 0: return self._head_without_rows(config, HeadStatus.NOTHING_STORED) - records = records.drop( - [ - c - for c in (constants.NODE_CONTENT_HASH_COL, self.HASH_COLUMN_NAME) - if c in records.column_names - ] - ) - stream = ArrowTableStream( - records, tag_columns=self.keys()[0], data_context=self.data_context - ) return stream.head(n, columns=config).for_stream(self, origin="stored") # ------------------------------------------------------------------ diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index d15727919..277c17c5d 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -16,6 +16,7 @@ from orcapod import contexts from orcapod.config import OrcapodConfig from orcapod.core.base import TraceableBase +from orcapod.core.streams.base import schema_requires_io from orcapod.core.streams.head import ( HeadStatus, StreamHead, @@ -618,6 +619,10 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: ) return self._bound_source.iter_data() + def _schema_requires_io(self) -> bool: + """Whether reading the schema would poll: as for the bound source.""" + return self._bound_source is not None and schema_requires_io(self._bound_source) + def head( self, n: int = 10, diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index 1fe741da2..807894516 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -8,6 +8,7 @@ from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream +from orcapod.core.streams.base import schema_requires_io from orcapod.core.streams.head import StreamHead from orcapod.contexts.stored import ( keep_rows_of_data_context, @@ -406,6 +407,10 @@ def as_table( assert self._cached_stream is not None return self._cached_stream.as_table(columns=columns, all_info=all_info) + def _schema_requires_io(self) -> bool: + """Whether reading the schema would poll: as for the wrapped source.""" + return schema_requires_io(self._source) + def head( self, n: int = 10, diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 87182cdf6..b93497415 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -1,5 +1,6 @@ from __future__ import annotations +import dataclasses from collections.abc import Mapping from typing import TYPE_CHECKING, Any, ClassVar @@ -14,6 +15,7 @@ from orcapod.core.streams.base import ( check_head_size, head_column_config, + schema_requires_io, ) from orcapod.errors import InputValidationError from orcapod.system_constants import constants @@ -395,6 +397,17 @@ def _build_derived_stream( has_rows = table is not None if table is None: table = self._empty_origin_table() + return self._derive_from(table, tags_as_data), has_rows + + def _derive_from( + self, table: "pa.Table", tags_as_data: Mapping[str, str] + ) -> ArrowTableStream: + """Append the tag copies to the origin rows ``table`` and enrich it as a source. + + Args: + table: Rows of the origin. + tags_as_data: Maps a tag column of the origin to the name of its copy. + """ for tag, name in tags_as_data.items(): # The copy is the tag's own Arrow column under a renamed field, so its # type, nullability and field metadata are the tag's exactly. @@ -408,7 +421,7 @@ def _build_derived_stream( source_id=self.source_id, record_id_column=self._record_id_column, ) - return result.stream, has_rows + return result.stream def _get_stream(self) -> ArrowTableStream: if self._tags_as_data is not None: @@ -430,6 +443,10 @@ def _get_stream(self) -> ArrowTableStream: tag_keys = self._origin.keys()[0] return ArrowTableStream(self._cached_table, tag_columns=tag_keys) + def _schema_requires_io(self) -> bool: + """Whether reading the schema would poll: as for the origin.""" + return schema_requires_io(self._origin) + def head( self, n: int = 10, @@ -439,27 +456,38 @@ def head( ) -> StreamHead: """Return the first rows of the origin's records, without computing. - The records are the ones this source already read, or else the origin's - records read now (a job node's stored rows) and not kept. An origin job node - that has not run gives no rows. + The records are the ones this source already read. Otherwise, an origin source + gives its own ``head(n)`` (so a polling or caching origin is never polled or + written), and an origin job node gives its stored rows, read now and not kept. + An origin job node that has not run gives no rows. """ check_head_size(n) config = head_column_config(columns, all_info) - if self._tags_as_data is not None: - if self._cached_stream is not None: - stream, has_rows = self._cached_stream, True - else: - stream, has_rows = self._build_derived_stream(self._tags_as_data) - else: - table = self._cached_table - if table is None: - table = self._origin_records() - has_rows = table is not None and table.num_rows > 0 - if table is None: - table = self._empty_origin_table() - stream = ArrowTableStream(table, tag_columns=self._origin.keys()[0]) - origin = "held" if isinstance(self._origin, RootSource) else "stored" - if not has_rows and origin == "stored": + origin_kind = "held" if isinstance(self._origin, RootSource) else "stored" + if self._tags_as_data is not None and self._cached_stream is not None: + return self._cached_stream.head(n, columns=config).for_stream( + self, origin=origin_kind + ) + if self._tags_as_data is None and self._cached_table is not None: + stream = ArrowTableStream( + self._cached_table, tag_columns=self._origin.keys()[0] + ) + return stream.head(n, columns=config).for_stream(self, origin=origin_kind) + if isinstance(self._origin, RootSource): + origin_head = self._origin.head(n) + if origin_head.status is not HeadStatus.ROWS: + return origin_head.for_stream(self) + stream = self._stream_over(origin_head.table) + return dataclasses.replace( + stream.head(n, columns=config).for_stream(self), + total_rows=origin_head.total_rows, + total_is_estimate=origin_head.total_is_estimate, + origin=origin_head.origin, + note=origin_head.note, + ) + records = self._origin_records() + if records is None or records.num_rows == 0: + stream = self._stream_over(self._empty_origin_table()) return StreamHead.without_rows( self, HeadStatus.NOTHING_STORED, @@ -467,7 +495,17 @@ def head( tag_columns=stream.keys(columns=config)[0], note="nothing stored: its origin has not run", ) - return stream.head(n, columns=config).for_stream(self, origin=origin) + return ( + self._stream_over(records) + .head(n, columns=config) + .for_stream(self, origin="stored") + ) + + def _stream_over(self, table: "pa.Table") -> ArrowTableStream: + """Return this source's stream over the origin rows ``table``.""" + if self._tags_as_data is not None: + return self._derive_from(table, self._tags_as_data) + return ArrowTableStream(table, tag_columns=self._origin.keys()[0]) def iter_data(self): return self._get_stream().iter_data() diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index ada1875f0..d9c89cbc2 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -444,6 +444,12 @@ def keys( self._sync_poll_and_commit() return self._get_combined_stream().keys(columns=columns, all_info=all_info) + def _schema_requires_io(self) -> bool: + """True while no schema was declared and nothing was polled: only a poll + tells the columns then.""" + no_schema = self._tag_schema is None or self._data_schema is None + return no_schema and not self._batches + def head( self, n: int = 10, @@ -461,11 +467,15 @@ def head( if self._batches: return self._get_combined_stream().head(n, columns=config).for_stream(self) if self._tag_schema is not None and self._data_schema is not None: + # The declared schema answers without a fetch, system tags included; the + # column groups that need data (source info, context, ...) are left out. + tag_schema, data_schema = self.output_schema( + columns=ColumnConfig(system_tags=config.system_tags) + ) table = arrow_utils.make_empty_table( - {**self._tag_schema, **self._data_schema}, - self.data_context.type_converter, + {**tag_schema, **data_schema}, self.data_context.type_converter ) - tag_columns = tuple(self._tag_schema.keys()) + tag_columns = tuple(tag_schema.keys()) else: table = pa.table({}) tag_columns = () diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index b5803eaf0..26f50b7d7 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -9,8 +9,10 @@ from orcapod.core.base import TraceableBase from orcapod.core.streams.head import ( + SCHEMA_NEEDS_POLL_NOTE, HeadStatus, StreamHead, + default_note, render_error_html, ) from orcapod.protocols.core_protocols import ( @@ -39,6 +41,15 @@ logger = logging.getLogger(__name__) +def schema_requires_io(stream: object) -> bool: + """Whether reading ``stream``'s schema would read or poll a source. + + A stream that does not say (``_schema_requires_io``) is taken not to. + """ + check = getattr(stream, "_schema_requires_io", None) + return bool(check()) if callable(check) else False + + def check_head_size(n: int) -> None: """Check the row count given to ``head``. @@ -530,7 +541,19 @@ def _head_without_rows( status: HeadStatus, note: str | None = None, ) -> StreamHead: - """Return a head with no rows: the stream's columns from its declared schema.""" + """Return a head with no rows: the stream's columns from its declared schema. + + When reading the schema would poll a source (``schema_requires_io``), the head + has no columns either, and the note says so. + """ + if schema_requires_io(self): + return StreamHead.without_rows( + self, + status, + pa.table({}), + tag_columns=(), + note=f"{note or default_note(status)}; {SCHEMA_NEEDS_POLL_NOTE}", + ) tag_schema, data_schema = self.output_schema(columns=config) table = arrow_utils.make_empty_table( {**tag_schema, **data_schema}, self.data_context.type_converter @@ -539,6 +562,14 @@ def _head_without_rows( self, status, table, tag_columns=tuple(tag_schema.keys()), note=note ) + def _schema_requires_io(self) -> bool: + """Whether reading this stream's schema would read or poll a source. + + True when an upstream's would: a ``PollingSource`` that declared no schema and + has not polled yet answers its schema only by polling. + """ + return any(schema_requires_io(upstream) for upstream in self.upstreams) + def _repr_html_(self) -> str: """Render ``head()`` for a notebook; never computes.""" try: diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index ee6ec7e92..d7751e049 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -62,6 +62,14 @@ class HeadStatus(str, Enum): HeadStatus.UNAVAILABLE: "no rows available", } +SCHEMA_NEEDS_POLL_NOTE = "columns not known until its source is polled" + + +def default_note(status: HeadStatus) -> str: + """Return the note shown for a head with ``status`` and no rows of its own.""" + return _DEFAULT_NOTES[status] + + # Words naming the rows of a ``ROWS`` head, by where they came from. _ORIGIN_NOUNS = { "held": "rows", @@ -292,17 +300,18 @@ def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: if isinstance(arrow_type, pa.ExtensionType): return _summarise_extension(column, arrow_type) if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): - return [ - None if v is None else f"bytes ({_size(len(v))})" - for v in column.to_pylist() - ] + return _byte_summaries(column, "bytes") if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): - values = column.to_pylist() - if all(v is None or len(v) <= _MAX_TEXT for v in values): + # Lengths come from Arrow; only the shortened text becomes Python strings. + lengths: list[int | None] = pc.utf8_length(column).to_pylist() + if all(n is None or n <= _MAX_TEXT for n in lengths): return None + heads: list[str | None] = pc.utf8_slice_codeunits( + column, 0, _MAX_TEXT - 3 + ).to_pylist() return [ - v if v is None or len(v) <= _MAX_TEXT else v[: _MAX_TEXT - 3] + "…" - for v in values + None if n is None or head is None else head + "…" if n > _MAX_TEXT else head + for head, n in zip(heads, lengths) ] if ( pa.types.is_list(arrow_type) @@ -339,14 +348,20 @@ def _summarise_extension( ] return [None if v is None else f"<{short_name}>" for v in values] if pa.types.is_binary(storage_type) or pa.types.is_large_binary(storage_type): - return [ - None if v is None else f"{short_name} ({_size(len(v))})" - for array in column.chunks - for v in cast("pa.ExtensionArray[Any]", array).storage.to_pylist() - ] + storage = pa.chunked_array( + [cast("pa.ExtensionArray[Any]", array).storage for array in column.chunks], + type=storage_type, + ) + return _byte_summaries(storage, short_name) return None +def _byte_summaries(column: pa.ChunkedArray[Any], name: str) -> list[str | None]: + """Return ``name (size)`` for each binary cell; sizes come from Arrow, not the bytes.""" + lengths: list[int | None] = pc.binary_length(column).to_pylist() + return [None if n is None else f"{name} ({_size(n)})" for n in lengths] + + def _struct_fields( array: pa.StructArray, names: list[str] ) -> list[dict[str, Any] | None]: @@ -378,14 +393,14 @@ def _summarise_list( lengths: list[int | None] = pc.list_value_length(column).to_pylist() if simple_items and all(n is None or n <= _MAX_LIST_ITEMS for n in lengths): return None - values = column.to_pylist() if simple_items else [None] * len(lengths) item = "list" if pa.types.is_nested(value_type) else None out: list[str | None] = [] - for value, length in zip(values, lengths): + for index, length in enumerate(lengths): if length is None: out.append(None) elif simple_items and length <= _MAX_LIST_ITEMS: - out.append(repr(value)) + # Only a short list is converted to Python, to show its items. + out.append(repr(column[index].as_py())) else: out.append(f"list (len {length})" + (f" of {item}" if item else "")) return out diff --git a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py index caff63dcc..d682d3a05 100644 --- a/tests/test_core/streams/test_dynamic_pod_stream_staleness.py +++ b/tests/test_core/streams/test_dynamic_pod_stream_staleness.py @@ -36,6 +36,7 @@ def _source() -> ArrowTableSource: def test_operator_stream_reruns_after_upstream_is_modified() -> None: + """A kept operator result is recomputed once an upstream changed.""" fn_stream = double.pod(_source()) assert isinstance(fn_stream, FunctionPodStream) out = SelectDataColumns(["y"])(fn_stream) diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 7ab6a172e..9dbf0f864 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -18,12 +18,18 @@ from orcapod import function_pod from orcapod.core.function_pod import FunctionPodStream from orcapod.core.operators import GroupBy, Join, PolarsFilter, SelectDataColumns -from orcapod.core.sources import ArrowTableSource, CachedSource, DictSource, ListSource +from orcapod.core.sources import ( + ArrowTableSource, + CachedSource, + DerivedSource, + DictSource, + ListSource, +) from orcapod.core.sources.polling_source import PollingSource from orcapod.databases import InMemoryArrowDatabase from orcapod.streams import HeadStatus, StreamHead from orcapod.side_effects import SideEffectPod -from orcapod.types import ColumnConfig, Cursor +from orcapod.types import ColumnConfig, Cursor, Schema ALL_COLUMNS = ColumnConfig( meta=True, context=True, source=True, system_tags=True, content_hash=True @@ -58,6 +64,7 @@ def _source(rows: int = 25, label: str = "readings") -> ArrowTableSource: def test_source_head_is_the_first_rows_with_the_exact_total() -> None: + """A source's head is the first rows of its table, with the exact row count.""" source = _source() head = source.head() assert isinstance(head, StreamHead) @@ -69,12 +76,14 @@ def test_source_head_is_the_first_rows_with_the_exact_total() -> None: def test_head_of_a_short_source_says_all_rows() -> None: + """A source no longer than n says it shows all of its rows.""" head = _source(rows=3).head() assert len(head) == 3 assert head.header() == "ArrowTableSource[readings] — all 3 rows" def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: + """With every column group, head is the prefix of as_table, in stream order.""" source = _source() head = source.head(4, columns=ALL_COLUMNS) assert head.table.equals(source.as_table(columns=ALL_COLUMNS).slice(0, 4)) @@ -85,6 +94,7 @@ def test_head_with_every_column_group_is_the_prefix_of_as_table() -> None: def test_head_refuses_negative_n_and_sorting() -> None: + """A negative n and sort_by_tags are refused.""" source = _source() with pytest.raises(ValueError, match="n"): source.head(-1) @@ -93,6 +103,7 @@ def test_head_refuses_negative_n_and_sorting() -> None: def test_head_of_zero_rows_keeps_the_columns() -> None: + """head(0) has no rows and keeps the columns.""" head = _source().head(0) assert head.status is HeadStatus.ROWS assert head.table.column_names == ["id", "x"] @@ -112,6 +123,7 @@ def test_head_of_zero_rows_keeps_the_columns() -> None: ids=["dict", "list"], ) def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: + """Dict and list sources serve the first rows they hold.""" source = make() head = source.head(5) assert head.status is HeadStatus.ROWS @@ -120,6 +132,7 @@ def test_other_in_memory_sources_serve_their_first_rows(make: Any) -> None: def test_preview_is_the_rendered_head() -> None: + """The notebook preview is head() rendered.""" source = _source() html = source._repr_html_() assert "ArrowTableSource[readings] — first 10 of 25 rows" in html @@ -141,6 +154,7 @@ def triple(x: int) -> int: def test_function_pod_stream_is_not_computed_by_head_or_preview() -> None: + """Neither head nor the preview calls the function of an uncomputed stream.""" calls: list[int] = [] stream = _counting_pod(calls).pod(_source(), label="triple") head = stream.head() @@ -154,6 +168,7 @@ def test_function_pod_stream_is_not_computed_by_head_or_preview() -> None: def test_function_pod_stream_shows_rows_computed_so_far() -> None: + """After an incomplete iteration, head shows the rows computed so far.""" calls: list[int] = [] stream = _counting_pod(calls).pod(_source()) rows = stream.iter_data() @@ -167,6 +182,7 @@ def test_function_pod_stream_shows_rows_computed_so_far() -> None: def test_function_pod_stream_shows_its_computed_rows() -> None: + """After as_table, head shows the computed rows without calling the function.""" calls: list[int] = [] stream = _counting_pod(calls).pod(_source()) table = stream.as_table() @@ -180,6 +196,7 @@ def test_function_pod_stream_shows_its_computed_rows() -> None: def test_function_pod_stream_rows_from_iteration_only() -> None: + """After a full iteration, head is the prefix of as_table with every column group.""" calls: list[int] = [] stream = _counting_pod(calls).pod(_source()) _ = list(stream.iter_data()) @@ -238,6 +255,7 @@ def _grouped_source() -> ArrowTableSource: @pytest.mark.parametrize("case", list(OPERATOR_CASES), ids=list(OPERATOR_CASES)) def test_operator_stream_is_not_run_by_head_or_preview(case: str) -> None: + """Neither head nor the preview runs the operator; its kept result is shown.""" operator_class, build = OPERATOR_CASES[case] stream = build() original = operator_class.static_process @@ -259,6 +277,7 @@ def test_operator_stream_is_not_run_by_head_or_preview(case: str) -> None: def test_an_empty_operator_result_is_not_not_computed() -> None: + """A computed empty result is ROWS with no rows, not NOT_COMPUTED.""" stream = PolarsFilter(constraints={"x": -1})(_source()) _ = stream.as_table() head = stream.head() @@ -268,6 +287,7 @@ def test_an_empty_operator_result_is_not_not_computed() -> None: def test_operator_stream_with_changed_inputs_is_not_computed() -> None: + """A kept result whose input changed is not shown.""" calls: list[int] = [] fn_stream = _counting_pod(calls).pod(_source()) assert isinstance(fn_stream, FunctionPodStream) @@ -285,6 +305,7 @@ def test_operator_stream_with_changed_inputs_is_not_computed() -> None: def test_side_effect_stream_preview_never_delivers() -> None: + """Showing a side-effect stream delivers nothing.""" delivered: list[int] = [] def deliver(x: int) -> None: @@ -301,9 +322,10 @@ def deliver(x: int) -> None: class _FakeDynamicSource: """A ``DynamicSourceProtocol`` double that records every poll and fetch.""" - def __init__(self) -> None: + def __init__(self, declared: Schema | None = None) -> None: self.polls = 0 self.fetches = 0 + self.declared = declared def identity(self) -> Any: return "fake" @@ -316,7 +338,7 @@ def from_config(cls, config: dict[str, Any]) -> _FakeDynamicSource: raise NotImplementedError def schema(self) -> Any: - return None + return self.declared async def poll(self, cursor: Cursor[int] | None = None) -> bool: self.polls += 1 @@ -334,6 +356,7 @@ async def close(self) -> None: def test_polling_source_head_never_polls() -> None: + """A polling source's head shows polled rows only and never polls.""" fake = _FakeDynamicSource() source = PollingSource(fake, tag_columns="id") head = source.head() @@ -349,7 +372,50 @@ def test_polling_source_head_never_polls() -> None: assert after.table.column("val").to_pylist() == [7] +def test_polling_source_head_shows_declared_system_tags_without_polling() -> None: + """A declared schema answers head's columns, system tags included, without a poll.""" + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + source = PollingSource(fake, tag_columns="id") + head = source.head(columns={"system_tags": True}) + assert (fake.polls, fake.fetches) == (0, 0) + tag_schema, data_schema = source.output_schema(columns={"system_tags": True}) + assert head.table.column_names == [*tag_schema, *data_schema] + assert any(name.startswith("_tag") for name in head.table.column_names) + + +def test_derived_source_over_a_polling_source_never_polls() -> None: + """A derived source's head asks its origin for its head, so nothing is polled.""" + fake = _FakeDynamicSource() + origin = PollingSource(fake, tag_columns="id") + # Building the derived source reads the origin's columns, which polls it once. + derived = DerivedSource(origin, tags_as_data={"id": "id_value"}) + seen = (fake.polls, fake.fetches) + head = derived.head() + _ = derived._repr_html_() + assert (fake.polls, fake.fetches) == seen + assert head.status is HeadStatus.ROWS + assert head.table.column("id_value").to_pylist() == [1] + + +def test_downstream_of_an_unpolled_source_never_polls() -> None: + """A stream over a source whose columns are known only by polling shows none.""" + fake = _FakeDynamicSource() + source = PollingSource(fake, tag_columns="id") + + def deliver(val: int) -> None: + raise AssertionError("delivered") + + stream = SideEffectPod(deliver)(source) + head = stream.head() + _ = stream._repr_html_() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.num_columns == 0 + assert "columns not known until its source is polled" in head.header() + + def test_cached_source_head_writes_nothing() -> None: + """A cached source's head writes nothing to its cache store.""" store = InMemoryArrowDatabase() source = CachedSource(_source(), cache_database=store) head = source.head(3) @@ -364,6 +430,7 @@ def test_cached_source_head_writes_nothing() -> None: def test_preview_summarises_large_cells_and_head_keeps_them() -> None: + """The preview summarises large cells; head's table keeps the full values.""" source = DictSource( [ { @@ -388,11 +455,13 @@ def test_preview_summarises_large_cells_and_head_keeps_them() -> None: def test_text_form_has_the_header() -> None: + """The text form of a head starts with its header.""" text = str(_source().head(2)) assert text.startswith("ArrowTableSource[readings] — first 2 of 25 rows") def test_cached_function_pod_stream_does_not_read_its_result_store() -> None: + """A cached pod's stream shows only computed rows and never calls the function.""" from orcapod.core.cached_function_pod import CachedFunctionPod calls: list[int] = [] diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 80d34786e..f40625962 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -74,11 +74,24 @@ def _job(store: ArrowDatabaseProtocol) -> PipelineJob: def test_nodes_before_a_run_show_no_rows_and_compute_nothing() -> None: + """Before a run, nodes show no rows and run neither the function nor the operator.""" job = _job(InMemoryArrowDatabase()) - fn_head = job.nodes["triple"].head() - op_head = job.nodes["sel"].head() - _ = job.nodes["triple"]._repr_html_() - _ = job.nodes["sel"]._repr_html_() + # The operator must not run: either entry point raises if it is called. + with ( + mock.patch.object( + SelectDataColumns, "process", side_effect=AssertionError("process ran") + ), + mock.patch.object( + SelectDataColumns, + "static_process", + side_effect=AssertionError("static_process ran"), + ), + ): + fn_head = job.nodes["triple"].head() + op_head = job.nodes["sel"].head() + _ = job.nodes["triple"]._repr_html_() + op_html = job.nodes["sel"]._repr_html_() + assert "could not read" not in op_html assert CALLS == [] assert fn_head.status is HeadStatus.NOTHING_STORED assert fn_head.table.column_names == ["id", "y"] @@ -91,6 +104,7 @@ def test_nodes_before_a_run_show_no_rows_and_compute_nothing() -> None: def test_nodes_after_a_run_show_their_first_rows( orchestrator: str, store_kind: str, tmp_path: Path ) -> None: + """Right after a run, nodes show their first rows from memory.""" store = _store(store_kind, tmp_path) job = _job(store) _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) @@ -113,6 +127,7 @@ def test_nodes_after_a_run_show_their_first_rows( def test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing( store_kind: str, tmp_path: Path ) -> None: + """A new job over a filled store shows stored rows without computing.""" store = _store(store_kind, tmp_path) _ = _job(store).run() CALLS.clear() @@ -128,6 +143,7 @@ def test_a_new_job_over_a_filled_store_reads_stored_rows_without_computing( def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None: + """A loaded job's nodes show stored rows, or say why there are none.""" store = DeltaTableDatabase(base_path=tmp_path / "store") job = _job(store) _ = job.run() @@ -146,6 +162,7 @@ def test_a_loaded_job_shows_stored_rows_and_never_raises(tmp_path: Path) -> None def test_blueprint_nodes_are_unavailable() -> None: + """Blueprint nodes have no rows to show.""" job = _job(InMemoryArrowDatabase()) for label in ("triple", "sel"): blueprint = job.nodes[label].as_node() @@ -155,6 +172,7 @@ def test_blueprint_nodes_are_unavailable() -> None: def test_source_nodes() -> None: + """Input slots and unbound source nodes show no rows; bound ones delegate.""" slot = SourceNode( name="nums", tag_schema=Schema({"id": int}), data_schema=Schema({"x": int}) ) @@ -183,6 +201,7 @@ def _operator_node( def test_operator_node_cache_modes() -> None: + """Operator nodes show stored rows in REPLAY and none in LOG before a run.""" store = InMemoryArrowDatabase() assert _operator_node(CacheMode.REPLAY, store).head().status is ( HeadStatus.NOTHING_STORED From 142db4a08c0b2d8da58661188d8276531d8f79c8 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 15:53:49 -0800 Subject: [PATCH 12/19] feat(streams): preview shows array cells from either stored form, long keys by both ends, unlabelled sources by source id (ITL-991) - An array column is recognised by the array logical type's extension name; its struct form (main) and its .npy form (ITL-981, PR #328) both show as 'ndarray ', reading the .npy header only. - Long strings keep their start and end around an ellipsis, and Polars shows up to 80 characters, so hash keys stay distinguishable. - A source given no label is shown by its source id in the header; the label itself is unchanged. - StreamProtocol.head returns StreamHeadProtocol, keeping head.py out of the protocol import chain; delegations go through head_of(). Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/nodes/operator_node.py | 3 +- src/orcapod/core/nodes/source_node.py | 7 +- .../core/operators/static_output_pod.py | 7 +- src/orcapod/core/sources/base.py | 8 + src/orcapod/core/sources/cached_source.py | 14 +- src/orcapod/core/sources/source_proxy.py | 7 +- src/orcapod/core/streams/base.py | 4 + src/orcapod/core/streams/head.py | 184 ++++++++++++++---- .../protocols/core_protocols/__init__.py | 3 +- .../protocols/core_protocols/streams.py | 41 +++- ...-itl-991-stream-head-and-preview-design.md | 14 ++ .../streams/test_head_and_preview.py | 74 +++++++ 12 files changed, 307 insertions(+), 59 deletions(-) diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 28b39643c..46e8457d0 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -36,6 +36,7 @@ BLUEPRINT_NOTE, HeadStatus, StreamHead, + head_of, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.errors import ( @@ -1332,7 +1333,7 @@ def head( check_head_size(n) config = head_column_config(columns, all_info) if self._cached_output_stream is not None: - return self._cached_output_stream.head(n, columns=config).for_stream(self) + return head_of(self._cached_output_stream.head(n, columns=config), self) if self._cache_mode == CacheMode.REPLAY and self._pipeline_database is not None: return self._head_from_store(n, config) if self._loaded: diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index 277c17c5d..e4cdbb20e 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -20,6 +20,7 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, + head_of, render_error_html, ) from orcapod.core.streams.base import ( @@ -638,9 +639,9 @@ def head( HeadStatus.UNAVAILABLE, note="no source bound", ) - return self._bound_source.head( - n, columns=columns, all_info=all_info - ).for_stream(self) + return head_of( + self._bound_source.head(n, columns=columns, all_info=all_info), self + ) async def async_iter_data( self, diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 83f40613b..9d8f90a27 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -17,9 +17,10 @@ head_column_config, ) from orcapod.core.streams.head import ( - STALE_NOTE, HeadStatus, + STALE_NOTE, StreamHead, + head_of, ) from orcapod.core.tracker import DEFAULT_TRACKER_MANAGER from orcapod.protocols.core_protocols import ( @@ -450,8 +451,8 @@ def head( had_result = self._cached_stream is not None self._update_cache_status() if self._cached_stream is not None and self._cached_time is not None: - return self._cached_stream.head(n, columns=config).for_stream( - self, origin="computed" + return head_of( + self._cached_stream.head(n, columns=config), self, origin="computed" ) return self._head_without_rows( config, diff --git a/src/orcapod/core/sources/base.py b/src/orcapod/core/sources/base.py index 2464fe774..195473241 100644 --- a/src/orcapod/core/sources/base.py +++ b/src/orcapod/core/sources/base.py @@ -187,6 +187,14 @@ def as_table( """Delegate to the underlying stream's as_table.""" return self._stream.as_table(columns=columns, all_info=all_info) + def _display_name(self) -> str: + """The name a preview header shows: the label when one was given, else the + source id (a source id is the source's name), e.g. a table's name. The label + itself is not changed.""" + if self.has_assigned_label or self._source_id is None: + return self.label + return self._source_id + def head( self, n: int = 10, diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index 807894516..9f04b778c 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -9,7 +9,10 @@ from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import schema_requires_io -from orcapod.core.streams.head import StreamHead +from orcapod.core.streams.head import ( + StreamHead, + head_of, +) from orcapod.contexts.stored import ( keep_rows_of_data_context, require_same_data_context, @@ -425,10 +428,11 @@ def head( read and written only on first use (``iter_data``, ``as_table``). """ if self._cached_stream is not None and not self.is_stale: - return self._cached_stream.head( - n, columns=columns, all_info=all_info - ).for_stream(self) - return self._source.head(n, columns=columns, all_info=all_info).for_stream( + return head_of( + self._cached_stream.head(n, columns=columns, all_info=all_info), self + ) + return head_of( + self._source.head(n, columns=columns, all_info=all_info), self, note="rows of the wrapped source; the cache is read and written on first use", ) diff --git a/src/orcapod/core/sources/source_proxy.py b/src/orcapod/core/sources/source_proxy.py index ed1c677e3..6714072c9 100644 --- a/src/orcapod/core/sources/source_proxy.py +++ b/src/orcapod/core/sources/source_proxy.py @@ -16,6 +16,7 @@ from orcapod.core.streams.head import ( HeadStatus, StreamHead, + head_of, ) from orcapod.core.streams.base import ( check_head_size, @@ -219,9 +220,9 @@ def head( ) -> StreamHead: """Return the bound source's head, or no rows when no live source is bound.""" if self._delegate is not None: - return self._delegate.head( - n, columns=columns, all_info=all_info - ).for_stream(self) + return head_of( + self._delegate.head(n, columns=columns, all_info=all_info), self + ) check_head_size(n) config = head_column_config(columns, all_info) return self._head_without_rows( diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 26f50b7d7..8222cf25f 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -562,6 +562,10 @@ def _head_without_rows( self, status, table, tag_columns=tuple(tag_schema.keys()), note=note ) + def _display_name(self) -> str: + """The name a preview header shows: the label. Never used as an identity.""" + return self.label + def _schema_requires_io(self) -> bool: """Whether reading this stream's schema would read or poll a source. diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index d7751e049..e06ef5b52 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -9,8 +9,11 @@ from __future__ import annotations +import ast import dataclasses import html +import importlib +from collections.abc import Sequence from dataclasses import dataclass from enum import Enum from typing import TYPE_CHECKING, Any, cast @@ -80,6 +83,8 @@ def default_note(status: HeadStatus) -> str: # Display limits of the preview. _MAX_TEXT = 80 +_TEXT_START = 56 +_TEXT_END = 20 _MAX_LIST_ITEMS = 5 @@ -92,7 +97,8 @@ class StreamHead: stream has no rows to show, a zero-row table with its columns. status: Where the rows came from, or why there are none. kind: Class name of the stream, source or node shown. - label: Its label. + label: The name shown for it: its label, or for a source given no label its + source id (``display_name``). Never used as an identity. tag_columns: The columns of ``table`` that are tags. total_rows: The stream's row count when it is known without computing, else ``None``. For ``PARTIAL``, the number of rows computed so far. @@ -132,7 +138,7 @@ def of_rows( table=table, status=status, kind=type(stream).__name__, - label=str(stream.label), + label=display_name(stream), tag_columns=tuple(c for c in tag_columns if c in table.column_names), total_rows=total_rows, total_is_estimate=total_is_estimate, @@ -155,7 +161,7 @@ def without_rows( table=table.slice(0, 0), status=status, kind=type(stream).__name__, - label=str(stream.label), + label=display_name(stream), tag_columns=tuple(c for c in tag_columns if c in table.column_names), note=note, ) @@ -168,7 +174,7 @@ def for_stream( return dataclasses.replace( self, kind=type(stream).__name__, - label=str(stream.label), + label=display_name(stream), origin=self.origin if origin is None else origin, note=self.note if note is None else note, ) @@ -254,7 +260,9 @@ def _repr_html_(self) -> str: if tags: parts.append(f"
{html.escape(tags)}
") if self.table.num_columns: - parts.append(self._display_frame()._repr_html_()) + # Polars shortens strings to its own default length; show up to _MAX_TEXT. + with pl.Config(fmt_str_lengths=_MAX_TEXT): + parts.append(self._display_frame()._repr_html_()) return "\n".join(parts) def __str__(self) -> str: @@ -263,7 +271,8 @@ def __str__(self) -> str: if tags: lines.append(tags) if self.table.num_columns: - lines.append(str(self._display_frame())) + with pl.Config(fmt_str_lengths=_MAX_TEXT): + lines.append(str(self._display_frame())) return "\n".join(lines) def __repr__(self) -> str: @@ -273,9 +282,45 @@ def __repr__(self) -> str: ) +_MAX_NAME = 48 + + +def display_name(stream: Any) -> str: + """Return the name the header shows for ``stream``: never its label changed. + + A stream that says (``_display_name``) gives its own, e.g. a source with no label + of its own is shown by its source id; otherwise its label. A long name keeps its + start and end around an ellipsis. + """ + name_of = getattr(stream, "_display_name", None) + name = str(name_of() if callable(name_of) else stream.label) + if len(name) > _MAX_NAME: + name = f"{name[:30]}…{name[-12:]}" + return name + + +def head_of( + head: Any, stream: Any, *, origin: str | None = None, note: str | None = None +) -> StreamHead: + """Return ``head`` (another stream's) shown as ``stream``. + + Args: + head: What another stream's ``head`` returned: a ``StreamHead``. + stream: The stream, source or node that shows those rows as its own. + origin: Another origin for the rows, if given. + note: Another note, if given. + + Raises: + TypeError: If ``head`` is not a ``StreamHead``. + """ + if not isinstance(head, StreamHead): + raise TypeError(f"head() returned {type(head).__name__}, not a StreamHead") + return head.for_stream(stream, origin=origin, note=note) + + def render_error_html(stream: Any, error: BaseException) -> str: """Return the preview of a stream whose ``head`` raised: its name and the error.""" - title = f"{type(stream).__name__}[{stream.label}] — could not read: {error}" + title = f"{type(stream).__name__}[{display_name(stream)}] — could not read: {error}" return f"
{html.escape(title)}
" @@ -300,18 +345,24 @@ def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: if isinstance(arrow_type, pa.ExtensionType): return _summarise_extension(column, arrow_type) if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): - return _byte_summaries(column, "bytes") + return _byte_summaries([column], "bytes") if pa.types.is_string(arrow_type) or pa.types.is_large_string(arrow_type): - # Lengths come from Arrow; only the shortened text becomes Python strings. + # Lengths come from Arrow; only the kept parts become Python strings. A long + # value keeps its start and its end (hash keys differ at the end), with an + # ellipsis between. lengths: list[int | None] = pc.utf8_length(column).to_pylist() if all(n is None or n <= _MAX_TEXT for n in lengths): return None - heads: list[str | None] = pc.utf8_slice_codeunits( - column, 0, _MAX_TEXT - 3 + shorts: list[str | None] = pc.utf8_slice_codeunits( + column, 0, _MAX_TEXT + ).to_pylist() + starts: list[str | None] = pc.utf8_slice_codeunits( + column, 0, _TEXT_START ).to_pylist() + ends: list[str | None] = pc.utf8_slice_codeunits(column, -_TEXT_END).to_pylist() return [ - None if n is None or head is None else head + "…" if n > _MAX_TEXT else head - for head, n in zip(heads, lengths) + None if n is None else f"{a}…{b}" if n > _MAX_TEXT else short + for n, short, a, b in zip(lengths, shorts, starts, ends) ] if ( pa.types.is_list(arrow_type) @@ -325,41 +376,92 @@ def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: def _summarise_extension( column: pa.ChunkedArray[Any], arrow_type: pa.ExtensionType ) -> list[str | None] | None: + """Return display strings for a logical-type column, or ``None`` to let Polars show it. + + An array column is recognised by its logical type's extension name, never by the + bytes of a cell: it shows as ``ndarray float64 (2, 3)``, read from its stored form + without decoding the data. Another logical type over binary storage shows its size, + over a struct its name; any other storage is shown by Polars. + """ + # The array logical type's module is read when a column is shown, not imported: + # it sits above this module in the import graph (types, protocols, side effects), + # and it loads numpy, which only an array column needs. + numpy_type = importlib.import_module("orcapod.logical_types.numpy_type") + array_extension_name: str = numpy_type.NDARRAY_EXTENSION_NAME + storage_type = arrow_type.storage_type short_name = arrow_type.extension_name.rsplit(".", 1)[-1] - if pa.types.is_struct(storage_type): - names = {storage_type.field(i).name for i in range(storage_type.num_fields)} - # Only the small fields are read; an ndarray's ``data`` is never decoded. - wanted = [n for n in ("dtype", "shape") if n in names] - values = [ - v - for array in column.chunks - for v in _struct_fields( - cast(pa.StructArray, cast("pa.ExtensionArray[Any]", array).storage), - wanted, - ) - ] - if {"dtype", "shape"} <= names: + storages = [ + cast("pa.ExtensionArray[Any]", array).storage for array in column.chunks + ] + is_binary = pa.types.is_binary(storage_type) or pa.types.is_large_binary( + storage_type + ) + if arrow_type.extension_name == array_extension_name: + if pa.types.is_struct(storage_type): return [ - None - if v is None - else f"{short_name} {v['dtype']} {_shape_text(v['shape'])}" - for v in values + None if v is None else f"ndarray {v['dtype']} {_shape_text(v['shape'])}" + for storage in storages + for v in _struct_fields( + cast(pa.StructArray, storage), ["dtype", "shape"] + ) ] - return [None if v is None else f"<{short_name}>" for v in values] - if pa.types.is_binary(storage_type) or pa.types.is_large_binary(storage_type): - storage = pa.chunked_array( - [cast("pa.ExtensionArray[Any]", array).storage for array in column.chunks], - type=storage_type, - ) - return _byte_summaries(storage, short_name) + if is_binary: + return [ + _npy_summary(storage[i]) + for storage in storages + for i in range(len(storage)) + ] + if is_binary: + return _byte_summaries(storages, short_name) + if pa.types.is_struct(storage_type): + return [ + f"<{short_name}>" if valid else None + for storage in storages + for valid in storage.is_valid().to_pylist() + ] return None -def _byte_summaries(column: pa.ChunkedArray[Any], name: str) -> list[str | None]: - """Return ``name (size)`` for each binary cell; sizes come from Arrow, not the bytes.""" - lengths: list[int | None] = pc.binary_length(column).to_pylist() - return [None if n is None else f"{name} ({_size(n)})" for n in lengths] +def _npy_summary(cell: Any) -> str | None: + """Return ``ndarray `` for one ``.npy`` cell, reading its header only. + + The ``.npy`` header is a magic string, a version, the header length and a Python + dict literal with ``descr``, ``fortran_order`` and ``shape``. The buffer is sliced + without copying, and the array data is never read. + """ + if not cell.is_valid: + return None + buffer = cell.as_buffer() + size = f"ndarray ({_size(buffer.size)})" + try: + prefix = buffer[:12].to_pybytes() + if not prefix.startswith(b"\x93NUMPY"): + return size + if prefix[6] == 1: + start, length = 10, int.from_bytes(prefix[8:10], "little") + else: + start, length = 12, int.from_bytes(prefix[8:12], "little") + text = buffer[start : start + length].to_pybytes().decode("utf-8") + header = ast.literal_eval(text) + import numpy as np + + dtype = np.dtype(header["descr"]) + return f"ndarray {dtype.name} {_shape_text(header['shape'])}" + except (ValueError, SyntaxError, KeyError, TypeError, IndexError): + return size + + +def _byte_summaries(arrays: Sequence[Any], name: str) -> list[str | None]: + """Return ``name (size)`` for each binary cell of ``arrays``, in order. + + Sizes come from Arrow (``binary_length``); the bytes are never read. + """ + return [ + None if n is None else f"{name} ({_size(n)})" + for array in arrays + for n in pc.binary_length(array).to_pylist() + ] def _struct_fields( diff --git a/src/orcapod/protocols/core_protocols/__init__.py b/src/orcapod/protocols/core_protocols/__init__.py index ed6b2acbf..9c27952bd 100644 --- a/src/orcapod/protocols/core_protocols/__init__.py +++ b/src/orcapod/protocols/core_protocols/__init__.py @@ -8,7 +8,7 @@ from .data_function import DataFunctionProtocol from .pod import ArgumentGroup, PodProtocol from .sources import DynamicSourceProtocol, SourceProtocol -from .streams import StreamProtocol +from .streams import StreamHeadProtocol, StreamProtocol from .side_effect_pod import SideEffectPodProtocol from .trackers import TrackerProtocol, TrackerManagerProtocol @@ -19,6 +19,7 @@ "DataProtocol", "DynamicSourceProtocol", "SourceProtocol", + "StreamHeadProtocol", "StreamProtocol", "PodProtocol", "ArgumentGroup", diff --git a/src/orcapod/protocols/core_protocols/streams.py b/src/orcapod/protocols/core_protocols/streams.py index 200d8b261..bb751c957 100644 --- a/src/orcapod/protocols/core_protocols/streams.py +++ b/src/orcapod/protocols/core_protocols/streams.py @@ -13,10 +13,47 @@ import polars as pl import pyarrow as pa - from orcapod.core.streams.head import StreamHead from orcapod.protocols.core_protocols.pod import PodProtocol +class StreamHeadProtocol(Protocol): + """The readily available first rows of a stream, as ``head`` returns them. + + ``orcapod.core.streams.head.StreamHead`` is the implementation. + """ + + @property + def table(self) -> "pa.Table": + """The rows, with the exact Arrow types the stream gives them.""" + ... + + @property + def status(self) -> str: + """Where the rows came from, or why there are none (a ``HeadStatus``).""" + ... + + @property + def total_rows(self) -> int | None: + """The stream's row count when it is known without computing.""" + ... + + def header(self) -> str: + """The one-line description shown above the rows.""" + ... + + def as_table(self) -> "pa.Table": + """The rows as a PyArrow table.""" + ... + + def as_polars_df(self) -> "pl.DataFrame": + """The rows as a Polars DataFrame.""" + ... + + def _repr_html_(self) -> str: + """The notebook preview of the rows.""" + ... + + @runtime_checkable class StreamProtocol(TraceableProtocol, PipelineElementProtocol, Protocol): """ @@ -164,7 +201,7 @@ def head( *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, - ) -> "StreamHead": + ) -> "StreamHeadProtocol": """Return at most ``n`` readily available rows, without computing. The rows are ones the stream already holds, has cached from an earlier diff --git a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md index 30b0e7466..07d398187 100644 --- a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md +++ b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md @@ -383,3 +383,17 @@ renders `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349`. - `head()` is part of `StreamProtocol`. - `_CachedFunctionPodStream` (added on main by ITL-988) drops stored meta columns in head as in `as_table`. +- `StreamProtocol.head` returns `StreamHeadProtocol` (in `protocols/core_protocols/streams.py`), + which `StreamHead` implements; that keeps `head.py` out of the protocol import chain. + Delegating heads go through `head_of(head, stream)`. +- An array column is recognised by the array logical type's extension name + (`numpy_type.NDARRAY_EXTENSION_NAME`, read at call time through `importlib` because that + module sits above `head.py` in the import graph), never by sniffing bytes. Both stored + forms are shown as `ndarray `: the `{dtype, shape, data}` struct on main + today, and one `.npy` file per cell after PR #328 (ITL-981), whose header alone is read + (zero-copy buffer slice). Other binary cells show `bytes (N B)`. +- Long strings keep their first 56 and last 20 characters around an ellipsis, so ~140 + character hash keys stay distinguishable; Polars' own string shortening is raised to 80 + characters while rendering (coordinator input from the ITL-990 workshop rundown). +- A rowless head does not read its schema when that would poll a source upstream + (`schema_requires_io`); it then has no columns and says so (CodeRabbit round 1). diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 9dbf0f864..0d42f3216 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -8,10 +8,12 @@ from __future__ import annotations +import io from typing import Any from unittest import mock import numpy as np +import numpy.typing as npt import pyarrow as pa import pytest @@ -479,3 +481,75 @@ def test_cached_function_pod_stream_does_not_read_its_result_store() -> None: assert after.status is HeadStatus.ROWS assert after.table.equals(table.slice(0, 5)) assert len(calls) == 25 + + +class _NpyArrayType(pa.ExtensionType): + """The array logical type stored as one ``.npy`` file per cell (the ITL-981 form).""" + + def __init__(self) -> None: + from orcapod.logical_types.numpy_type import NDARRAY_EXTENSION_NAME + + super().__init__(pa.large_binary(), NDARRAY_EXTENSION_NAME) + + def __arrow_ext_serialize__(self) -> bytes: + return b"" + + @classmethod + def __arrow_ext_deserialize__( + cls, storage_type: pa.DataType, serialized: bytes + ) -> _NpyArrayType: + return cls() + + +def _npy(array: npt.NDArray[Any]) -> bytes: + buffer = io.BytesIO() + np.lib.format.write_array(buffer, array, version=(3, 0)) + return buffer.getvalue() + + +def test_preview_reads_an_npy_cell_header_only() -> None: + """An array cell stored as ``.npy`` shows its dtype and shape from the header.""" + storage = pa.array( + [_npy(np.zeros(40, dtype=np.float32)), _npy(np.ones((2, 3))), None], + type=pa.large_binary(), + ) + table = pa.table( + { + "id": pa.array([1, 2, 3], type=pa.int64()), + "frame": pa.ExtensionArray.from_storage(_NpyArrayType(), storage), + "jpeg": pa.array([b"\xff\xd8\xff\xe0" * 10, b"", None], pa.large_binary()), + } + ) + head = StreamHead( + table=table, + status=HeadStatus.ROWS, + kind="ArrowTableStream", + label="frames", + tag_columns=("id",), + total_rows=3, + ) + html = head._repr_html_() + assert "ndarray float32 (40,)" in html + assert "ndarray float64 (2, 3)" in html + assert "bytes (40 B)" in html + assert "NUMPY" not in html + + +def test_preview_keeps_both_ends_of_a_long_key() -> None: + """A long hash key keeps its start and its distinguishing end in the preview.""" + key = "derived:arrow_v1:" + "ab" * 61 + "XYZ" + source = DictSource([{"key": key, "x": 1}], tag_columns=["key"], label="keys") + html = source._repr_html_() + assert "derived:arrow_v1:abab" in html + assert "XYZ" in html + assert key not in html + assert source.head().table.column("key").to_pylist() == [key] + + +def test_an_unlabelled_source_is_shown_by_its_source_id() -> None: + """A source given no label is named by its source id in the header; its label is kept.""" + source = DictSource( + [{"id": 1, "x": 2}], tag_columns=["id"], source_id="spike_event" + ) + assert source.head().header() == "DictSource[spike_event] — all 1 row" + assert source.label == "DictSource" From b16f15953761f1452623be4bd862fe90c9d7ce7d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 16:05:05 -0800 Subject: [PATCH 13/19] fix(sources): address CodeRabbit round 2 on head() (ITL-991) - DerivedSource.head enriches only the shown stored rows (the total is the whole stored count) and, when its origin has no rows but known columns, shows its own columns (tag copies included) with the origin's status and note. - PollingSource.head combines only the batches holding the first n rows and counts the rest. - CachedSource.head keeps the wrapped source's explanation when it has no rows. - FunctionPodStream builds an empty result with the declared columns. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 11 +++- src/orcapod/core/sources/cached_source.py | 13 +++-- src/orcapod/core/sources/derived_source.py | 22 ++++++-- src/orcapod/core/sources/polling_source.py | 17 ++++++- .../streams/test_head_and_preview.py | 51 +++++++++++++++++++ 5 files changed, 103 insertions(+), 11 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 910f1c97a..6c1b0186a 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1237,7 +1237,16 @@ def _computed_prefix( def _table_from_pairs( self, pairs: list[tuple[TagProtocol, DataProtocol]] ) -> pa.Table: - """Return the output rows ``pairs`` as one table with every column group.""" + """Return the output rows ``pairs`` as one table with every column group. + + With no rows the table has the declared columns (tags, system tags and data), + typed from the declared schema, as the rest of a head without rows is. + """ + if not pairs: + tag_schema, data_schema = self.output_schema(all_info=True) + return arrow_utils.make_empty_table( + {**tag_schema, **data_schema}, self.data_context.type_converter + ) all_tags = [] all_data = [] tag_schema, data_schema = None, None diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index 9f04b778c..6138577de 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -1,5 +1,7 @@ from __future__ import annotations +import dataclasses + import logging from collections.abc import Iterator from typing import TYPE_CHECKING, Any @@ -431,9 +433,14 @@ def head( return head_of( self._cached_stream.head(n, columns=columns, all_info=all_info), self ) - return head_of( - self._source.head(n, columns=columns, all_info=all_info), - self, + wrapped = head_of( + self._source.head(n, columns=columns, all_info=all_info), self + ) + if wrapped.table.num_rows == 0: + # Keep the wrapped source's own explanation of why there are no rows. + return wrapped + return dataclasses.replace( + wrapped, note="rows of the wrapped source; the cache is read and written on first use", ) diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index b93497415..a473943aa 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -476,7 +476,19 @@ def head( if isinstance(self._origin, RootSource): origin_head = self._origin.head(n) if origin_head.status is not HeadStatus.ROWS: - return origin_head.for_stream(self) + if origin_head.table.num_columns == 0: + # The origin's columns are not known without polling it. + return origin_head.for_stream(self) + # No rows: this source's own columns (tag copies included), with the + # origin's status and explanation. + empty = self._stream_over(origin_head.table.slice(0, 0)) + return StreamHead.without_rows( + self, + origin_head.status, + empty.as_table(columns=config), + tag_columns=empty.keys(columns=config)[0], + note=origin_head.note, + ) stream = self._stream_over(origin_head.table) return dataclasses.replace( stream.head(n, columns=config).for_stream(self), @@ -495,10 +507,10 @@ def head( tag_columns=stream.keys(columns=config)[0], note="nothing stored: its origin has not run", ) - return ( - self._stream_over(records) - .head(n, columns=config) - .for_stream(self, origin="stored") + # Only the shown rows are enriched; the total is the whole stored count. + head = self._stream_over(records.slice(0, n)).head(n, columns=config) + return dataclasses.replace( + head.for_stream(self, origin="stored"), total_rows=records.num_rows ) def _stream_over(self, table: "pa.Table") -> ArrowTableStream: diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index d9c89cbc2..78689622a 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -464,8 +464,21 @@ def head( """ check_head_size(n) config = head_column_config(columns, all_info) - if self._batches: - return self._get_combined_stream().head(n, columns=config).for_stream(self) + batches = list(self._batches) # snapshot: the list is append-only + if batches: + # Combine only the batches that hold the first n rows; count the rest. + sizes = [batch.head(0).total_rows or 0 for batch in batches] + needed, covered = 0, 0 + while needed < len(batches) and (covered < n or needed == 0): + covered += sizes[needed] + needed += 1 + prefix = batches[0] + for batch in batches[1:needed]: + prefix = self._combine(prefix, batch) + return dataclasses.replace( + prefix.head(n, columns=config).for_stream(self), + total_rows=sum(sizes), + ) if self._tag_schema is not None and self._data_schema is not None: # The declared schema answers without a fetch, system tags included; the # column groups that need data (source info, context, ...) are left out. diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 0d42f3216..9f5997e6f 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -553,3 +553,54 @@ def test_an_unlabelled_source_is_shown_by_its_source_id() -> None: ) assert source.head().header() == "DictSource[spike_event] — all 1 row" assert source.label == "DictSource" + + +class _BatchedDynamicSource(_FakeDynamicSource): + """Serves three batches of two rows, one per poll that reports new data.""" + + async def poll(self, cursor: Cursor[int] | None = None) -> bool: + self.polls += 1 + return (cursor.value if cursor is not None else 0) < 3 + + async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + self.fetches += 1 + index = cursor.value if cursor is not None else 0 + return Cursor(value=index + 1), { + "id": pa.array([2 * index, 2 * index + 1], type=pa.int64()), + "val": pa.array([10 * index, 10 * index + 1], type=pa.int64()), + } + + +def test_polling_source_head_over_several_batches_is_the_prefix() -> None: + """With several polled batches, head is the prefix of all rows, with the full count.""" + fake = _BatchedDynamicSource() + source = PollingSource(fake, tag_columns="id") + for _ in range(3): + _ = list(source.iter_data()) + seen = (fake.polls, fake.fetches) + head = source.head(3) + assert (fake.polls, fake.fetches) == seen + assert head.total_rows == 6 + assert head.table.equals(source.as_table().slice(0, 3)) + + +def test_derived_source_rowless_head_has_its_own_columns() -> None: + """Before its origin polls, a derived source shows its columns, tag copies included.""" + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + origin = PollingSource(fake, tag_columns="id") + derived = DerivedSource(origin, tags_as_data={"id": "id_value"}) + head = derived.head() + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert "id_value" in head.table.column_names + assert "nothing polled yet" in head.header() + + +def test_cached_source_keeps_the_wrapped_explanation_when_it_has_no_rows() -> None: + """A cached source over an unpolled source says why there are no rows.""" + source = CachedSource( + PollingSource(_FakeDynamicSource(), tag_columns="id"), + cache_database=InMemoryArrowDatabase(), + ) + head = source.head() + assert "nothing polled yet" in head.header() From ff279bcf9e6098504ae50be38c61098c300d462b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 16:25:51 -0800 Subject: [PATCH 14/19] fix(nodes): address CodeRabbit round 3 on head() (ITL-991) - An operator node whose async run produced no rows keeps a typed zero-row result, as the sync path does, so head() reads it as computed and empty rather than not run. Nothing is stored for it. - head_of() adapts any StreamHeadProtocol result instead of refusing one that is not a StreamHead. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/nodes/operator_node.py | 10 ++++++- src/orcapod/core/streams/head.py | 17 +++++++---- .../streams/test_head_and_preview.py | 29 +++++++++++++++++++ tests/test_pipeline/test_node_head.py | 17 +++++++++++ 4 files changed, 67 insertions(+), 6 deletions(-) diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 46e8457d0..71b4aff20 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -1563,12 +1563,20 @@ async def forward() -> None: # does, and store it in LOG mode (sync DB write, post-hoc). A run # with no rows replaces an earlier run's output too: the node then # reads as a zero-row table. - self._cached_output_stream = None if collected: stream = StaticOutputOperatorPod._materialize_to_stream(collected) self._cached_output_stream = stream if should_store: self._store_output_stream(stream) + else: + # A completed run with no rows keeps a typed zero-row result, as the + # sync path does, so the node reads as computed and empty (not as + # never run). Nothing is stored for it. + self._cached_output_stream = ArrowTableStream( + self._make_empty_table(), + tag_columns=self.keys()[0], + data_context=self.data_context, + ) self._update_modified_time() diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index 93e384583..b40cba7e4 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -312,17 +312,24 @@ def head_of( ) -> StreamHead: """Return ``head`` (another stream's) shown as ``stream``. + A ``StreamHead`` keeps every field. Any other result of ``head`` (a + ``StreamHeadProtocol``) is adapted from what that protocol gives: its rows, status + and total. + Args: - head: What another stream's ``head`` returned: a ``StreamHead``. + head: What another stream's ``head`` returned. stream: The stream, source or node that shows those rows as its own. origin: Another origin for the rows, if given. note: Another note, if given. - - Raises: - TypeError: If ``head`` is not a ``StreamHead``. """ if not isinstance(head, StreamHead): - raise TypeError(f"head() returned {type(head).__name__}, not a StreamHead") + head = StreamHead( + table=head.table, + status=HeadStatus(head.status), + kind=type(stream).__name__, + label=display_name(stream), + total_rows=head.total_rows, + ) return head.for_stream(stream, origin=origin, note=note) diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 9f5997e6f..12dfb6764 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -604,3 +604,32 @@ def test_cached_source_keeps_the_wrapped_explanation_when_it_has_no_rows() -> No ) head = source.head() assert "nothing polled yet" in head.header() + + +def test_head_of_adapts_any_head_the_protocol_allows() -> None: + """A head that is not a ``StreamHead`` but follows the protocol is adapted.""" + from orcapod.core.streams.head import head_of + + class _PlainHead: + """A minimal ``StreamHeadProtocol`` result.""" + + table = _table(2) + status = "rows" + total_rows = 2 + + def header(self) -> str: + return "plain" + + def as_table(self) -> pa.Table: + return self.table + + def as_polars_df(self) -> Any: + return None + + def _repr_html_(self) -> str: + return "" + + source = _source() + head = head_of(_PlainHead(), source) + assert head.status is HeadStatus.ROWS + assert head.header() == "ArrowTableSource[readings] — all 2 rows" diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index f40625962..1115962da 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -222,3 +222,20 @@ def test_operator_node_cache_modes() -> None: log_head = log.head() assert log_head.status is HeadStatus.NOT_COMPUTED assert "not replayed" in log_head.header() + + +@pytest.mark.parametrize("orchestrator", list(ORCHESTRATORS)) +def test_an_operator_node_with_an_empty_result_reads_as_computed( + orchestrator: str, +) -> None: + """After a run that produced no rows, an operator node is computed and empty.""" + from orcapod.core.operators import PolarsFilter + + job = PipelineJob(name="itl991_empty", store=InMemoryArrowDatabase()) + with job: + _ = PolarsFilter(constraints={"x": -1})(_source(), label="none") + _ = job.run(orchestrator=ORCHESTRATORS[orchestrator]()) + head = job.nodes["none"].head() + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert head.table.column_names == ["id", "x"] From b5499db3ff5284fac772e109e32d3712c965030e Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 16:40:05 -0800 Subject: [PATCH 15/19] fix(streams): address CodeRabbit review of the #333 merge (ITL-991) - A computed empty function-pod result has the columns as_table gives, content hashes included. - A loaded function node's head says how many results were kept in an ephemeral store and are not shown. - An operator node whose input changed shows NOT_COMPUTED, not its kept rows. - A CachedSource not yet used shows its columns and NOT_COMPUTED: its rows merge earlier cached rows with the wrapped source's and are unknown until read. - A derived source's rowless head keeps its origin's total and wording. - The preview shows all n rows of head(n); the end of a long string is read from the end of its buffer only. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 16 +++++++--- src/orcapod/core/nodes/function_node.py | 18 +++++++++-- src/orcapod/core/nodes/operator_node.py | 6 ++++ src/orcapod/core/sources/cached_source.py | 22 ++++++++------ src/orcapod/core/sources/derived_source.py | 17 +++++++---- src/orcapod/core/streams/head.py | 20 +++++++++++-- ...-itl-991-stream-head-and-preview-design.md | 5 ++++ .../streams/test_head_and_preview.py | 30 ++++++++++++++++++- tests/test_pipeline/test_node_head.py | 14 +++++++++ 9 files changed, 123 insertions(+), 25 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 6c1b0186a..c135e224b 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1,7 +1,6 @@ from __future__ import annotations import asyncio -import dataclasses import logging from abc import abstractmethod from collections.abc import Callable, Collection, Iterator, Sequence @@ -1187,9 +1186,18 @@ def shown_content_hashes() -> pa.Array[Any]: if not shown: # Nothing to show (an empty result, inputs that all gave no output so far, # or n == 0): the declared columns. - empty = self._head_without_rows(config, HeadStatus.ROWS) - return dataclasses.replace( - empty, status=status, total_rows=total, origin="computed" + table = self._shape_table( + self._table_from_pairs([]), + config, + lambda: pa.array([], type=pa.large_string()), + ) + return StreamHead.of_rows( + self, + table, + tag_columns=tag_columns, + total_rows=total, + status=status, + origin="computed", ) return StreamHead.of_rows( self, diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 6f47e95f3..acfc2cf7f 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -2985,11 +2985,25 @@ def _head_from_store(self, n: int, config: ColumnConfig) -> StreamHead: self._require_expected_records() fetched = self._fetch_joined_records() stored = None if fetched is None else self._stored_rows_stream(fetched) + stored_ids = set(stored[0]) if stored is not None else set() + missing = 0 + if fetched is not None: + missing = sum( + 1 for k in fetched.empty_data_tokens if k not in stored_ids + ) + note = ( + f"{missing:,} {'result' if missing == 1 else 'results'} not shown: kept " + "in an ephemeral store, which is not saved with the job" + if missing + else None + ) if stored is not None: return ( - stored[1].head(n, columns=config).for_stream(self, origin="stored") + stored[1] + .head(n, columns=config) + .for_stream(self, origin="stored", note=note) ) - return self._head_without_rows(config, HeadStatus.NOTHING_STORED) + return self._head_without_rows(config, HeadStatus.NOTHING_STORED, note=note) if self._cached_function_pod is None: return self._head_without_rows(config, HeadStatus.NOT_COMPUTED) entries = list(self._load_cached_entries().values()) diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 71b4aff20..9f22703ec 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -33,6 +33,7 @@ head_column_config, ) from orcapod.core.streams.head import ( + STALE_NOTE, BLUEPRINT_NOTE, HeadStatus, StreamHead, @@ -1333,6 +1334,11 @@ def head( check_head_size(n) config = head_column_config(columns, all_info) if self._cached_output_stream is not None: + if not self._loaded and self.is_stale: + # run() would recompute; the kept rows are not the current result. + return self._head_without_rows( + config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE + ) return head_of(self._cached_output_stream.head(n, columns=config), self) if self._cache_mode == CacheMode.REPLAY and self._pipeline_database is not None: return self._head_from_store(n, config) diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index 6138577de..c258bde6a 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -1,6 +1,5 @@ from __future__ import annotations -import dataclasses import logging from collections.abc import Iterator @@ -12,6 +11,7 @@ from orcapod.core.streams.arrow_table_stream import ArrowTableStream from orcapod.core.streams.base import schema_requires_io from orcapod.core.streams.head import ( + HeadStatus, StreamHead, head_of, ) @@ -426,22 +426,26 @@ def head( """Return the first rows without reading or writing the cache. Once the source has been used, its merged stream is in memory and this is a slice - of it. Before that, the rows are the wrapped source's head: the cache table is - read and written only on first use (``iter_data``, ``as_table``). + of it. Before that its rows are not known: they are the cache's earlier rows + merged with the wrapped source's, and reading them writes the cache. The head + then has the wrapped source's columns, no rows and ``NOT_COMPUTED`` (or the + wrapped source's own reason when it has none to give). """ if self._cached_stream is not None and not self.is_stale: return head_of( self._cached_stream.head(n, columns=columns, all_info=all_info), self ) wrapped = head_of( - self._source.head(n, columns=columns, all_info=all_info), self + self._source.head(0, columns=columns, all_info=all_info), self ) - if wrapped.table.num_rows == 0: - # Keep the wrapped source's own explanation of why there are no rows. + if wrapped.status is not HeadStatus.ROWS: return wrapped - return dataclasses.replace( - wrapped, - note="rows of the wrapped source; the cache is read and written on first use", + return StreamHead.without_rows( + self, + HeadStatus.NOT_COMPUTED, + wrapped.table, + tag_columns=wrapped.tag_columns, + note="not read yet; as_table() or iter_data() reads and updates the cache", ) def get_all_records( diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index a473943aa..1788afc6f 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -482,12 +482,17 @@ def head( # No rows: this source's own columns (tag copies included), with the # origin's status and explanation. empty = self._stream_over(origin_head.table.slice(0, 0)) - return StreamHead.without_rows( - self, - origin_head.status, - empty.as_table(columns=config), - tag_columns=empty.keys(columns=config)[0], - note=origin_head.note, + return dataclasses.replace( + StreamHead.without_rows( + self, + origin_head.status, + empty.as_table(columns=config), + tag_columns=empty.keys(columns=config)[0], + note=origin_head.note, + ), + total_rows=origin_head.total_rows, + total_is_estimate=origin_head.total_is_estimate, + description=origin_head.description, ) stream = self._stream_over(origin_head.table) return dataclasses.replace( diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index b40cba7e4..b3f542f21 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -269,7 +269,7 @@ def _repr_html_(self) -> str: parts.append(f"
{html.escape(tags, quote=False)}
") if self.table.num_columns: # Polars shortens strings to its own default length; show up to _MAX_TEXT. - with pl.Config(fmt_str_lengths=_MAX_TEXT): + with pl.Config(fmt_str_lengths=_MAX_TEXT, tbl_rows=-1): parts.append(self._display_frame()._repr_html_()) return "\n".join(parts) @@ -279,7 +279,7 @@ def __str__(self) -> str: if tags: lines.append(tags) if self.table.num_columns: - with pl.Config(fmt_str_lengths=_MAX_TEXT): + with pl.Config(fmt_str_lengths=_MAX_TEXT, tbl_rows=-1): lines.append(str(self._display_frame())) return "\n".join(lines) @@ -374,7 +374,10 @@ def _summarise(column: pa.ChunkedArray[Any]) -> list[str | None] | None: starts: list[str | None] = pc.utf8_slice_codeunits( column, 0, _TEXT_START ).to_pylist() - ends: list[str | None] = pc.utf8_slice_codeunits(column, -_TEXT_END).to_pylist() + ends = [ + None if n is None or n <= _MAX_TEXT else _text_end(column[i]) + for i, n in enumerate(lengths) + ] return [ None if n is None else f"{a}…{b}" if n > _MAX_TEXT else short for n, short, a, b in zip(lengths, shorts, starts, ends) @@ -498,6 +501,17 @@ def _shape_text(shape: Any) -> str: return "(" + ", ".join(str(d) for d in dims) + ")" +def _text_end(cell: Any) -> str: + """Return the last ``_TEXT_END`` characters of a string cell, reading only its end. + + A character is at most 4 bytes in UTF-8, so the last ``4 * _TEXT_END`` bytes hold + them; a character cut at the start of that window is dropped when decoding. + """ + buffer = cell.as_buffer() + window = buffer[max(0, buffer.size - 4 * _TEXT_END) :].to_pybytes() + return window.decode("utf-8", errors="ignore")[-_TEXT_END:] + + def _summarise_list( column: pa.ChunkedArray[Any], value_type: pa.DataType ) -> list[str | None] | None: diff --git a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md index 439f48ad6..209df2dc6 100644 --- a/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md +++ b/superpowers/specs/2026-11-12-itl-991-stream-head-and-preview-design.md @@ -402,3 +402,8 @@ renders `SpiralDBTableSource[spike_event] — first 10 rows of about 63,349`. `_head_note` (still the one place for its wording) as the header text, through `StreamHead.description`. An unlabelled Spiral source is shown by its table name. #333's display tests now go through `head()` / `header()`; their scan assertions are unchanged. +- CodeRabbit rounds 3-4: a `CachedSource` not yet used shows its columns and `NOT_COMPUTED` + (its rows merge earlier cached rows with the wrapped source's, unknown until read); an + operator node whose input changed shows `NOT_COMPUTED` ("inputs changed") rather than its + kept rows; an async operator run with no rows keeps a typed empty result; `head_of` adapts + any `StreamHeadProtocol` result; the preview shows all `n` rows (Polars `tbl_rows=-1`). diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 12dfb6764..972d39991 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -423,7 +423,14 @@ def test_cached_source_head_writes_nothing() -> None: head = source.head(3) _ = source._repr_html_() assert store.list_table_paths() == [] - assert head.table.column("x").to_pylist() == [0, 10, 20] + # Before first use its rows (earlier cached rows merged with the wrapped source's) + # are not known: the columns are shown, with no rows. + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.column_names == ["id", "x"] + table = source.as_table() + after = source.head(3) + assert after.status is HeadStatus.ROWS + assert after.table.equals(table.slice(0, 3)) # --------------------------------------------------------------------------- @@ -633,3 +640,24 @@ def _repr_html_(self) -> str: head = head_of(_PlainHead(), source) assert head.status is HeadStatus.ROWS assert head.header() == "ArrowTableSource[readings] — all 2 rows" + + +def test_head_of_n_rows_renders_all_n_rows() -> None: + """The preview of ``head(n)`` shows all ``n`` rows, not Polars' default of 10.""" + text = str(_source().head(25)) + assert "shape: (25, 2)" in text + assert "…" not in text.split("\n", 2)[2] + + +def test_an_empty_computed_result_has_the_requested_columns() -> None: + """A computed empty result has the columns ``as_table`` gives, hashes included.""" + calls: list[int] = [] + stream = _counting_pod(calls).pod(PolarsFilter(constraints={"x": -1})(_source())) + _ = list(stream.iter_data()) + head = stream.head(columns={"content_hash": True}) + assert head.status is HeadStatus.ROWS + assert len(head) == 0 + assert ( + head.table.column_names + == stream.as_table(columns={"content_hash": True}).column_names + ) diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 1115962da..7a71b2c17 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -14,6 +14,7 @@ import pytest from orcapod import function_pod +from orcapod.core.function_pod import FunctionPodStream from orcapod.core.nodes import OperatorJobNode from orcapod.core.nodes.source_node import SourceJobNode, SourceNode from orcapod.core.operators import SelectDataColumns @@ -239,3 +240,16 @@ def test_an_operator_node_with_an_empty_result_reads_as_computed( assert head.status is HeadStatus.ROWS assert len(head) == 0 assert head.table.column_names == ["id", "x"] + + +def test_an_operator_node_with_changed_inputs_shows_no_stale_rows() -> None: + """After an input changed, an operator node's kept rows are not shown.""" + fn_stream = triple.pod(_source()) + assert isinstance(fn_stream, FunctionPodStream) + node = OperatorJobNode(operator=SelectDataColumns(["y"]), input_streams=[fn_stream]) + _ = node.execute(fn_stream) + assert node.head().status is HeadStatus.ROWS + fn_stream.clear_cache() # the input is modified after the run + head = node.head() + assert head.status is HeadStatus.NOT_COMPUTED + assert "inputs changed" in head.header() From 53b2f9375294b440509f67643f06d39925d49e3d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 17:01:02 -0800 Subject: [PATCH 16/19] fix(streams): address CodeRabbit round 5 on head() (ITL-991) - A function pod's head over an input whose columns are known only by polling shows the computed result without columns, never calling keys() on that input. - CachedSource.head checks n before choosing a path. - Docstrings for the head.py helpers and the test helpers and doubles. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 19 +++++++++++++- src/orcapod/core/sources/cached_source.py | 3 ++- src/orcapod/core/streams/head.py | 10 +++++++ .../streams/test_head_and_preview.py | 26 +++++++++++++++++++ tests/test_pipeline/test_node_head.py | 5 ++++ 5 files changed, 61 insertions(+), 2 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index c135e224b..c7865a645 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1,6 +1,7 @@ from __future__ import annotations import asyncio +import dataclasses import logging from abc import abstractmethod from collections.abc import Callable, Collection, Iterator, Sequence @@ -16,8 +17,10 @@ StreamBase, check_head_size, head_column_config, + schema_requires_io, ) from orcapod.core.streams.head import ( + SCHEMA_NEEDS_POLL_NOTE, STALE_NOTE, HeadStatus, StreamHead, @@ -1157,7 +1160,21 @@ def head( return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE ) - # Rows exist only once the input was read, so the schema is known without I/O. + if has_results and schema_requires_io(self): + # The input gave no rows, so its columns are still known only by polling + # it: show the result as computed, without columns. + finished = not self._needs_iterator and self._cached_input_iterator is None + return dataclasses.replace( + StreamHead.without_rows( + self, + HeadStatus.ROWS if finished else HeadStatus.PARTIAL, + pa.table({}), + tag_columns=(), + note=SCHEMA_NEEDS_POLL_NOTE, + ), + total_rows=0, + origin="computed", + ) tag_columns = self.keys(columns=config)[0] if has_results else () if self._cached_output_table is not None: # A complete computation: its table holds the rows; only the shown rows' diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index c258bde6a..e1446fc85 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -9,7 +9,7 @@ from orcapod.config import OrcapodConfig from orcapod.core.sources.base import RootSource from orcapod.core.streams.arrow_table_stream import ArrowTableStream -from orcapod.core.streams.base import schema_requires_io +from orcapod.core.streams.base import check_head_size, schema_requires_io from orcapod.core.streams.head import ( HeadStatus, StreamHead, @@ -431,6 +431,7 @@ def head( then has the wrapped source's columns, no rows and ``NOT_COMPUTED`` (or the wrapped source's own reason when it has none to give). """ + check_head_size(n) if self._cached_stream is not None and not self.is_stale: return head_of( self._cached_stream.head(n, columns=columns, all_info=all_info), self diff --git a/src/orcapod/core/streams/head.py b/src/orcapod/core/streams/head.py index b3f542f21..a794fcc1a 100644 --- a/src/orcapod/core/streams/head.py +++ b/src/orcapod/core/streams/head.py @@ -198,6 +198,7 @@ def as_polars_df(self) -> pl.DataFrame: return pl.DataFrame(self.table) def __len__(self) -> int: + """Return the number of rows in ``table``.""" return self.table.num_rows # ------------------------------------------------------------------ @@ -209,6 +210,7 @@ def header(self) -> str: return f"{self.kind}[{self.label}] — {self.description or self._describe()}" def _describe(self) -> str: + """Return the header text after the name, for this status.""" shown = self.table.num_rows if self.status is HeadStatus.ROWS: text = self._describe_rows(shown) @@ -226,6 +228,7 @@ def _describe(self) -> str: return text def _describe_rows(self, shown: int) -> str: + """Return the header text for a ``ROWS`` head showing ``shown`` rows.""" total = self.total_rows noun = _ORIGIN_NOUNS.get(self.origin, "rows") if shown == 0 and (total is None or total == 0): @@ -256,11 +259,13 @@ def _display_frame(self) -> pl.DataFrame: return pl.DataFrame(pa.table(columns, names=table.column_names)) def _tags_line(self) -> str: + """Return the line naming the tag columns, or an empty string when there are none.""" if not self.tag_columns: return "" return "tags: " + ", ".join(self.tag_columns) def _repr_html_(self) -> str: + """Return the notebook preview: header, tag line and the rows (large cells summarised).""" parts = [ f"
{html.escape(self.header(), quote=False)}
" ] @@ -274,6 +279,7 @@ def _repr_html_(self) -> str: return "\n".join(parts) def __str__(self) -> str: + """Return the preview as text: header, tag line and the rows.""" lines = [self.header()] tags = self._tags_line() if tags: @@ -284,6 +290,7 @@ def __str__(self) -> str: return "\n".join(lines) def __repr__(self) -> str: + """Return a short description: kind, label, status and row count.""" return ( f"StreamHead({self.kind}[{self.label}], status={self.status.value}, " f"rows={self.table.num_rows})" @@ -340,6 +347,7 @@ def render_error_html(stream: Any, error: BaseException) -> str: def _rows(count: int) -> str: + """Return ``row`` or ``rows`` for ``count``.""" return "row" if count == 1 else "rows" @@ -495,6 +503,7 @@ def _struct_fields( def _shape_text(shape: Any) -> str: + """Return an array shape as Python writes a tuple: ``(40,)``, ``(2, 3)``.""" dims = list(shape or []) if len(dims) == 1: return f"({dims[0]},)" @@ -515,6 +524,7 @@ def _text_end(cell: Any) -> str: def _summarise_list( column: pa.ChunkedArray[Any], value_type: pa.DataType ) -> list[str | None] | None: + """Return display strings for a list column: short lists of scalars as written, others by length.""" simple_items = not ( isinstance(value_type, pa.ExtensionType) or pa.types.is_nested(value_type) diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 972d39991..9df2688a6 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -55,6 +55,7 @@ def _table(rows: int, offset: int = 0) -> pa.Table: def _source(rows: int = 25, label: str = "readings") -> ArrowTableSource: + """An ``ArrowTableSource`` over ``_table(rows)``, labelled ``label``.""" return ArrowTableSource( _table(rows), tag_columns=["id"], source_id="readings", label=label ) @@ -147,8 +148,11 @@ def test_preview_is_the_rendered_head() -> None: def _counting_pod(calls: list[int]) -> Any: + """A function pod tripling ``x`` that appends each input to ``calls``.""" + @function_pod(output_keys="y") def triple(x: int) -> int: + """Triple ``x``, recording the call.""" calls.append(x) return 3 * x @@ -215,6 +219,7 @@ def test_function_pod_stream_rows_from_iteration_only() -> None: def _other_source() -> ArrowTableSource: + """A source sharing ``id`` with ``_source``, with a ``z`` column for joins.""" table = pa.table( {"id": list(range(25)), "z": [i * i for i in range(25)]}, schema=pa.schema( @@ -228,6 +233,7 @@ def _other_source() -> ArrowTableSource: def _grouped_source() -> ArrowTableSource: + """A source with an ``id`` and a ``grp`` tag, for grouping.""" table = pa.table( { "id": list(range(25)), @@ -311,6 +317,7 @@ def test_side_effect_stream_preview_never_delivers() -> None: delivered: list[int] = [] def deliver(x: int) -> None: + """The side effect; recording or refusing a delivery.""" delivered.append(x) stream = SideEffectPod(deliver)(_source()) @@ -325,28 +332,35 @@ class _FakeDynamicSource: """A ``DynamicSourceProtocol`` double that records every poll and fetch.""" def __init__(self, declared: Schema | None = None) -> None: + """Set up the double.""" self.polls = 0 self.fetches = 0 self.declared = declared def identity(self) -> Any: + """Return the double's identity.""" return "fake" def to_config(self) -> dict[str, Any] | None: + """Return no config: the double is not serializable.""" return None @classmethod def from_config(cls, config: dict[str, Any]) -> _FakeDynamicSource: + """Refuse: the double is not serializable.""" raise NotImplementedError def schema(self) -> Any: + """Return the declared schema, if any.""" return self.declared async def poll(self, cursor: Cursor[int] | None = None) -> bool: + """Count the poll and say whether new data is there.""" self.polls += 1 return False async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + """Count the fetch and return the next batch.""" self.fetches += 1 return Cursor(value=1), { "id": pa.array([1], type=pa.int64()), @@ -354,6 +368,7 @@ async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], A } async def close(self) -> None: + """Release nothing.""" pass @@ -405,6 +420,7 @@ def test_downstream_of_an_unpolled_source_never_polls() -> None: source = PollingSource(fake, tag_columns="id") def deliver(val: int) -> None: + """The side effect; recording or refusing a delivery.""" raise AssertionError("delivered") stream = SideEffectPod(deliver)(source) @@ -494,21 +510,25 @@ class _NpyArrayType(pa.ExtensionType): """The array logical type stored as one ``.npy`` file per cell (the ITL-981 form).""" def __init__(self) -> None: + """Set up the double.""" from orcapod.logical_types.numpy_type import NDARRAY_EXTENSION_NAME super().__init__(pa.large_binary(), NDARRAY_EXTENSION_NAME) def __arrow_ext_serialize__(self) -> bytes: + """Return no extension metadata.""" return b"" @classmethod def __arrow_ext_deserialize__( cls, storage_type: pa.DataType, serialized: bytes ) -> _NpyArrayType: + """Rebuild the type.""" return cls() def _npy(array: npt.NDArray[Any]) -> bytes: + """Return ``array`` as the bytes of a ``.npy`` file (format 3.0).""" buffer = io.BytesIO() np.lib.format.write_array(buffer, array, version=(3, 0)) return buffer.getvalue() @@ -566,10 +586,12 @@ class _BatchedDynamicSource(_FakeDynamicSource): """Serves three batches of two rows, one per poll that reports new data.""" async def poll(self, cursor: Cursor[int] | None = None) -> bool: + """Count the poll and say whether new data is there.""" self.polls += 1 return (cursor.value if cursor is not None else 0) < 3 async def fetch(self, cursor: Cursor[int] | None = None) -> tuple[Cursor[int], Any]: + """Count the fetch and return the next batch.""" self.fetches += 1 index = cursor.value if cursor is not None else 0 return Cursor(value=index + 1), { @@ -625,15 +647,19 @@ class _PlainHead: total_rows = 2 def header(self) -> str: + """Return a fixed header.""" return "plain" def as_table(self) -> pa.Table: + """Return the rows.""" return self.table def as_polars_df(self) -> Any: + """Return nothing: not used by the test.""" return None def _repr_html_(self) -> str: + """Return an empty preview.""" return "" source = _source() diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 7a71b2c17..129f9a723 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -44,10 +44,12 @@ def triple(x: int) -> int: @pytest.fixture(autouse=True) def _reset_calls() -> None: + """Clear the record of calls before each test.""" CALLS.clear() def _source(rows: int = 25) -> ArrowTableSource: + """A source of ``rows`` rows with ``id`` and ``x``, both non-nullable ``int64``.""" table = pa.table( {"id": list(range(rows)), "x": list(range(rows))}, schema=pa.schema( @@ -61,12 +63,14 @@ def _source(rows: int = 25) -> ArrowTableSource: def _store(kind: str, tmp_path: Path) -> ArrowDatabaseProtocol: + """A fresh store of the given kind (``in_memory`` or ``delta``).""" if kind == "in_memory": return InMemoryArrowDatabase() return DeltaTableDatabase(base_path=tmp_path / "store") def _job(store: ArrowDatabaseProtocol) -> PipelineJob: + """A job of ``triple`` over the source, then a select of ``y``.""" job = PipelineJob(name="itl991", store=store) with job: tripled = triple.pod(_source(), label="triple") @@ -193,6 +197,7 @@ def test_source_nodes() -> None: def _operator_node( cache_mode: CacheMode, store: InMemoryArrowDatabase ) -> OperatorJobNode: + """An operator node selecting ``x`` from the source, on ``store``.""" return OperatorJobNode( operator=SelectDataColumns(["x"]), input_streams=[_source()], From 19ad76ce6ce81a62227428117b8fa3037982bab3 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 18:02:10 -0800 Subject: [PATCH 17/19] docs(plan): record the merge of origin/main into ITL-991 Co-Authored-By: Claude Opus 5.5 --- .../plans/2026-11-12-itl-991-stream-head.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md index 9e744fc0e..72954aca8 100644 --- a/superpowers/plans/2026-11-12-itl-991-stream-head.md +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -185,19 +185,25 @@ Header wording (exact; #333 depends on the estimate line): Resolution agreed in advance by the #334 and #335 workers (coordinator brief): -- [ ] `git merge origin/main` (merge commit, no rebase). Conflict expected in +- [x] `git merge origin/main` (merge commit, no rebase). Conflict expected in `core/nodes/function_node.py`; check `core/function_pod.py` for a duplicated `_table_from_pairs` left by the auto-merge. -- [ ] Keep #334's structure: `head`, `_shape_table`, `_computed_prefix`, `as_table` = +- [x] Keep #334's structure: `head`, `_shape_table`, `_computed_prefix`, `as_table` = `_table_from_pairs(list(self.iter_data()))` then `_shape_table`. -- [ ] Take main's (#335) `_table_from_pairs` bodies as the non-empty build in both classes +- [x] Take main's (#335) `_table_from_pairs` bodies as the non-empty build in both classes (per-row `tag.as_table` / `data.as_table` concatenation, `_concat_row_tables`, mixed Arrow types raise `InputValidationError`). -- [ ] Empty pairs: `FunctionPodStream._table_from_pairs` keeps #334's declared-columns branch +- [x] Empty pairs: `FunctionPodStream._table_from_pairs` keeps #334's declared-columns branch (`make_empty_table`); `FunctionNodeBase._table_from_pairs` keeps main's empty branch. -- [ ] If the resolution is not mechanical, ask the coordinator before pushing. -- [ ] Tests: #335's (tag types kept on both stores, mixed rows error, join after function +- [x] (not needed: the resolution was mechanical) If the resolution is not mechanical, ask the coordinator before pushing. +- [x] Tests: #335's (tag types kept on both stores, mixed rows error, join after function node), #334's (no computation on display, prefix invariant in every ROWS state), #328's array tests + ndarray `.npy` preview summary; full suite on 3.12, 3.11, 3.13; ruff, format, basedpyright; golden unchanged. - [ ] One push; CodeRabbit reviews the new head; answer every thread; worker_done with sha. + +Result: function_pod.py auto-merged into two `_table_from_pairs`; the #334 one was removed +and its empty branch moved into main's (the now-dead "no tag tables" fallback dropped). +function_node.py: #334's `_table_from_pairs` removed, main's kept in place. New test +`test_head_keeps_the_tag_arrow_type_of_the_input` (int8 tag, both orchestrators and stores). +Suite 10712 passed on 3.11, 3.12, 3.13; ruff, format, basedpyright exit 0; golden unchanged. From 36a8cc8493bef83554423c385fcc17b5308c468e Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 18:59:33 -0800 Subject: [PATCH 18/19] fix(streams): a rowless head never polls for data-dependent columns (ITL-991) A head with source info, meta, context or content hashes asked for, downstream of a PollingSource that declared its schema and has not polled yet, read the schema with those column groups and so fetched a batch. The schema-IO check now takes the head's column configuration: with no batches, a polling source answers that its schema needs a poll when it declared none or when the configuration asks for a group known only from rows. The head then shows no columns, with the existing note. Addresses CodeRabbit on 19ad76ce; also adds a docstring to a nested test helper. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/core/function_pod.py | 2 +- src/orcapod/core/nodes/source_node.py | 8 ++-- src/orcapod/core/sources/cached_source.py | 6 +-- src/orcapod/core/sources/derived_source.py | 6 +-- src/orcapod/core/sources/polling_source.py | 19 ++++++--- src/orcapod/core/streams/base.py | 24 +++++++---- .../plans/2026-11-12-itl-991-stream-head.md | 9 ++++ .../streams/test_head_and_preview.py | 42 +++++++++++++++++++ tests/test_pipeline/test_node_head.py | 1 + 9 files changed, 93 insertions(+), 24 deletions(-) diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 25bc331c9..2768b1e53 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -1203,7 +1203,7 @@ def head( return self._head_without_rows( config, HeadStatus.NOT_COMPUTED, note=STALE_NOTE ) - if has_results and schema_requires_io(self): + if has_results and schema_requires_io(self, config): # The input gave no rows, so its columns are still known only by polling # it: show the result as computed, without columns. finished = not self._needs_iterator and self._cached_input_iterator is None diff --git a/src/orcapod/core/nodes/source_node.py b/src/orcapod/core/nodes/source_node.py index e4cdbb20e..eaa888091 100644 --- a/src/orcapod/core/nodes/source_node.py +++ b/src/orcapod/core/nodes/source_node.py @@ -620,9 +620,11 @@ def iter_data(self) -> Iterator[tuple[TagProtocol, DataProtocol]]: ) return self._bound_source.iter_data() - def _schema_requires_io(self) -> bool: - """Whether reading the schema would poll: as for the bound source.""" - return self._bound_source is not None and schema_requires_io(self._bound_source) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the bound source.""" + return self._bound_source is not None and schema_requires_io( + self._bound_source, config + ) def head( self, diff --git a/src/orcapod/core/sources/cached_source.py b/src/orcapod/core/sources/cached_source.py index e1446fc85..4e3847172 100644 --- a/src/orcapod/core/sources/cached_source.py +++ b/src/orcapod/core/sources/cached_source.py @@ -412,9 +412,9 @@ def as_table( assert self._cached_stream is not None return self._cached_stream.as_table(columns=columns, all_info=all_info) - def _schema_requires_io(self) -> bool: - """Whether reading the schema would poll: as for the wrapped source.""" - return schema_requires_io(self._source) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the wrapped source.""" + return schema_requires_io(self._source, config) def head( self, diff --git a/src/orcapod/core/sources/derived_source.py b/src/orcapod/core/sources/derived_source.py index 26520c89b..e2c3b4291 100644 --- a/src/orcapod/core/sources/derived_source.py +++ b/src/orcapod/core/sources/derived_source.py @@ -445,9 +445,9 @@ def _get_stream(self) -> ArrowTableStream: tag_keys = self._origin.keys()[0] return ArrowTableStream(self._cached_table, tag_columns=tag_keys) - def _schema_requires_io(self) -> bool: - """Whether reading the schema would poll: as for the origin.""" - return schema_requires_io(self._origin) + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll: as for the origin.""" + return schema_requires_io(self._origin, config) def head( self, diff --git a/src/orcapod/core/sources/polling_source.py b/src/orcapod/core/sources/polling_source.py index 78689622a..b45b0a335 100644 --- a/src/orcapod/core/sources/polling_source.py +++ b/src/orcapod/core/sources/polling_source.py @@ -444,11 +444,20 @@ def keys( self._sync_poll_and_commit() return self._get_combined_stream().keys(columns=columns, all_info=all_info) - def _schema_requires_io(self) -> bool: - """True while no schema was declared and nothing was polled: only a poll - tells the columns then.""" - no_schema = self._tag_schema is None or self._data_schema is None - return no_schema and not self._batches + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading the schema with ``config`` would poll. + + True while nothing was polled and either no schema was declared, or ``config`` + asks for a column group known only from rows (source info, meta, context, + content hashes): only a poll tells those columns then. + """ + if self._batches: + return False + if self._tag_schema is None or self._data_schema is None: + return True + return bool( + config.meta or config.source or config.context or config.content_hash + ) def head( self, diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 8222cf25f..af13d5540 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -41,13 +41,18 @@ logger = logging.getLogger(__name__) -def schema_requires_io(stream: object) -> bool: - """Whether reading ``stream``'s schema would read or poll a source. +def schema_requires_io(stream: object, config: ColumnConfig) -> bool: + """Whether reading ``stream``'s schema with ``config`` would read or poll a source. A stream that does not say (``_schema_requires_io``) is taken not to. + + Args: + stream: The stream whose schema would be read. + config: The column groups the schema would be read with: source info, meta, + context and content hashes may be known only from rows. """ check = getattr(stream, "_schema_requires_io", None) - return bool(check()) if callable(check) else False + return bool(check(config)) if callable(check) else False def check_head_size(n: int) -> None: @@ -546,7 +551,7 @@ def _head_without_rows( When reading the schema would poll a source (``schema_requires_io``), the head has no columns either, and the note says so. """ - if schema_requires_io(self): + if schema_requires_io(self, config): return StreamHead.without_rows( self, status, @@ -566,13 +571,14 @@ def _display_name(self) -> str: """The name a preview header shows: the label. Never used as an identity.""" return self.label - def _schema_requires_io(self) -> bool: - """Whether reading this stream's schema would read or poll a source. + def _schema_requires_io(self, config: ColumnConfig) -> bool: + """Whether reading this stream's schema with ``config`` would read or poll a source. - True when an upstream's would: a ``PollingSource`` that declared no schema and - has not polled yet answers its schema only by polling. + True when an upstream's would: a ``PollingSource`` that has not polled yet + answers its schema only by polling when it declared no schema, or when + ``config`` asks for a column group known only from rows. """ - return any(schema_requires_io(upstream) for upstream in self.upstreams) + return any(schema_requires_io(upstream, config) for upstream in self.upstreams) def _repr_html_(self) -> str: """Render ``head()`` for a notebook; never computes.""" diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md index 72954aca8..ff1565e08 100644 --- a/superpowers/plans/2026-11-12-itl-991-stream-head.md +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -207,3 +207,12 @@ and its empty branch moved into main's (the now-dead "no tag tables" fallback dr function_node.py: #334's `_table_from_pairs` removed, main's kept in place. New test `test_head_keeps_the_tag_arrow_type_of_the_input` (int8 tag, both orchestrators and stores). Suite 10712 passed on 3.11, 3.12, 3.13; ruff, format, basedpyright exit 0; golden unchanged. + +## CodeRabbit round on 19ad76ce (coordinator approved a 2nd push: F3, F4 only) + +- [ ] F3: rowless head downstream of an unpolled declared-schema `PollingSource` fetched + when `source`/`meta`/`context`/`content_hash`/`all_info` was asked for. The private + schema-IO check takes the head's `ColumnConfig`; regression test first (zero polls and + fetches, `FunctionPodStream` and `SideEffectPodStream`). +- [ ] F4: docstring on the nested `build` helper in `test_node_head.py`. +- [ ] Decline F1 (ITL-1002), F2 (follow-up), linked-issues check (D2) in the threads. diff --git a/tests/test_core/streams/test_head_and_preview.py b/tests/test_core/streams/test_head_and_preview.py index 9df2688a6..2dac4d642 100644 --- a/tests/test_core/streams/test_head_and_preview.py +++ b/tests/test_core/streams/test_head_and_preview.py @@ -432,6 +432,48 @@ def deliver(val: int) -> None: assert "columns not known until its source is polled" in head.header() +@pytest.mark.parametrize( + ("columns", "all_info"), + [ + (ColumnConfig(source=True), False), + (ColumnConfig(meta=True), False), + (ColumnConfig(context=True), False), + (ColumnConfig(content_hash=True), False), + (None, True), + ], + ids=["source", "meta", "context", "content_hash", "all_info"], +) +@pytest.mark.parametrize("kind", ["function_pod", "side_effect"]) +def test_downstream_head_never_polls_for_data_columns( + kind: str, columns: ColumnConfig | None, all_info: bool +) -> None: + """Asking for data-dependent columns downstream of an unpolled source never polls. + + A declared schema gives the user columns, but source info, meta, context and + content hashes are known only from polled rows: the head then shows no columns. + """ + fake = _FakeDynamicSource(declared=Schema({"id": int, "val": int})) + source = PollingSource(fake, tag_columns="id") + + def deliver(val: int) -> None: + """The side effect; refusing a delivery.""" + raise AssertionError(f"delivered {val}") + + @function_pod(output_keys="out") + def copy(val: int) -> int: + """Return ``val``; refusing a call.""" + raise AssertionError(f"called with {val}") + + stream = ( + copy.pod(source) if kind == "function_pod" else SideEffectPod(deliver)(source) + ) + head = stream.head(columns=columns, all_info=all_info) + assert (fake.polls, fake.fetches) == (0, 0) + assert head.status is HeadStatus.NOT_COMPUTED + assert head.table.num_columns == 0 + assert "columns not known until its source is polled" in head.header() + + def test_cached_source_head_writes_nothing() -> None: """A cached source's head writes nothing to its cache store.""" store = InMemoryArrowDatabase() diff --git a/tests/test_pipeline/test_node_head.py b/tests/test_pipeline/test_node_head.py index 08934784c..028881ab0 100644 --- a/tests/test_pipeline/test_node_head.py +++ b/tests/test_pipeline/test_node_head.py @@ -281,6 +281,7 @@ def test_head_keeps_the_tag_arrow_type_of_the_input( ) def build(store: ArrowDatabaseProtocol) -> PipelineJob: + """Build a job of ``triple`` over the narrow-tag source on ``store``.""" job = PipelineJob(name="itl991_types", store=store) with job: _ = triple.pod( From a291e008c22a19ebaa54b04788a149c964f8c14f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 19:37:32 -0800 Subject: [PATCH 19/19] docs(plan): record the CodeRabbit round on 19ad76ce Co-Authored-By: Claude Opus 5.5 --- superpowers/plans/2026-11-12-itl-991-stream-head.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/superpowers/plans/2026-11-12-itl-991-stream-head.md b/superpowers/plans/2026-11-12-itl-991-stream-head.md index ff1565e08..ea827f3c2 100644 --- a/superpowers/plans/2026-11-12-itl-991-stream-head.md +++ b/superpowers/plans/2026-11-12-itl-991-stream-head.md @@ -210,9 +210,12 @@ Suite 10712 passed on 3.11, 3.12, 3.13; ruff, format, basedpyright exit 0; golde ## CodeRabbit round on 19ad76ce (coordinator approved a 2nd push: F3, F4 only) -- [ ] F3: rowless head downstream of an unpolled declared-schema `PollingSource` fetched +- [x] F3: rowless head downstream of an unpolled declared-schema `PollingSource` fetched when `source`/`meta`/`context`/`content_hash`/`all_info` was asked for. The private schema-IO check takes the head's `ColumnConfig`; regression test first (zero polls and fetches, `FunctionPodStream` and `SideEffectPodStream`). -- [ ] F4: docstring on the nested `build` helper in `test_node_head.py`. +- [x] F4: docstring on the nested `build` helper in `test_node_head.py`. - [ ] Decline F1 (ITL-1002), F2 (follow-up), linked-issues check (D2) in the threads. + +Also merged origin/main #337, #338 and #286 (no conflicts; golden and contexts identical to +main). Suite 11029 passed on 3.11, 3.12, 3.13; basedpyright exit 0.