Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions changelog.d/ITL-1065.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
### Fixed

#### Two reads of a pod stream open at once compute each row once (ITL-1065)

With two `iter_data()` iterators open on one function pod stream fed by an operator stream
(for example the column selection a pod call adds when its input has a column the function
does not take), the second iterator threw away the rows the first had computed and computed
them again: one call too many, and a row the first iterator had already handed out was no
longer the stream's row. A read left partway and read again did the same. Such reads now share
the stream's results, whatever feeds it: each row is computed once and every read yields the
same rows. An upstream changed with `clear_cache()` between the reads still makes the next
read start again. (ITL-1065)
3 changes: 3 additions & 0 deletions design_issues/ITL-1052-F18.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ so a chain through an operator was not affected; a pod reading a pod directly wa
its results are complete, that is when its input is read to the end, before its reader sees
the end; never when a read starts. `clear_cache()` stamps (a reset is a change). A failed or
partial read does not stamp.
Between a read's first row and its end, the stream is judged by its input's noted state, not
by modified-time order: an upstream completing for the first time during the read is not a
change (F19, ITL-1065).

**Fix:** the stamp moved from `_ensure_iterator` to the end of the input, on the sequential
and the concurrent read paths. A reset of the input during a read must not be reported as
Expand Down
58 changes: 58 additions & 0 deletions design_issues/ITL-1065-F19.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
## `src/orcapod/core/function_pod.py`

### F19 — Interleaved reads of a pod stream fed by an operator stream recomputed a yielded row
**Status:** resolved
**Severity:** high
**Issue:** ITL-1065

A regression from F18 (PR #355). With two iterators open on one function pod stream whose
input is an operator stream (for example the automatic `SelectDataColumns` that `pod(src)`
inserts when `src` has a data column the function does not take), the second iterator dropped
the stream's results and computed them again: 7 calls of the function for 6 rows, and the row
the first iterator had already yielded was no longer the stream's row. A read left after one
row and then read again did the same.

**Cause** (modified times printed at each step): the first read found the stream stale (its
input, the operator stream, had no modified time before it ran), so `clear_cache()` stamped
the stream at t1; then opening the input made the operator stream compute and take t2 > t1.
The stream stamps again only when its input is read to the end (F18), so until then it was
older than its input: `last_modified` was `None` and `is_stale` True in the middle of the read,
and the next `iter_data()` cleared it. Before F18 the stream stamped right after opening (and
so computing) the operator input, which hid this. A source is older than the stream, and a
pod-stream input stamps only at its end, so neither showed it directly; a pod stream over an
operator-fed pod stream did.

**Rule:** a pod stream's results are judged in one of two ways.

- Completed results (the input read to the end, or nothing read since a reset) are current
while neither a direct upstream nor the pod is newer or without a modified time (F18).
- Results in progress (a read has noted its input's state and not reached the input's end) are
current while that state is unchanged, the input is not stale itself and the pod did not
change. The state is the reset generation of a pod-stream input, and the modified time of
any other input, noted when its first row (or its end) arrives. An upstream completing, or
computing, for the first time during the read is not a change of the stream's input. A
pod-stream input is also asked whether it is stale, because its generation does not move
when a stream further up is reset; any other input's modified time already reflects its
upstreams.
- A read in progress ends when any read reaches the input's end (the stream stamps, or clears
its modified time if the input changed) or when the stream is reset (`clear_cache()`, also
the one a stale stream's next read makes). An abandoned iterator does not end it: the
stream keeps its partial results and its open input, and the next read resumes it
(ITL-1039) unless the input changed, in which case that read finds the stream stale and
starts again.

An operator stream needs no reset generation of its own: it computes when it is opened, so by
the time the pod stream notes its modified time it has computed, and that time moves only when
it is invalidated or recomputed, which are real changes.

**Fix:** `FunctionPodStream.is_stale` applies the in-progress rule while a noted input state
exists; `_input_read_to_end` ends the read in progress after stamping (or clearing) the
modified time. A reset upstream during a read still changes the noted state: the next read of
the stream starts again, and the stream ends a read that saw the reset stale (F18 unchanged).
In-memory bookkeeping only: no hash, saved file or store sees it. Regressions:
`tests/test_core/function_pod/test_pod_stream_interleaved_reads.py` (source, operator, pod
stream and pod-over-operator inputs; sequential and concurrent paths; plain and cached pods;
interleaved reads, an abandoned partial read, a reset one or two links up between
interleaved reads or after an abandoned read, `head()` after an abandoned read (partial, not
stale), and `as_table()` or `head()` while an iterator is
open).
4 changes: 4 additions & 0 deletions docs/concepts/streams.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@ run: `head()`, `as_table()` and `iter_data()` serve them from memory, without re
store (a function node whose run left no rows in memory reads its stored rows). Likewise, once a chain of streams such as `g.pod(f.pod(source))` has been read to the
end, reading it again computes nothing. A change upstream (`clear_cache()`) makes every stream
after it out of date, and its next read computes it again.
Two reads of one stream open at the same time, or a read that picks up one left partway, share
the stream's results: each row is computed once and every read yields the same rows, whatever
feeds the stream. An upstream finishing its first computation during the read is not a change;
a `clear_cache()` upstream between the reads is, and the next read starts again.

In the preview, large cells are summarised (an array as `ndarray float64 (2, 3)`, bytes as
`bytes (300 B)`, long text and long lists shortened); `head().table` keeps the full values.
Expand Down
28 changes: 28 additions & 0 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -1109,6 +1109,34 @@ def _input_read_to_end(self) -> None:
self._set_modified_time(None)
else:
self._update_modified_time()
# The read is complete: from now on the stream is judged by its modified time.
self._input_state = None

@property
def is_stale(self) -> bool:
"""Whether the stream's results must be discarded before it is read again.

Results in progress (a read has noted its input's state and not reached the
input's end) stay current while that state is unchanged, the input is not
stale itself and the pod did not change: an upstream completing or computing for
the first time while the stream is read is not a change of its input (ITL-1065).
A pod-stream input is asked whether it is stale because its noted state, the
reset generation, does not move when a stream further up is reset; any other
input's modified time already reflects its upstreams. Otherwise the stream is
stale when it, an upstream or its pod has no modified time or an upstream or its
pod is newer (``StreamBase.is_stale``).
"""
if self._input_state is None:
return super().is_stale
current = self._current_input_state()
if current != self._input_state or None in current:
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
return True
upstream = self._input_stream
if isinstance(upstream, FunctionPodStream) and upstream.is_stale:
return True
own_time = self._modified_time
pod_time = self._function_pod.last_modified
return own_time is None or pod_time is None or pod_time > own_time

def clear_cache(self) -> None:
"""Discard all in-memory cached state."""
Expand Down
127 changes: 127 additions & 0 deletions superpowers/plans/2026-11-13-itl-1065-interleaved-reads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# ITL-1065 Interleaved reads of a pod stream recompute a yielded row — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan
> task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Interleaved and resumed reads of a function pod stream call the function once per
input row and yield the same datagrams as a later full read, whatever its upstream kind, while
every staleness guarantee of PR #355 (ITL-1042, ITL-1052) still holds.

**Architecture:** `FunctionPodStream.is_stale` gets two rules: completed results are judged by
modified-time order (as now); results in progress (a read has noted its input's state and has
not reached the end) are judged by whether the input's noted state changed. In-memory
bookkeeping only.

**Tech Stack:** Python 3.12, pytest, `uv run`.

---

## Cause (confirmed on 89160c47, probe in the session scratchpad)

`s = f.pod(DictSource(rows with an extra column))` reads an automatic `SelectDataColumns`
stream (`DynamicPodStream`, last_modified `None` until it runs).

```
start s.own=.655051 s.lm=None gen=0 stale=True up.lm=None calls=0
next(first) s.own=.655087 s.lm=None gen=1 stale=True up.lm=.655445 calls=1
3 x next(second) s.own=.656093 s.lm=.656093 gen=2 stale=False up.lm=.655445 calls=4
drain first/second/third: calls=7, first read's row 0 is not the third read's
```

1. `first`'s `iter_data` sees the stream stale (input time unknown), `clear_cache()` stamps
`s` at t1 (generation 1).
2. `_ensure_iterator` opens the operator stream, which computes now and takes t2 > t1.
3. Until the read completes, `s` is older than its input, so `last_modified` is `None` and
`is_stale` is True mid-read. `second`'s `iter_data` clears the cache (generation 2) and
recomputes row 0, which `first` had already yielded.

Before #355, `_ensure_iterator` stamped `s` after opening (and so computing) the operator
input, which hid this. Without the extra column the input is a source, older than `s`.

## Rule after the fix

- Completed results (input read to the end and stamped, or nothing read since a reset) are
current while no direct upstream and not the pod is newer: modified-time order, as #355.
- Results in progress (a read has noted its input's state, the end not reached) are current
while the input's state is the noted one and the pod did not change: a pod-stream input's
reset generation; any other input's modified time, noted once it has delivered a row (an
operator stream computes when opened, so its time then moves only when it is recomputed or
invalidated). An upstream completing for the first time during the read is not a change.
- So the operator stream needs no reset generation: its modified time, noted after it has
computed, is one.
- A reset during a read still changes the noted state: the next read of the stream clears it,
and the stream ends its read stale (unchanged from #355).

## Files

- Modify: `src/orcapod/core/function_pod.py` (`FunctionPodStream`: `is_stale` override,
`_input_read_to_end` ends the read in progress).
- Create: `tests/test_core/function_pod/test_pod_stream_interleaved_reads.py`.
- Create: `design_issues/ITL-1065-F19.md`; update `design_issues/ITL-1052-F18.md` (rule).
- Create: `changelog.d/ITL-1065.md`.

### Task 1: Failing tests

Public interface only: `iter_data`, `is_stale`, `clear_cache`, call counts, `is` identity of
yielded datagrams.

Upstream kinds: `source` (`pod.process(src)`), `operator` (`pod(src_with_extra_column)`, the
automatic select), `pod_stream` (`g.pod(f.pod(src))`), `pod_over_operator`
(`g.pod(f.pod(src_with_extra))`). Executors: sequential, concurrent. Pods: plain, cached
(`CachedFunctionPod`, fresh in-memory store).

- [ ] `test_interleaved_reads_compute_each_row_once`: `first`, `second` iterators; one row from
`first`, three from `second`; drain both; a third full read. Every function called once per
row; the three reads yield the same datagram objects; nothing stale.
- [ ] `test_a_read_after_an_abandoned_partial_read_computes_each_row_once`: one row, abandon,
full read: once per row, same row 0 object.
- [ ] `test_a_reset_between_interleaved_reads_gives_one_state`: stamped (epoch) upstream pod,
directly or through a join + automatic select; reset it between the two iterators; drain;
third read is all epoch 1, derived from the upstream's rows, computes nothing, not stale.
- [ ] Run, expect `operator` / `pod_over_operator` sequential cases to fail (7 calls).
- [ ] WIP commit.

### Task 2: Fix

```python
@property
def is_stale(self) -> bool:
if self._input_state is None:
return super().is_stale
current = self._current_input_state()
if current != self._input_state or None in current:
return True
own = self._modified_time
pod_time = self._function_pod.last_modified
return own is None or pod_time is None or pod_time > own
```

`_input_read_to_end` sets `self._input_state = None` after stamping (the read is no longer in
progress). `clear_cache` already clears it.

- [ ] Implement; run the new tests, `test_pod_stream_staleness.py`,
`tests/test_pipeline/test_node_staleness_after_run.py`, `tests/test_core/function_pod`,
`tests/test_core/streams` (one pytest process, Python 3.12).
- [ ] Re-run the probe: 6 calls.
- [ ] Commit.

### Task 3: Docs and records

- [ ] `design_issues/ITL-1065-F19.md` (cause, rule, fix); F18 rule paragraph points to F19.
- [ ] `changelog.d/ITL-1065.md` under `### Fixed`.
- [ ] `uv run python scripts/design_issues.py --check`, `scripts/assemble_changelog.py --check`,
ruff, basedpyright.
- [ ] Commit, merge origin/main, push, PR (Fixes ITL-1065), Copilot review.

## Notes for a successor

- Status (2026-11-13): tests (b5247136) failed 8 sequential cases before the fix; fix in
`FunctionPodStream.is_stale` / `_input_read_to_end`; targeted suites pass (1472); repro and
the ITL-1034 suite's interleaved cases (`--runxfail`, 117) pass. Docs, F19, F18 pointer and
changelog fragment written. Next: merge origin/main, push, PR, Copilot.
- Coordinator follow-up (msg_7bf21fb00ea3): added `as_table`/`head` during an open read and
an abandoned read followed by a reset. The latter found a gap two links up (`s <- s1 <- p`,
only `s1`'s generation noted): the in-progress rule now also asks a pod-stream input whether
it is stale. Merge origin/main after #358 (as_table clears a stale cache) and rerun.
- The ITL-1034 branch's strict `ITL_1065` xfail marks will XPASS once this merges: that branch
must drop them.
Loading
Loading