Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions superpowers/plans/2026-11-13-itl-1061-rerun-cost.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# ITL-1061 re-run cost investigation plan

> **For agentic workers:** investigation only. No change under `src/` or `tests/`. Scripts go
> under `superpowers/specs/itl-1061/`, the report goes in `superpowers/specs/2026-11-13-itl-1061-rerun-cost.md`.

**Goal:** Pinpoint, with counters and file:line references, where a second run of an unchanged
pipeline does work again, measured against the developer's model (each node holds a timestamped
in-memory cache; a re-run with no newer upstream is instantaneous).

**Architecture:** One instrumentation module (`counters.py`) monkeypatches the library at run time
and counts calls per node. One driver (`rerun_counts.py`) builds a workshop-shaped pipeline (two
`ArrowTableSource`s of a few thousand rows, drop_nulls, group_by, map_tags, join, two function pods)
and runs it through each path twice, printing a table of counts per run.

**Tech stack:** `uv run python`, pyarrow, orcapod at `origin/main` (97263e5b); optionally PR #355
(963310fb) in a scratch worktree.

**Constraints:** one python process at a time, small inputs, no test suites, no live Spiral.

---

## Task 1: Read the code paths
- [x] Orchestrators: `pipeline/sync_orchestrator.py`, `pipeline/async_orchestrator.py`, `pipeline/job.py`, `pipeline/graph.py`
- [x] Nodes: `core/nodes/source_node.py`, `operator_node.py`, `function_node.py`
- [x] Streams: `core/streams/base.py` (`is_stale`, `last_modified`), `core/static_output_pod.py`, `core/function_pod.py`
- [x] Sources: `arrow_table_source.py`, `dict_source.py`, `csv_source.py`, `delta_table_source.py`, Spiral source, polling sources

## Task 2: Counters and driver
- [x] `superpowers/specs/itl-1061/counters.py`: patch functions with wrappers that increment `Counter[(label, node)]`
- [x] `superpowers/specs/itl-1061/rerun_counts.py`: paths (a) lazy streams read twice, (b) `PipelineJob.run()` twice sync and async, (c) new job same store, (d) `Pipeline.run()`
- [x] Run each once (`uv run python superpowers/specs/itl-1061/rerun_counts.py <path>`), save output under `superpowers/specs/itl-1061/out/`
- [x] Commit WIP and push

## Task 3: PR #355 comparison (lazy path only)
- [x] Scratch worktree at 963310fb, run path (a) there, record commit

## Task 4: History
- [x] `git log -S is_stale`, `-S _cached_output_stream`, `--follow` on orchestrators / nodes; v0.1.x tags

## Task 5: Headline findings to coordinator, then the report (<500 lines), then PR (docs only, Refs ITL-1061)

## Notes for a successor
- Done 2026-11-13: report at `superpowers/specs/2026-11-13-itl-1061-rerun-cost.md`, raw output in
`superpowers/specs/itl-1061/out/`. Branch merged with origin/main 89160c47 (PR #355 included).
- Headline findings sent to the coordinator (orchestration status message).
- The developer decided the job-path fix lands after the v0.3.0 tag.
455 changes: 455 additions & 0 deletions superpowers/specs/2026-11-13-itl-1061-rerun-cost.md

Large diffs are not rendered by default.

254 changes: 254 additions & 0 deletions superpowers/specs/itl-1061/counters.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
"""Count what a pipeline run does, per node, by monkeypatching orcapod at run time (ITL-1061).

Investigation aid only: nothing under ``src/`` is changed. ``install()`` wraps library
functions with counting wrappers; every count is attributed to the node whose
``execute`` / ``async_execute`` is running (a ``ContextVar``, so async tasks keep their
own node), or to ``-`` outside any node (lazy streams, the orchestrator itself).

Metrics (one row each in the report):

- ``node.execute``: calls of a node's ``execute`` / ``async_execute``.
- ``rows.out``: rows a sync ``execute`` returned (the orchestrator's buffer).
- ``source.iter``: ``iter_data`` calls on a concrete source (a read of the source).
- ``spiral.scan``: scans a ``SpiralDBTableSource`` started (``_batch_streams``).
- ``record_id``: per-row source record ids built (``stream_builder._make_record_id``).
- ``base_entry_id``: per-row function-node entry ids (``compute_base_entry_id``).
- ``hash_table``: Arrow table hashes (``SemanticArrowHasher.hash_table``); per-row
data hashes, entry ids and source table hashes all go through it.
- ``Tag.new`` / ``Data.new``: datagram objects constructed.
- ``ats.new``: ``ArrowTableStream`` objects constructed.
- ``ats.iter.build`` / ``ats.iter.cached``: ``ArrowTableStream.iter_data`` calls that
built per-row Tag/Data objects vs. served its ``_cached_elements``.
- ``op.static_process``: operator computations (``static_process`` of any operator).
- ``op.process``: ``StaticOutputOperatorPod.process`` (a new ``DynamicPodStream``).
- ``rows_to_table`` / ``materialize_to_stream`` / ``hstack_tables``: table rebuilds.
- ``fn.call``: user function bodies run (counted by the driver's functions).
- ``db.read`` / ``db.write`` / ``db.flush``: store calls.
"""

from __future__ import annotations

import contextvars
import functools
import inspect
from collections import Counter, defaultdict
from typing import Any, Callable

NODE: contextvars.ContextVar[str] = contextvars.ContextVar("node", default="-")
COUNTS: defaultdict[str, Counter[str]] = defaultdict(Counter)


def bump(metric: str, n: int = 1) -> None:
"""Add ``n`` to ``metric`` for the node now running."""
COUNTS[NODE.get()][metric] += n


def reset() -> None:
"""Forget every count."""
COUNTS.clear()


def snapshot() -> dict[str, dict[str, int]]:
"""Return a copy of the counts, per node."""
return {node: dict(c) for node, c in COUNTS.items()}


def _wrap(owner: Any, name: str, metric: str | Callable[..., str]) -> None:
"""Replace ``owner.name`` with a wrapper that counts each call."""
raw = owner.__dict__[name]
is_static = isinstance(raw, staticmethod)
func = raw.__func__ if is_static else raw

if inspect.isgeneratorfunction(func):

@functools.wraps(func)
def gen_wrapper(*args, **kwargs):
bump(metric if isinstance(metric, str) else metric(*args, **kwargs))
yield from func(*args, **kwargs)

wrapper: Callable[..., Any] = gen_wrapper
else:

@functools.wraps(func)
def call_wrapper(*args, **kwargs):
bump(metric if isinstance(metric, str) else metric(*args, **kwargs))
return func(*args, **kwargs)

wrapper = call_wrapper

setattr(owner, name, staticmethod(wrapper) if is_static else wrapper)


def _wrap_node(owner: Any, name: str) -> None:
"""Count a node's ``execute`` / ``async_execute`` and attribute nested counts to it."""
func = owner.__dict__[name]
if inspect.iscoroutinefunction(func):

@functools.wraps(func)
async def async_wrapper(self, *args, **kwargs):
token = NODE.set(self.label)
try:
bump("node.execute")
return await func(self, *args, **kwargs)
finally:
NODE.reset(token)

setattr(owner, name, async_wrapper)
return

@functools.wraps(func)
def wrapper(self, *args, **kwargs):
token = NODE.set(self.label)
try:
bump("node.execute")
out = func(self, *args, **kwargs)
if isinstance(out, list):
bump("rows.out", len(out))
return out
finally:
NODE.reset(token)

setattr(owner, name, wrapper)


def install() -> None:
"""Patch the library with the counting wrappers (idempotent per process: call once)."""
from orcapod.core.datagrams.tag_data import Data, Tag
from orcapod.core.nodes import function_node, operator_node, source_node
from orcapod.core.operators import static_output_pod as sop
from orcapod.core.operators.base import (
BinaryOperator,
NonZeroInputOperator,
UnaryOperator,
)
from orcapod.core.operators.join import Join
from orcapod.core.sources import stream_builder
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.spiraldb_table_source import SpiralDBTableSource
from orcapod.core.streams.arrow_table_stream import ArrowTableStream
from orcapod.databases.delta_lake_databases import DeltaTableDatabase
from orcapod.databases.in_memory_databases import InMemoryArrowDatabase
from orcapod.hashing import arrow_hashers
from orcapod.utils import arrow_utils

node_classes = [
getattr(module, name)
for module in (function_node, operator_node, source_node)
for name in dir(module)
if name.endswith(("Node", "NodeBase"))
and isinstance(getattr(module, name), type)
and getattr(module, name).__module__ == module.__name__
]
for cls in node_classes:
for name in ("execute", "async_execute"):
if name in cls.__dict__:
_wrap_node(cls, name)

_wrap(Tag, "__init__", "Tag.new")
_wrap(Data, "__init__", "Data.new")
_wrap(ArrowTableStream, "__init__", "ats.new")
_wrap(
ArrowTableStream,
"iter_data",
lambda self: (
"ats.iter.cached" if self._cached_elements is not None else "ats.iter.build"
),
)
_wrap(RootSource, "iter_data", "source.iter")
_wrap(SpiralDBTableSource, "iter_data", "source.iter")
_wrap(SpiralDBTableSource, "_batch_streams", "spiral.scan")

original_record_id = stream_builder._make_record_id

def counted_record_id(*args, **kwargs):
bump("record_id")
return original_record_id(*args, **kwargs)

stream_builder._make_record_id = counted_record_id

for cls in node_classes:
if "compute_base_entry_id" in cls.__dict__:
_wrap(cls, "compute_base_entry_id", "base_entry_id")
for cls_name in dir(arrow_hashers):
cls = getattr(arrow_hashers, cls_name)
if (
isinstance(cls, type)
and cls.__module__ == arrow_hashers.__name__ # not starfix's digester
and "hash_table" in cls.__dict__
):
_wrap(cls, "hash_table", "hash_table")

for cls in (UnaryOperator, BinaryOperator, NonZeroInputOperator, Join):
if "static_process" in cls.__dict__:
_wrap(cls, "static_process", "op.static_process")
_wrap(sop.StaticOutputOperatorPod, "process", "op.process")
_wrap(sop.StaticOutputOperatorPod, "_rows_to_table", "rows_to_table")
_wrap(
sop.StaticOutputOperatorPod, "_materialize_to_stream", "materialize_to_stream"
)

original_hstack = arrow_utils.hstack_tables

def counted_hstack(*args, **kwargs):
bump("hstack_tables")
return original_hstack(*args, **kwargs)

arrow_utils.hstack_tables = counted_hstack

for db in (InMemoryArrowDatabase, DeltaTableDatabase):
for name in ("add_record", "add_records"):
if name in db.__dict__:
_wrap(db, name, "db.write")
for name in (
"get_all_records",
"get_records_by_ids",
"get_record_by_id",
"get_records_with_column_value",
):
if name in db.__dict__:
_wrap(db, name, "db.read")
if "flush" in db.__dict__:
_wrap(db, "flush", "db.flush")


METRICS = (
"node.execute",
"rows.out",
"source.iter",
"spiral.scan",
"record_id",
"base_entry_id",
"hash_table",
"Tag.new",
"Data.new",
"ats.new",
"ats.iter.build",
"ats.iter.cached",
"op.process",
"op.static_process",
"rows_to_table",
"materialize_to_stream",
"hstack_tables",
"fn.call",
"db.read",
"db.write",
"db.flush",
)


def report(title: str, counts: dict[str, dict[str, int]]) -> str:
"""Format ``counts`` as a table: one column per node, one row per metric."""
nodes = sorted(counts, key=lambda n: (n == "-", n))
width = max(12, *(len(n) for n in nodes)) if nodes else 12
lines = [f"### {title}", ""]
header = f"{'metric':<22}" + "".join(f"{n:>{width + 2}}" for n in nodes)
header += f"{'TOTAL':>{width + 2}}"
lines.append(header)
for metric in METRICS:
values = [counts[n].get(metric, 0) for n in nodes]
if not any(values):
continue
row = f"{metric:<22}" + "".join(f"{v:>{width + 2}}" for v in values)
row += f"{sum(values):>{width + 2}}"
lines.append(row)
return "\n".join(lines) + "\n"
97 changes: 97 additions & 0 deletions superpowers/specs/itl-1061/node_state.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
"""Show each job node's in-memory state and staleness after one and two runs (ITL-1061).

uv run python superpowers/specs/itl-1061/node_state.py

For each node: its kind, ``last_modified``, ``is_stale``, its upstreams (the objects its
staleness is computed against), and the in-memory result it keeps (operator
``_cached_output_stream`` rows, function ``_cached_output_datas`` entries). Also shows what a
re-bound source does to the source node's modified time. Private attributes are read here
only because what they hold is the subject of the investigation.
"""

from __future__ import annotations

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
sys.path.insert(0, str(Path(__file__).resolve().parent))

import rerun_counts # noqa: E402 # pyright: ignore[reportImplicitRelativeImport] # a sibling script, run from this folder's path

from orcapod import PipelineJob # noqa: E402
from orcapod.databases import InMemoryArrowDatabase # noqa: E402


def describe(job: PipelineJob, title: str) -> None:
print(f"### {title}")
for label in (
"sessions",
"experiments",
"drop_nulls",
"group_by",
"map_tags",
"join",
"accuracy",
"grade",
):
node = job.nodes[label]
kept = ""
if hasattr(node, "_cached_output_stream"):
s = node._cached_output_stream
kept = f"_cached_output_stream={'None' if s is None else type(s).__name__}"
if hasattr(node, "_cached_output_datas"):
kept = f"_cached_output_datas={len(node._cached_output_datas)}"
if hasattr(node, "bound_source"):
kept = f"bound_source={type(node.bound_source).__name__} (src last_modified {node.bound_source.last_modified:%H:%M:%S.%f})"
ups = ",".join(type(u).__name__ for u in node.upstreams)
lm = node.last_modified
print(
f"{label:<12} {type(node).__name__:<16} last_modified="
f"{'None' if lm is None else format(lm, '%H:%M:%S.%f')} is_stale={getattr(node, 'is_stale', 'n/a')!s:<5} "
f"upstreams=[{ups}] {kept}"
)
print()


sources = rerun_counts.arrow_sources()
job = PipelineJob(name="ws", store=InMemoryArrowDatabase())
with job:
rerun_counts.build(sources[0], sources[1], rerun_counts.make_pods())
describe(job, "after compile, before any run")
job.run()
describe(job, "after job.run() #1")
job.run()
describe(job, "after job.run() #2")
# A source whose data really changed: experiments with different ``correct`` values.
import pyarrow as pa # noqa: E402
import pyarrow.compute as pc # noqa: E402

from orcapod.sources import ArrowTableSource # noqa: E402

_, experiments = rerun_counts.tables()
changed = experiments.set_column(
experiments.schema.get_field_index("correct"),
"correct",
pc.subtract(pa.scalar(10, pa.int64()), experiments["correct"]),
)
before = job.nodes["accuracy"].as_table()["accuracy"].to_pylist()
job.bind(
sources={
"experiments": ArrowTableSource(
changed,
tag_columns=["subject_id", "session", "experiment_index"],
source_id="experiments",
)
}
)
describe(job, "after job.bind(sources={'experiments': same id, CHANGED rows})")
would_skip = [
label
for label in ("drop_nulls", "group_by", "map_tags", "join", "accuracy", "grade")
if not job.nodes[label].is_stale
]
print("a skip on 'not is_stale and holds a result' would skip:", would_skip)
job.run()
after = job.nodes["accuracy"].as_table()["accuracy"].to_pylist()
print("today's run #3 picked up the change:", before != after)
Loading
Loading