diff --git a/changelog.d/ITL-1111.md b/changelog.d/ITL-1111.md new file mode 100644 index 000000000..745a398f0 --- /dev/null +++ b/changelog.d/ITL-1111.md @@ -0,0 +1,7 @@ +### Fixed + +- Defining a pipeline whose stages split a stream into two branches and join them again no longer takes time exponential in the number of stages (about 4x per stage before: 9 stages took 13 seconds to build, with no row computed; 20 stages now build in well under a second, as streams and inside a `PipelineJob`) (ITL-1111). Three graph walks asked a shared upstream once per path: every operator stream, function pod stream and node predicted its output schema again from its upstreams on each call, and a join asks each input several times; `head()` and the notebook preview walked every path to check whether reading the schema would poll a source; and each operator or function pod call formatted its input streams, whose description nests every upstream, into a debug log message even with debug logging off. A stream or node now keeps its predicted output schema per column selection (`output_schema()`, `keys()`), the poll check visits each upstream once, and the log message is formatted only when debug logging is on (see Changed for the description). Job compilation, `save`, `PipelineJob.load` and a run's schema queries are linear in the nodes as well: each node is asked a bounded number of times. A schema that includes the system tags is still as wide as the joins make it, because each join keeps both inputs' system-tag columns: stacked split-and-join stages over one source double their number per stage (ITL-1114). `head()` and the notebook preview of a stream that has not run build their table of no rows from a table that carries the system tags whatever columns are shown, so over such stacked stages they are as wide as the joins make them (ITL-1132). No hash, stored table or golden value changes. `PipelineJob.bind(sources=...)` makes every node of the job predict its schema again, so binding a source whose column has another type still changes the downstream schemas; assigning `bound_source` on a source node directly, or setting a node's `upstreams`, does not reach the nodes downstream of it, so use `bind`. + +### Changed + +- The `repr` of an operator stream, an operator node and a function node no longer nests the `repr` of its upstreams (ITL-1111). It shows its own class, label and operator or data function, and names each upstream by class and label only, for example `DynamicPodStream(label='Join', kernel=Join, upstreams=(, ))`. The nested form named every path through the graph, so its length doubled with each stage that splits a stream and joins it again. diff --git a/design_issues/ITL-1111-GW1.md b/design_issues/ITL-1111-GW1.md new file mode 100644 index 000000000..e96c27484 --- /dev/null +++ b/design_issues/ITL-1111-GW1.md @@ -0,0 +1,35 @@ +## `src/orcapod/core/streams/` + +### GW1 — Defining stacked split-and-join stages took time exponential in the stages +**Status:** resolved +**Severity:** high +**Issue:** ITL-1111 + +A pipeline whose every stage splits a stream into two branches and joins them again took +about 4x longer to define per stage (9 stages: 13 s, no row computed), as streams and inside a +`PipelineJob`. Three walks asked a shared upstream once per path instead of once: +`DynamicPodStream`, `FunctionPodStream`, `FunctionNode` and `OperatorNode` predicted their +output schema from their upstreams on every call, and `Join.output_schema` asks each input +several times (`keys`, `output_schema`, the system-tag schema); `StreamBase._schema_requires_io` +(`head()`, the notebook preview) recursed into every upstream path; and `process` / `__call__` +of operators and function pods formatted the input streams, whose `repr` nests every upstream, +into an f-string debug log on every call, even with debug logging off. + +**Fix:** `StreamBase._memoized_output_schema` keeps the predicted `(tag_schema, data_schema)` +per normalized `ColumnConfig` (an unhashable config is predicted every time) on the four +owners; each states what its prediction depends on (the pod and the upstreams' schemas and +pipeline hashes, never rows). The node `upstreams` setters and `PipelineJob.bind(sources=...)` +clear it. `_schema_requires_io` visits each upstream once per call and keeps nothing, because +a polling source's answer changes once it has polled. The debug logs take `%s` arguments, and the `repr` of `DynamicPodStream`, `OperatorNode` and +`FunctionNode` names each upstream by class and label only (`short_stream_repr`), so printing +a stream or logging at debug level is bounded too. +Call-count tests over 20 stages: `tests/test_pipeline/test_graph_walk_cost.py`. + +Left as they are: assigning `SourceJobNode.bound_source` directly, or setting a node's +`upstreams`, clears no memo of the nodes downstream of it, which keep their schema (their +pipeline hashes are equally stale today; strict xfail tests record both). The risk is bounded: +nothing in `src` calls the `upstreams` setter (the loaders build a node with `__new__` and set +`_input_streams` before any schema is asked; only tests call it), and after ITL-1112 a bind +cannot change a slot's schema. Cascading the clear (a registry of dependents, or a setter that +refuses an upstream of another schema) is for the developer to decide. And each join keeps both inputs' system-tag columns, so their number doubles per +stage (2 ** (stages + 1)), which is the width of the output, not a walk. diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 912a8456d..c5f112b78 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -724,6 +724,12 @@ topology suffix, encoding the full join lineage into each column name. | **Uniqueness guarantee** | Each post-join system tag column name uniquely identifies `(original schema, input topology, canonical join position)`; no collision even when joining streams with identical schemas | | **Known exclusions** | `SemiJoin` passes system tags through unchanged. Aggregating operators (`Batch`, `GroupBy`) use the fold described in site 6b instead. | +Every input keeps its own system tag columns, so the count reflects the join depth: a stream +split into two branches that are joined again carries each of its system tag columns twice, +and `k` such stages stacked carry each one `2 ** k` times (ITL-1114). The table of no rows that +`head()` and the notebook preview of a stream that has not run are built from carries the system +tag columns too, so those calls are as wide as the joins make them (ITL-1132). + --- ### Site 6b — Aggregating operator system tag fold diff --git a/src/orcapod/core/function_pod.py b/src/orcapod/core/function_pod.py index 874f80ab9..8f5e7bf28 100644 --- a/src/orcapod/core/function_pod.py +++ b/src/orcapod/core/function_pod.py @@ -733,7 +733,7 @@ def __call__( type is incompatible with its parameter, or (``strict=True``) the input carries a data column that is not a parameter. """ - logger.debug(f"Invoking pod {self} on streams through __call__: {streams}") + logger.debug("Invoking pod %s on streams through __call__: %s", self, streams) # A function that accepts arbitrary keywords takes every column, so # nothing is unused and no select is added (developer, November 12, 2026, # ITL-807). No data function kind accepts them today. @@ -866,7 +866,7 @@ def process( Returns: A ``FunctionPodStream`` wrapping the computation. """ - logger.debug(f"Invoking kernel {self} on streams: {streams}") + logger.debug("Invoking kernel %s on streams: %s", self, streams) input_stream = self.handle_input_streams(*streams) @@ -1098,8 +1098,15 @@ def output_schema( columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False, ) -> tuple[Schema, Schema]: - return self._function_pod.output_schema( - self._input_stream, columns=columns, all_info=all_info + # Memoized (ITL-1111). The prediction depends on the function pod (its declared + # output types) and on the input stream's schema, both fixed for the life of + # this stream; never on rows. + return self._memoized_output_schema( + columns, + all_info, + lambda: self._function_pod.output_schema( + self._input_stream, columns=columns, all_info=all_info + ), ) def _ensure_iterator(self) -> None: diff --git a/src/orcapod/core/nodes/function_node.py b/src/orcapod/core/nodes/function_node.py index 5357b1c2c..fda08a165 100644 --- a/src/orcapod/core/nodes/function_node.py +++ b/src/orcapod/core/nodes/function_node.py @@ -34,6 +34,7 @@ ZERO_ROW_INPUT_COLUMNS, StreamBase, check_head_size, + short_stream_repr, ) from orcapod.core.streams.view import ( BLUEPRINT_NOTE, @@ -426,6 +427,7 @@ def upstreams(self, value: tuple[StreamProtocol, ...]) -> None: if len(value) != 1: raise ValueError("FunctionPod can only have one upstream") self._input_stream = value[0] + self._forget_output_schema() # ------------------------------------------------------------------ # Read-only overrides (for deserialized nodes without live function_pod) @@ -481,10 +483,17 @@ def output_schema( f"Cannot restore the output schema of loaded node {self.label!r}: " f"{exc}" ) from exc - tag_schema = self._input_stream.output_schema( - columns=columns, all_info=all_info - )[0] - return tag_schema, self._data_function.output_data_schema + # Memoized (ITL-1111). The prediction depends on the data function's declared + # output types and on the input stream's schema; the ``upstreams`` setter and a + # job's ``bind`` clear it when the input changes. Never on rows. + return self._memoized_output_schema( + columns, + all_info, + lambda: ( + self._input_stream.output_schema(columns=columns, all_info=all_info)[0], + self._data_function.output_data_schema, + ), + ) def keys( self, @@ -752,9 +761,11 @@ def _head_of_pairs( ) def __repr__(self) -> str: + # The input by class and label only: its own repr would nest every path (ITL-1111). return ( - f"{type(self).__name__}(data_function={self._data_function!r}, " - f"input_stream={self._input_stream!r})" + f"{type(self).__name__}(label={self.label!r}, " + f"data_function={self._data_function!r}, " + f"input_stream={short_stream_repr(self._input_stream)})" ) @abstractmethod diff --git a/src/orcapod/core/nodes/operator_node.py b/src/orcapod/core/nodes/operator_node.py index 2b98053e5..e92574e01 100644 --- a/src/orcapod/core/nodes/operator_node.py +++ b/src/orcapod/core/nodes/operator_node.py @@ -31,6 +31,7 @@ ZERO_ROW_INPUT_COLUMNS, StreamBase, check_head_size, + short_stream_repr, ) from orcapod.core.streams.view import ( STALE_NOTE, @@ -319,6 +320,7 @@ def upstreams(self) -> tuple[StreamProtocol, ...]: @upstreams.setter def upstreams(self, value: tuple[StreamProtocol, ...]) -> None: self._input_streams = value + self._forget_output_schema() def keys( self, @@ -367,10 +369,16 @@ def output_schema( f"Cannot restore the output schema of loaded node {self.label!r}: " f"{exc}" ) from exc - return self._operator.output_schema( - *self._input_streams, - columns=columns, - all_info=all_info, + # Memoized (ITL-1111). The prediction depends on the operator's configuration + # and on the input streams' schemas and pipeline hashes; the ``upstreams`` + # setter and a job's ``bind`` clear it when an input changes. Never on rows. + operator = self._operator + return self._memoized_output_schema( + columns, + all_info, + lambda: operator.output_schema( + *self._input_streams, columns=columns, all_info=all_info + ), ) # ------------------------------------------------------------------ @@ -424,9 +432,11 @@ def clear_cache(self) -> None: self._update_modified_time() def __repr__(self) -> str: + # Upstreams by class and label only: their own repr would nest every path (ITL-1111). + upstreams = ", ".join(short_stream_repr(s) for s in self._input_streams) return ( - f"{type(self).__name__}(operator={self._operator!r}, " - f"upstreams={self._input_streams!r})" + f"{type(self).__name__}(label={self.label!r}, operator={self._operator!r}, " + f"upstreams=({upstreams}))" ) @abstractmethod diff --git a/src/orcapod/core/operators/static_output_pod.py b/src/orcapod/core/operators/static_output_pod.py index 467ad80c6..3c990610f 100644 --- a/src/orcapod/core/operators/static_output_pod.py +++ b/src/orcapod/core/operators/static_output_pod.py @@ -18,6 +18,7 @@ ZERO_ROW_INPUT_COLUMNS, StreamBase, check_head_size, + short_stream_repr, typed_zero_row_input, typed_zero_row_table, ) @@ -165,7 +166,7 @@ def process( Returns: A ``DynamicPodStream`` wrapping the computation. """ - logger.debug(f"Invoking kernel {self} on streams: {streams}") + logger.debug("Invoking kernel %s on streams: %s", self, streams) # perform input stream validation self.validate_inputs(*streams) @@ -208,7 +209,7 @@ def from_config(cls, config: dict[str, Any]) -> "StaticOutputOperatorPod": def __call__(self, *streams: StreamProtocol, **kwargs) -> DynamicPodStream: """Convenience alias for ``process``.""" - logger.debug(f"Invoking pod {self} on streams through __call__: {streams}") + logger.debug("Invoking pod %s on streams through __call__: %s", self, streams) # perform input stream validation return self.process(*streams, **kwargs) @@ -684,11 +685,7 @@ def keys( all_info: bool = False, ) -> tuple[tuple[str, ...], tuple[str, ...]]: """Return the (tag_keys, data_keys) column names for this stream.""" - tag_schema, data_schema = self._pod.output_schema( - *self.upstreams, - columns=columns, - all_info=all_info, - ) + tag_schema, data_schema = self.output_schema(columns=columns, all_info=all_info) return tuple(tag_schema.keys()), tuple(data_schema.keys()) def output_schema( @@ -698,10 +695,15 @@ def output_schema( all_info: bool = False, ) -> tuple[Schema, Schema]: """Return the (tag_schema, data_schema) for this stream.""" - return self._pod.output_schema( - *self.upstreams, - columns=columns, - all_info=all_info, + # Memoized (ITL-1111). The prediction depends on the operator's configuration + # and on the upstreams' schemas and pipeline hashes, all fixed for the life of + # this stream; never on rows, so recomputing the rows when stale leaves it valid. + return self._memoized_output_schema( + columns, + all_info, + lambda: self._pod.output_schema( + *self.upstreams, columns=columns, all_info=all_info + ), ) @property @@ -820,7 +822,14 @@ def head( ) def __repr__(self) -> str: - return f"{self.__class__.__name__}(kernel={self.producer}, upstreams={self.upstreams})" + # Upstreams by class and label only: their own repr would nest every path (ITL-1111). + upstreams = ", ".join( + short_stream_repr(upstream) for upstream in self.upstreams + ) + return ( + f"{self.__class__.__name__}(label={self.label!r}, kernel={self.producer}, " + f"upstreams=({upstreams}))" + ) def _upstream_state(stream: StreamProtocol) -> tuple[str, object]: diff --git a/src/orcapod/core/streams/base.py b/src/orcapod/core/streams/base.py index 9882ac6e2..2449511cf 100644 --- a/src/orcapod/core/streams/base.py +++ b/src/orcapod/core/streams/base.py @@ -2,7 +2,7 @@ import logging from abc import abstractmethod -from collections.abc import AsyncIterator, Collection, Iterator, Mapping +from collections.abc import AsyncIterator, Callable, Collection, Iterator, Mapping from contextvars import ContextVar from datetime import datetime from typing import TYPE_CHECKING, Any, cast @@ -55,6 +55,24 @@ def schema_requires_io(stream: object, config: ColumnConfig) -> bool: return bool(check(config)) if callable(check) else False +def short_stream_repr(stream: object) -> str: + """Name a stream by its class and label only, as a derived stream's ``repr`` lists it. + + A derived stream's ``repr`` names its upstreams this way, never by their own + ``repr``: that one names their upstreams in turn, so a graph whose stages split + and join again would print every path, 2 ** stages long (ITL-1111). + + Args: + stream: An upstream stream. + + Returns: + ````, or ```` when the stream has no label. + """ + label = getattr(stream, "label", None) + name = type(stream).__name__ + return f"<{name} {label!r}>" if label is not None else f"<{name}>" + + ZERO_ROW_INPUT_COLUMNS = ColumnConfig(system_tags=True, source=True) """The column groups of a zero-row table: what a node's rows carry downstream.""" @@ -587,6 +605,56 @@ def output_schema( all_info: bool = False, ) -> tuple[Schema, Schema]: ... + # Predicted output schemas by normalized column config; see + # ``_memoized_output_schema``. Set on first use, so a subclass that asks for its + # schema before ``StreamBase.__init__`` runs still works. + _output_schema_memo: dict[ColumnConfig, tuple[Schema, Schema]] | None = None + + def _memoized_output_schema( + self, + columns: ColumnConfig | dict[str, Any] | None, + all_info: bool, + predict: Callable[[], tuple[Schema, Schema]], + ) -> tuple[Schema, Schema]: + """Return ``predict()``, computed once per normalized column config (ITL-1111). + + A stream derived from upstreams predicts its schema by asking them for + theirs. Without a memo, a graph in which one stream feeds two branches that + join again asks the shared upstream once per path, so stacked + split-and-join stages cost 4x per stage. With it, every graph walk that + asks for schemas is linear in the nodes. + + Only the prediction is kept: a ``(tag_schema, data_schema)`` pair of + immutable ``Schema`` objects, never rows or a table. The caller states what + its prediction depends on, and clears the memo + (``_forget_output_schema``) when one of those things changes. A config + that cannot be a dict key (``meta`` given as a list of names) is predicted + every time. + + Args: + columns: The column config as the caller received it. + all_info: The ``all_info`` flag as the caller received it. + predict: Computes the schema from ``columns`` and ``all_info``. + + Returns: + The predicted ``(tag_schema, data_schema)``. + """ + config = ColumnConfig.handle_config(columns, all_info=all_info) + try: + hash(config) + except TypeError: + return predict() + if self._output_schema_memo is None: + self._output_schema_memo = {} + memo = self._output_schema_memo + if config not in memo: + memo[config] = predict() + return memo[config] + + def _forget_output_schema(self) -> None: + """Drop the memoized output schemas: an input of the prediction changed.""" + self._output_schema_memo = None + def __iter__( self, ) -> Iterator[tuple[TagProtocol, DataProtocol]]: @@ -831,8 +899,25 @@ def _schema_requires_io(self, config: ColumnConfig) -> bool: True when an upstream's would: a ``PollingSource`` that has not polled yet answers its schema only by polling when it declared no schema, or when ``config`` asks for a column group known only from rows. - """ - return any(schema_requires_io(upstream, config) for upstream in self.upstreams) + + Each upstream is visited once, however many paths lead to it (ITL-1111): a + stream that answers like this one is passed through to its own upstreams, + any other is asked. The answer is not kept, because a source's changes once + it has polled. + """ + seen: set[int] = {id(self)} + pending: list[StreamProtocol] = list(self.upstreams) + while pending: + stream = pending.pop() + if id(stream) in seen: + continue + seen.add(id(stream)) + hook = getattr(type(stream), "_schema_requires_io", None) + if hook is StreamBase._schema_requires_io: + pending.extend(stream.upstreams) + elif schema_requires_io(stream, config): + return True + return False def _zero_row_table(self, config: ColumnConfig) -> pa.Table | None: """This stream's table with no rows and the column groups of ``config``. diff --git a/src/orcapod/pipeline/job.py b/src/orcapod/pipeline/job.py index c38146edc..a077015fa 100644 --- a/src/orcapod/pipeline/job.py +++ b/src/orcapod/pipeline/job.py @@ -15,6 +15,7 @@ from orcapod.core.nodes.function_node import FunctionJobNode from orcapod.core.nodes.operator_node import OperatorJobNode from orcapod.core.nodes.source_node import SourceJobNode +from orcapod.core.streams.base import StreamBase from orcapod.side_effects import SideEffectJobNode from orcapod.core.tracker import AutoRegisteringContextBasedTracker from orcapod.contexts.stored import ( @@ -465,6 +466,11 @@ def bind( schema_hash = job_node.as_node().content_hash().to_string() self._sources_by_hash[schema_hash] = sources[job_node.name] self._sources.update(sources) + # ``validate`` checks column names only, so a rebound source may give a + # column another type: every node predicts its schema again (ITL-1111). + for job_node in (self._persistent_node_map or {}).values(): + if isinstance(job_node, StreamBase): + job_node._forget_output_schema() # pyright: ignore[reportPrivateUsage] # the job owns its nodes' bindings if execution_context is not None: self._execution_context = execution_context diff --git a/superpowers/plans/2026-11-14-itl-1111-definition-time.md b/superpowers/plans/2026-11-14-itl-1111-definition-time.md new file mode 100644 index 000000000..c6978c158 --- /dev/null +++ b/superpowers/plans/2026-11-14-itl-1111-definition-time.md @@ -0,0 +1,66 @@ +# ITL-1111: defining stacked split-and-join stages is exponential + +Branch `eywalker/itl-1111-defining-a-pipeline-with-stacked-split-and`, from `origin/main` `a9b50350`. +Cost fix only: nothing hashed, stored or golden changes. + +## Cause (measured, cProfile + per-node call counts) + +Scratch tools (not committed): `walks.py` in the session scratchpad wraps `output_schema`, `keys`, +`content_hash`, `pipeline_hash`, identity structures, `argument_symmetry`, `validate_inputs`, +`is_stale`, `last_modified`, `_update_cache_status`, `_predict_system_tag_schema` on every orcapod +class and counts calls per object. + +- No derived stream keeps its output schema. `DynamicPodStream.output_schema()` / `keys()` + (`core/operators/static_output_pod.py`), `FunctionPodStream.output_schema()` + (`core/function_pod.py`), `FunctionNode.output_schema()` (`core/nodes/function_node.py`) and + `OperatorNode.output_schema()` (`core/nodes/operator_node.py`) recompute from their pod over + their upstreams on every call. +- `Join.output_schema(a, b)` asks each input several times: `keys()` in + `_require_tag_data_roles_agree`, `output_schema()` in `_joined_user_schemas`, again + `output_schema(system_tags)` in `_predict_system_tag_schema`. A split-and-join stage therefore + calls its upstream about 4 times per call of its own: 4x per stage. +- 8-stage operator build (41 nodes): 101,930 `Join.output_schema`, 509,572 + `DynamicPodStream.output_schema`, 203,890 `keys`, 305,830 source `output_schema` calls. + Max calls on one node: 4x per stage (294, 1190, 4762 for 3, 4, 5 stages). +- `content_hash` and `pipeline_hash` are already memoized (`core/base.py`): constant per node. +- Same 4x growth for job compile, `job.save()`, `PipelineJob.load()` (operators), async + `job.run()` (function pods), `head()`, preview, stream `run()`: all through `output_schema`. +- Staleness (`last_modified` / `is_stale`) in stream `run()` of a function-pod diamond is also + exponential on main: PR #380's territory; measure on its head, do not fix here. + +## Proposal (sent to coordinator; wait for reply) + +Memoize the output schema on each derived stream that owns a fan-in walk, keyed by the +normalized `ColumnConfig` (skip the memo when the config is unhashable), `keys()` derived from +it. A shared private helper on `StreamBase`. + +Depends on: the pod (immutable configuration, incl. `system_tag_n_char`), the upstream tuple, +and the upstreams' schemas and pipeline hashes. Never on rows: a schema is predicted, never read +from computed data, so a `DynamicPodStream` recomputing its rows when stale gives the same +schema. Invalidation: node `upstreams` setter; `PipelineJob.bind(sources=...)` (and +`SourceJobNode.bound_source` setter) because bind validates column names only, so a rebound +source of other types changes downstream schemas (downstream pipeline-hash memos already have +this same latent gap: report it). + +## Steps after approval + +1. Failing tests (call counts, linear in nodes): 20 stacked stages as streams and in a + `PipelineJob`; per-node call count of `output_schema` bounded by a constant. +2. Helper + memo in the four owners; invalidation hooks. +3. Re-measure every walk at 20 stages; table for the PR and worker_done. +4. Golden + hash label rule tests unchanged; full suite; changelog fragment; design_issues entry. + +## Progress (2026-11-14) + +- Coordinator approved option A with invalidation (upstreams setter, `job.bind`); direct + `bound_source` assignment left stale (strict xfail test), reported. +- Two more exponential walks found and fixed: eager f-string debug logs formatting nested + stream reprs in `process` / `__call__` (now `%s`), and `_schema_requires_io` (now visits + each upstream once, no memo). `repr` of derived streams / nodes bounded to one level + (`short_stream_repr`), approved as option A. +- System-tag columns double per join stage: expected (developer); docs sentence in + `docs/reference/hashing.md` (ITL-1114). Tests: 20 stages for schema walks, 8 stages for + all_info and job.run. +- Stream `run()` staleness walk still exponential on #380 head + this branch: reported, not + fixed (operators 2x per stage, function pods 4x per stage). +- Done: tests, fix, changelog fragment, design_issues/ITL-1111-GW1.md. Next: PR, CI, CodeRabbit. diff --git a/tests/test_pipeline/test_graph_walk_cost.py b/tests/test_pipeline/test_graph_walk_cost.py new file mode 100644 index 000000000..7e820f802 --- /dev/null +++ b/tests/test_pipeline/test_graph_walk_cost.py @@ -0,0 +1,595 @@ +"""Graph walks over stacked split-and-join stages are linear in the nodes (ITL-1111). + +Each stage splits a stream into two branches and joins them again, so every +stage is a diamond over the previous one. A walk that asks a shared upstream +once per path visits the source 2**stages times or more; a walk that keeps +what it has computed asks each node a bounded number of times. + +The tests count calls of the schema queries (``output_schema`` and ``keys``) +and of the other graph walks (``__repr__``, ``_schema_requires_io``) on every +stream, source and node class, per object, never wall-clock time. A +counter stops the walk with ``WalkBudgetExceeded`` as soon as the total passes +a budget linear in the node count, so an exponential walk fails fast. + +Walks over the system tags (``all_info``, a run) are tested at fewer stages: +every join keeps both inputs' system-tag columns, so a stage doubles their +number, and at ``STAGES`` a row would be millions of columns wide. That width is +the output, not a walk; only the number of schema queries per node is asserted. +""" + +from __future__ import annotations + +import collections +import functools +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +import pytest + +from orcapod import PipelineJob, function_pod +from orcapod.core.sources import DictSource +from orcapod.core.streams.base import StreamBase +from orcapod.databases import InMemoryArrowDatabase +from orcapod.errors import InputValidationError +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator + +STAGES = 20 + +# Stages for the walks whose output carries the system tags (2 ** (stages + 1) +# system-tag columns). +SYSTEM_TAG_STAGES = 8 + +# Upper bound on the schema queries one node may receive in one walk. +MAX_CALLS_PER_NODE = 30 + +# The schema queries, plus the two other walks found at construction (ITL-1111): +# ``__repr__`` nests every upstream (it was formatted eagerly into a debug log on +# every invocation), and the private ``_schema_requires_io`` hook, counted because +# ``head()`` and the preview walk the graph through it. +_COUNTED_METHODS = ("output_schema", "keys", "__repr__", "_schema_requires_io") + + +class WalkBudgetExceeded(AssertionError): + """More schema queries than the linear budget allows: the walk revisits nodes.""" + + +def _all_stream_classes() -> set[type]: + """Return ``StreamBase`` and every subclass of it defined so far. + + Returns: + The stream, source and node classes whose methods are counted. + """ + seen: set[type] = set() + pending: list[type] = [StreamBase] + while pending: + cls = pending.pop() + if cls in seen: + continue + seen.add(cls) + pending.extend(cls.__subclasses__()) + return seen + + +class _CallCounter: + """Count calls per object and stop a walk that passes a budget. + + Records one call per wrapped method call, keyed by the object called. Every + counted object is kept until the counter is released, so the id of a + short-lived stream (a run makes many) is never reused by another object and + two objects' counts never merge. + + Attributes: + budget: The most calls allowed in total before ``WalkBudgetExceeded``. + per_object: Calls per object, keyed by ``id``. + total: Calls in total. + """ + + def __init__(self, budget: int) -> None: + """Start with no calls counted. + + Args: + budget: The most calls allowed in total. + """ + self.budget = budget + self.per_object: collections.Counter[int] = collections.Counter() + self.total = 0 + # Every counted object is kept alive while counting, so that the id of a + # short-lived stream (a run makes many) is never reused by another one. + self._counted: dict[int, object] = {} + + def record(self, obj: object) -> None: + """Count one call on ``obj``. + + Args: + obj: The object whose method was called. + + Raises: + WalkBudgetExceeded: If the total passes the budget. + """ + self._counted[id(obj)] = obj + self.per_object[id(obj)] += 1 + self.total += 1 + if self.total > self.budget: + raise WalkBudgetExceeded( + f"more than {self.budget} schema queries: the walk revisits nodes" + ) + + @property + def max_per_object(self) -> int: + """The most calls any one object received, 0 when none was counted.""" + return max(self.per_object.values(), default=0) + + +@contextmanager +def _count_schema_queries( + monkeypatch: pytest.MonkeyPatch, + budget: int, + methods: tuple[str, ...] = _COUNTED_METHODS, +) -> Iterator[_CallCounter]: + """Count calls of ``methods`` on every stream class, per object.""" + counter = _CallCounter(budget) + + def wrap(method: Callable[..., Any]) -> Callable[..., Any]: + """Return ``method`` wrapped to count each call on its object.""" + + @functools.wraps(method) + def counted(self: object, *args: Any, **kwargs: Any) -> Any: + """Count the call, then call the original method.""" + counter.record(self) + return method(self, *args, **kwargs) + + return counted + + with monkeypatch.context() as patch: + for cls in _all_stream_classes(): + for name in methods: + if name in cls.__dict__: + patch.setattr(cls, name, wrap(cls.__dict__[name])) + yield counter + + +@function_pod(output_keys="a") +def _branch_a(x: int) -> int: + """One branch of a function-pod stage.""" + return x + 1 + + +@function_pod(output_keys="b") +def _branch_b(x: int) -> int: + """The other branch of a function-pod stage.""" + return x + 2 + + +@function_pod(output_keys="x") +def _merge(a: int, b: int) -> int: + """Join the two branches of a function-pod stage.""" + return a + b + + +def _operator_stage(stream: Any) -> Any: + """Split ``stream`` into two renamed branches and join them again with operators. + + Args: + stream: The previous stage's output, with tag ``id`` and data ``x``. + + Returns: + A stream with the same columns: five operator streams per stage. + """ + a = stream.map_data({"x": "a"}) + b = stream.map_data({"x": "b"}) + joined = a.join(b) + return joined.map_data({"a": "x"}).drop_data_columns("b") + + +def _function_pod_stage(stream: Any) -> Any: + """Split ``stream`` into two function-pod branches and merge them. + + Args: + stream: The previous stage's output, with tag ``id`` and data ``x``. + + Returns: + A stream with the same columns: three stream nodes per stage. + """ + return _merge.pod(_branch_a.pod(stream), _branch_b.pod(stream)) + + +# shape -> (stage, nodes per stage) +SHAPES: dict[str, tuple[Callable[[Any], Any], int]] = { + "operators": (_operator_stage, 5), + "function_pods": (_function_pod_stage, 3), +} + + +def _node_count(shape: str, stages: int) -> int: + """Return the number of nodes ``stages`` stages of ``shape`` make, with the source.""" + return SHAPES[shape][1] * stages + 1 + + +def _budget(shape: str, stages: int = STAGES) -> int: + """Return the most calls a walk over ``stages`` stages of ``shape`` may make.""" + return MAX_CALLS_PER_NODE * _node_count(shape, stages) + + +def _source() -> DictSource: + """Return the three-row source every graph starts from.""" + return DictSource( + [{"id": i, "x": i} for i in range(3)], tag_columns=["id"], source_id="src" + ) + + +def _build(shape: str, stages: int = STAGES) -> Any: + """Build ``stages`` stacked stages of ``shape`` as plain streams. + + Args: + shape: A key of ``SHAPES``. + stages: The number of stages. + + Returns: + The last stage's output stream. + """ + stage, _ = SHAPES[shape] + stream = _source() + for _ in range(stages): + stream = stage(stream) + return stream + + +def _build_job(shape: str, stages: int = STAGES) -> PipelineJob: + """Record ``stages`` stacked stages of ``shape`` in a job and compile it. + + Args: + shape: A key of ``SHAPES``. + stages: The number of stages. + + Returns: + The compiled job, over an in-memory database. + """ + job = PipelineJob(name="stacked", database=InMemoryArrowDatabase()) + with job: + _build(shape, stages) + return job + + +def _guarded_build( + shape: str, monkeypatch: pytest.MonkeyPatch, stages: int = STAGES +) -> Any: + """Build the stream graph, failing fast if building is not linear.""" + with _count_schema_queries(monkeypatch, _budget(shape, stages)): + return _build(shape, stages) + + +def _guarded_build_job( + shape: str, monkeypatch: pytest.MonkeyPatch, stages: int = STAGES +) -> PipelineJob: + """Record and compile the job, failing fast if that is not linear.""" + with _count_schema_queries(monkeypatch, _budget(shape, stages)): + return _build_job(shape, stages) + + +def _assert_linear(counter: _CallCounter, shape: str, stages: int = STAGES) -> None: + """Assert that no object received more than its share and the total is in budget.""" + assert counter.max_per_object <= MAX_CALLS_PER_NODE + assert counter.total <= _budget(shape, stages) + + +@pytest.fixture(params=sorted(SHAPES)) +def shape(request: pytest.FixtureRequest) -> str: + """Run each test once per stage shape.""" + return request.param + + +class TestStreams: + """Walks over plain streams, outside a job.""" + + def test_building_the_graph_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Building the stages asks each stream a bounded number of times.""" + with _count_schema_queries(monkeypatch, _budget(shape)) as counter: + _build(shape) + _assert_linear(counter, shape) + + @pytest.mark.parametrize( + ("walk", "stages"), + [ + pytest.param(lambda s: s.output_schema(), STAGES, id="output_schema"), + pytest.param(lambda s: s.keys(), STAGES, id="keys"), + # head() and the preview of a stream that has not run build their table + # from its zero-row table, which carries the system tags (2 ** (stages + 1) + # columns): the system-tag stage count (ITL-1114, ITL-1132). + pytest.param(lambda s: s.head(), SYSTEM_TAG_STAGES, id="head"), + # The notebook preview is the Jupyter display hook. + pytest.param(lambda s: s._repr_html_(), SYSTEM_TAG_STAGES, id="preview"), + ], + ) + def test_a_schema_walk_is_linear( + self, + shape: str, + walk: Callable[[Any], Any], + stages: int, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """A schema query, ``head()`` or the preview asks each stream a bounded number of times.""" + stream = _guarded_build(shape, monkeypatch, stages) + with _count_schema_queries(monkeypatch, _budget(shape, stages)) as counter: + walk(stream) + _assert_linear(counter, shape, stages) + + def test_the_system_tag_schema_walk_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Predicting the schema with system tags asks each stream a bounded number of times.""" + stream = _guarded_build(shape, monkeypatch, SYSTEM_TAG_STAGES) + budget = _budget(shape, SYSTEM_TAG_STAGES) + with _count_schema_queries(monkeypatch, budget) as counter: + stream.output_schema(all_info=True) + _assert_linear(counter, shape, SYSTEM_TAG_STAGES) + + def test_the_zero_row_derivation_asks_each_stream_once( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A stream's zero-row table (ITL-912) is derived once per node, with its schema walks bounded.""" + # typed_zero_row_table is the zero-row derivation head() makes on a stream that + # has not run; _typed_zero_row_table is its per-stream hook. + from orcapod.core.streams.base import typed_zero_row_table + + # The zero-row table carries the system tags (2 ** (stages + 1) columns), so + # the stage count is the system-tag one (ITL-1114, ITL-1132). + stages = SYSTEM_TAG_STAGES + stream = _guarded_build(shape, monkeypatch, stages) + methods = (*_COUNTED_METHODS, "_typed_zero_row_table", "_zero_row_table") + with _count_schema_queries(monkeypatch, _budget(shape, stages), methods) as c: + table = typed_zero_row_table(stream) + assert table is not None and table.num_rows == 0 + _assert_linear(c, shape, stages) + with _count_schema_queries( + monkeypatch, _budget(shape, stages), ("_typed_zero_row_table",) + ) as once: + typed_zero_row_table(stream) + assert once.max_per_object == 1 + + @pytest.mark.parametrize("all_info", [False, True]) + def test_head_of_a_stream_that_has_not_run_is_linear_with_the_zero_row_table( + self, shape: str, all_info: bool, monkeypatch: pytest.MonkeyPatch + ) -> None: + """``head()`` builds its typed zero-row table by walking each stream a bounded number of times.""" + # The head's table is built from the zero-row table, which carries the system + # tags whatever the config: the system-tag stage count (ITL-1114, ITL-1132). + stages = SYSTEM_TAG_STAGES + stream = _guarded_build(shape, monkeypatch, stages) + methods = (*_COUNTED_METHODS, "_typed_zero_row_table", "_zero_row_table") + with _count_schema_queries(monkeypatch, _budget(shape, stages), methods) as c: + view = stream.head(all_info=all_info) + assert view.as_table().num_rows == 0 + _assert_linear(c, shape, stages) + + def test_the_poll_check_asks_each_stream_once( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """``head()`` asks each stream once whether reading its schema would poll a source.""" + # The private hook is the walk head() makes to learn whether reading the + # schema would poll a source. + # head() builds a zero-row table with the system tags: 8 stages (ITL-1132). + stream = _guarded_build(shape, monkeypatch, SYSTEM_TAG_STAGES) + with _count_schema_queries( + monkeypatch, + _budget(shape, SYSTEM_TAG_STAGES), + methods=("_schema_requires_io",), + ) as counter: + stream.head() + assert counter.max_per_object == 1 + + def test_repr_is_bounded(self, shape: str, monkeypatch: pytest.MonkeyPatch) -> None: + """A stream's ``repr`` names its upstreams by class and label, not by their ``repr``.""" + stream = _guarded_build(shape, monkeypatch) + with _count_schema_queries(monkeypatch, _budget(shape)): + text = repr(stream) + assert len(text) < 500 + assert type(stream).__name__ in text + + def test_a_join_error_is_bounded_and_names_the_inputs_and_the_column( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A join of two inputs sharing a data column fails with a short message naming both.""" + stream = _guarded_build(shape, monkeypatch) + left = stream.map_data({"x": "y"}, label="left_branch") + right = stream.map_data({"x": "y"}, label="right_branch") + with _count_schema_queries(monkeypatch, _budget(shape)): + with pytest.raises(InputValidationError) as excinfo: + left.join(right) + message = str(excinfo.value) + assert len(message) < 1000 + assert "left_branch" in message + assert "right_branch" in message + assert "'y'" in message + + def test_hashes_are_unchanged_by_asking_twice( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Two builds of the same graph have equal hashes and schemas, memo or not.""" + first = _guarded_build(shape, monkeypatch) + with _count_schema_queries(monkeypatch, _budget(shape)): + first.output_schema() + second = _guarded_build(shape, monkeypatch) + assert first.content_hash() == second.content_hash() + assert first.pipeline_hash() == second.pipeline_hash() + assert first.output_schema() == second.output_schema() + + +class TestPipelineJob: + """Walks over the nodes of a ``PipelineJob``.""" + + def test_recording_and_compiling_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Recording the stages in a job and compiling it asks each node a bounded number of times.""" + with _count_schema_queries(monkeypatch, _budget(shape)) as counter: + _build_job(shape) + _assert_linear(counter, shape) + + def test_save_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """Saving the job asks each node a bounded number of times.""" + job = _guarded_build_job(shape, monkeypatch) + with _count_schema_queries(monkeypatch, _budget(shape)) as counter: + job.save(tmp_path / "job.json") + _assert_linear(counter, shape) + + def test_load_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + """Loading the saved job asks each node a bounded number of times.""" + job = _guarded_build_job(shape, monkeypatch) + with _count_schema_queries(monkeypatch, _budget(shape)): + job.save(tmp_path / "job.json") + with _count_schema_queries(monkeypatch, _budget(shape)) as counter: + PipelineJob.load(tmp_path / "job.json") + _assert_linear(counter, shape) + + @pytest.mark.parametrize( + "orchestrator", + [ + pytest.param(SyncPipelineOrchestrator, id="sync"), + pytest.param(AsyncPipelineOrchestrator, id="async"), + ], + ) + def test_run_is_linear( + self, + shape: str, + orchestrator: type, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + """A run on either orchestrator asks each node a bounded number of times.""" + job = _guarded_build_job(shape, monkeypatch, SYSTEM_TAG_STAGES) + budget = _budget(shape, SYSTEM_TAG_STAGES) + with _count_schema_queries(monkeypatch, budget) as counter: + job.run(orchestrator=orchestrator()) + _assert_linear(counter, shape, SYSTEM_TAG_STAGES) + + def test_node_reprs_are_bounded( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Every job node's ``repr`` names its upstreams by class and label only.""" + job = _guarded_build_job(shape, monkeypatch) + for node in job.nodes.values(): + with _count_schema_queries(monkeypatch, _budget(shape)): + text = repr(node) + assert len(text) < 500 + + def test_a_node_schema_walk_is_linear( + self, shape: str, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Asking the last node for its schema asks each node a bounded number of times.""" + job = _guarded_build_job(shape, monkeypatch) + last = list(job.nodes.values())[-1] + with _count_schema_queries(monkeypatch, _budget(shape)) as counter: + last.output_schema() + last.keys() + _assert_linear(counter, shape) + + +# --------------------------------------------------------------------------- +# A memoized schema is predicted again when an input of the prediction changes +# --------------------------------------------------------------------------- + + +def _one_row_source(value: int | float | str) -> DictSource: + """Return a one-row source named ``src`` whose data column ``x`` holds ``value``.""" + return DictSource([{"id": 1, "x": value}], tag_columns=["id"], source_id="src") + + +def _rename_job() -> PipelineJob: + """Return a job ``src`` -> ``renamed`` (``x`` to ``y``) -> ``renamed_again`` (``y`` to ``z``).""" + job = PipelineJob(name="rebind", database=InMemoryArrowDatabase()) + with job: + renamed = _one_row_source(1).map_data({"x": "y"}, label="renamed") + renamed.map_data({"y": "z"}, label="renamed_again") + return job + + +def _function_job() -> PipelineJob: + """Return a job with one function node, ``plus_one``, over ``src``.""" + job = PipelineJob(name="rebind_function", database=InMemoryArrowDatabase()) + with job: + _branch_a.pod(_one_row_source(1), label="plus_one") + return job + + +class TestMemoInvalidation: + """A memoized schema is predicted again when an input of the prediction changes.""" + + def test_bind_to_a_source_of_another_column_type_changes_downstream_schemas( + self, + ) -> None: + """``job.bind`` to a source with another column type changes every downstream schema.""" + job = _rename_job() + assert job.nodes["renamed"].output_schema()[1]["y"] is int + assert job.nodes["renamed_again"].output_schema()[1]["z"] is int + assert job.nodes["renamed_again"].keys() == (("id",), ("z",)) + + job.bind(sources={"src": _one_row_source("a")}) + + assert job.nodes["renamed"].output_schema()[1]["y"] is str + assert job.nodes["renamed_again"].output_schema()[1]["z"] is str + + def test_setting_an_operator_nodes_upstreams_changes_its_schema(self) -> None: + """Setting an operator node's ``upstreams`` changes its own schema.""" + job = _rename_job() + node = job.nodes["renamed"] + assert node.output_schema()[1]["y"] is int + + node.upstreams = (_one_row_source(1.5),) + + assert node.output_schema()[1]["y"] is float + + def test_setting_a_function_nodes_upstreams_changes_its_schema(self) -> None: + """Setting a function node's ``upstreams`` changes its own schema.""" + job = _function_job() + node = job.nodes["plus_one"] + assert node.output_schema()[0]["id"] is int + other = DictSource( + [{"id": "one", "x": 1}], tag_columns=["id"], source_id="other" + ) + + node.upstreams = (other,) + + assert node.output_schema()[0]["id"] is str + + @pytest.mark.xfail( + strict=True, + reason=( + "A direct bound_source assignment reaches no downstream node, so their " + "memoized schemas are not predicted again; use job.bind (ITL-1111). " + "Their pipeline hashes are equally stale today." + ), + ) + def test_assigning_bound_source_directly_changes_downstream_schemas( + self, + ) -> None: + """A direct ``bound_source`` assignment should change the downstream schemas.""" + job = _rename_job() + assert job.nodes["renamed_again"].output_schema()[1]["z"] is int + + job.nodes["src"].bound_source = _one_row_source("a") + + assert job.nodes["renamed_again"].output_schema()[1]["z"] is str + + @pytest.mark.xfail( + strict=True, + reason=( + "Setting a wired node's upstreams clears its own memo only; the nodes " + "downstream of it keep theirs (ITL-1111). Nothing in orcapod sets " + "upstreams after construction." + ), + ) + def test_setting_upstreams_changes_downstream_schemas(self) -> None: + """Setting a node's ``upstreams`` should change the schemas downstream of it.""" + job = _rename_job() + assert job.nodes["renamed_again"].output_schema()[1]["z"] is int + + job.nodes["renamed"].upstreams = (_one_row_source(1.5),) + + assert job.nodes["renamed_again"].output_schema()[1]["z"] is float