Repository navigation
fix(streams): linear graph walks when defining stacked split-and-join stages (ITL-1111) #387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 11 commits into
main
from
eywalker/itl-1111-defining-a-pipeline-with-stacked-split-and
Nov 14, 2026
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
cfd6468
docs(plan): ITL-1111 cause and proposal
f8e79e4
wip(streams): memoize predicted output schemas; call-count tests (ITL…
648dfa7
fix(streams): linear graph walks at definition time (ITL-1111)
82d0197
docs(hashing): system-tag columns per join depth; changelog and desig…
d48d7c1
Merge remote-tracking branch 'origin/main' into eywalker/itl-1111-def…
e02b517
test(pipeline): type the one-row source value (ITL-1111)
dcfe44d
fix(streams): bound the repr of derived streams and nodes to one leve…
d20c22d
test(pipeline): keep counted streams alive so an id is never reused (…
079af91
docs(changelog): qualify linear schema walks; document test helpers; …
a369fd4
merge: origin/main into ITL-1111 branch (keeps the schema memo and th…
959dce6
test(pipeline): count the zero-row derivation over stacked stages; he…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| ### Fixed | ||
|
|
||
| - Defining a pipeline whose stages split a stream into two branches and join them again no longer takes time exponential in the number of stages (about 4x per stage before: 9 stages took 13 seconds to build, with no row computed; 20 stages now build in well under a second, as streams and inside a `PipelineJob`) (ITL-1111). Three graph walks asked a shared upstream once per path: every operator stream, function pod stream and node predicted its output schema again from its upstreams on each call, and a join asks each input several times; `head()` and the notebook preview walked every path to check whether reading the schema would poll a source; and each operator or function pod call formatted its input streams, whose description nests every upstream, into a debug log message even with debug logging off. A stream or node now keeps its predicted output schema per column selection (`output_schema()`, `keys()`), the poll check visits each upstream once, and the log message is formatted only when debug logging is on (see Changed for the description). Job compilation, `save`, `PipelineJob.load` and a run's schema queries are linear in the nodes as well: each node is asked a bounded number of times. A schema that includes the system tags is still as wide as the joins make it, because each join keeps both inputs' system-tag columns: stacked split-and-join stages over one source double their number per stage (ITL-1114). `head()` and the notebook preview of a stream that has not run build their table of no rows from a table that carries the system tags whatever columns are shown, so over such stacked stages they are as wide as the joins make them (ITL-1132). No hash, stored table or golden value changes. `PipelineJob.bind(sources=...)` makes every node of the job predict its schema again, so binding a source whose column has another type still changes the downstream schemas; assigning `bound_source` on a source node directly, or setting a node's `upstreams`, does not reach the nodes downstream of it, so use `bind`. | ||
|
|
||
| ### Changed | ||
|
|
||
| - The `repr` of an operator stream, an operator node and a function node no longer nests the `repr` of its upstreams (ITL-1111). It shows its own class, label and operator or data function, and names each upstream by class and label only, for example `DynamicPodStream(label='Join', kernel=Join, upstreams=(<DynamicPodStream 'MapData'>, <DynamicPodStream 'MapData'>))`. The nested form named every path through the graph, so its length doubled with each stage that splits a stream and joins it again. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| ## `src/orcapod/core/streams/` | ||
|
|
||
| ### GW1 — Defining stacked split-and-join stages took time exponential in the stages | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1111 | ||
|
|
||
| A pipeline whose every stage splits a stream into two branches and joins them again took | ||
| about 4x longer to define per stage (9 stages: 13 s, no row computed), as streams and inside a | ||
| `PipelineJob`. Three walks asked a shared upstream once per path instead of once: | ||
| `DynamicPodStream`, `FunctionPodStream`, `FunctionNode` and `OperatorNode` predicted their | ||
| output schema from their upstreams on every call, and `Join.output_schema` asks each input | ||
| several times (`keys`, `output_schema`, the system-tag schema); `StreamBase._schema_requires_io` | ||
| (`head()`, the notebook preview) recursed into every upstream path; and `process` / `__call__` | ||
| of operators and function pods formatted the input streams, whose `repr` nests every upstream, | ||
| into an f-string debug log on every call, even with debug logging off. | ||
|
|
||
| **Fix:** `StreamBase._memoized_output_schema` keeps the predicted `(tag_schema, data_schema)` | ||
| per normalized `ColumnConfig` (an unhashable config is predicted every time) on the four | ||
| owners; each states what its prediction depends on (the pod and the upstreams' schemas and | ||
| pipeline hashes, never rows). The node `upstreams` setters and `PipelineJob.bind(sources=...)` | ||
| clear it. `_schema_requires_io` visits each upstream once per call and keeps nothing, because | ||
| a polling source's answer changes once it has polled. The debug logs take `%s` arguments, and the `repr` of `DynamicPodStream`, `OperatorNode` and | ||
| `FunctionNode` names each upstream by class and label only (`short_stream_repr`), so printing | ||
| a stream or logging at debug level is bounded too. | ||
| Call-count tests over 20 stages: `tests/test_pipeline/test_graph_walk_cost.py`. | ||
|
|
||
| Left as they are: assigning `SourceJobNode.bound_source` directly, or setting a node's | ||
| `upstreams`, clears no memo of the nodes downstream of it, which keep their schema (their | ||
| pipeline hashes are equally stale today; strict xfail tests record both). The risk is bounded: | ||
| nothing in `src` calls the `upstreams` setter (the loaders build a node with `__new__` and set | ||
| `_input_streams` before any schema is asked; only tests call it), and after ITL-1112 a bind | ||
| cannot change a slot's schema. Cascading the clear (a registry of dependents, or a setter that | ||
| refuses an upstream of another schema) is for the developer to decide. And each join keeps both inputs' system-tag columns, so their number doubles per | ||
| stage (2 ** (stages + 1)), which is the width of the output, not a walk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: metamorphic-labs/orcapod-python
Length of output: 9816
Invalidate downstream schema memos when an upstream changes.
Setting
upstreamsclears only the changed node's memo. The same local-only reset appears infunction_node.py:429. A downstream node keeps its memo entry. Its memo hit atbase.py:547-549returns the cached prediction without callingpredict(), so it never reads the new upstream schema.For example, after
renamed_again.output_schema()runs, setrenamed.upstreams = (_one_row_source(1.5),).renamedthen reportsy: float, butrenamed_againstill reports the cachedz: int. Before this memo, the downstream schema was recomputed on each call.The
job.bind()path clears every job node (job.py:473), so it is not affected. Only direct reassignment ofupstreamson a wired node leaves dependents stale.The setter must also invalidate dependent memos. Nodes do not currently track their downstream nodes, so a direct local-only reset cannot reach them. The fix needs a mechanism for downstream invalidation, or a memo key that reflects the upstream identity or version. The change is not a localized one-line edit, so no diff is proposed here.
Add a downstream assertion to
test_setting_an_operator_nodes_upstreams_changes_its_schemaand to the function-node setter test. Assert thatrenamed_again(or the downstream node in the function-node case) reports the new schema.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed, and declined as a code change in this PR, by decision of the coordinator. The gap is recorded instead (079af91):
test_setting_upstreams_changes_downstream_schemas, asserts what you describe:renamed_againshould reportz: floatafterrenamed.upstreamsis reassigned. A second one covers the same gap for a directSourceJobNode.bound_sourceassignment.design_issues/ITL-1111-GW1.mdand the changelog fragment name both. The fragment says to usejob.bind, which clears every node of the job.Why no cascade here: a version stamp on the direct upstreams reaches only one level, so a correct fix needs a registry of dependents, or a setter that refuses an upstream of another schema. Both are new design, and the developer will decide between them on a separate issue. The risk until then is bounded:
srccalls theupstreamssetter. The loaders (from_descriptor) build a node with__new__and set_input_streamsbefore any schema is asked, and only tests call the setter.maintoday, without this PR.Posted by a Claude Code agent on behalf of @eywalker.