Skip to content

fix(pipeline): loaded side-effect node passes run_alone the unchecked source ids (#381 x #384) - #396

Merged
eywalker merged 1 commit into
mainfrom
eywalker/hotfix-loaded-side-effect-run-alone
Nov 15, 2026
Merged

eywalker merged 1 commit into
mainfrom
eywalker/hotfix-loaded-side-effect-run-alone

Conversation

@eywalker

@eywalker eywalker commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

PR #384 (ITL-1102) changed run_job_node's run_alone to take the run's unchecked source ids (a tuple). PR #381 (ITL-1094), merged earlier, still passed lambda: self._run_alone(load_only=True) from LoadedSideEffectJobNode.run: a basedpyright error on main (reportArgumentType), and a TypeError if reached (a loaded node whose upstream is not a job node). It now takes the ids and ignores them, as the live SideEffectJobNode.run does. A regression test builds a loaded node over a source's stream and runs it load-only, which reaches run_alone.

Refs ITL-1094, ITL-1102

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Fixed load-only runs for pipelines that read directly from a data source. The resulting view now includes the source rows, and the run does not deliver them. This makes the behavior consistent when loading data without an upstream job, while leaving normal delivery behavior unchanged. The fix is covered by a regression test.

… source ids (#381 x #384)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
ITL-1094 A loaded job has no side-effect nodes: PipelineJob.load drops them

Overview

Found on November 13, 2026 by the ITL-1091 worker on main b4fde330; not yet reproduced by the coordinator.

A job with a side-effect pod node saves that node (the saved file has an entry with node_type side_effect), but PipelineJob.load rebuilds the job without it: Pipeline.load has no branch for side_effect. In the worker's case the saved job had nodes s, delivered (the side-effect node) and down; the loaded job's nodes are ['s', 'down'], and loaded.nodes['delivered'] is a KeyError.

So a loaded job silently lacks its side-effect nodes: they cannot be inspected, and whatever is downstream of one is connected to something else or to nothing (to be established).

Goals & Success Criteria

  • A reproduction through the public interface (save, load, nodes), and a statement of what a node downstream of the dropped side-effect node is connected to after the load.
  • A decision by the developer: a loaded job restores its side-effect nodes (read-only, as the other loaded nodes are; a read delivers nothing, per ITL-1091), or the load refuses or warns clearly when the saved job has one.
  • A test for the chosen behaviour; the docs on saving and loading a job say it.
  • No hash, golden value or stored table changes; if the saved file format must change, it is raised first.

Scope & Boundaries

In scope:

  • PipelineJob.save / load and Pipeline.load for side-effect nodes.

Out of scope:

  • What a read of a side-effect job node does after a run (ITL-1091).

ITL-1102 A run checks each source for new data first: polling sources poll, table-backed sources check and reload; a failed check warns

Overview

Split out of ITL-1095 (PR #380) on November 14, 2026, by the developer's instruction not to grow that pull request further ("let's stop bloating 380. Any new aspects should just become new issues").

ITL-1095 makes job.run() and node.run() state-driven: a node that holds a current completed run is not run again, and staleness (the streams' modified-time mechanism) is the only trigger. Without more, a source whose data changes while its object stays the same is never noticed by a re-run. The developer's rules, November 14, 2026:

  • "polling source should update its modified time the moment it finds more data compared to the last time."
  • "For table back database, run should trigger a check / reload attempt to see if there is any new data and if so, update modified time."
  • "For check that fails, issues a warning."

Goals & Success Criteria

  • At the start of a run (job.run, run_async, a node.run() cascade, load_only included) every source feeding the run is given the chance to notice new data; its modified time moves if and only if it found some; then staleness is judged. head() and the preview never check.
  • One hook on the source base class; in-memory sources answer "nothing can change"; a polling source polls; each table-backed kind (Delta, SQLite, PostgreSQL, CSV, SpiralDB, cached and derived sources, proxies) uses the cheapest exact signal of change it has and reloads when it changed. A design per kind is agreed before it is built, including what an explicitly pinned SpiralDB asof does (the coordinator's suggestion: it never moves) and what a reload does to record identity and reuse of stored results.
  • A check that fails does not raise: one warning per source per run in a dedicated category, the source's modified time and held data unchanged, the run proceeds; the run summary names the sources that could not be checked. A later failure to read the source still raises.
  • A poll or check that finds nothing new does not move the modified time, so a current job stays current.
  • Tests on both orchestrators: nothing new means a second run computes nothing; new rows mean the second run processes them and reuses stored results for the rest.
  • Staged so that polling, Delta and SpiralDB land first if the whole set is too much for the v0.3.0 tag.

Scope & Boundaries

In scope:

  • The start-of-run source check and the per-kind implementations.

Out of scope:

  • The state rule itself, load_only, loaded jobs (ITL-1095).
  • The runtime restriction (ITL-1092).

Dependencies & Risks

Review in Linear

@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

LoadedSideEffectJobNode.run now passes unchecked source IDs to its run_alone callback. A regression test checks load-only behavior for a node over a non-job source stream.

Changes: Loaded side-effect run

Layer / File(s) Summary
Callback and load-only test
src/orcapod/pipeline/loaded_side_effects.py, tests/test_pipeline/loaded_job/test_loaded_side_effect_nodes.py, superpowers/plans/hotfix-loaded-side-effect-run-alone.md
The callback accepts unchecked source IDs and continues to run the node in load-only mode. The test checks that the returned view is LOADED_ONLY with ROWS rows and that recorded deliveries remain unchanged. The plan describes the reported mismatch, test scenario, and validation steps.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Possibly related PRs

Merge Risk: Minimal · up to 069ec

This fixes a mismatch between the loaded side-effect node's run callback and the arguments its caller supplies. It does not change load-only behavior, and a regression test covers the fixed path. Merge risk is minimal.

Security architecture risk: Minimal · up to 069ec

The newly working path holds input rows in memory but retains the existing restrictions against delivering side effects or writing the invocation log. No introduced or worsened security or failure-containment risk was established.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the actual change: LoadedSideEffectJobNode.run now passes run_alone the unchecked source IDs instead of using a zero-argument callable. It also names the fix type and scope, and it cit…
Linked Issues check Passed Both linked issues, ITL-1102 and ITL-1094, are marked Done and as closed context. Neither supplies an active requirement, so no linked-issue coding requirements apply to this PR. The PR objectives sta…
Out of Scope Changes check Passed The changes are limited to the reported defect. In src/orcapod/pipeline/loaded_side_effects.py, the lambda passed to run_alone in LoadedSideEffectJobNode.run now accepts one unused argument for the un…
Full details: Title check

Explanation

The title names the actual change: LoadedSideEffectJobNode.run now passes run_alone the unchecked source IDs instead of using a zero-argument callable. It also names the fix type and scope, and it cites the related issue pair. The title is specific and matches the changeset, so it passes.

Full details: Linked Issues check

Explanation

Both linked issues, ITL-1102 and ITL-1094, are marked Done and as closed context. Neither supplies an active requirement, so no linked-issue coding requirements apply to this PR. The PR objectives state the same. The PR description references both issues only as background for the regression fix.

Full details: Out of Scope Changes check

Explanation

The changes are limited to the reported defect. In src/orcapod/pipeline/loaded_side_effects.py, the lambda passed to run_alone in LoadedSideEffectJobNode.run now accepts one unused argument for the unchecked source ids (+2/-1), matching the live SideEffectJobNode.run. The regression test in tests/test_pipeline/loaded_job/test_loaded_side_effect_nodes.py exercises that path. The plan document in superpowers/plans/ describes the same fix and its checks. The PR description and objectives match these changes. Limitation: the text diff could not be read because changed blobs are missing from the checkout. This verdict relies on the AI-generated summary and the PR description, not on direct source inspection.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e42ab469-9a13-497e-9898-318834181ad4
Commits

Reviewing files that changed from the base of the PR and between a899d1e and 069ec85.

Files selected for processing: 3

  • src/orcapod/pipeline/loaded_side_effects.py
  • superpowers/plans/hotfix-loaded-side-effect-run-alone.md
  • tests/test_pipeline/loaded_job/test_loaded_side_effect_nodes.py

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Recent review details

Context from checks skipped due to timeout: 12

  • GitHub Check: test (3.11)
  • GitHub Check: test (3.13)
  • GitHub Check: test (3.12)
  • GitHub Check: CodeRabbit
  • GitHub Check: typecheck / typecheck
  • GitHub Check: pytest / pytest (3.12)
  • GitHub Check: pytest / pytest (3.13)
  • GitHub Check: pytest / pytest (3.11)
  • GitHub Check: test (3.13)
  • GitHub Check: test (3.11)
  • GitHub Check: test (3.13)
  • GitHub Check: test (3.12)
Additional context used

Code guidelines: 1

CLAUDE.md — auto-discovered
Path-based instructions: 2

Place all superpowers-related artifacts (design specs, plans, etc.) in the `superpowers/` directory at the project root — **not** under `docs/`.

CodeRabbit inference engine (CLAUDE.md)

Files:

  • superpowers/plans/hotfix-loaded-side-effect-run-alone.md
Source excerpt: Use [Google style](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) Python docstrings everywhere.

CodeRabbit inference engine (CLAUDE.md)

Files:

  • src/orcapod/pipeline/loaded_side_effects.py
  • tests/test_pipeline/loaded_job/test_loaded_side_effect_nodes.py
Learnings: 1

Common learnings

Learnt from: CR
Repo: metamorphic-labs/orcapod-python

Timestamp: 2026-11-15T02:38:46.232Z
Learning: Source excerpt:
# Claude Code instructions for orcapod-python

## Lint, format and type checks

- Existing type errors and warnings are recorded in `.basedpyright/baseline.json`, a debt
  list to pay down. Never regenerate the baseline (`basedpyright --writebaseline`) to
  hide a new error; regenerate it only to **remove** entries you fixed. Fix the
  new diagnostic, or, when it is a genuine false positive, suppress that one line with a
  targeted `# pyright: ignore[<rule>]` and a reason.
Learnt from: CR
Repo: metamorphic-labs/orcapod-python

Timestamp: 2026-11-15T02:38:46.232Z
Learning: Source excerpt:
# Claude Code instructions for orcapod-python

## Lint, format and type checks

- New and changed code must be ruff-clean and ruff-formatted, and must add no
  basedpyright errors. Warnings are reported but do not fail the check
  (`failOnWarnings = false`); still do not add them needlessly. Run
  `uv run ruff format .` and `uv run ruff check .` before committing;
  `uv run pre-commit install` once per clone runs both on every commit.
Additional comments: 3

src/orcapod/pipeline/loaded_side_effects.py: 1

346-347: LGTM!

tests/test_pipeline/loaded_job/test_loaded_side_effect_nodes.py: 1

408-425: LGTM!

superpowers/plans/hotfix-loaded-side-effect-run-alone.md: 1

1-11: LGTM!

@eywalker
eywalker merged commit 118b037 into main Nov 15, 2026
21 checks passed
@eywalker
eywalker deleted the eywalker/hotfix-loaded-side-effect-run-alone branch November 15, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant