Repository navigation
fix(pipeline): reuse completed work on unchanged reruns #359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
9954640
c58d687
f08ed22
5382d5a
23e1273
64e7a98
0cbc892
73db288
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| """Measure work retained by a job, independently of function execution cost. | ||
|
|
||
| Run with ``uv run python bench/bench_pipeline_cached_rerun.py --rows 1000``. | ||
| Counters, rather than timing thresholds, distinguish reuse from fast recomputation. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import time | ||
| from unittest.mock import patch | ||
|
|
||
| from orcapod import PipelineJob, function_pod | ||
| from orcapod.core.operators.group_by import GroupBy | ||
| from orcapod.databases import InMemoryArrowDatabase | ||
| from orcapod.hashing.arrow_hashers import StarfixArrowHasher | ||
| from orcapod.pipeline import AsyncPipelineOrchestrator | ||
| from orcapod.sources import DictSource | ||
|
|
||
|
|
||
| def measure(rows: int, grouped: bool, asynchronous: bool) -> None: | ||
| """Print cold and warm run times and counts for one independent job.""" | ||
| counts = {"functions": 0, "arrow_hashes": 0, "group_by": 0} | ||
|
|
||
| @function_pod(output_keys="y") | ||
| def increment(x: int) -> int: | ||
| counts["functions"] += 1 | ||
| return x + 1 | ||
|
|
||
| @function_pod(output_keys="y") | ||
| def total(x: list[int]) -> int: | ||
| counts["functions"] += 1 | ||
| return sum(x) | ||
|
|
||
| source = DictSource( | ||
| data=[{"g": i // 100, "k": i, "x": i} for i in range(rows)], | ||
| tag_columns=["g", "k"], | ||
| source_id="benchmark", | ||
| ) | ||
| with PipelineJob(name="cache_benchmark", store=InMemoryArrowDatabase()) as job: | ||
| if grouped: | ||
| total.pod(source.group_by(["g"]).drop_tag_columns(["k"]), label="result") | ||
| else: | ||
| increment.pod(source, label="result") | ||
|
|
||
| original_hash = StarfixArrowHasher.hash_table | ||
| original_group = GroupBy.unary_static_process | ||
|
|
||
| def hash_table(self, *args, **kwargs): | ||
|
Check warning on line 50 in bench/bench_pipeline_cached_rerun.py
|
||
| counts["arrow_hashes"] += 1 | ||
| return original_hash(self, *args, **kwargs) | ||
|
Check warning on line 52 in bench/bench_pipeline_cached_rerun.py
|
||
|
|
||
| def group(self, *args, **kwargs): | ||
| counts["group_by"] += 1 | ||
| return original_group(self, *args, **kwargs) | ||
|
|
||
| with ( | ||
| patch.object(StarfixArrowHasher, "hash_table", hash_table), | ||
| patch.object(GroupBy, "unary_static_process", group), | ||
| ): | ||
| for run in ("cold", "warm", "warm_again"): | ||
| counts.update(functions=0, arrow_hashes=0, group_by=0) | ||
| started = time.perf_counter() | ||
| summary = job.run( | ||
| orchestrator=AsyncPipelineOrchestrator() if asynchronous else None | ||
| ) | ||
| elapsed = time.perf_counter() - started | ||
| if not summary.ok: | ||
| raise RuntimeError(str(summary)) | ||
| print( | ||
| json.dumps( | ||
| { | ||
| "rows": rows, | ||
| "grouped": grouped, | ||
| "async": asynchronous, | ||
| "run": run, | ||
| "seconds": elapsed, | ||
| **counts, | ||
| } | ||
| ), | ||
| flush=True, | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| parser.add_argument("--rows", type=int, default=1000) | ||
| options = parser.parse_args() | ||
| if options.rows < 1: | ||
| parser.error("--rows must be positive") | ||
| for is_grouped in (False, True): | ||
| for is_async in (False, True): | ||
| measure(options.rows, is_grouped, is_async) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| ### Fixed | ||
|
|
||
| - Repeated pipeline runs reuse completed operator work and retained row hashes when | ||
| their input snapshots are unchanged. Sync execution preserves row objects across | ||
| node boundaries; async execution reuses valid in-memory function results. Unknown | ||
| or changing inputs, explicit invalidation and failed work remain subject to normal | ||
| execution and recovery. | ||
| - A SpiralDB source retains a successfully read pinned snapshot for later sync, | ||
| async and table reads. Construction remains lazy; partial, failed and cancelled | ||
| reads do not publish a snapshot. Retention uses memory proportional to the snapshot; | ||
| `batch_rows` bounds processing slices, not total retained memory. Construct a new | ||
| source to read a newer table version. | ||
| - Unchanged job reruns preserve node modification times so external lazy readers | ||
| remain current. Source nodes follow their bound source's freshness. Node-owned | ||
| stored row views preserve declared types without changing live Python objects. | ||
| - `PipelineJob.clear_cache()` releases retained working sets without deleting stored | ||
| results. Spiral sources also expose `clear_cache()` and a `retain_snapshot=False` | ||
| streaming opt-out; clearing during a scan prevents that scan republishing a cache. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| ## `src/orcapod/pipeline/` — completed work on unchanged reruns | ||
|
|
||
| ### CR1 — Job reruns rebuild rows and hashes despite retained function results | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1063 | ||
|
|
||
| An unchanged job visits every node again. The sync orchestrator materializes retained | ||
| `(Tag, Data)` objects as a new Arrow stream, losing their memoized hashes. Operator | ||
| execution constructs a new stream despite holding a previous output. Async function | ||
| execution reloads stored results on every run. SpiralDB sources pin a table version but | ||
| rescan it and rebuild provenance on each read. | ||
|
|
||
| Measured at `89160c47`, a 1,000-row flat job's warm sync run performs 2,000 Arrow hashes | ||
| and its async run 1,000, with no user-function calls. Grouping the source into 10 rows still | ||
| reruns GroupBy and performs 20 hashes on both orchestrators. A retained standalone stream | ||
| read reuses its objects and does not repeat this work. | ||
|
|
||
| Approved design: `superpowers/specs/2026-11-13-pipeline-cache-reuse-design.md`. | ||
| Reproduction: `uv run python bench/bench_pipeline_cached_rerun.py --rows 1000`. | ||
|
|
||
| Completed graph execution now retains operator results and exact-row entry IDs against | ||
| known unchanged snapshots. Sync boundaries retain Arrow-decoded row views to preserve | ||
| declared-schema behavior; async functions reuse their valid in-memory results. Pinned | ||
| Spiral sources publish retained batch streams only after complete successful reads. | ||
| Unknown sources, direct node calls, changed bindings and failed work remain conservative. | ||
|
|
||
| Validated with 11,625 passing tests (119 skipped, 19 expected failures), zero type-check | ||
| errors, clean Ruff checks and independent review. Warm benchmark runs perform zero new | ||
| Arrow hashes and zero GroupBy computations. Remaining traversal/materialization costs | ||
| and potential further optimizations are tracked by ITL-1064. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
F5. Everything here is under
### Fixed. The new API (PipelineJob.clear_cache(include_sources=...),SpiralDBTableSource(retain_snapshot=...),SpiralDBTableSource.clear_cache()) belongs under### Added. Retaining pinned Spiral snapshots by default changes memory behaviour, so it belongs under### Changed, which is where a user who relies on bounded streaming will look.Posted by a Claude Code agent on behalf of @eywalker.