Skip to content

feat(hashing)!: v0.3 hash step (ITL-967) - #324

Merged
eywalker merged 116 commits into
mainfrom
eywalker/itl-967-v03-hash-step-combine-the-hash-changing
Nov 12, 2026
Merged

eywalker merged 116 commits into
mainfrom
eywalker/itl-967-v03-hash-step-combine-the-hash-changing

Conversation

@eywalker

@eywalker eywalker commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes ITL-967
Fixes ITL-910
Fixes ITL-944
Fixes ITL-954
Fixes ITL-945
Fixes ITL-942
Fixes ITL-935

Supersedes #309, #313, #318, #314, #315, #319

Summary

The v0.3 hash step in one pull request: the six hash-changing branches merged with their history (git merge --no-ff, no squash), and one golden regeneration. The design is superpowers/specs/2026-11-11-itl-938-versioning-system-design.md (Parts 5 to 9); every conflict was resolved to agree with it.

Not included: #301 (ITL-905, dict and set order). Two of its points are implemented provisionally and still await the developer (the key-ordered in-memory copy of dict values; raising on NaN or colliding keys), so it would bring an open design question.

What each issue does

Issue Original PR What it does
ITL-910 #309 starfix 0.5 (hash format 2): every Arrow-derived hash moves; fixed_size_list columns are hashable; types the hasher refuses raise ValueError.
ITL-944 #313 Data context std:v1:default with the manifest contexts/data/v1.json; __data_context on every stored row; layouts pdb_v2, rdb_v2, tdb_v2, new odb_v1 and sdb_v1; reads keep rows of the current data context; old-layout tables are reported and never read; side-effecting nodes of both kinds raise SideEffectLogDataContextError; __pf_var_function_version.
ITL-954 #318 ArrowDatabaseProtocol.list_table_paths; tables written before v0.3 are found by name under the job and function or pod name and told apart by layout; a store that cannot list warns and proceeds.
ITL-945 #314 Hash label rule: labels semantic_v1, arrow_v1, directory_v1 (file hash keeps sha256), each defined once in code; no label in any hash input; bare digests inside names; labelled row keys for operator-cache and cached-source tables; default derived source ids keep the label.
ITL-942 #315 Wrong-hash fixes: Batch/GroupBy source-id fold, declared nested Arrow types, set order, dict key types, "__type__" forgery (Encoded), column-selection identity, record-id type, namedtuple field order, one default function version.
ITL-935 #319 Logical type identifiers: extension name plus metadata, written by every built-in type ({"version":1}) and hashed in full; native list / set containers carry no version; [] and [None] no longer hash alike in a list of a value-hashed type.

Merge points and how each was resolved

Point Resolution Test
a. ITL-944 x ITL-945 v1.json and context_schema.json carry no hasher_id; v1.json changes records the label rule. ITL-944's rewrites of operator_node._read_stored_records and cached_source._read_cached_records win; ITL-945's labelled row keys, raw digests in hash inputs and bare path digests are kept. Files ITL-944 removed stay removed. test_hash_label_rule.py (single definition, paths, row keys); test_hash_step_combined.py::TestStoredPathsAndKeys
b. ITL-945 x ITL-935, cell token utf8("<name>;<hex metadata>") + b"::" + raw digest; _encode_container_elements kept. test_extension_type_hashing.py, test_array_value_format.py::test_table_token_carries_extension_...
c. ITL-945 x ITL-942 GenericAliasHandler and UnionTypeHandler return Encoded whose fields hold ContentHash objects; the strict xfail on test_hash_key_differs_from_its_string_spelling is removed (it passes). test_wrong_hash_findings.py (test_nested_hash_differs_from_literal_hash_dict, test_hash_key_differs_from_its_string_spelling); test_logical_type_identifiers.py::test_native_container_hashes_like_any_generic_alias_in_the_schema_hash
d. ITL-935 x ITL-944 No manifest edit needed from ITL-935; native containers carry no version, other built-ins {"version":1}. test_logical_type_identifiers.py
e. Default source ids DerivedSource and node.as_source() keep the labelled form, label semantic_v1. test_hash_label_rule.py::test_default_derived_source_ids_keep_the_hash_label
f. Changelog One fragment per issue in changelog.d/ (ITL-910, 944, 954, 945, 942, 935). CHANGELOG.md differs from main only by the two corrective edits to the ITL-888 text (the second as reworded by #313). ITL-945.md, ITL-944.md, ITL-910.md and main's ITL-926.md were corrected where a sentence was no longer true of the combined result. assemble_changelog.py --check: 12 fragments OK
g. CLAUDE.md / .zed/rules Main's text plus each branch's sections; same sections in both files.
h. CodeRabbit findings on #309's code (raised on #315) Plan approved by the coordinator. Fixed: a read-only OperatorJobNode takes its data context from its descriptor (was AttributeError); SideEffectJobNode checks for unmatched completions before every run, not once per node object; the side-effect error, docs and fragment no longer offer a fresh store as a safe retry; two test nitpicks. Moot: two findings on files ITL-944 removed. Declined: moving the check off the event loop in async_execute (every store access in the async paths is synchronous). test_hash_step_combined.py
i. ITL-954 after combining Passes on the real v0.2.0 store, see below.
main x ITL-910 (found here) main's ITL-926 wrapper that names unhashable columns caught NotImplementedError, which starfix 0.5 no longer raises. It now catches ValueError and raises ValueError naming every refused column and its type (coordinator decision). test_stream_builder.py::TestUnhashableColumnTypes
main (#316 ITL-955, #317 ITL-939, #321 ITL-968, #310, #311 ITL-943) Merged, golden regenerated last each time. #317: as_source() keeps the labelled id without tags_as_data; the default ids derived:<label>:<hex> and tags_as_data:<label>:<hex> take the label semantic_v1 through its single definition; the label-rename test allows exactly the 8 entries built from such an id (rule 5a). #321: this branch's design-issue entries are files under design_issues/ and pass scripts/design_issues.py --check. test_hash_label_rule.py, test_derived_source_tags_as_data.py
CodeRabbit round 1 on this PR, with the findings raised on #322 and #323 Plan approved by the coordinator; see the replies on the threads and the top-level comment. Main fixes: the logical type identity fix (below); identifier checks on already extension-typed columns; DataFunctionWrapper inherits its function's data context; side-effecting function nodes check before every run; the async operator path checks input contexts before forwarding; Delta listing does not follow symlinks and obeys AWS_ALLOW_HTTP; read-only function nodes name their data context; docstrings on every new definition. tests named in the replies

Design-issue entries of the six issues are one file each under design_issues/ (ITL-968); the edit to the existing entry SP2 is in design_issues/ITL-926-SP2.md.

Golden report: relative to main (c8ab0c67)

Method: the golden values were generated at main and at each of the six merge commits, then before and after the logical type identity fix, and consecutive states compared. Every entry that moves is attributed to the step or steps at which it moved. No entry moves without a cause; ITL-954 moves nothing. The golden files were regenerated once for the six branches, and again as the last step after each merge of main and after the identity fix.

Entries
On main 833
Unchanged 36
Changed: label text only, digest identical 41
Changed: value (digest) moved 514 (76 of them also renamed)
Renamed only (the entry name embeds a hash that moved; value identical) 242
Removed 0
Added 71 (21 ITL-910, 32 ITL-942, 18 identity fix)
Now 904

Entries touched per cause (an entry can have several): ITL-945 649, ITL-910 205, ITL-935 92, ITL-942 39, ITL-944 9, identity fix (ITL-942 follow-up) 27 (9 changed, 18 added). Entries that ITL-939 (#317) and ITL-943 (#311) added on main after the six branches were cut are regenerated under the hash step and shown as one step each: 38 and 109 entries.

Logical type identity fix (follow-up to ITL-942 finding I, found by CodeRabbit on #323, decided by the developer through the coordinator on November 12). Since ITL-942 a column of a value-hashed logical type (UUID, File, Directory, array, DataFrame, Series, lists and sets of them) with zero rows or only nulls hashed like a plain large_binary column and like every other such type; main gave distinct hashes. The hashed field now keeps the logical type's identifier as its ARROW:extension:name / ARROW:extension:metadata metadata (confirmed on starfix 0.5.0 at top level and inside list, struct, fixed-size list, list of struct and struct of list; map only at schema level, since starfix refuses map data). It moves 9 entries, by group: arrow:extension_table 1 (extension.uuid); datagram:uuid 2, datagram:ndarray 2, datagram:dataframe 2, datagram:set_uuid 2 (the row data hashes). It adds 18: arrow:extension_no_values (zero-row and all-null columns of each value-hashed type and of plain large_binary, all 18 distinct). design_issues/ITL-967-WH2.md.

In the tables, ITL-xxx(what): digest the value's digest moved, label only the label text, layout only a layout component inside the value, key the entry's name.

arrow.json (53 entries on main, 112 now)

Status Cause (step, what moved) Entries
added ITL-910 21
added ITL-942 20
added ITL-942 follow-up (ITL-967) 18
value changed ITL-910(digest), ITL-945(digest), ITL-935(digest), ITL-942 follow-up (ITL-967)(digest) 1
value changed ITL-910(digest), ITL-945(label) 51
value changed ITL-910(digest), ITL-945(label), ITL-935(digest) 1

By group:

Group Unchanged Label/layout text only Value changed Key renamed (hash in key) Added Removed Causes
extension_no_values 0 0 0 0 18 0 ITL-942 follow-up
extension_schema 0 0 0 0 5 0 ITL-910
extension_table 0 0 3 0 2 0 ITL-910, ITL-935, ITL-942 follow-up , ITL-945
schema 0 0 25 0 17 0 ITL-910, ITL-942, ITL-945
table 0 0 25 0 17 0 ITL-910, ITL-942, ITL-945

datagram.json (171 entries on main, 171 now)

Status Cause (step, what moved) Entries
key renamed only ITL-945(key) 48
key renamed only ITL-945(key), ITL-935(key) 32
value changed ITL-910(digest), ITL-945(digest), ITL-935(digest), ITL-942 follow-up (ITL-967)(digest) 8
value changed ITL-910(digest), ITL-945(label) 28
value changed ITL-910(digest), ITL-945(label), ITL-935(digest) 4
value changed ITL-945(digest) 27
value changed ITL-945(digest), ITL-935(digest) 24

By group:

Group Unchanged Label/layout text only Value changed Key renamed (hash in key) Added Removed Causes
arrow_source 0 0 36 36 0 0 ITL-910, ITL-945
dataclass 0 0 5 4 0 0 ITL-910, ITL-935, ITL-945
dataframe 0 0 5 4 0 0 ITL-910, ITL-935, ITL-942 follow-up , ITL-945
list_path 0 0 5 4 0 0 ITL-910, ITL-935, ITL-945
ndarray 0 0 5 4 0 0 ITL-910, ITL-935, ITL-942 follow-up , ITL-945
nested 0 0 5 4 0 0 ITL-910, ITL-945
optional 0 0 5 4 0 0 ITL-910, ITL-945
path 0 0 5 4 0 0 ITL-910, ITL-935, ITL-945
pydantic 0 0 5 4 0 0 ITL-910, ITL-935, ITL-945
scalars 0 0 5 4 0 0 ITL-910, ITL-945
set_uuid 0 0 5 4 0 0 ITL-910, ITL-935, ITL-942 follow-up , ITL-945
uuid 0 0 5 4 0 0 ITL-910, ITL-935, ITL-942 follow-up , ITL-945

pipeline.json (414 entries on main, 420 now)

Status Cause (step, what moved) Entries
added ITL-942 6
key renamed only ITL-945(key) 96
key renamed only hash step on entries ITL-943 added on main(key) 48
label/layout text only hash step on entries ITL-943 added on main(label) 4
unchanged - 1
value changed ITL-910(digest), ITL-945(label) 64
value changed ITL-944(layout), ITL-945(digest) 2
value changed ITL-945(digest) 86
value changed hash step on entries ITL-943 added on main(digest) 55
value changed, key renamed ITL-910(digest), ITL-944(key), ITL-945(digest+key) 2
value changed, key renamed ITL-910(digest), ITL-944(key), ITL-945(key+label) 2
value changed, key renamed ITL-945(digest+key) 48
value changed, key renamed ITL-945(key), ITL-942(digest) 4
value changed, key renamed hash step on entries ITL-943 added on main(digest+key) 2

By group:

Group Unchanged Label/layout text only Value changed Key renamed (hash in key) Added Removed Causes
batch 0 0 7 4 0 0 ITL-910, ITL-942, ITL-945
column_selection 1 0 21 16 4 0 ITL-910, ITL-942, ITL-945
drop_nulls 0 2 24 24 0 0 hash step on entries ITL-943 added on main
drop_nulls_all 0 2 8 4 0 0 hash step on entries ITL-943 added on main
drop_nulls_job 0 0 25 22 0 0 hash step on entries ITL-943 added on main
group_by 0 0 7 4 0 0 ITL-910, ITL-942, ITL-945
index_pick 0 0 10 8 0 0 ITL-910, ITL-945
job 0 0 40 28 0 0 ITL-910, ITL-944, ITL-945
join_pod 0 0 26 16 0 0 ITL-910, ITL-945
map_data 0 0 6 6 0 0 ITL-910, ITL-945
merge_join 0 0 13 8 0 0 ITL-910, ITL-945
pod 0 0 6 6 0 0 ITL-910, ITL-945
pod_default_version 0 0 0 0 2 0 ITL-942
polars_filter 0 0 5 4 0 0 ITL-910, ITL-945
polars_filter_expr 0 0 10 8 0 0 ITL-910, ITL-945
polars_filter_expr_job 0 0 35 24 0 0 ITL-910, ITL-944, ITL-945
semi_join 0 0 5 4 0 0 ITL-910, ITL-945
sources 0 0 17 16 0 0 ITL-910, ITL-945

schema.json (131 entries on main, 131 now)

Status Cause (step, what moved) Entries
key renamed only ITL-945(key) 12
key renamed only hash step on entries ITL-939 added on main(key) 6
label/layout text only ITL-945(label) 1
unchanged - 21
value changed ITL-910(digest) 1
value changed ITL-910(digest), ITL-944(digest), ITL-945(digest), ITL-942(digest) 3
value changed ITL-910(digest), ITL-945(label) 12
value changed ITL-945(digest) 23
value changed ITL-945(digest), ITL-935(digest) 14
value changed hash step on entries ITL-939 added on main(digest) 20
value changed, key renamed ITL-910(digest), ITL-945(key) 6
value changed, key renamed hash step on entries ITL-939 added on main(digest+key) 12

By group:

Group Unchanged Label/layout text only Value changed Key renamed (hash in key) Added Removed Causes
compute_schema_hash 0 0 2 0 0 0 ITL-935, ITL-945
derived_source_tags_as_data 1 0 32 18 0 0 hash step on entries ITL-939 added on main
multi_field 0 1 5 0 0 0 ITL-935, ITL-945
single_field 18 0 18 0 0 0 ITL-935, ITL-945
source_schema_hash 0 0 3 0 0 0 ITL-935, ITL-945
source_stream_builder 2 0 31 18 0 0 ITL-910, ITL-942, ITL-944, ITL-945

semantic.json (64 entries on main, 70 now)

Status Cause (step, what moved) Entries
added ITL-942 6
label/layout text only ITL-945(label) 36
unchanged - 14
value changed ITL-910(digest), ITL-945(label) 1
value changed ITL-945(digest) 5
value changed ITL-945(digest), ITL-935(digest) 2
value changed ITL-945(label), ITL-935(digest) 6

By group:

Group Unchanged Label/layout text only Value changed Key renamed (hash in key) Added Removed Causes
polars_expr 10 0 0 0 0 0 -
type 0 5 13 0 0 0 ITL-935, ITL-945
value 4 31 1 0 6 0 ITL-910, ITL-942, ITL-945

The full per-entry list (every moved entry with its status and cause) is superpowers/plans/2026-11-12-itl-967-golden-entries.md.

Pinned values outside the golden directory, re-recorded once: hash_samples/schema_hash_golden.json (6 schemas with File, Directory, Path: ITL-935); three pins in test_canonical_type_names.py (ITL-945, ITL-935); the producer-less stream content hashes in test_struct_layout_identity.py (ITL-944, ITL-945); the Spiral source pins made on main (ITL-910, ITL-945).

ITL-954 re-verification on the combined branch

$ uv run pytest tests/test_pipeline/test_pre_v03_tables.py tests/test_databases/test_list_table_paths.py -v
================= 85 passed, 1 skipped, 37 warnings in 10.43s ==================

Skipped: test_s3_lists_tables_and_nested_tables (MinIO needs Docker; runs in CI). The as_written cases now exercise really moved hashes: on the v0.2.0 fixture, the function tables v0.2 wrote are at v02full/plus_one/semantic_v0.1:f65f2aaa…/v0/python.function.v0/schema:semantic_v0.1:b2d5c5d7…/pdb_v1 and this branch writes v02full/plus_one/eb1f77fe…/v0/python.function.v0/schema:8bfcc700…/pdb_v2. The sink and the ctx_arg pod raise before delivering, in sync and async runs, and the v0.2 files stay byte-identical.

A v0.2 operator cache is not replayed (asked after a CodeRabbit finding on #316): test_operator_cache_of_v02_is_not_replayed, on the same fixture as written and with hashes moved. The v0.2 cache of SelectDataColumns (cache_mode=LOG) holds rows; a v0.3 node in REPLAY mode returns none of them and the old files are unchanged. So a GroupBy cache written before ITL-955 cannot be replayed in the old shape by v0.3.

Known gaps

  • fix(hashing): dict and set values hash the same whatever order they were built in (ITL-905) #301 (ITL-905) is not part of the step (see above).
  • The older-completions check runs synchronously inside async_execute, like every other store access there.
  • A read-only operator node loaded from a job file still cannot find its rows: the file carries no pipeline_path and a content hash that differs from the one the run stored (ITL-904); only the AttributeError is fixed here.
  • A saved operator descriptor carries no cache_mode.
  • v1.json stays open (released_in: null, no golden digest) until the v0.3.0 release commit.
  • Carrying a data context in memory is incomplete (ITL-951): a non-default data context is not preserved through streams, cached sources' configs or operator materialization. Only std:v1:default ships.
  • A third-party logical type registered for a generic alias enters the schema hash as origin and arguments; a logical type used inside list[T] / set[T] must have UTF-8 metadata.
  • ITL-953 (function defaults) and ITL-957 (remaining default source ids) are separate changes that move further golden entries.

Test plan

  • uv run pytest tests/ on Python 3.11, 3.12 and 3.13, one at a time, at 6dc2a519: 8895 passed, 101 skipped, 18 xfailed on each
  • Skips by reason: 77 PostgreSQL (opt-in, --postgres), 9 SpiralDB integration (opt-in), 8 MinIO (no Docker), 4 deltalake cannot write a table at a path with # or ?, 3 Graphviz dot not installed
  • uv run --with ruff ruff check . and ruff format --check .
  • uv run --with basedpyright basedpyright: exit 0, 0 errors
  • Doc samples and mkdocs build --strict (tests/test_docs, in the suite)
  • uv run python scripts/assemble_changelog.py --check (14 fragments) and uv run python scripts/design_issues.py --check (113 entries)
  • Golden suite, including test_hash_label_rule.py, which recomputes every golden value under renamed labels

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added versioned data contexts and logical-type identifiers, with compatibility checks when reading stored data.
    • Added context-aware storage and reads, plus discovery and reporting of older tables.
    • Expanded Arrow hashing support for fixed-size lists and extension types; unsupported columns now produce clearer errors.
  • Behavior Changes
    • Orcapod v0.3 does not reuse or modify v0.2 results; it recomputes inputs and leaves older data unchanged.
    • Side-effecting nodes stop before delivery when older or mismatched-context logs are found.
    • Removed legacy migration commands and APIs.
  • Documentation
    • Updated v0.3 guides and release notes for storage, hashing, and compatibility behavior.

Edgar Y. Walker and others added 30 commits November 11, 2026 15:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash
pyarrow's canonical extension types and extension-typed schemas with their
extension identity instead of as their storage type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910)

Explicitly approved regeneration (developer decision 3, November 11, 2026):
184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline
68/305, schema 22/92, semantic 1/64); no key added or removed among them.
21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a
chunked Boolean column, canonical extension types and extension schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910)

Adds orcapod.errors.StoreHashLabelError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

A v0.3 node whose pipeline, result, operator, cached-source or side-effect
table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now
raises StoreHashLabelError on read and before its first write, and
PipelineJob.run checks every node before the first one runs, so a refused
store is left exactly as it was. No opt-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndings (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910)

Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses
arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove
StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run
pre-flight. Every whole-table read keeps rows of the current label only
(keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910)

Developer decisions of November 11, 2026: a read that leaves out rows of another
hash label logs one warning with the total; a side-effect node whose invocation
log holds deliveries logged under another Arrow label raises
SideEffectLogHashLabelError before delivering anything.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reported, kept)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…5-hash-format-2-in-orcapod' into eywalker/itl-942-fix-the-wrong-hash-findings-of-the-hash
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…eck (ITL-944)

Add the manifest v1.json (key std:v1:default) and make it the default data
context. released_manifests.json pins the sha256 of every released manifest;
v1 stays open (null) until the release-preparation commit sets its digest.
The manifest schema accepts v<int> as well as v<int>.<int>.

The content hash of a stream without a producer covers its _context_key
column, so three golden entries (schema:source_stream_builder.*.content_hash)
and the stream hashes pinned in test_struct_layout_identity move with the key.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d-layout report (ITL-944)

Work in progress: existing tests that name the old paths, columns and
ResultCache signature are not updated yet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ITL-944)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@eywalker

Copy link
Copy Markdown
Contributor Author

Replies to the findings of the CodeRabbit reviews of 9743004d and a0a3aeac that have no thread. Changes are in 6dc2a519.

Filter both result tables with keep_rows_of_data_context() before dropping the data context column (function_node.py:2377, outside the diff)

Fixed (declined in the first round as unverified; accepted now because every read is meant to keep rows of the current data context only). Both result reads, persistent and ephemeral, keep rows of the node's data context; a current pipeline entry whose result row is of another data context counts as having no stored result. Test test_a_result_of_another_data_context_is_not_joined_to_a_current_entry (failed before).

Recognize struct children whose type is a pa.ExtensionType (database_hooks.py:235-237, outside the diff)

Fixed: such a member is visited, so a readable older identifier is rewrapped as the registered type. An unreadable one was already refused when the type was registered. Test test_extension_typed_struct_child_with_an_unreadable_identifier_is_refused.

Validate the extension metadata before returning the storage decoder (universal_converter.py:1669-1675, outside the diff)

Fixed: the Arrow-to-Python value converter calls require_readable before it hands values to the registered logical type. Test test_values_of_an_unreadable_identifier_are_not_decoded.

Persist the data context key in DerivedSource.to_config() (derived_source.py:324-329, outside the diff, review of 9743004d)

Declined here: this is code of #317 that reached the branch through main, and carrying a data context through configs and in-memory objects is tracked as ITL-951. Only std:v1:default ships, so a reloaded source gets the same context.

Posted by a Claude Code agent on behalf of @eywalker.

@eywalker
eywalker merged commit e140862 into main Nov 12, 2026
20 checks passed
eywalker pushed a commit that referenced this pull request Nov 12, 2026
The squash commit on main has the same tree as 6dc2a51, which this branch
already contains, so every conflict resolves to this branch's side and the
merged tree is identical to the tree of 4f6e6e0.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eywalker pushed a commit that referenced this pull request Nov 12, 2026
The squash commit on main has the same tree as the hash-step head 6dc2a51.
The merged tree is the automatic (conflict-free) merge of this branch with
6dc2a51, whose merge base with this branch is 9743004; no hand resolution.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eywalker pushed a commit that referenced this pull request Nov 12, 2026
The squash commit on main has the same tree as the hash-step head 6dc2a51;
the merge was computed against 6dc2a51 (merge base c4b1e4a). Main now has
#317 (DerivedSource tags_as_data). Resolved by hand:

- derived_source.py: one SOURCE_KIND = 'derived' (config type key and kind of
  the plain default id); the plain default is hash_source_id(SOURCE_KIND, ...);
  the tags_as_data default id is unchanged.
- function_node.py, operator_node.py: as_source(tags_as_data=None) passes no
  source_id and takes the DerivedSource default in both forms.
- test_hash_label_rule.py: the rule 5a allowance covers exactly the default-id
  golden cases (ITL-957's two and the two tags_as_data cases of #317).
- golden/cases.py: both sides' cases; pipeline.json regenerated (10 added).
- docs/reference/hashing.md: source id rows of both sides.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eywalker pushed a commit that referenced this pull request Nov 14, 2026
…bles (ITL-1095)

8314c93 (CodeRabbit O2) listed pre-v0.3 tables only when the run ran a node. A
run that runs nothing but leaves a node out below an unbound source must still
report that node's tables in the job total, as the PR #324 test
test_job_total_counts_the_tables_of_an_excluded_node guards. The listing is now
skipped only for a run that neither runs nor leaves out any node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eywalker added a commit that referenced this pull request Nov 14, 2026
…e) (ITL-1095) (#380)

* test(function_node): failing tests for the rows a job node returns after a run (ITL-1093)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): after a run a job node returns exactly the rows of that run's inputs (ITL-1093)

The node records the base entry ids of its last completed run's inputs and every
read of its rows goes through one helper that keeps only those inputs' rows. The
async run loads every stored record into the cache, so it returned every row ever
stored for the node; a re-run of the same node objects did so on the sync path
too. A run that raised defines no rows. Nothing stored, no hash, no layout changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): ITL-1095 diagnosis and proposal (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(nodes): failing tests: a job node reads empty and stale before a run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)

A function or operator job node shows only the rows of its current run in this
process: before a run, after clear_cache(), after a run that raised, and once stale
(any job node it depends on, or a rebound source, changed since) every read returns
zero typed rows and head() points to run() and node.as_source(). REPLAY no longer
shows stored rows before a run; a loaded node reads empty instead of every stored row
(function) or raising (operator).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read job nodes after a run, not before (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read stored results of loaded jobs through as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): count a run input only when it gave a row; docstrings for the new tests (ITL-1093)

Addresses the CodeRabbit review of PR #378: of two equal inputs where one fails
and the other succeeds, the node now holds one row, as downstream received.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: update operator, delta and stored-context reads to the current-run rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(operator_node): the stored view of a loaded operator raises when its run's rows cannot be named (ITL-1095)

A loaded operator node read empty before a run, so its stored view (as_source(),
get_all_records) is now the way to its run's rows; it raises, as the node's reads
did (ITL-1043), when the job file lacks the run's content hash and nothing is stored
under the computed one. Also stamps an async REPLAY node after its inputs close, so
its replayed rows are not judged stale. Loaded-operator tests read the stored view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(loaded_job): read loaded nodes' stored results through as_source() (ITL-1095)

Four tests that need the loaded node's own load path (one row per entry, EmptyData
placeholders, stream columns) are strict xfails until loaded.run(load_only=True).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a job node shows its current run, as_source() what is stored (ITL-1095)

Concept pages, PipelineJob.load docstring, CLAUDE.md and .zed/rules rule (replaces the
ITL-1093 sentence), changelog fragment and design issue CR1. Staleness across the job
graph moves into a RunHoldingNode mixin (no import cycle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): job.run(load_only=True) fills the nodes from what is stored (ITL-1095)

job.run, job.run_async and both orchestrators take keyword-only load_only. Sources
are read, operators run on the rows that reach them (a LOG cache writes nothing),
each function node takes the results stored for this run's inputs (its own records
under its content hash) and skips and counts the others, and a side-effect node
passes its rows on uncalled. No function call, no delivery, no write.
RunSummary.load_only, NodeRunSummary.not_stored and .inputs_incomplete; the text
and HTML displays say what was loaded and skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: loading without computing, job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: node.run() cascade, state rule and load-only status (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes): node.run() runs the job nodes it depends on, by their state (ITL-1095)

A job node's run(*, load_only=False) is state-driven: a node that holds a completed
run and is not stale (and, for a full run, was not filled load-only, nor anything
above it) returns its rows and runs nothing. Otherwise it runs, after every job node
it depends on that holds no current run, each once, on the sync orchestrator
restricted to that sub-graph (held rows feed the nodes after them), and commits the
run's writes as job.run() does. node.last_run reports the counts; failed rows warn.
A load-only run's rows have their own status, ViewStatus.LOADED_ONLY, captioned
'N loaded, M inputs without a stored result' (and the incomplete inputs above).
CACHE_ONLY nodes and nodes that cannot compute raise from run() unless load_only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run whose end-of-run commit fails leaves no node holding its rows (ITL-1095)

job.run() (both orchestrators) and the node.run() cascade clear every function and
operator job node of the run (not run, stale) before the commit's error propagates.
A load-only run writes nothing, so its commit has nothing to fail on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: running one node, node states, and the run() argument rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): any raise of a run clears the node's rows, also a refusal or an observer failure (ITL-1093)

Addresses the second CodeRabbit review of PR #378: a sync run refused for its
input's data context, or failing in on_node_end, kept the previous run's rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(function_node): a CACHE_ONLY node's run() raises and points to as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: #378's new raised-run tests read the stored view (ITL-1095); load-only path clears on a refused run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): a loaded job is filled from its store by run(load_only=True) (ITL-1095)

loaded.run(load_only=True) and a loaded node's run(load_only=True) read the job's
sources (rebuilt or bound), run the rebuilt operators (a loaded REPLAY operator
replays) and give each function node the results its store holds for those inputs;
nodes below an unbound source stay empty. loaded.run() still raises, now naming
run(load_only=True) and as_source(). The four loaded-job tests that waited for this
stage read through a load-only run again (no xfails left).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(objective): a second node runs before it is read (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): an async run whose output close fails defines no rows (ITL-1093)

Addresses the third CodeRabbit review of PR #378 (outside-diff finding).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(side_effects): a side-effect job node follows the job node state rule (ITL-1095)

After #377 (ITL-1091) a SideEffectJobNode holds the rows its last run passed on; it
now mixes in RunHoldingNode: stale before a run, transitive staleness, a current
node delivers nothing on run() (clear_cache() to deliver again), run() cascades as
every job node's does (shared run_job_node), and a load-only run passes its rows on,
holds them as load-only and delivers nothing. The orchestrators no longer special-case
side-effect nodes in a load-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run() follows the state rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)

Developer, November 14, 2026: job.run() and run_async() use the same plan as
node.run() (node_run.plan_run): a node that holds a current run is held, not run;
its rows feed its successors on both orchestrators (the async one takes held rows
too); a source is read only for nodes that run. A second run of an unchanged job
computes, reads, writes and delivers nothing; the summary reports held nodes
(NodeRunSummary.held: 'held from the previous run'). A failed end-of-run commit
clears only the nodes that ran. Tests that ran one job twice to re-run a node now
clear that node first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nodes): review fixes for the current-run rule and load-only runs (ITL-1095)

CodeRabbit review of c1a97de, plan approved by the coordinator:
- a load-only run never flushes (job.run prepare/failure/finish, the node cascade,
  a node or side-effect node run alone): pending writes of others stay pending;
- a run that raises (an observer at a node's end, a failed store write or commit, a
  failed or cancelled output close) leaves no node it ran holding rows; held nodes
  keep theirs; a failed side-effect run makes the nodes below it stale;
- a standalone LOG operator run commits its write; a full standalone run after a
  load-only run computes and stores; an operator filled by a load-only run shows
  LOADED_ONLY; an operator rebuilt without its operator keeps its saved columns;
- a source slot bound to a stream with no modified time has none (stale below);
- staleness and currency checks visit each ancestor once per traversal;
- a held dependency keeps its incomplete-input ancestry in the summary (test);
- a run with every node held says so; a loaded job that lost a node names
  ITL-1094 instead of failing in the orchestrator;
- staleness tests start from a run (nitpick).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): second review round for the current-run rule (ITL-1095)

CodeRabbit review of b1b9944 and 3cd6bef, plan approved by the coordinator:
- a run that left failed rows (error_policy continue) is never current: the next
  run computes only the failed rows and reads the rest (sync and async);
- a run that runs no node touches no store: no pre-run or end-of-run flush;
- a held load-only node keeps its missing inputs in the summary;
- an interrupted or cancelled run clears the nodes it ran (job.run and cascade);
- a loaded node whose saved inputs were not all restored raises naming ITL-1094
  (checked against the saved graph, in one place);
- an async CACHE_ONLY node holds the rows it sent;
- docs/changelog: only the nodes that ran are cleared; clear each branch or rebind;
  failed rows are retried; docstrings for the planner helpers and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): #379's async error-policy tests read a failed run's nodes as empty (ITL-1095)

After a fail_fast raise every node the run ran reads as not run (empty, stale) under
the current-run rule; the stored rows are checked through as_source(), sync and
async alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(side_effects): a side-effect node filled by a load-only run shows loaded_only (ITL-1095)

Every node kind a load-only run fills (function, operator, side-effect) reports
ViewStatus.LOADED_ONLY; a source node is read as in any run and shows rows. The
side-effect node's async load-only branch now comes before it reads its pod's
configuration (a load-only run delivers nothing), so a podless node can reuse it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): third review round for the current-run rule (ITL-1095)

CodeRabbit review of f02d72a, plan approved by the coordinator:
- the async orchestrator drains the inputs of a held node, so a direct
  run_async(dag, held_rows=...) never blocks a producer (both error policies);
- the loaded-job missing-input check (ITL-1094) covers only the nodes the run
  includes, and exclusion below an unbound source passes through a node the load
  did not restore;
- a commit interrupted by a non-Exception clears the nodes the run ran;
- a CACHE_ONLY node's as_table judges staleness as head and iter_data do;
- pre-v0.3 tables are listed only for a run that runs a node;
- docs and changelog: a current side-effect node is not delivered again by any run
  until cleared; a source holding the data it read when built needs a new source
  object, clear_cache() re-reads only a polling source; test docstrings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): an interrupted end-of-run commit of job.run clears the nodes it ran (ITL-1095)

CodeRabbit review of 8314c93 (outside diff, job.py:1318): a KeyboardInterrupt or
cancellation during _finish_run's flush bypassed both handlers, so the nodes the
run ran kept rows of a commit whose outcome is unknown. A BaseException there now
clears plan.to_run and is re-raised unchanged, as commit_run already does for
node.run().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run that leaves a node out still reports pre-v0.3 tables (ITL-1095)

8314c93 (CodeRabbit O2) listed pre-v0.3 tables only when the run ran a node. A
run that runs nothing but leaves a node out below an unbound source must still
report that node's tables in the job total, as the PR #324 test
test_job_total_counts_the_tables_of_an_excluded_node guards. The listing is now
skipped only for a run that neither runs nor leaves out any node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): a job node's current-run view has one Arrow schema in every state (ITL-1095)

Function, group_by and join nodes over an int8 non-null tag and int32 data:
as_table(all_info=True) has the populated table's schema before a run,
after a run, after clear_cache, after a failed run, while stale and after a
load-only run, sync and async. A join that receives no rows lists its
columns in another order (pre-existing, ITL-1131): strict xfail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): docstrings for the current-run view schema tests (ITL-1095)

CodeRabbit review of 7de3512..410420e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Edgar Y. Walker <>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
eywalker added a commit that referenced this pull request Nov 14, 2026
…ITL-1121) (#388)

* test(function_node): failing tests for the rows a job node returns after a run (ITL-1093)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): after a run a job node returns exactly the rows of that run's inputs (ITL-1093)

The node records the base entry ids of its last completed run's inputs and every
read of its rows goes through one helper that keeps only those inputs' rows. The
async run loads every stored record into the cache, so it returned every row ever
stored for the node; a re-run of the same node objects did so on the sync path
too. A run that raised defines no rows. Nothing stored, no hash, no layout changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): ITL-1095 diagnosis and proposal (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(nodes): failing tests: a job node reads empty and stale before a run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)

A function or operator job node shows only the rows of its current run in this
process: before a run, after clear_cache(), after a run that raised, and once stale
(any job node it depends on, or a rebound source, changed since) every read returns
zero typed rows and head() points to run() and node.as_source(). REPLAY no longer
shows stored rows before a run; a loaded node reads empty instead of every stored row
(function) or raising (operator).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read job nodes after a run, not before (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read stored results of loaded jobs through as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): count a run input only when it gave a row; docstrings for the new tests (ITL-1093)

Addresses the CodeRabbit review of PR #378: of two equal inputs where one fails
and the other succeeds, the node now holds one row, as downstream received.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: update operator, delta and stored-context reads to the current-run rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(operator_node): the stored view of a loaded operator raises when its run's rows cannot be named (ITL-1095)

A loaded operator node read empty before a run, so its stored view (as_source(),
get_all_records) is now the way to its run's rows; it raises, as the node's reads
did (ITL-1043), when the job file lacks the run's content hash and nothing is stored
under the computed one. Also stamps an async REPLAY node after its inputs close, so
its replayed rows are not judged stale. Loaded-operator tests read the stored view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(loaded_job): read loaded nodes' stored results through as_source() (ITL-1095)

Four tests that need the loaded node's own load path (one row per entry, EmptyData
placeholders, stream columns) are strict xfails until loaded.run(load_only=True).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a job node shows its current run, as_source() what is stored (ITL-1095)

Concept pages, PipelineJob.load docstring, CLAUDE.md and .zed/rules rule (replaces the
ITL-1093 sentence), changelog fragment and design issue CR1. Staleness across the job
graph moves into a RunHoldingNode mixin (no import cycle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): job.run(load_only=True) fills the nodes from what is stored (ITL-1095)

job.run, job.run_async and both orchestrators take keyword-only load_only. Sources
are read, operators run on the rows that reach them (a LOG cache writes nothing),
each function node takes the results stored for this run's inputs (its own records
under its content hash) and skips and counts the others, and a side-effect node
passes its rows on uncalled. No function call, no delivery, no write.
RunSummary.load_only, NodeRunSummary.not_stored and .inputs_incomplete; the text
and HTML displays say what was loaded and skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: loading without computing, job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: node.run() cascade, state rule and load-only status (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes): node.run() runs the job nodes it depends on, by their state (ITL-1095)

A job node's run(*, load_only=False) is state-driven: a node that holds a completed
run and is not stale (and, for a full run, was not filled load-only, nor anything
above it) returns its rows and runs nothing. Otherwise it runs, after every job node
it depends on that holds no current run, each once, on the sync orchestrator
restricted to that sub-graph (held rows feed the nodes after them), and commits the
run's writes as job.run() does. node.last_run reports the counts; failed rows warn.
A load-only run's rows have their own status, ViewStatus.LOADED_ONLY, captioned
'N loaded, M inputs without a stored result' (and the incomplete inputs above).
CACHE_ONLY nodes and nodes that cannot compute raise from run() unless load_only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run whose end-of-run commit fails leaves no node holding its rows (ITL-1095)

job.run() (both orchestrators) and the node.run() cascade clear every function and
operator job node of the run (not run, stale) before the commit's error propagates.
A load-only run writes nothing, so its commit has nothing to fail on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: running one node, node states, and the run() argument rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): any raise of a run clears the node's rows, also a refusal or an observer failure (ITL-1093)

Addresses the second CodeRabbit review of PR #378: a sync run refused for its
input's data context, or failing in on_node_end, kept the previous run's rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(function_node): a CACHE_ONLY node's run() raises and points to as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: #378's new raised-run tests read the stored view (ITL-1095); load-only path clears on a refused run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): a loaded job is filled from its store by run(load_only=True) (ITL-1095)

loaded.run(load_only=True) and a loaded node's run(load_only=True) read the job's
sources (rebuilt or bound), run the rebuilt operators (a loaded REPLAY operator
replays) and give each function node the results its store holds for those inputs;
nodes below an unbound source stay empty. loaded.run() still raises, now naming
run(load_only=True) and as_source(). The four loaded-job tests that waited for this
stage read through a load-only run again (no xfails left).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(objective): a second node runs before it is read (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): an async run whose output close fails defines no rows (ITL-1093)

Addresses the third CodeRabbit review of PR #378 (outside-diff finding).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(side_effects): a side-effect job node follows the job node state rule (ITL-1095)

After #377 (ITL-1091) a SideEffectJobNode holds the rows its last run passed on; it
now mixes in RunHoldingNode: stale before a run, transitive staleness, a current
node delivers nothing on run() (clear_cache() to deliver again), run() cascades as
every job node's does (shared run_job_node), and a load-only run passes its rows on,
holds them as load-only and delivers nothing. The orchestrators no longer special-case
side-effect nodes in a load-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run() follows the state rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)

Developer, November 14, 2026: job.run() and run_async() use the same plan as
node.run() (node_run.plan_run): a node that holds a current run is held, not run;
its rows feed its successors on both orchestrators (the async one takes held rows
too); a source is read only for nodes that run. A second run of an unchanged job
computes, reads, writes and delivers nothing; the summary reports held nodes
(NodeRunSummary.held: 'held from the previous run'). A failed end-of-run commit
clears only the nodes that ran. Tests that ran one job twice to re-run a node now
clear that node first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nodes): review fixes for the current-run rule and load-only runs (ITL-1095)

CodeRabbit review of c1a97de, plan approved by the coordinator:
- a load-only run never flushes (job.run prepare/failure/finish, the node cascade,
  a node or side-effect node run alone): pending writes of others stay pending;
- a run that raises (an observer at a node's end, a failed store write or commit, a
  failed or cancelled output close) leaves no node it ran holding rows; held nodes
  keep theirs; a failed side-effect run makes the nodes below it stale;
- a standalone LOG operator run commits its write; a full standalone run after a
  load-only run computes and stores; an operator filled by a load-only run shows
  LOADED_ONLY; an operator rebuilt without its operator keeps its saved columns;
- a source slot bound to a stream with no modified time has none (stale below);
- staleness and currency checks visit each ancestor once per traversal;
- a held dependency keeps its incomplete-input ancestry in the summary (test);
- a run with every node held says so; a loaded job that lost a node names
  ITL-1094 instead of failing in the orchestrator;
- staleness tests start from a run (nitpick).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): second review round for the current-run rule (ITL-1095)

CodeRabbit review of b1b9944 and 3cd6bef, plan approved by the coordinator:
- a run that left failed rows (error_policy continue) is never current: the next
  run computes only the failed rows and reads the rest (sync and async);
- a run that runs no node touches no store: no pre-run or end-of-run flush;
- a held load-only node keeps its missing inputs in the summary;
- an interrupted or cancelled run clears the nodes it ran (job.run and cascade);
- a loaded node whose saved inputs were not all restored raises naming ITL-1094
  (checked against the saved graph, in one place);
- an async CACHE_ONLY node holds the rows it sent;
- docs/changelog: only the nodes that ran are cleared; clear each branch or rebind;
  failed rows are retried; docstrings for the planner helpers and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): #379's async error-policy tests read a failed run's nodes as empty (ITL-1095)

After a fail_fast raise every node the run ran reads as not run (empty, stale) under
the current-run rule; the stored rows are checked through as_source(), sync and
async alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(side_effects): a side-effect node filled by a load-only run shows loaded_only (ITL-1095)

Every node kind a load-only run fills (function, operator, side-effect) reports
ViewStatus.LOADED_ONLY; a source node is read as in any run and shows rows. The
side-effect node's async load-only branch now comes before it reads its pod's
configuration (a load-only run delivers nothing), so a podless node can reuse it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): third review round for the current-run rule (ITL-1095)

CodeRabbit review of f02d72a, plan approved by the coordinator:
- the async orchestrator drains the inputs of a held node, so a direct
  run_async(dag, held_rows=...) never blocks a producer (both error policies);
- the loaded-job missing-input check (ITL-1094) covers only the nodes the run
  includes, and exclusion below an unbound source passes through a node the load
  did not restore;
- a commit interrupted by a non-Exception clears the nodes the run ran;
- a CACHE_ONLY node's as_table judges staleness as head and iter_data do;
- pre-v0.3 tables are listed only for a run that runs a node;
- docs and changelog: a current side-effect node is not delivered again by any run
  until cleared; a source holding the data it read when built needs a new source
  object, clear_cache() re-reads only a polling source; test docstrings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): an interrupted end-of-run commit of job.run clears the nodes it ran (ITL-1095)

CodeRabbit review of 8314c93 (outside diff, job.py:1318): a KeyboardInterrupt or
cancellation during _finish_run's flush bypassed both handlers, so the nodes the
run ran kept rows of a commit whose outcome is unknown. A BaseException there now
clears plan.to_run and is re-raised unchanged, as commit_run already does for
node.run().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run that leaves a node out still reports pre-v0.3 tables (ITL-1095)

8314c93 (CodeRabbit O2) listed pre-v0.3 tables only when the run ran a node. A
run that runs nothing but leaves a node out below an unbound source must still
report that node's tables in the job total, as the PR #324 test
test_job_total_counts_the_tables_of_an_excluded_node guards. The listing is now
skipped only for a run that neither runs nor leaves out any node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(repro): reproduction scripts for the ITL-1121 review comments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plan): ITL-1121 implementation plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): stamp a node before on_node_end, so an observer reading it keeps its rows (ITL-1121)

An observer that read a function job node in on_node_end found it with no modified time
(stale) and cleared the run's rows; the node then read empty after the run. Sync,
async and both load-only paths now stamp before the observer hears the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(operator_node): a run whose lazy output fails to compute holds no run (ITL-1121)

Without a database, and in CacheMode.OFF, an operator node's run() computes its output
while building the view, which was outside the try that clears a failed run: the node
then claimed a current run and its reads raised. The view of a run that computed is now
built inside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(run_summary): say 'nothing to do' only when the run ran no node (ITL-1121)

The headline judged only the function nodes' counts, so a run that ran a side-effect node
(and delivered) or an operator node below a held function node said 'nothing to do;
every node holds a current run'. RunSummary takes ran_nothing from the run's plan
(job.run, run_async and node.run).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(side_effects): a load-only execute whose input raises clears the node (ITL-1121)

The sync load-only branch dropped its held rows but kept its modified time, so the node and
the nodes below it were not stale after the failure. It now clears the node, as the async
load-only branch and the delivering branches do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(repro): ruff-clean the ITL-1121 reproduction scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node_run): a load-only run refuses an input that delivers when read (ITL-1121)

A job node built over a side-effect pod's stream outside a job (SideEffectPodStream) read
it in node.run(load_only=True), which delivered every row although a load-only run
delivers nothing. The run plan now refuses such an input with a ValueError before any
stream is read, for function, operator and side-effect job nodes, as the node run or in
the cascade of a node below it. A full run is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(function_pods): a held side-effect node delivers nothing on a second run (ITL-1121)

Also assert cache reuse with a recording function in test_caching_flows, and record
progress in the plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): ITL-1121 fragment and design issue entry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(itl-1121): type-clean the regression tests and reproduction scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style: ruff format

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(repro): reproduce the PR #388 review's load-only delivery paths (ITL-1121)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): PR #388 review round 1: more load-only delivery paths, operator and log-commit state (ITL-1121)

- A load-only run refuses a side-effect pod's stream outside a job also when it is read
  through a bound source slot (job.run, run_async, node.run, the slot's own run) or
  through another stream outside a job (node_run walks the streams a run reads).
- An operator job node without a database records a load-only run as one.
- A standalone side-effect job node interrupted while committing its invocation log
  is cleared, so the next run runs and commits again.
- The c3 reproduction script orders commits by the write calls, not timestamps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(repro): a held node.run() keeps the previous last_run (observation, ITL-1121)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(node_run): a load-only run refuses an input that computes when read (ITL-1121)

Developer decision (November 14, 2026, 'Refuse is consistent'): a load-only run reads only
job nodes, sources and streams that neither compute nor deliver. A function pod's stream
outside a job (a cached pod's included) is now refused before anything is read, as a
side-effect pod's stream already was: as a node's input, bound to a source slot, or read
through an operator's stream outside the job; on job.run, run_async, node.run and a
source slot's run. One rule, node_run.refuse_unloadable_inputs. Docs say what to do instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plan): ITL-1121 progress notes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(function_pods): name ITL-1130, which re-evaluates the load-only refusal (ITL-1121)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): a job node's current-run view has one Arrow schema in every state (ITL-1095)

Function, group_by and join nodes over an int8 non-null tag and int32 data:
as_table(all_info=True) has the populated table's schema before a run,
after a run, after clear_cache, after a failed run, while stale and after a
load-only run, sync and async. A join that receives no rows lists its
columns in another order (pre-existing, ITL-1131): strict xfail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): docstrings for the current-run view schema tests (ITL-1095)

CodeRabbit review of 7de3512..410420e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(claude): a failed end-of-run commit of node.run() clears only the nodes that ran (ITL-1121)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plan): ITL-1121 merge of main after #380 merged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(node_run): a job node behind an outside stream is read, not run (ITL-1121)

The _streams_read_by docstring said such a node is run as a node of its
own; node.run() reads the rows it holds. Adds the c11 reproduction of
CodeRabbit's finding 4575098786 (#380 behaviour, out of scope here).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(scripts): type the c11 reproduction's pods for basedpyright (ITL-1121)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plan): ITL-1121 merges of #390 and #389

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(node_run): the load-only refusal reads a held table and refuses a SideEffectNode (ITL-1121)

CodeRabbit round 2 on #388. An ArrowTableStream reads only its table, so the
walk no longer enters its upstreams (a load-only run over one built with a
function pod's stream among its upstreams was refused). A SideEffectNode
outside a job delivers when it is read, so a load-only run refuses it as it
refuses a side-effect pod's stream (it delivered as a node's input and bound
to a source slot). Reproduction: superpowers/scripts/itl_1121/c12_*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Edgar Y. Walker <>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
eywalker added a commit that referenced this pull request Nov 15, 2026
…TL-1094) (#381)

* test(function_node): failing tests for the rows a job node returns after a run (ITL-1093)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): after a run a job node returns exactly the rows of that run's inputs (ITL-1093)

The node records the base entry ids of its last completed run's inputs and every
read of its rows goes through one helper that keeps only those inputs' rows. The
async run loads every stored record into the cache, so it returned every row ever
stored for the node; a re-run of the same node objects did so on the sync path
too. A run that raised defines no rows. Nothing stored, no hash, no layout changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): ITL-1095 diagnosis and proposal (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(nodes): failing tests: a job node reads empty and stale before a run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes)!: a job node reads empty and stale before a run (ITL-1095)

A function or operator job node shows only the rows of its current run in this
process: before a run, after clear_cache(), after a run that raised, and once stale
(any job node it depends on, or a rebound source, changed since) every read returns
zero typed rows and head() points to run() and node.as_source(). REPLAY no longer
shows stored rows before a run; a loaded node reads empty instead of every stored row
(function) or raising (operator).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read job nodes after a run, not before (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: read stored results of loaded jobs through as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): count a run input only when it gave a row; docstrings for the new tests (ITL-1093)

Addresses the CodeRabbit review of PR #378: of two equal inputs where one fails
and the other succeeds, the node now holds one row, as downstream received.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: update operator, delta and stored-context reads to the current-run rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(operator_node): the stored view of a loaded operator raises when its run's rows cannot be named (ITL-1095)

A loaded operator node read empty before a run, so its stored view (as_source(),
get_all_records) is now the way to its run's rows; it raises, as the node's reads
did (ITL-1043), when the job file lacks the run's content hash and nothing is stored
under the computed one. Also stamps an async REPLAY node after its inputs close, so
its replayed rows are not judged stale. Loaded-operator tests read the stored view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(loaded_job): read loaded nodes' stored results through as_source() (ITL-1095)

Four tests that need the loaded node's own load path (one row per entry, EmptyData
placeholders, stream columns) are strict xfails until loaded.run(load_only=True).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a job node shows its current run, as_source() what is stored (ITL-1095)

Concept pages, PipelineJob.load docstring, CLAUDE.md and .zed/rules rule (replaces the
ITL-1093 sentence), changelog fragment and design issue CR1. Staleness across the job
graph moves into a RunHoldingNode mixin (no import cycle).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): job.run(load_only=True) fills the nodes from what is stored (ITL-1095)

job.run, job.run_async and both orchestrators take keyword-only load_only. Sources
are read, operators run on the rows that reach them (a LOG cache writes nothing),
each function node takes the results stored for this run's inputs (its own records
under its content hash) and skips and counts the others, and a side-effect node
passes its rows on uncalled. No function call, no delivery, no write.
RunSummary.load_only, NodeRunSummary.not_stored and .inputs_incomplete; the text
and HTML displays say what was loaded and skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: loading without computing, job.run(load_only=True) (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: node.run() cascade, state rule and load-only status (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(nodes): node.run() runs the job nodes it depends on, by their state (ITL-1095)

A job node's run(*, load_only=False) is state-driven: a node that holds a completed
run and is not stale (and, for a full run, was not filled load-only, nor anything
above it) returns its rows and runs nothing. Otherwise it runs, after every job node
it depends on that holds no current run, each once, on the sync orchestrator
restricted to that sub-graph (held rows feed the nodes after them), and commits the
run's writes as job.run() does. node.last_run reports the counts; failed rows warn.
A load-only run's rows have their own status, ViewStatus.LOADED_ONLY, captioned
'N loaded, M inputs without a stored result' (and the incomplete inputs above).
CACHE_ONLY nodes and nodes that cannot compute raise from run() unless load_only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run whose end-of-run commit fails leaves no node holding its rows (ITL-1095)

job.run() (both orchestrators) and the node.run() cascade clear every function and
operator job node of the run (not run, stale) before the commit's error propagates.
A load-only run writes nothing, so its commit has nothing to fail on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: running one node, node states, and the run() argument rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): any raise of a run clears the node's rows, also a refusal or an observer failure (ITL-1093)

Addresses the second CodeRabbit review of PR #378: a sync run refused for its
input's data context, or failing in on_node_end, kept the previous run's rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(function_node): a CACHE_ONLY node's run() raises and points to as_source() (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: #378's new raised-run tests read the stored view (ITL-1095); load-only path clears on a refused run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline): a loaded job is filled from its store by run(load_only=True) (ITL-1095)

loaded.run(load_only=True) and a loaded node's run(load_only=True) read the job's
sources (rebuilt or bound), run the rebuilt operators (a loaded REPLAY operator
replays) and give each function node the results its store holds for those inputs;
nodes below an unbound source stay empty. loaded.run() still raises, now naming
run(load_only=True) and as_source(). The four loaded-job tests that waited for this
stage read through a load-only run again (no xfails left).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(objective): a second node runs before it is read (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(arrays): read the v0.2-store jobs after a load-only run (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): type-narrowing in the node.run() cascade (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(function_node): an async run whose output close fails defines no rows (ITL-1093)

Addresses the third CodeRabbit review of PR #378 (outside-diff finding).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(side_effects): a side-effect job node follows the job node state rule (ITL-1095)

After #377 (ITL-1091) a SideEffectJobNode holds the rows its last run passed on; it
now mixes in RunHoldingNode: stale before a run, transitive staleness, a current
node delivers nothing on run() (clear_cache() to deliver again), run() cascades as
every job node's does (shared run_job_node), and a load-only run passes its rows on,
holds them as load-only and delivers nothing. The orchestrators no longer special-case
side-effect nodes in a load-only run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): ITL-1094 reproduction and proposal (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): failing tests: job.run() follows the state rule (ITL-1095)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a loaded job restores its side-effect nodes, read-only (ITL-1094) (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): restore loaded side-effect nodes from their own module (ITL-1094) (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: a loaded job keeps its side-effect nodes; changelog fragment and design issue (ITL-1094)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(pipeline)!: job.run() follows the job node state rule (ITL-1095)

Developer, November 14, 2026: job.run() and run_async() use the same plan as
node.run() (node_run.plan_run): a node that holds a current run is held, not run;
its rows feed its successors on both orchestrators (the async one takes held rows
too); a source is read only for nodes that run. A second run of an unchanged job
computes, reads, writes and delivers nothing; the summary reports held nodes
(NodeRunSummary.held: 'held from the previous run'). A failed end-of-run commit
clears only the nodes that ran. Tests that ran one job twice to re-run a node now
clear that node first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(nodes): review fixes for the current-run rule and load-only runs (ITL-1095)

CodeRabbit review of c1a97de, plan approved by the coordinator:
- a load-only run never flushes (job.run prepare/failure/finish, the node cascade,
  a node or side-effect node run alone): pending writes of others stay pending;
- a run that raises (an observer at a node's end, a failed store write or commit, a
  failed or cancelled output close) leaves no node it ran holding rows; held nodes
  keep theirs; a failed side-effect run makes the nodes below it stale;
- a standalone LOG operator run commits its write; a full standalone run after a
  load-only run computes and stores; an operator filled by a load-only run shows
  LOADED_ONLY; an operator rebuilt without its operator keeps its saved columns;
- a source slot bound to a stream with no modified time has none (stale below);
- staleness and currency checks visit each ancestor once per traversal;
- a held dependency keeps its incomplete-input ancestry in the summary (test);
- a run with every node held says so; a loaded job that lost a node names
  ITL-1094 instead of failing in the orchestrator;
- staleness tests start from a run (nitpick).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): second review round for the current-run rule (ITL-1095)

CodeRabbit review of b1b9944 and 3cd6bef, plan approved by the coordinator:
- a run that left failed rows (error_policy continue) is never current: the next
  run computes only the failed rows and reads the rest (sync and async);
- a run that runs no node touches no store: no pre-run or end-of-run flush;
- a held load-only node keeps its missing inputs in the summary;
- an interrupted or cancelled run clears the nodes it ran (job.run and cascade);
- a loaded node whose saved inputs were not all restored raises naming ITL-1094
  (checked against the saved graph, in one place);
- an async CACHE_ONLY node holds the rows it sent;
- docs/changelog: only the nodes that ran are cleared; clear each branch or rebind;
  failed rows are retried; docstrings for the planner helpers and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): #379's async error-policy tests read a failed run's nodes as empty (ITL-1095)

After a fail_fast raise every node the run ran reads as not run (empty, stale) under
the current-run rule; the stored rows are checked through as_source(), sync and
async alike.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(side_effects): a side-effect node filled by a load-only run shows loaded_only (ITL-1095)

Every node kind a load-only run fills (function, operator, side-effect) reports
ViewStatus.LOADED_ONLY; a source node is read as in any run and shows rows. The
side-effect node's async load-only branch now comes before it reads its pod's
configuration (a load-only run delivers nothing), so a podless node can reuse it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): third review round for the current-run rule (ITL-1095)

CodeRabbit review of f02d72a, plan approved by the coordinator:
- the async orchestrator drains the inputs of a held node, so a direct
  run_async(dag, held_rows=...) never blocks a producer (both error policies);
- the loaded-job missing-input check (ITL-1094) covers only the nodes the run
  includes, and exclusion below an unbound source passes through a node the load
  did not restore;
- a commit interrupted by a non-Exception clears the nodes the run ran;
- a CACHE_ONLY node's as_table judges staleness as head and iter_data do;
- pre-v0.3 tables are listed only for a run that runs a node;
- docs and changelog: a current side-effect node is not delivered again by any run
  until cleared; a source holding the data it read when built needs a new source
  object, clear_cache() re-reads only a polling source; test docstrings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): an interrupted end-of-run commit of job.run clears the nodes it ran (ITL-1095)

CodeRabbit review of 8314c93 (outside diff, job.py:1318): a KeyboardInterrupt or
cancellation during _finish_run's flush bypassed both handlers, so the nodes the
run ran kept rows of a commit whose outcome is unknown. A BaseException there now
clears plan.to_run and is re-raised unchanged, as commit_run already does for
node.run().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a run that leaves a node out still reports pre-v0.3 tables (ITL-1095)

8314c93 (CodeRabbit O2) listed pre-v0.3 tables only when the run ran a node. A
run that runs nothing but leaves a node out below an unbound source must still
report that node's tables in the job total, as the PR #324 test
test_job_total_counts_the_tables_of_an_excluded_node guards. The listing is now
skipped only for a run that neither runs nor leaves out any node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plans): ITL-1094 merge of #380 rounds 3 and 4; stacked PR (WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): review round 1 for loaded side-effect nodes (ITL-1094)

- Pipeline.load: a side-effect node whose upstream was not restored (a node of
  a kind this version does not know) is skipped with a warning instead of
  failing the whole load (CodeRabbit, graph.py).
- to_invocations(): a loaded side-effect node with no pod raises the explicit
  RuntimeError the loaded function and operator stubs raise, instead of
  building a SideEffectInvocation with pod None (CodeRabbit).
- Tests for both, public interface.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): a job node's current-run view has one Arrow schema in every state (ITL-1095)

Function, group_by and join nodes over an int8 non-null tag and int32 data:
as_table(all_info=True) has the populated table's schema before a run,
after a run, after clear_cache, after a failed run, while stale and after a
load-only run, sync and async. A join that receives no rows lists its
columns in another order (pre-existing, ITL-1131): strict xfail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(pipeline): docstrings for the current-run view schema tests (ITL-1095)

CodeRabbit review of 7de3512..410420e.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a loaded side-effect node has the saved job's content hash (ITL-1094)

PipelineJob.save records each side-effect job node's content hash in the
existing run.node_job_content_hashes map, as for function nodes, and
PipelineJob.load gives it to LoadedSideEffectJobNode; a file without the
entry falls back to the descriptor's (blueprint) hash. A load-only run of
a loaded job whose end node is a side-effect node now records the live
job's pipeline_uri. No hash value or stored table changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(plan): ITL-1094 follow-up, CodeRabbit schema finding (WIP notes)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pipeline): a restored side-effect node passes its upstream's columns through (ITL-1094)

CodeRabbit round on 5185317. A loaded side-effect node returned the saved
default schema and keys whatever the column configuration asked for; it now
inherits the live pass-through to its restored upstream, so system tags are
included and the operators below predict the live job's schemas.

On the sync path, an operator below a node with no producer (a restored
side-effect node, a loaded function node) named its system tags after a
root's schema-only hash, so a load-only run of a loaded job left the function
nodes below a Join unloaded (0 of 4; async 4 of 4). The rebuilt upstream
stream now reports the node's own pipeline hash, as the async path does; no
hash value changes for live nodes or sources.

as_pipeline() of a loaded job keeps a restored side-effect node as a
LoadedSideEffectNode built from the saved entry instead of raising.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Edgar Y. Walker <>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@eywalker
eywalker deleted the eywalker/itl-967-v03-hash-step-combine-the-hash-changing branch November 17, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant