Skip to content

fix(function_node): a row read back from the store carries its stored __data_id (ITL-1060) - #356

Merged
eywalker merged 1 commit into
mainfrom
eywalker/itl-1060-a-result-row-read-back-from-the-store-gets
Nov 13, 2026
Merged

eywalker merged 1 commit into
mainfrom
eywalker/itl-1060-a-result-row-read-back-from-the-store-gets

Conversation

@eywalker

@eywalker eywalker commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes ITL-1060

What

A result row a function node read back from its store (sync iter_data, the async cache-only path, a loaded node) got a fresh uuid7 as its datagram_uuid, not the stored __data_id. _load_cached_entries built the rows through a stream over a table whose __data_id column was dropped. _stored_rows_stream now also returns each row's stored __data_id, and _load_cached_entries rebuilds each Data with it (same columns, types, source info, context; the test checks content hash, source info and table equal the computed row's). Reading the same stored row twice gives the same id.

Evidence

  • Failing-first tests (tests/test_core/function_pod/test_readback_datagram_id.py, public interface, tags not in order): second sync run, second async run, a never-run job on the same store: all 3 failed before the fix (ids differed from the stored __data_id), pass after. Plus: a pod-cached row read twice (already correct: the result-cache lookup passes record_uuid), and read-back row equals the computed row apart from its id.
  • Where datagram_uuid is used (grep of every .datagram_uuid in src): only on the OUTPUT of process_data: the pipeline record's data_record_id (5 sites in function_node.py), result_cache.store (result record id), FunctionPod post-run payload record_id_hash. Never on an input datagram or a read-back row. A source-info token is built from the new output's own uuid when it is computed (data_function.py). So no stored value, source-info token, meta column or operator-cache row could differ between runs because of this: it was an in-memory identity defect.
  • Operator node cache and DerivedSource store no __data_id, so there is no stored id to carry; DerivedSource over a function node reads get_all_records() into a plain table.
  • No hash, golden entry, record id, stored schema, public API or dependency change; golden tests pass. Targeted tests only locally (function_pod, pipeline, golden hashes, design issues, changelog: 3707 passed on 3.12); the full suite on 3.11, 3.12 and 3.13 is verified by CI. ruff, format, basedpyright exit 0.

Coordination

The row-alignment suite (#346) marks 162 read-back cases xfail(strict) with ITL-1060: whichever of #346 and this PR lands second must drop those marks (strict xfail fails once the fix is in).

🤖 Generated with Claude Code

… __data_id (ITL-1060)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
ITL-1060 A result row read back from the store gets a fresh datagram id instead of its stored __data_id

Overview

Found on November 13, 2026 by the row-alignment suite (ITL-1034), through a check added in review: a stored record's __data_id should be the id of the node's own output for that record. Reported as pre-existing (same result on 003d2e82 and cdb4f160); not verified by the coordinator.

Rows that a function node reads back from its store are given a fresh datagram_uuid (a new uuid7), not the stored __data_id, while the row's own source-info token still names the stored __data_id. So for every read-back row, data.datagram_uuid differs from the stored id and from the id inside data.source_info()['y']. It holds for any read-back, including a plain sync re-run that takes every row from the store.

Reproduction (self-contained script on the ITL-1034 branch, superpowers/specs/itl-1034-sweep/datagram_uuid_after_readback.py): a source with tag k and data x, s; a function f(x) labelled n; run a job; build a second job on the same store and run it ("2 rows: 0 computed, 2 from store"); for every row of node.iter_data(), data.datagram_uuid == stored __data_id is false.

It is not a misalignment: values, tokens, record ids and hashes stay with their rows. It is an identity defect: a read-back result is not the datagram its own provenance names, and anything that records data.datagram_uuid downstream records an id that exists nowhere and differs on every read.

Goals & Success Criteria

  • A row read back from the store carries the stored __data_id as its datagram_uuid, on every read-back path: function node (sync, async, loaded), cached pod stream, operator node cache, DerivedSource. Failing-first tests through the public interface.
  • Stated with evidence: where datagram_uuid is used after read-back (source-info tokens of downstream outputs, meta columns written by a downstream operator cache, anything stored), and whether any stored value could have differed between runs because of this.
  • Reading the same stored row twice gives the same datagram id.
  • No hash, golden entry or record id changes (datagram ids are run-time ids, not hash inputs; confirm).

Scope & Boundaries

In scope:

  • The read-back construction of Data in nodes and streams.

Out of scope:

Dependencies

  • Should merge before the v0.3.0 tag if any stored value is affected; otherwise it can follow.

Review in Linear

@eywalker

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Posted by a Claude Code agent on behalf of @eywalker.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

The included review limit has been reached and this organization has disabled usage-based review continuation. Wait for reviews to reset or ask a billing admin to change After included review limits.

Next included review available in 21 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5ae492da-daab-4a05-95ed-a76f221a73d0
Commits

Reviewing files that changed from the base of the PR and between 97263e5 and f4e48d8.

Files selected for processing: 5

  • changelog.d/ITL-1060.md
  • design_issues/ITL-1060-FN1.md
  • src/orcapod/core/nodes/function_node.py
  • superpowers/plans/2026-11-13-itl-1060-readback-datagram-id.md
  • tests/test_core/function_pod/test_readback_datagram_id.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change preserving stored result IDs during datagram readback.

No open issues 🎉

What changed in this PR

Fixes ITL-1060 by preserving a function result’s stored __data_id as its datagram_uuid when reading it back from storage, keeping in-memory identity consistent with provenance.

Changes:

  • Carries stored result IDs through cache loading and rebuilds datagrams with those IDs.
  • Adds regression tests for sync and async readback, fresh jobs, pod caching, and preserved row content.
  • Documents the fix without changing stored schemas, hashes, or public APIs.
File Description
tests/​test_core/​function_pod/​test_readback_datagram_id.py Tests stable readback IDs and preservation of row content and provenance.
superpowers/​plans/​2026-11-13-itl-1060-readback-datagram-id.md Records the implementation approach and evidence.
src/​orcapod/​core/​nodes/​function_node.py Restores stored result IDs when rebuilding cached datagrams.
design_issues/​ITL-1060-FN1.md Records the identity defect as resolved and explains its scope.
changelog.d/​ITL-1060.md Adds a release note for the readback identity fix.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change preserving stored datagram identities on read-back.

No open issues 🎉

Review effort: Balanced

@eywalker
eywalker merged commit 14b0e92 into main Nov 13, 2026
23 checks passed
eywalker pushed a commit that referenced this pull request Nov 13, 2026
…nfiguration (ITL-1034)

Merge main at 5c7a058 (#351 ITL-1050, #354 ITL-1053, #356 ITL-1060, #358
ITL-1045, #360 ITL-1065, #361). With --runxfail only the ITL-1066 cases fail,
so every ITL-1050, ITL-1060, ITL-1065 and ITL-1045 mark is removed.

head(n, cfg) now takes sort_by_tags and all_info: node, operator and stream
heads are checked under every configuration, must be sorted when asked and
must equal as_table(cfg).slice(0, n). Previews before a full read sort the
rows the stream holds; the check allows any order among tied tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@eywalker
eywalker deleted the eywalker/itl-1060-a-result-row-read-back-from-the-store-gets branch November 17, 2026 07:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants