Repository navigation
fix(datagrams): a Data's source info follows its data columns; sync and async store the same records (ITL-1040) #350
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
eywalker
merged 6 commits into
main
from
eywalker/itl-1040-async-runs-store-an-extra-_source_-column
Nov 13, 2026
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
2d05860
fix(datagrams): a Data's source info follows its data columns (ITL-1040)
597fbb8
docs(design_issues): ITL-1040 entry and changelog fragment
6820feb
fix(datagrams): select keeps source info in column order; ITL-838 xfa…
1aca39a
Merge remote-tracking branch 'origin/main' into eywalker/itl-1040-asy…
3f107ac
fix(datagrams): a Data's source-info table is in the order of its dat…
e7b07bc
Merge remote-tracking branch 'origin/main' into eywalker/itl-1040-asy…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ### Fixed | ||
|
|
||
| - Async job runs no longer store an extra `_source_<column>` column for a data column dropped before a function pod, by the automatic select of `pod(stream)` or by an explicit `SelectDataColumns` or `DropDataColumns` (ITL-1040, ITL-838). `Data.drop` and `Data.select` kept the source info of the removed columns, and `Data.as_table(columns={"source": True})` still showed it. A selected `Data`'s source info is now in the order of its columns. Sync and async runs now store the same pipeline records. No hash, record id or stored path changes. Existing tables are not migrated in this release. A Delta pipeline table written by an async run before this fix keeps the extra column, and rows written from now on have null in it. Migrating stores written by earlier versions is the job of the migration tool planned after v0.3 (ITL-1048). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| ## `src/orcapod/core/datagrams/tag_data.py` | ||
|
|
||
| ### DG2 — A dropped data column kept its source info, so async runs stored an extra column | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-1040 | ||
|
|
||
| `Data` did not override `Datagram.drop` or `Datagram.select`, so both left the removed | ||
| columns' entries in `Data._source_info`. The source-info table that `as_table(source=True)` | ||
| and `arrow_schema(source=True)` use was built from all of `_source_info`, while `as_dict` and | ||
| `source_info()` filtered it to the current columns. | ||
|
|
||
| The async paths of `SelectDataColumns` and `DropDataColumns` drop columns one datagram at a | ||
| time with `data.drop`. Their sync paths drop the `_source_` columns from the table | ||
| explicitly. So an async run stored an extra `_source_<dropped column>` in the function node's | ||
| pipeline records that a sync run did not, and the stored table's schema depended on the | ||
| orchestrator. This happened after the automatic select in front of a pod, and after any | ||
| explicit select or drop. On one store, mixing orchestrators then either widened a Delta table | ||
| (with nulls) or failed an in-memory commit with a schema mismatch. The cell values were always | ||
| the row's own. The row-alignment suite found it (ITL-1034). It had been reported for `Data` alone as ITL-838. | ||
|
|
||
| **Fix:** `Data.drop` and `Data.select` remove the source info of the columns they remove, and | ||
| the source-info table holds only the entries of current columns. Sync and async runs now | ||
| store identical records (`tests/test_pipeline/test_sync_async_stored_records.py`). No hash, | ||
| record id or stored path depends on source-info columns. | ||
|
|
||
| A Delta table that an async run wrote before the fix keeps its extra column. Rows written | ||
| after the fix have null in it, which is what a sync run appending to such a table already did | ||
| before the fix. No migration in this release; migrating stores written by earlier versions is the job of the migration tool planned after v0.3 (ITL-1048). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
28 changes: 28 additions & 0 deletions
28
superpowers/plans/2026-11-12-itl-1040-dropped-column-source-info.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # ITL-1040 Dropped data column keeps its source info: plan | ||
|
|
||
| **Goal:** a `Data` without a column carries no source info for it in any view, so sync and | ||
| async runs store identical pipeline records. | ||
|
|
||
| **Cause:** `Datagram.drop` (and `select`) are not overridden by `Data`; they leave the dropped | ||
| column's entry in `Data._source_info`, and `_ensure_source_info_table` builds the source-info | ||
| table (used by `as_table(source=True)` and `arrow_schema(source=True)`) from all of | ||
| `_source_info`, while `as_dict` / `source_info()` filter to `keys()`. The async | ||
| `SelectDataColumns` / `DropDataColumns` call `data.drop`; their sync paths drop `_source_*` | ||
| explicitly. | ||
|
|
||
| **Steps:** | ||
| 1. MEASURE FIRST (send to coordinator before any push): with `main`, an async run of a job | ||
| whose pod input is auto-selected writes a pipeline table with `_source_s`; then the fixed | ||
| build appends new rows to that table (Delta and in memory). Record: succeeds / fails / | ||
| writes nulls; what a read returns. Also state with evidence whether any hash, record id or | ||
| stored path depends on the extra column. | ||
| 2. Failing-first tests: `Data.drop` / `Data.select` + `as_table(source=True)`, | ||
| `arrow_schema(source=True)`; sync == async stored records (auto select, explicit | ||
| select, explicit drop; in-memory and Delta); other Datagram operations (`rename`, | ||
| `update`, `with_columns`, `copy`) and `Tag` system tags checked for the same drift. | ||
| 3. Fix: `Data` keeps `_source_info` to its own keys: override `drop` and `select` to prune, | ||
| and build the source-info table from the entries of current keys only. | ||
| 4. design_issues entry, changelog fragment, full suite, PR "Fixes ITL-1040". | ||
|
|
||
| No hash, golden, API change expected; stored schema of async-written tables changes (the | ||
| extra column disappears from new rows) - the developer decides handling from step 1. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # ITL-1040 measurement: appending to a pipeline table an async run wrote before the fix | ||
|
|
||
| Evidence for the ITL-1040 decision. These scripts are not tests, and the test run does not | ||
| collect them. | ||
|
|
||
| `measure.py` runs each step as a subcommand (see its docstring). The job is source (`k`; `x`, | ||
| `s`) → `times_ten(x)`; the pod's automatic select drops `s`. An in-memory store is carried | ||
| from one build to another by dumping its tables to Arrow files and reloading them. | ||
|
|
||
| - `phase1 <dir>` (run on `main` before the fix): async runs write rows `r0..r3` to Delta | ||
| stores and to a dumped in-memory store. It also measures the baseline on `main`: a sync run | ||
| appending rows `r4, r5` to an async-written table. | ||
| - `phase1b` (run on `main` before the fix): the reverse baseline, an async run appending to | ||
| a sync-written table. | ||
| - `phase2 <dir>` (run on the fixed build): async and sync runs append `r4, r5` to the | ||
| tables phase 1 wrote. | ||
| - `operator_node`: a cached operator node downstream of an async select. Are its stored | ||
| records the same after a sync and an async run? | ||
|
|
||
| Results (November 12, 2026): | ||
|
|
||
| | Store | Who appends | Result | | ||
| |---|---|---| | ||
| | Delta | fixed build, async or sync | succeeds; the old `_source_s` column stays, old rows keep their tokens, new rows get NULL | | ||
| | in memory | fixed build, async or sync | `StoreWriteError` (schema mismatch); cannot happen across processes, since an in-memory store dies with its process | | ||
| | Delta | `main`, sync after async | succeeds; new rows get NULL in `_source_s` (the same state as the fixed build) | | ||
| | Delta | `main`, async after sync | succeeds; the OLD rows get NULL in `_source_s` | | ||
| | in memory | `main`, sync after async, or async after sync | fails (`StoreWriteError` / `ExceptionGroup`) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,248 @@ | ||
| """ITL-1040 measurement: appending to a pipeline table an async run wrote before the fix. | ||
|
|
||
| Usage (from the repository root): | ||
|
|
||
| uv run python superpowers/specs/itl-1040/measure.py phase1 <dir> # on main, unfixed | ||
| uv run python superpowers/specs/itl-1040/measure.py phase1b # on main, unfixed | ||
| uv run python superpowers/specs/itl-1040/measure.py phase2 <dir> # on the fixed build | ||
| uv run python superpowers/specs/itl-1040/measure.py operator_node # either build | ||
|
|
||
| See README.md in this folder for what each phase does and the results. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import shutil | ||
| import sys | ||
| import tempfile | ||
| from collections.abc import Callable | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| import pyarrow as pa | ||
| import pyarrow.feather as feather | ||
|
|
||
| from orcapod import PipelineJob, function_pod | ||
| from orcapod.core.operators import Join, SelectDataColumns | ||
| from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase | ||
| from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator | ||
| from orcapod.sources import ArrowTableSource | ||
| from orcapod.types import CacheMode | ||
|
|
||
| DUMP_ID = "__dump_record_id" | ||
|
|
||
|
|
||
| @function_pod(output_keys="y") | ||
| def times_ten(x: int) -> int: | ||
| """Return ``10 * x``.""" | ||
| return 10 * x | ||
|
|
||
|
|
||
| def source(n: int) -> ArrowTableSource: | ||
| """``n`` rows: tag ``k``; data ``x`` (the pod's input) and ``s`` (dropped).""" | ||
| table = pa.table( | ||
| { | ||
| "k": [f"r{i}" for i in range(n)], | ||
| "x": list(range(n)), | ||
| "s": [f"s{i}" for i in range(n)], | ||
| }, | ||
| schema=pa.schema( | ||
| [ | ||
| pa.field("k", pa.large_string(), nullable=False), | ||
| pa.field("x", pa.int64(), nullable=False), | ||
| pa.field("s", pa.large_string(), nullable=False), | ||
| ] | ||
| ), | ||
| ) | ||
| return ArrowTableSource( | ||
| table, tag_columns=["k"], source_id="src", record_id_column="x" | ||
| ) | ||
|
|
||
|
|
||
| def run(store: Any, n: int, mode: str) -> tuple[PipelineJob, Any]: | ||
| """Run ``times_ten`` over ``n`` rows on ``store`` (``mode`` "sync" or "async").""" | ||
| job = PipelineJob(name="m", store=store) | ||
| with job: | ||
| times_ten.pod(source(n), label="n") # the automatic select of x drops s | ||
| orchestrator = ( | ||
| AsyncPipelineOrchestrator() if mode == "async" else SyncPipelineOrchestrator() | ||
| ) | ||
| return job, job.run(orchestrator=orchestrator) | ||
|
|
||
|
|
||
| def dump_memory(store: InMemoryArrowDatabase, out: Path) -> None: | ||
| """Write every table of ``store`` to ``out`` as Arrow files.""" | ||
| out.mkdir(parents=True, exist_ok=True) | ||
| index = [] | ||
| for i, path in enumerate(store.list_table_paths()): | ||
| table = store.get_all_records(path, record_id_column=DUMP_ID) | ||
| assert table is not None | ||
| feather.write_feather(table, out / f"{i}.arrow") | ||
| index.append(list(path)) | ||
| (out / "paths.json").write_text(json.dumps(index)) | ||
|
|
||
|
|
||
| def load_memory(out: Path) -> InMemoryArrowDatabase: | ||
| """An in-memory store holding the tables ``dump_memory`` wrote.""" | ||
| store = InMemoryArrowDatabase() | ||
| for i, path in enumerate(json.loads((out / "paths.json").read_text())): | ||
| table = feather.read_table(out / f"{i}.arrow") | ||
| store.add_records(tuple(path), table, record_id_column=DUMP_ID, flush=True) | ||
| return store | ||
|
|
||
|
|
||
| def describe(job: PipelineJob, label: str) -> None: | ||
| """Print the node's stored source-info columns and its table.""" | ||
| node = job.nodes["n"] | ||
| records = node.get_all_records(all_info=True) | ||
| assert records is not None | ||
| source_columns = [c for c in records.column_names if c.startswith("_source")] | ||
| print( | ||
| f" [{label}] records: {records.num_rows} rows; source columns {source_columns}" | ||
| ) | ||
| if "_source_s" in records.column_names: | ||
| values = dict( | ||
| zip( | ||
| records.column("k").to_pylist(), records.column("_source_s").to_pylist() | ||
| ) | ||
| ) | ||
| print(f" [{label}] _source_s by row: {dict(sorted(values.items()))}") | ||
| table = node.as_table(all_info=True) | ||
| print(f" [{label}] as_table(all_info): {table.num_rows} rows") | ||
| print(f" [{label}] y: {sorted(table.column('y').to_pylist())}") | ||
|
|
||
|
|
||
| def attempt(label: str, append: Callable[[], tuple[PipelineJob, Any]]) -> None: | ||
| """Run ``append`` and describe the result, or print how it failed.""" | ||
| print(label) | ||
| try: | ||
| job, summary = append() | ||
| except BaseException as exc: # noqa: BLE001 # the measurement reports any failure | ||
| inner = exc.exceptions[0] if isinstance(exc, BaseExceptionGroup) else exc | ||
| print(" FAILS:", type(exc).__name__, "/", type(inner).__name__) | ||
| print(" ", str(inner).splitlines()[0][:220]) | ||
| return | ||
| print(" run:", str(summary).splitlines()[-1].strip()) | ||
| describe(job, label) | ||
|
|
||
|
|
||
| def phase1(out: Path) -> None: | ||
| """On main: async runs write r0..r3; the sync-after-async baseline.""" | ||
| shutil.rmtree(out, ignore_errors=True) | ||
| for name in ("delta_then_async", "delta_then_sync", "delta_main_sync_append"): | ||
| job, _ = run(DeltaTableDatabase(out / name), 4, "async") | ||
| describe(job, f"main, async write, {name}") | ||
| memory = InMemoryArrowDatabase() | ||
| job, _ = run(memory, 4, "async") | ||
| describe(job, "main, async write, memory") | ||
| dump_memory(memory, out / "memory_dump") | ||
| attempt( | ||
| "baseline on main: sync run appends r4, r5 to the async-written Delta table", | ||
| lambda: run(DeltaTableDatabase(out / "delta_main_sync_append"), 6, "sync"), | ||
| ) | ||
| other = InMemoryArrowDatabase() | ||
| run(other, 4, "async") | ||
| attempt( | ||
| "baseline on main: sync run appends r4, r5 to the async-written in-memory table", | ||
| lambda: run(other, 6, "sync"), | ||
| ) | ||
|
|
||
|
|
||
| def phase1b() -> None: | ||
| """On main: an async run appends r4, r5 to a sync-written table.""" | ||
| for kind in ("delta", "memory"): | ||
| store: Any = ( | ||
| DeltaTableDatabase(Path(tempfile.mkdtemp())) | ||
| if kind == "delta" | ||
| else InMemoryArrowDatabase() | ||
| ) | ||
| run(store, 4, "sync") | ||
| attempt( | ||
| f"baseline on main: async run appends to a sync-written {kind} table", | ||
| lambda store=store: run(store, 6, "async"), | ||
| ) | ||
|
|
||
|
|
||
| def phase2(out: Path) -> None: | ||
| """On the fixed build: append r4, r5 to the tables phase 1 wrote.""" | ||
| attempt( | ||
| "delta, fixed async append", | ||
| lambda: run(DeltaTableDatabase(out / "delta_then_async"), 6, "async"), | ||
| ) | ||
| attempt( | ||
| "delta, fixed sync append", | ||
| lambda: run(DeltaTableDatabase(out / "delta_then_sync"), 6, "sync"), | ||
| ) | ||
| attempt( | ||
| "memory, fixed async append", | ||
| lambda: run(load_memory(out / "memory_dump"), 6, "async"), | ||
| ) | ||
| attempt( | ||
| "memory, fixed sync append", | ||
| lambda: run(load_memory(out / "memory_dump"), 6, "sync"), | ||
| ) | ||
|
|
||
|
|
||
| def operator_node() -> None: | ||
| """A cached Join downstream of an async select: stored records, sync vs async.""" | ||
|
|
||
| def stored(orchestrator: Any) -> pa.Table: | ||
| a = pa.table( | ||
| {"k": ["a", "b"], "x": [1, 2], "s": ["p", "q"]}, | ||
| schema=pa.schema( | ||
| [ | ||
| pa.field("k", pa.large_string(), nullable=False), | ||
| pa.field("x", pa.int64(), nullable=False), | ||
| pa.field("s", pa.large_string(), nullable=False), | ||
| ] | ||
| ), | ||
| ) | ||
| b = pa.table( | ||
| {"k": ["a", "b"], "z": [10, 20]}, | ||
| schema=pa.schema( | ||
| [ | ||
| pa.field("k", pa.large_string(), nullable=False), | ||
| pa.field("z", pa.int64(), nullable=False), | ||
| ] | ||
| ), | ||
| ) | ||
| store = InMemoryArrowDatabase() | ||
| job = PipelineJob(name="j", store=store) | ||
| with job: | ||
| selected = SelectDataColumns(["x"])( | ||
| ArrowTableSource(a, tag_columns=["k"], source_id="a") | ||
| ) | ||
| Join()( | ||
| selected, | ||
| ArrowTableSource(b, tag_columns=["k"], source_id="b"), | ||
| label="op", | ||
| ) | ||
| job.nodes["op"].attach_databases( | ||
| pipeline_database=store.at("j"), cache_mode=CacheMode.LOG | ||
| ) | ||
| job.run(orchestrator=orchestrator) | ||
| records = job.nodes["op"].get_all_records(all_info=True) | ||
| assert records is not None | ||
| return records.sort_by("k") | ||
|
|
||
| sync = stored(SyncPipelineOrchestrator()) | ||
| async_ = stored(AsyncPipelineOrchestrator()) | ||
| for name, table in (("sync", sync), ("async", async_)): | ||
| columns = [c for c in table.column_names if c.startswith("_source")] | ||
| print(f"{name}: stored operator records, source-info columns {columns}") | ||
| print("stored operator tables equal:", sync.equals(async_)) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| command = sys.argv[1] | ||
| if command == "phase1": | ||
| phase1(Path(sys.argv[2])) | ||
| elif command == "phase1b": | ||
| phase1b() | ||
| elif command == "phase2": | ||
| phase2(Path(sys.argv[2])) | ||
| elif command == "operator_node": | ||
| operator_node() | ||
| else: | ||
| raise SystemExit(f"unknown command {command!r}") |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rebuild dropped-column provenance in the remaining data-column order.
For
Data({"x": 1, "s": 2, "m": 3}, source_info={"m": "M", "s": "S", "x": "X"}).drop("s"), this comprehension retains source entries inm, xorder. The source-table builder preserves that order. However,as_dict,keys, andschemaexpose source columns inx, morder.Build this mapping from
Datagram.keys(new_p), asselectdoes. This keeps the retained source columns consistent across views, as required by ITL-1040.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid, and wider than
drop: it held for anyDatawhose source info was given in another order than its columns. Fixed in 3f107ac (head e7b07bc).Before the fix,
as_tableandarrow_schemalisted the source columns in insertion order, whileas_dict,keysandschemaused column order._ensure_source_info_tablenow builds the source-info table from the data columns in order, keeping only those that have source info, so every view gives the same order after construction,droporselect. The library's callers that pass source info (function outputs indata_function.py, the asyncJoinmerge injoin.py) already pass it in column order, so their stored tables do not change.Regression test:
test_source_info_given_out_of_column_order_follows_the_columns(construct, drop, select). The construct and drop cases failed before and pass now. Full suite on currentmain: 11358 passed, 0 failed.Posted by a Claude Code agent on behalf of @eywalker.