From 3307270fa9d3f6163b742f75f3479195a5ce945e Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 20:52:21 -0800 Subject: [PATCH 1/9] docs(itl-1004): WIP audit plan and measurement scripts Co-Authored-By: Claude Opus 5.5 --- ...6-11-13-itl-1004-referential-tags-audit.md | 57 +++++ .../itl-1004-audit/01_today_same_name.out | 81 +++++++ .../itl-1004-audit/01_today_same_name.py | 106 +++++++++ .../itl-1004-audit/02_prototype_unified.out | 138 ++++++++++++ .../itl-1004-audit/02_prototype_unified.py | 203 ++++++++++++++++++ .../specs/itl-1004-audit/04_identity.out | 34 +++ .../specs/itl-1004-audit/04_identity.py | 127 +++++++++++ superpowers/specs/itl-1004-audit/_common.py | 83 +++++++ 8 files changed, 829 insertions(+) create mode 100644 superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md create mode 100644 superpowers/specs/itl-1004-audit/01_today_same_name.out create mode 100644 superpowers/specs/itl-1004-audit/01_today_same_name.py create mode 100644 superpowers/specs/itl-1004-audit/02_prototype_unified.out create mode 100644 superpowers/specs/itl-1004-audit/02_prototype_unified.py create mode 100644 superpowers/specs/itl-1004-audit/04_identity.out create mode 100644 superpowers/specs/itl-1004-audit/04_identity.py create mode 100644 superpowers/specs/itl-1004-audit/_common.py diff --git a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md new file mode 100644 index 000000000..ebe4733e7 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md @@ -0,0 +1,57 @@ +# ITL-1004 referential tags: implementation-scope audit (plan) + +> **For agentic workers:** this plan produces a design document, not library code. No change under +> `src/` or `tests/`, no golden file. + +**Goal:** One document, `superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md`, +that makes the v0.4 referential-tags specification (Linear document attached to ITL-1004) +dispatchable: where the code assumes tag/data disjointness, which hashes would see a binding, +what is stored, operator-by-operator rules and change sizes, sources, function pods, answers to +the specification's open questions, a sliced delivery plan, and the decisions for the developer. + +**Architecture:** read the code; measure today's behaviour with small scripts under +`superpowers/specs/itl-1004-audit/` (run with `uv run`, outputs saved beside them); write the +document from the evidence. + +**Branch:** `eywalker/itl-1004-referential-tags-v04-a-tag-may-refer-to` from `origin/main` +(`f51ce67c`). PR: docs only, `Refs ITL-1004`. + +--- + +### Task 1: Representation sites +- [ ] Read `core/streams/arrow_table_stream.py`, `core/datagrams/*`, `types.py` (Schema, ColumnConfig), + `utils/schema_utils.py`, `core/streams/base.py`. +- [ ] grep every `keys()` / `output_schema` / `identity_structure` / `pipeline_identity_structure` + definition; classify each site as works / change / reject. +- [ ] Script `01_representation.py`: build an `ArrowTableStream` with a column in both tag and + data lists (if constructible) and record what each public method returns. + +### Task 2: Identity and hashes +- [ ] Read `core/base.py`, `core/sources/base.py`, `core/nodes/function_node.py` + (`_build_record_id_preimage`), `core/function_pod.py`, `core/datagrams` content hash, + `derived_source.py` (PR #336/#338 pattern). +- [ ] Script `02_identity.py`: show which hash inputs contain the tag/data split. + +### Task 3: Storage +- [ ] Read FunctionNode / OperatorNode / CachedSource / DerivedSource write and read paths. +- [ ] Script `03_storage.py`: stored columns of each table kind for a small pipeline. + +### Task 4: Operators +- [ ] For every file in `core/operators` and the FunctionPod call path, measure today's behaviour + with a same-named tag and data column (`04_operators.py`), sync and async. +- [ ] Join source-info of merged column: options. + +### Task 5–6: Sources and function pods +- [ ] Read `arrow_table_source.py`, `stream_builder.py`, `spiraldb_table_source.py`, + `db_table_source.py`, `derived_source.py`; `function_pod.py` output collision (`05_function_pod.py`). + +### Task 7–9: Open questions, delivery plan, decisions +- [ ] Answer section 17 from code; slice plan with sizes, dependencies, tests; decisions list. + +### Task 10: Finish +- [ ] Doc under ~900 lines; ruff on audit scripts; `uv run pytest tests/test_docs` not affected + (superpowers is not a docs page); merge origin/main; push; PR `Refs ITL-1004`; + `orca linear attach`; `orca worktree set --pr`; worker_done. + +## Progress notes +(updated as work proceeds, for a successor) diff --git a/superpowers/specs/itl-1004-audit/01_today_same_name.out b/superpowers/specs/itl-1004-audit/01_today_same_name.out new file mode 100644 index 000000000..892fbf2af --- /dev/null +++ b/superpowers/specs/itl-1004-audit/01_today_same_name.out @@ -0,0 +1,81 @@ +=== R1. A stream cannot name one column as both tag and data +--- ArrowTableSource(x, y; tag_columns=[x]) + keys: tags=('x',) data=('y',) + schema: tags={'x': int | None} data={'y': float | None} + as_table columns: ['x', 'y'] + rows: [({'x': 1}, {'y': 0.1}), ({'x': 2}, {'y': 0.2})] +--- ArrowTableStream over a table with two physical columns named x, tag_columns=[x] + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) + +=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape) +--- Join().output_schema(A, B) + (Schema({'k': int | None, 'x': int | None}), Schema({'x': int | None, 'b': float | None, 'a': float | None})) +--- Join()(A, B) read + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- Join async rows + [({'k': 1, 'x': 1}, {'x': 1, 'b': 0.5, 'a': 0.1}), ({'k': 2, 'x': 1}, {'x': 2, 'b': 0.6, 'a': 0.1}), ({'k': 1, 'x': 2}, {'x': 1, 'b': 0.5, 'a': 0.2}), ({'k': 2, 'x': 2}, {'x': 2, 'b': 0.6, 'a': 0.2})] +--- MergeJoin()(A, B) read + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- SemiJoin()(A, B) read + keys: tags=('x',) data=('a',) + schema: tags={'x': int | None} data={'a': float | None} + as_table columns: ['x', 'a'] + rows: [({'x': 1}, {'a': 0.1}), ({'x': 2}, {'a': 0.2})] + +=== R3. Join: both inputs have DATA x (the collision) +--- Join().output_schema(C, D) + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'v': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- Join async rows + RAISES ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) + (raised at /Users/eywalker/.local/share/uv/python/cpython-3.12.15-macos-aarch64-none/lib/python3.12/asyncio/taskgroups.py:164) +--- MergeJoin()(C, D) read (colliding data -> list) + keys: tags=('u',) data=('x', 'v') + schema: tags={'u': int | None} data={'x': list[int | None], 'v': float | None} + as_table columns: ['u', 'x', 'v'] + rows: [({'u': 5}, {'x': [1, 1], 'v': 1.0}), ({'u': 6}, {'x': [2, 2], 'v': 2.0})] + +=== R4. MapTags / MapData renaming onto a name of the other namespace +--- MapTags({x: y})(E) + RAISES InputValidationError: Renaming {'x': 'y'} would cause collisions with existing columns: overlapping data columns: {'y'}. + (raised at orcapod/core/operators/mappers.py:254) +--- MapData({y: x})(E) + RAISES InputValidationError: Renaming {'y': 'x'} would cause collisions with existing columns: overlapping tag columns: {'x'}. + (raised at orcapod/core/operators/mappers.py:104) + +=== R5. Function pod output named like an input TAG +--- make_x.pod.output_schema(E) + (Schema({'x': int | None}), Schema({'x': float})) +--- make_x.pod(E) read + RAISES ValueError: Duplicate column name 'x' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) + +=== R6. DerivedSource(tags_as_data={x: x}) (same name) +--- DerivedSource(E, tags_as_data={'x': 'x'}) + RAISES InputValidationError: tags_as_data: cannot copy tag 'x' to 'x', which is already a column. Choose a new name. + (raised at orcapod/core/sources/derived_source.py:99) +--- DerivedSource(E, tags_as_data={'x': 'x_copy'}) + keys: tags=('x',) data=('y', 'x_copy') + schema: tags={'x': int | None} data={'y': float | None, 'x_copy': int | None} + as_table columns: ['x', 'y', 'x_copy'] + rows: [({'x': 1}, {'y': 0.1, 'x_copy': 1}), ({'x': 2}, {'y': 0.2, 'x_copy': 2})] + +=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x) +--- GroupBy(by=[t_unit])(G) + keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp') + schema: tags={'t_unit': int | None, 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]} + as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp'] + rows: [({'t_unit': 1, 't_time': [10, 30]}, {'unit': [1, 1], 'time': [10, 30], 'amp': [0.1, 0.3]}), ({'t_unit': 2, 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})] +--- Batch(batch_size=2)(G) + keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp') + schema: tags={'t_unit': list[int | None], 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]} + as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp'] + rows: [({'t_unit': [1, 1], 't_time': [30, 10]}, {'unit': [1, 1], 'time': [30, 10], 'amp': [0.3, 0.1]}), ({'t_unit': [2], 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})] + +=== R8. Function pod with a tag name as a parameter (tags are not inputs) +--- uses_tag.pod(E) (x is a tag of E) + RAISES ValueError: Incoming data schema Schema({'y': float | None}) is not compatible with expected input schema Schema({'x': int | None}): column 'y' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) diff --git a/superpowers/specs/itl-1004-audit/01_today_same_name.py b/superpowers/specs/itl-1004-audit/01_today_same_name.py new file mode 100644 index 000000000..d1baed8f1 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/01_today_same_name.py @@ -0,0 +1,106 @@ +"""ITL-1004 audit, measurement 01: what v0.3 code does today when a tag and a data column +share a name, anywhere it can be built through the public interface. + +Run: uv run python superpowers/specs/itl-1004-audit/01_today_same_name.py +Output: 01_today_same_name.out +""" + +from __future__ import annotations + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src + +from orcapod import function_pod +from orcapod.core.streams import ArrowTableStream +from orcapod.operators import ( + Batch, + GroupBy, + Join, + MapData, + MapTags, + MergeJoin, + SemiJoin, +) +from orcapod.sources import DerivedSource + +print("=== R1. A stream cannot name one column as both tag and data") +one = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "one") +show("ArrowTableSource(x, y; tag_columns=[x])", lambda: describe(one)) + +dup = pa.table([i64(1, 2), i64(1, 2), f64(0.1, 0.2)], names=["x", "x", "y"]) +show( + "ArrowTableStream over a table with two physical columns named x, tag_columns=[x]", + lambda: describe(ArrowTableStream(dup, tag_columns=["x"])), +) + +print("\n=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape)") +a = src({"x": i64(1, 2), "a": f64(0.1, 0.2)}, ["x"], "A") +b = src({"k": i64(1, 2), "x": i64(1, 2), "b": f64(0.5, 0.6)}, ["k"], "B") +show("Join().output_schema(A, B)", lambda: Join().output_schema(a, b)) +show("Join()(A, B) read", lambda: describe(Join()(a, b))) +show("Join async rows", lambda: async_rows(Join(), a, b)) +show("MergeJoin()(A, B) read", lambda: describe(MergeJoin()(a, b))) +show("SemiJoin()(A, B) read", lambda: describe(SemiJoin()(a, b))) + +print("\n=== R3. Join: both inputs have DATA x (the collision)") +c = src({"x": i64(1, 2), "u": i64(5, 6)}, ["u"], "C") +d = src({"x": i64(1, 2), "u": i64(5, 6), "v": f64(1.0, 2.0)}, ["u"], "D") +show("Join().output_schema(C, D)", lambda: Join().output_schema(c, d)) +show("Join async rows", lambda: async_rows(Join(), c, d)) +show( + "MergeJoin()(C, D) read (colliding data -> list)", + lambda: describe(MergeJoin()(c, d)), +) + +print("\n=== R4. MapTags / MapData renaming onto a name of the other namespace") +e = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "E") +show("MapTags({x: y})(E)", lambda: describe(MapTags({"x": "y"})(e))) +show("MapData({y: x})(E)", lambda: describe(MapData({"y": "x"})(e))) + + +print("\n=== R5. Function pod output named like an input TAG") + + +@function_pod(output_keys="x") +def make_x(y: float | None) -> float: + return (y or 0.0) * 2 + + +show("make_x.pod.output_schema(E)", lambda: make_x.pod.output_schema(e)) +show("make_x.pod(E) read", lambda: describe(make_x.pod(e))) + +print("\n=== R6. DerivedSource(tags_as_data={x: x}) (same name)") +show( + "DerivedSource(E, tags_as_data={'x': 'x'})", + lambda: describe(DerivedSource(e, tags_as_data={"x": "x"})), +) +show( + "DerivedSource(E, tags_as_data={'x': 'x_copy'})", + lambda: describe(DerivedSource(e, tags_as_data={"x": "x_copy"})), +) + +print("\n=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x)") +g = src( + { + "t_unit": i64(1, 1, 2), + "t_time": i64(30, 10, 20), + "unit": i64(1, 1, 2), + "time": i64(30, 10, 20), + "amp": f64(0.3, 0.1, 0.2), + }, + ["t_unit", "t_time"], + "G", +) +show("GroupBy(by=[t_unit])(G)", lambda: describe(GroupBy(by=["t_unit"])(g))) +show("Batch(batch_size=2)(G)", lambda: describe(Batch(batch_size=2)(g))) + +print("\n=== R8. Function pod with a tag name as a parameter (tags are not inputs)") + + +@function_pod(output_keys="z") +def uses_tag(x: int | None) -> int | None: + return x + + +show("uses_tag.pod(E) (x is a tag of E)", lambda: describe(uses_tag.pod(e))) +_ = s diff --git a/superpowers/specs/itl-1004-audit/02_prototype_unified.out b/superpowers/specs/itl-1004-audit/02_prototype_unified.out new file mode 100644 index 000000000..6776a4943 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/02_prototype_unified.out @@ -0,0 +1,138 @@ +=== P0. The prototype stream R: tags session, x(ref); data x, y +--- describe(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'x': 20, 'y': 0.2}), ({'session': 's1', 'x': 10}, {'x': 10, 'y': 0.1}), ({'session': 's2', 'x': 30}, {'x': 30, 'y': 0.3})] +--- R.pipeline_hash() + semantic_v1:cc818c22c966b6399a83946db7fd429c71819ad4524a7f74f265b101cfb3746c +--- R.as_table(all_info=True) columns + ['session', 'x', 'y', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_content_hash', '_context_key', '_source_y', '_source_x'] + +=== P1. Row filters (spec: preserve) +--- PolarsFilter(constraints={'x': 20})(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2})] +--- DropNulls(['x'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': } data={'x': , 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2}), ({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] +--- DropNulls()(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': , 'x': } data={'x': , 'y': } + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2}), ({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] +--- SemiJoin()(R, M[tag x materialized]) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] + +=== P2. Select / Drop (spec: removing the target materializes the tag) +--- SelectDataColumns(['y'])(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- DropDataColumns(['x'])(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- DropTagColumns(['x'])(R) (spec: data x kept) + keys: tags=('session',) data=('x', 'y') + schema: tags={'session': str | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'y'] + rows: [({'session': 's1'}, {'y': 0.2}), ({'session': 's1'}, {'y': 0.1}), ({'session': 's2'}, {'y': 0.3})] +--- SelectTagColumns(['session'])(R) + keys: tags=('session',) data=('x', 'y') + schema: tags={'session': str | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'y'] + rows: [({'session': 's1'}, {'y': 0.2}), ({'session': 's1'}, {'y': 0.1}), ({'session': 's2'}, {'y': 0.3})] + +=== P3. Renames +--- MapTags({'x': 'x2'})(R) + RAISES ValueError: Specified tag columns {'x2'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- MapData({'x': 'x2'})(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- MapTags({'session': 'sess'})(R) + keys: tags=('sess', 'x') data=('x', 'y') + schema: tags={'sess': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['sess', 'x', 'y'] + rows: [({'sess': 's1', 'x': 20}, {'y': 0.2}), ({'sess': 's1', 'x': 10}, {'y': 0.1}), ({'sess': 's2', 'x': 30}, {'y': 0.3})] + +=== P4. GroupBy (spec: key scalar in both roles, non-key list in both, one order) +--- GroupBy(['x'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': list[str | None], 'x': int | None} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': ['s1'], 'x': 10}, {'y': [0.1]}), ({'session': ['s1'], 'x': 20}, {'y': [0.2]}), ({'session': ['s2'], 'x': 30}, {'y': [0.3]})] +--- GroupBy(['x']).output_schema(R) (prediction) + (Schema({'session': list[str | None], 'x': int | None}), Schema({'x': list[int | None], 'y': list[float | None]})) +--- GroupBy(['session'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': list[int | None]} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': [10, 20]}, {'y': [0.1, 0.2]}), ({'session': 's2', 'x': [30]}, {'y': [0.3]})] +--- GroupBy(['session']).output_schema(R) + (Schema({'session': str | None, 'x': list[int | None]}), Schema({'x': list[int | None], 'y': list[float | None]})) +--- GroupBy(['session']) async rows + RAISES ValueError: Duplicate column name 'x' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) +--- Batch(2)(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': list[str | None], 'x': list[int | None]} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': ['s1', 's1'], 'x': [20, 10]}, {'y': [0.2, 0.1]}), ({'session': ['s2'], 'x': [30]}, {'y': [0.3]})] + +=== P5. Join +--- Join().output_schema(R, M) + (Schema({'x': int | None, 'session': str | None}), Schema({'w': float | None, 'x': int | None, 'y': float | None})) +--- Join()(R, M) [ref x + materialized x] + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- Join async (R, M) + [({'x': 10, 'session': 's1'}, {'w': 1.0, 'x': 10, 'y': 0.1}), ({'x': 30, 'session': 's2'}, {'w': 3.0, 'x': 30, 'y': 0.3})] +--- Join().output_schema(R, R2) [ref + ref] + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'z': float | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- Join()(R, R2) + RAISES InputValidationError: Input streams are not compatible: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'z': float | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:46) +--- Join().output_schema(R, N[data x, no tag x]) + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- MergeJoin()(R, R2) + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) + +=== P6. Function pods +--- uses_x.pod(R) + keys: tags=('session', 'x') data=('z',) + schema: tags={'session': str | None, 'x': int | None} data={'z': } + as_table columns: ['session', 'x', 'z'] + rows: [({'session': 's1', 'x': 20}, {'z': 20.2}), ({'session': 's1', 'x': 10}, {'z': 10.1}), ({'session': 's2', 'x': 30}, {'z': 30.3})] +--- uses_y.pod(R) (automatic select drops data x) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- uses_y.pod.process(R) (strict: extra data x) + RAISES ValueError: Incoming data schema Schema({'x': int | None, 'y': float | None}) is not compatible with expected input schema Schema({'y': float | None}): column 'x' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) +--- writes_x.pod.output_schema(R) + RAISES ValueError: Incoming data schema Schema({'x': int | None, 'y': float | None}) is not compatible with expected input schema Schema({'y': float | None}): column 'x' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) +--- writes_x.pod(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- data hash of the row passed to uses_y: same with and without the reference? + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) + +=== P7. Index over a grouped referential list +--- Index('x', 0)(GroupBy(session)(R)) + RAISES KeyError: 'x' + (raised at orcapod/core/datagrams/datagram.py:334) +--- Index('x', 0, out='x0')(...) + RAISES KeyError: 'x' + (raised at orcapod/core/datagrams/datagram.py:334) diff --git a/superpowers/specs/itl-1004-audit/02_prototype_unified.py b/superpowers/specs/itl-1004-audit/02_prototype_unified.py new file mode 100644 index 000000000..e5d73e4b1 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/02_prototype_unified.py @@ -0,0 +1,203 @@ +"""ITL-1004 audit, measurement 02: run every operator over a PROTOTYPE referential stream. + +The prototype is today's ``ArrowTableStream`` with one physical column ``x`` listed both as +a tag and as a data column (the "name in both lists" representation). It is built by +setting the stream's private column lists after construction: this is a measurement of +what today's operator code does with such a stream, not a proposed implementation. Nothing +under ``src/`` is changed. + +Run: uv run python superpowers/specs/itl-1004-audit/02_prototype_unified.py +Output: 02_prototype_unified.out +""" + +from __future__ import annotations + +from typing import Any + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src + +from orcapod import function_pod +from orcapod.operators import ( + Batch, + DropDataColumns, + DropNulls, + DropTagColumns, + GroupBy, + Index, + Join, + MapData, + MapTags, + MergeJoin, + PolarsFilter, + SelectDataColumns, + SelectTagColumns, + SemiJoin, +) +from orcapod.system_constants import constants + + +def make_ref(source: Any, refs: list[str]) -> Any: + """Turn tags ``refs`` of ``source`` into referential tags (prototype, private attrs). + + The physical column stays single. It is added to the data column list and given a + source-info column, as a data column would have. + """ + st = source._stream # the source's inner ArrowTableStream + names = st._table.column_names + st._data_columns = tuple(c for c in names if c in refs or c in st._data_columns) + st._data_schema = pa.schema( + f for f in st._table.schema if f.name in st._data_columns + ) + for r in refs: + col = f"{constants.SOURCE_PREFIX}{r}" + if col not in st._source_info_table.column_names: + n = st._table.num_rows + st._source_info_table = st._source_info_table.append_column( + pa.field(col, pa.large_string(), nullable=False), + pa.array( + [f"{source.source_id}::row_{i}::{r}" for i in range(n)], + pa.large_string(), + ), + ) + return source + + +def ref( + columns: dict[str, pa.Array], tags: list[str], refs: list[str], sid: str +) -> Any: + return make_ref(src(columns, tags, sid), refs) + + +R = ref( + { + "session": s("s1", "s1", "s2"), + "x": i64(20, 10, 30), + "y": f64(0.2, 0.1, 0.3), + }, + ["session", "x"], + ["x"], + "R", +) +print("=== P0. The prototype stream R: tags session, x(ref); data x, y") +show("describe(R)", lambda: describe(R)) +show("R.pipeline_hash()", lambda: R.pipeline_hash().to_string()) +show( + "R.as_table(all_info=True) columns", lambda: R.as_table(all_info=True).column_names +) + +print("\n=== P1. Row filters (spec: preserve)") +show( + "PolarsFilter(constraints={'x': 20})(R)", + lambda: describe(PolarsFilter(constraints={"x": 20})(R)), +) +show("DropNulls(['x'])(R)", lambda: describe(DropNulls(["x"])(R))) +show("DropNulls()(R)", lambda: describe(DropNulls()(R))) +M = src({"x": i64(10, 30), "w": f64(1.0, 3.0)}, ["x"], "M") +show("SemiJoin()(R, M[tag x materialized])", lambda: describe(SemiJoin()(R, M))) + +print("\n=== P2. Select / Drop (spec: removing the target materializes the tag)") +show("SelectDataColumns(['y'])(R)", lambda: describe(SelectDataColumns(["y"])(R))) +show("DropDataColumns(['x'])(R)", lambda: describe(DropDataColumns(["x"])(R))) +show( + "DropTagColumns(['x'])(R) (spec: data x kept)", + lambda: describe(DropTagColumns(["x"])(R)), +) +show( + "SelectTagColumns(['session'])(R)", + lambda: describe(SelectTagColumns(["session"])(R)), +) + +print("\n=== P3. Renames") +show("MapTags({'x': 'x2'})(R)", lambda: describe(MapTags({"x": "x2"})(R))) +show("MapData({'x': 'x2'})(R)", lambda: describe(MapData({"x": "x2"})(R))) +show( + "MapTags({'session': 'sess'})(R)", lambda: describe(MapTags({"session": "sess"})(R)) +) + +print( + "\n=== P4. GroupBy (spec: key scalar in both roles, non-key list in both, one order)" +) +show("GroupBy(['x'])(R)", lambda: describe(GroupBy(["x"])(R))) +show( + "GroupBy(['x']).output_schema(R) (prediction)", + lambda: GroupBy(["x"]).output_schema(R), +) +show("GroupBy(['session'])(R)", lambda: describe(GroupBy(["session"])(R))) +show( + "GroupBy(['session']).output_schema(R)", + lambda: GroupBy(["session"]).output_schema(R), +) +show("GroupBy(['session']) async rows", lambda: async_rows(GroupBy(["session"]), R)) +show("Batch(2)(R)", lambda: describe(Batch(batch_size=2)(R))) + +print("\n=== P5. Join") +show("Join().output_schema(R, M)", lambda: Join().output_schema(R, M)) +show("Join()(R, M) [ref x + materialized x]", lambda: describe(Join()(R, M))) +show("Join async (R, M)", lambda: async_rows(Join(), R, M)) +R2 = ref( + {"x": i64(10, 20), "z": f64(5.0, 6.0)}, + ["x"], + ["x"], + "R2", +) +show("Join().output_schema(R, R2) [ref + ref]", lambda: Join().output_schema(R, R2)) +show("Join()(R, R2)", lambda: describe(Join()(R, R2))) +N = src({"k": i64(1, 2), "x": i64(10, 20)}, ["k"], "N") +show("Join().output_schema(R, N[data x, no tag x])", lambda: Join().output_schema(R, N)) +show("MergeJoin()(R, R2)", lambda: describe(MergeJoin()(R, R2))) + +print("\n=== P6. Function pods") + + +@function_pod(output_keys="z") +def uses_x(x: int | None, y: float | None) -> float: + return float(x or 0) + (y or 0.0) + + +@function_pod(output_keys="z") +def uses_y(y: float | None) -> float: + return (y or 0.0) * 2 + + +@function_pod(output_keys="x") +def writes_x(y: float | None) -> float: + return (y or 0.0) * 2 + + +show("uses_x.pod(R)", lambda: describe(uses_x.pod(R))) +show("uses_y.pod(R) (automatic select drops data x)", lambda: describe(uses_y.pod(R))) +show( + "uses_y.pod.process(R) (strict: extra data x)", + lambda: describe(uses_y.pod.process(R)), +) +show("writes_x.pod.output_schema(R)", lambda: writes_x.pod.output_schema(R)) +show("writes_x.pod(R)", lambda: describe(writes_x.pod(R))) +show( + "data hash of the row passed to uses_y: same with and without the reference?", + lambda: ( + [ + d.content_hash().to_string() + for _, d in SelectDataColumns(["y"])(R).iter_data() + ] + == [ + d.content_hash().to_string() + for _, d in SelectDataColumns(["y"])( + src( + { + "session": s("s1", "s1", "s2"), + "x": i64(20, 10, 30), + "y": f64(0.2, 0.1, 0.3), + }, + ["session", "x"], + "R", + ) + ).iter_data() + ] + ), +) + +print("\n=== P7. Index over a grouped referential list") +GR = GroupBy(["session"])(R) +show("Index('x', 0)(GroupBy(session)(R))", lambda: describe(Index("x", 0)(GR))) +show("Index('x', 0, out='x0')(...)", lambda: describe(Index("x", 0, out="x0")(GR))) diff --git a/superpowers/specs/itl-1004-audit/04_identity.out b/superpowers/specs/itl-1004-audit/04_identity.out new file mode 100644 index 000000000..ddaae0c42 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/04_identity.out @@ -0,0 +1,34 @@ +=== I1. pipeline_hash of a root = hash((tag_schema, data_schema)) +--- x tag only: output_schema / pipeline_hash + (Schema({'session': str | None, 'x': int | None}), Schema({'y': float | None})) + b9e64f95eca6847a +--- x data only: output_schema / pipeline_hash + (Schema({'session': str | None}), Schema({'x': int | None, 'y': float | None})) + 95a52202a35fa95b +--- x referential: output_schema / pipeline_hash + (Schema({'session': str | None, 'x': int | None}), Schema({'x': int | None, 'y': float | None})) + cc818c22c966b639 + +=== I2. Schema-pair hash: the three shapes are distinct, no extra identity term +--- hash_object((tag_schema, data_schema)) + tag{x} data{y}: e3b13da3a40743d1 + tag{} data{x,y}: 8ccaa63afa168bd3 + tag{x} data{x,y}: 3d4503d18110c78b + +=== I3. Root content_hash = (class, output_schema, source_id): sees the binding +--- content hashes (same source_id 'S') + tag only: 006efb368b7aec35 + data only: ca14dfaa13bab7e2 + referential: c53cd5c9cfed363d + +=== I4. Function-result identity: the Data hash a function would receive +--- per-row Data content hash, x untagged vs x referential (data columns x, y) + True +--- per-row Tag content hash, x materialized vs x referential (tag columns session, x) + True + +=== I5. Record-id preimage input = tag.as_table(system_tags) + input data hash +--- tag table of row 0, x materialized vs x referential: same columns? + (['session', 'x', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4'], ['session', 'x', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4']) +--- system tag column names (embed the source schema hash, computed at build) + (('_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4'), ('_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4')) diff --git a/superpowers/specs/itl-1004-audit/04_identity.py b/superpowers/specs/itl-1004-audit/04_identity.py new file mode 100644 index 000000000..727bd710e --- /dev/null +++ b/superpowers/specs/itl-1004-audit/04_identity.py @@ -0,0 +1,127 @@ +"""ITL-1004 audit, measurement 04: which identities would see a referential binding. + +Uses the prototype of 02 (one physical column ``x`` listed as tag and data, set through +private attributes after construction). Nothing under ``src/`` changes. + +Run: uv run python superpowers/specs/itl-1004-audit/04_identity.py +Output: 04_identity.out +""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path + +from _common import f64, i64, s, show, src + +from orcapod.contexts import resolve_context +from orcapod.types import Schema + +_spec = importlib.util.spec_from_file_location( + "proto", Path(__file__).with_name("02_prototype_unified.py") +) +make_ref = None # loaded lazily below so 02's prints do not run here + + +def _make_ref(): + import io + import contextlib + + assert _spec is not None and _spec.loader is not None + mod = importlib.util.module_from_spec(_spec) + with contextlib.redirect_stdout(io.StringIO()): + _spec.loader.exec_module(mod) + return mod.make_ref + + +make_ref = _make_ref() +hasher = resolve_context(None).semantic_hasher + + +def cols(): + return {"session": s("s1", "s2"), "x": i64(10, 20), "y": f64(0.1, 0.2)} + + +TAGGED = src(cols(), ["session", "x"], "S") # v0.3: x materialized tag only +UNTAGGED = src(cols(), ["session"], "S") # v0.3: x data only +REF = make_ref(src(cols(), ["session", "x"], "S"), ["x"]) # prototype: x tag and data + +print("=== I1. pipeline_hash of a root = hash((tag_schema, data_schema))") +for name, st in [ + ("x tag only", TAGGED), + ("x data only", UNTAGGED), + ("x referential", REF), +]: + show( + f"{name}: output_schema / pipeline_hash", + lambda st=st: f"{st.output_schema()}\n{st.pipeline_hash().to_hex(16)}", + ) + +print( + "\n=== I2. Schema-pair hash: the three shapes are distinct, no extra identity term" +) +pairs = { + "tag{x} data{y}": (Schema({"session": str, "x": int}), Schema({"y": float})), + "tag{} data{x,y}": (Schema({"session": str}), Schema({"x": int, "y": float})), + "tag{x} data{x,y}": ( + Schema({"session": str, "x": int}), + Schema({"x": int, "y": float}), + ), +} +show( + "hash_object((tag_schema, data_schema))", + lambda: "\n".join( + f"{k}: {hasher.hash_object(v).to_hex(16)}" for k, v in pairs.items() + ), +) + +print( + "\n=== I3. Root content_hash = (class, output_schema, source_id): sees the binding" +) +show( + "content hashes (same source_id 'S')", + lambda: "\n".join( + f"{n}: {st.content_hash().to_hex(16)}" + for n, st in [ + ("tag only", TAGGED), + ("data only", UNTAGGED), + ("referential", REF), + ] + ), +) + +print("\n=== I4. Function-result identity: the Data hash a function would receive") +show( + "per-row Data content hash, x untagged vs x referential (data columns x, y)", + lambda: ( + [d.content_hash().to_hex(16) for _, d in UNTAGGED.iter_data()] + == [d.content_hash().to_hex(16) for _, d in REF.iter_data()] + ), +) +show( + "per-row Tag content hash, x materialized vs x referential (tag columns session, x)", + lambda: ( + [t.content_hash().to_hex(16) for t, _ in TAGGED.iter_data()] + == [t.content_hash().to_hex(16) for t, _ in REF.iter_data()] + ), +) + +print( + "\n=== I5. Record-id preimage input = tag.as_table(system_tags) + input data hash" +) +show( + "tag table of row 0, x materialized vs x referential: same columns?", + lambda: ( + next(iter(TAGGED.iter_data()))[0] + .as_table(columns={"system_tags": True}) + .column_names, + next(iter(REF.iter_data()))[0] + .as_table(columns={"system_tags": True}) + .column_names, + ), +) +show( + "system tag column names (embed the source schema hash, computed at build)", + lambda: (TAGGED.keys(all_info=True)[0][2:], REF.keys(all_info=True)[0][2:]), +) +_ = i64 diff --git a/superpowers/specs/itl-1004-audit/_common.py b/superpowers/specs/itl-1004-audit/_common.py new file mode 100644 index 000000000..0d307873d --- /dev/null +++ b/superpowers/specs/itl-1004-audit/_common.py @@ -0,0 +1,83 @@ +"""Shared helpers for the ITL-1004 audit scripts (measurements only, no library change).""" + +from __future__ import annotations + +import asyncio +import traceback +from collections.abc import Callable +from typing import Any + +import pyarrow as pa + +from orcapod.sources import ArrowTableSource + + +def src(columns: dict[str, pa.Array], tags: list[str], sid: str) -> ArrowTableSource: + """Build an ArrowTableSource with explicit Arrow types and a fixed source id.""" + return ArrowTableSource(pa.table(columns), tag_columns=tags, source_id=sid) + + +def i64(*v: Any) -> pa.Array: + return pa.array(list(v), type=pa.int64()) + + +def f64(*v: Any) -> pa.Array: + return pa.array(list(v), type=pa.float64()) + + +def s(*v: Any) -> pa.Array: + return pa.array(list(v), type=pa.large_string()) + + +def show(title: str, fn: Callable[[], Any]) -> Any: + """Run ``fn`` and print its result or the exception type and message.""" + print(f"--- {title}") + try: + out = fn() + except Exception as exc: # noqa: BLE001 - a measurement records any failure + last = traceback.extract_tb(exc.__traceback__)[-1] + print(f" RAISES {type(exc).__name__}: {str(exc)[:300]}") + print(f" (raised at {last.filename.split('src/')[-1]}:{last.lineno})") + return None + if out is not None: + for line in str(out).splitlines(): + print(f" {line}") + return out + + +def describe(stream: Any) -> str: + """Keys, schema and rows of a stream, read through its public methods.""" + tk, dk = stream.keys() + ts, ds = stream.output_schema() + rows = [(t.as_dict(), d.as_dict()) for t, d in stream.iter_data()] + table = stream.as_table() + return ( + f"keys: tags={tk} data={dk}\n" + f"schema: tags={dict(ts)} data={dict(ds)}\n" + f"as_table columns: {table.column_names}\n" + f"rows: {rows}" + ) + + +def async_rows(op: Any, *streams: Any) -> list[tuple[dict, dict]]: + """Run an operator's ``async_execute`` over channels and collect the rows. + + Same harness as ``tests/test_core/operators/test_async_sync_equivalence.py``. + """ + from orcapod.channels import Channel + + async def run() -> list[tuple[dict, dict]]: + chans = [Channel(buffer_size=64) for _ in streams] + out = Channel(buffer_size=64) + for st, ch in zip(streams, chans): + for row in st.iter_data(): + await ch.writer.send(row) + await ch.writer.close() + await op.async_execute( + [c.reader for c in chans], + out.writer, + input_pipeline_hashes=[st.pipeline_hash() for st in streams], + ) + return [(t.as_dict(), d.as_dict()) for t, d in await out.reader.collect()] + + return asyncio.run(run()) From eac0744f58c233fd7062f9b07c34f67ea0f4c642 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:05:02 -0800 Subject: [PATCH 2/9] docs(itl-1004): referential tags implementation scope audit Design audit of the v0.4 referential-tags specification against the code: representation (one physical column in both role lists), identity and hash verdict (plain-tag mode byte-identical to v0.3, no data context version), storage, operator-by-operator rules, sources with the referential default, function pods, open questions, delivery plan mapped onto ITL-1009..1023, and decisions. Measurements under superpowers/specs/itl-1004-audit/. Refs ITL-1004 Co-Authored-By: Claude Opus 5.5 --- ...6-11-13-itl-1004-referential-tags-audit.md | 6 + ...4-referential-tags-implementation-scope.md | 621 ++++++++++++++++++ .../itl-1004-audit/01_today_same_name.py | 2 +- .../itl-1004-audit/02_prototype_unified.py | 6 +- .../specs/itl-1004-audit/04_identity.py | 2 +- .../05_migration_and_collisions.out | 16 + .../05_migration_and_collisions.py | 43 ++ .../itl-1004-audit/06_default_referential.out | 36 + .../itl-1004-audit/06_default_referential.py | 132 ++++ superpowers/specs/itl-1004-audit/_common.py | 16 +- 10 files changed, 867 insertions(+), 13 deletions(-) create mode 100644 superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md create mode 100644 superpowers/specs/itl-1004-audit/05_migration_and_collisions.out create mode 100644 superpowers/specs/itl-1004-audit/05_migration_and_collisions.py create mode 100644 superpowers/specs/itl-1004-audit/06_default_referential.out create mode 100644 superpowers/specs/itl-1004-audit/06_default_referential.py diff --git a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md index ebe4733e7..9cbffb0eb 100644 --- a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md +++ b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md @@ -55,3 +55,9 @@ document from the evidence. ## Progress notes (updated as work proceeds, for a successor) +- 2026-11-13: Tasks 1-9 done. Measurements 01-06 in superpowers/specs/itl-1004-audit/ (outputs .out). + Coordinator redirects applied (default referential sources; plain-tag mode = v0.3 exactly; + three Join cases equal; decisions only for Open/code-impossible points; slices mapped onto + ITL-1009..1023). Representation + hash verdict sent to the coordinator (msg_a8cdbe370954). + Pod-boundary rule (D1) asked; coordinator: keep (a) recommended, put to developer. + Full suite: 11029 passed, 117 skipped, 23 xfailed. Remaining: PR, linear attach, worker_done. diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md new file mode 100644 index 000000000..a1bf98b62 --- /dev/null +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -0,0 +1,621 @@ +# ITL-1004 — Referential tags (v0.4): implementation scope + +**Status:** design audit for the developer's review. No library code changes. +**Issue:** ITL-1004. **Specification:** the Linear document "Orcapod v0.4 — Referential Tags: +design specification and implementation guidance" attached to the issue ("the spec"; §9.3 +refers to its section numbers). The spec is the authority. This document scopes how the code +implements what the spec marks Required or Proposed, and raises decisions only for points the +spec marks Open or that the code makes impossible or ambiguous. +**Developer decisions included** (November 12, 2026): +- Every source's declared tags are referential **by default**. +- An explicitly chosen plain (materialized) tag mode reproduces v0.3 exactly. This is proven + by the unchanged golden values, so the referential path is additive. +- The change is a change of default behaviour. +- `Join`'s three cases (§9.1) have equal standing. + +**Code audited:** `origin/main` at `f51ce67c`. +**Measurements:** `superpowers/specs/itl-1004-audit/` holds each script (run with `uv run`) and +its output (`.out`). Every example below comes from those outputs. + +| Script | What it measures | +|---|---| +| `01_today_same_name.py` | What v0.3 does when a tag and a data column share a name | +| `02_prototype_unified.py` | Every operator over a **prototype** referential stream: today's stream with one physical column put in both role lists through private attributes. A probe of today's operator code, not an implementation | +| `03_arrow_table_stream_sites.txt` | The 32 places that construct an `ArrowTableStream` | +| `04_identity.py` | Which hashes see a binding | +| `05_migration_and_collisions.py` | Function-result identity across the move from `tags_as_data`; today's output-name checks | +| `06_default_referential.py` | Consequences of the referential default; today's plain-with-plain `Join` | + +--- + +## 1. Verdicts in brief + +1. **Representation (spec §13.3, open question "how Arrow storage avoids duplicated values").** + A referential tag `x` is **one physical column `x` whose name is in both role lists** of the + stream. The binding is the name intersection `tag_keys ∩ data_keys`, which is the spec's + §4.3 statement as code. + - The current table abstraction already supports this. At stream level Orcapod is the + spec's option B: an `ArrowTableStream` holds one table, tags are a name list, and data is + every other column (`core/streams/arrow_table_stream.py:85`, `:113-115`). At row level it + is option A: separate `Tag` and `Data` datagrams. + - `types.Schema` and the `(tag_schema, data_schema)` tuple do not change. A referential tag + appears in both halves of `output_schema()` when the graph is built. +2. **Hashes.** No data context version is needed, and no golden entry moves. + - A stream with only materialized tags produces byte-identical hash inputs. The binding has + no term of its own, so the pair `(tag_schema, data_schema)` is exactly the v0.3 pair. + - A referential tag changes the pair, so it enters `pipeline_hash`, the root `content_hash` + and the source schema hash only when present. + - A function that does not read the referenced column receives a `Data` whose hash equals + the v0.3 one (measured, §3.3). Its stored results are reused. +3. **The default, for an unchanged v0.3 script.** Moved: source content hash, pipeline hashes, + system-tag column names, function-node pipeline record ids, and pipeline/operator/cached + table paths. Unchanged: source ids, source record ids, row `Data` hashes and results of + functions that read no key, and result table paths. A user reproduces v0.3 exactly by + naming the tags with the explicit plain-tag option (§6.2). +4. **Storage.** No stored-table layout changes. A referenced value is written once in every + table kind. The binding is recovered on read by the same intersection rule from the saved + `{tag, data}` descriptor. +5. **Work.** + - The prototype shows filters, `DropNulls`, `SemiJoin`, `Batch` and non-key `GroupBy` + already produce coherent physical results, ordering included. + - What breaks is generic: every operator rebuilds its output with + `ArrowTableStream(table, tag_columns=...)`, which drops the data role (32 sites), and two + row-to-table helpers `hstack` tag and data (duplicate column). + - The new rules are local: `Join` cases 2 and 3, materialization on select or drop, the + `GroupBy` key, and output collisions. +6. **Prerequisite: ITL-940 (Backlog).** Today `Join`, `MergeJoin`, the function pod and + `Index(out=)` accept a tag `x` beside a data `x` at build time. Under the intersection rule + such shapes would look referential while holding different values. The async `Join` already + emits such rows silently (`01 R2`). +7. **One Required rule the code makes impossible as written** (spec §6.1 against + `core/function_pod.py:747`): a pod's output data is only the function's outputs, so a + reference's target never survives a pod. This is decision D1, sent to the coordinator. + +--- + +## 2. Representation (question 1) + +### 2.1 What the code is today + +- **Stream.** + - `ArrowTableStream` stores one table. Tags are the given names + (`arrow_table_stream.py:85`); a missing tag name raises (`:102-105`); data is "every + column not a tag" (`:113-115`); the schemas are filtered by those sets (`:127-136`). + - `iter_data` selects each row's tag and data columns from the same record batch + (`:352-353`). `RecordBatch.select` is zero-copy, so a name in both selections shares one + buffer. +- **Disjointness is by construction.** A table can hold two columns of one name, but the + stream then fails on its first lookup (`01 R1`: + `KeyError: 'Field "x" exists 2 times in schema'`). The source builder splits the same way + (`data_schema = non_sys.drop(tag_columns)`, `core/sources/stream_builder.py:219-220`) and + gives source info only to non-tags (`:336-338`). +- **Rows.** `Tag` and `Data` are independent datagrams with their own tables + (`core/datagrams/tag_data.py:53`, `:274`). A row can already hold `x` in both. +- **Schema.** A plain mapping (`types.py:84`). The `(tag, data)` pair is the only record of a + column's role. No Arrow field metadata marks a role, and the hasher strips non-extension + metadata (`hashing/schema_cleaner.py:141-162`). + +### 2.2 Every site that assumes disjointness + +Legend: +- **OK**: works unchanged. +- **Change**: needs references passed or deduplicated. +- **Reject**: must refuse a shape. + +The prototype (`02`) verified the rows marked †. + +| Site (file:line) | Assumption | Verdict | +|---|---|---| +| `ArrowTableStream.__init__` (`arrow_table_stream.py:113-115`) | data = non-tags | **Change**: new `reference_tags` argument. Data = non-tags ∪ references; validate references ⊆ tags and present. The "at least one data column" rule (`:120-123`) then also counts references, so an all-tags source with referential tags is valid (`06 H3` shows v0.3 raises for a plain one, and still will) | +| `keys` / `output_schema` (`:182-236`) | — | OK†: `x` in both (`02 P0`) | +| `iter_data` (`:319-353`) | — | OK†: zero-copy selects | +| `as_table` / `_assemble_table` (`:256-310`) | — | OK†: one column, emitted once | +| `identity_structure` without producer (`:146-163`) | `self._tag_columns` alone | **Change**: add `("reference_tags", refs)` only when non-empty (§3) | +| `SourceStreamBuilder.build` (`stream_builder.py:143-295`, `:336-338`, `:346`) | data = table minus tags | **Change**: explicit `reference_tags`; schema hash over the pair with `x` in both; source info for referenced columns | +| `StaticOutputOperatorPod._rows_to_table` (`core/operators/static_output_pod.py:208-252`) | `hstack(tags, data)` | **Change**: drop a data column already present as a tag; return the reference set. `02 P4`: the async `GroupBy` raises `Duplicate column name 'x'` (`utils/arrow_utils.py:678-682`) | +| `_materialize_to_stream` / `_materialize_input_stream` (`static_output_pod.py:256-300`); `pipeline/sync_orchestrator.py:216`, `:236` | `tag_columns` only | **Change**: pass references (the empty branch takes them from `upstream_node.output_schema`) | +| 15 operator rebuilds (`03_arrow_table_stream_sites.txt`) | `ArrowTableStream(table, tag_columns=…)` | **Change**†: today the data role is lost silently. `02 P1`: `PolarsFilter` keys say data `('x','y')`, but rows carry `{'y': 0.2}` | +| `FunctionPodStream._table_from_pairs` (`core/function_pod.py:1111-1152`) | `hstack(tags, data)` | OK, given the output-collision rule (§7) | +| `Join._joined_user_schemas` (`core/operators/join.py:78-119`), `_merge_data` (`:759-790`) | shared data name = error | **Change** (§5.1) | +| `MergeJoin` (`merge_join.py:161-175`, `:325-420`) | shared data name = list merge | **Change** (§5.2, open) | +| `SemiJoin` (`semijoin.py:76-130`) | matches on overlapping tag **and** data names | OK† | +| `GroupBy.unary_output_schema` (`group_by.py:302`) | every data column becomes a list | **Change**: a referenced key stays scalar | +| `column_selection.py` (classes at `:37`, `:158`, `:286`, `:413`; rebuilds at `:76`, `:202`, `:328`, `:459`; async at `:113`, `:241`, `:364`, `:494`) | dropping a name drops the column | **Change**: drop a role (§5.3). The async row paths are already right | +| `MapTags`/`MapData` (`mappers.py:83`, `:104`, `:226`, `:254`) | a rename moves the column | **Change** (§5.4) | +| `Index`/`Pick` (`index.py:365`, `pick.py:379`) | data replaced in place | **Reject** in-place on a referenced column (§5.5) | +| `FunctionPod.output_schema` (`function_pod.py:733-747`) | output data = function outputs | **Change**: reject output names equal to a tag; D1 | +| `FunctionNode` read (`core/nodes/function_node.py:2820-2835`, `:2891`) | tags = pipeline-table columns | OK under D1(a) | +| `OperatorNode` read (`operator_node.py:1116`, `:1228`, `:1317`, `:1581`; loaded stub `:235-238`) | `tag_keys = self.keys()[0]` | **Change**: pass references (from `keys()` or descriptor) | +| `CachedSource` (`cached_source.py:360-363`), `DerivedSource` plain (`derived_source.py:462`, `:490`, `:543`), `PollingSource` (`polling_source.py:288-301`, `:865`), `SpiralDBTableSource` (`spiraldb_table_source.py:623`) | tags vs rest | **Change** (§6) | +| `SourceProxy.keys` (`source_proxy.py:177-185`); node descriptor (`pipeline/graph.py:178-199`) | — | OK: two name→type dicts; a name may be in both | +| `schema_utils.union_schemas` / `intersection_schemas` | — | OK: equal types merge (`02 P1`, `SemiJoin`) | + +### 2.3 How a binding is declared (ITL-1009) + +**The rule, in one place:** `reference_names(tag_schema, data_schema) = tag ∩ data` (sorted). +For each name the two Python types must be equal. Because there is one physical column, the +Arrow type, nullability, nesting and values are identical by construction, so §4.4 and §4.5 +hold without a check on values. + +- **Schema model.** No change to `Schema` or the tuple. Add: + - `schema_utils.reference_names(...)`; + - `StreamBase.reference_tags()`; + - one validator for the namespace rules, run on every predicted output schema from the base + wrappers (`StaticOutputOperatorPod`, `_FunctionPodBase`), per spec §13.4. +- **Why inference is sound.** §4.3: "the presence of a tag and data column with the same name + must be sufficient to identify the tag as referential". The only unsound inputs are shapes + made by accident: tag `x` from one input with data `x` from another (ITL-940), or an output + named like a tag. The validator rejects both when the graph is built. +- **Rows.** A row-level consumer, meaning the async paths, applies the same intersection to a + row's tag and data keys. It agrees with the sync path without new channel metadata. +- **Cost.** 32 constructor sites pass references, and 2 helpers deduplicate. + +**Comparison with the spec's options (§13.3).** +- **Option A** (separate tag/data storage plus binding metadata) does not match the code: + Orcapod has no separate tag storage at stream, operator-cache or cached-source level. +- **Option B** (one table, role metadata) is what the code already is. The "role metadata" + is the two name lists. + +What remains open is whether to add an *explicit* marker on `Schema` (decision D3). + +--- + +## 3. Identity and hashes (question 2) + +### 3.1 Which identities see a binding + +| Identity | Hash input (file:line) | Sees a binding? | Explicit plain-tag source vs v0.3 | +|---|---|---|---| +| Root `pipeline_hash` | `(tag_schema, data_schema)` (`core/sources/base.py:132-139`) | **Yes**, through the pair. `04 I1`: tag-only `b9e6…`, data-only `95a5…`, referential `cc81…` | identical | +| Root `content_hash` | `(class, output_schema(), source_id)` (`base.py:88-90`) | **Yes** (`04 I3`) | identical | +| Source schema hash (names system-tag columns) | `compute_source_schema_hash(tag, data)` (`stream_builder.py:227`) | **Yes**. `06 H1`: `a7ad…` → `6a4b…` | identical | +| `ArrowTableStream` without producer, `content_hash` | `(class, table, tag_columns)` (`arrow_table_stream.py:146-163`) | **must add** `("reference_tags", refs)` when non-empty, or a referential and a plain stream over one table collide | identical (term absent) | +| Downstream hashes | `(producer, argument_symmetry(upstreams))` (`core/streams/base.py:106-113`; `static_output_pod.py:62-64`) | Yes, transitively. An automatic materialization is a function of (operator identity, input schema), so it is inside the Merkle chain (§12.3: "automatic materialization likewise changes the resulting pipeline structure") | identical | +| Default source id | raw table hash (`stream_builder.py:237`, `:257-265`) | No | identical | +| Source record id | `uuid5(source_id::token)` (`stream_builder.py:97-116`, `:333-335`) | No (no schema input) | identical | +| Function-node record id | `tag.as_table(system_tags)` + input `Data` digest (`function_node.py:115-141`) | Through the system-tag column *names* (schema hash). The table path is scoped by `pipeline_hash` (`function_node.py:475`) | identical | +| Row `Data` hash (memo key, `core/result_cache.py:216-227`) | data table (`core/datagrams/datagram.py:445-454`) | **No** (`04 I4`, `06 H2`) | identical | +| Function-result identity | `DataFunction.uri` (`core/data_function.py:200-212`) | No (`06 H2`: the uri holds name, output-schema hash and version only) | identical | + +### 3.2 Additive identity: how it is achieved + +The pattern of PRs #336 and #338 adds a term to an identity structure only when the option is +given. Here the pattern needs only one explicit term, the `ArrowTableStream` one; everywhere +else the binding enters as the name in both schema halves, absent in the plain mode. Proof +obligations for ITL-1011: +- every existing golden entry is unchanged, with the fixtures switched to the plain-tag option + (§6.2); +- new golden entries pin the referential default. + +A data context version is **not** needed. No algorithm changes, and every new value belongs +to a shape v0.3 cannot build (`01 R1`); the data context key is the global hash version, and +nothing about it moves. + +### 3.3 The default, for an unchanged v0.3 script (spec §12.4, §16.3) + +| | Moves? | Why / evidence | +|---|---|---| +| Source `content_hash`, `pipeline_hash` | moves | schema pair gains the tags as data (`04 I1`, `I3`) | +| System-tag column names | moves | the source schema hash moves (`06 H1`) | +| Source id, source record ids | stays | built from the table hash and row token only | +| Function-node pipeline record ids | moves | the preimage holds system-tag columns, whose names moved | +| Pipeline table path (`/schema:/pdb_v2`), operator cache (`odb_v1`), cached source (`sdb_v1`) | moves | the paths contain the pipeline or content hash | +| Result table path (`/rdb_v2`) | stays | `06 H2`: `('double_y', '872e…', 'v0', 'python.function.v0')` | +| Row `Data` hash for a function that reads no key | stays | the automatic select drops the referenced keys (materializing them), and what remains hashes as in v0.3 (`06 H2`: `True`) | +| A function that reads a key | new | no v0.3 function could read a key under its own name: tags were never data | +| A function that read a `tags_as_data` copy | moves once | the column name is in the hash (`05 M1`) | + +So retagging works as the spec intends (§2.2, §12.2): pipeline records are re-established +under new paths, and results are found again. No migration step is required: the v0.3 +pipeline tables stay where they are, with current layouts, so they are not reported stale and +are not read. The upgrade note names the plain-tag option for anyone who wants the old paths. + +One consequence is not covered by the result cache. `pod.process(source)` and +`pod(source, strict=True)` (ITL-807's exact form) now see every key as an extra data column +and raise, unless the function takes all keys. That is decision D2. + +--- + +## 4. Storage (question 3) + +| Table | Columns today | With referential `x` | Layout change | +|---|---|---|---| +| Function pipeline `pdb_v2` (`function_node.py:2337-2461`) | user and system tags; the input's `_source_*` (`:2389-2396`); meta | `x` once, as a tag. `_source_x` once, if `x` was an input | none | +| Function result `rdb_v2` (`result_cache.py:269-347`) | outputs, their `_source_*`, `__input_data_hash`, `__data_context` | unchanged (`x` is never an output, §7) | none | +| Operator cache `odb_v1` (`operator_node.py:930-1019`) | flat `as_table(source, system_tags, context)`, row hash, `__data_context` | `x` once (`02 P0`), plus `_source_x` | none | +| Cached source `sdb_v1` (`cached_source.py:278-311`) | flat source table | `x` once | none | +| Side-effect log `tdb_v2` | ids only | unchanged | none | + +**Getting the binding back.** +- A live node passes `reference_tags()` where it now passes `keys()[0]` + (`operator_node.py:1114-1117`). +- A loaded job computes `tag ∩ data` from the saved descriptor + (`pipeline/graph.py:178-190`; stub at `operator_node.py:235-238`). `SourceProxy` already + returns `x` in both `keys()`. +- `DerivedSource` and `CachedSource` take the binding from the live origin. + +**Stored-schema changes, named exactly.** +- No stored table changes, and the descriptor format is unchanged. +- Source configs gain the plain-tag option (§6.2), written only when used. `from_config` for a + v0.3 config therefore gives the **new default**, consistent with the change of default. + +**Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as +plain tags only (§16.1). Nothing is reinterpreted. + +--- + +## 5. Operator by operator (question 4) + +Sizes: +- **S**: up to about a day with tests. +- **M**: 2-4 days. +- **L**: a week or more. + +"Rebuild" means passing references to the constructed `ArrowTableStream`; ITL-1010 makes +that change at every site. + +| Operator | Today (measured) | Spec rule | Change | Size | +|---|---|---|---|---| +| `PolarsFilter` (`filters.py:80`) | prototype: data `x` lost from rows (`02 P1`) | preserve (§10) | rebuild | S | +| `DropNulls` (`drop_nulls.py:306`) | prototype: narrows `x` in both roles, so both read `int` | preserve | rebuild (one column, one flag) | S | +| `SemiJoin` (`semijoin.py:130`) | prototype: rows right, data `x` lost | preserve (§10) | rebuild. v0.3 `SemiJoin` already matches overlapping tag and data names (`:76-83`) | S | +| `Batch` (`batch.py:107`) | prototype: tag and data `x` both `list[int]`, in input order | preserve when both are restructured alike (§10.2) | rebuild. Every member column is listed in input order (`:60-100`), and `x` is one column | S | +| `GroupBy` non-key (`group_by.py:258`) | prototype: predicted list in both (correct) | list in both, same order (§8.2-8.3) | rebuild. Members sort on non-key tags (`:192-215`); one column means one order | S | +| `GroupBy` key | prototype: data **predicted `list[int]`** (`02 P4`) | scalar in both (§8.1) | rebuild, plus `unary_output_schema` keeps a referenced key scalar (`:302`). Physically keys are not member columns (`:187`) | S | +| `GroupBy`/`Batch` async | `Duplicate column name 'x'` (`02 P4`) | as sync | `_rows_to_table` (ITL-1010) | — | +| Select/Drop data (`column_selection.py:158`, `:413`) | raise when dropping referenced `x` (`02 P2`) | materialize (§7.2) | §5.3 | S | +| Select/Drop tag (`:37`, `:286`) | the physical column dropped, so data `x` is lost (`02 P2`) | remove tag, keep data (§10) | §5.3 | S | +| `MapTags`/`MapData` | renaming a referenced tag raises (`02 P3`) | §10.1 (proposed) | §5.4 | M | +| `Index`/`Pick` | §5.5 | §4.5 | §5.5 | S | +| `Join` | §5.1 | §9 | §5.1 | M | +| `MergeJoin` | §5.2 | open (§17) | §5.2 | M | +| Function pod + automatic join and select (ITL-807) | §7 | §6 | §7 | M | + +### 5.1 `Join`: the three cases of §9.1, with equal standing + +**Today.** `Join` joins on **all** shared tag names (`common_tag_keys`, `join.py:202`), inner, +through Polars (`:219`), with key types checked (`:206`; async `:648`). It refuses any shared +data name (`:116`; async `:781`). Because every shared tag is a join key, the condition "the +join established equality" (§9.5, point 3) always holds for a shared referential name. + +| Case | Today (measured) | v0.4 (spec §9.2-9.3) | Code change | +|---|---|---|---| +| Plain with plain | `06 J1`: tags `('subject',)`, data `('b','a')`, pipeline hash `d12c05799eccd043`, four name-extended system tags, sync = async | **byte-identical** | none. Pin the `06 J1` values as a regression test; the golden join cases already cover it | +| Plain with referential | prototype `02 P5`: `output_schema` already right (tag `x`, data `x`); sync read raises `KeyError` in the column layout (`table.select(tag_order + data_order)`, `:238-246`); async rows already right | referential tag; data `x` from the referential side; nothing made from the plain tag | Polars keeps one key column `x`, which *is* the referential side's data. Lay out each name once; the output's references are the inputs' references among shared keys | +| Referential with referential | prototype: `InputValidationError: Datas should not have overlapping keys` | referential tag, one merged data `x` | `_joined_user_schemas` and `_merge_data` exempt a name that is a tag on both inputs and referential on at least one; one key column survives | +| Referential tag with data `x` but no tag `x` | — | error (§9.5) | none: not a key, so an ordinary collision | +| Plain tag `x` with data `x` but no tag `x` | `01 R2`: build accepts; sync read `KeyError`; **async yields rows where tag `x` ≠ data `x`** | invalid (§4.1) | reject (ITL-940) | + +**Provenance of the merged column (§9.4: "must retain the relevant source relationships").** +How the code retains them: +1. **Row level.** The name-extended system-tag columns already keep both inputs' `source_id` + and `record_id` per row (`06 J1` shows four columns), and they are the pipeline + provenance. Nothing changes. +2. **Field level.** Each referential input brings `_source_x`; Polars suffixes the right one + `_source_x_right`. `MergeJoin` already handles this suffix (`merge_join.py:408-409`). + Retaining both relationships means keeping **both tokens**. The one code-level choice is + the column's shape (D5): + - (a) a sorted two-element list, as `MergeJoin` does; + - (b) two scalar columns. +3. **Plain with referential.** Only the referential side has `_source_x`, so there is nothing + to merge. + +**Under the referential default.** Two sources sharing tag columns now meet case 3 on every +join, including the automatic join of `pod(a, b)` (ITL-807), which builds the same `Join`. +The automatic select that follows then materializes the keys the function does not read, so +the function's `Data` is the v0.3 one (§3.3). + +### 5.2 `MergeJoin` (spec §17 open, ITL-1018) + +**From the code.** An inner join on shared tags (`merge_join.py:393`); every shared data name +becomes a sorted `large_list` (`:325`, `:404-420`), with source info reordered to match. +- A shared referential name cannot be list-merged: tag `x` would be scalar beside data + `[v, v]`. +- Materializing first is impossible: a plain tag and data of one name is invalid. + +**Options (D4):** +- (a) **Recommended.** An equated referential name stays one scalar column, as in `Join`; a + referential name colliding with a non-tag data name is rejected. +- (b) Reject every referential input until the user materializes. + +There are no unmatched rows: `how="inner"`. + +### 5.3 Select and drop: roles, not columns (spec §7, ITL-1015) + +`SelectDataColumns`/`DropDataColumns` removing a referenced `x` remove the data role. The +physical column stays as a plain tag, with the same buffer, type and values (§7.2, §4.4). The +tag operators remove the tag role and keep data. The async row paths already behave this way, +because they drop per datagram. + +**Diagnostic (§7.4, Proposed).** +- **Config.** A new `OrcapodConfig` section field + `reference_materialization: Literal["warn", "silent"] = "warn"`, beside `hashing`, `display` + and `datetime` (`config.py:103-112`). No hash reads it. +- **Where it fires.** Once per operator application, from one helper in the operator base that + compares input and output reference sets. It emits a `warnings.warn` category the user can + filter. It is recommended silent for the automatic select of `pod(...)` and at a pod + boundary (D1). +- **Inspection (§7.3, §13.5).** The output schema shows the transition. The same helper + exposes `binding_transitions(...)` → `{name: kept | materialized | removed}` for pipeline + views. + +### 5.4 Renames (spec §10.1, Proposed: how) + +`MapTags` and `MapData` rename within their own namespace and refuse a name of the other one +(`01 R4`; `mappers.py:104`, `:254`). Per §10.1: +- **Renaming only the tag (or only the data) of a reference materializes it.** The renamed + side gets a zero-copy duplicate of the column under the new name, using the idiom + `table.append_column(field.with_name(new), table.column(x))` from `derived_source.py:429-434`. + The other side keeps the original. The diagnostic fires. +- **Renaming both together preserves the reference.** No operator does this today. Add one + explicit form, for example `MapTags({...}, with_references=True)` or a `MapReferences` + operator. The name is the developer's call at ITL-1018. + +### 5.5 `Index` and `Pick` + +- **`out=None` on a referenced column.** The operator replaces the data in place, which would + leave tag `x` ≠ data `x`. Materializing would create a plain tag and data of one name (§4.3). + So per §4.5, **reject**, naming the remedy (`out=`, or drop or select first). +- **`out=` equal to a tag name.** Accepted at build today; reading fails with + `Duplicate column name 't'` (`05 M2`). Reject at build (ITL-940). + +--- + +## 6. Sources (question 5) + +### 6.1 The referential default + +A source applies the default; the builder stays explicit. +- `SourceStreamBuilder.build(..., reference_tags=...)` takes the list. It computes the schema + hash from the pair with references in both halves, gives source info to referenced columns + (exclude only `tags − references`, `stream_builder.py:336-338`), and passes the list on. +- Each source computes `reference_tags` = its declared tags minus the ones asked for as plain. + +**Every source:** + +| Source | Declared tags today | v0.4 default | +|---|---|---| +| `ArrowTableSource`, `DictSource`, `CSVSource`, `DataFrameSource`, `DeltaTableSource` | `tag_columns` | referential | +| `DBTableSource`, `SQLiteTableSource`, `PostgreSQLTableSource` | `tag_columns`, else primary keys (`db_table_source.py:98-104`; SQLite `rowid` fallback `sqlite_table_source.py:92-96`) | referential (§11.4: keys are a construction policy) | +| `SpiralDBTableSource` | `tag_columns`, else keys (`spiraldb_table_source.py:307-311`) | referential. Keys are non-null already (`:466-486`); a reference keeps the exact field | +| `PollingSource` | `tag_columns` (`polling_source.py:257-301`) | referential; its own identity `(impl.identity(), tag_columns)` (`:351`) gains the plain list only when used | +| `ListSource` | `element_index` or `tag_function` output (`list_source.py:49-76`): **values computed by the source, not columns** | code-ambiguous, D6 | +| `DerivedSource`, `CachedSource`, `SourceProxy` | the origin's or saved roles | keep the origin's bindings; no default applied afresh | + +Two further consequences: +- **All-tags source.** With referential tags the "one data column" rule is met, so it is valid + (ITL-1012). A plain all-tags source still raises, as in v0.3 (`06 H3`). +- **The automatic join and select (ITL-807)** work as described in §5.1 and §3.3. + +### 6.2 The explicit plain-tag option (API shape, ITL-1012) + +The previous behaviour must be reachable on every source through one option with one name. +Options: + +| | Form | v0.3 equivalent | Mixed plain and referential in one source (§11.3: `lab` plain, `timestamp` referential) | Primary-key default (DB sources) | +|---|---|---|---|---| +| (a) **Recommended** | `tag_columns=[...]` (referential) and `plain_tag_columns=[...]` (materialized); a name in at most one | `ArrowTableSource(t, plain_tag_columns=["k"])` | yes | `plain_tag_columns=None` and `tag_columns=None` → keys referential; v0.3 = list the keys in `plain_tag_columns` | +| (b) | `tag_columns=[...]` and `tag_binding="referential" \| "plain"` (source-wide) | `tag_columns=["k"], tag_binding="plain"` | no: needs a select or drop afterwards | `tag_binding="plain"` alone reproduces v0.3 | +| (c) | `tag_columns={"k": "plain", "t": "ref"}` or a list | a mapping | yes | awkward | + +(a) is recommended because §11.3 needs both kinds on one source, and the name says what the +column is. Name alternatives are `materialized_tag_columns` (the spec's word) and +`label_columns`; "plain" is shorter and is the developer's word. For DB sources, (a) can add +`plain_tag_columns="keys"` so v0.3 keys need not be listed by hand; that is the developer's +call. + +**Golden fixtures (`tests/test_hashing/golden/cases.py`).** +- The 16 `tag_columns=` call sites switch to `plain_tag_columns=`. No fixture moves or is + renamed, and every entry stays byte-identical. +- The two `SourceStreamBuilder` / `DerivedSource` cases (`:584-633`) need no edit: the builder + is explicit, and `DerivedSource` keeps its origin's binding once the origin fixture is + plain. +- `ListSource` (`:1699`) depends on D6. +- New entries pin the referential default for each source kind, `Join` case 3, and `GroupBy` + on a referential key. + +**Upgrade note (changelog fragment of ITL-1012), draft:** "Declared tag columns are now +referential: each is also a data column of the same name, stored once, so a function pod can +read it. This changes the schemas and hashes of pipelines that use `tag_columns`; computed +function results are reused. To keep a tag as a plain label, with the v0.3 schemas and hashes, +pass it as `plain_tag_columns=[...]`." + +### 6.3 `tags_as_data` (spec §11.5, §16: legacy meaning kept, explicit migration) + +**How the code keeps the meaning.** +- `tags_as_data={tag: new_name}` still adds an independent renamed copy + (`derived_source.py:429-434`; Spiral `:375-388`). +- Its identity terms are unchanged (`derived_source.py:274-293`; + `spiraldb_table_source.py:407-421`). +- `validate_tags_as_data` already refuses a name that is a column (`derived_source.py:98-101`, + `01 R6`), so the copy can never collide with a reference. +- Beside the default, a `tags_as_data` copy of a referential tag is simply a second data + column with a different name. + +**Explicit migration (ITL-1020, ITL-1021):** + +| v0.3 | v0.4 | +|---|---| +| `SpiralDBTableSource(..., tags_as_data={"spike_time": "spike_time_us"})` | `SpiralDBTableSource(...)`, with keys referential by default | +| pod parameter `spike_time_us` | pod parameter `spike_time` | + +Functions that read the copy recompute once, because the column name is in the `Data` hash +(`05 M1`). That is spec §16.3: the computational input changed. + +--- + +## 7. Function pods (question 6) + +- **Input binding by name.** It works unchanged (`02 P6`: `uses_x.pod(R)` gives `z = 20.2`, + …). +- **Automatic select (ITL-807, `function_pod.py:669-727`).** For a function that does not + read `x`, the automatic select removes data `x`, and the select's materialization rule + applies (§6.5), as ITL-1013 says. Today it raises on the prototype (`02 P6`); with §5.3 it + materializes. +- **Tags pass through** (`function_pod.py:739-747`). The output data is only the function's + outputs (`:747`), so a reference's target never survives a pod, whether or not it was read. + Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. This is **D1**. +- **An output name that collides with a tag (§6.2, Required).** + - Today `output_schema` accepts it (`01 R5`: + `(Schema({'x': int | None}), Schema({'x': float}))`), and reading fails with + `Duplicate column name 'x'`. Under the intersection rule it would look like a reference + with unequal types. + - Implement the rejection in `_FunctionPodBase.output_schema`, for plain and referential + tags, with the spec's message. A `PythonDataFunction`'s output schema is static, so + §6.4's dynamic case does not arise. +- **Replacing plain data with no tag (§6.3).** Unchanged: a pod's outputs are new columns. + +--- + +## 8. The spec's open questions (§17, §9.7) + +| Question | Answer | +|---|---| +| Outer and unmatched-row joins (§9.7) | **The code decides:** none exist. `Join` and `MergeJoin` use `how="inner"` (`join.py:219`, `merge_join.py:393`), `SemiJoin` uses `how="semi"` (`semijoin.py:120`). Record the §9.7 policy in the operator contract for future ones | +| Rename both namespaces | §5.4: preserve, through an explicit both-sides form (name at ITL-1018) | +| Rename one side materializes? | §5.4: yes, as a zero-copy duplicate; the existing contract renames one namespace (`mappers.py:104`, `:254`) | +| `Batch` | **The code decides:** every member column becomes a list in input order (`batch.py:60-100`). A reference is one column, so it is preserved (`02 P4`) | +| `MergeJoin` | §5.2; D4 | +| Avoiding duplicate stored values | **The code decides:** one physical column, and rows share buffers (`arrow_table_stream.py:352-353`). Option B, which the code already is; D3 asks only whether to add an explicit marker | +| Database columns referential by default | **Developer decided:** every declared tag of every source (§6.1) | +| `tags_as_data` migration | §6.3 | +| Which hashes change | §3: none in plain mode (golden-proven); additive for references; no data context version | +| Operators relying on unique tags per row | None require it. `GroupBy` documents the expectation and falls back to `record_id` (`group_by.py:71-81`, `:192-215`); `Join` is many-to-many. References do not change this | + +--- + +## 9. Delivery plan (question 8), mapped onto the project's issues + +The project has ITL-1009 to ITL-1023. Recommended order, with changes to the issue set: + +| Order | Issue | Scope from this audit | Size | Depends on | Parallel with | +|---|---|---|---|---|---| +| 0 | **ITL-940** (add to the project) | reject a tag/data name collision across inputs (`Join`, `MergeJoin`, sync and async), pod output = tag name, `Index`/`Pick` `out=` = tag name; regression tests `01 R2`, `R5`, `05 M2` | M | — | — | +| 1 | **ITL-1009 + ITL-1010 (merge)** | `reference_names`, the validator, `StreamBase.reference_tags()`; `ArrowTableStream(reference_tags=)`; its identity term; `_rows_to_table` and `_materialize_*` deduplication; all 32 rebuild sites pass references; spec §15.1 tests | M | 0 | — | +| 2a | **ITL-1011 + ITL-1012 (land together)** | builder `reference_tags`; every source's default; `plain_tag_columns` (§6.2); `to_config`; golden fixtures switched, new entries; upgrade note | M | 1 | 2b-2e | +| 2b | **ITL-1015** | select and drop by role; config, diagnostic, `binding_transitions`; §15.5 tests | S | 1 | 2a, 2c-2e | +| 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | +| 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | +| 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | +| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); D1; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | +| 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | +| 5 | **ITL-1018** | `MergeJoin` (D4); rename forms (§5.4) | M | 1 | 6 | +| 6 | **ITL-1020** | `tags_as_data` beside references (§6.3): docs and tests; no behaviour change | S | 2a | 5 | +| 7 | **ITL-1021** | ediop and the notebook, without copies; EDI-shaped integration test (§15.7) | M | 2a, 2c, 2d, 3 | — | +| 8 | **ITL-1022** | operator table in `docs/`; `streams.md:10-25`, `operators.md:31-33`, `sources.md`, `hashing.md`; CLAUDE.md and `.zed/rules` rule | S | all | — | + +**ITL-1023 (workshop slice, go/no-go November 14).** The thinnest path is ITL-1010 (+1009), +1012 (sources only), 1015, 1013, 1016 and 1017, plus guard errors ("not supported with +referential tags yet") in: +- `Batch`, `MergeJoin`, `MapTags`, `MapData`, `Index`, `Pick` when an input has references; +- `OperatorNode` reload and `DerivedSource`; +- `pod.process`/`strict=True` with references (D2), if D2 is not settled. + +Each guard is one check against `reference_tags()` at `validate_inputs`. Estimate: about +5-7 working days for one worker, which is tight for November 14. The audit's measurements +(`02`, `06`) give the test shapes. The riskiest part is the 32-site rebuild; it is mechanical +but touches every operator. + +**Splits and merges.** Merge ITL-1009 into ITL-1010: with the intersection rule ITL-1009 is a +helper and a validator, testable only with a stream. Land ITL-1011 with ITL-1012: the golden +fixture switch is the proof ITL-1011 asks for, and it needs the plain-tag option. ITL-1013 +depends on ITL-1015 (the automatic select). Add ITL-940 to the project as the first issue. + +**Blocked by other issues?** +- **ITL-858 (channels):** no. Rows carry both datagrams, and the intersection rule needs + nothing else. +- **ITL-835 (Arrow types in `Schema`):** no. A reference is one column, so its Arrow types + cannot disagree. +- **ITL-846 (metadata preservation):** no. The binding is not field metadata. +- **ITL-980:** unrelated. +- **ITL-940:** blocks step 1. + +--- + +## 10. Decisions for the developer (most consequential first) + +D0 is stated first, as asked. It is the consequence of the developer's default, for +confirmation. D1 is the first open decision. + +**D0. The hash consequence of the default (measured, §3.3).** An unchanged v0.3 script gets: +- new source content and pipeline hashes, system-tag names, function-node record ids, and + pipeline/operator/cached-source table paths; +- the same source ids, source record ids, result table paths, and results for every function + that reads no key. + +Plain mode is byte-identical, proven by the golden values with fixtures switched. No data +context version. To confirm: +- (a) **Recommended.** Ship as described. No migration tool; v0.3 tables are left in place, + unread. +- (b) Also provide a one-off tool that re-keys v0.3 pipeline records to the new paths. The + results are reused anyway, so this only saves re-establishing records. + +**D1. A reference after a function pod** (Required §6.1 is impossible as written, because a pod +outputs only its function's outputs, `function_pod.py:747`). The coordinator has put it to the +developer. The §6.1 example is conditional ("assuming the output stream retains the input data +`x`"); in Orcapod it never does. +- (a) **Recommended.** The pod's discarding of input data is the structural projection of + §6.5: the tag is materialized at the pod boundary, which the pod's output schema shows + (`binding_transitions`). No storage or result-identity change. + - **The diagnostic at a pod boundary.** With referential sources by default, *every* pod + would trigger it, so recommend **none** there. The same goes for the automatic select of + `pod(...)` (ITL-807), which also fires on every call. The `reference_materialization` + warning is for a select, drop or rename the user wrote. The materialization is still + recorded in the schema and the pipeline hash; §7.4 makes the diagnostic a matter of + configuration, never of result. +- (b) Pass referenced input columns through to the pod's output data (the §6.1 example + literally). This changes the `FunctionPod` output contract and the pipeline hashes of pods + over references; `FunctionNode` rebuilds data `x` from the stored tag on read. +- (c) Reject a pod over references until materialized. + +**D2. `pod.process(...)` and `strict=True` with referenced keys (ITL-807).** Under the default, +every key is an extra data column for a function that does not read it. +- (a) **Recommended.** Keep ITL-807's contract: `process` and `strict` reject. `pod(...)` is + the convenient form and materializes. +- (b) `process` and `strict` ignore referenced columns that are not parameters, materializing + them like the automatic select. This is friendlier, but `process` would no longer be the + exact primitive. + +**D3. An explicit binding marker on `Schema`?** (§17 "avoid duplicated values" and +§13.1-13.2.) +- (a) **Recommended.** No. The binding is the name intersection (§4.3), with the build-time + validator; `Schema` and every schema hash are untouched. +- (b) Add a `reference_fields` marker beside `optional_fields`. It is explicit, but it touches + `Schema` equality and hashing (it must stay out of the hash when empty; + `hashing/semantic_hashing/builtin_handlers.py:290-293` shows how `optional_fields` is left + out) and every operator's schema code. + +**D4. `MergeJoin` with references (§17 open).** +- (a) **Recommended.** An equated reference stays one scalar column; a reference colliding with + a non-tag data name is rejected. +- (b) Reject referential inputs. + +**D5. Field-level shape of the merged column's provenance (§9.4 says retain both; the code +needs a shape).** +- (a) **Recommended.** `_source_x` becomes a sorted two-element `large_list`, + as `MergeJoin` does. Downstream `GroupBy` then nests one level, as it does for `MergeJoin` + output. +- (b) Two scalar columns, `_source_x` and a second name. This needs a new naming convention + for source-info columns. + +**D6. `ListSource` under the default.** Its tags are values the source computes +(`element_index`, or `tag_function`), not columns of the user's data. §11.1 speaks of source +*columns*. +- (a) **Recommended.** `ListSource` tags stay plain. Its golden entry is unchanged without a + fixture edit, and the upgrade note says so. +- (b) Referential like every source, with a plain option added to `ListSource`. + +**D7. The plain-tag option's name and form (§6.2).** +- (a) **Recommended.** `plain_tag_columns=[...]` beside `tag_columns`, the same on every + source, plus optionally `plain_tag_columns="keys"` on database sources. +- (b) The source-wide `tag_binding="plain"`. +- (c) A mapping. + +--- + +## Appendix — out-of-scope findings (reported to the coordinator, not fixed) + +1. **Async `Join` of tag `x` with data `x` from another input** yields rows where the two + differ, silently. Sync raises `KeyError` on read (`01 R2`). This extends ITL-940 to async. +2. **`Index(out=)`** is accepted at build and fails on read (`05 M2`). This extends + ITL-940. diff --git a/superpowers/specs/itl-1004-audit/01_today_same_name.py b/superpowers/specs/itl-1004-audit/01_today_same_name.py index d1baed8f1..61d2d2fa5 100644 --- a/superpowers/specs/itl-1004-audit/01_today_same_name.py +++ b/superpowers/specs/itl-1004-audit/01_today_same_name.py @@ -8,7 +8,7 @@ from __future__ import annotations import pyarrow as pa -from _common import async_rows, describe, f64, i64, s, show, src +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder from orcapod import function_pod from orcapod.core.streams import ArrowTableStream diff --git a/superpowers/specs/itl-1004-audit/02_prototype_unified.py b/superpowers/specs/itl-1004-audit/02_prototype_unified.py index e5d73e4b1..b7b2b2cd8 100644 --- a/superpowers/specs/itl-1004-audit/02_prototype_unified.py +++ b/superpowers/specs/itl-1004-audit/02_prototype_unified.py @@ -15,7 +15,7 @@ from typing import Any import pyarrow as pa -from _common import async_rows, describe, f64, i64, s, show, src +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder from orcapod import function_pod from orcapod.operators import ( @@ -63,9 +63,7 @@ def make_ref(source: Any, refs: list[str]) -> Any: return source -def ref( - columns: dict[str, pa.Array], tags: list[str], refs: list[str], sid: str -) -> Any: +def ref(columns: dict[str, Any], tags: list[str], refs: list[str], sid: str) -> Any: return make_ref(src(columns, tags, sid), refs) diff --git a/superpowers/specs/itl-1004-audit/04_identity.py b/superpowers/specs/itl-1004-audit/04_identity.py index 727bd710e..8b10a3623 100644 --- a/superpowers/specs/itl-1004-audit/04_identity.py +++ b/superpowers/specs/itl-1004-audit/04_identity.py @@ -12,7 +12,7 @@ import importlib.util from pathlib import Path -from _common import f64, i64, s, show, src +from _common import f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder from orcapod.contexts import resolve_context from orcapod.types import Schema diff --git a/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out new file mode 100644 index 000000000..87902721b --- /dev/null +++ b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out @@ -0,0 +1,16 @@ +=== M1. Function-result identity follows the data COLUMN NAME (Arrow hash) +--- Data{spike_time: 10} hash == Data{spike_time_us: 10} hash + False + +=== M2. Index / Pick: out= equal to an existing TAG name (today) +--- grouped stream + keys: tags=('u', 't') data=('v',) + schema: tags={'u': int | None, 't': list[int | None]} data={'v': list[float | None]} + as_table columns: ['u', 't', 'v'] + rows: [({'u': 1, 't': [5, 6]}, {'v': [0.1, 0.2]})] +--- Index('v', 0, out='t')(g) (t is a tag) + RAISES ValueError: Duplicate column name 't' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) +--- Pick('d', 't', out='u')(st) (u is a tag) + RAISES InputValidationError: Pick: column 'd' has type orcapod.semantic_types.universal_converter.Struct_t_int | None which is not a supported pick target (not dict[K,V] and no registered logical type). + (raised at orcapod/core/operators/pick.py:174) diff --git a/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py new file mode 100644 index 000000000..75d1a9e5f --- /dev/null +++ b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py @@ -0,0 +1,43 @@ +"""ITL-1004 audit, measurement 05: migration from tags_as_data, and today's output-name checks. + +Run: uv run python superpowers/specs/itl-1004-audit/05_migration_and_collisions.py +Output: 05_migration_and_collisions.out +""" + +from __future__ import annotations + +import pyarrow as pa +from _common import describe, f64, i64, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod.core.datagrams import Data +from orcapod.operators import GroupBy, Index, Pick + +print("=== M1. Function-result identity follows the data COLUMN NAME (Arrow hash)") +same = Data(pa.table({"spike_time": i64(10)})) +renamed = Data(pa.table({"spike_time_us": i64(10)})) +show( + "Data{spike_time: 10} hash == Data{spike_time_us: 10} hash", + lambda: same.content_hash() == renamed.content_hash(), +) + +print("\n=== M2. Index / Pick: out= equal to an existing TAG name (today)") +g = GroupBy(["u"])( + src({"u": i64(1, 1), "t": i64(5, 6), "v": f64(0.1, 0.2)}, ["u", "t"], "G") +) +show("grouped stream", lambda: describe(g)) +show( + "Index('v', 0, out='t')(g) (t is a tag)", + lambda: describe(Index("v", 0, out="t")(g)), +) +st = src( + { + "u": i64(1, 2), + "d": pa.array([{"t": 1}, {"t": 2}], pa.struct([("t", pa.int64())])), + }, + ["u"], + "S", +) +show( + "Pick('d', 't', out='u')(st) (u is a tag)", + lambda: describe(Pick("d", "t", out="u")(st)), +) diff --git a/superpowers/specs/itl-1004-audit/06_default_referential.out b/superpowers/specs/itl-1004-audit/06_default_referential.out new file mode 100644 index 000000000..fdf60b44d --- /dev/null +++ b/superpowers/specs/itl-1004-audit/06_default_referential.out @@ -0,0 +1,36 @@ +=== J1. Today: materialized with materialized Join (must stay byte-identical) +--- Join()(A, B) + keys: tags=('subject',) data=('b', 'a') + schema: tags={'subject': str | None} data={'b': float | None, 'a': float | None} + as_table columns: ['subject', 'b', 'a'] + rows: [({'subject': 'm1'}, {'b': 1.0, 'a': 0.1}), ({'subject': 'm2'}, {'b': 2.0, 'a': 0.2})] +--- Join()(A, B).pipeline_hash() + d12c05799eccd043 +--- system tag columns of Join()(A, B) + ['_tag_source_id::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127:0', '_tag_record_id::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127:0', '_tag_source_id::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588:1', '_tag_record_id::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588:1'] +--- sync rows == async rows + True + +=== H1. Source schema hash (names the system-tag columns) moves under the default +--- compute_source_schema_hash: x materialized vs x referential + ('a7ada39f09464539483466a1e3176e697a755fcfe2d763c100730e525d915d40', '6a4bcb1d6bf4061ec4c4840fd7c7f8cc876265be2f16a1dcc9696570655b2e04') + +=== H2. Function-result identity for a function that does not read the keys +--- Data hash after dropping referenced keys == v0.3 Data hash, per row + True +--- the same, as values + ['000002c339da0c5c', '00000278727a2921'] +--- result table path (function uri) is independent of the input stream + ('double_y', '872e0cf0d9d842b03280a5017222ea774267c87b7799511c47caefd6d04fd25f', 'v0', 'python.function.v0') + +=== H3. A source whose columns are all tags (v0.3) +--- ArrowTableSource(subject, x; tag_columns=[subject, x]) + RAISES ValueError: No data columns found in the table. At least one data column is required. + (raised at orcapod/core/streams/arrow_table_stream.py:121) + +=== H4. ListSource: tags come from a tag function, not from columns +--- ListSource('weight', [20.5, 22.25]) + keys: tags=('element_index',) data=('weight',) + schema: tags={'element_index': } data={'weight': } + as_table columns: ['element_index', 'weight'] + rows: [({'element_index': 0}, {'weight': 20.5}), ({'element_index': 1}, {'weight': 22.25})] diff --git a/superpowers/specs/itl-1004-audit/06_default_referential.py b/superpowers/specs/itl-1004-audit/06_default_referential.py new file mode 100644 index 000000000..d6afc37bc --- /dev/null +++ b/superpowers/specs/itl-1004-audit/06_default_referential.py @@ -0,0 +1,132 @@ +"""ITL-1004 audit, measurement 06: consequences of "declared source tags are referential by +default" (developer, November 12, 2026), and today's materialized-with-materialized Join. + +Uses the prototype of 02 for the referential side (private attributes set after +construction; nothing under ``src/`` changes). + +Run: uv run python superpowers/specs/itl-1004-audit/06_default_referential.py +Output: 06_default_referential.out +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +from pathlib import Path +from typing import Any + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod import function_pod +from orcapod.config import DEFAULT_CONFIG +from orcapod.contexts import resolve_context +from orcapod.core.datagrams import Data +from orcapod.operators import Join +from orcapod.sources import ArrowTableSource, ListSource +from orcapod.types import Schema +from orcapod.utils.schema_utils import compute_source_schema_hash + + +def _load_make_ref() -> Any: + spec = importlib.util.spec_from_file_location( + "proto", Path(__file__).with_name("02_prototype_unified.py") + ) + assert spec is not None and spec.loader is not None + mod = importlib.util.module_from_spec(spec) + with contextlib.redirect_stdout(io.StringIO()): + spec.loader.exec_module(mod) + return mod.make_ref + + +make_ref = _load_make_ref() +ctx = resolve_context(None) + +print("=== J1. Today: materialized with materialized Join (must stay byte-identical)") +A = src({"subject": s("m1", "m2"), "a": f64(0.1, 0.2)}, ["subject"], "A") +B = src({"subject": s("m1", "m2"), "b": f64(1.0, 2.0)}, ["subject"], "B") +show("Join()(A, B)", lambda: describe(Join()(A, B))) +show("Join()(A, B).pipeline_hash()", lambda: Join()(A, B).pipeline_hash().to_hex(16)) +show( + "system tag columns of Join()(A, B)", + lambda: [c for c in Join()(A, B).keys(all_info=True)[0] if c.startswith("_tag")], +) +show( + "sync rows == async rows", + lambda: ( + sorted( + (repr(t.as_dict()), repr(d.as_dict())) for t, d in Join()(A, B).iter_data() + ) + == sorted((repr(t), repr(d)) for t, d in async_rows(Join(), A, B)) + ), +) + +print( + "\n=== H1. Source schema hash (names the system-tag columns) moves under the default" +) +tag = Schema({"subject": str, "x": int}) +show( + "compute_source_schema_hash: x materialized vs x referential", + lambda: ( + compute_source_schema_hash(tag, Schema({"y": float}), ctx, DEFAULT_CONFIG), + compute_source_schema_hash( + tag, Schema({"x": int, "y": float}), ctx, DEFAULT_CONFIG + ), + ), +) + + +def cols() -> dict[str, Any]: + return {"subject": s("m1", "m2"), "x": i64(10, 20), "y": f64(0.1, 0.2)} + + +V03 = src(cols(), ["subject", "x"], "S") # explicit materialized = v0.3 +REF = make_ref(src(cols(), ["subject", "x"], "S"), ["subject", "x"]) # default ref + +print("\n=== H2. Function-result identity for a function that does not read the keys") +# The automatic select keeps only y; on the referential stream that is the row's Data +# without the referenced columns. Compare with the v0.3 row's Data (y only). +show( + "Data hash after dropping referenced keys == v0.3 Data hash, per row", + lambda: ( + [ + Data(d.as_table().drop_columns(["subject", "x"])).content_hash().to_hex(16) + for _, d in REF.iter_data() + ] + == [d.content_hash().to_hex(16) for _, d in V03.iter_data()] + ), +) +show( + "the same, as values", + lambda: [d.content_hash().to_hex(16) for _, d in V03.iter_data()], +) + + +@function_pod(output_keys="z") +def double_y(y: float | None) -> float: + return (y or 0.0) * 2 + + +show( + "result table path (function uri) is independent of the input stream", + lambda: double_y.pod.data_function.uri, +) + +print("\n=== H3. A source whose columns are all tags (v0.3)") +show( + "ArrowTableSource(subject, x; tag_columns=[subject, x])", + lambda: describe( + ArrowTableSource( + pa.table({"subject": s("m1"), "x": i64(1)}), + tag_columns=["subject", "x"], + source_id="T", + ) + ), +) + +print("\n=== H4. ListSource: tags come from a tag function, not from columns") +show( + "ListSource('weight', [20.5, 22.25])", + lambda: describe(ListSource("weight", [20.5, 22.25])), +) diff --git a/superpowers/specs/itl-1004-audit/_common.py b/superpowers/specs/itl-1004-audit/_common.py index 0d307873d..aad6c3bcc 100644 --- a/superpowers/specs/itl-1004-audit/_common.py +++ b/superpowers/specs/itl-1004-audit/_common.py @@ -12,20 +12,22 @@ from orcapod.sources import ArrowTableSource -def src(columns: dict[str, pa.Array], tags: list[str], sid: str) -> ArrowTableSource: +def src(columns: dict[str, Any], tags: list[str], sid: str) -> ArrowTableSource: """Build an ArrowTableSource with explicit Arrow types and a fixed source id.""" - return ArrowTableSource(pa.table(columns), tag_columns=tags, source_id=sid) + return ArrowTableSource( + pa.Table.from_pydict(columns), tag_columns=tags, source_id=sid + ) -def i64(*v: Any) -> pa.Array: +def i64(*v: Any) -> "pa.Array[Any]": return pa.array(list(v), type=pa.int64()) -def f64(*v: Any) -> pa.Array: +def f64(*v: Any) -> "pa.Array[Any]": return pa.array(list(v), type=pa.float64()) -def s(*v: Any) -> pa.Array: +def s(*v: Any) -> "pa.Array[Any]": return pa.array(list(v), type=pa.large_string()) @@ -59,14 +61,14 @@ def describe(stream: Any) -> str: ) -def async_rows(op: Any, *streams: Any) -> list[tuple[dict, dict]]: +def async_rows(op: Any, *streams: Any) -> list[tuple[dict[str, Any], dict[str, Any]]]: """Run an operator's ``async_execute`` over channels and collect the rows. Same harness as ``tests/test_core/operators/test_async_sync_equivalence.py``. """ from orcapod.channels import Channel - async def run() -> list[tuple[dict, dict]]: + async def run() -> list[tuple[dict[str, Any], dict[str, Any]]]: chans = [Channel(buffer_size=64) for _ in streams] out = Channel(buffer_size=64) for st, ch in zip(streams, chans): From e00797272c7c4282930c6c57ee000b1861e5fe36 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:06:55 -0800 Subject: [PATCH 3/9] docs(itl-1004): record the developer's pod-boundary decision (D1) Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 51 +++++++++++-------- 1 file changed, 29 insertions(+), 22 deletions(-) diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index a1bf98b62..6b3c225c0 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -69,7 +69,7 @@ its output (`.out`). Every example below comes from those outputs. emits such rows silently (`01 R2`). 7. **One Required rule the code makes impossible as written** (spec §6.1 against `core/function_pod.py:747`): a pod's output data is only the function's outputs, so a - reference's target never survives a pod. This is decision D1, sent to the coordinator. + reference's target never survives a pod. Decided (D1): the tag becomes plain at the pod boundary. --- @@ -124,7 +124,7 @@ The prototype (`02`) verified the rows marked †. | `MapTags`/`MapData` (`mappers.py:83`, `:104`, `:226`, `:254`) | a rename moves the column | **Change** (§5.4) | | `Index`/`Pick` (`index.py:365`, `pick.py:379`) | data replaced in place | **Reject** in-place on a referenced column (§5.5) | | `FunctionPod.output_schema` (`function_pod.py:733-747`) | output data = function outputs | **Change**: reject output names equal to a tag; D1 | -| `FunctionNode` read (`core/nodes/function_node.py:2820-2835`, `:2891`) | tags = pipeline-table columns | OK under D1(a) | +| `FunctionNode` read (`core/nodes/function_node.py:2820-2835`, `:2891`) | tags = pipeline-table columns | OK under D1 (tags are plain after a pod) | | `OperatorNode` read (`operator_node.py:1116`, `:1228`, `:1317`, `:1581`; loaded stub `:235-238`) | `tag_keys = self.keys()[0]` | **Change**: pass references (from `keys()` or descriptor) | | `CachedSource` (`cached_source.py:360-363`), `DerivedSource` plain (`derived_source.py:462`, `:490`, `:543`), `PollingSource` (`polling_source.py:288-301`, `:865`), `SpiralDBTableSource` (`spiraldb_table_source.py:623`) | tags vs rest | **Change** (§6) | | `SourceProxy.keys` (`source_proxy.py:177-185`); node descriptor (`pipeline/graph.py:178-199`) | — | OK: two name→type dicts; a name may be in both | @@ -454,7 +454,7 @@ Functions that read the copy recompute once, because the column name is in the ` materializes. - **Tags pass through** (`function_pod.py:739-747`). The output data is only the function's outputs (`:747`), so a reference's target never survives a pod, whether or not it was read. - Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. This is **D1**. + Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. Decided (D1): the tag becomes plain at the pod boundary; a fall-through / append convenience is a planned follow-up. - **An output name that collides with a tag (§6.2, Required).** - Today `output_schema` accepts it (`01 R5`: `(Schema({'x': int | None}), Schema({'x': float}))`), and reading fails with @@ -497,7 +497,7 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss | 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | | 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | | 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | -| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); D1; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | +| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | | 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | | 5 | **ITL-1018** | `MergeJoin` (D4); rename forms (§5.4) | M | 1 | 6 | | 6 | **ITL-1020** | `tags_as_data` beside references (§6.3): docs and tests; no behaviour change | S | 2a | 5 | @@ -535,7 +535,7 @@ depends on ITL-1015 (the automatic select). Add ITL-940 to the project as the fi ## 10. Decisions for the developer (most consequential first) D0 is stated first, as asked. It is the consequence of the developer's default, for -confirmation. D1 is the first open decision. +confirmation. D1 is decided and recorded here; D2 onward are open. **D0. The hash consequence of the default (measured, §3.3).** An unchanged v0.3 script gets: - new source content and pipeline hashes, system-tag names, function-node record ids, and @@ -550,23 +550,30 @@ context version. To confirm: - (b) Also provide a one-off tool that re-keys v0.3 pipeline records to the new paths. The results are reused anyway, so this only saves re-establishing records. -**D1. A reference after a function pod** (Required §6.1 is impossible as written, because a pod -outputs only its function's outputs, `function_pod.py:747`). The coordinator has put it to the -developer. The §6.1 example is conditional ("assuming the output stream retains the input data -`x`"); in Orcapod it never does. -- (a) **Recommended.** The pod's discarding of input data is the structural projection of - §6.5: the tag is materialized at the pod boundary, which the pod's output schema shows - (`binding_transitions`). No storage or result-identity change. - - **The diagnostic at a pod boundary.** With referential sources by default, *every* pod - would trigger it, so recommend **none** there. The same goes for the automatic select of - `pod(...)` (ITL-807), which also fires on every call. The `reference_materialization` - warning is for a select, drop or rename the user wrote. The materialization is still - recorded in the schema and the pipeline hash; §7.4 makes the diagnostic a matter of - configuration, never of result. -- (b) Pass referenced input columns through to the pod's output data (the §6.1 example - literally). This changes the `FunctionPod` output contract and the pipeline hashes of pods - over references; `FunctionNode` rebuilds data `x` from the stored tag on read. -- (c) Reject a pod over references until materialized. +**D1. A reference after a function pod: DECIDED (developer, November 12, 2026).** Required +§6.1 cannot hold as written, because a pod outputs only its function's outputs +(`function_pod.py:747`); the §6.1 example is conditional ("assuming the output stream retains +the input data `x`"), and in Orcapod it never does. + +The decision: at a pod boundary, a referential tag whose data column does not survive becomes +a **plain tag** with the same value and type, visible in the pod's output schema +(`binding_transitions`). A pod's output data stays the function's outputs. Storage and result +identity do not change. + +The options considered and not taken: +- passing referenced input columns through to the output data; +- rejecting a pod over references. + +Two parts remain: +- **The diagnostic at a pod boundary (the coordinator's call).** With referential sources by + default, *every* pod would trigger it, so the recommendation is **none** there. The same + goes for the automatic select of `pod(...)` (ITL-807), which fires on every call. The + `reference_materialization` warning is for a select, drop or rename the user wrote. The + transition is still recorded in the schema and the pipeline hash (§7.4: the diagnostic + never changes a result). +- **Planned follow-up, out of scope now.** A separate project issue covers a "fall-through / + append" convenience: the pod result joined back to its input stream so the tags stay + referential. Not to be built in the slices above. **D2. `pod.process(...)` and `strict=True` with referenced keys (ITL-807).** Under the default, every key is an extra data column for a function that does not read it. From d0ea69cf9e8a5d36ba6676c1a91439c3eac9552a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:09:51 -0800 Subject: [PATCH 4/9] docs(itl-1004): name the tag-only option tag_only_columns (developer decision) Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 77 ++++++++++--------- 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index 6b3c225c0..fd1095448 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -8,10 +8,12 @@ implements what the spec marks Required or Proposed, and raises decisions only f spec marks Open or that the code makes impossible or ambiguous. **Developer decisions included** (November 12, 2026): - Every source's declared tags are referential **by default**. -- An explicitly chosen plain (materialized) tag mode reproduces v0.3 exactly. This is proven - by the unchanged golden values, so the referential path is additive. +- Tags named in `tag_only_columns` (a subset of `tag_columns`) stay materialized and reproduce + v0.3 exactly. This is proven by the unchanged golden values, so the referential path is + additive. - The change is a change of default behaviour. - `Join`'s three cases (§9.1) have equal standing. +- At a pod boundary a reference whose data column does not survive becomes a plain tag (D1). **Code audited:** `origin/main` at `f51ce67c`. **Measurements:** `superpowers/specs/itl-1004-audit/` holds each script (run with `uv run`) and @@ -51,7 +53,7 @@ its output (`.out`). Every example below comes from those outputs. system-tag column names, function-node pipeline record ids, and pipeline/operator/cached table paths. Unchanged: source ids, source record ids, row `Data` hashes and results of functions that read no key, and result table paths. A user reproduces v0.3 exactly by - naming the tags with the explicit plain-tag option (§6.2). + also naming the tags in `tag_only_columns` (§6.2). 4. **Storage.** No stored-table layout changes. A referenced value is written once in every table kind. The binding is recovered on read by the same intersection rule from the saved `{tag, data}` descriptor. @@ -164,7 +166,7 @@ What remains open is whether to add an *explicit* marker on `Schema` (decision D ### 3.1 Which identities see a binding -| Identity | Hash input (file:line) | Sees a binding? | Explicit plain-tag source vs v0.3 | +| Identity | Hash input (file:line) | Sees a binding? | Source with `tag_only_columns` vs v0.3 | |---|---|---|---| | Root `pipeline_hash` | `(tag_schema, data_schema)` (`core/sources/base.py:132-139`) | **Yes**, through the pair. `04 I1`: tag-only `b9e6…`, data-only `95a5…`, referential `cc81…` | identical | | Root `content_hash` | `(class, output_schema(), source_id)` (`base.py:88-90`) | **Yes** (`04 I3`) | identical | @@ -183,7 +185,7 @@ The pattern of PRs #336 and #338 adds a term to an identity structure only when given. Here the pattern needs only one explicit term, the `ArrowTableStream` one; everywhere else the binding enters as the name in both schema halves, absent in the plain mode. Proof obligations for ITL-1011: -- every existing golden entry is unchanged, with the fixtures switched to the plain-tag option +- every existing golden entry is unchanged, with the fixtures switched to `tag_only_columns` (§6.2); - new golden entries pin the referential default. @@ -208,7 +210,7 @@ nothing about it moves. So retagging works as the spec intends (§2.2, §12.2): pipeline records are re-established under new paths, and results are found again. No migration step is required: the v0.3 pipeline tables stay where they are, with current layouts, so they are not reported stale and -are not read. The upgrade note names the plain-tag option for anyone who wants the old paths. +are not read. The upgrade note names `tag_only_columns` for anyone who wants the old paths. One consequence is not covered by the result cache. `pod.process(source)` and `pod(source, strict=True)` (ITL-807's exact form) now see every key as an extra data column @@ -236,7 +238,7 @@ and raise, unless the function takes all keys. That is decision D2. **Stored-schema changes, named exactly.** - No stored table changes, and the descriptor format is unchanged. -- Source configs gain the plain-tag option (§6.2), written only when used. `from_config` for a +- Source configs gain `tag_only_columns` (§6.2), written only when non-empty. `from_config` for a v0.3 config therefore gives the **new default**, consistent with the change of default. **Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as @@ -387,38 +389,40 @@ Two further consequences: (ITL-1012). A plain all-tags source still raises, as in v0.3 (`06 H3`). - **The automatic join and select (ITL-807)** work as described in §5.1 and §3.3. -### 6.2 The explicit plain-tag option (API shape, ITL-1012) +### 6.2 The explicit tag-only option: `tag_only_columns` (DECIDED, ITL-1012) -The previous behaviour must be reachable on every source through one option with one name. -Options: +The developer chose the name and shape (November 12, 2026): +- `tag_columns=[...]` lists **all** tags, referential by default. +- `tag_only_columns=[...]` names the subset kept as tags only (materialized, not data). +- A name in `tag_only_columns` that is not in `tag_columns` raises `ValueError`. +- The option has the same name on every source. +- In prose the spec's terms "materialized" and "referential" stay for the concepts. -| | Form | v0.3 equivalent | Mixed plain and referential in one source (§11.3: `lab` plain, `timestamp` referential) | Primary-key default (DB sources) | -|---|---|---|---|---| -| (a) **Recommended** | `tag_columns=[...]` (referential) and `plain_tag_columns=[...]` (materialized); a name in at most one | `ArrowTableSource(t, plain_tag_columns=["k"])` | yes | `plain_tag_columns=None` and `tag_columns=None` → keys referential; v0.3 = list the keys in `plain_tag_columns` | -| (b) | `tag_columns=[...]` and `tag_binding="referential" \| "plain"` (source-wide) | `tag_columns=["k"], tag_binding="plain"` | no: needs a select or drop afterwards | `tag_binding="plain"` alone reproduces v0.3 | -| (c) | `tag_columns={"k": "plain", "t": "ref"}` or a list | a mapping | yes | awkward | +| Need | Call | +|---|---| +| v0.3 behaviour, byte-identical | `ArrowTableSource(t, tag_columns=["k"], tag_only_columns=["k"])` | +| Mixed (§11.3: `lab` a label, `timestamp` referential) | `tag_columns=["lab", "timestamp"], tag_only_columns=["lab"]` | +| DB sources with primary keys as the default tags | `tag_columns=None` gives the keys, referential. The v0.3 form lists the keys in both arguments. A shorthand for "all declared tags" (for example `tag_only_columns="all"`) would save listing them; that is the ITL-1012 worker's proposal to make | -(a) is recommended because §11.3 needs both kinds on one source, and the name says what the -column is. Name alternatives are `materialized_tag_columns` (the spec's word) and -`label_columns`; "plain" is shorter and is the developer's word. For DB sources, (a) can add -`plain_tag_columns="keys"` so v0.3 keys need not be listed by hand; that is the developer's -call. +Implementation: each source computes `reference_tags = tag_columns − tag_only_columns` +and passes it to the builder (§6.1). `to_config` writes `tag_only_columns` only when it is +non-empty. **Golden fixtures (`tests/test_hashing/golden/cases.py`).** -- The 16 `tag_columns=` call sites switch to `plain_tag_columns=`. No fixture moves or is - renamed, and every entry stays byte-identical. -- The two `SourceStreamBuilder` / `DerivedSource` cases (`:584-633`) need no edit: the builder - is explicit, and `DerivedSource` keeps its origin's binding once the origin fixture is - plain. +- Each of the 15 source constructions with `tag_columns=` gains `tag_only_columns=` naming + the same columns. No fixture moves or is renamed, and every entry stays byte-identical. +- The builder call (`:596`) and the `DerivedSource` case (`:618-633`) need no edit: the + builder is explicit, and `DerivedSource` keeps its origin's binding once the origin fixture + is tag-only. - `ListSource` (`:1699`) depends on D6. - New entries pin the referential default for each source kind, `Join` case 3, and `GroupBy` on a referential key. -**Upgrade note (changelog fragment of ITL-1012), draft:** "Declared tag columns are now -referential: each is also a data column of the same name, stored once, so a function pod can -read it. This changes the schemas and hashes of pipelines that use `tag_columns`; computed -function results are reused. To keep a tag as a plain label, with the v0.3 schemas and hashes, -pass it as `plain_tag_columns=[...]`." +**Upgrade note (changelog fragment of ITL-1012), draft.** "Declared tag columns are now +referential by default: each is also a data column of the same name, stored once, so a +function pod can read it. This changes the schemas and hashes of pipelines that use +`tag_columns`; stored function results are reused. To keep a tag as a label only, with the +v0.3 schemas and hashes, also name it in `tag_only_columns=[...]`." ### 6.3 `tags_as_data` (spec §11.5, §16: legacy meaning kept, explicit migration) @@ -492,7 +496,7 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss |---|---|---|---|---|---| | 0 | **ITL-940** (add to the project) | reject a tag/data name collision across inputs (`Join`, `MergeJoin`, sync and async), pod output = tag name, `Index`/`Pick` `out=` = tag name; regression tests `01 R2`, `R5`, `05 M2` | M | — | — | | 1 | **ITL-1009 + ITL-1010 (merge)** | `reference_names`, the validator, `StreamBase.reference_tags()`; `ArrowTableStream(reference_tags=)`; its identity term; `_rows_to_table` and `_materialize_*` deduplication; all 32 rebuild sites pass references; spec §15.1 tests | M | 0 | — | -| 2a | **ITL-1011 + ITL-1012 (land together)** | builder `reference_tags`; every source's default; `plain_tag_columns` (§6.2); `to_config`; golden fixtures switched, new entries; upgrade note | M | 1 | 2b-2e | +| 2a | **ITL-1011 + ITL-1012 (land together)** | builder `reference_tags`; every source's default; `tag_only_columns` (§6.2); `to_config`; golden fixtures switched, new entries; upgrade note | M | 1 | 2b-2e | | 2b | **ITL-1015** | select and drop by role; config, diagnostic, `binding_transitions`; §15.5 tests | S | 1 | 2a, 2c-2e | | 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | | 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | @@ -518,7 +522,7 @@ but touches every operator. **Splits and merges.** Merge ITL-1009 into ITL-1010: with the intersection rule ITL-1009 is a helper and a validator, testable only with a stream. Land ITL-1011 with ITL-1012: the golden -fixture switch is the proof ITL-1011 asks for, and it needs the plain-tag option. ITL-1013 +fixture switch is the proof ITL-1011 asks for, and it needs `tag_only_columns`. ITL-1013 depends on ITL-1015 (the automatic select). Add ITL-940 to the project as the first issue. **Blocked by other issues?** @@ -612,11 +616,10 @@ needs a shape).** fixture edit, and the upgrade note says so. - (b) Referential like every source, with a plain option added to `ListSource`. -**D7. The plain-tag option's name and form (§6.2).** -- (a) **Recommended.** `plain_tag_columns=[...]` beside `tag_columns`, the same on every - source, plus optionally `plain_tag_columns="keys"` on database sources. -- (b) The source-wide `tag_binding="plain"`. -- (c) A mapping. +**D7. The tag-only option: DECIDED** (developer, November 12, 2026). It is +`tag_only_columns=[...]`, a subset of `tag_columns` (§6.2). Open for the ITL-1012 worker to +propose: a shorthand for "all declared tags", useful for database sources whose tags default +to primary keys. --- From 0385596e40ef2d595ee32a695b9979276bc9ddce Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:10:44 -0800 Subject: [PATCH 5/9] docs(itl-1004): record the developer's rename rules (spec 10.1) Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 41 +++++++++++-------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index fd1095448..e3dd7ebed 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -14,6 +14,8 @@ spec marks Open or that the code makes impossible or ambiguous. - The change is a change of default behaviour. - `Join`'s three cases (§9.1) have equal standing. - At a pod boundary a reference whose data column does not survive becomes a plain tag (D1). +- Renames (§5.4): `map_tags` moves both roles of a reference, `tag_only=True` and `map_data` + leave a plain tag. **Code audited:** `origin/main` at `f51ce67c`. **Measurements:** `superpowers/specs/itl-1004-audit/` holds each script (run with `uv run`) and @@ -267,7 +269,7 @@ that change at every site. | `GroupBy`/`Batch` async | `Duplicate column name 'x'` (`02 P4`) | as sync | `_rows_to_table` (ITL-1010) | — | | Select/Drop data (`column_selection.py:158`, `:413`) | raise when dropping referenced `x` (`02 P2`) | materialize (§7.2) | §5.3 | S | | Select/Drop tag (`:37`, `:286`) | the physical column dropped, so data `x` is lost (`02 P2`) | remove tag, keep data (§10) | §5.3 | S | -| `MapTags`/`MapData` | renaming a referenced tag raises (`02 P3`) | §10.1 (proposed) | §5.4 | M | +| `MapTags`/`MapData` | renaming a referenced tag raises (`02 P3`) | decided (§5.4) | §5.4 | M | | `Index`/`Pick` | §5.5 | §4.5 | §5.5 | S | | `Join` | §5.1 | §9 | §5.1 | M | | `MergeJoin` | §5.2 | open (§17) | §5.2 | M | @@ -341,17 +343,23 @@ because they drop per datagram. exposes `binding_transitions(...)` → `{name: kept | materialized | removed}` for pipeline views. -### 5.4 Renames (spec §10.1, Proposed: how) +### 5.4 Renames: DECIDED (developer, November 12, 2026; closes spec §10.1) -`MapTags` and `MapData` rename within their own namespace and refuse a name of the other one -(`01 R4`; `mappers.py:104`, `:254`). Per §10.1: -- **Renaming only the tag (or only the data) of a reference materializes it.** The renamed - side gets a zero-copy duplicate of the column under the new name, using the idiom - `table.append_column(field.with_name(new), table.column(x))` from `derived_source.py:429-434`. - The other side keeps the original. The diagnostic fires. -- **Renaming both together preserves the reference.** No operator does this today. Add one - explicit form, for example `MapTags({...}, with_references=True)` or a `MapReferences` - operator. The name is the developer's call at ITL-1018. +Today `MapTags` and `MapData` rename within their own namespace and refuse a name of the +other one (`01 R4`; `mappers.py:104`, `:254`). With references (prototype `02 P3`), renaming +either side of a reference raises. The decided rules: + +| Call | On a referential tag `x` → `y` | On a plain tag | Diagnostic | +|---|---|---|---| +| `map_tags({x: y})` (default) | renames the column in **both** roles; the reference survives as `y` (one physical column: rename the column, then rename it in both role lists) | as today | none (nothing materializes) | +| `map_tags({x: y}, tag_only=True)` (spelling is the coordinator's; the developer may rename it) | the tag becomes plain `y`, a zero-copy duplicate (`table.append_column(field.with_name(y), table.column(x))`, the `derived_source.py:429-434` idiom); data `x` keeps its name | as today | none (the user asked) | +| `map_data({x: y})` | always a data-only move: data becomes `y`; tag `x` stays as a plain tag under the old name (zero-copy duplicate) | n/a | none (the user asked) | + +A rename that would produce a plain tag and a data column of one name, or collide with an +existing column, raises as today (`mappers.py:104`, `:254`). `tag_only` enters `MapTags`'s +identity only when true, so the identity of a plain-tag rename stays v0.3's. The EDI test +renames `timestamp` to `spike_time` / `gaze_time` with `map_tags`. Size M (ITL-1018, or the +ITL-1023 slice). ### 5.5 `Index` and `Pick` @@ -476,8 +484,8 @@ Functions that read the copy recompute once, because the column name is in the ` | Question | Answer | |---|---| | Outer and unmatched-row joins (§9.7) | **The code decides:** none exist. `Join` and `MergeJoin` use `how="inner"` (`join.py:219`, `merge_join.py:393`), `SemiJoin` uses `how="semi"` (`semijoin.py:120`). Record the §9.7 policy in the operator contract for future ones | -| Rename both namespaces | §5.4: preserve, through an explicit both-sides form (name at ITL-1018) | -| Rename one side materializes? | §5.4: yes, as a zero-copy duplicate; the existing contract renames one namespace (`mappers.py:104`, `:254`) | +| Rename both namespaces | **Decided:** `map_tags` on a referential tag renames both roles; the reference survives (§5.4) | +| Rename one side materializes? | **Decided:** `map_tags(..., tag_only=True)` and `map_data` leave a plain tag (zero-copy), with no warning (§5.4) | | `Batch` | **The code decides:** every member column becomes a list in input order (`batch.py:60-100`). A reference is one column, so it is preserved (`02 P4`) | | `MergeJoin` | §5.2; D4 | | Avoiding duplicate stored values | **The code decides:** one physical column, and rows share buffers (`arrow_table_stream.py:352-353`). Option B, which the code already is; D3 asks only whether to add an explicit marker | @@ -503,7 +511,7 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss | 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | | 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | | 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | -| 5 | **ITL-1018** | `MergeJoin` (D4); rename forms (§5.4) | M | 1 | 6 | +| 5 | **ITL-1018** | `MergeJoin` (D4); the decided rename rules (§5.4), unless the ITL-1023 slice lands them | M | 1 | 6 | | 6 | **ITL-1020** | `tags_as_data` beside references (§6.3): docs and tests; no behaviour change | S | 2a | 5 | | 7 | **ITL-1021** | ediop and the notebook, without copies; EDI-shaped integration test (§15.7) | M | 2a, 2c, 2d, 3 | — | | 8 | **ITL-1022** | operator table in `docs/`; `streams.md:10-25`, `operators.md:31-33`, `sources.md`, `hashing.md`; CLAUDE.md and `.zed/rules` rule | S | all | — | @@ -511,7 +519,8 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss **ITL-1023 (workshop slice, go/no-go November 14).** The thinnest path is ITL-1010 (+1009), 1012 (sources only), 1015, 1013, 1016 and 1017, plus guard errors ("not supported with referential tags yet") in: -- `Batch`, `MergeJoin`, `MapTags`, `MapData`, `Index`, `Pick` when an input has references; +- `Batch`, `MergeJoin`, `Index`, `Pick` when an input has references (the rename rules of §5.4 + are in the slice); - `OperatorNode` reload and `DerivedSource`; - `pod.process`/`strict=True` with references (D2), if D2 is not settled. @@ -572,7 +581,7 @@ Two parts remain: - **The diagnostic at a pod boundary (the coordinator's call).** With referential sources by default, *every* pod would trigger it, so the recommendation is **none** there. The same goes for the automatic select of `pod(...)` (ITL-807), which fires on every call. The - `reference_materialization` warning is for a select, drop or rename the user wrote. The + `reference_materialization` warning is for a select or drop the user wrote (a rename never warns, §5.4). The transition is still recorded in the schema and the pipeline hash (§7.4: the diagnostic never changes a result). - **Planned follow-up, out of scope now.** A separate project issue covers a "fall-through / From ace3bb0de242e1b631ad5991ba42ae5b67b9fc3a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:21:31 -0800 Subject: [PATCH 6/9] docs(itl-1004): address CodeRabbit round 1 - tag_only_columns v0.3 reproduction stated as a proof obligation of the golden fixture switch, not as already proven - from_config rule for a config without tag_only_columns as decision D8 - the default flip is the last step of ITL-1012, after 1013/1015/1016/1017/1019 - new section 11: rules under reconsideration (pod boundary, GroupBy non-key) - audit harness: feeders, operator and collector run concurrently; no quoted annotations; plan heading levels Co-Authored-By: Claude Opus 5.5 --- ...6-11-13-itl-1004-referential-tags-audit.md | 14 +-- ...4-referential-tags-implementation-scope.md | 110 ++++++++++++++++-- superpowers/specs/itl-1004-audit/_common.py | 36 ++++-- 3 files changed, 134 insertions(+), 26 deletions(-) diff --git a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md index 9cbffb0eb..284868271 100644 --- a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md +++ b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md @@ -18,7 +18,7 @@ document from the evidence. --- -### Task 1: Representation sites +## Task 1: Representation sites - [ ] Read `core/streams/arrow_table_stream.py`, `core/datagrams/*`, `types.py` (Schema, ColumnConfig), `utils/schema_utils.py`, `core/streams/base.py`. - [ ] grep every `keys()` / `output_schema` / `identity_structure` / `pipeline_identity_structure` @@ -26,29 +26,29 @@ document from the evidence. - [ ] Script `01_representation.py`: build an `ArrowTableStream` with a column in both tag and data lists (if constructible) and record what each public method returns. -### Task 2: Identity and hashes +## Task 2: Identity and hashes - [ ] Read `core/base.py`, `core/sources/base.py`, `core/nodes/function_node.py` (`_build_record_id_preimage`), `core/function_pod.py`, `core/datagrams` content hash, `derived_source.py` (PR #336/#338 pattern). - [ ] Script `02_identity.py`: show which hash inputs contain the tag/data split. -### Task 3: Storage +## Task 3: Storage - [ ] Read FunctionNode / OperatorNode / CachedSource / DerivedSource write and read paths. - [ ] Script `03_storage.py`: stored columns of each table kind for a small pipeline. -### Task 4: Operators +## Task 4: Operators - [ ] For every file in `core/operators` and the FunctionPod call path, measure today's behaviour with a same-named tag and data column (`04_operators.py`), sync and async. - [ ] Join source-info of merged column: options. -### Task 5–6: Sources and function pods +## Task 5–6: Sources and function pods - [ ] Read `arrow_table_source.py`, `stream_builder.py`, `spiraldb_table_source.py`, `db_table_source.py`, `derived_source.py`; `function_pod.py` output collision (`05_function_pod.py`). -### Task 7–9: Open questions, delivery plan, decisions +## Task 7–9: Open questions, delivery plan, decisions - [ ] Answer section 17 from code; slice plan with sizes, dependencies, tests; decisions list. -### Task 10: Finish +## Task 10: Finish - [ ] Doc under ~900 lines; ruff on audit scripts; `uv run pytest tests/test_docs` not affected (superpowers is not a docs page); merge origin/main; push; PR `Refs ITL-1004`; `orca linear attach`; `orca worktree set --pr`; worker_done. diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index e3dd7ebed..792c632d1 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -8,9 +8,11 @@ implements what the spec marks Required or Proposed, and raises decisions only f spec marks Open or that the code makes impossible or ambiguous. **Developer decisions included** (November 12, 2026): - Every source's declared tags are referential **by default**. -- Tags named in `tag_only_columns` (a subset of `tag_columns`) stay materialized and reproduce - v0.3 exactly. This is proven by the unchanged golden values, so the referential path is - additive. +- Tags named in `tag_only_columns` (a subset of `tag_columns`) stay materialized and must + reproduce v0.3 exactly, so the referential path is additive. This is a requirement and a + proof obligation: it is proven once the golden fixtures, switched to `tag_only_columns`, pass + unchanged (ITL-1011, ITL-1012). This audit shows only that no hash input of a tag-only + stream changes (§3). - The change is a change of default behaviour. - `Join`'s three cases (§9.1) have equal standing. - At a pod boundary a reference whose data column does not survive becomes a plain tag (D1). @@ -240,8 +242,16 @@ and raise, unless the function takes all keys. That is decision D2. **Stored-schema changes, named exactly.** - No stored table changes, and the descriptor format is unchanged. -- Source configs gain `tag_only_columns` (§6.2), written only when non-empty. `from_config` for a - v0.3 config therefore gives the **new default**, consistent with the change of default. +- Source configs gain `tag_only_columns` (§6.2). **Recommended rule (decision D8):** + - v0.4 `to_config` always writes the key, as an empty list when every tag is referential. + - A config **without** the key is a v0.3 config, and `from_config` / + `resolve_source_from_config` restores it with every declared tag tag-only, its v0.3 + meaning. + - So a saved v0.3 job reloads with the content and pipeline hashes it saved. Otherwise its + sources would be rebuilt referentially and silently disagree with the saved descriptors + (`pipeline/job.py` `PipelineJob.save` embeds each source's `to_config()` beside the node + hashes). + - The referential default applies to sources built in code. **Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as plain tags only (§16.1). Nothing is reinterpreted. @@ -504,16 +514,17 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss |---|---|---|---|---|---| | 0 | **ITL-940** (add to the project) | reject a tag/data name collision across inputs (`Join`, `MergeJoin`, sync and async), pod output = tag name, `Index`/`Pick` `out=` = tag name; regression tests `01 R2`, `R5`, `05 M2` | M | — | — | | 1 | **ITL-1009 + ITL-1010 (merge)** | `reference_names`, the validator, `StreamBase.reference_tags()`; `ArrowTableStream(reference_tags=)`; its identity term; `_rows_to_table` and `_materialize_*` deduplication; all 32 rebuild sites pass references; spec §15.1 tests | M | 0 | — | -| 2a | **ITL-1011 + ITL-1012 (land together)** | builder `reference_tags`; every source's default; `tag_only_columns` (§6.2); `to_config`; golden fixtures switched, new entries; upgrade note | M | 1 | 2b-2e | +| 2a | **ITL-1011 + ITL-1012, part 1** | builder `reference_tags`; `tag_only_columns` on every source (§6.2); `to_config` per D8; referential sources available but **not yet the default** | M | 1 | 2b-2e | | 2b | **ITL-1015** | select and drop by role; config, diagnostic, `binding_transitions`; §15.5 tests | S | 1 | 2a, 2c-2e | | 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | | 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | | 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | | 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | | 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | +| 4b | **ITL-1012, part 2 (its last step), with ITL-1011** | **flip the default** to referential; golden fixtures switched to `tag_only_columns`, new entries; upgrade note | S | **1013, 1015, 1016, 1017, 1019** | — | | 5 | **ITL-1018** | `MergeJoin` (D4); the decided rename rules (§5.4), unless the ITL-1023 slice lands them | M | 1 | 6 | | 6 | **ITL-1020** | `tags_as_data` beside references (§6.3): docs and tests; no behaviour change | S | 2a | 5 | -| 7 | **ITL-1021** | ediop and the notebook, without copies; EDI-shaped integration test (§15.7) | M | 2a, 2c, 2d, 3 | — | +| 7 | **ITL-1021** | ediop and the notebook, without copies; EDI-shaped integration test (§15.7) | M | 4b | — | | 8 | **ITL-1022** | operator table in `docs/`; `streams.md:10-25`, `operators.md:31-33`, `sources.md`, `hashing.md`; CLAUDE.md and `.zed/rules` rule | S | all | — | **ITL-1023 (workshop slice, go/no-go November 14).** The thinnest path is ITL-1010 (+1009), @@ -529,6 +540,11 @@ Each guard is one check against `reference_tags()` at `validate_inputs`. Estimat (`02`, `06`) give the test shapes. The riskiest part is the 32-site rebuild; it is mechanical but touches every operator. +**The default flip is the last step of ITL-1012.** It depends on ITL-1013, 1015, 1016, 1017 +and 1019. Flipped earlier, `main` would break: `pod(source)` (the automatic select raises), +`source_a.join(source_b)` (shared keys collide), `GroupBy` on a key, and reloading a node. +ITL-1023's integration branch carries everything at once and is unaffected. + **Splits and merges.** Merge ITL-1009 into ITL-1010: with the intersection rule ITL-1009 is a helper and a validator, testable only with a stream. Land ITL-1011 with ITL-1012: the golden fixture switch is the proof ITL-1011 asks for, and it needs `tag_only_columns`. ITL-1013 @@ -629,6 +645,86 @@ needs a shape).** `tag_only_columns=[...]`, a subset of `tag_columns` (§6.2). Open for the ITL-1012 worker to propose: a shorthand for "all declared tags", useful for database sources whose tags default to primary keys. +**D8. How a stored source config without `tag_only_columns` is restored** (a stored-config +rule, §4). +- (a) **Recommended.** v0.4 always writes the key; a config without it is v0.3 and restores + with every declared tag tag-only. Saved v0.3 jobs reload with their saved hashes. +- (b) A config without the key gets the new default. Simpler, but a reloaded v0.3 job + rebuilds its sources referentially and no longer matches its saved hashes and table paths. + +--- + +## 11. Rules under reconsideration (November 12) + +The developer may change two rules. For each, here is what changes against this document. + +### A. A referential tag stays referential through a function pod (instead of D1) + +The pod's output keeps the data role: output data = function outputs + the input's referenced +columns. The result store is unchanged, and the node gives the data role back from the stored +tag column on read. + +**Operator table (§5, §7):** +- `FunctionPod.output_schema` adds the referenced input columns to the data schema. +- The automatic select of `pod(...)` must **keep** referenced columns, so they are not made + plain before the pod. The pod then passes only its parameters to the function. +- Output rows must carry `x` (from the input row) beside the outputs, and `_table_from_pairs` + must deduplicate it like `_rows_to_table`. +- The output-collision rule is unchanged. +- Downstream, a pod's output meets `Join` case 3 with sources. +- Size: ITL-1013 grows from M to L. + +**Hash verdict (§3):** +- Result identity is at risk. The memo key is the whole input `Data` hash + (`result_cache.py:217`, `function_node.py:140`, `:2407-2408`). If referenced keys stay in + `Data`, a function that does not read them gets a new key, and v0.3 results would not be + reused (spec §12.4). +- So the pod must hash **its parameter projection** of the input `Data`, never the whole + row. For a tag-only stream that projection is the whole `Data` (ITL-807's strict contract), + so no golden value moves. +- Pipeline hashes of pods over references change (new shapes only). No data context + version. + +**Node storage (§4):** +- `pdb_v2` already holds `x` as a tag, and `rdb_v2` holds only outputs, so neither changes. +- On read, `FunctionNode` rebuilds the output stream with `reference_tags` = the input's + references, and the data `x` *is* the stored tag column (`function_node.py:2891`). +- The value is still written once. + +### B. `GroupBy`: a non-key referential tag becomes a data-only list column + +Under B: +- A non-key referential tag loses its tag role and becomes a data-only list. +- A non-key plain tag stays a list-valued tag, as in v0.3. +- `Join` refuses a shared list-valued tag. + +**Operator table (§5):** +- `GroupBy` on a non-key reference: output tags omit `x`, data `x` is `list[T]`, and the + output references are the referenced keys only. `unary_output_schema` (`group_by.py:302`) + and the rebuild (`:258`) change together. +- The member sort still reads the input's non-key tags (`:192-215`), so the order is + unchanged. +- `Batch` is not covered by B. Batch has no keys, so B's analogue would make every reference + a data-only list; it needs a ruling. +- `Join` gets a new check: a shared tag that is list-valued in either input is refused when + the graph is built, from `output_schema`. + +**Hash verdict (§3):** +- No golden entry moves. Plain non-key tags stay lists in v0.3 form, and the golden cases + join only ungrouped streams (`cases.py:1113`). +- The new `Join` refusal is a behaviour change for v0.3 pipelines that join two grouped + streams on a shared list-valued tag. They get an error where they ran before. No hash + moves, but it belongs in the upgrade note. + +**Node storage (§4):** +- Unchanged. `odb_v1` stores one flat table, and roles come from the descriptor. + +### Reconsideration: where each rule touches the slices + +- **A** touches only ITL-1013 and ITL-1019. +- **B** touches only ITL-1017 and ITL-1016. + +Neither changes the representation (§2) or the stores (§4). --- diff --git a/superpowers/specs/itl-1004-audit/_common.py b/superpowers/specs/itl-1004-audit/_common.py index aad6c3bcc..53b12ffc1 100644 --- a/superpowers/specs/itl-1004-audit/_common.py +++ b/superpowers/specs/itl-1004-audit/_common.py @@ -19,15 +19,15 @@ def src(columns: dict[str, Any], tags: list[str], sid: str) -> ArrowTableSource: ) -def i64(*v: Any) -> "pa.Array[Any]": +def i64(*v: Any) -> pa.Array[Any]: return pa.array(list(v), type=pa.int64()) -def f64(*v: Any) -> "pa.Array[Any]": +def f64(*v: Any) -> pa.Array[Any]: return pa.array(list(v), type=pa.float64()) -def s(*v: Any) -> "pa.Array[Any]": +def s(*v: Any) -> pa.Array[Any]: return pa.array(list(v), type=pa.large_string()) @@ -68,18 +68,30 @@ def async_rows(op: Any, *streams: Any) -> list[tuple[dict[str, Any], dict[str, A """ from orcapod.channels import Channel + async def feed(stream: Any, channel: Any) -> None: + for row in stream.iter_data(): + await channel.writer.send(row) + await channel.writer.close() + async def run() -> list[tuple[dict[str, Any], dict[str, Any]]]: chans = [Channel(buffer_size=64) for _ in streams] out = Channel(buffer_size=64) - for st, ch in zip(streams, chans): - for row in st.iter_data(): - await ch.writer.send(row) - await ch.writer.close() - await op.async_execute( - [c.reader for c in chans], - out.writer, - input_pipeline_hashes=[st.pipeline_hash() for st in streams], + # Feeders, the operator and the collector run together, so a bounded channel + # never blocks a feeder before its reader has started. + collected: list[Any] = [] + + async def collect() -> None: + collected.extend(await out.reader.collect()) + + await asyncio.gather( + *(feed(st, ch) for st, ch in zip(streams, chans)), + op.async_execute( + [c.reader for c in chans], + out.writer, + input_pipeline_hashes=[st.pipeline_hash() for st in streams], + ), + collect(), ) - return [(t.as_dict(), d.as_dict()) for t, d in await out.reader.collect()] + return [(t.as_dict(), d.as_dict()) for t, d in collected] return asyncio.run(run()) From c669a3478c1c2e4a5cd5f9ac05834c35089d578b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:26:13 -0800 Subject: [PATCH 7/9] docs(itl-1004): record the settled pod-boundary and GroupBy rules, version pinning - section 11 replaced: GroupBy unchanged; references fall through on pod(...) only, with a row-aligned ReferenceFallThrough step proposed and its duplicate-tag test - D1 final form; D8 is now version pinning of saved files - measurement 07: the version information a saved job holds today Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 232 +++++++++--------- .../itl-1004-audit/07_saved_job_versions.out | 13 + .../itl-1004-audit/07_saved_job_versions.py | 60 +++++ 3 files changed, 193 insertions(+), 112 deletions(-) create mode 100644 superpowers/specs/itl-1004-audit/07_saved_job_versions.out create mode 100644 superpowers/specs/itl-1004-audit/07_saved_job_versions.py diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index 792c632d1..3a74a5f33 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -15,7 +15,12 @@ spec marks Open or that the code makes impossible or ambiguous. stream changes (§3). - The change is a change of default behaviour. - `Join`'s three cases (§9.1) have equal standing. -- At a pod boundary a reference whose data column does not survive becomes a plain tag (D1). +- Pod boundary (D1). With `pod.process(...)`, and `pod(..., strict=True)`, a reference whose + data column does not survive becomes a plain tag. With `pod(...)`, an automatic step around + the pod gives the input's references back their data role (§11). Full append is deferred + (ITL-1025). +- `GroupBy` follows spec §8 unchanged; `Join` gets no special rule for list-valued tags. +- Saved jobs carry version information that pins behaviour (§4). - Renames (§5.4): `map_tags` moves both roles of a reference, `tag_only=True` and `map_data` leave a plain tag. @@ -242,16 +247,31 @@ and raise, unless the function takes all keys. That is decision D2. **Stored-schema changes, named exactly.** - No stored table changes, and the descriptor format is unchanged. -- Source configs gain `tag_only_columns` (§6.2). **Recommended rule (decision D8):** - - v0.4 `to_config` always writes the key, as an empty list when every tag is referential. - - A config **without** the key is a v0.3 config, and `from_config` / - `resolve_source_from_config` restores it with every declared tag tag-only, its v0.3 - meaning. - - So a saved v0.3 job reloads with the content and pipeline hashes it saved. Otherwise its - sources would be rebuilt referentially and silently disagree with the saved descriptors - (`pipeline/job.py` `PipelineJob.save` embeds each source's `to_config()` beside the node - hashes). - - The referential default applies to sources built in code. +- Source configs gain `tag_only_columns` (§6.2). +- **Version pinning (developer, November 12, 2026: "saved jobs should clearly come with + version info to pin behavior").** Measured (`07`), a saved job today holds: + - `orcapod_pipeline_job_version = "0.1.0"` (written `pipeline/job.py:1356`, checked + `:1422-1426`); + - the embedded pipeline's `orcapod_pipeline_version = "0.1.0"` (`pipeline/graph.py:217`, + checked `:258`); + - a `data_context_key` per node (`std:v1:default`). + + It holds no Orcapod package version, and the source configs in `bindings.sources` carry + none. Both format versions were already `"0.1.0"` at v0.2.0, so they were not bumped + across the v0.2-to-v0.3 hash changes. The context key will not change in v0.4 (§3.2), so + it cannot tell v0.3 from v0.4. +- **The rule (decision D8):** + - v0.4 bumps both format versions in the change that flips the source default (the last + step of ITL-1012). + - Loading a `"0.1.0"` file pins v0.3 meaning: every declared tag tag-only. The saved + content and pipeline hashes then match. + - A standalone source config (a `to_config()` dict outside a job or pipeline file) carries + no version, so v0.4 adds a `config_version` to source configs if those must reload + across versions. + - Sent to the coordinator before the v0.3.0 tag: nothing is strictly required in v0.3.0, + but writing `orcapod_version` (the package version) beside both format versions costs + about two lines and no hash. It lets v0.4 tell v0.3.0 files from files written by the + ITL-1023 integration branch, which uses format `"0.1.0"` with referential tags. **Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as plain tags only (§16.1). Nothing is reinterpreted. @@ -476,7 +496,9 @@ Functions that read the copy recompute once, because the column name is in the ` materializes. - **Tags pass through** (`function_pod.py:739-747`). The output data is only the function's outputs (`:747`), so a reference's target never survives a pod, whether or not it was read. - Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. Decided (D1): the tag becomes plain at the pod boundary; a fall-through / append convenience is a planned follow-up. + Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. Decided (D1): with + `process` the tag becomes plain at the boundary; with `pod(...)` an automatic step gives the + references back their data role (§11). - **An output name that collides with a tag (§6.2, Required).** - Today `output_schema` accepts it (`01 R5`: `(Schema({'x': int | None}), Schema({'x': float}))`), and reading fails with @@ -519,7 +541,7 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss | 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | | 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | | 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | -| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule; automatic join and select with references; §15.2 and §15.6 result-identity tests | M | 1, 2b; 2c for the automatic join | 4 | +| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule (plain under `process`); automatic join and select with references; the `ReferenceFallThrough` step on `pod(...)` (§11, after its duplicate-tag test is agreed); §15.2 and §15.6 result-identity tests | M+M | 1, 2b; 2c and 2d for the automatic join and the fall-through | 4 | | 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | | 4b | **ITL-1012, part 2 (its last step), with ITL-1011** | **flip the default** to referential; golden fixtures switched to `tag_only_columns`, new entries; upgrade note | S | **1013, 1015, 1016, 1017, 1019** | — | | 5 | **ITL-1018** | `MergeJoin` (D4); the decided rename rules (§5.4), unless the ITL-1023 slice lands them | M | 1 | 6 | @@ -581,28 +603,19 @@ context version. To confirm: **D1. A reference after a function pod: DECIDED (developer, November 12, 2026).** Required §6.1 cannot hold as written, because a pod outputs only its function's outputs -(`function_pod.py:747`); the §6.1 example is conditional ("assuming the output stream retains -the input data `x`"), and in Orcapod it never does. - -The decision: at a pod boundary, a referential tag whose data column does not survive becomes -a **plain tag** with the same value and type, visible in the pod's output schema -(`binding_transitions`). A pod's output data stays the function's outputs. Storage and result -identity do not change. - -The options considered and not taken: -- passing referenced input columns through to the output data; -- rejecting a pod over references. - -Two parts remain: -- **The diagnostic at a pod boundary (the coordinator's call).** With referential sources by - default, *every* pod would trigger it, so the recommendation is **none** there. The same - goes for the automatic select of `pod(...)` (ITL-807), which fires on every call. The - `reference_materialization` warning is for a select or drop the user wrote (a rename never warns, §5.4). The - transition is still recorded in the schema and the pipeline hash (§7.4: the diagnostic - never changes a result). -- **Planned follow-up, out of scope now.** A separate project issue covers a "fall-through / - append" convenience: the pod result joined back to its input stream so the tags stay - referential. Not to be built in the slices above. +(`function_pod.py:747`). The §6.1 example is conditional ("assuming the output stream +retains the input data `x`"), and in Orcapod it never does. The decision: +- **`pod.process(stream)` and `pod(stream, strict=True)`:** a reference whose data column + does not survive becomes a **plain tag** (same value and type), visible in the output + schema. No automation. The `strict` reading is the coordinator's; the developer may + correct it. +- **`pod(stream)`:** an automatic step **around** the pod, of the ITL-807 family and visible + in the graph, gives the input's referential keys their data role back. The pod node, what + it stores and function-result identity are unchanged. Only referential key columns fall + through; full append is deferred (ITL-1025). The mechanism is proposed in §11. +- **The diagnostic** is the coordinator's call. Recommended none at the pod boundary or in + the automatic select: the `reference_materialization` warning is for a select or drop the + user wrote (a rename never warns, §5.4). **D2. `pod.process(...)` and `strict=True` with referenced keys (ITL-807).** Under the default, every key is an extra data column for a function that does not read it. @@ -645,86 +658,81 @@ needs a shape).** `tag_only_columns=[...]`, a subset of `tag_columns` (§6.2). Open for the ITL-1012 worker to propose: a shorthand for "all declared tags", useful for database sources whose tags default to primary keys. -**D8. How a stored source config without `tag_only_columns` is restored** (a stored-config -rule, §4). -- (a) **Recommended.** v0.4 always writes the key; a config without it is v0.3 and restores - with every declared tag tag-only. Saved v0.3 jobs reload with their saved hashes. -- (b) A config without the key gets the new default. Simpler, but a reloaded v0.3 job - rebuilds its sources referentially and no longer matches its saved hashes and table paths. +**D8. Version pinning of saved files (decided in principle by the developer; the mechanism +needs approval).** +- (a) **Recommended.** v0.4 bumps `PIPELINE_JOB_FORMAT_VERSION` and `PIPELINE_FORMAT_VERSION` + with the default flip, and `"0.1.0"` loads with v0.3 tag meaning. v0.3.0 adds + `orcapod_version` before its tag (informational). v0.4 adds `config_version` to standalone + source configs. +- (b) Rely on the format-version bump alone. Files from the ITL-1023 integration branch + (format `"0.1.0"`, referential tags) would then be misread as v0.3. --- -## 11. Rules under reconsideration (November 12) - -The developer may change two rules. For each, here is what changes against this document. - -### A. A referential tag stays referential through a function pod (instead of D1) - -The pod's output keeps the data role: output data = function outputs + the input's referenced -columns. The result store is unchanged, and the node gives the data role back from the stored -tag column on read. - -**Operator table (§5, §7):** -- `FunctionPod.output_schema` adds the referenced input columns to the data schema. -- The automatic select of `pod(...)` must **keep** referenced columns, so they are not made - plain before the pod. The pod then passes only its parameters to the function. -- Output rows must carry `x` (from the input row) beside the outputs, and `_table_from_pairs` - must deduplicate it like `_rows_to_table`. -- The output-collision rule is unchanged. -- Downstream, a pod's output meets `Join` case 3 with sources. -- Size: ITL-1013 grows from M to L. - -**Hash verdict (§3):** -- Result identity is at risk. The memo key is the whole input `Data` hash - (`result_cache.py:217`, `function_node.py:140`, `:2407-2408`). If referenced keys stay in - `Data`, a function that does not read them gets a new key, and v0.3 results would not be - reused (spec §12.4). -- So the pod must hash **its parameter projection** of the input `Data`, never the whole - row. For a tag-only stream that projection is the whole `Data` (ITL-807's strict contract), - so no golden value moves. -- Pipeline hashes of pods over references change (new shapes only). No data context - version. - -**Node storage (§4):** -- `pdb_v2` already holds `x` as a tag, and `rdb_v2` holds only outputs, so neither changes. -- On read, `FunctionNode` rebuilds the output stream with `reference_tags` = the input's - references, and the data `x` *is* the stored tag column (`function_node.py:2891`). -- The value is still written once. - -### B. `GroupBy`: a non-key referential tag becomes a data-only list column - -Under B: -- A non-key referential tag loses its tag role and becomes a data-only list. -- A non-key plain tag stays a list-valued tag, as in v0.3. -- `Join` refuses a shared list-valued tag. - -**Operator table (§5):** -- `GroupBy` on a non-key reference: output tags omit `x`, data `x` is `list[T]`, and the - output references are the referenced keys only. `unary_output_schema` (`group_by.py:302`) - and the rebuild (`:258`) change together. -- The member sort still reads the input's non-key tags (`:192-215`), so the order is - unchanged. -- `Batch` is not covered by B. Batch has no keys, so B's analogue would make every reference - a data-only list; it needs a ruling. -- `Join` gets a new check: a shared tag that is list-valued in either input is refused when - the graph is built, from `output_schema`. - -**Hash verdict (§3):** -- No golden entry moves. Plain non-key tags stay lists in v0.3 form, and the golden cases - join only ungrouped streams (`cases.py:1113`). -- The new `Join` refusal is a behaviour change for v0.3 pipelines that join two grouped - streams on a shared list-valued tag. They get an error where they ran before. No hash - moves, but it belongs in the upgrade note. - -**Node storage (§4):** -- Unchanged. `odb_v1` stores one flat table, and roles come from the descriptor. - -### Reconsideration: where each rule touches the slices - -- **A** touches only ITL-1013 and ITL-1019. -- **B** touches only ITL-1017 and ITL-1016. - -Neither changes the representation (§2) or the stores (§4). +## 11. The two rules settled late (developer, November 12) + +### B. `GroupBy`: no change + +Spec §8.2 stands: a non-key referential tag becomes the same list in both roles and stays +referential, and plain tags behave as in v0.3 (§5). `Join` gets no special refusal for a +shared list-valued tag. `Join` matches same-named tags whatever their type, and streams +whose same-named tags mean different things must be renamed. What `Join` does today with a +shared list-valued tag (correct match, wrong match, or a join-engine error, sync and async) +is for the ITL-1023 worker to measure and report. + +### A. Pod boundary: references fall through on `pod(...)` only + +What the code must do (D1): +- `process` and `strict`: the tag becomes plain. +- `pod(...)`: an automatic step after the pod restores the data role of the input's + referential keys. + +**Correctness constraints for that step:** +- Pod output rows are one-to-one with input rows, except rows whose function returned + nothing, which the pod skips (`function_pod.py:1047-1054`). +- User tags are not unique per row, so a tag join of the output back onto the input can + multiply rows. +- A literal `Join` of a stream with its own ancestor would extend and duplicate system-tag + names. + +**Proposed mechanism: a binary structural step `ReferenceFallThrough(names)(pod_output, +pod_input)`**, inserted by `_FunctionPodBase.__call__` beside ITL-807's automatic join and +select: +- **Row alignment by the full system tag tuple.** + - The pod passes each input row's tag through unchanged, system tags included + (`function_pod.py:1047-1054`: it yields the input `tag` object). + - The tuple of system-tag columns (`source_id`/`record_id` per input position) identifies + the input row. Source record ids are `uuid5(source_id::token)` and unique per source row + (`stream_builder.py:97-116`). A join pairs them, and `GroupBy`/`Batch` fold them into one + digest per group. + - The step matches output to input on that tuple. It needs no user tag, so duplicate user + tags cannot multiply rows. + - An input row the pod skipped has no output row and is dropped, as the pod dropped it. +- **System tags are name-preserving (rule 1).** Both inputs carry the same system-tag + columns, so the output keeps the pod output's columns unchanged. This is not a `Join`: + nothing is renamed or extended. +- **Data comes from data.** Each restored data element and its `_source_x` come from the + pod *input's* data column `x`. No data element comes from a tag value (ITL-955), and + field-level provenance survives. A unary "re-reference the plain tag column" step was + considered and rejected: it would turn a tag column into data (against ITL-955), and it + cannot restore `_source_x`, which only the input's `Data` holds. +- **Identity.** `("ReferenceFallThrough", sorted(names))`, with ordered arguments (output, + input). It is a new node in the graph. The pod node, its pipeline table and its results are + unchanged. A stream without references inserts no step, so no v0.3 graph or hash changes. +- **Storage.** As an operator node it writes an `odb_v1` table like any operator. It stores + `x` once, as the referential column. +- **Size: M**, sync plus async (barrier `async_execute` first). It belongs in ITL-1013, after + ITL-1016 and ITL-1017. + +**Test, required before building (duplicate user tags):** +- A source with rows `(session=s1, x=10, y=1)` and `(session=s1, x=20, y=2)`: user tag + `session` duplicated, `x` referential, `y` data. +- `pod(...)` with a function reading only `y` returns exactly two rows, with data + `x = [10, 20]` aligned to `y`. +- `pod.process(...)` (or `strict=True`) returns two rows with `x` a plain tag. +- The pod node's stored records equal those of the `process` form. +- The same holds after a `Join` of two sources (two record ids per row), on the sync and + async paths. --- diff --git a/superpowers/specs/itl-1004-audit/07_saved_job_versions.out b/superpowers/specs/itl-1004-audit/07_saved_job_versions.out new file mode 100644 index 000000000..77f4c7298 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/07_saved_job_versions.out @@ -0,0 +1,13 @@ +=== V1. Top-level keys of a saved job file + ['bindings', 'orcapod_pipeline_job_version', 'pipeline', 'run'] + +=== V2. Every field naming a version or a data context + orcapod_pipeline_job_version = "0.1.0" + pipeline.orcapod_pipeline_version = "0.1.0" + pipeline.nodes.semantic_v1:c472fc04f813764db9cb4d1567c03ce3ee70a48a47cdbcd0a165a01589d74596.data_context_key = "std:v1:default" + pipeline.nodes.semantic_v1:a55f98a143f589ff66e8430a3b9eb26d5beb703dc30a2b7c5267f9f89d3bdd29.data_context_key = "std:v1:default" + pipeline.nodes.semantic_v1:a55f98a143f589ff66e8430a3b9eb26d5beb703dc30a2b7c5267f9f89d3bdd29.function_config.data_function.config.version = "v0.0" + +=== V3. A source config inside the job (bindings.sources) + src: keys ['content_hash', 'data_schema', 'pipeline_hash', 'source_id', 'source_type', 'tag_columns', 'tag_schema'] + src.config keys: [] diff --git a/superpowers/specs/itl-1004-audit/07_saved_job_versions.py b/superpowers/specs/itl-1004-audit/07_saved_job_versions.py new file mode 100644 index 000000000..ea187cb00 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/07_saved_job_versions.py @@ -0,0 +1,60 @@ +"""ITL-1004 audit, measurement 07: the version information a saved job holds today. + +Saves a small job (a ``DictSource`` and one function pod, in-memory store) and prints every +field that could identify the Orcapod version that wrote it. + +Run: uv run python superpowers/specs/itl-1004-audit/07_saved_job_versions.py +Output: 07_saved_job_versions.out +""" + +from __future__ import annotations + +import json +import tempfile +from pathlib import Path +from typing import Any + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@function_pod(output_keys="doubled") +def double(v: int) -> int: + return v * 2 + + +def version_fields(obj: Any, path: str = "") -> list[str]: + """Every key path whose name mentions a version or a context, with its value.""" + found: list[str] = [] + if isinstance(obj, dict): + for key, value in obj.items(): + here = f"{path}.{key}" if path else str(key) + if any(w in str(key).lower() for w in ("version", "context")): + found.append(f"{here} = {json.dumps(value)[:80]}") + found.extend(version_fields(value, here)) + elif isinstance(obj, list): + for i, value in enumerate(obj): + found.extend(version_fields(value, f"{path}[{i}]")) + return found + + +job = PipelineJob(name="versions", store=InMemoryArrowDatabase()) +with job: + double.pod(DictSource(data=[{"k": 1, "v": 2}], tag_columns=["k"], source_id="src")) +_ = job.run() + +with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "job.json" + job.save(str(path)) + saved = json.loads(path.read_text()) + +print("=== V1. Top-level keys of a saved job file") +print(f" {sorted(saved)}") +print("\n=== V2. Every field naming a version or a data context") +for line in version_fields(saved): + print(f" {line}") +print("\n=== V3. A source config inside the job (bindings.sources)") +for name, config in saved["bindings"]["sources"].items(): + print(f" {name}: keys {sorted(config)}") + print(f" {name}.config keys: {sorted(config.get('config', {}))}") From 5b6bd2dfe9173246a15d41a05a8b83899d7ff01d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:29:48 -0800 Subject: [PATCH 8/9] docs(itl-1004): describe the fall-through as a variant of the function pod Developer ruling: the fall-through of referential tags on pod(...) is a per-row variant of the function pod that abbreviates a legal composition (temporary row tag, pod, join back, drop), not a join or inserted operator. Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 142 ++++++++++-------- 1 file changed, 79 insertions(+), 63 deletions(-) diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index 3a74a5f33..22d2f9b8a 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -15,9 +15,10 @@ spec marks Open or that the code makes impossible or ambiguous. stream changes (§3). - The change is a change of default behaviour. - `Join`'s three cases (§9.1) have equal standing. -- Pod boundary (D1). With `pod.process(...)`, and `pod(..., strict=True)`, a reference whose - data column does not survive becomes a plain tag. With `pod(...)`, an automatic step around - the pod gives the input's references back their data role (§11). Full append is deferred +- Pod boundary (D1). With `pod.process(...)` and `pod(..., strict=True)`, a reference whose + data column does not survive becomes a plain tag. `pod(...)` uses a **variant of the function + pod** that emits each row's referenced columns beside the function's outputs, so they stay + referential. It is not a join and not an inserted operator (§11). Full append is deferred (ITL-1025). - `GroupBy` follows spec §8 unchanged; `Join` gets no special rule for list-valued tags. - Saved jobs carry version information that pins behaviour (§4). @@ -497,8 +498,8 @@ Functions that read the copy recompute once, because the column name is in the ` - **Tags pass through** (`function_pod.py:739-747`). The output data is only the function's outputs (`:747`), so a reference's target never survives a pod, whether or not it was read. Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. Decided (D1): with - `process` the tag becomes plain at the boundary; with `pod(...)` an automatic step gives the - references back their data role (§11). + `process` the tag becomes plain at the boundary; `pod(...)` uses the fall-through variant of + the pod, and the references stay referential (§11). - **An output name that collides with a tag (§6.2, Required).** - Today `output_schema` accepts it (`01 R5`: `(Schema({'x': int | None}), Schema({'x': float}))`), and reading fails with @@ -541,7 +542,7 @@ The project has ITL-1009 to ITL-1023. Recommended order, with changes to the iss | 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | | 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | | 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | -| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule (plain under `process`); automatic join and select with references; the `ReferenceFallThrough` step on `pod(...)` (§11, after its duplicate-tag test is agreed); §15.2 and §15.6 result-identity tests | M+M | 1, 2b; 2c and 2d for the automatic join and the fall-through | 4 | +| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule (plain under `process`); automatic join and select with references; the fall-through variant of the pod on `pod(...)` (§11) with its equivalence test; §15.2 and §15.6 result-identity tests | M+M | 1, 2b; 2c and 2d for the automatic join and the fall-through | 4 | | 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | | 4b | **ITL-1012, part 2 (its last step), with ITL-1011** | **flip the default** to referential; golden fixtures switched to `tag_only_columns`, new entries; upgrade note | S | **1013, 1015, 1016, 1017, 1019** | — | | 5 | **ITL-1018** | `MergeJoin` (D4); the decided rename rules (§5.4), unless the ITL-1023 slice lands them | M | 1 | 6 | @@ -609,10 +610,11 @@ retains the input data `x`"), and in Orcapod it never does. The decision: does not survive becomes a **plain tag** (same value and type), visible in the output schema. No automation. The `strict` reading is the coordinator's; the developer may correct it. -- **`pod(stream)`:** an automatic step **around** the pod, of the ITL-807 family and visible - in the graph, gives the input's referential keys their data role back. The pod node, what - it stores and function-result identity are unchanged. Only referential key columns fall - through; full append is deferred (ITL-1025). The mechanism is proposed in §11. +- **`pod(stream)`:** a **variant of the function pod** (or a thin wrapper) processes one + (tag, data) pair at a time. It calls the function exactly as the plain pod does, then emits + the tag unchanged and, as data, the outputs plus the row's own referenced columns. Function + results are shared with the plain pod. Only referential key columns fall through; full + append is deferred (ITL-1025). See §11. - **The diagnostic** is the coordinator's call. Recommended none at the pod boundary or in the automatic select: the `reference_materialization` warning is for a select or drop the user wrote (a rename never warns, §5.4). @@ -680,59 +682,73 @@ whose same-named tags mean different things must be renamed. What `Join` does to shared list-valued tag (correct match, wrong match, or a join-engine error, sync and async) is for the ITL-1023 worker to measure and report. -### A. Pod boundary: references fall through on `pod(...)` only - -What the code must do (D1): -- `process` and `strict`: the tag becomes plain. -- `pod(...)`: an automatic step after the pod restores the data role of the input's - referential keys. - -**Correctness constraints for that step:** -- Pod output rows are one-to-one with input rows, except rows whose function returned - nothing, which the pod skips (`function_pod.py:1047-1054`). -- User tags are not unique per row, so a tag join of the output back onto the input can - multiply rows. -- A literal `Join` of a stream with its own ancestor would extend and duplicate system-tag - names. - -**Proposed mechanism: a binary structural step `ReferenceFallThrough(names)(pod_output, -pod_input)`**, inserted by `_FunctionPodBase.__call__` beside ITL-807's automatic join and -select: -- **Row alignment by the full system tag tuple.** - - The pod passes each input row's tag through unchanged, system tags included - (`function_pod.py:1047-1054`: it yields the input `tag` object). - - The tuple of system-tag columns (`source_id`/`record_id` per input position) identifies - the input row. Source record ids are `uuid5(source_id::token)` and unique per source row - (`stream_builder.py:97-116`). A join pairs them, and `GroupBy`/`Batch` fold them into one - digest per group. - - The step matches output to input on that tuple. It needs no user tag, so duplicate user - tags cannot multiply rows. - - An input row the pod skipped has no output row and is dropped, as the pod dropped it. -- **System tags are name-preserving (rule 1).** Both inputs carry the same system-tag - columns, so the output keeps the pod output's columns unchanged. This is not a `Join`: - nothing is renamed or extended. -- **Data comes from data.** Each restored data element and its `_source_x` come from the - pod *input's* data column `x`. No data element comes from a tag value (ITL-955), and - field-level provenance survives. A unary "re-reference the plain tag column" step was - considered and rejected: it would turn a tag column into data (against ITL-955), and it - cannot restore `_source_x`, which only the input's `Data` holds. -- **Identity.** `("ReferenceFallThrough", sorted(names))`, with ordered arguments (output, - input). It is a new node in the graph. The pod node, its pipeline table and its results are - unchanged. A stream without references inserts no step, so no v0.3 graph or hash changes. -- **Storage.** As an operator node it writes an `odb_v1` table like any operator. It stores - `x` once, as the referential column. -- **Size: M**, sync plus async (barrier `async_execute` first). It belongs in ITL-1013, after - ITL-1016 and ITL-1017. - -**Test, required before building (duplicate user tags):** -- A source with rows `(session=s1, x=10, y=1)` and `(session=s1, x=20, y=2)`: user tag - `session` duplicated, `x` referential, `y` data. -- `pod(...)` with a function reading only `y` returns exactly two rows, with data - `x = [10, 20]` aligned to `y`. -- `pod.process(...)` (or `strict=True`) returns two rows with `x` a plain tag. -- The pod node's stored records equal those of the `process` form. -- The same holds after a `Join` of two sources (two record ids per row), on the sync and - async paths. +### A. Pod boundary: references fall through on `pod(...)`, as a variant of the pod + +The developer's ruling: the fall-through is an abbreviated form of a legal composition, done +by a variant of the function pod. It is **not** a join and **not** an inserted operator. + +**The composition it abbreviates** (each step obeys the boundary: no tag enters a function; +no operator makes data that is not existing data): +1. Add a unique temporary tag per row. +2. Send one copy through the plain pod. The automatic select makes the unread references + plain; the read ones are consumed, and either way the pod's output has them as plain tags. +3. Join the pod's output back onto the input, restricted to its tags and referenced columns, + on the temporary tag. This is `Join` case 2, plain with referential (§5.1), so the + referenced data comes from the input's data column. +4. Drop the temporary tag. + +**The variant, one (tag, data) pair at a time** (`pod(stream)`, the `__call__` path that +already adds ITL-807's automatic join and select): +- **The function sees exactly what the plain pod gives it.** The variant projects the row's + `Data` to the function's parameters and runs the plain pod's per-row path on that + projection. The memo key (`result_cache.py:217`) and the result store (`rdb_v2` at the + function uri) are therefore the plain pod's, so results are shared. +- **What it emits.** The input tag unchanged (system tags included, as the plain pod does, + `function_pod.py:1043-1054`). As data: the function's outputs plus the row's referenced + columns, with their `_source_*` taken from the row's `Data`. +- **What carries the referenced columns.** ITL-807's automatic select must keep referenced + columns in the variant's input. Only non-referenced extra columns are dropped. +- **Rows.** Row count and order are the pod's: a row whose function returned nothing is + skipped, as in the plain pod. There is no real join, no extended system-tag name, and no + row multiplication from duplicate user tags. + +**Consequences:** +- **Output schema.** Data = outputs ∪ referenced input columns. Tags = input tags. The + references of the output = those of the input. +- **A function output named like a referenced column (or any tag)** still raises (§7). +- **Pipeline identity.** It differs from the plain pod's **only when references are + present**. Add the term `("reference_fall_through", sorted(names))` to the pod stream's or + node's identity only when the set is non-empty (the #336/#338 pattern). With no referential + tag in the input, the variant is indistinguishable from the plain pod: same schema, same + hashes, same node path and stored tables, no golden move. +- **Function-result identity and the result store.** Shared with the plain pod. A retagged + pipeline, or a `process` call over the same rows, finds the same results. +- **`FunctionNode` and the job path.** + - The pipeline record (`pdb_v2`, `function_node.py:2337-2461`) stores the row's tags, + which include `x`, plus the input's `_source_*` columns (`:2390-2396`). + - For the variant those must be the `_source_*` of the full input `Data`, referenced + columns included, not only the projection's, so `_source_x` is kept. + - The value of `x` is stored once, as the tag column. On read the node gives that column + its data role back (`reference_tags` = the input's references, `function_node.py:2891`). + - Nothing is stored twice, and the table is at its own `pipeline_hash` path whenever + references are present. +- **`strict=True`.** It uses the plain pod, as `process` does (the coordinator's reading of + ITL-807; the developer may correct it). +- **Size: M**, in ITL-1013, after ITL-1016, because its equivalence test uses `Join` case 2. + +**Tests:** +- **Equivalence.** On a source with duplicate user tags (`session=s1` on two rows, `x` + referential, `y` data), compare `pod(source)` with the composition above, built with a + temporary tag-only row-id column, `pod.process`, `Join` and `DropTagColumns`. User tags, + data, references and rows must match. System-tag *names* differ by construction (the + composition's `Join` extends them), so compare those separately: the variant keeps the + pod's names. +- **No references.** `pod(source)` over a tag-only source equals today's `pod(source)` in + schema, `pipeline_hash`, `content_hash`, node path and stored tables. +- **Result sharing.** Run `pod(source)` and then `pod.process(source)` (or the reverse) over + one result store. The second run computes nothing. +- **Job path.** Saved, reloaded and read back, the variant node gives the same table with + `x` referential. Both sync and async. --- From 83b3d2a080a70b7be563813d39d93811f7084c16 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:34:12 -0800 Subject: [PATCH 9/9] docs(itl-1004): format versions: v0.3.0 writes 0.2.0, referential default 0.3.0 Co-Authored-By: Claude Opus 5.5 --- ...4-referential-tags-implementation-scope.md | 37 +++++++++---------- 1 file changed, 18 insertions(+), 19 deletions(-) diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md index 22d2f9b8a..fe1f79a50 100644 --- a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -261,18 +261,19 @@ and raise, unless the function takes all keys. That is decision D2. none. Both format versions were already `"0.1.0"` at v0.2.0, so they were not bumped across the v0.2-to-v0.3 hash changes. The context key will not change in v0.4 (§3.2), so it cannot tell v0.3 from v0.4. -- **The rule (decision D8):** - - v0.4 bumps both format versions in the change that flips the source default (the last - step of ITL-1012). - - Loading a `"0.1.0"` file pins v0.3 meaning: every declared tag tag-only. The saved - content and pipeline hashes then match. +- **The rule (D8, decided by the developer).** + - v0.3.0 itself (ITL-1026) bumps `PIPELINE_JOB_FORMAT_VERSION` and + `PIPELINE_FORMAT_VERSION` from `"0.1.0"` to `"0.2.0"`, and writes an informational + `orcapod_version` field. + - The referential-tag default (the last step of ITL-1012) bumps both to **`"0.3.0"`**, the + first format in which a source may be referential. + - Files at `"0.1.0"` and `"0.2.0"` load with v0.3 meaning: every declared source tag is + tag-only, and the saved content and pipeline hashes match. + - v0.4 accepts `"0.1.0"`, `"0.2.0"` and `"0.3.0"` on load. The ITL-1023 integration branch + writes `"0.3.0"` already. - A standalone source config (a `to_config()` dict outside a job or pipeline file) carries - no version, so v0.4 adds a `config_version` to source configs if those must reload - across versions. - - Sent to the coordinator before the v0.3.0 tag: nothing is strictly required in v0.3.0, - but writing `orcapod_version` (the package version) beside both format versions costs - about two lines and no hash. It lets v0.4 tell v0.3.0 files from files written by the - ITL-1023 integration branch, which uses format `"0.1.0"` with referential tags. + no version. If such configs must reload across versions, v0.4 adds a `config_version` + to them. **Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as plain tags only (§16.1). Nothing is reinterpreted. @@ -660,14 +661,12 @@ needs a shape).** `tag_only_columns=[...]`, a subset of `tag_columns` (§6.2). Open for the ITL-1012 worker to propose: a shorthand for "all declared tags", useful for database sources whose tags default to primary keys. -**D8. Version pinning of saved files (decided in principle by the developer; the mechanism -needs approval).** -- (a) **Recommended.** v0.4 bumps `PIPELINE_JOB_FORMAT_VERSION` and `PIPELINE_FORMAT_VERSION` - with the default flip, and `"0.1.0"` loads with v0.3 tag meaning. v0.3.0 adds - `orcapod_version` before its tag (informational). v0.4 adds `config_version` to standalone - source configs. -- (b) Rely on the format-version bump alone. Files from the ITL-1023 integration branch - (format `"0.1.0"`, referential tags) would then be misread as v0.3. +**D8. Version pinning of saved files: DECIDED** (developer, November 12, 2026; §4). +- v0.3.0 writes format `"0.2.0"` and `orcapod_version` (ITL-1026). +- The referential default writes `"0.3.0"`. +- `"0.1.0"` and `"0.2.0"` load with v0.3 tag meaning. +- Open: a `config_version` on standalone source configs, if they must reload across + versions. ---