Skip to content
63 changes: 63 additions & 0 deletions superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# ITL-1004 referential tags: implementation-scope audit (plan)

> **For agentic workers:** this plan produces a design document, not library code. No change under
> `src/` or `tests/`, no golden file.

**Goal:** One document, `superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md`,
that makes the v0.4 referential-tags specification (Linear document attached to ITL-1004)
dispatchable: where the code assumes tag/data disjointness, which hashes would see a binding,
what is stored, operator-by-operator rules and change sizes, sources, function pods, answers to
the specification's open questions, a sliced delivery plan, and the decisions for the developer.

**Architecture:** read the code; measure today's behaviour with small scripts under
`superpowers/specs/itl-1004-audit/` (run with `uv run`, outputs saved beside them); write the
document from the evidence.

**Branch:** `eywalker/itl-1004-referential-tags-v04-a-tag-may-refer-to` from `origin/main`
(`f51ce67c`). PR: docs only, `Refs ITL-1004`.

---

## Task 1: Representation sites
- [ ] Read `core/streams/arrow_table_stream.py`, `core/datagrams/*`, `types.py` (Schema, ColumnConfig),
`utils/schema_utils.py`, `core/streams/base.py`.
- [ ] grep every `keys()` / `output_schema` / `identity_structure` / `pipeline_identity_structure`
definition; classify each site as works / change / reject.
- [ ] Script `01_representation.py`: build an `ArrowTableStream` with a column in both tag and
data lists (if constructible) and record what each public method returns.

## Task 2: Identity and hashes
- [ ] Read `core/base.py`, `core/sources/base.py`, `core/nodes/function_node.py`
(`_build_record_id_preimage`), `core/function_pod.py`, `core/datagrams` content hash,
`derived_source.py` (PR #336/#338 pattern).
- [ ] Script `02_identity.py`: show which hash inputs contain the tag/data split.

## Task 3: Storage
- [ ] Read FunctionNode / OperatorNode / CachedSource / DerivedSource write and read paths.
- [ ] Script `03_storage.py`: stored columns of each table kind for a small pipeline.

## Task 4: Operators
- [ ] For every file in `core/operators` and the FunctionPod call path, measure today's behaviour
with a same-named tag and data column (`04_operators.py`), sync and async.
- [ ] Join source-info of merged column: options.

## Task 5–6: Sources and function pods
- [ ] Read `arrow_table_source.py`, `stream_builder.py`, `spiraldb_table_source.py`,
`db_table_source.py`, `derived_source.py`; `function_pod.py` output collision (`05_function_pod.py`).

## Task 7–9: Open questions, delivery plan, decisions
- [ ] Answer section 17 from code; slice plan with sizes, dependencies, tests; decisions list.

## Task 10: Finish
- [ ] Doc under ~900 lines; ruff on audit scripts; `uv run pytest tests/test_docs` not affected
(superpowers is not a docs page); merge origin/main; push; PR `Refs ITL-1004`;
`orca linear attach`; `orca worktree set --pr`; worker_done.

## Progress notes
(updated as work proceeds, for a successor)
- 2026-11-13: Tasks 1-9 done. Measurements 01-06 in superpowers/specs/itl-1004-audit/ (outputs .out).
Coordinator redirects applied (default referential sources; plain-tag mode = v0.3 exactly;
three Join cases equal; decisions only for Open/code-impossible points; slices mapped onto
ITL-1009..1023). Representation + hash verdict sent to the coordinator (msg_a8cdbe370954).
Pod-boundary rule (D1) asked; coordinator: keep (a) recommended, put to developer.
Full suite: 11029 passed, 117 skipped, 23 xfailed. Remaining: PR, linear attach, worker_done.

Large diffs are not rendered by default.

81 changes: 81 additions & 0 deletions superpowers/specs/itl-1004-audit/01_today_same_name.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
=== R1. A stream cannot name one column as both tag and data
--- ArrowTableSource(x, y; tag_columns=[x])
keys: tags=('x',) data=('y',)
schema: tags={'x': int | None} data={'y': float | None}
as_table columns: ['x', 'y']
rows: [({'x': 1}, {'y': 0.1}), ({'x': 2}, {'y': 0.2})]
--- ArrowTableStream over a table with two physical columns named x, tag_columns=[x]
RAISES KeyError: 'Field "x" exists 2 times in schema'
(raised at pyarrow/table.pxi:1735)

=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape)
--- Join().output_schema(A, B)
(Schema({'k': int | None, 'x': int | None}), Schema({'x': int | None, 'b': float | None, 'a': float | None}))
--- Join()(A, B) read
RAISES KeyError: 'Field "x" exists 2 times in schema'
(raised at pyarrow/table.pxi:1735)
--- Join async rows
[({'k': 1, 'x': 1}, {'x': 1, 'b': 0.5, 'a': 0.1}), ({'k': 2, 'x': 1}, {'x': 2, 'b': 0.6, 'a': 0.1}), ({'k': 1, 'x': 2}, {'x': 1, 'b': 0.5, 'a': 0.2}), ({'k': 2, 'x': 2}, {'x': 2, 'b': 0.6, 'a': 0.2})]
--- MergeJoin()(A, B) read
RAISES KeyError: 'Field "x" exists 2 times in schema'
(raised at pyarrow/table.pxi:1735)
--- SemiJoin()(A, B) read
keys: tags=('x',) data=('a',)
schema: tags={'x': int | None} data={'a': float | None}
as_table columns: ['x', 'a']
rows: [({'x': 1}, {'a': 0.1}), ({'x': 2}, {'a': 0.2})]

=== R3. Join: both inputs have DATA x (the collision)
--- Join().output_schema(C, D)
RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'v': float | None})) found in ArrowTableSource and ArrowTableSource.
(raised at orcapod/core/operators/join.py:116)
--- Join async rows
RAISES ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
(raised at /Users/eywalker/.local/share/uv/python/cpython-3.12.15-macos-aarch64-none/lib/python3.12/asyncio/taskgroups.py:164)
--- MergeJoin()(C, D) read (colliding data -> list)
keys: tags=('u',) data=('x', 'v')
schema: tags={'u': int | None} data={'x': list[int | None], 'v': float | None}
as_table columns: ['u', 'x', 'v']
rows: [({'u': 5}, {'x': [1, 1], 'v': 1.0}), ({'u': 6}, {'x': [2, 2], 'v': 2.0})]

=== R4. MapTags / MapData renaming onto a name of the other namespace
--- MapTags({x: y})(E)
RAISES InputValidationError: Renaming {'x': 'y'} would cause collisions with existing columns: overlapping data columns: {'y'}.
(raised at orcapod/core/operators/mappers.py:254)
--- MapData({y: x})(E)
RAISES InputValidationError: Renaming {'y': 'x'} would cause collisions with existing columns: overlapping tag columns: {'x'}.
(raised at orcapod/core/operators/mappers.py:104)

=== R5. Function pod output named like an input TAG
--- make_x.pod.output_schema(E)
(Schema({'x': int | None}), Schema({'x': float}))
--- make_x.pod(E) read
RAISES ValueError: Duplicate column name 'x' found in input tables.
(raised at orcapod/utils/arrow_utils.py:682)

=== R6. DerivedSource(tags_as_data={x: x}) (same name)
--- DerivedSource(E, tags_as_data={'x': 'x'})
RAISES InputValidationError: tags_as_data: cannot copy tag 'x' to 'x', which is already a column. Choose a new name.
(raised at orcapod/core/sources/derived_source.py:99)
--- DerivedSource(E, tags_as_data={'x': 'x_copy'})
keys: tags=('x',) data=('y', 'x_copy')
schema: tags={'x': int | None} data={'y': float | None, 'x_copy': int | None}
as_table columns: ['x', 'y', 'x_copy']
rows: [({'x': 1}, {'y': 0.1, 'x_copy': 1}), ({'x': 2}, {'y': 0.2, 'x_copy': 2})]

=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x)
--- GroupBy(by=[t_unit])(G)
keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp')
schema: tags={'t_unit': int | None, 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]}
as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp']
rows: [({'t_unit': 1, 't_time': [10, 30]}, {'unit': [1, 1], 'time': [10, 30], 'amp': [0.1, 0.3]}), ({'t_unit': 2, 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})]
--- Batch(batch_size=2)(G)
keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp')
schema: tags={'t_unit': list[int | None], 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]}
as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp']
rows: [({'t_unit': [1, 1], 't_time': [30, 10]}, {'unit': [1, 1], 'time': [30, 10], 'amp': [0.3, 0.1]}), ({'t_unit': [2], 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})]

=== R8. Function pod with a tag name as a parameter (tags are not inputs)
--- uses_tag.pod(E) (x is a tag of E)
RAISES ValueError: Incoming data schema Schema({'y': float | None}) is not compatible with expected input schema Schema({'x': int | None}): column 'y' is not a parameter of the function
(raised at orcapod/core/function_pod.py:259)
106 changes: 106 additions & 0 deletions superpowers/specs/itl-1004-audit/01_today_same_name.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
"""ITL-1004 audit, measurement 01: what v0.3 code does today when a tag and a data column
share a name, anywhere it can be built through the public interface.

Run: uv run python superpowers/specs/itl-1004-audit/01_today_same_name.py
Output: 01_today_same_name.out
"""

from __future__ import annotations

import pyarrow as pa
from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder

from orcapod import function_pod
from orcapod.core.streams import ArrowTableStream
from orcapod.operators import (
Batch,
GroupBy,
Join,
MapData,
MapTags,
MergeJoin,
SemiJoin,
)
from orcapod.sources import DerivedSource

print("=== R1. A stream cannot name one column as both tag and data")
one = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "one")
show("ArrowTableSource(x, y; tag_columns=[x])", lambda: describe(one))

dup = pa.table([i64(1, 2), i64(1, 2), f64(0.1, 0.2)], names=["x", "x", "y"])
show(
"ArrowTableStream over a table with two physical columns named x, tag_columns=[x]",
lambda: describe(ArrowTableStream(dup, tag_columns=["x"])),
)

print("\n=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape)")
a = src({"x": i64(1, 2), "a": f64(0.1, 0.2)}, ["x"], "A")
b = src({"k": i64(1, 2), "x": i64(1, 2), "b": f64(0.5, 0.6)}, ["k"], "B")
show("Join().output_schema(A, B)", lambda: Join().output_schema(a, b))
show("Join()(A, B) read", lambda: describe(Join()(a, b)))
show("Join async rows", lambda: async_rows(Join(), a, b))
show("MergeJoin()(A, B) read", lambda: describe(MergeJoin()(a, b)))
show("SemiJoin()(A, B) read", lambda: describe(SemiJoin()(a, b)))

print("\n=== R3. Join: both inputs have DATA x (the collision)")
c = src({"x": i64(1, 2), "u": i64(5, 6)}, ["u"], "C")
d = src({"x": i64(1, 2), "u": i64(5, 6), "v": f64(1.0, 2.0)}, ["u"], "D")
show("Join().output_schema(C, D)", lambda: Join().output_schema(c, d))
show("Join async rows", lambda: async_rows(Join(), c, d))
show(
"MergeJoin()(C, D) read (colliding data -> list)",
lambda: describe(MergeJoin()(c, d)),
)

print("\n=== R4. MapTags / MapData renaming onto a name of the other namespace")
e = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "E")
show("MapTags({x: y})(E)", lambda: describe(MapTags({"x": "y"})(e)))
show("MapData({y: x})(E)", lambda: describe(MapData({"y": "x"})(e)))


print("\n=== R5. Function pod output named like an input TAG")


@function_pod(output_keys="x")
def make_x(y: float | None) -> float:
return (y or 0.0) * 2


show("make_x.pod.output_schema(E)", lambda: make_x.pod.output_schema(e))
show("make_x.pod(E) read", lambda: describe(make_x.pod(e)))

print("\n=== R6. DerivedSource(tags_as_data={x: x}) (same name)")
show(
"DerivedSource(E, tags_as_data={'x': 'x'})",
lambda: describe(DerivedSource(e, tags_as_data={"x": "x"})),
)
show(
"DerivedSource(E, tags_as_data={'x': 'x_copy'})",
lambda: describe(DerivedSource(e, tags_as_data={"x": "x_copy"})),
)

print("\n=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x)")
g = src(
{
"t_unit": i64(1, 1, 2),
"t_time": i64(30, 10, 20),
"unit": i64(1, 1, 2),
"time": i64(30, 10, 20),
"amp": f64(0.3, 0.1, 0.2),
},
["t_unit", "t_time"],
"G",
)
show("GroupBy(by=[t_unit])(G)", lambda: describe(GroupBy(by=["t_unit"])(g)))
show("Batch(batch_size=2)(G)", lambda: describe(Batch(batch_size=2)(g)))

print("\n=== R8. Function pod with a tag name as a parameter (tags are not inputs)")


@function_pod(output_keys="z")
def uses_tag(x: int | None) -> int | None:
return x


show("uses_tag.pod(E) (x is a tag of E)", lambda: describe(uses_tag.pod(e)))
_ = s
Loading
Loading