From 44927e90030ce5e66649e8790e0be5b1377bf4a2 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 16:15:35 -0800 Subject: [PATCH 01/14] docs(itl-939): spike plan and experiments on today's behaviour Co-Authored-By: Claude Opus 5.5 --- ...11-itl-939-tag-and-data-same-name-spike.md | 72 +++++++ .../exp01_same_name_today.py | 186 +++++++++++++++++ .../exp02_workarounds_today.py | 189 ++++++++++++++++++ 3 files changed, 447 insertions(+) create mode 100644 superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md create mode 100644 superpowers/specs/itl-939-experiments/exp01_same_name_today.py create mode 100644 superpowers/specs/itl-939-experiments/exp02_workarounds_today.py diff --git a/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md b/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md new file mode 100644 index 000000000..b98ec3c60 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md @@ -0,0 +1,72 @@ +# ITL-939 design spike plan: a tag and a data column sharing a name + +> **For agentic workers:** this is a design spike. No library code changes. Execute inline, +> task by task; tick the boxes and commit after each task so a successor can continue. + +**Goal:** a design document (`superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md`) +that establishes, by running code, what Orcapod does today when a value is wanted both as a tag +and as data, compares four designs, and recommends the smallest safe feature. + +**Architecture:** throwaway experiment scripts under `superpowers/specs/itl-939-experiments/` +drive only the public API (sources, stream methods, operators, `function_pod`) on a small +EDI-shaped table (`session`, `experiment`, keys `subject_id`, `session_start_datetime`, +`experiment_index`). Each script prints what it observes; the outputs are pasted into the doc. + +**Tech Stack:** orcapod (this repo), pyarrow, `uv run`. + +--- + +### Task 1: Experiment 01, what happens today with a shared name + +**Files:** Create `superpowers/specs/itl-939-experiments/exp01_same_name_today.py` + +- [ ] Build an Arrow table with duplicate field names (`session_start_datetime` twice) and pass + it to `ArrowTableSource` with the name in `tag_columns`; record the error or behaviour. +- [ ] `MapData({"x": ""})` and `MapTags({"": ""})` collisions. +- [ ] `Join` of two streams where one stream's tag name equals the other's data name. +- [ ] `function_pod` whose output key equals a tag name. +- [ ] Run with `uv run python ...`; save stdout to `exp01_same_name_today.out.txt`. + +### Task 2: Experiment 02, today's workarounds on the EDI-shaped table + +**Files:** Create `superpowers/specs/itl-939-experiments/exp02_workarounds_today.py` + +- [ ] W1 choose `tag_columns` so the column stays data. +- [ ] W2 duplicate the column under another name before building the source. +- [ ] W3 read the source twice (once with the column as data), join. +- [ ] W4 anything else found (e.g. `MapTags` cannot copy; `PolarsFilter` sees both). +- [ ] For each: compute elapsed time (`experiment_start - session_start_datetime`) with a pod, + then Join with a second table keyed on the same columns; record lines, renames, + `pipeline_hash` / `content_hash` changes and downstream Join behaviour. +- [ ] Save stdout to `exp02_workarounds_today.out.txt`. + +### Task 3: Experiment 03, prototype of the smallest feature (design C), out of tree + +**Files:** Create `superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py` + +- [ ] A throwaway subclass of `UnaryOperator` defined in the script (not in `src/`) that copies + named tag columns into the data group under chosen names, with source-info for the copy. +- [ ] Show `keys()`, `output_schema()`, `as_table()`, `iter_data()`, a pod reading the copy, + a downstream Join keyed on the original tags, and that streams not using it keep their + hashes. +- [ ] Save stdout to `exp03_copy_tag_prototype.out.txt`. + +### Task 4: Need analysis from the EDI schemas + +- [ ] List key columns that are plausibly computation inputs; 3-5 workshop computations. + +### Task 5: Design document + +- [ ] Sections: today (1), need (2), designs A-D table + prose (3), recommendation and numbered + decisions (4). Commit and push WIP. + +### Task 6: Report and PR + +- [ ] `ask` the coordinator with the summary; wait for the reply. +- [ ] Append the DESIGN_ISSUES.md entry for ITL-939 at the end. +- [ ] Merge origin/main, push, open docs-only PR ("Part of ITL-939"), attach to Linear, + `orca worktree set --pr`. CodeRabbit rounds. `worker_done`. + +## Progress notes + +(successor: read this section first) diff --git a/superpowers/specs/itl-939-experiments/exp01_same_name_today.py b/superpowers/specs/itl-939-experiments/exp01_same_name_today.py new file mode 100644 index 000000000..46fd07e73 --- /dev/null +++ b/superpowers/specs/itl-939-experiments/exp01_same_name_today.py @@ -0,0 +1,186 @@ +"""ITL-939 experiment 01: what Orcapod does TODAY when a tag and a data column share a name. + +Throwaway spike script; public interface only. Run: + + uv run python superpowers/specs/itl-939-experiments/exp01_same_name_today.py + +Every probe prints OK (with what it saw) or the exception type and message. +""" + +from __future__ import annotations + +import datetime as dt +import traceback + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.operators import Join, MapData, MapTags, MergeJoin, SemiJoin +from orcapod.sources import ArrowTableSource + +US = pa.timestamp("us", tz="UTC") +T0 = dt.datetime(2026, 1, 5, 9, 0, 0, tzinfo=dt.timezone.utc) +T1 = dt.datetime(2026, 1, 6, 9, 30, 0, tzinfo=dt.timezone.utc) + + +def probe(label: str, fn) -> None: + print(f"\n--- {label}") + try: + out = fn() + print("OK:", out) + except Exception as exc: # noqa: BLE001 - a probe reports every failure + print(f"RAISES {type(exc).__name__}: {exc}") + frames = traceback.extract_tb(exc.__traceback__) + inner = [f for f in frames if "orcapod" in f.filename] + if inner: + f = inner[-1] + print(f" raised at {f.filename.split('src/')[-1]}:{f.lineno} ({f.name})") + + +def session_table() -> pa.Table: + return pa.table( + { + "subject_id": pa.array(["m1", "m2"], pa.large_string()), + "session_start_datetime": pa.array([T0, T1], US), + "session_end_datetime": pa.array( + [T0 + dt.timedelta(hours=2), T1 + dt.timedelta(hours=1)], US + ), + } + ) + + +# 1. An Arrow table that physically carries the name twice (Arrow allows duplicate +# field names), with the name declared as a tag. +def dup_name_source(): + t = session_table() + dup = pa.Table.from_arrays( + [t["subject_id"], t["session_start_datetime"], t["session_start_datetime"]], + names=["subject_id", "session_start_datetime", "session_start_datetime"], + ) + src = ArrowTableSource( + dup, tag_columns=["subject_id", "session_start_datetime"], source_id="session", infer_nullable=True + ) + return src.keys() + + +probe("1. ArrowTableSource on a table with a duplicated field name", dup_name_source) + + +# 2. tag_columns naming a column -> it simply leaves the data group. +def baseline(): + src = ArrowTableSource( + session_table(), + tag_columns=["subject_id", "session_start_datetime"], + source_id="session", + ) + return src.keys() + + +probe("2. Baseline: a key column is a tag and nothing else", baseline) + +base = ArrowTableSource( + session_table(), + tag_columns=["subject_id", "session_start_datetime"], + source_id="session", + infer_nullable=True, +) + +# 3. MapData renaming a data column onto a tag name. +probe( + "3. MapData renames data 'session_end_datetime' -> tag name 'session_start_datetime'", + lambda: MapData({"session_end_datetime": "session_start_datetime"})(base).keys(), +) + +# 4. MapTags renaming a tag onto a data name. +probe( + "4. MapTags renames tag 'session_start_datetime' -> data name 'session_end_datetime'", + lambda: MapTags({"session_start_datetime": "session_end_datetime"})(base).keys(), +) + + +# 5. A function pod whose output key equals a tag name. +@function_pod(output_keys="session_start_datetime") +def echo_end(session_end_datetime: dt.datetime) -> dt.datetime: + return session_end_datetime + + +probe( + "5. function pod output key == tag name: building the stream", + lambda: echo_end.pod(base).keys(), +) +probe( + "5b. ... and materialising it (as_table)", + lambda: echo_end.pod(base).as_table().column_names, +) + + +# 6. Join: stream A has the name as a tag, stream B has the same name as data. +def b_stream(): + t = pa.table( + { + "subject_id": pa.array(["m1", "m2"], pa.large_string()), + "session_start_datetime": pa.array([T0, T1], US), + "weight_g": pa.array([25.0, 27.5], pa.float64()), + } + ) + return ArrowTableSource( + t, tag_columns=["subject_id"], source_id="weights", infer_nullable=True + ) + + +probe( + "6. Join(A[tag: session_start_datetime], B[data: session_start_datetime]) keys", + lambda: Join()(base, b_stream()).keys(), +) +probe( + "6b. ... and materialising it", + lambda: Join()(base, b_stream()).as_table().column_names, +) +probe( + "7. MergeJoin(A, B) with the same shapes", + lambda: MergeJoin()(base, b_stream()).keys(), +) +probe( + "7b. ... and materialising it", + lambda: MergeJoin()(base, b_stream()).as_table().column_names, +) +probe( + "8. SemiJoin(A, B) with the same shapes", + lambda: SemiJoin()(base, b_stream()).keys(), +) +probe( + "8b. ... and materialising it", + lambda: SemiJoin()(base, b_stream()).as_table().column_names, +) + + +# 9. A pod parameter named after a tag: can a pod read a tag today? +@function_pod(output_keys="elapsed_s") +def elapsed(session_start_datetime: dt.datetime, session_end_datetime: dt.datetime) -> float: + return (session_end_datetime - session_start_datetime).total_seconds() + + +probe( + "9. pod with a parameter named after a TAG column", + lambda: elapsed.pod(base).keys(), +) + + +# 10. Per-row view: Tag and Data are separate datagrams, so a shared name may survive +# iter_data() even though as_table() cannot hold it. +def iter_pod_rows(): + return [(t.as_dict(), d.as_dict()) for t, d in echo_end.pod(base).iter_data()] + + +probe("10. iter_data() of the pod whose output key == tag name", iter_pod_rows) + + +def iter_join_rows(): + return [(t.as_dict(), d.as_dict()) for t, d in Join()(base, b_stream()).iter_data()] + + +probe("10b. iter_data() of Join(A[tag], B[data]) sharing the name", iter_join_rows) +probe( + "10c. output_schema() of that Join (graph-build time, no error)", + lambda: Join()(base, b_stream()).output_schema(), +) diff --git a/superpowers/specs/itl-939-experiments/exp02_workarounds_today.py b/superpowers/specs/itl-939-experiments/exp02_workarounds_today.py new file mode 100644 index 000000000..d2efbd716 --- /dev/null +++ b/superpowers/specs/itl-939-experiments/exp02_workarounds_today.py @@ -0,0 +1,189 @@ +"""ITL-939 experiment 02: every way to get a tag value into a function pod TODAY. + +Throwaway spike script; public interface only. EDI-shaped fixtures: + +* ``session`` keys (subject_id, session_start_datetime); data session_end_datetime, booth +* ``experiment`` keys (subject_id, session_start_datetime, experiment_index); + data experiment_start_timestamp (u64, microseconds since the epoch), task_id + +Subject m1 has two sessions, each with experiment_index 0, so experiment_index alone does +not identify an experiment (as in the real EDI tables). + +The computation: elapsed seconds from session start to experiment start, then a Join with +``session`` on the shared keys, as a workshop pipeline would do. + +Run: + + uv run python superpowers/specs/itl-939-experiments/exp02_workarounds_today.py +""" + +from __future__ import annotations + +import datetime as dt + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.operators import Join, MapData +from orcapod.sources import ArrowTableSource + +UTC = dt.timezone.utc +TS = pa.timestamp("us", tz="UTC") +S1 = dt.datetime(2026, 1, 5, 9, 0, tzinfo=UTC) +S2 = dt.datetime(2026, 1, 6, 9, 30, tzinfo=UTC) +S3 = dt.datetime(2026, 1, 5, 14, 0, tzinfo=UTC) + + +def us(t: dt.datetime) -> int: + return int(t.timestamp() * 1_000_000) + + +def session_table() -> pa.Table: + return pa.table( + { + "subject_id": pa.array(["m1", "m1", "m2"], pa.large_string()), + "session_start_datetime": pa.array([S1, S2, S3], TS), + "session_end_datetime": pa.array( + [S1 + dt.timedelta(hours=2), S2 + dt.timedelta(hours=2), S3 + dt.timedelta(hours=1)], + TS, + ), + "booth": pa.array(["A", "B", "A"], pa.large_string()), + } + ) + + +def experiment_table() -> pa.Table: + starts = [S1, S1, S2, S2, S3] + offsets_s = [60, 900, 120, 1500, 30] + return pa.table( + { + "subject_id": pa.array(["m1", "m1", "m1", "m1", "m2"], pa.large_string()), + "session_start_datetime": pa.array(starts, TS), + "experiment_index": pa.array([0, 1, 0, 1, 0], pa.uint32()), + "experiment_start_timestamp": pa.array( + [us(s) + o * 1_000_000 for s, o in zip(starts, offsets_s)], pa.uint64() + ), + "task_id": pa.array(["fv", "fv", "fv", "gz", "fv"], pa.large_string()), + } + ) + + +KEYS_SESSION = ["subject_id", "session_start_datetime"] +KEYS_EXPERIMENT = ["subject_id", "session_start_datetime", "experiment_index"] + + +def src(table: pa.Table, tags: list[str], source_id: str) -> ArrowTableSource: + return ArrowTableSource(table, tag_columns=tags, source_id=source_id, infer_nullable=True) + + +session = src(session_table(), KEYS_SESSION, "session") + + +def show(label: str, stream) -> None: + print(f"\n=== {label}") + tags, data = stream.keys() + print("keys():", tags, "|", data) + sys_cols = [c for c in stream.as_table(all_info=True).column_names if c.startswith("_tag_")] + print("system tag columns:", sys_cols) + print("pipeline_hash:", stream.pipeline_hash().to_hex(16)) + print("content_hash: ", stream.content_hash().to_hex(16)) + + +def rows(stream, cols: list[str]) -> None: + df = stream.as_table().select(cols).to_pandas() + print(df.to_string(index=False)) + + +# --- baseline: the DB-source default (primary keys become the tags) -------------------- +baseline = src(experiment_table(), KEYS_EXPERIMENT, "experiment") +show("baseline: experiment, primary keys as tags", baseline) + + +# ---------------------------------------------------------------------------------------- +# W1. Choose tag_columns so session_start_datetime stays data. +# ---------------------------------------------------------------------------------------- +@function_pod(output_keys="elapsed_s") +def elapsed_s(session_start_datetime: dt.datetime, experiment_start_timestamp: int) -> float: + return experiment_start_timestamp / 1e6 - session_start_datetime.timestamp() + + +w1_src = src(experiment_table(), ["subject_id", "experiment_index"], "experiment") +show("W1 source: tag_columns without session_start_datetime", w1_src) +try: + w1 = elapsed_s.pod(w1_src.select_data_columns(["session_start_datetime", "experiment_start_timestamp"])) + print("W1 pod output:") + rows(w1, ["subject_id", "experiment_index", "elapsed_s"]) +except Exception as exc: # noqa: BLE001 + print(f"W1 pod RAISES {type(exc).__name__}: {exc}") +# Downstream: Join with session on the shared keys. W1's tags lack session_start_datetime. +try: + j = Join()(w1, session) + print("W1 Join keys():", j.keys()) + print("W1 Join rows:", j.as_table().num_rows, "(5 experiments; correct answer is 5)") + rows(j, ["subject_id", "experiment_index", "session_start_datetime", "elapsed_s"]) +except Exception as exc: # noqa: BLE001 + print(f"W1 Join RAISES {type(exc).__name__}: {exc}") +# The pod's output no longer carries session_start_datetime at all, so the result has lost +# the session identity: m1 experiment 0 appears twice with nothing to tell them apart. + + +# ---------------------------------------------------------------------------------------- +# W2. Duplicate the column under another name BEFORE building the source. +# (A DB source such as SpiralDBTableSource runs SELECT * with no column hook, so for a +# DB table this means: read it, copy the column, build an ArrowTableSource.) +# ---------------------------------------------------------------------------------------- +@function_pod(output_keys="elapsed_s") +def elapsed_s2(session_start: dt.datetime, experiment_start_timestamp: int) -> float: + return experiment_start_timestamp / 1e6 - session_start.timestamp() + + +t = experiment_table() +t = t.append_column("session_start", t["session_start_datetime"]) # the one extra line +w2_src = src(t, KEYS_EXPERIMENT, "experiment") +show("W2 source: column copied to 'session_start' before the source", w2_src) +w2 = elapsed_s2.pod(w2_src.select_data_columns(["session_start", "experiment_start_timestamp"])) +print("W2 pod output:") +rows(w2, ["subject_id", "session_start_datetime", "experiment_index", "elapsed_s"]) +j2 = Join()(w2, session) +print("W2 Join keys():", j2.keys()) +print("W2 Join rows:", j2.as_table().num_rows, "(correct answer is 5)") + + +# ---------------------------------------------------------------------------------------- +# W2'. Same, but mid-pipeline (after an operator), by re-rooting: as_table() + new source. +# ---------------------------------------------------------------------------------------- +mid = Join()(baseline, session) # some upstream work +mt = mid.as_table() +mt = mt.append_column("session_start", mt["session_start_datetime"]) +w2b_src = ArrowTableSource(mt, tag_columns=KEYS_EXPERIMENT, source_id="rerooted", infer_nullable=True) +show("W2' re-rooted mid-pipeline (lineage to 'experiment'/'session' is cut)", w2b_src) +print("W2' upstreams of the new source:", w2b_src.upstreams) + + +# ---------------------------------------------------------------------------------------- +# W3. Read the source twice: once keyed normally, once with the column as data, then join. +# ---------------------------------------------------------------------------------------- +second = src(experiment_table(), ["subject_id", "experiment_index"], "experiment_as_data") +second = MapData({"session_start_datetime": "session_start"}, drop_unmapped=True)(second) +try: + j3 = Join()(baseline, second) + print("\n=== W3: Join(keys-as-tags read, column-as-data read)") + print("W3 Join keys():", j3.keys()) + print("W3 Join rows:", j3.as_table().num_rows, "(5 experiments; >5 means a wrong pairing)") + rows(j3, ["subject_id", "session_start_datetime", "experiment_index", "session_start"]) +except Exception as exc: # noqa: BLE001 + print(f"W3 Join RAISES {type(exc).__name__}: {exc}") + + +# ---------------------------------------------------------------------------------------- +# W4. Operators that could copy a tag into data today? None: MapTags renames (moves) a tag +# within the tag group; MapData renames within the data group; Select/Drop only remove; +# Pick/Index read data columns only; a pod never receives tags. +# ---------------------------------------------------------------------------------------- +from orcapod.operators import MapTags # noqa: E402 + +try: + MapTags({"session_start_datetime": "session_start"})(baseline).keys() + print("\nW4 MapTags keys():", MapTags({"session_start_datetime": "session_start"})(baseline).keys()) +except Exception as exc: # noqa: BLE001 + print(f"W4 MapTags RAISES {type(exc).__name__}: {exc}") From 4394f52d39339aceb44f9daba970e38eb749d7d3 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 16:18:16 -0800 Subject: [PATCH 02/14] docs(itl-939): copy-operator prototype and foreign-key experiments Co-Authored-By: Claude Opus 5.5 --- .../exp03_copy_tag_prototype.py | 174 ++++++++++++++++++ .../exp04_foreign_key_promotion.py | 76 ++++++++ 2 files changed, 250 insertions(+) create mode 100644 superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py create mode 100644 superpowers/specs/itl-939-experiments/exp04_foreign_key_promotion.py diff --git a/superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py b/superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py new file mode 100644 index 000000000..312e7e2f5 --- /dev/null +++ b/superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py @@ -0,0 +1,174 @@ +"""ITL-939 experiment 03: an out-of-tree prototype of design C, a tag-to-data COPY operator. + +Throwaway spike script. ``CopyTagsToData`` is defined here, not in ``src/``; it is the +smallest feature the design document proposes. Everything else uses the public interface. + +Run: + + uv run python superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py +""" + +from __future__ import annotations + +import contextlib +import datetime as dt +import io +import sys +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.core.operators.base import UnaryOperator +from orcapod.core.streams import ArrowTableStream +from orcapod.errors import InputValidationError +from orcapod.operators import GroupBy, Join +from orcapod.sources import ArrowTableSource +from orcapod.system_constants import constants +from orcapod.types import ColumnConfig, Schema + +sys.path.insert(0, str(Path(__file__).parent)) +with contextlib.redirect_stdout(io.StringIO()): # reuse exp02's fixtures, not its output + import exp02_workarounds_today as fx + + +class CopyTagsToData(UnaryOperator): + """Copy tag columns into the data group under new names; the tags stay as they are. + + Prototype only. The copy takes the tag's Arrow type and nullability unchanged. Its + source-info value is null: an operator does not know which source a tag value came + from (after a Join it came from several). + + Args: + name_map: Tag column name to the name of its data copy. + """ + + def __init__(self, name_map: Mapping[str, str], **kwargs: Any) -> None: + self.name_map = dict(name_map) + super().__init__(**kwargs) + + def identity_structure(self) -> Any: + return (self.__class__.__name__, tuple(sorted(self.name_map.items()))) + + def validate_unary_input(self, stream) -> None: + tag_columns, data_columns = stream.keys() + missing = set(self.name_map) - set(tag_columns) + if missing: + raise InputValidationError(f"Not tag columns: {sorted(missing)}") + taken = set(self.name_map.values()) & (set(tag_columns) | set(data_columns)) + if taken: + raise InputValidationError(f"Copy names already used: {sorted(taken)}") + + def unary_output_schema( + self, stream, *, columns: ColumnConfig | dict[str, Any] | None = None, all_info: bool = False + ) -> tuple[Schema, Schema]: + tag_schema, data_schema = stream.output_schema(columns=columns, all_info=all_info) + base_tags, _ = stream.output_schema() + copies = {new: base_tags[old] for old, new in self.name_map.items()} + return tag_schema, Schema({**data_schema, **copies}) + + def unary_static_process(self, stream): + tag_columns, _ = stream.keys() + table = stream.as_table(columns={"source": True, "system_tags": True, "sort_by_tags": False}) + for old, new in self.name_map.items(): + col = table.column(old) + field = table.schema.field(old) + table = table.append_column(pa.field(new, field.type, field.nullable), col) + table = table.append_column( + pa.field(f"{constants.SOURCE_PREFIX}{new}", pa.large_string(), True), + pa.nulls(table.num_rows, pa.large_string()), + ) + return ArrowTableStream(table, tag_columns=tag_columns) + + +def section(title: str) -> None: + print(f"\n=== {title}") + + +experiment = fx.baseline # keys subject_id, session_start_datetime, experiment_index +session = fx.session + +# 1. What the user sees. +copied = CopyTagsToData({"session_start_datetime": "session_start"})(experiment) +section("1. CopyTagsToData({'session_start_datetime': 'session_start'})(experiment)") +print("keys():", copied.keys()) +print("output_schema():", copied.output_schema()) +print("as_table() columns:", copied.as_table().column_names) +print("as_table(all_info) source cols:", [c for c in copied.as_table(all_info=True).column_names if c.startswith("_source_")]) +tag, data = next(iter(copied.iter_data())) +print("first iter_data():", tag.as_dict(), "|", data.as_dict()) +# Schema prediction vs the computed stream (non-public static_process, the CLAUDE.md +# standing exception for checking a schema prediction). +op = CopyTagsToData({"session_start_datetime": "session_start"}) +print("prediction matches static_process:", op.output_schema(experiment) == op.static_process(experiment).output_schema()) + +# 2. The pod reads the copy; the tags still identify the row. +section("2. pod on the copy, then Join with session on the shared keys") +elapsed = fx.elapsed_s2.pod(copied.select_data_columns(["session_start", "experiment_start_timestamp"])) +fx.rows(elapsed, ["subject_id", "session_start_datetime", "experiment_index", "elapsed_s"]) +j = Join()(elapsed, session) +print("Join keys():", j.keys()) +print("Join rows:", j.as_table().num_rows, "(correct answer is 5)") + +# 3. Hash comparison with the W2 recipe (column duplicated before the source). +section("3. data content hashes: copy operator vs W2 recipe") +w2_in = fx.w2_src.select_data_columns(["session_start", "experiment_start_timestamp"]) +c_in = copied.select_data_columns(["session_start", "experiment_start_timestamp"]) +w2_hashes = [d.content_hash().to_hex(16) for _, d in w2_in.iter_data()] +c_hashes = [d.content_hash().to_hex(16) for _, d in c_in.iter_data()] +print("W2 :", w2_hashes) +print("copy :", c_hashes) +print("per-row data hashes equal (so pod result caches agree):", w2_hashes == c_hashes) +print("pipeline_hash experiment (unchanged by the feature):", experiment.pipeline_hash().to_hex(16)) +print("pipeline_hash after copy:", copied.pipeline_hash().to_hex(16)) +print("pipeline_hash W2 source :", fx.w2_src.pipeline_hash().to_hex(16)) + +# 4. Why design D (pods read tags directly) is unsafe with today's result cache. +section("4. two rows, same data, different tags -> same input data hash") +t = pa.table( + { + "subject_id": pa.array(["m1", "m1"], pa.large_string()), + "session_start_datetime": pa.array([fx.S1, fx.S2], fx.TS), + "experiment_start_timestamp": pa.array([fx.us(fx.S2) + 60_000_000] * 2, pa.uint64()), + } +) +s = ArrowTableSource(t, tag_columns=["subject_id", "session_start_datetime"], source_id="d", infer_nullable=True) +print("data hashes:", [d.content_hash().to_hex(16) for _, d in s.iter_data()]) +print("-> a pod that read session_start_datetime from the TAG would get one cached result") +print(" for both rows (ResultCache keys on the input data hash only).") +s_c = CopyTagsToData({"session_start_datetime": "session_start"})(s) +print("with the copy, data hashes:", [d.content_hash().to_hex(16) for _, d in s_c.iter_data()]) + +# 5. Nullable keys (every EDI key column is nullable in the DB schema). +section("5. nullable tag -> nullable copy") +tn = pa.Table.from_arrays( + [t["subject_id"], t["session_start_datetime"], t["experiment_start_timestamp"]], + schema=pa.schema( + [ + pa.field("subject_id", pa.large_string(), True), + pa.field("session_start_datetime", fx.TS, True), + pa.field("experiment_start_timestamp", pa.uint64(), True), + ] + ), +) +sn = ArrowTableSource(tn, tag_columns=["subject_id", "session_start_datetime"], source_id="n") +cn = CopyTagsToData({"session_start_datetime": "session_start"})(sn) +print("tag schema :", cn.output_schema()[0]) +print("data schema:", cn.output_schema()[1]) +print("arrow field:", cn.as_table().schema.field("session_start")) + +# 6. Copy before GroupBy: grouped key values reach the pod as a list. +section("6. copy, then GroupBy(by=['subject_id', 'session_start_datetime'])") +g = GroupBy(by=["subject_id", "session_start_datetime"])( + CopyTagsToData({"experiment_index": "experiment_idx"})(experiment) +) +print("keys():", g.keys()) +print("output_schema():", g.output_schema()) +fx.rows(g, ["subject_id", "session_start_datetime", "experiment_index", "experiment_idx"]) + +# 7. Equality is a snapshot: after a later operator the two may diverge. +section("7. the copy is ordinary data afterwards") +renamed = fx.MapData({"session_start": "t0"})(copied) +print("MapData on the copy leaves the tag alone:", renamed.keys()) diff --git a/superpowers/specs/itl-939-experiments/exp04_foreign_key_promotion.py b/superpowers/specs/itl-939-experiments/exp04_foreign_key_promotion.py new file mode 100644 index 000000000..581cc30ae --- /dev/null +++ b/superpowers/specs/itl-939-experiments/exp04_foreign_key_promotion.py @@ -0,0 +1,76 @@ +"""ITL-939 experiment 04: the inverse direction, a DATA column wanted as a TAG (foreign key). + +EDI ``penetration`` is keyed (subject_id, session_start_datetime, recording_index, +probe_index) and carries ``probe_id`` as data; ``probe`` is keyed by ``probe_id``. Joining +them needs ``probe_id`` as a tag on both sides. Throwaway spike script, public interface only. + +Run: + + uv run python superpowers/specs/itl-939-experiments/exp04_foreign_key_promotion.py +""" + +from __future__ import annotations + +import datetime as dt + +import pyarrow as pa + +from orcapod import function_pod +from orcapod.operators import Join +from orcapod.sources import ArrowTableSource + +UTC = dt.timezone.utc +TS = pa.timestamp("us", tz="UTC") +S1 = dt.datetime(2026, 1, 5, 9, 0, tzinfo=UTC) + +penetration = pa.table( + { + "subject_id": pa.array(["m1", "m1"], pa.large_string()), + "session_start_datetime": pa.array([S1, S1], TS), + "recording_index": pa.array([0, 0], pa.uint32()), + "probe_index": pa.array([0, 1], pa.uint32()), + "probe_id": pa.array(["P-17", "P-22"], pa.large_string()), + "probe_plan_chamber_z_um": pa.array([3100.0, 2950.0], pa.float64()), + } +) +probe = pa.table( + { + "probe_id": pa.array(["P-17", "P-22"], pa.large_string()), + "probe_model": pa.array(["NP1.0", "NP2.0"], pa.large_string()), + } +) +PK = ["subject_id", "session_start_datetime", "recording_index", "probe_index"] + +pen_pk = ArrowTableSource(penetration, tag_columns=PK, source_id="penetration", infer_nullable=True) +probe_src = ArrowTableSource(probe, tag_columns=["probe_id"], source_id="probe", infer_nullable=True) + +print("=== F1. primary keys only: Join(penetration, probe) shares no tag") +try: + j = Join()(pen_pk, probe_src) + print("keys():", j.keys()) + print("rows:", j.as_table().num_rows, "(2 penetrations; 4 means a cross product)") +except Exception as exc: # noqa: BLE001 + print(f"RAISES {type(exc).__name__}: {exc}") + +print("\n=== F2. today's recipe: list probe_id among the tag_columns at the source") +pen_fk = ArrowTableSource( + penetration, tag_columns=[*PK, "probe_id"], source_id="penetration", infer_nullable=True +) +j2 = Join()(pen_fk, probe_src) +print("keys():", j2.keys()) +print("rows:", j2.as_table().num_rows, "(correct answer is 2)") + + +print("\n=== F3. after a pod, a data column cannot become a tag (no operator does it)") + + +@function_pod(output_keys=["probe_id", "depth_mm"]) +def plan(probe_id: str, probe_plan_chamber_z_um: float) -> tuple[str, float]: + return probe_id, probe_plan_chamber_z_um / 1000.0 + + +out = plan.pod(pen_pk.select_data_columns(["probe_id", "probe_plan_chamber_z_um"])) +print("pod output keys():", out.keys()) +print("Join(pod output, probe) keys():", Join()(out, probe_src).keys()) +print("-> probe_id stays data; joining on it needs the column as a tag, which only") +print(" re-rooting (as_table() + a new source) can do today.") From 8c9ef7ae048da91638b26f2cb660e01bdbc4b8ed Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 16:21:25 -0800 Subject: [PATCH 03/14] docs(itl-939): design spike for a value that is both tag and data Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 18 + ...11-itl-939-tag-and-data-same-name-spike.md | 42 +- ...1-itl-939-tag-and-data-same-name-design.md | 378 ++++++++++++++++++ 3 files changed, 419 insertions(+), 19 deletions(-) create mode 100644 superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..017cfc502 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2442,3 +2442,21 @@ results. Known compromise: the struct's children are nullable at the Arrow level, because Polars drops nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null instead. Tighten under a new storage version once nested nullability survives Polars. + +--- + +## Tag and data groups — a value wanted as both (ITL-939) + +### TD1 — A key column cannot also be a pod input; same-name collisions fail late +**Status:** in progress +**Severity:** medium +**Issue:** ITL-939 (design spike) + +A name cannot be in both the tag and the data group, so a key that a pod needs as data (EDI +`session_start_datetime`, for elapsed-time arithmetic) cannot reach a pod without a workaround, +and two of the obvious workarounds (keeping the column out of `tag_columns`, or reading the +table twice and joining) give silently wrong joins. Nothing checks the rule at graph-build time: +`Join`/`MergeJoin` across a tag-vs-data name, and a pod output key equal to a tag name, are +accepted by `keys()` and fail only at materialisation. Design spike and recommendation (a +`CopyTagsToData` operator) in `superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md`, +awaiting the developer's decisions. diff --git a/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md b/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md index b98ec3c60..a74f5d9fc 100644 --- a/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md +++ b/superpowers/plans/2026-11-11-itl-939-tag-and-data-same-name-spike.md @@ -20,53 +20,57 @@ EDI-shaped table (`session`, `experiment`, keys `subject_id`, `session_start_dat **Files:** Create `superpowers/specs/itl-939-experiments/exp01_same_name_today.py` -- [ ] Build an Arrow table with duplicate field names (`session_start_datetime` twice) and pass +- [x] Build an Arrow table with duplicate field names (`session_start_datetime` twice) and pass it to `ArrowTableSource` with the name in `tag_columns`; record the error or behaviour. -- [ ] `MapData({"x": ""})` and `MapTags({"": ""})` collisions. -- [ ] `Join` of two streams where one stream's tag name equals the other's data name. -- [ ] `function_pod` whose output key equals a tag name. -- [ ] Run with `uv run python ...`; save stdout to `exp01_same_name_today.out.txt`. +- [x] `MapData({"x": ""})` and `MapTags({"": ""})` collisions. +- [x] `Join` of two streams where one stream's tag name equals the other's data name. +- [x] `function_pod` whose output key equals a tag name. +- [x] Run with `uv run python ...`; save stdout to `exp01_same_name_today.out.txt`. ### Task 2: Experiment 02, today's workarounds on the EDI-shaped table **Files:** Create `superpowers/specs/itl-939-experiments/exp02_workarounds_today.py` -- [ ] W1 choose `tag_columns` so the column stays data. -- [ ] W2 duplicate the column under another name before building the source. -- [ ] W3 read the source twice (once with the column as data), join. -- [ ] W4 anything else found (e.g. `MapTags` cannot copy; `PolarsFilter` sees both). -- [ ] For each: compute elapsed time (`experiment_start - session_start_datetime`) with a pod, +- [x] W1 choose `tag_columns` so the column stays data. +- [x] W2 duplicate the column under another name before building the source. +- [x] W3 read the source twice (once with the column as data), join. +- [x] W4 anything else found (e.g. `MapTags` cannot copy; `PolarsFilter` sees both). +- [x] For each: compute elapsed time (`experiment_start - session_start_datetime`) with a pod, then Join with a second table keyed on the same columns; record lines, renames, `pipeline_hash` / `content_hash` changes and downstream Join behaviour. -- [ ] Save stdout to `exp02_workarounds_today.out.txt`. +- [x] Save stdout to `exp02_workarounds_today.out.txt`. ### Task 3: Experiment 03, prototype of the smallest feature (design C), out of tree **Files:** Create `superpowers/specs/itl-939-experiments/exp03_copy_tag_prototype.py` -- [ ] A throwaway subclass of `UnaryOperator` defined in the script (not in `src/`) that copies +- [x] A throwaway subclass of `UnaryOperator` defined in the script (not in `src/`) that copies named tag columns into the data group under chosen names, with source-info for the copy. -- [ ] Show `keys()`, `output_schema()`, `as_table()`, `iter_data()`, a pod reading the copy, +- [x] Show `keys()`, `output_schema()`, `as_table()`, `iter_data()`, a pod reading the copy, a downstream Join keyed on the original tags, and that streams not using it keep their hashes. -- [ ] Save stdout to `exp03_copy_tag_prototype.out.txt`. +- [x] Save stdout to `exp03_copy_tag_prototype.out.txt`. ### Task 4: Need analysis from the EDI schemas -- [ ] List key columns that are plausibly computation inputs; 3-5 workshop computations. +- [x] List key columns that are plausibly computation inputs; 3-5 workshop computations. ### Task 5: Design document -- [ ] Sections: today (1), need (2), designs A-D table + prose (3), recommendation and numbered +- [x] Sections: today (1), need (2), designs A-D table + prose (3), recommendation and numbered decisions (4). Commit and push WIP. ### Task 6: Report and PR -- [ ] `ask` the coordinator with the summary; wait for the reply. -- [ ] Append the DESIGN_ISSUES.md entry for ITL-939 at the end. -- [ ] Merge origin/main, push, open docs-only PR ("Part of ITL-939"), attach to Linear, +- [x] `ask` the coordinator with the summary; wait for the reply. +- [x] Append the DESIGN_ISSUES.md entry for ITL-939 at the end. +- [x] Merge origin/main, push, open docs-only PR ("Part of ITL-939"), attach to Linear, `orca worktree set --pr`. CodeRabbit rounds. `worker_done`. ## Progress notes (successor: read this section first) +- Tasks 1-5 done (exp01-exp04 with .out.txt; design doc written). Findings: Join/MergeJoin + tag-vs-data collision crashes late (J1), pod output key == tag name accepted (P1); W1/W3 + give 9 rows instead of 5; GroupBy already turns non-key tags into data (precedent for C). +- Next: ask coordinator with the summary (Task 6), then PR. diff --git a/superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md b/superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md new file mode 100644 index 000000000..799729051 --- /dev/null +++ b/superpowers/specs/2026-11-11-itl-939-tag-and-data-same-name-design.md @@ -0,0 +1,378 @@ +# ITL-939 design spike: a value that is both a tag and data + +**Status:** proposal, awaiting the developer's decisions (section 6). +**Issue:** ITL-939. **Related:** ITL-929 (record ids from primary keys), ITL-938 (versioning). +**Date:** November 11, 2026. +**Experiments:** `superpowers/specs/itl-939-experiments/` (each `expNN_*.py` has its captured +output next to it as `expNN_*.out.txt`; rerun with `uv run python