diff --git a/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md new file mode 100644 index 000000000..284868271 --- /dev/null +++ b/superpowers/plans/2026-11-13-itl-1004-referential-tags-audit.md @@ -0,0 +1,63 @@ +# ITL-1004 referential tags: implementation-scope audit (plan) + +> **For agentic workers:** this plan produces a design document, not library code. No change under +> `src/` or `tests/`, no golden file. + +**Goal:** One document, `superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md`, +that makes the v0.4 referential-tags specification (Linear document attached to ITL-1004) +dispatchable: where the code assumes tag/data disjointness, which hashes would see a binding, +what is stored, operator-by-operator rules and change sizes, sources, function pods, answers to +the specification's open questions, a sliced delivery plan, and the decisions for the developer. + +**Architecture:** read the code; measure today's behaviour with small scripts under +`superpowers/specs/itl-1004-audit/` (run with `uv run`, outputs saved beside them); write the +document from the evidence. + +**Branch:** `eywalker/itl-1004-referential-tags-v04-a-tag-may-refer-to` from `origin/main` +(`f51ce67c`). PR: docs only, `Refs ITL-1004`. + +--- + +## Task 1: Representation sites +- [ ] Read `core/streams/arrow_table_stream.py`, `core/datagrams/*`, `types.py` (Schema, ColumnConfig), + `utils/schema_utils.py`, `core/streams/base.py`. +- [ ] grep every `keys()` / `output_schema` / `identity_structure` / `pipeline_identity_structure` + definition; classify each site as works / change / reject. +- [ ] Script `01_representation.py`: build an `ArrowTableStream` with a column in both tag and + data lists (if constructible) and record what each public method returns. + +## Task 2: Identity and hashes +- [ ] Read `core/base.py`, `core/sources/base.py`, `core/nodes/function_node.py` + (`_build_record_id_preimage`), `core/function_pod.py`, `core/datagrams` content hash, + `derived_source.py` (PR #336/#338 pattern). +- [ ] Script `02_identity.py`: show which hash inputs contain the tag/data split. + +## Task 3: Storage +- [ ] Read FunctionNode / OperatorNode / CachedSource / DerivedSource write and read paths. +- [ ] Script `03_storage.py`: stored columns of each table kind for a small pipeline. + +## Task 4: Operators +- [ ] For every file in `core/operators` and the FunctionPod call path, measure today's behaviour + with a same-named tag and data column (`04_operators.py`), sync and async. +- [ ] Join source-info of merged column: options. + +## Task 5–6: Sources and function pods +- [ ] Read `arrow_table_source.py`, `stream_builder.py`, `spiraldb_table_source.py`, + `db_table_source.py`, `derived_source.py`; `function_pod.py` output collision (`05_function_pod.py`). + +## Task 7–9: Open questions, delivery plan, decisions +- [ ] Answer section 17 from code; slice plan with sizes, dependencies, tests; decisions list. + +## Task 10: Finish +- [ ] Doc under ~900 lines; ruff on audit scripts; `uv run pytest tests/test_docs` not affected + (superpowers is not a docs page); merge origin/main; push; PR `Refs ITL-1004`; + `orca linear attach`; `orca worktree set --pr`; worker_done. + +## Progress notes +(updated as work proceeds, for a successor) +- 2026-11-13: Tasks 1-9 done. Measurements 01-06 in superpowers/specs/itl-1004-audit/ (outputs .out). + Coordinator redirects applied (default referential sources; plain-tag mode = v0.3 exactly; + three Join cases equal; decisions only for Open/code-impossible points; slices mapped onto + ITL-1009..1023). Representation + hash verdict sent to the coordinator (msg_a8cdbe370954). + Pod-boundary rule (D1) asked; coordinator: keep (a) recommended, put to developer. + Full suite: 11029 passed, 117 skipped, 23 xfailed. Remaining: PR, linear attach, worker_done. diff --git a/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md new file mode 100644 index 000000000..fe1f79a50 --- /dev/null +++ b/superpowers/specs/2026-11-13-itl-1004-referential-tags-implementation-scope.md @@ -0,0 +1,759 @@ +# ITL-1004 — Referential tags (v0.4): implementation scope + +**Status:** design audit for the developer's review. No library code changes. +**Issue:** ITL-1004. **Specification:** the Linear document "Orcapod v0.4 — Referential Tags: +design specification and implementation guidance" attached to the issue ("the spec"; §9.3 +refers to its section numbers). The spec is the authority. This document scopes how the code +implements what the spec marks Required or Proposed, and raises decisions only for points the +spec marks Open or that the code makes impossible or ambiguous. +**Developer decisions included** (November 12, 2026): +- Every source's declared tags are referential **by default**. +- Tags named in `tag_only_columns` (a subset of `tag_columns`) stay materialized and must + reproduce v0.3 exactly, so the referential path is additive. This is a requirement and a + proof obligation: it is proven once the golden fixtures, switched to `tag_only_columns`, pass + unchanged (ITL-1011, ITL-1012). This audit shows only that no hash input of a tag-only + stream changes (§3). +- The change is a change of default behaviour. +- `Join`'s three cases (§9.1) have equal standing. +- Pod boundary (D1). With `pod.process(...)` and `pod(..., strict=True)`, a reference whose + data column does not survive becomes a plain tag. `pod(...)` uses a **variant of the function + pod** that emits each row's referenced columns beside the function's outputs, so they stay + referential. It is not a join and not an inserted operator (§11). Full append is deferred + (ITL-1025). +- `GroupBy` follows spec §8 unchanged; `Join` gets no special rule for list-valued tags. +- Saved jobs carry version information that pins behaviour (§4). +- Renames (§5.4): `map_tags` moves both roles of a reference, `tag_only=True` and `map_data` + leave a plain tag. + +**Code audited:** `origin/main` at `f51ce67c`. +**Measurements:** `superpowers/specs/itl-1004-audit/` holds each script (run with `uv run`) and +its output (`.out`). Every example below comes from those outputs. + +| Script | What it measures | +|---|---| +| `01_today_same_name.py` | What v0.3 does when a tag and a data column share a name | +| `02_prototype_unified.py` | Every operator over a **prototype** referential stream: today's stream with one physical column put in both role lists through private attributes. A probe of today's operator code, not an implementation | +| `03_arrow_table_stream_sites.txt` | The 32 places that construct an `ArrowTableStream` | +| `04_identity.py` | Which hashes see a binding | +| `05_migration_and_collisions.py` | Function-result identity across the move from `tags_as_data`; today's output-name checks | +| `06_default_referential.py` | Consequences of the referential default; today's plain-with-plain `Join` | + +--- + +## 1. Verdicts in brief + +1. **Representation (spec §13.3, open question "how Arrow storage avoids duplicated values").** + A referential tag `x` is **one physical column `x` whose name is in both role lists** of the + stream. The binding is the name intersection `tag_keys ∩ data_keys`, which is the spec's + §4.3 statement as code. + - The current table abstraction already supports this. At stream level Orcapod is the + spec's option B: an `ArrowTableStream` holds one table, tags are a name list, and data is + every other column (`core/streams/arrow_table_stream.py:85`, `:113-115`). At row level it + is option A: separate `Tag` and `Data` datagrams. + - `types.Schema` and the `(tag_schema, data_schema)` tuple do not change. A referential tag + appears in both halves of `output_schema()` when the graph is built. +2. **Hashes.** No data context version is needed, and no golden entry moves. + - A stream with only materialized tags produces byte-identical hash inputs. The binding has + no term of its own, so the pair `(tag_schema, data_schema)` is exactly the v0.3 pair. + - A referential tag changes the pair, so it enters `pipeline_hash`, the root `content_hash` + and the source schema hash only when present. + - A function that does not read the referenced column receives a `Data` whose hash equals + the v0.3 one (measured, §3.3). Its stored results are reused. +3. **The default, for an unchanged v0.3 script.** Moved: source content hash, pipeline hashes, + system-tag column names, function-node pipeline record ids, and pipeline/operator/cached + table paths. Unchanged: source ids, source record ids, row `Data` hashes and results of + functions that read no key, and result table paths. A user reproduces v0.3 exactly by + also naming the tags in `tag_only_columns` (§6.2). +4. **Storage.** No stored-table layout changes. A referenced value is written once in every + table kind. The binding is recovered on read by the same intersection rule from the saved + `{tag, data}` descriptor. +5. **Work.** + - The prototype shows filters, `DropNulls`, `SemiJoin`, `Batch` and non-key `GroupBy` + already produce coherent physical results, ordering included. + - What breaks is generic: every operator rebuilds its output with + `ArrowTableStream(table, tag_columns=...)`, which drops the data role (32 sites), and two + row-to-table helpers `hstack` tag and data (duplicate column). + - The new rules are local: `Join` cases 2 and 3, materialization on select or drop, the + `GroupBy` key, and output collisions. +6. **Prerequisite: ITL-940 (Backlog).** Today `Join`, `MergeJoin`, the function pod and + `Index(out=)` accept a tag `x` beside a data `x` at build time. Under the intersection rule + such shapes would look referential while holding different values. The async `Join` already + emits such rows silently (`01 R2`). +7. **One Required rule the code makes impossible as written** (spec §6.1 against + `core/function_pod.py:747`): a pod's output data is only the function's outputs, so a + reference's target never survives a pod. Decided (D1): the tag becomes plain at the pod boundary. + +--- + +## 2. Representation (question 1) + +### 2.1 What the code is today + +- **Stream.** + - `ArrowTableStream` stores one table. Tags are the given names + (`arrow_table_stream.py:85`); a missing tag name raises (`:102-105`); data is "every + column not a tag" (`:113-115`); the schemas are filtered by those sets (`:127-136`). + - `iter_data` selects each row's tag and data columns from the same record batch + (`:352-353`). `RecordBatch.select` is zero-copy, so a name in both selections shares one + buffer. +- **Disjointness is by construction.** A table can hold two columns of one name, but the + stream then fails on its first lookup (`01 R1`: + `KeyError: 'Field "x" exists 2 times in schema'`). The source builder splits the same way + (`data_schema = non_sys.drop(tag_columns)`, `core/sources/stream_builder.py:219-220`) and + gives source info only to non-tags (`:336-338`). +- **Rows.** `Tag` and `Data` are independent datagrams with their own tables + (`core/datagrams/tag_data.py:53`, `:274`). A row can already hold `x` in both. +- **Schema.** A plain mapping (`types.py:84`). The `(tag, data)` pair is the only record of a + column's role. No Arrow field metadata marks a role, and the hasher strips non-extension + metadata (`hashing/schema_cleaner.py:141-162`). + +### 2.2 Every site that assumes disjointness + +Legend: +- **OK**: works unchanged. +- **Change**: needs references passed or deduplicated. +- **Reject**: must refuse a shape. + +The prototype (`02`) verified the rows marked †. + +| Site (file:line) | Assumption | Verdict | +|---|---|---| +| `ArrowTableStream.__init__` (`arrow_table_stream.py:113-115`) | data = non-tags | **Change**: new `reference_tags` argument. Data = non-tags ∪ references; validate references ⊆ tags and present. The "at least one data column" rule (`:120-123`) then also counts references, so an all-tags source with referential tags is valid (`06 H3` shows v0.3 raises for a plain one, and still will) | +| `keys` / `output_schema` (`:182-236`) | — | OK†: `x` in both (`02 P0`) | +| `iter_data` (`:319-353`) | — | OK†: zero-copy selects | +| `as_table` / `_assemble_table` (`:256-310`) | — | OK†: one column, emitted once | +| `identity_structure` without producer (`:146-163`) | `self._tag_columns` alone | **Change**: add `("reference_tags", refs)` only when non-empty (§3) | +| `SourceStreamBuilder.build` (`stream_builder.py:143-295`, `:336-338`, `:346`) | data = table minus tags | **Change**: explicit `reference_tags`; schema hash over the pair with `x` in both; source info for referenced columns | +| `StaticOutputOperatorPod._rows_to_table` (`core/operators/static_output_pod.py:208-252`) | `hstack(tags, data)` | **Change**: drop a data column already present as a tag; return the reference set. `02 P4`: the async `GroupBy` raises `Duplicate column name 'x'` (`utils/arrow_utils.py:678-682`) | +| `_materialize_to_stream` / `_materialize_input_stream` (`static_output_pod.py:256-300`); `pipeline/sync_orchestrator.py:216`, `:236` | `tag_columns` only | **Change**: pass references (the empty branch takes them from `upstream_node.output_schema`) | +| 15 operator rebuilds (`03_arrow_table_stream_sites.txt`) | `ArrowTableStream(table, tag_columns=…)` | **Change**†: today the data role is lost silently. `02 P1`: `PolarsFilter` keys say data `('x','y')`, but rows carry `{'y': 0.2}` | +| `FunctionPodStream._table_from_pairs` (`core/function_pod.py:1111-1152`) | `hstack(tags, data)` | OK, given the output-collision rule (§7) | +| `Join._joined_user_schemas` (`core/operators/join.py:78-119`), `_merge_data` (`:759-790`) | shared data name = error | **Change** (§5.1) | +| `MergeJoin` (`merge_join.py:161-175`, `:325-420`) | shared data name = list merge | **Change** (§5.2, open) | +| `SemiJoin` (`semijoin.py:76-130`) | matches on overlapping tag **and** data names | OK† | +| `GroupBy.unary_output_schema` (`group_by.py:302`) | every data column becomes a list | **Change**: a referenced key stays scalar | +| `column_selection.py` (classes at `:37`, `:158`, `:286`, `:413`; rebuilds at `:76`, `:202`, `:328`, `:459`; async at `:113`, `:241`, `:364`, `:494`) | dropping a name drops the column | **Change**: drop a role (§5.3). The async row paths are already right | +| `MapTags`/`MapData` (`mappers.py:83`, `:104`, `:226`, `:254`) | a rename moves the column | **Change** (§5.4) | +| `Index`/`Pick` (`index.py:365`, `pick.py:379`) | data replaced in place | **Reject** in-place on a referenced column (§5.5) | +| `FunctionPod.output_schema` (`function_pod.py:733-747`) | output data = function outputs | **Change**: reject output names equal to a tag; D1 | +| `FunctionNode` read (`core/nodes/function_node.py:2820-2835`, `:2891`) | tags = pipeline-table columns | OK under D1 (tags are plain after a pod) | +| `OperatorNode` read (`operator_node.py:1116`, `:1228`, `:1317`, `:1581`; loaded stub `:235-238`) | `tag_keys = self.keys()[0]` | **Change**: pass references (from `keys()` or descriptor) | +| `CachedSource` (`cached_source.py:360-363`), `DerivedSource` plain (`derived_source.py:462`, `:490`, `:543`), `PollingSource` (`polling_source.py:288-301`, `:865`), `SpiralDBTableSource` (`spiraldb_table_source.py:623`) | tags vs rest | **Change** (§6) | +| `SourceProxy.keys` (`source_proxy.py:177-185`); node descriptor (`pipeline/graph.py:178-199`) | — | OK: two name→type dicts; a name may be in both | +| `schema_utils.union_schemas` / `intersection_schemas` | — | OK: equal types merge (`02 P1`, `SemiJoin`) | + +### 2.3 How a binding is declared (ITL-1009) + +**The rule, in one place:** `reference_names(tag_schema, data_schema) = tag ∩ data` (sorted). +For each name the two Python types must be equal. Because there is one physical column, the +Arrow type, nullability, nesting and values are identical by construction, so §4.4 and §4.5 +hold without a check on values. + +- **Schema model.** No change to `Schema` or the tuple. Add: + - `schema_utils.reference_names(...)`; + - `StreamBase.reference_tags()`; + - one validator for the namespace rules, run on every predicted output schema from the base + wrappers (`StaticOutputOperatorPod`, `_FunctionPodBase`), per spec §13.4. +- **Why inference is sound.** §4.3: "the presence of a tag and data column with the same name + must be sufficient to identify the tag as referential". The only unsound inputs are shapes + made by accident: tag `x` from one input with data `x` from another (ITL-940), or an output + named like a tag. The validator rejects both when the graph is built. +- **Rows.** A row-level consumer, meaning the async paths, applies the same intersection to a + row's tag and data keys. It agrees with the sync path without new channel metadata. +- **Cost.** 32 constructor sites pass references, and 2 helpers deduplicate. + +**Comparison with the spec's options (§13.3).** +- **Option A** (separate tag/data storage plus binding metadata) does not match the code: + Orcapod has no separate tag storage at stream, operator-cache or cached-source level. +- **Option B** (one table, role metadata) is what the code already is. The "role metadata" + is the two name lists. + +What remains open is whether to add an *explicit* marker on `Schema` (decision D3). + +--- + +## 3. Identity and hashes (question 2) + +### 3.1 Which identities see a binding + +| Identity | Hash input (file:line) | Sees a binding? | Source with `tag_only_columns` vs v0.3 | +|---|---|---|---| +| Root `pipeline_hash` | `(tag_schema, data_schema)` (`core/sources/base.py:132-139`) | **Yes**, through the pair. `04 I1`: tag-only `b9e6…`, data-only `95a5…`, referential `cc81…` | identical | +| Root `content_hash` | `(class, output_schema(), source_id)` (`base.py:88-90`) | **Yes** (`04 I3`) | identical | +| Source schema hash (names system-tag columns) | `compute_source_schema_hash(tag, data)` (`stream_builder.py:227`) | **Yes**. `06 H1`: `a7ad…` → `6a4b…` | identical | +| `ArrowTableStream` without producer, `content_hash` | `(class, table, tag_columns)` (`arrow_table_stream.py:146-163`) | **must add** `("reference_tags", refs)` when non-empty, or a referential and a plain stream over one table collide | identical (term absent) | +| Downstream hashes | `(producer, argument_symmetry(upstreams))` (`core/streams/base.py:106-113`; `static_output_pod.py:62-64`) | Yes, transitively. An automatic materialization is a function of (operator identity, input schema), so it is inside the Merkle chain (§12.3: "automatic materialization likewise changes the resulting pipeline structure") | identical | +| Default source id | raw table hash (`stream_builder.py:237`, `:257-265`) | No | identical | +| Source record id | `uuid5(source_id::token)` (`stream_builder.py:97-116`, `:333-335`) | No (no schema input) | identical | +| Function-node record id | `tag.as_table(system_tags)` + input `Data` digest (`function_node.py:115-141`) | Through the system-tag column *names* (schema hash). The table path is scoped by `pipeline_hash` (`function_node.py:475`) | identical | +| Row `Data` hash (memo key, `core/result_cache.py:216-227`) | data table (`core/datagrams/datagram.py:445-454`) | **No** (`04 I4`, `06 H2`) | identical | +| Function-result identity | `DataFunction.uri` (`core/data_function.py:200-212`) | No (`06 H2`: the uri holds name, output-schema hash and version only) | identical | + +### 3.2 Additive identity: how it is achieved + +The pattern of PRs #336 and #338 adds a term to an identity structure only when the option is +given. Here the pattern needs only one explicit term, the `ArrowTableStream` one; everywhere +else the binding enters as the name in both schema halves, absent in the plain mode. Proof +obligations for ITL-1011: +- every existing golden entry is unchanged, with the fixtures switched to `tag_only_columns` + (§6.2); +- new golden entries pin the referential default. + +A data context version is **not** needed. No algorithm changes, and every new value belongs +to a shape v0.3 cannot build (`01 R1`); the data context key is the global hash version, and +nothing about it moves. + +### 3.3 The default, for an unchanged v0.3 script (spec §12.4, §16.3) + +| | Moves? | Why / evidence | +|---|---|---| +| Source `content_hash`, `pipeline_hash` | moves | schema pair gains the tags as data (`04 I1`, `I3`) | +| System-tag column names | moves | the source schema hash moves (`06 H1`) | +| Source id, source record ids | stays | built from the table hash and row token only | +| Function-node pipeline record ids | moves | the preimage holds system-tag columns, whose names moved | +| Pipeline table path (`/schema:/pdb_v2`), operator cache (`odb_v1`), cached source (`sdb_v1`) | moves | the paths contain the pipeline or content hash | +| Result table path (`/rdb_v2`) | stays | `06 H2`: `('double_y', '872e…', 'v0', 'python.function.v0')` | +| Row `Data` hash for a function that reads no key | stays | the automatic select drops the referenced keys (materializing them), and what remains hashes as in v0.3 (`06 H2`: `True`) | +| A function that reads a key | new | no v0.3 function could read a key under its own name: tags were never data | +| A function that read a `tags_as_data` copy | moves once | the column name is in the hash (`05 M1`) | + +So retagging works as the spec intends (§2.2, §12.2): pipeline records are re-established +under new paths, and results are found again. No migration step is required: the v0.3 +pipeline tables stay where they are, with current layouts, so they are not reported stale and +are not read. The upgrade note names `tag_only_columns` for anyone who wants the old paths. + +One consequence is not covered by the result cache. `pod.process(source)` and +`pod(source, strict=True)` (ITL-807's exact form) now see every key as an extra data column +and raise, unless the function takes all keys. That is decision D2. + +--- + +## 4. Storage (question 3) + +| Table | Columns today | With referential `x` | Layout change | +|---|---|---|---| +| Function pipeline `pdb_v2` (`function_node.py:2337-2461`) | user and system tags; the input's `_source_*` (`:2389-2396`); meta | `x` once, as a tag. `_source_x` once, if `x` was an input | none | +| Function result `rdb_v2` (`result_cache.py:269-347`) | outputs, their `_source_*`, `__input_data_hash`, `__data_context` | unchanged (`x` is never an output, §7) | none | +| Operator cache `odb_v1` (`operator_node.py:930-1019`) | flat `as_table(source, system_tags, context)`, row hash, `__data_context` | `x` once (`02 P0`), plus `_source_x` | none | +| Cached source `sdb_v1` (`cached_source.py:278-311`) | flat source table | `x` once | none | +| Side-effect log `tdb_v2` | ids only | unchanged | none | + +**Getting the binding back.** +- A live node passes `reference_tags()` where it now passes `keys()[0]` + (`operator_node.py:1114-1117`). +- A loaded job computes `tag ∩ data` from the saved descriptor + (`pipeline/graph.py:178-190`; stub at `operator_node.py:235-238`). `SourceProxy` already + returns `x` in both `keys()`. +- `DerivedSource` and `CachedSource` take the binding from the live origin. + +**Stored-schema changes, named exactly.** +- No stored table changes, and the descriptor format is unchanged. +- Source configs gain `tag_only_columns` (§6.2). +- **Version pinning (developer, November 12, 2026: "saved jobs should clearly come with + version info to pin behavior").** Measured (`07`), a saved job today holds: + - `orcapod_pipeline_job_version = "0.1.0"` (written `pipeline/job.py:1356`, checked + `:1422-1426`); + - the embedded pipeline's `orcapod_pipeline_version = "0.1.0"` (`pipeline/graph.py:217`, + checked `:258`); + - a `data_context_key` per node (`std:v1:default`). + + It holds no Orcapod package version, and the source configs in `bindings.sources` carry + none. Both format versions were already `"0.1.0"` at v0.2.0, so they were not bumped + across the v0.2-to-v0.3 hash changes. The context key will not change in v0.4 (§3.2), so + it cannot tell v0.3 from v0.4. +- **The rule (D8, decided by the developer).** + - v0.3.0 itself (ITL-1026) bumps `PIPELINE_JOB_FORMAT_VERSION` and + `PIPELINE_FORMAT_VERSION` from `"0.1.0"` to `"0.2.0"`, and writes an informational + `orcapod_version` field. + - The referential-tag default (the last step of ITL-1012) bumps both to **`"0.3.0"`**, the + first format in which a source may be referential. + - Files at `"0.1.0"` and `"0.2.0"` load with v0.3 meaning: every declared source tag is + tag-only, and the saved content and pipeline hashes match. + - v0.4 accepts `"0.1.0"`, `"0.2.0"` and `"0.3.0"` on load. The ITL-1023 integration branch + writes `"0.3.0"` already. + - A standalone source config (a `to_config()` dict outside a job or pipeline file) carries + no version. If such configs must reload across versions, v0.4 adds a `config_version` + to them. + +**Legacy data.** A v0.3 table or saved job has disjoint `tag`/`data` sets, so it reads as +plain tags only (§16.1). Nothing is reinterpreted. + +--- + +## 5. Operator by operator (question 4) + +Sizes: +- **S**: up to about a day with tests. +- **M**: 2-4 days. +- **L**: a week or more. + +"Rebuild" means passing references to the constructed `ArrowTableStream`; ITL-1010 makes +that change at every site. + +| Operator | Today (measured) | Spec rule | Change | Size | +|---|---|---|---|---| +| `PolarsFilter` (`filters.py:80`) | prototype: data `x` lost from rows (`02 P1`) | preserve (§10) | rebuild | S | +| `DropNulls` (`drop_nulls.py:306`) | prototype: narrows `x` in both roles, so both read `int` | preserve | rebuild (one column, one flag) | S | +| `SemiJoin` (`semijoin.py:130`) | prototype: rows right, data `x` lost | preserve (§10) | rebuild. v0.3 `SemiJoin` already matches overlapping tag and data names (`:76-83`) | S | +| `Batch` (`batch.py:107`) | prototype: tag and data `x` both `list[int]`, in input order | preserve when both are restructured alike (§10.2) | rebuild. Every member column is listed in input order (`:60-100`), and `x` is one column | S | +| `GroupBy` non-key (`group_by.py:258`) | prototype: predicted list in both (correct) | list in both, same order (§8.2-8.3) | rebuild. Members sort on non-key tags (`:192-215`); one column means one order | S | +| `GroupBy` key | prototype: data **predicted `list[int]`** (`02 P4`) | scalar in both (§8.1) | rebuild, plus `unary_output_schema` keeps a referenced key scalar (`:302`). Physically keys are not member columns (`:187`) | S | +| `GroupBy`/`Batch` async | `Duplicate column name 'x'` (`02 P4`) | as sync | `_rows_to_table` (ITL-1010) | — | +| Select/Drop data (`column_selection.py:158`, `:413`) | raise when dropping referenced `x` (`02 P2`) | materialize (§7.2) | §5.3 | S | +| Select/Drop tag (`:37`, `:286`) | the physical column dropped, so data `x` is lost (`02 P2`) | remove tag, keep data (§10) | §5.3 | S | +| `MapTags`/`MapData` | renaming a referenced tag raises (`02 P3`) | decided (§5.4) | §5.4 | M | +| `Index`/`Pick` | §5.5 | §4.5 | §5.5 | S | +| `Join` | §5.1 | §9 | §5.1 | M | +| `MergeJoin` | §5.2 | open (§17) | §5.2 | M | +| Function pod + automatic join and select (ITL-807) | §7 | §6 | §7 | M | + +### 5.1 `Join`: the three cases of §9.1, with equal standing + +**Today.** `Join` joins on **all** shared tag names (`common_tag_keys`, `join.py:202`), inner, +through Polars (`:219`), with key types checked (`:206`; async `:648`). It refuses any shared +data name (`:116`; async `:781`). Because every shared tag is a join key, the condition "the +join established equality" (§9.5, point 3) always holds for a shared referential name. + +| Case | Today (measured) | v0.4 (spec §9.2-9.3) | Code change | +|---|---|---|---| +| Plain with plain | `06 J1`: tags `('subject',)`, data `('b','a')`, pipeline hash `d12c05799eccd043`, four name-extended system tags, sync = async | **byte-identical** | none. Pin the `06 J1` values as a regression test; the golden join cases already cover it | +| Plain with referential | prototype `02 P5`: `output_schema` already right (tag `x`, data `x`); sync read raises `KeyError` in the column layout (`table.select(tag_order + data_order)`, `:238-246`); async rows already right | referential tag; data `x` from the referential side; nothing made from the plain tag | Polars keeps one key column `x`, which *is* the referential side's data. Lay out each name once; the output's references are the inputs' references among shared keys | +| Referential with referential | prototype: `InputValidationError: Datas should not have overlapping keys` | referential tag, one merged data `x` | `_joined_user_schemas` and `_merge_data` exempt a name that is a tag on both inputs and referential on at least one; one key column survives | +| Referential tag with data `x` but no tag `x` | — | error (§9.5) | none: not a key, so an ordinary collision | +| Plain tag `x` with data `x` but no tag `x` | `01 R2`: build accepts; sync read `KeyError`; **async yields rows where tag `x` ≠ data `x`** | invalid (§4.1) | reject (ITL-940) | + +**Provenance of the merged column (§9.4: "must retain the relevant source relationships").** +How the code retains them: +1. **Row level.** The name-extended system-tag columns already keep both inputs' `source_id` + and `record_id` per row (`06 J1` shows four columns), and they are the pipeline + provenance. Nothing changes. +2. **Field level.** Each referential input brings `_source_x`; Polars suffixes the right one + `_source_x_right`. `MergeJoin` already handles this suffix (`merge_join.py:408-409`). + Retaining both relationships means keeping **both tokens**. The one code-level choice is + the column's shape (D5): + - (a) a sorted two-element list, as `MergeJoin` does; + - (b) two scalar columns. +3. **Plain with referential.** Only the referential side has `_source_x`, so there is nothing + to merge. + +**Under the referential default.** Two sources sharing tag columns now meet case 3 on every +join, including the automatic join of `pod(a, b)` (ITL-807), which builds the same `Join`. +The automatic select that follows then materializes the keys the function does not read, so +the function's `Data` is the v0.3 one (§3.3). + +### 5.2 `MergeJoin` (spec §17 open, ITL-1018) + +**From the code.** An inner join on shared tags (`merge_join.py:393`); every shared data name +becomes a sorted `large_list` (`:325`, `:404-420`), with source info reordered to match. +- A shared referential name cannot be list-merged: tag `x` would be scalar beside data + `[v, v]`. +- Materializing first is impossible: a plain tag and data of one name is invalid. + +**Options (D4):** +- (a) **Recommended.** An equated referential name stays one scalar column, as in `Join`; a + referential name colliding with a non-tag data name is rejected. +- (b) Reject every referential input until the user materializes. + +There are no unmatched rows: `how="inner"`. + +### 5.3 Select and drop: roles, not columns (spec §7, ITL-1015) + +`SelectDataColumns`/`DropDataColumns` removing a referenced `x` remove the data role. The +physical column stays as a plain tag, with the same buffer, type and values (§7.2, §4.4). The +tag operators remove the tag role and keep data. The async row paths already behave this way, +because they drop per datagram. + +**Diagnostic (§7.4, Proposed).** +- **Config.** A new `OrcapodConfig` section field + `reference_materialization: Literal["warn", "silent"] = "warn"`, beside `hashing`, `display` + and `datetime` (`config.py:103-112`). No hash reads it. +- **Where it fires.** Once per operator application, from one helper in the operator base that + compares input and output reference sets. It emits a `warnings.warn` category the user can + filter. It is recommended silent for the automatic select of `pod(...)` and at a pod + boundary (D1). +- **Inspection (§7.3, §13.5).** The output schema shows the transition. The same helper + exposes `binding_transitions(...)` → `{name: kept | materialized | removed}` for pipeline + views. + +### 5.4 Renames: DECIDED (developer, November 12, 2026; closes spec §10.1) + +Today `MapTags` and `MapData` rename within their own namespace and refuse a name of the +other one (`01 R4`; `mappers.py:104`, `:254`). With references (prototype `02 P3`), renaming +either side of a reference raises. The decided rules: + +| Call | On a referential tag `x` → `y` | On a plain tag | Diagnostic | +|---|---|---|---| +| `map_tags({x: y})` (default) | renames the column in **both** roles; the reference survives as `y` (one physical column: rename the column, then rename it in both role lists) | as today | none (nothing materializes) | +| `map_tags({x: y}, tag_only=True)` (spelling is the coordinator's; the developer may rename it) | the tag becomes plain `y`, a zero-copy duplicate (`table.append_column(field.with_name(y), table.column(x))`, the `derived_source.py:429-434` idiom); data `x` keeps its name | as today | none (the user asked) | +| `map_data({x: y})` | always a data-only move: data becomes `y`; tag `x` stays as a plain tag under the old name (zero-copy duplicate) | n/a | none (the user asked) | + +A rename that would produce a plain tag and a data column of one name, or collide with an +existing column, raises as today (`mappers.py:104`, `:254`). `tag_only` enters `MapTags`'s +identity only when true, so the identity of a plain-tag rename stays v0.3's. The EDI test +renames `timestamp` to `spike_time` / `gaze_time` with `map_tags`. Size M (ITL-1018, or the +ITL-1023 slice). + +### 5.5 `Index` and `Pick` + +- **`out=None` on a referenced column.** The operator replaces the data in place, which would + leave tag `x` ≠ data `x`. Materializing would create a plain tag and data of one name (§4.3). + So per §4.5, **reject**, naming the remedy (`out=`, or drop or select first). +- **`out=` equal to a tag name.** Accepted at build today; reading fails with + `Duplicate column name 't'` (`05 M2`). Reject at build (ITL-940). + +--- + +## 6. Sources (question 5) + +### 6.1 The referential default + +A source applies the default; the builder stays explicit. +- `SourceStreamBuilder.build(..., reference_tags=...)` takes the list. It computes the schema + hash from the pair with references in both halves, gives source info to referenced columns + (exclude only `tags − references`, `stream_builder.py:336-338`), and passes the list on. +- Each source computes `reference_tags` = its declared tags minus the ones asked for as plain. + +**Every source:** + +| Source | Declared tags today | v0.4 default | +|---|---|---| +| `ArrowTableSource`, `DictSource`, `CSVSource`, `DataFrameSource`, `DeltaTableSource` | `tag_columns` | referential | +| `DBTableSource`, `SQLiteTableSource`, `PostgreSQLTableSource` | `tag_columns`, else primary keys (`db_table_source.py:98-104`; SQLite `rowid` fallback `sqlite_table_source.py:92-96`) | referential (§11.4: keys are a construction policy) | +| `SpiralDBTableSource` | `tag_columns`, else keys (`spiraldb_table_source.py:307-311`) | referential. Keys are non-null already (`:466-486`); a reference keeps the exact field | +| `PollingSource` | `tag_columns` (`polling_source.py:257-301`) | referential; its own identity `(impl.identity(), tag_columns)` (`:351`) gains the plain list only when used | +| `ListSource` | `element_index` or `tag_function` output (`list_source.py:49-76`): **values computed by the source, not columns** | code-ambiguous, D6 | +| `DerivedSource`, `CachedSource`, `SourceProxy` | the origin's or saved roles | keep the origin's bindings; no default applied afresh | + +Two further consequences: +- **All-tags source.** With referential tags the "one data column" rule is met, so it is valid + (ITL-1012). A plain all-tags source still raises, as in v0.3 (`06 H3`). +- **The automatic join and select (ITL-807)** work as described in §5.1 and §3.3. + +### 6.2 The explicit tag-only option: `tag_only_columns` (DECIDED, ITL-1012) + +The developer chose the name and shape (November 12, 2026): +- `tag_columns=[...]` lists **all** tags, referential by default. +- `tag_only_columns=[...]` names the subset kept as tags only (materialized, not data). +- A name in `tag_only_columns` that is not in `tag_columns` raises `ValueError`. +- The option has the same name on every source. +- In prose the spec's terms "materialized" and "referential" stay for the concepts. + +| Need | Call | +|---|---| +| v0.3 behaviour, byte-identical | `ArrowTableSource(t, tag_columns=["k"], tag_only_columns=["k"])` | +| Mixed (§11.3: `lab` a label, `timestamp` referential) | `tag_columns=["lab", "timestamp"], tag_only_columns=["lab"]` | +| DB sources with primary keys as the default tags | `tag_columns=None` gives the keys, referential. The v0.3 form lists the keys in both arguments. A shorthand for "all declared tags" (for example `tag_only_columns="all"`) would save listing them; that is the ITL-1012 worker's proposal to make | + +Implementation: each source computes `reference_tags = tag_columns − tag_only_columns` +and passes it to the builder (§6.1). `to_config` writes `tag_only_columns` only when it is +non-empty. + +**Golden fixtures (`tests/test_hashing/golden/cases.py`).** +- Each of the 15 source constructions with `tag_columns=` gains `tag_only_columns=` naming + the same columns. No fixture moves or is renamed, and every entry stays byte-identical. +- The builder call (`:596`) and the `DerivedSource` case (`:618-633`) need no edit: the + builder is explicit, and `DerivedSource` keeps its origin's binding once the origin fixture + is tag-only. +- `ListSource` (`:1699`) depends on D6. +- New entries pin the referential default for each source kind, `Join` case 3, and `GroupBy` + on a referential key. + +**Upgrade note (changelog fragment of ITL-1012), draft.** "Declared tag columns are now +referential by default: each is also a data column of the same name, stored once, so a +function pod can read it. This changes the schemas and hashes of pipelines that use +`tag_columns`; stored function results are reused. To keep a tag as a label only, with the +v0.3 schemas and hashes, also name it in `tag_only_columns=[...]`." + +### 6.3 `tags_as_data` (spec §11.5, §16: legacy meaning kept, explicit migration) + +**How the code keeps the meaning.** +- `tags_as_data={tag: new_name}` still adds an independent renamed copy + (`derived_source.py:429-434`; Spiral `:375-388`). +- Its identity terms are unchanged (`derived_source.py:274-293`; + `spiraldb_table_source.py:407-421`). +- `validate_tags_as_data` already refuses a name that is a column (`derived_source.py:98-101`, + `01 R6`), so the copy can never collide with a reference. +- Beside the default, a `tags_as_data` copy of a referential tag is simply a second data + column with a different name. + +**Explicit migration (ITL-1020, ITL-1021):** + +| v0.3 | v0.4 | +|---|---| +| `SpiralDBTableSource(..., tags_as_data={"spike_time": "spike_time_us"})` | `SpiralDBTableSource(...)`, with keys referential by default | +| pod parameter `spike_time_us` | pod parameter `spike_time` | + +Functions that read the copy recompute once, because the column name is in the `Data` hash +(`05 M1`). That is spec §16.3: the computational input changed. + +--- + +## 7. Function pods (question 6) + +- **Input binding by name.** It works unchanged (`02 P6`: `uses_x.pod(R)` gives `z = 20.2`, + …). +- **Automatic select (ITL-807, `function_pod.py:669-727`).** For a function that does not + read `x`, the automatic select removes data `x`, and the select's materialization rule + applies (§6.5), as ITL-1013 says. Today it raises on the prototype (`02 P6`); with §5.3 it + materializes. +- **Tags pass through** (`function_pod.py:739-747`). The output data is only the function's + outputs (`:747`), so a reference's target never survives a pod, whether or not it was read. + Measured: `uses_x.pod(R)` outputs tags `(session, x)`, data `(z,)`. Decided (D1): with + `process` the tag becomes plain at the boundary; `pod(...)` uses the fall-through variant of + the pod, and the references stay referential (§11). +- **An output name that collides with a tag (§6.2, Required).** + - Today `output_schema` accepts it (`01 R5`: + `(Schema({'x': int | None}), Schema({'x': float}))`), and reading fails with + `Duplicate column name 'x'`. Under the intersection rule it would look like a reference + with unequal types. + - Implement the rejection in `_FunctionPodBase.output_schema`, for plain and referential + tags, with the spec's message. A `PythonDataFunction`'s output schema is static, so + §6.4's dynamic case does not arise. +- **Replacing plain data with no tag (§6.3).** Unchanged: a pod's outputs are new columns. + +--- + +## 8. The spec's open questions (§17, §9.7) + +| Question | Answer | +|---|---| +| Outer and unmatched-row joins (§9.7) | **The code decides:** none exist. `Join` and `MergeJoin` use `how="inner"` (`join.py:219`, `merge_join.py:393`), `SemiJoin` uses `how="semi"` (`semijoin.py:120`). Record the §9.7 policy in the operator contract for future ones | +| Rename both namespaces | **Decided:** `map_tags` on a referential tag renames both roles; the reference survives (§5.4) | +| Rename one side materializes? | **Decided:** `map_tags(..., tag_only=True)` and `map_data` leave a plain tag (zero-copy), with no warning (§5.4) | +| `Batch` | **The code decides:** every member column becomes a list in input order (`batch.py:60-100`). A reference is one column, so it is preserved (`02 P4`) | +| `MergeJoin` | §5.2; D4 | +| Avoiding duplicate stored values | **The code decides:** one physical column, and rows share buffers (`arrow_table_stream.py:352-353`). Option B, which the code already is; D3 asks only whether to add an explicit marker | +| Database columns referential by default | **Developer decided:** every declared tag of every source (§6.1) | +| `tags_as_data` migration | §6.3 | +| Which hashes change | §3: none in plain mode (golden-proven); additive for references; no data context version | +| Operators relying on unique tags per row | None require it. `GroupBy` documents the expectation and falls back to `record_id` (`group_by.py:71-81`, `:192-215`); `Join` is many-to-many. References do not change this | + +--- + +## 9. Delivery plan (question 8), mapped onto the project's issues + +The project has ITL-1009 to ITL-1023. Recommended order, with changes to the issue set: + +| Order | Issue | Scope from this audit | Size | Depends on | Parallel with | +|---|---|---|---|---|---| +| 0 | **ITL-940** (add to the project) | reject a tag/data name collision across inputs (`Join`, `MergeJoin`, sync and async), pod output = tag name, `Index`/`Pick` `out=` = tag name; regression tests `01 R2`, `R5`, `05 M2` | M | — | — | +| 1 | **ITL-1009 + ITL-1010 (merge)** | `reference_names`, the validator, `StreamBase.reference_tags()`; `ArrowTableStream(reference_tags=)`; its identity term; `_rows_to_table` and `_materialize_*` deduplication; all 32 rebuild sites pass references; spec §15.1 tests | M | 0 | — | +| 2a | **ITL-1011 + ITL-1012, part 1** | builder `reference_tags`; `tag_only_columns` on every source (§6.2); `to_config` per D8; referential sources available but **not yet the default** | M | 1 | 2b-2e | +| 2b | **ITL-1015** | select and drop by role; config, diagnostic, `binding_transitions`; §15.5 tests | S | 1 | 2a, 2c-2e | +| 2c | **ITL-1016** | `Join` three cases, sync and async; field-level provenance per D5; §15.4 tests; `06 J1` pinned | M | 1 | 2a, 2b, 2d, 2e | +| 2d | **ITL-1017** | `GroupBy` key scalar in data; `Batch` tests; §15.3 tests | S | 1 | others | +| 2e | **ITL-1014** | tests only for the filters, `SemiJoin` and `DropNulls`; `Index`/`Pick` in-place rejection | S | 1 | others | +| 3 | **ITL-1013** | pod output collision (shares code with ITL-940); the D1 pod-boundary rule (plain under `process`); automatic join and select with references; the fall-through variant of the pod on `pod(...)` (§11) with its equivalence test; §15.2 and §15.6 result-identity tests | M+M | 1, 2b; 2c and 2d for the automatic join and the fall-through | 4 | +| 4 | **ITL-1019** | nodes, stores and saved jobs pass references; descriptor intersection; reuse across retagging (§15.6) | S | 1 | 3 | +| 4b | **ITL-1012, part 2 (its last step), with ITL-1011** | **flip the default** to referential; golden fixtures switched to `tag_only_columns`, new entries; upgrade note | S | **1013, 1015, 1016, 1017, 1019** | — | +| 5 | **ITL-1018** | `MergeJoin` (D4); the decided rename rules (§5.4), unless the ITL-1023 slice lands them | M | 1 | 6 | +| 6 | **ITL-1020** | `tags_as_data` beside references (§6.3): docs and tests; no behaviour change | S | 2a | 5 | +| 7 | **ITL-1021** | ediop and the notebook, without copies; EDI-shaped integration test (§15.7) | M | 4b | — | +| 8 | **ITL-1022** | operator table in `docs/`; `streams.md:10-25`, `operators.md:31-33`, `sources.md`, `hashing.md`; CLAUDE.md and `.zed/rules` rule | S | all | — | + +**ITL-1023 (workshop slice, go/no-go November 14).** The thinnest path is ITL-1010 (+1009), +1012 (sources only), 1015, 1013, 1016 and 1017, plus guard errors ("not supported with +referential tags yet") in: +- `Batch`, `MergeJoin`, `Index`, `Pick` when an input has references (the rename rules of §5.4 + are in the slice); +- `OperatorNode` reload and `DerivedSource`; +- `pod.process`/`strict=True` with references (D2), if D2 is not settled. + +Each guard is one check against `reference_tags()` at `validate_inputs`. Estimate: about +5-7 working days for one worker, which is tight for November 14. The audit's measurements +(`02`, `06`) give the test shapes. The riskiest part is the 32-site rebuild; it is mechanical +but touches every operator. + +**The default flip is the last step of ITL-1012.** It depends on ITL-1013, 1015, 1016, 1017 +and 1019. Flipped earlier, `main` would break: `pod(source)` (the automatic select raises), +`source_a.join(source_b)` (shared keys collide), `GroupBy` on a key, and reloading a node. +ITL-1023's integration branch carries everything at once and is unaffected. + +**Splits and merges.** Merge ITL-1009 into ITL-1010: with the intersection rule ITL-1009 is a +helper and a validator, testable only with a stream. Land ITL-1011 with ITL-1012: the golden +fixture switch is the proof ITL-1011 asks for, and it needs `tag_only_columns`. ITL-1013 +depends on ITL-1015 (the automatic select). Add ITL-940 to the project as the first issue. + +**Blocked by other issues?** +- **ITL-858 (channels):** no. Rows carry both datagrams, and the intersection rule needs + nothing else. +- **ITL-835 (Arrow types in `Schema`):** no. A reference is one column, so its Arrow types + cannot disagree. +- **ITL-846 (metadata preservation):** no. The binding is not field metadata. +- **ITL-980:** unrelated. +- **ITL-940:** blocks step 1. + +--- + +## 10. Decisions for the developer (most consequential first) + +D0 is stated first, as asked. It is the consequence of the developer's default, for +confirmation. D1 is decided and recorded here; D2 onward are open. + +**D0. The hash consequence of the default (measured, §3.3).** An unchanged v0.3 script gets: +- new source content and pipeline hashes, system-tag names, function-node record ids, and + pipeline/operator/cached-source table paths; +- the same source ids, source record ids, result table paths, and results for every function + that reads no key. + +Plain mode is byte-identical, proven by the golden values with fixtures switched. No data +context version. To confirm: +- (a) **Recommended.** Ship as described. No migration tool; v0.3 tables are left in place, + unread. +- (b) Also provide a one-off tool that re-keys v0.3 pipeline records to the new paths. The + results are reused anyway, so this only saves re-establishing records. + +**D1. A reference after a function pod: DECIDED (developer, November 12, 2026).** Required +§6.1 cannot hold as written, because a pod outputs only its function's outputs +(`function_pod.py:747`). The §6.1 example is conditional ("assuming the output stream +retains the input data `x`"), and in Orcapod it never does. The decision: +- **`pod.process(stream)` and `pod(stream, strict=True)`:** a reference whose data column + does not survive becomes a **plain tag** (same value and type), visible in the output + schema. No automation. The `strict` reading is the coordinator's; the developer may + correct it. +- **`pod(stream)`:** a **variant of the function pod** (or a thin wrapper) processes one + (tag, data) pair at a time. It calls the function exactly as the plain pod does, then emits + the tag unchanged and, as data, the outputs plus the row's own referenced columns. Function + results are shared with the plain pod. Only referential key columns fall through; full + append is deferred (ITL-1025). See §11. +- **The diagnostic** is the coordinator's call. Recommended none at the pod boundary or in + the automatic select: the `reference_materialization` warning is for a select or drop the + user wrote (a rename never warns, §5.4). + +**D2. `pod.process(...)` and `strict=True` with referenced keys (ITL-807).** Under the default, +every key is an extra data column for a function that does not read it. +- (a) **Recommended.** Keep ITL-807's contract: `process` and `strict` reject. `pod(...)` is + the convenient form and materializes. +- (b) `process` and `strict` ignore referenced columns that are not parameters, materializing + them like the automatic select. This is friendlier, but `process` would no longer be the + exact primitive. + +**D3. An explicit binding marker on `Schema`?** (§17 "avoid duplicated values" and +§13.1-13.2.) +- (a) **Recommended.** No. The binding is the name intersection (§4.3), with the build-time + validator; `Schema` and every schema hash are untouched. +- (b) Add a `reference_fields` marker beside `optional_fields`. It is explicit, but it touches + `Schema` equality and hashing (it must stay out of the hash when empty; + `hashing/semantic_hashing/builtin_handlers.py:290-293` shows how `optional_fields` is left + out) and every operator's schema code. + +**D4. `MergeJoin` with references (§17 open).** +- (a) **Recommended.** An equated reference stays one scalar column; a reference colliding with + a non-tag data name is rejected. +- (b) Reject referential inputs. + +**D5. Field-level shape of the merged column's provenance (§9.4 says retain both; the code +needs a shape).** +- (a) **Recommended.** `_source_x` becomes a sorted two-element `large_list`, + as `MergeJoin` does. Downstream `GroupBy` then nests one level, as it does for `MergeJoin` + output. +- (b) Two scalar columns, `_source_x` and a second name. This needs a new naming convention + for source-info columns. + +**D6. `ListSource` under the default.** Its tags are values the source computes +(`element_index`, or `tag_function`), not columns of the user's data. §11.1 speaks of source +*columns*. +- (a) **Recommended.** `ListSource` tags stay plain. Its golden entry is unchanged without a + fixture edit, and the upgrade note says so. +- (b) Referential like every source, with a plain option added to `ListSource`. + +**D7. The tag-only option: DECIDED** (developer, November 12, 2026). It is +`tag_only_columns=[...]`, a subset of `tag_columns` (§6.2). Open for the ITL-1012 worker to +propose: a shorthand for "all declared tags", useful for database sources whose tags default +to primary keys. +**D8. Version pinning of saved files: DECIDED** (developer, November 12, 2026; §4). +- v0.3.0 writes format `"0.2.0"` and `orcapod_version` (ITL-1026). +- The referential default writes `"0.3.0"`. +- `"0.1.0"` and `"0.2.0"` load with v0.3 tag meaning. +- Open: a `config_version` on standalone source configs, if they must reload across + versions. + +--- + +## 11. The two rules settled late (developer, November 12) + +### B. `GroupBy`: no change + +Spec §8.2 stands: a non-key referential tag becomes the same list in both roles and stays +referential, and plain tags behave as in v0.3 (§5). `Join` gets no special refusal for a +shared list-valued tag. `Join` matches same-named tags whatever their type, and streams +whose same-named tags mean different things must be renamed. What `Join` does today with a +shared list-valued tag (correct match, wrong match, or a join-engine error, sync and async) +is for the ITL-1023 worker to measure and report. + +### A. Pod boundary: references fall through on `pod(...)`, as a variant of the pod + +The developer's ruling: the fall-through is an abbreviated form of a legal composition, done +by a variant of the function pod. It is **not** a join and **not** an inserted operator. + +**The composition it abbreviates** (each step obeys the boundary: no tag enters a function; +no operator makes data that is not existing data): +1. Add a unique temporary tag per row. +2. Send one copy through the plain pod. The automatic select makes the unread references + plain; the read ones are consumed, and either way the pod's output has them as plain tags. +3. Join the pod's output back onto the input, restricted to its tags and referenced columns, + on the temporary tag. This is `Join` case 2, plain with referential (§5.1), so the + referenced data comes from the input's data column. +4. Drop the temporary tag. + +**The variant, one (tag, data) pair at a time** (`pod(stream)`, the `__call__` path that +already adds ITL-807's automatic join and select): +- **The function sees exactly what the plain pod gives it.** The variant projects the row's + `Data` to the function's parameters and runs the plain pod's per-row path on that + projection. The memo key (`result_cache.py:217`) and the result store (`rdb_v2` at the + function uri) are therefore the plain pod's, so results are shared. +- **What it emits.** The input tag unchanged (system tags included, as the plain pod does, + `function_pod.py:1043-1054`). As data: the function's outputs plus the row's referenced + columns, with their `_source_*` taken from the row's `Data`. +- **What carries the referenced columns.** ITL-807's automatic select must keep referenced + columns in the variant's input. Only non-referenced extra columns are dropped. +- **Rows.** Row count and order are the pod's: a row whose function returned nothing is + skipped, as in the plain pod. There is no real join, no extended system-tag name, and no + row multiplication from duplicate user tags. + +**Consequences:** +- **Output schema.** Data = outputs ∪ referenced input columns. Tags = input tags. The + references of the output = those of the input. +- **A function output named like a referenced column (or any tag)** still raises (§7). +- **Pipeline identity.** It differs from the plain pod's **only when references are + present**. Add the term `("reference_fall_through", sorted(names))` to the pod stream's or + node's identity only when the set is non-empty (the #336/#338 pattern). With no referential + tag in the input, the variant is indistinguishable from the plain pod: same schema, same + hashes, same node path and stored tables, no golden move. +- **Function-result identity and the result store.** Shared with the plain pod. A retagged + pipeline, or a `process` call over the same rows, finds the same results. +- **`FunctionNode` and the job path.** + - The pipeline record (`pdb_v2`, `function_node.py:2337-2461`) stores the row's tags, + which include `x`, plus the input's `_source_*` columns (`:2390-2396`). + - For the variant those must be the `_source_*` of the full input `Data`, referenced + columns included, not only the projection's, so `_source_x` is kept. + - The value of `x` is stored once, as the tag column. On read the node gives that column + its data role back (`reference_tags` = the input's references, `function_node.py:2891`). + - Nothing is stored twice, and the table is at its own `pipeline_hash` path whenever + references are present. +- **`strict=True`.** It uses the plain pod, as `process` does (the coordinator's reading of + ITL-807; the developer may correct it). +- **Size: M**, in ITL-1013, after ITL-1016, because its equivalence test uses `Join` case 2. + +**Tests:** +- **Equivalence.** On a source with duplicate user tags (`session=s1` on two rows, `x` + referential, `y` data), compare `pod(source)` with the composition above, built with a + temporary tag-only row-id column, `pod.process`, `Join` and `DropTagColumns`. User tags, + data, references and rows must match. System-tag *names* differ by construction (the + composition's `Join` extends them), so compare those separately: the variant keeps the + pod's names. +- **No references.** `pod(source)` over a tag-only source equals today's `pod(source)` in + schema, `pipeline_hash`, `content_hash`, node path and stored tables. +- **Result sharing.** Run `pod(source)` and then `pod.process(source)` (or the reverse) over + one result store. The second run computes nothing. +- **Job path.** Saved, reloaded and read back, the variant node gives the same table with + `x` referential. Both sync and async. + +--- + +## Appendix — out-of-scope findings (reported to the coordinator, not fixed) + +1. **Async `Join` of tag `x` with data `x` from another input** yields rows where the two + differ, silently. Sync raises `KeyError` on read (`01 R2`). This extends ITL-940 to async. +2. **`Index(out=)`** is accepted at build and fails on read (`05 M2`). This extends + ITL-940. diff --git a/superpowers/specs/itl-1004-audit/01_today_same_name.out b/superpowers/specs/itl-1004-audit/01_today_same_name.out new file mode 100644 index 000000000..892fbf2af --- /dev/null +++ b/superpowers/specs/itl-1004-audit/01_today_same_name.out @@ -0,0 +1,81 @@ +=== R1. A stream cannot name one column as both tag and data +--- ArrowTableSource(x, y; tag_columns=[x]) + keys: tags=('x',) data=('y',) + schema: tags={'x': int | None} data={'y': float | None} + as_table columns: ['x', 'y'] + rows: [({'x': 1}, {'y': 0.1}), ({'x': 2}, {'y': 0.2})] +--- ArrowTableStream over a table with two physical columns named x, tag_columns=[x] + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) + +=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape) +--- Join().output_schema(A, B) + (Schema({'k': int | None, 'x': int | None}), Schema({'x': int | None, 'b': float | None, 'a': float | None})) +--- Join()(A, B) read + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- Join async rows + [({'k': 1, 'x': 1}, {'x': 1, 'b': 0.5, 'a': 0.1}), ({'k': 2, 'x': 1}, {'x': 2, 'b': 0.6, 'a': 0.1}), ({'k': 1, 'x': 2}, {'x': 1, 'b': 0.5, 'a': 0.2}), ({'k': 2, 'x': 2}, {'x': 2, 'b': 0.6, 'a': 0.2})] +--- MergeJoin()(A, B) read + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- SemiJoin()(A, B) read + keys: tags=('x',) data=('a',) + schema: tags={'x': int | None} data={'a': float | None} + as_table columns: ['x', 'a'] + rows: [({'x': 1}, {'a': 0.1}), ({'x': 2}, {'a': 0.2})] + +=== R3. Join: both inputs have DATA x (the collision) +--- Join().output_schema(C, D) + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'v': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- Join async rows + RAISES ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) + (raised at /Users/eywalker/.local/share/uv/python/cpython-3.12.15-macos-aarch64-none/lib/python3.12/asyncio/taskgroups.py:164) +--- MergeJoin()(C, D) read (colliding data -> list) + keys: tags=('u',) data=('x', 'v') + schema: tags={'u': int | None} data={'x': list[int | None], 'v': float | None} + as_table columns: ['u', 'x', 'v'] + rows: [({'u': 5}, {'x': [1, 1], 'v': 1.0}), ({'u': 6}, {'x': [2, 2], 'v': 2.0})] + +=== R4. MapTags / MapData renaming onto a name of the other namespace +--- MapTags({x: y})(E) + RAISES InputValidationError: Renaming {'x': 'y'} would cause collisions with existing columns: overlapping data columns: {'y'}. + (raised at orcapod/core/operators/mappers.py:254) +--- MapData({y: x})(E) + RAISES InputValidationError: Renaming {'y': 'x'} would cause collisions with existing columns: overlapping tag columns: {'x'}. + (raised at orcapod/core/operators/mappers.py:104) + +=== R5. Function pod output named like an input TAG +--- make_x.pod.output_schema(E) + (Schema({'x': int | None}), Schema({'x': float})) +--- make_x.pod(E) read + RAISES ValueError: Duplicate column name 'x' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) + +=== R6. DerivedSource(tags_as_data={x: x}) (same name) +--- DerivedSource(E, tags_as_data={'x': 'x'}) + RAISES InputValidationError: tags_as_data: cannot copy tag 'x' to 'x', which is already a column. Choose a new name. + (raised at orcapod/core/sources/derived_source.py:99) +--- DerivedSource(E, tags_as_data={'x': 'x_copy'}) + keys: tags=('x',) data=('y', 'x_copy') + schema: tags={'x': int | None} data={'y': float | None, 'x_copy': int | None} + as_table columns: ['x', 'y', 'x_copy'] + rows: [({'x': 1}, {'y': 0.1, 'x_copy': 1}), ({'x': 2}, {'y': 0.2, 'x_copy': 2})] + +=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x) +--- GroupBy(by=[t_unit])(G) + keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp') + schema: tags={'t_unit': int | None, 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]} + as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp'] + rows: [({'t_unit': 1, 't_time': [10, 30]}, {'unit': [1, 1], 'time': [10, 30], 'amp': [0.1, 0.3]}), ({'t_unit': 2, 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})] +--- Batch(batch_size=2)(G) + keys: tags=('t_unit', 't_time') data=('unit', 'time', 'amp') + schema: tags={'t_unit': list[int | None], 't_time': list[int | None]} data={'unit': list[int | None], 'time': list[int | None], 'amp': list[float | None]} + as_table columns: ['t_unit', 't_time', 'unit', 'time', 'amp'] + rows: [({'t_unit': [1, 1], 't_time': [30, 10]}, {'unit': [1, 1], 'time': [30, 10], 'amp': [0.3, 0.1]}), ({'t_unit': [2], 't_time': [20]}, {'unit': [2], 'time': [20], 'amp': [0.2]})] + +=== R8. Function pod with a tag name as a parameter (tags are not inputs) +--- uses_tag.pod(E) (x is a tag of E) + RAISES ValueError: Incoming data schema Schema({'y': float | None}) is not compatible with expected input schema Schema({'x': int | None}): column 'y' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) diff --git a/superpowers/specs/itl-1004-audit/01_today_same_name.py b/superpowers/specs/itl-1004-audit/01_today_same_name.py new file mode 100644 index 000000000..61d2d2fa5 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/01_today_same_name.py @@ -0,0 +1,106 @@ +"""ITL-1004 audit, measurement 01: what v0.3 code does today when a tag and a data column +share a name, anywhere it can be built through the public interface. + +Run: uv run python superpowers/specs/itl-1004-audit/01_today_same_name.py +Output: 01_today_same_name.out +""" + +from __future__ import annotations + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod import function_pod +from orcapod.core.streams import ArrowTableStream +from orcapod.operators import ( + Batch, + GroupBy, + Join, + MapData, + MapTags, + MergeJoin, + SemiJoin, +) +from orcapod.sources import DerivedSource + +print("=== R1. A stream cannot name one column as both tag and data") +one = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "one") +show("ArrowTableSource(x, y; tag_columns=[x])", lambda: describe(one)) + +dup = pa.table([i64(1, 2), i64(1, 2), f64(0.1, 0.2)], names=["x", "x", "y"]) +show( + "ArrowTableStream over a table with two physical columns named x, tag_columns=[x]", + lambda: describe(ArrowTableStream(dup, tag_columns=["x"])), +) + +print("\n=== R2. Join: input A has TAG x, input B has DATA x (ITL-940 shape)") +a = src({"x": i64(1, 2), "a": f64(0.1, 0.2)}, ["x"], "A") +b = src({"k": i64(1, 2), "x": i64(1, 2), "b": f64(0.5, 0.6)}, ["k"], "B") +show("Join().output_schema(A, B)", lambda: Join().output_schema(a, b)) +show("Join()(A, B) read", lambda: describe(Join()(a, b))) +show("Join async rows", lambda: async_rows(Join(), a, b)) +show("MergeJoin()(A, B) read", lambda: describe(MergeJoin()(a, b))) +show("SemiJoin()(A, B) read", lambda: describe(SemiJoin()(a, b))) + +print("\n=== R3. Join: both inputs have DATA x (the collision)") +c = src({"x": i64(1, 2), "u": i64(5, 6)}, ["u"], "C") +d = src({"x": i64(1, 2), "u": i64(5, 6), "v": f64(1.0, 2.0)}, ["u"], "D") +show("Join().output_schema(C, D)", lambda: Join().output_schema(c, d)) +show("Join async rows", lambda: async_rows(Join(), c, d)) +show( + "MergeJoin()(C, D) read (colliding data -> list)", + lambda: describe(MergeJoin()(c, d)), +) + +print("\n=== R4. MapTags / MapData renaming onto a name of the other namespace") +e = src({"x": i64(1, 2), "y": f64(0.1, 0.2)}, ["x"], "E") +show("MapTags({x: y})(E)", lambda: describe(MapTags({"x": "y"})(e))) +show("MapData({y: x})(E)", lambda: describe(MapData({"y": "x"})(e))) + + +print("\n=== R5. Function pod output named like an input TAG") + + +@function_pod(output_keys="x") +def make_x(y: float | None) -> float: + return (y or 0.0) * 2 + + +show("make_x.pod.output_schema(E)", lambda: make_x.pod.output_schema(e)) +show("make_x.pod(E) read", lambda: describe(make_x.pod(e))) + +print("\n=== R6. DerivedSource(tags_as_data={x: x}) (same name)") +show( + "DerivedSource(E, tags_as_data={'x': 'x'})", + lambda: describe(DerivedSource(e, tags_as_data={"x": "x"})), +) +show( + "DerivedSource(E, tags_as_data={'x': 'x_copy'})", + lambda: describe(DerivedSource(e, tags_as_data={"x": "x_copy"})), +) + +print("\n=== R7. GroupBy / Batch over the 'copy' shape (tag t_x, data x)") +g = src( + { + "t_unit": i64(1, 1, 2), + "t_time": i64(30, 10, 20), + "unit": i64(1, 1, 2), + "time": i64(30, 10, 20), + "amp": f64(0.3, 0.1, 0.2), + }, + ["t_unit", "t_time"], + "G", +) +show("GroupBy(by=[t_unit])(G)", lambda: describe(GroupBy(by=["t_unit"])(g))) +show("Batch(batch_size=2)(G)", lambda: describe(Batch(batch_size=2)(g))) + +print("\n=== R8. Function pod with a tag name as a parameter (tags are not inputs)") + + +@function_pod(output_keys="z") +def uses_tag(x: int | None) -> int | None: + return x + + +show("uses_tag.pod(E) (x is a tag of E)", lambda: describe(uses_tag.pod(e))) +_ = s diff --git a/superpowers/specs/itl-1004-audit/02_prototype_unified.out b/superpowers/specs/itl-1004-audit/02_prototype_unified.out new file mode 100644 index 000000000..6776a4943 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/02_prototype_unified.out @@ -0,0 +1,138 @@ +=== P0. The prototype stream R: tags session, x(ref); data x, y +--- describe(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'x': 20, 'y': 0.2}), ({'session': 's1', 'x': 10}, {'x': 10, 'y': 0.1}), ({'session': 's2', 'x': 30}, {'x': 30, 'y': 0.3})] +--- R.pipeline_hash() + semantic_v1:cc818c22c966b6399a83946db7fd429c71819ad4524a7f74f265b101cfb3746c +--- R.as_table(all_info=True) columns + ['session', 'x', 'y', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_content_hash', '_context_key', '_source_y', '_source_x'] + +=== P1. Row filters (spec: preserve) +--- PolarsFilter(constraints={'x': 20})(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2})] +--- DropNulls(['x'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': } data={'x': , 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2}), ({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] +--- DropNulls()(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': , 'x': } data={'x': , 'y': } + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 20}, {'y': 0.2}), ({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] +--- SemiJoin()(R, M[tag x materialized]) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': 10}, {'y': 0.1}), ({'session': 's2', 'x': 30}, {'y': 0.3})] + +=== P2. Select / Drop (spec: removing the target materializes the tag) +--- SelectDataColumns(['y'])(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- DropDataColumns(['x'])(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- DropTagColumns(['x'])(R) (spec: data x kept) + keys: tags=('session',) data=('x', 'y') + schema: tags={'session': str | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'y'] + rows: [({'session': 's1'}, {'y': 0.2}), ({'session': 's1'}, {'y': 0.1}), ({'session': 's2'}, {'y': 0.3})] +--- SelectTagColumns(['session'])(R) + keys: tags=('session',) data=('x', 'y') + schema: tags={'session': str | None} data={'x': int | None, 'y': float | None} + as_table columns: ['session', 'y'] + rows: [({'session': 's1'}, {'y': 0.2}), ({'session': 's1'}, {'y': 0.1}), ({'session': 's2'}, {'y': 0.3})] + +=== P3. Renames +--- MapTags({'x': 'x2'})(R) + RAISES ValueError: Specified tag columns {'x2'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- MapData({'x': 'x2'})(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- MapTags({'session': 'sess'})(R) + keys: tags=('sess', 'x') data=('x', 'y') + schema: tags={'sess': str | None, 'x': int | None} data={'x': int | None, 'y': float | None} + as_table columns: ['sess', 'x', 'y'] + rows: [({'sess': 's1', 'x': 20}, {'y': 0.2}), ({'sess': 's1', 'x': 10}, {'y': 0.1}), ({'sess': 's2', 'x': 30}, {'y': 0.3})] + +=== P4. GroupBy (spec: key scalar in both roles, non-key list in both, one order) +--- GroupBy(['x'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': list[str | None], 'x': int | None} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': ['s1'], 'x': 10}, {'y': [0.1]}), ({'session': ['s1'], 'x': 20}, {'y': [0.2]}), ({'session': ['s2'], 'x': 30}, {'y': [0.3]})] +--- GroupBy(['x']).output_schema(R) (prediction) + (Schema({'session': list[str | None], 'x': int | None}), Schema({'x': list[int | None], 'y': list[float | None]})) +--- GroupBy(['session'])(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': str | None, 'x': list[int | None]} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': 's1', 'x': [10, 20]}, {'y': [0.1, 0.2]}), ({'session': 's2', 'x': [30]}, {'y': [0.3]})] +--- GroupBy(['session']).output_schema(R) + (Schema({'session': str | None, 'x': list[int | None]}), Schema({'x': list[int | None], 'y': list[float | None]})) +--- GroupBy(['session']) async rows + RAISES ValueError: Duplicate column name 'x' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) +--- Batch(2)(R) + keys: tags=('session', 'x') data=('x', 'y') + schema: tags={'session': list[str | None], 'x': list[int | None]} data={'x': list[int | None], 'y': list[float | None]} + as_table columns: ['session', 'x', 'y'] + rows: [({'session': ['s1', 's1'], 'x': [20, 10]}, {'y': [0.2, 0.1]}), ({'session': ['s2'], 'x': [30]}, {'y': [0.3]})] + +=== P5. Join +--- Join().output_schema(R, M) + (Schema({'x': int | None, 'session': str | None}), Schema({'w': float | None, 'x': int | None, 'y': float | None})) +--- Join()(R, M) [ref x + materialized x] + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) +--- Join async (R, M) + [({'x': 10, 'session': 's1'}, {'w': 1.0, 'x': 10, 'y': 0.1}), ({'x': 30, 'session': 's2'}, {'w': 3.0, 'x': 30, 'y': 0.3})] +--- Join().output_schema(R, R2) [ref + ref] + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'z': float | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- Join()(R, R2) + RAISES InputValidationError: Input streams are not compatible: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'z': float | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:46) +--- Join().output_schema(R, N[data x, no tag x]) + RAISES InputValidationError: Datas should not have overlapping keys, but KeysView(Schema({'x': int | None, 'y': float | None})) found in ArrowTableSource and ArrowTableSource. + (raised at orcapod/core/operators/join.py:116) +--- MergeJoin()(R, R2) + RAISES KeyError: 'Field "x" exists 2 times in schema' + (raised at pyarrow/table.pxi:1735) + +=== P6. Function pods +--- uses_x.pod(R) + keys: tags=('session', 'x') data=('z',) + schema: tags={'session': str | None, 'x': int | None} data={'z': } + as_table columns: ['session', 'x', 'z'] + rows: [({'session': 's1', 'x': 20}, {'z': 20.2}), ({'session': 's1', 'x': 10}, {'z': 10.1}), ({'session': 's2', 'x': 30}, {'z': 30.3})] +--- uses_y.pod(R) (automatic select drops data x) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- uses_y.pod.process(R) (strict: extra data x) + RAISES ValueError: Incoming data schema Schema({'x': int | None, 'y': float | None}) is not compatible with expected input schema Schema({'y': float | None}): column 'x' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) +--- writes_x.pod.output_schema(R) + RAISES ValueError: Incoming data schema Schema({'x': int | None, 'y': float | None}) is not compatible with expected input schema Schema({'y': float | None}): column 'x' is not a parameter of the function + (raised at orcapod/core/function_pod.py:259) +--- writes_x.pod(R) + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) +--- data hash of the row passed to uses_y: same with and without the reference? + RAISES ValueError: Specified tag columns {'x'} are not present in the table. + (raised at orcapod/core/streams/arrow_table_stream.py:104) + +=== P7. Index over a grouped referential list +--- Index('x', 0)(GroupBy(session)(R)) + RAISES KeyError: 'x' + (raised at orcapod/core/datagrams/datagram.py:334) +--- Index('x', 0, out='x0')(...) + RAISES KeyError: 'x' + (raised at orcapod/core/datagrams/datagram.py:334) diff --git a/superpowers/specs/itl-1004-audit/02_prototype_unified.py b/superpowers/specs/itl-1004-audit/02_prototype_unified.py new file mode 100644 index 000000000..b7b2b2cd8 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/02_prototype_unified.py @@ -0,0 +1,201 @@ +"""ITL-1004 audit, measurement 02: run every operator over a PROTOTYPE referential stream. + +The prototype is today's ``ArrowTableStream`` with one physical column ``x`` listed both as +a tag and as a data column (the "name in both lists" representation). It is built by +setting the stream's private column lists after construction: this is a measurement of +what today's operator code does with such a stream, not a proposed implementation. Nothing +under ``src/`` is changed. + +Run: uv run python superpowers/specs/itl-1004-audit/02_prototype_unified.py +Output: 02_prototype_unified.out +""" + +from __future__ import annotations + +from typing import Any + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod import function_pod +from orcapod.operators import ( + Batch, + DropDataColumns, + DropNulls, + DropTagColumns, + GroupBy, + Index, + Join, + MapData, + MapTags, + MergeJoin, + PolarsFilter, + SelectDataColumns, + SelectTagColumns, + SemiJoin, +) +from orcapod.system_constants import constants + + +def make_ref(source: Any, refs: list[str]) -> Any: + """Turn tags ``refs`` of ``source`` into referential tags (prototype, private attrs). + + The physical column stays single. It is added to the data column list and given a + source-info column, as a data column would have. + """ + st = source._stream # the source's inner ArrowTableStream + names = st._table.column_names + st._data_columns = tuple(c for c in names if c in refs or c in st._data_columns) + st._data_schema = pa.schema( + f for f in st._table.schema if f.name in st._data_columns + ) + for r in refs: + col = f"{constants.SOURCE_PREFIX}{r}" + if col not in st._source_info_table.column_names: + n = st._table.num_rows + st._source_info_table = st._source_info_table.append_column( + pa.field(col, pa.large_string(), nullable=False), + pa.array( + [f"{source.source_id}::row_{i}::{r}" for i in range(n)], + pa.large_string(), + ), + ) + return source + + +def ref(columns: dict[str, Any], tags: list[str], refs: list[str], sid: str) -> Any: + return make_ref(src(columns, tags, sid), refs) + + +R = ref( + { + "session": s("s1", "s1", "s2"), + "x": i64(20, 10, 30), + "y": f64(0.2, 0.1, 0.3), + }, + ["session", "x"], + ["x"], + "R", +) +print("=== P0. The prototype stream R: tags session, x(ref); data x, y") +show("describe(R)", lambda: describe(R)) +show("R.pipeline_hash()", lambda: R.pipeline_hash().to_string()) +show( + "R.as_table(all_info=True) columns", lambda: R.as_table(all_info=True).column_names +) + +print("\n=== P1. Row filters (spec: preserve)") +show( + "PolarsFilter(constraints={'x': 20})(R)", + lambda: describe(PolarsFilter(constraints={"x": 20})(R)), +) +show("DropNulls(['x'])(R)", lambda: describe(DropNulls(["x"])(R))) +show("DropNulls()(R)", lambda: describe(DropNulls()(R))) +M = src({"x": i64(10, 30), "w": f64(1.0, 3.0)}, ["x"], "M") +show("SemiJoin()(R, M[tag x materialized])", lambda: describe(SemiJoin()(R, M))) + +print("\n=== P2. Select / Drop (spec: removing the target materializes the tag)") +show("SelectDataColumns(['y'])(R)", lambda: describe(SelectDataColumns(["y"])(R))) +show("DropDataColumns(['x'])(R)", lambda: describe(DropDataColumns(["x"])(R))) +show( + "DropTagColumns(['x'])(R) (spec: data x kept)", + lambda: describe(DropTagColumns(["x"])(R)), +) +show( + "SelectTagColumns(['session'])(R)", + lambda: describe(SelectTagColumns(["session"])(R)), +) + +print("\n=== P3. Renames") +show("MapTags({'x': 'x2'})(R)", lambda: describe(MapTags({"x": "x2"})(R))) +show("MapData({'x': 'x2'})(R)", lambda: describe(MapData({"x": "x2"})(R))) +show( + "MapTags({'session': 'sess'})(R)", lambda: describe(MapTags({"session": "sess"})(R)) +) + +print( + "\n=== P4. GroupBy (spec: key scalar in both roles, non-key list in both, one order)" +) +show("GroupBy(['x'])(R)", lambda: describe(GroupBy(["x"])(R))) +show( + "GroupBy(['x']).output_schema(R) (prediction)", + lambda: GroupBy(["x"]).output_schema(R), +) +show("GroupBy(['session'])(R)", lambda: describe(GroupBy(["session"])(R))) +show( + "GroupBy(['session']).output_schema(R)", + lambda: GroupBy(["session"]).output_schema(R), +) +show("GroupBy(['session']) async rows", lambda: async_rows(GroupBy(["session"]), R)) +show("Batch(2)(R)", lambda: describe(Batch(batch_size=2)(R))) + +print("\n=== P5. Join") +show("Join().output_schema(R, M)", lambda: Join().output_schema(R, M)) +show("Join()(R, M) [ref x + materialized x]", lambda: describe(Join()(R, M))) +show("Join async (R, M)", lambda: async_rows(Join(), R, M)) +R2 = ref( + {"x": i64(10, 20), "z": f64(5.0, 6.0)}, + ["x"], + ["x"], + "R2", +) +show("Join().output_schema(R, R2) [ref + ref]", lambda: Join().output_schema(R, R2)) +show("Join()(R, R2)", lambda: describe(Join()(R, R2))) +N = src({"k": i64(1, 2), "x": i64(10, 20)}, ["k"], "N") +show("Join().output_schema(R, N[data x, no tag x])", lambda: Join().output_schema(R, N)) +show("MergeJoin()(R, R2)", lambda: describe(MergeJoin()(R, R2))) + +print("\n=== P6. Function pods") + + +@function_pod(output_keys="z") +def uses_x(x: int | None, y: float | None) -> float: + return float(x or 0) + (y or 0.0) + + +@function_pod(output_keys="z") +def uses_y(y: float | None) -> float: + return (y or 0.0) * 2 + + +@function_pod(output_keys="x") +def writes_x(y: float | None) -> float: + return (y or 0.0) * 2 + + +show("uses_x.pod(R)", lambda: describe(uses_x.pod(R))) +show("uses_y.pod(R) (automatic select drops data x)", lambda: describe(uses_y.pod(R))) +show( + "uses_y.pod.process(R) (strict: extra data x)", + lambda: describe(uses_y.pod.process(R)), +) +show("writes_x.pod.output_schema(R)", lambda: writes_x.pod.output_schema(R)) +show("writes_x.pod(R)", lambda: describe(writes_x.pod(R))) +show( + "data hash of the row passed to uses_y: same with and without the reference?", + lambda: ( + [ + d.content_hash().to_string() + for _, d in SelectDataColumns(["y"])(R).iter_data() + ] + == [ + d.content_hash().to_string() + for _, d in SelectDataColumns(["y"])( + src( + { + "session": s("s1", "s1", "s2"), + "x": i64(20, 10, 30), + "y": f64(0.2, 0.1, 0.3), + }, + ["session", "x"], + "R", + ) + ).iter_data() + ] + ), +) + +print("\n=== P7. Index over a grouped referential list") +GR = GroupBy(["session"])(R) +show("Index('x', 0)(GroupBy(session)(R))", lambda: describe(Index("x", 0)(GR))) +show("Index('x', 0, out='x0')(...)", lambda: describe(Index("x", 0, out="x0")(GR))) diff --git a/superpowers/specs/itl-1004-audit/04_identity.out b/superpowers/specs/itl-1004-audit/04_identity.out new file mode 100644 index 000000000..ddaae0c42 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/04_identity.out @@ -0,0 +1,34 @@ +=== I1. pipeline_hash of a root = hash((tag_schema, data_schema)) +--- x tag only: output_schema / pipeline_hash + (Schema({'session': str | None, 'x': int | None}), Schema({'y': float | None})) + b9e64f95eca6847a +--- x data only: output_schema / pipeline_hash + (Schema({'session': str | None}), Schema({'x': int | None, 'y': float | None})) + 95a52202a35fa95b +--- x referential: output_schema / pipeline_hash + (Schema({'session': str | None, 'x': int | None}), Schema({'x': int | None, 'y': float | None})) + cc818c22c966b639 + +=== I2. Schema-pair hash: the three shapes are distinct, no extra identity term +--- hash_object((tag_schema, data_schema)) + tag{x} data{y}: e3b13da3a40743d1 + tag{} data{x,y}: 8ccaa63afa168bd3 + tag{x} data{x,y}: 3d4503d18110c78b + +=== I3. Root content_hash = (class, output_schema, source_id): sees the binding +--- content hashes (same source_id 'S') + tag only: 006efb368b7aec35 + data only: ca14dfaa13bab7e2 + referential: c53cd5c9cfed363d + +=== I4. Function-result identity: the Data hash a function would receive +--- per-row Data content hash, x untagged vs x referential (data columns x, y) + True +--- per-row Tag content hash, x materialized vs x referential (tag columns session, x) + True + +=== I5. Record-id preimage input = tag.as_table(system_tags) + input data hash +--- tag table of row 0, x materialized vs x referential: same columns? + (['session', 'x', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4'], ['session', 'x', '_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4']) +--- system tag column names (embed the source schema hash, computed at build) + (('_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4'), ('_tag_source_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4', '_tag_record_id::b9e64f95eca6847ae34f9d4993c3baea005192b7d2f8a7fc545cbc2c2b4ac8c4')) diff --git a/superpowers/specs/itl-1004-audit/04_identity.py b/superpowers/specs/itl-1004-audit/04_identity.py new file mode 100644 index 000000000..8b10a3623 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/04_identity.py @@ -0,0 +1,127 @@ +"""ITL-1004 audit, measurement 04: which identities would see a referential binding. + +Uses the prototype of 02 (one physical column ``x`` listed as tag and data, set through +private attributes after construction). Nothing under ``src/`` changes. + +Run: uv run python superpowers/specs/itl-1004-audit/04_identity.py +Output: 04_identity.out +""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path + +from _common import f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod.contexts import resolve_context +from orcapod.types import Schema + +_spec = importlib.util.spec_from_file_location( + "proto", Path(__file__).with_name("02_prototype_unified.py") +) +make_ref = None # loaded lazily below so 02's prints do not run here + + +def _make_ref(): + import io + import contextlib + + assert _spec is not None and _spec.loader is not None + mod = importlib.util.module_from_spec(_spec) + with contextlib.redirect_stdout(io.StringIO()): + _spec.loader.exec_module(mod) + return mod.make_ref + + +make_ref = _make_ref() +hasher = resolve_context(None).semantic_hasher + + +def cols(): + return {"session": s("s1", "s2"), "x": i64(10, 20), "y": f64(0.1, 0.2)} + + +TAGGED = src(cols(), ["session", "x"], "S") # v0.3: x materialized tag only +UNTAGGED = src(cols(), ["session"], "S") # v0.3: x data only +REF = make_ref(src(cols(), ["session", "x"], "S"), ["x"]) # prototype: x tag and data + +print("=== I1. pipeline_hash of a root = hash((tag_schema, data_schema))") +for name, st in [ + ("x tag only", TAGGED), + ("x data only", UNTAGGED), + ("x referential", REF), +]: + show( + f"{name}: output_schema / pipeline_hash", + lambda st=st: f"{st.output_schema()}\n{st.pipeline_hash().to_hex(16)}", + ) + +print( + "\n=== I2. Schema-pair hash: the three shapes are distinct, no extra identity term" +) +pairs = { + "tag{x} data{y}": (Schema({"session": str, "x": int}), Schema({"y": float})), + "tag{} data{x,y}": (Schema({"session": str}), Schema({"x": int, "y": float})), + "tag{x} data{x,y}": ( + Schema({"session": str, "x": int}), + Schema({"x": int, "y": float}), + ), +} +show( + "hash_object((tag_schema, data_schema))", + lambda: "\n".join( + f"{k}: {hasher.hash_object(v).to_hex(16)}" for k, v in pairs.items() + ), +) + +print( + "\n=== I3. Root content_hash = (class, output_schema, source_id): sees the binding" +) +show( + "content hashes (same source_id 'S')", + lambda: "\n".join( + f"{n}: {st.content_hash().to_hex(16)}" + for n, st in [ + ("tag only", TAGGED), + ("data only", UNTAGGED), + ("referential", REF), + ] + ), +) + +print("\n=== I4. Function-result identity: the Data hash a function would receive") +show( + "per-row Data content hash, x untagged vs x referential (data columns x, y)", + lambda: ( + [d.content_hash().to_hex(16) for _, d in UNTAGGED.iter_data()] + == [d.content_hash().to_hex(16) for _, d in REF.iter_data()] + ), +) +show( + "per-row Tag content hash, x materialized vs x referential (tag columns session, x)", + lambda: ( + [t.content_hash().to_hex(16) for t, _ in TAGGED.iter_data()] + == [t.content_hash().to_hex(16) for t, _ in REF.iter_data()] + ), +) + +print( + "\n=== I5. Record-id preimage input = tag.as_table(system_tags) + input data hash" +) +show( + "tag table of row 0, x materialized vs x referential: same columns?", + lambda: ( + next(iter(TAGGED.iter_data()))[0] + .as_table(columns={"system_tags": True}) + .column_names, + next(iter(REF.iter_data()))[0] + .as_table(columns={"system_tags": True}) + .column_names, + ), +) +show( + "system tag column names (embed the source schema hash, computed at build)", + lambda: (TAGGED.keys(all_info=True)[0][2:], REF.keys(all_info=True)[0][2:]), +) +_ = i64 diff --git a/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out new file mode 100644 index 000000000..87902721b --- /dev/null +++ b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.out @@ -0,0 +1,16 @@ +=== M1. Function-result identity follows the data COLUMN NAME (Arrow hash) +--- Data{spike_time: 10} hash == Data{spike_time_us: 10} hash + False + +=== M2. Index / Pick: out= equal to an existing TAG name (today) +--- grouped stream + keys: tags=('u', 't') data=('v',) + schema: tags={'u': int | None, 't': list[int | None]} data={'v': list[float | None]} + as_table columns: ['u', 't', 'v'] + rows: [({'u': 1, 't': [5, 6]}, {'v': [0.1, 0.2]})] +--- Index('v', 0, out='t')(g) (t is a tag) + RAISES ValueError: Duplicate column name 't' found in input tables. + (raised at orcapod/utils/arrow_utils.py:682) +--- Pick('d', 't', out='u')(st) (u is a tag) + RAISES InputValidationError: Pick: column 'd' has type orcapod.semantic_types.universal_converter.Struct_t_int | None which is not a supported pick target (not dict[K,V] and no registered logical type). + (raised at orcapod/core/operators/pick.py:174) diff --git a/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py new file mode 100644 index 000000000..75d1a9e5f --- /dev/null +++ b/superpowers/specs/itl-1004-audit/05_migration_and_collisions.py @@ -0,0 +1,43 @@ +"""ITL-1004 audit, measurement 05: migration from tags_as_data, and today's output-name checks. + +Run: uv run python superpowers/specs/itl-1004-audit/05_migration_and_collisions.py +Output: 05_migration_and_collisions.out +""" + +from __future__ import annotations + +import pyarrow as pa +from _common import describe, f64, i64, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod.core.datagrams import Data +from orcapod.operators import GroupBy, Index, Pick + +print("=== M1. Function-result identity follows the data COLUMN NAME (Arrow hash)") +same = Data(pa.table({"spike_time": i64(10)})) +renamed = Data(pa.table({"spike_time_us": i64(10)})) +show( + "Data{spike_time: 10} hash == Data{spike_time_us: 10} hash", + lambda: same.content_hash() == renamed.content_hash(), +) + +print("\n=== M2. Index / Pick: out= equal to an existing TAG name (today)") +g = GroupBy(["u"])( + src({"u": i64(1, 1), "t": i64(5, 6), "v": f64(0.1, 0.2)}, ["u", "t"], "G") +) +show("grouped stream", lambda: describe(g)) +show( + "Index('v', 0, out='t')(g) (t is a tag)", + lambda: describe(Index("v", 0, out="t")(g)), +) +st = src( + { + "u": i64(1, 2), + "d": pa.array([{"t": 1}, {"t": 2}], pa.struct([("t", pa.int64())])), + }, + ["u"], + "S", +) +show( + "Pick('d', 't', out='u')(st) (u is a tag)", + lambda: describe(Pick("d", "t", out="u")(st)), +) diff --git a/superpowers/specs/itl-1004-audit/06_default_referential.out b/superpowers/specs/itl-1004-audit/06_default_referential.out new file mode 100644 index 000000000..fdf60b44d --- /dev/null +++ b/superpowers/specs/itl-1004-audit/06_default_referential.out @@ -0,0 +1,36 @@ +=== J1. Today: materialized with materialized Join (must stay byte-identical) +--- Join()(A, B) + keys: tags=('subject',) data=('b', 'a') + schema: tags={'subject': str | None} data={'b': float | None, 'a': float | None} + as_table columns: ['subject', 'b', 'a'] + rows: [({'subject': 'm1'}, {'b': 1.0, 'a': 0.1}), ({'subject': 'm2'}, {'b': 2.0, 'a': 0.2})] +--- Join()(A, B).pipeline_hash() + d12c05799eccd043 +--- system tag columns of Join()(A, B) + ['_tag_source_id::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127:0', '_tag_record_id::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127::09145c4fb338d217d713d494648dfa233289617e81a05a5b0cac3b76314ef127:0', '_tag_source_id::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588:1', '_tag_record_id::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588::53834b01c7f038b4464ab8c754790d7bce1d4b3ff783514c34e00d1c8c7e8588:1'] +--- sync rows == async rows + True + +=== H1. Source schema hash (names the system-tag columns) moves under the default +--- compute_source_schema_hash: x materialized vs x referential + ('a7ada39f09464539483466a1e3176e697a755fcfe2d763c100730e525d915d40', '6a4bcb1d6bf4061ec4c4840fd7c7f8cc876265be2f16a1dcc9696570655b2e04') + +=== H2. Function-result identity for a function that does not read the keys +--- Data hash after dropping referenced keys == v0.3 Data hash, per row + True +--- the same, as values + ['000002c339da0c5c', '00000278727a2921'] +--- result table path (function uri) is independent of the input stream + ('double_y', '872e0cf0d9d842b03280a5017222ea774267c87b7799511c47caefd6d04fd25f', 'v0', 'python.function.v0') + +=== H3. A source whose columns are all tags (v0.3) +--- ArrowTableSource(subject, x; tag_columns=[subject, x]) + RAISES ValueError: No data columns found in the table. At least one data column is required. + (raised at orcapod/core/streams/arrow_table_stream.py:121) + +=== H4. ListSource: tags come from a tag function, not from columns +--- ListSource('weight', [20.5, 22.25]) + keys: tags=('element_index',) data=('weight',) + schema: tags={'element_index': } data={'weight': } + as_table columns: ['element_index', 'weight'] + rows: [({'element_index': 0}, {'weight': 20.5}), ({'element_index': 1}, {'weight': 22.25})] diff --git a/superpowers/specs/itl-1004-audit/06_default_referential.py b/superpowers/specs/itl-1004-audit/06_default_referential.py new file mode 100644 index 000000000..d6afc37bc --- /dev/null +++ b/superpowers/specs/itl-1004-audit/06_default_referential.py @@ -0,0 +1,132 @@ +"""ITL-1004 audit, measurement 06: consequences of "declared source tags are referential by +default" (developer, November 12, 2026), and today's materialized-with-materialized Join. + +Uses the prototype of 02 for the referential side (private attributes set after +construction; nothing under ``src/`` changes). + +Run: uv run python superpowers/specs/itl-1004-audit/06_default_referential.py +Output: 06_default_referential.out +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +from pathlib import Path +from typing import Any + +import pyarrow as pa +from _common import async_rows, describe, f64, i64, s, show, src # pyright: ignore[reportImplicitRelativeImport] # run as a file from this folder + +from orcapod import function_pod +from orcapod.config import DEFAULT_CONFIG +from orcapod.contexts import resolve_context +from orcapod.core.datagrams import Data +from orcapod.operators import Join +from orcapod.sources import ArrowTableSource, ListSource +from orcapod.types import Schema +from orcapod.utils.schema_utils import compute_source_schema_hash + + +def _load_make_ref() -> Any: + spec = importlib.util.spec_from_file_location( + "proto", Path(__file__).with_name("02_prototype_unified.py") + ) + assert spec is not None and spec.loader is not None + mod = importlib.util.module_from_spec(spec) + with contextlib.redirect_stdout(io.StringIO()): + spec.loader.exec_module(mod) + return mod.make_ref + + +make_ref = _load_make_ref() +ctx = resolve_context(None) + +print("=== J1. Today: materialized with materialized Join (must stay byte-identical)") +A = src({"subject": s("m1", "m2"), "a": f64(0.1, 0.2)}, ["subject"], "A") +B = src({"subject": s("m1", "m2"), "b": f64(1.0, 2.0)}, ["subject"], "B") +show("Join()(A, B)", lambda: describe(Join()(A, B))) +show("Join()(A, B).pipeline_hash()", lambda: Join()(A, B).pipeline_hash().to_hex(16)) +show( + "system tag columns of Join()(A, B)", + lambda: [c for c in Join()(A, B).keys(all_info=True)[0] if c.startswith("_tag")], +) +show( + "sync rows == async rows", + lambda: ( + sorted( + (repr(t.as_dict()), repr(d.as_dict())) for t, d in Join()(A, B).iter_data() + ) + == sorted((repr(t), repr(d)) for t, d in async_rows(Join(), A, B)) + ), +) + +print( + "\n=== H1. Source schema hash (names the system-tag columns) moves under the default" +) +tag = Schema({"subject": str, "x": int}) +show( + "compute_source_schema_hash: x materialized vs x referential", + lambda: ( + compute_source_schema_hash(tag, Schema({"y": float}), ctx, DEFAULT_CONFIG), + compute_source_schema_hash( + tag, Schema({"x": int, "y": float}), ctx, DEFAULT_CONFIG + ), + ), +) + + +def cols() -> dict[str, Any]: + return {"subject": s("m1", "m2"), "x": i64(10, 20), "y": f64(0.1, 0.2)} + + +V03 = src(cols(), ["subject", "x"], "S") # explicit materialized = v0.3 +REF = make_ref(src(cols(), ["subject", "x"], "S"), ["subject", "x"]) # default ref + +print("\n=== H2. Function-result identity for a function that does not read the keys") +# The automatic select keeps only y; on the referential stream that is the row's Data +# without the referenced columns. Compare with the v0.3 row's Data (y only). +show( + "Data hash after dropping referenced keys == v0.3 Data hash, per row", + lambda: ( + [ + Data(d.as_table().drop_columns(["subject", "x"])).content_hash().to_hex(16) + for _, d in REF.iter_data() + ] + == [d.content_hash().to_hex(16) for _, d in V03.iter_data()] + ), +) +show( + "the same, as values", + lambda: [d.content_hash().to_hex(16) for _, d in V03.iter_data()], +) + + +@function_pod(output_keys="z") +def double_y(y: float | None) -> float: + return (y or 0.0) * 2 + + +show( + "result table path (function uri) is independent of the input stream", + lambda: double_y.pod.data_function.uri, +) + +print("\n=== H3. A source whose columns are all tags (v0.3)") +show( + "ArrowTableSource(subject, x; tag_columns=[subject, x])", + lambda: describe( + ArrowTableSource( + pa.table({"subject": s("m1"), "x": i64(1)}), + tag_columns=["subject", "x"], + source_id="T", + ) + ), +) + +print("\n=== H4. ListSource: tags come from a tag function, not from columns") +show( + "ListSource('weight', [20.5, 22.25])", + lambda: describe(ListSource("weight", [20.5, 22.25])), +) diff --git a/superpowers/specs/itl-1004-audit/07_saved_job_versions.out b/superpowers/specs/itl-1004-audit/07_saved_job_versions.out new file mode 100644 index 000000000..77f4c7298 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/07_saved_job_versions.out @@ -0,0 +1,13 @@ +=== V1. Top-level keys of a saved job file + ['bindings', 'orcapod_pipeline_job_version', 'pipeline', 'run'] + +=== V2. Every field naming a version or a data context + orcapod_pipeline_job_version = "0.1.0" + pipeline.orcapod_pipeline_version = "0.1.0" + pipeline.nodes.semantic_v1:c472fc04f813764db9cb4d1567c03ce3ee70a48a47cdbcd0a165a01589d74596.data_context_key = "std:v1:default" + pipeline.nodes.semantic_v1:a55f98a143f589ff66e8430a3b9eb26d5beb703dc30a2b7c5267f9f89d3bdd29.data_context_key = "std:v1:default" + pipeline.nodes.semantic_v1:a55f98a143f589ff66e8430a3b9eb26d5beb703dc30a2b7c5267f9f89d3bdd29.function_config.data_function.config.version = "v0.0" + +=== V3. A source config inside the job (bindings.sources) + src: keys ['content_hash', 'data_schema', 'pipeline_hash', 'source_id', 'source_type', 'tag_columns', 'tag_schema'] + src.config keys: [] diff --git a/superpowers/specs/itl-1004-audit/07_saved_job_versions.py b/superpowers/specs/itl-1004-audit/07_saved_job_versions.py new file mode 100644 index 000000000..ea187cb00 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/07_saved_job_versions.py @@ -0,0 +1,60 @@ +"""ITL-1004 audit, measurement 07: the version information a saved job holds today. + +Saves a small job (a ``DictSource`` and one function pod, in-memory store) and prints every +field that could identify the Orcapod version that wrote it. + +Run: uv run python superpowers/specs/itl-1004-audit/07_saved_job_versions.py +Output: 07_saved_job_versions.out +""" + +from __future__ import annotations + +import json +import tempfile +from pathlib import Path +from typing import Any + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@function_pod(output_keys="doubled") +def double(v: int) -> int: + return v * 2 + + +def version_fields(obj: Any, path: str = "") -> list[str]: + """Every key path whose name mentions a version or a context, with its value.""" + found: list[str] = [] + if isinstance(obj, dict): + for key, value in obj.items(): + here = f"{path}.{key}" if path else str(key) + if any(w in str(key).lower() for w in ("version", "context")): + found.append(f"{here} = {json.dumps(value)[:80]}") + found.extend(version_fields(value, here)) + elif isinstance(obj, list): + for i, value in enumerate(obj): + found.extend(version_fields(value, f"{path}[{i}]")) + return found + + +job = PipelineJob(name="versions", store=InMemoryArrowDatabase()) +with job: + double.pod(DictSource(data=[{"k": 1, "v": 2}], tag_columns=["k"], source_id="src")) +_ = job.run() + +with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "job.json" + job.save(str(path)) + saved = json.loads(path.read_text()) + +print("=== V1. Top-level keys of a saved job file") +print(f" {sorted(saved)}") +print("\n=== V2. Every field naming a version or a data context") +for line in version_fields(saved): + print(f" {line}") +print("\n=== V3. A source config inside the job (bindings.sources)") +for name, config in saved["bindings"]["sources"].items(): + print(f" {name}: keys {sorted(config)}") + print(f" {name}.config keys: {sorted(config.get('config', {}))}") diff --git a/superpowers/specs/itl-1004-audit/_common.py b/superpowers/specs/itl-1004-audit/_common.py new file mode 100644 index 000000000..53b12ffc1 --- /dev/null +++ b/superpowers/specs/itl-1004-audit/_common.py @@ -0,0 +1,97 @@ +"""Shared helpers for the ITL-1004 audit scripts (measurements only, no library change).""" + +from __future__ import annotations + +import asyncio +import traceback +from collections.abc import Callable +from typing import Any + +import pyarrow as pa + +from orcapod.sources import ArrowTableSource + + +def src(columns: dict[str, Any], tags: list[str], sid: str) -> ArrowTableSource: + """Build an ArrowTableSource with explicit Arrow types and a fixed source id.""" + return ArrowTableSource( + pa.Table.from_pydict(columns), tag_columns=tags, source_id=sid + ) + + +def i64(*v: Any) -> pa.Array[Any]: + return pa.array(list(v), type=pa.int64()) + + +def f64(*v: Any) -> pa.Array[Any]: + return pa.array(list(v), type=pa.float64()) + + +def s(*v: Any) -> pa.Array[Any]: + return pa.array(list(v), type=pa.large_string()) + + +def show(title: str, fn: Callable[[], Any]) -> Any: + """Run ``fn`` and print its result or the exception type and message.""" + print(f"--- {title}") + try: + out = fn() + except Exception as exc: # noqa: BLE001 - a measurement records any failure + last = traceback.extract_tb(exc.__traceback__)[-1] + print(f" RAISES {type(exc).__name__}: {str(exc)[:300]}") + print(f" (raised at {last.filename.split('src/')[-1]}:{last.lineno})") + return None + if out is not None: + for line in str(out).splitlines(): + print(f" {line}") + return out + + +def describe(stream: Any) -> str: + """Keys, schema and rows of a stream, read through its public methods.""" + tk, dk = stream.keys() + ts, ds = stream.output_schema() + rows = [(t.as_dict(), d.as_dict()) for t, d in stream.iter_data()] + table = stream.as_table() + return ( + f"keys: tags={tk} data={dk}\n" + f"schema: tags={dict(ts)} data={dict(ds)}\n" + f"as_table columns: {table.column_names}\n" + f"rows: {rows}" + ) + + +def async_rows(op: Any, *streams: Any) -> list[tuple[dict[str, Any], dict[str, Any]]]: + """Run an operator's ``async_execute`` over channels and collect the rows. + + Same harness as ``tests/test_core/operators/test_async_sync_equivalence.py``. + """ + from orcapod.channels import Channel + + async def feed(stream: Any, channel: Any) -> None: + for row in stream.iter_data(): + await channel.writer.send(row) + await channel.writer.close() + + async def run() -> list[tuple[dict[str, Any], dict[str, Any]]]: + chans = [Channel(buffer_size=64) for _ in streams] + out = Channel(buffer_size=64) + # Feeders, the operator and the collector run together, so a bounded channel + # never blocks a feeder before its reader has started. + collected: list[Any] = [] + + async def collect() -> None: + collected.extend(await out.reader.collect()) + + await asyncio.gather( + *(feed(st, ch) for st, ch in zip(streams, chans)), + op.async_execute( + [c.reader for c in chans], + out.writer, + input_pipeline_hashes=[st.pipeline_hash() for st in streams], + ), + collect(), + ) + return [(t.as_dict(), d.as_dict()) for t, d in collected] + + return asyncio.run(run())