Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,18 @@ GroupBy is the only operator that reduces row count many→one: it collapses N r
tag tuple into one row with list-valued members. It synthesizes no new data values and keys
only on tags, never on data. Its non-key tags stay tags, list-valued, as Batch keeps them.

Operator or derived source? (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
by a source, DerivedSource(origin, tags_as_data={tag: data_name}) (ITL-939), never by an
operator. That form is a new root: fresh provenance through SourceStreamBuilder, schema-only
pipeline_hash, and a default source_id of the form
tags_as_data:<algorithm label>:<hex> (the kind names the operation; the config type key stays
derived), the labelled content hash an operator node doing the
derivation would have (DerivedSource._default_source_id, the one place that defines it;
the string is used as-is in system-tag values and source-info tokens). DerivedSource without tags_as_data (a job node's stored records)
keeps its old identity; do not change it (ITL-956).

### Two identity chains

Every pipeline element has two parallel hashes:
Expand Down Expand Up @@ -508,7 +520,8 @@ data. The rules:
- FunctionNode Phase 1 returns ALL records in the shared pipeline_path DB table.
Phase 2 skips inputs whose hash is already in the DB.
- Empty data → ArrowTableSource raises ValueError("Table is empty").
- DerivedSource before run() → raises ValueError (no computed records).
- DerivedSource over a job node before run() → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with tags_as_data.
- Join requires non-overlapping data columns; raises InputValidationError on collision.
- MergeJoin requires colliding data columns to have identical Python types; merges into sorted
large_list<T> with source columns reordered to match, where T is the large-normalized input
Expand Down
15 changes: 14 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,18 @@ tag tuple into one row with list-valued members. It still synthesizes no new dat
every emitted element came from an input row — and it keys only on tags, never on data. Its
non-key tags stay tags, list-valued, as `Batch` keeps them.

**Operator or derived source?** (developer, November 11, 2026) The test is: does any data
element come from a tag value, or from anywhere other than the input's data elements? If yes,
the step is not an operator: it is a derived source. So a tag's value is offered as data only
by a source, `DerivedSource(origin, tags_as_data={tag: data_name})` (ITL-939), never by an
operator. That form is a new root: fresh provenance through `SourceStreamBuilder`, schema-only
`pipeline_hash`, and a default `source_id` of the form
`tags_as_data:<algorithm label>:<hex>` (the kind names the operation; the config type key stays
`derived`), the labelled content hash an operator node doing the
derivation would have (`DerivedSource._default_source_id`, the one place that defines it;
the string is used as-is in system-tag values and source-info tokens). `DerivedSource` without `tags_as_data` (a job node's stored records)
keeps its old identity; do not change it (ITL-956).

### Two identity chains

Every pipeline element has two parallel hashes:
Expand Down Expand Up @@ -550,7 +562,8 @@ data. The rules:
DB table (not filtered to current inputs). Phase 2 skips inputs whose hash is already
in the DB.
- Empty data → `ArrowTableSource` raises `ValueError("Table is empty")`.
- `DerivedSource` before `run()` → raises `ValueError` (no computed records).
- `DerivedSource` over a job node before `run()` → presents zero rows with the origin's columns
(no computed records; no error), in the plain form and with `tags_as_data`.
- Join requires non-overlapping data columns; raises `InputValidationError` on collision.
- MergeJoin requires colliding data columns to have identical Python types; merges into sorted
`large_list<T>` with source columns reordered to match, where `T` is the large-normalized input
Expand Down
12 changes: 12 additions & 0 deletions changelog.d/ITL-939.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
### Added

- **`DerivedSource(origin, tags_as_data={tag: data_name})`** — a source derived from
another source or from a job node that keeps the origin's tags and also offers chosen tag
values as data columns, so a key (for example a session start time) can be an input to a
function pod. Also available as `node.as_source(tags_as_data=...)`. The copy has the tag's
exact Arrow type and nullability. The result is a new root source with its own provenance;
its default `source_id` is `tags_as_data:<label>:<hex>`, a hash of the origin's identity and the
mapping. A saved job
reloads it whenever its origin can be reloaded. `DerivedSource` without `tags_as_data` is
unchanged, except that giving it a source (rather than a job node) now raises `TypeError`
when it is built instead of `AttributeError` when it is first read. (ITL-939)
26 changes: 26 additions & 0 deletions design_issues/ITL-939-TD1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## Tag and data groups — a value wanted as both (ITL-939)

### TD1 — A key column cannot also be a pod input
**Status:** resolved
**Severity:** medium
**Issue:** ITL-939

A name cannot be in both the tag and the data group, so a key that a pod needs as data (EDI
`session_start_datetime`, for elapsed-time arithmetic) could not reach a pod without a
workaround, and two of the obvious workarounds (keeping the column out of `tag_columns`, or
reading the table twice and joining) give silently wrong joins. The developer rejected an
operator for this (an operator must not put new content into the data group) and chose a
source derived from other pods.

**Fix:** `DerivedSource(origin, tags_as_data={tag: data_name})`, also
`node.as_source(tags_as_data=...)`: a new root source over a source or a job node whose data
columns are the origin's plus a copy of each mapped tag, with the tag's exact Arrow type and
nullability, fresh source provenance, a schema-only `pipeline_hash`, and a default `source_id`
`tags_as_data:<label>:<hex>`, the content hash an operator node doing the derivation would have. Reloadable from a
saved job when its origin is. No existing hash or stored schema changed.

Left open, tracked elsewhere: the same-name collisions that fail late in `Join`/`MergeJoin`
and for a pod output key equal to a tag name (ITL-940); the gaps of the plain
`node.as_source()` form, which has no system tags and cannot be saved, and whose zero-row
table before the node has run takes its Arrow types from the Python schema (ITL-956);
`GroupBy` promoting non-key tags into data (ITL-955, fixed separately).
87 changes: 86 additions & 1 deletion docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,12 +211,97 @@ column as nullable.

### `DerivedSource`

A `DerivedSource` reads the stored results of a node of a
A `DerivedSource` is a source whose rows come from something Orcapod already has: another
source, or a node of a job. It has two uses.

**Reading a job's results.** A `DerivedSource` reads the stored results of a node of a
[`PipelineJob`](function-pods.md#pipelinejob-database-backed-cached-execution) that has run
and presents them as a new source. You get one by calling `as_source()` on the node, for
example `job.nodes["bmi"].as_source()`. This is useful for chaining pipeline stages: run a
job, then use one of its outputs as the input of a new pipeline.

**Using a tag value as an input to a computation.** Tags identify a row and data is what a
function computes on, so a function pod never receives a tag. Sometimes one value is needed
for both: a session's start time identifies the session, and it is also what you subtract to
get an elapsed time. Pass `tags_as_data` to get a source that keeps the tag and also offers
its value as a data column under a name you choose:

```python
from datetime import datetime, timezone

from orcapod import function_pod
from orcapod.sources import DerivedSource, DictSource

start_1 = datetime(2026, 1, 5, 9, 0, tzinfo=timezone.utc)
start_2 = datetime(2026, 1, 6, 9, 30, tzinfo=timezone.utc)

experiments = DictSource(
data=[
{"subject_id": "m1", "session_start": start_1, "experiment_index": 0,
"experiment_start": datetime(2026, 1, 5, 9, 1, tzinfo=timezone.utc)},
{"subject_id": "m1", "session_start": start_1, "experiment_index": 1,
"experiment_start": datetime(2026, 1, 5, 9, 15, tzinfo=timezone.utc)},
{"subject_id": "m1", "session_start": start_2, "experiment_index": 0,
"experiment_start": datetime(2026, 1, 6, 9, 32, tzinfo=timezone.utc)},
],
tag_columns=["subject_id", "session_start", "experiment_index"],
source_id="experiments",
)

with_start = DerivedSource(
experiments, tags_as_data={"session_start": "session_start_value"}
)

tag_keys, data_keys = with_start.keys()
print(tag_keys)
# ('subject_id', 'session_start', 'experiment_index')
print(data_keys)
# ('experiment_start', 'session_start_value')

@function_pod(output_keys="onset_s")
def onset_s(session_start_value: datetime, experiment_start: datetime) -> float:
return (experiment_start - session_start_value).total_seconds()

onsets = onset_s.pod(with_start)
print(onsets.as_table().select(["subject_id", "experiment_index", "onset_s"]).to_pandas())
# subject_id experiment_index onset_s
# 0 m1 0 60.0
# 1 m1 1 900.0
# 2 m1 0 120.0
```

The result still has `session_start` as a tag, so it joins with other tables on that key as
before. Things to know:

- The mapping is `{tag column: new data column}`. The new name must not already be a column
and must not start with `_`; a wrong name raises `InputValidationError` when the source is
built.
- The copy has exactly the tag's type, including whether it may be null. If the tag may be
null, annotate the function parameter as optional (`datetime | None`).
- The copy equals the tag when the source is read. After that it is ordinary data: a function
may replace it and `map_data` may rename it, while the tag stays as it was.
- It works on any source (`DerivedSource(source, tags_as_data=...)`) and on a job node
(`job.nodes["bmi"].as_source(tags_as_data=...)`).
- It is a **new source**, not a step in the origin's pipeline: `upstreams` is empty, and it
has provenance columns of its own, like the result of handing the same rows to
`ArrowTableSource`. Use it in place of the origin. Its default `source_id` is
`tags_as_data:` followed by a labelled hash built from the origin's identity and the mapping, so
it changes when either changes; pass `source_id=` to choose a name.
- A saved job reloads a derived source whenever its origin can be reloaded (a CSV file, a
Delta table, a database table). Over an in-memory source it is saved but not reloaded, like
the in-memory source itself.

```python
print(with_start.upstreams)
# ()
print(with_start.source_id)
# tags_as_data:...:...
```

Only a source can do this. An operator rearranges rows and columns, and every data value it
emits was already a data value of its input; turning a tag value into a data value creates
new data, which is what a source is for.

## How sources add provenance columns

When you create a source, Orcapod automatically adds two kinds of hidden columns to track
Expand Down
1 change: 1 addition & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ The table below shows what each class returns from `identity_structure()`.
| `ArrowTableStream` | `(producer, argument_symmetry(upstreams))` | Falls back to `(class_name, as_table(all_info=True), tag_columns)` when no producer or producer is not a `PipelineElementProtocol` |
| `RootSource` (`ArrowTableSource`) | `(class_name, (tag_schema, data_schema), source_id)` | `source_id` is the explicit user-supplied value; when not provided it defaults to the Arrow hash of the raw table (site 4). Data-inclusive base case of the Merkle chain. |
| `DerivedSource` | Origin node's `content_hash` | Inherits its generating node's identity |
| `DerivedSource` with `tags_as_data` | `(class_name, output_schema, source_id)`; the default `source_id` is `tags_as_data:<label>:<hex>` of the hash of `(("DerivedSource", sorted tags_as_data items), (origin,))`, the origin entering through its `content_hash`; a `record_id_column`, when set, is appended to the parameters as `("record_id_column", name)` | The rule of every root source; the origin and the mapping enter through the default `source_id`, which is the content hash an operator node doing the derivation would have |
| Operators (unary) | `(operator_class_name, upstream_stream)` | Stream reference resolved via `content_resolver` |
| Operators (binary/N-ary) | `(operator_class_name, argument_symmetry(streams))` | `frozenset` for commutative (Join, MergeJoin); `tuple` for ordered (SemiJoin) |

Expand Down
20 changes: 16 additions & 4 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
import logging
import uuid
from abc import abstractmethod
from collections.abc import Iterator
from collections.abc import Iterator, Mapping
from typing import TYPE_CHECKING, Any, Literal, NamedTuple, cast

from orcapod import contexts
Expand Down Expand Up @@ -2103,22 +2103,34 @@ def get_all_records(

return joined if joined.num_rows > 0 else None

def as_source(self):
def as_source(self, tags_as_data: "Mapping[str, str] | None" = None):
"""Return a DerivedSource backed by the DB records of this node.

Args:
tags_as_data: Maps a tag column of this node to the name of a data
column that receives a copy of it. When given, the result is a
new root source with its own provenance and a ``source_id``
derived from this node's content hash and the mapping; see
``DerivedSource``.

Raises:
RuntimeError: If no database is attached.
InputValidationError: If ``tags_as_data`` is not valid for this
node's columns.
"""
if self._pipeline_database is None:
raise RuntimeError("Cannot create a DerivedSource without a database")

from orcapod.core.sources.derived_source import DerivedSource

path_str = "/".join(self.node_identity_path)
source_id = f"{path_str}:{self.content_hash().to_string()}"
source_id: str | None = None
if tags_as_data is None:
path_str = "/".join(self.node_identity_path)
source_id = f"{path_str}:{self.content_hash().to_string()}"
return DerivedSource(
origin=self,
source_id=source_id,
tags_as_data=tags_as_data,
data_context=self.data_context_key,
config=self.orcapod_config,
)
Expand Down
20 changes: 16 additions & 4 deletions src/orcapod/core/nodes/operator_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
import asyncio
import logging
from abc import abstractmethod
from collections.abc import Collection, Iterator, Sequence
from collections.abc import Collection, Iterator, Mapping, Sequence
from typing import TYPE_CHECKING, Any, Literal

from orcapod import contexts
Expand Down Expand Up @@ -1166,22 +1166,34 @@ def get_all_records(
# DerivedSource
# ------------------------------------------------------------------

def as_source(self):
def as_source(self, tags_as_data: "Mapping[str, str] | None" = None):
"""Return a DerivedSource backed by the DB records of this node.

Args:
tags_as_data: Maps a tag column of this node to the name of a data
column that receives a copy of it. When given, the result is a
new root source with its own provenance and a ``source_id``
derived from this node's content hash and the mapping; see
``DerivedSource``.

Raises:
RuntimeError: If no database is attached.
InputValidationError: If ``tags_as_data`` is not valid for this
node's columns.
"""
if self._pipeline_database is None:
raise RuntimeError("Cannot create a DerivedSource without a database")

from orcapod.core.sources.derived_source import DerivedSource

path_str = "/".join(self.node_identity_path)
source_id = f"{path_str}:{self.content_hash().to_string()}"
source_id: str | None = None
if tags_as_data is None:
path_str = "/".join(self.node_identity_path)
source_id = f"{path_str}:{self.content_hash().to_string()}"
return DerivedSource(
origin=self,
source_id=source_id,
tags_as_data=tags_as_data,
data_context=self.data_context_key,
config=self.orcapod_config,
)
Expand Down
Loading
Loading