Skip to content
Merged
22 changes: 22 additions & 0 deletions changelog.d/ITL-1022.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
### Added

- `SelectColumns` and `DropColumns` (`select_columns`, `drop_columns`) handle referential
tags (ITL-1022). A referential tag is one column in both roles, so naming it acts on both:
selecting it keeps it referential, dropping it (or leaving it out of a select) removes the
column from both roles. The roles are never split and nothing is materialized. Before,
both operators raised "not supported with referential tags yet".

### Changed

- Errors that a referential tag causes name it and the way out; no behaviour changes.
- A bind refused because a column is a referential tag on one side and a plain tag or a
data column on the other (listed as a missing or unexpected column) says which side
holds the referential tag, and how to make the two agree: `tag_only_columns` on the
source, or a slot declared with the column in both roles (ITL-1142). The bind stays
strict.
- `pod.process(stream)` and `pod(stream, strict=True)` refusing a referential tag the
function does not read name the referential tags and the two ways out: `pod(stream)`,
or `tag_only_columns`, each only where it can work (not when no data column of the
input is a parameter of the function, except a plain `FunctionPod`'s call).
- "Not supported with referential tags yet" names `tag_only_columns=True`, which keeps
every tag plain as in v0.3, beside its issue.
58 changes: 55 additions & 3 deletions docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,11 +82,12 @@ does not make it compute again.
| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the non-key tags, so that list is in value order and the other data lists follow it. |
| `SelectDataColumns`, `DropDataColumns` | Removing the data column leaves `x` a plain tag with the same values, with a `ReferenceMaterializationWarning`. |
| `SelectTagColumns`, `DropTagColumns` | Removing the tag leaves `x` a data column. |
| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` referential, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. |
| `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. |
| `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. |
| Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the reference. |
| `Batch` | Keeps every tag, list-valued, as for plain tags: `x` becomes the same list in both roles and stays referential. Its source info is one token per member, like any data column: nothing is merged. |
| `MergeJoin`, `SelectColumns`, `DropColumns` | Not supported with referential tags yet: they raise. |
| `MergeJoin` | Not supported with referential tags yet: it raises. |

```python
units = op.sources.DictSource(
Expand All @@ -99,6 +100,13 @@ print(joined.keys())
# (('unit', 'time'), ('unit', 'area', 'time', 'amp'))
```

```python
print(spikes.select_columns(["time", "amp"]).keys())
# (('time',), ('time', 'amp'))
print(spikes.drop_columns(["time"]).keys())
# (('unit',), ('unit', 'amp'))
```

`Batch` lists every column, a referential tag included, so the tag and its data stay one
column:

Expand All @@ -111,6 +119,24 @@ for tag, data in batched.iter_data():
# [10, 20] [10, 20]
```

### After `GroupBy`: rename a non-key column with `map_data`

`GroupBy` keeps only its key tags, and drops every other tag itself: there is nothing left
to drop with `drop_tag_columns` afterwards (that raises, since the tag is gone). A
referential non-key column loses its tag role there, but keeps its data column, a list
like every other data column. So after the grouping it is data only, and is renamed with
`map_data`; `map_tags` would find no tag of that name and rename nothing.

```python
by_unit = spikes.group_by(["unit"])
print(by_unit.keys())
# (('unit',), ('unit', 'time', 'amp'))

renamed = by_unit.map_data({"time": "times"})
print(renamed.keys())
# (('unit',), ('unit', 'times', 'amp'))
```

### Source info of a merged value

A source-info column belongs to its data column and has its shape: a scalar value has one
Expand Down Expand Up @@ -149,6 +175,32 @@ referential_tags.reference_materialization = "warn" # the default
## Not supported yet

These raise "not supported with referential tags yet" when a referential tag reaches them:
`MergeJoin`, `SelectColumns`, `DropColumns`, storing an operator node's results (`LOG` and `REPLAY` cache modes),
`MergeJoin`, storing an operator node's results (`LOG` and `REPLAY` cache modes),
`DerivedSource` over a stream with referential tags, and saving a pipeline or job.
`PollingSource` keeps its tags plain.
`PollingSource` keeps its tags plain. Each error names its issue and the way out: build the
source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not
data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer
take it from that column: as in v0.3, the value must also be a data column of its own, for
example a copy in the source's data under another name, or
`DerivedSource(origin, tags_as_data={...})`.

## Errors that name a referential tag

- `pod.process(stream)` and `pod(stream, strict=True)` are exact: they hand every data
column to the function, so a referential tag the function does not read is refused like
any extra column. The error names the referential tags and the two ways out: call the
pod, `pod(stream)`, which for a plain `FunctionPod` lets them pass through as
referential tags (a wrapper pod such as `CachedFunctionPod` has no fall-through: its
call keeps them as plain tags), or keep them plain with `tag_only_columns`. It offers
only a way out that can work: when no data column of the input is a parameter, a
wrapper pod's call and `tag_only_columns` would leave the function nothing to read,
and the error says so instead.
- Binding a source to a job's input slot is strict: a column that is a referential tag on
one side and a plain tag or a data column on the other is refused (it is listed as a
missing or unexpected column). The error says which side holds it as a referential tag,
and how to make the two agree, in either direction. A referential source tag for a
plain slot tag: keep it plain with `tag_only_columns`, or declare the slot with the
column in both roles. A plain source tag (or a data-only source column) for a
referential slot tag: make it a declared tag that `tag_only_columns` does not name, or
declare the slot with it as the source has it (a plain tag, or a data column only). `tag_only_columns` lists every plain tag of the source, so the error
names the exact list.
4 changes: 1 addition & 3 deletions src/orcapod/core/cached_function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,9 +431,7 @@ def process(
active trackers (so a ``PipelineJob`` records this pod).
"""
input_stream = self.handle_input_streams(*streams)
tag_schema, data_schema = input_stream.output_schema()
self._validate_input_schema(data_schema)
self._validate_output_keys(tag_schema.keys())
self._validate_input(*input_stream.output_schema())
# The pod has no ``from_config`` (it cannot be rebuilt without its cache
# database), the one member of FunctionPodProtocol it lacks; the tracker uses
# none of it.
Expand Down
105 changes: 92 additions & 13 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from orcapod.core.referential_tags import (
materialization_diagnostic_off,
not_supported_yet,
reference_names,
stream_references,
)
from orcapod.core.streams.base import (
Expand Down Expand Up @@ -252,9 +253,70 @@ def validate_inputs(self, *streams: StreamProtocol) -> None:
ValueError: If inputs are incompatible with the data function schema.
"""
input_stream = self.handle_input_streams(*streams)
incoming_tag_schema, incoming_data_schema = input_stream.output_schema()
self._validate_input_schema(incoming_data_schema)
self._validate_output_keys(incoming_tag_schema.keys())
self._validate_input(*input_stream.output_schema())

def _validate_input(self, tag_schema: Schema, data_schema: Schema) -> None:
"""Validate a (joined) input's schemas against the function.

Its data must fit the function's parameters (``_validate_input_schema``) and no
output key may be one of its tags (``_validate_output_keys``). When the data
does not fit and the input has referential tags the function does not read,
the error says so and names the ways out: this is the exact form, which hands
every data column to the function (ITL-1022).

Args:
tag_schema: The input's tag schema.
data_schema: The input's data schema.

Raises:
ValueError: If the data does not fit the function's parameters.
InputValidationError: If an output key of the function is a tag name.
"""
try:
self._validate_input_schema(data_schema)
except ValueError as error:
parameters = self.input_data_schema
unread = [
name
for name in reference_names(tag_schema, data_schema)
if name not in parameters
]
if not unread:
raise
# Name only the ways out that can work for this input. Only a plain
# ``FunctionPod``'s call lets references fall through (``__call__``),
# whatever the other columns. A wrapper pod's call drops their data
# columns, and keeping them plain removes them: either leaves the
# function a column to read only when a parameter column remains.
has_parameter_column = any(name in parameters for name in data_schema)
remedies: list[str] = []
if isinstance(self, FunctionPod):
remedies.append(
"call the pod as pod(stream) to let them pass through as "
"referential tags"
)
elif has_parameter_column:
remedies.append(
"call the pod as pod(stream), which keeps them as plain tags"
)
if has_parameter_column:
remedies.append(
f"add {unread} to the tag_only_columns of the source that "
"declares them (tag_only_columns=True keeps every tag plain)"
)
way_out = (
", or ".join(remedies)
if remedies
else "no data column of the input is a parameter of the function, "
"so neither calling the pod nor keeping them plain leaves it one"
)
raise ValueError(
f"{error}. {unread} are referential tags of the input (each a tag "
"and a data column of one name) that the function does not read, and "
"the exact form, pod.process(stream) or pod(stream, strict=True), "
f"hands every data column to the function: {way_out}"
) from error
self._validate_output_keys(tag_schema.keys())

def _validate_output_keys(self, tag_names: Collection[str]) -> None:
"""Reject an output key that is also a tag column of the input.
Expand Down Expand Up @@ -808,9 +870,7 @@ def output_schema(
tag_schema, incoming_data_schema = self.multi_stream_handler().output_schema(
*streams, columns=columns, all_info=all_info
)
# validate that incoming_data_schema is valid
self._validate_input_schema(incoming_data_schema)
self._validate_output_keys(tag_schema.keys())
self._validate_input(tag_schema, incoming_data_schema)
# The output schema of the FunctionPodProtocol is determined by the data function
# TODO: handle and extend to include additional columns
# Namely, the source columns
Expand Down Expand Up @@ -907,9 +967,7 @@ def process(
input_stream = self.handle_input_streams(*streams)

# perform input stream schema validation
tag_schema, data_schema = input_stream.output_schema()
self._validate_input_schema(data_schema)
self._validate_output_keys(tag_schema.keys())
self._validate_input(*input_stream.output_schema())
self.tracker_manager.record_function_pod_invocation(
self, input_stream, label=label
)
Expand Down Expand Up @@ -1861,7 +1919,24 @@ def uri(self) -> tuple[str, ...]:
return self._function_pod.uri

def validate_inputs(self, *streams: StreamProtocol) -> None:
self._function_pod.validate_inputs(*streams)
try:
self._function_pod.validate_inputs(*streams)
except ValueError:
self._refuse_as_this_pod(*streams)
raise

def _refuse_as_this_pod(self, *streams: StreamProtocol) -> None:
"""Validate ``streams`` as this pod, so a refusal names this pod's call.

The inner pod words a referential-tag refusal for its own call, which may let
references fall through where this pod's does not (ITL-1022). Called only when
the inner pod refused; when this check passes, the caller re-raises the inner
pod's error.

Raises:
ValueError: If the input does not fit this pod, with this pod's advice.
"""
self._validate_input(*self.handle_input_streams(*streams).output_schema())

def argument_symmetry(self, streams: Collection[StreamProtocol]) -> ArgumentGroup:
return self._function_pod.argument_symmetry(streams)
Expand All @@ -1872,9 +1947,13 @@ def output_schema(
columns: ColumnConfig | dict[str, Any] | None = None,
all_info: bool = False,
) -> tuple[Schema, Schema]:
return self._function_pod.output_schema(
*streams, columns=columns, all_info=all_info
)
try:
return self._function_pod.output_schema(
*streams, columns=columns, all_info=all_info
)
except ValueError:
self._refuse_as_this_pod(*streams)
raise

# TODO: reconsider whether to return FunctionPodStream here in the signature
def process(
Expand Down
105 changes: 105 additions & 0 deletions src/orcapod/core/nodes/source_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,94 @@ def _one_system_tag_pair_issue(
return f"{issue}: {remedy} (binding such a stream is planned, ITL-1127)"


_TAG_ROLE = frozenset({"tag"})
_BOTH_ROLES = frozenset({"tag", "data"})


def _roles(
name: str, tag_schema: Mapping[str, Any], data_schema: Mapping[str, Any]
) -> frozenset[str]:
"""Return the roles ``name`` has in a pair of schemas: ``"tag"``, ``"data"``, both, none.

A name in both is a referential tag (ITL-1023): one column in both roles.
"""
return frozenset(
role
for role, schema in (("tag", tag_schema), ("data", data_schema))
if name in schema
)


def _referential_role_hint(
name: str,
*,
source_roles: frozenset[str],
slot_roles: frozenset[str],
source_plain_tags: list[str],
) -> str | None:
"""Explain a role difference that a referential tag on one side causes, with the remedy.

The bind refuses every role difference (ITL-1142: strict for now). The missing and
unexpected columns it lists do not say that one side holds the column as a
referential tag, a tag and a data column of one name; this does, and names how to
make the two agree (ITL-1022).

Args:
name: The column name.
source_roles: The roles the column has in the source (see ``_roles``).
slot_roles: The roles the column has in the slot.
source_plain_tags: The source's plain tags, in tag order. ``tag_only_columns``
lists every plain tag, so the remedy names the exact list that changes
``name`` only.

Returns:
The explanation, or ``None`` when the roles agree, when neither side holds the
column as a referential tag, or when one side lacks the column altogether.
"""
if source_roles == slot_roles or not source_roles or not slot_roles:
return None
if source_roles == _BOTH_ROLES:
if slot_roles == _TAG_ROLE:
remedy = (
f"build the source with tag_only_columns={[*source_plain_tags, name]} "
"to keep "
f"it a plain tag, or declare the slot with {name!r} as both a tag and "
"a data column"
)
slot_role = "a plain tag"
else:
remedy = (
f"build the source without {name!r} in tag_columns, or declare the "
f"slot with {name!r} as both a tag and a data column"
)
slot_role = "a data column only"
return (
f"{name!r} is a referential tag of the source (a tag and a data column of "
f"one name) but {slot_role} of the slot: {remedy}"
)
if slot_roles == _BOTH_ROLES:
# A declared tag is referential unless ``tag_only_columns`` names it.
others = [tag for tag in source_plain_tags if tag != name]
tag_only = f"tag_only_columns={others}" if others else "no tag_only_columns"
if source_roles == _TAG_ROLE:
source_role = "a plain tag"
remedy = (
f"build the source with {tag_only} (a declared tag is referential "
f"unless named there), or declare the slot with {name!r} as a plain tag"
)
else:
source_role = "a data column only"
remedy = (
f"build the source with {name!r} in tag_columns and {tag_only}, or "
f"declare the slot with {name!r} as a data column only"
)
return (
f"{name!r} is {source_role} of the source but a referential tag of the "
f"slot (a tag and a data column of one name): {remedy}"
)
return None


class SourceNodeBase(TraceableBase, ABC):
"""Abstract base for SourceNode and SourceJobNode.

Expand Down Expand Up @@ -657,6 +745,23 @@ def _fit(self, source: "StreamProtocol") -> tuple[Schema, Schema, frozenset[str]
f"{_type_text(given[name])}"
)

# A name whose roles differ only because one side holds it as a referential tag
# (a tag and a data column of one name) is refused like any role difference
# (ITL-1142: strict for now); say so, and how to make the two agree (ITL-1022).
for name in sorted(
{*self._tag_schema, *self._data_schema, *source_tag, *source_data}
):
hint = _referential_role_hint(
name,
source_roles=_roles(name, source_tag, source_data),
slot_roles=_roles(name, self._tag_schema, self._data_schema),
# ``tag_only_columns`` lists every plain tag, so a remedy keeps the
# source's other plain tags plain.
source_plain_tags=[tag for tag in source_tag if tag not in source_data],
)
if hint is not None:
issues.append(hint)

# The slot's rows carry one system-tag pair under the slot's names, which only
# a source with exactly one pair can take. A joined or merged stream (one pair
# per input) or a stream with none is refused (developer, November 14, 2026,
Expand Down
Loading
Loading