Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions changelog.d/ITL-1019.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
### Added

- A stored table states which of its tags are referential (ITL-1019): a function node's
pipeline table marks each referential tag column with Arrow field metadata
(`orcapod:tag_kind` = `referential`, `referential_tags.REFERENTIAL_TAG_METADATA_KEY`); a
plain tag carries no marker. The result table never records referential status. The
marker moves no hash and no stored path, and needs no new table layout.
- A function node reads its bindings back from its pipeline table and refuses
(`InputValidationError`, naming the table and both lists) a table whose marker disagrees
with what the node declares: its pod's fall-through tags, or a loaded job's saved record.
- A job or pipeline holding referential tags and fall-through pods can be saved and
loaded; it keeps its hashes and bindings. `ReferenceFallThroughPod.to_config` and
`from_config` work.

### Changed

- Saved files of format `0.3.0` state every binding: each source config that takes the
option writes `tag_only_columns` (`True` when every tag is plain), and each function
node writes `fall_through_references` (empty for a plain pod). The format version decides
what a file means, never a missing key: a file at `0.1.0` or `0.2.0` holds only plain
tags, and a `0.3.0` file without these keys, or whose keys disagree with its saved
schemas, is refused on load.

### Known limitations

- The SQLite and PostgreSQL databases keep no Arrow field metadata, so a function node
that would store a referential tag there is refused (`InputValidationError`, naming the
database and the two ways out: a database that keeps field metadata, or
`tag_only_columns`) before anything is written.
4 changes: 2 additions & 2 deletions changelog.d/ITL-1023.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

### Known limitations

- `MergeJoin`, operator-node result storage, `DerivedSource` over a referential
stream and saving a pipeline or job raise "not supported with referential tags yet".
- `MergeJoin`, operator-node result storage and `DerivedSource` over a referential
stream raise "not supported with referential tags yet".
`PollingSource` keeps its tags plain. The identities of referential streams may still
change.
2 changes: 1 addition & 1 deletion design_issues/ITL-1023-RT1.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
**Severity:** medium
**Issue:** ITL-1023

The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `Batch` (ITL-1017), `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, `DerivedSource` over a referential stream and saving a pipeline or job (ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
47 changes: 46 additions & 1 deletion docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,11 +172,56 @@ referential_tags.reference_materialization = "silent"
referential_tags.reference_materialization = "warn" # the default
```

## Stored tables and saved jobs

A stored table states which of its tags are referential. A function node stores each
row's tags in its pipeline table, and a referential tag is stored there once, as a tag
column that carries Arrow field metadata: the key `orcapod:tag_kind` with the value
`referential` (`referential_tags.REFERENTIAL_TAG_METADATA_KEY`). A plain tag carries no
marker. The function's result table never records referential status, so plain and
referential runs of the same function share their results. The marker moves no hash, and
the stores keep it on every write (first write, append, merge, schema evolution; in
memory and on Delta). The SQLite and PostgreSQL databases keep no Arrow
metadata at all: a function node that would store a referential tag there is refused when
its database is attached, before anything is written. Use a database that keeps field
metadata, or keep those tags plain with `tag_only_columns`.

```python
from orcapod.core.referential_tags import stored_references
from orcapod.databases import InMemoryArrowDatabase
from orcapod.pipeline import PipelineJob

database = InMemoryArrowDatabase()
job = PipelineJob(database=database)
with job:
scale.pod(spikes, label="scale")
job.run()
pipeline_table = next(
database.get_all_records(path)
for path in database.list_table_paths()
if "_result" not in path
)
print(stored_references(pipeline_table.schema))
# ('unit', 'time')
```

A node reads the binding back from its table, and refuses a table whose marker disagrees
with what the node declares (its pod's fall-through tags, or for a loaded job the saved
file's).

A saved job or pipeline (format `0.3.0`) states every binding: each source config writes
`tag_only_columns` (`True` when every tag is plain), and each function node writes
`fall_through_references` (empty for a plain pod). The format version, never a missing
key, decides what a file means: a file at `0.1.0` or `0.2.0` holds only plain tags, and a
`0.3.0` file without these keys is refused. A loaded job keeps its hashes and bindings, and
the file is checked against the stored tables when a node reads them.

## Not supported yet

These raise "not supported with referential tags yet" when a referential tag reaches them:
`MergeJoin`, storing an operator node's results (`LOG` and `REPLAY` cache modes),
`DerivedSource` over a stream with referential tags, and saving a pipeline or job.
and `DerivedSource` over a stream with referential tags (so `node.as_source()` of a node
with referential tags).
`PollingSource` keeps its tags plain. Each error names its issue and the way out: build the
source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not
data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer
Expand Down
43 changes: 34 additions & 9 deletions src/orcapod/core/function_pod.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
from orcapod.core.datagrams import Data
from orcapod.core.referential_tags import (
materialization_diagnostic_off,
not_supported_yet,
reference_names,
stream_references,
)
Expand Down Expand Up @@ -2004,21 +2003,47 @@ def plain_pod(self) -> FunctionPod:
return self._plain_pod

def to_config(self) -> dict[str, Any]:
"""Raise: a pipeline holding referential tags cannot be saved yet (ITL-1019).
"""Serialize this pod: the plain pod's config and the references (ITL-1019).

Raises:
NotImplementedError: Always.
Returns:
The plain pod's config with ``fall_through_references``, the referential
tags that fall through.
"""
raise not_supported_yet("Saving a fall-through function pod", "ITL-1019")
from orcapod.pipeline.serialization import FALL_THROUGH_REFERENCES_KEY

return {
**self._plain_pod.to_config(),
FALL_THROUGH_REFERENCES_KEY: list(self._references),
}

@classmethod
def from_config(cls, config: dict[str, Any]) -> "ReferenceFallThroughPod":
"""Raise: a fall-through pod is never saved, so never loaded (ITL-1019).
def from_config(
cls, config: dict[str, Any], *, fallback_to_proxy: bool = False
) -> "ReferenceFallThroughPod":
"""Rebuild the pod from ``to_config``'s output.

Args:
config: A config written by ``to_config``.
fallback_to_proxy: Passed to ``FunctionPod.from_config``.

Returns:
The fall-through pod over the rebuilt plain pod.

Raises:
NotImplementedError: Always.
ValueError: If the config names no ``fall_through_references``.
"""
raise not_supported_yet("Loading a fall-through function pod", "ITL-1019")
from orcapod.pipeline.serialization import FALL_THROUGH_REFERENCES_KEY

if FALL_THROUGH_REFERENCES_KEY not in config:
raise ValueError(
f"a fall-through pod config names {FALL_THROUGH_REFERENCES_KEY}; "
"this one does not"
)
plain = {k: v for k, v in config.items() if k != FALL_THROUGH_REFERENCES_KEY}
return cls(
FunctionPod.from_config(plain, fallback_to_proxy=fallback_to_proxy),
config[FALL_THROUGH_REFERENCES_KEY],
)

def identity_structure(self) -> Any:
return (self._plain_pod, ("fall_through_references", self._references))
Expand Down
104 changes: 96 additions & 8 deletions src/orcapod/core/nodes/function_node.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,12 @@
result_pod,
with_fall_through,
)
from orcapod.core.referential_tags import (
mark_references,
require_database_keeps_marker,
stored_references,
without_reference_marks,
)
from orcapod.channels import Channel, ReadableChannel, WritableChannel
from orcapod.config import OrcapodConfig
from orcapod.core.cached_function_pod import CachedFunctionPod
Expand Down Expand Up @@ -431,6 +437,31 @@ def __init__(
self._table_scope = table_scope
self._node_identity_path_cache: tuple[str, ...] | None = None

def declared_references(self) -> tuple[str, ...]:
"""Return the referential tags this node lets fall through, as it declares them.

A node with a pod declares the pod's (``ReferenceFallThroughPod``); a node
loaded from a saved file declares the file's record (``fall_through_references``
of its descriptor, ITL-1019). They are what the node's stored pipeline table
must mark.

Returns:
The referential tags; empty for a plain pod.

Raises:
ValueError: If the node has neither a pod nor a saved record.
"""
if self._function_pod is not None:
return fall_through_references(self._function_pod)
from orcapod.pipeline.serialization import FALL_THROUGH_REFERENCES_KEY

if FALL_THROUGH_REFERENCES_KEY not in self._descriptor:
raise ValueError(
f"Function node {self.label!r} has no pod and no saved "
f"{FALL_THROUGH_REFERENCES_KEY}: its referential tags are unknown"
)
return tuple(self._descriptor[FALL_THROUGH_REFERENCES_KEY])

# ------------------------------------------------------------------
# load_status
# ------------------------------------------------------------------
Expand Down Expand Up @@ -1142,12 +1173,15 @@ class _JoinedRecords(NamedTuple):
empty_taginfo_rows: Mapping from ``base_entry_id`` bytes to the raw
taginfo row dict for each entry in ``empty_data_tokens``. Used
by ``_load_cached_entries`` to reconstruct the tag for each token.
references: The referential tags the pipeline table marks (ITL-1019),
checked against the node's pod when it has one.
"""

table: pa.Table
taginfo_columns: tuple[str, ...]
empty_data_tokens: dict[bytes, EmptyData]
empty_taginfo_rows: dict[bytes, dict]
references: tuple[str, ...]


class FunctionJobNode(RunHoldingNode, FunctionNodeBase):
Expand Down Expand Up @@ -1428,6 +1462,11 @@ def attach_databases(
result_database = pipeline_database.at("_result")

if self._function_pod is not None:
# A table must state its referential tags: refuse a database that
# cannot, before anything is written (ITL-1019).
require_database_keeps_marker(
pipeline_database, fall_through_references(self._function_pod)
)
# Normal path: wrap in CachedFunctionPod for compute + cache.
# A fall-through pod's results are its plain pod's (ITL-1023).
self._cached_function_pod = CachedFunctionPod(
Expand Down Expand Up @@ -2911,11 +2950,16 @@ def _add_pipeline_record(
}
)

# Combine: tag (with system tags) + input source columns + meta columns
combined_record = arrow_utils.hstack_tables(
tag.as_table(columns={"system_tags": True}),
input_source_table,
meta_table,
# Combine: tag (with system tags) + input source columns + meta columns. The
# table states which tags are referential: the ones that fall through this
# node's pod, marked on their stored column (ITL-1019).
combined_record = mark_references(
arrow_utils.hstack_tables(
tag.as_table(columns={"system_tags": True}),
input_source_table,
meta_table,
),
fall_through_references(self._function_pod),
)

try:
Expand Down Expand Up @@ -3117,6 +3161,7 @@ def _fetch_joined_records(

taginfo_columns = tuple(taginfo.column_names)
taginfo_schema = taginfo.schema
references = self._stored_references(taginfo_schema)

is_ephemeral_col = constants.IS_EPHEMERAL_COL
taginfo_df = pl.DataFrame(taginfo)
Expand Down Expand Up @@ -3275,6 +3320,7 @@ def _fetch_joined_records(
taginfo_columns=taginfo_columns,
empty_data_tokens=empty_data_tokens,
empty_taginfo_rows=empty_taginfo_rows,
references=references,
)

# Apply base_entry_id filter if requested
Expand All @@ -3298,8 +3344,48 @@ def _fetch_joined_records(
taginfo_columns=taginfo_columns,
empty_data_tokens=empty_data_tokens,
empty_taginfo_rows=empty_taginfo_rows,
references=references,
)

def _stored_references(self, taginfo_schema: "pa.Schema") -> tuple[str, ...]:
"""Return the referential tags the pipeline table marks, checked against the pod.

The stored table states its bindings (developer, November 16, 2026, ITL-1019):
a referential tag column carries the marker of ``referential_tags``. The node
reads the table only when it agrees with what the node declares
(``declared_references``: its pod's, or a loaded node's saved record, an
independent point of verification).

Args:
taginfo_schema: The schema of the stored pipeline table.

Returns:
The referential tags, in the order the node declares them.

Raises:
InputValidationError: If the table's marker is invalid, or the table
and the node disagree on which tags are referential.
"""
table_path = self._require_pipeline_database().base_path + (
self._versioned_pipeline_path
)
try:
stored = stored_references(taginfo_schema)
except ValueError as exc:
raise InputValidationError(
f"Function node {self.label!r}, pipeline table {table_path}: {exc}"
) from exc
expected = self.declared_references()
if set(stored) != set(expected):
source = "pod" if self._function_pod is not None else "saved job"
raise InputValidationError(
f"Function node {self.label!r}: the pipeline table {table_path} marks "
f"the referential tags {sorted(stored)}, but the node's {source} "
f"declares {sorted(expected)}. The table was not written by this "
"node's pipeline; nothing was read."
)
return expected

def _stored_tag_keys(self, fetched: _JoinedRecords) -> tuple[str, ...]:
"""Return the user tag columns of the records in *fetched*.

Expand Down Expand Up @@ -3376,11 +3462,13 @@ def _stored_rows_stream(
uuid.UUID(bytes=bytes(cast(bytes, record_id)))
for record_id in joined.column(constants.DATA_RECORD_ID).to_pylist()
]
# A fall-through column is the stored tag column in its data role (ITL-1023).
# A fall-through column is the stored tag column in its data role (ITL-1023):
# the table states which (ITL-1019). In the stream the binding is the shared
# name, so the marker stays with the stored table.
stream = ArrowTableStream(
joined.drop(drop_cols),
without_reference_marks(joined.drop(drop_cols)),
tag_columns=self._stored_tag_keys(fetched),
reference_tags=fall_through_references(self._function_pod),
reference_tags=fetched.references,
)
return base_entry_ids, stream, data_ids

Expand Down
Loading
Loading