Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,11 @@ Use a non-public entry point (`unary_static_process`, `static_process`,
entry point. The standing exception is verifying a schema prediction, which must compare
against `static_process(...).output_schema(...)`; see "Important implementation details".

Tests and golden fixtures that build a source with tag columns pass `tag_only_columns=True`
unless the test is about the default (developer, November 16, 2026, ITL-1159). This covers the
sources that take the option (the table, file and database sources); `ListSource`,
`DerivedSource`, `CachedSource` and `PollingSource` do not take it.

---

## Project layout
Expand Down
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,11 @@ Use a non-public entry point (`unary_static_process`, `static_process`,
entry point. The standing exception is verifying a schema prediction, which must compare
against `static_process(...).output_schema(...)`; see "Important implementation details".

Tests and golden fixtures that build a source with tag columns pass `tag_only_columns=True`
unless the test is about the default (developer, November 16, 2026, ITL-1159). This covers the
sources that take the option (the table, file and database sources); `ListSource`,
`DerivedSource`, `CachedSource` and `PollingSource` do not take it.

---

## Project layout
Expand Down
21 changes: 21 additions & 0 deletions changelog.d/ITL-1159.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
### Added

- Every source built from a table takes the keyword-only option `tag_only_columns`:
`ArrowTableSource`, `DictSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource`,
`DBTableSource`, `SQLiteTableSource`, `PostgreSQLTableSource` and `SpiralDBTableSource`
(ITL-1159). In v0.3 every tag is plain (a tag only, never also a data column), and the option
has one accepted value, `True` (all tags plain), which is also the default. A source built
with `tag_only_columns=True` is the source built without it: the same content and pipeline
hashes, source id, record ids, rows, stored paths and saved config. Any other value (`False`,
an empty collection, column names, even all of them) raises `ValueError`, because referential
tags arrive in v0.4. In v0.4 a declared tag becomes referential by default (a tag that is also
the data column of the same name) and `tag_only_columns` keeps a tag plain, so code that must
keep today's behaviour can pass `tag_only_columns=True` now. `ListSource`, `DerivedSource`,
`CachedSource` and `PollingSource` do not take the option. No hash or golden value changes;
the golden fixtures that build a source with tag columns now pass `tag_only_columns=True`
explicitly.
- A saved pipeline or job file does not record `tag_only_columns`: v0.3 writes no such key, and
both file format versions stay `0.2.0`. Plain tags are decided by
the format version: every source in a file at `0.1.0` or `0.2.0` has all its tags plain. The
format that brings referential tags (v0.4) writes `tag_only_columns` for every source, and its
reader refuses a source config without it, so a v0.3 file reads back plain in v0.4.
19 changes: 19 additions & 0 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,25 @@ Two construction rules apply to every source and are the most common first error
- At least one column must be left over as a data column, otherwise it raises
`ValueError: No data columns found in the table`.

**Plain tags and `tag_only_columns`.** The sources built from a table (`ArrowTableSource`,
`DictSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource` and the database sources) also
take the keyword-only option `tag_only_columns`. In v0.3 every tag column is plain: it is a tag only,
never also a data column. `tag_only_columns=True` says so explicitly; it is the default and
the only value v0.3 accepts, and it changes no hash, source id, record id or saved config. Any
other value is refused. In v0.4 tags become referential by default (a tag that is also the
data column of the same name), and `tag_only_columns` keeps a tag plain, so code that must
keep today's behaviour passes `tag_only_columns=True` now.

```python
from orcapod.sources import DictSource

plain = DictSource(
data=[{"k": "a", "x": 1}], tag_columns=["k"], tag_only_columns=True
)
print(plain.keys())
# (('k',), ('x',))
```

Data with no rows has no columns to read, so `DictSource(data=[], ...)` raises
`ValueError: Table is empty`. To build a source with no rows, pass `data_schema=` so the
columns are known.
Expand Down
6 changes: 6 additions & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -689,6 +689,12 @@ sequence before the stream is returned:
3. **Site 5 — Per-row `record_id`:** a UUID v5 computed from `(source_id, row_provenance_token)` and stored per row.
4. Two system tag columns, `_tag_source_id::{schema_hash}` and `_tag_record_id::{schema_hash}`, are appended to every row.

`tag_only_columns=True` (every tag plain; the default and the only value v0.3 accepts) enters
none of these sites: a source built with it has the schema hash, `source_id`, record ids and
saved config of the same source built without it. Every golden fixture that builds a source
with tag columns passes `tag_only_columns=True`, so the golden values of data context `v1` pin
plain tags, and stay pinned when v0.4 makes tags referential by default.

At Join time (site 6), all existing system tag columns are **renamed** by appending a
topology suffix, encoding the full join lineage into each column name.

Expand Down
5 changes: 5 additions & 0 deletions docs/tutorial/01-first-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ Every column of a source is either a tag column or a data column.
uses tags to match rows when it combines tables.
- **Data columns** are everything else: the values that were measured or recorded.

A tag column is a tag only, never also a data column. `CSVSource`, like every source built
from a table, accepts `tag_only_columns=True`, which says exactly that and is the default in
this version. From Orcapod v0.4 tags become referential by default (a tag that is also the
data column of the same name), and `tag_only_columns=True` keeps them plain as they are here.

You choose the tag columns; all remaining columns become data columns. `output_schema()`
returns both groups, with their types:

Expand Down
39 changes: 39 additions & 0 deletions src/orcapod/core/referential_tags.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""The ``tag_only_columns`` option of a source (ITL-1159).

Referential tags arrive in Orcapod v0.4: a declared source tag will then be referential by
default (a handle to the data column of the same name), and ``tag_only_columns`` names the
tags kept plain. In v0.3 every tag is plain. Every source that declares tag columns already
accepts the option, with exactly one value, ``True`` (all tags plain), which is also the
default, so code and tests written for v0.3 can state the plain behaviour explicitly and
keep it when the default changes. Any other value would make a tag referential and is
refused.
"""

from __future__ import annotations

from collections.abc import Iterable

TagOnlyColumns = str | Iterable[str] | bool
"""The ``tag_only_columns`` option of a source: names, or ``True`` for every tag."""


def check_tag_only_columns(tag_only_columns: TagOnlyColumns) -> None:
"""Refuse a ``tag_only_columns`` value that would make a tag referential.

``tag_only_columns=True`` keeps every declared tag plain (tag only, not data), which is
what every source does in v0.3; it is also the default. Any other value (``False``,
names, an empty collection, even a collection listing every tag) is refused:
referential tags arrive in v0.4.

Args:
tag_only_columns: The tags kept plain; only ``True`` (all of them) is accepted.

Raises:
ValueError: If ``tag_only_columns`` is anything but ``True``.
"""
if tag_only_columns is not True:
raise ValueError(
f"tag_only_columns={tag_only_columns!r}: every tag is plain in Orcapod v0.3, "
"so only True (the default) is accepted. Referential tags (tag_only_columns "
"naming some of the tags, an empty collection, or False) arrive in v0.4."
)
9 changes: 9 additions & 0 deletions src/orcapod/core/sources/arrow_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from collections.abc import Collection
from typing import TYPE_CHECKING, Any, ClassVar

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.stream_builder import SourceStreamBuilder
from orcapod.utils import arrow_utils
Expand Down Expand Up @@ -67,6 +68,11 @@ class ArrowTableSource(RootSource):
table hash, which changes with row order.
Whenever row order can vary between runs, pass an explicit ``source_id``
and set ``record_id_column`` to a column that is unique per row.

Plain tags: ``tag_only_columns`` (keyword-only) is ``True``, the default, and keeps
every tag column a tag only, not data, as every source does in v0.3. Any other value
is refused: referential tags arrive in v0.4, where tags become referential by default
and ``tag_only_columns`` names the tags kept plain.
"""

# The config type key; also the kind prefix of the default source id.
Expand All @@ -79,8 +85,11 @@ def __init__(
system_tag_columns: Collection[str] = (),
record_id_column: str | None = None,
infer_nullable: bool = False,
*,
tag_only_columns: TagOnlyColumns = True,
**kwargs: Any,
) -> None:
check_tag_only_columns(tag_only_columns)
super().__init__(**kwargs)
tag_columns = _normalize_column_list(tag_columns)

Expand Down
10 changes: 10 additions & 0 deletions src/orcapod/core/sources/csv_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from typing_extensions import override

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.reload import keep_nullable_columns, same_rows
from orcapod.core.sources.stream_builder import (
Expand Down Expand Up @@ -41,6 +42,11 @@ class CSVSource(RootSource):
positions are record ids; in any order with one), holds the new rows and
moves its modified time. Record ids keep their rule: the ``record_id_column``
value, or the row's position.

Plain tags: ``tag_only_columns`` (keyword-only) is ``True``, the default, and keeps
every tag column a tag only, not data, as every source does in v0.3. Any other value
is refused: referential tags arrive in v0.4, where tags become referential by default
and ``tag_only_columns`` names the tags kept plain.
"""

def __init__(
Expand All @@ -51,8 +57,11 @@ def __init__(
record_id_column: str | None = None,
source_id: str | None = None,
schema: pa.Schema | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
**kwargs: Any,
) -> None:
check_tag_only_columns(tag_only_columns)
if source_id is None:
source_id = file_path
super().__init__(source_id=source_id, **kwargs)
Expand Down Expand Up @@ -159,6 +168,7 @@ def from_config(cls, config: dict[str, Any], db_registry=None) -> CSVSource:
return cls(
file_path=config["file_path"],
tag_columns=config.get("tag_columns", ()),
tag_only_columns=config.get("tag_only_columns", True),
system_tag_columns=config.get("system_tag_columns", ()),
record_id_column=config.get("record_id_column"),
source_id=config.get("source_id"),
Expand Down
9 changes: 9 additions & 0 deletions src/orcapod/core/sources/data_frame_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from collections.abc import Collection
from typing import TYPE_CHECKING, Any, ClassVar

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.stream_builder import SourceStreamBuilder
from orcapod.utils import arrow_utils, polars_data_utils
Expand All @@ -26,6 +27,11 @@ class DataFrameSource(RootSource):

The DataFrame is converted to an Arrow table and enriched by
``SourceStreamBuilder`` (source-info, schema-hash, system tags).

Plain tags: ``tag_only_columns`` (keyword-only) is ``True``, the default, and keeps
every tag column a tag only, not data, as every source does in v0.3. Any other value
is refused: referential tags arrive in v0.4, where tags become referential by default
and ``tag_only_columns`` names the tags kept plain.
"""

# The config type key; also the kind prefix of the default source id.
Expand All @@ -38,8 +44,11 @@ def __init__(
system_tag_columns: Collection[str] = (),
source_id: str | None = None,
schema: pa.Schema | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
**kwargs: Any,
) -> None:
check_tag_only_columns(tag_only_columns)
super().__init__(source_id=source_id, **kwargs)

df = pl.DataFrame(data)
Expand Down
8 changes: 8 additions & 0 deletions src/orcapod/core/sources/db_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

from typing_extensions import override

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.reload import keep_nullable_columns, same_rows
from orcapod.core.sources.stream_builder import (
Expand Down Expand Up @@ -63,6 +64,11 @@ class DBTableSource(RootSource):
label: Human-readable label for this source node.
data_context: Data context governing type conversion and hashing.
config: Orcapod configuration (controls hash character counts, etc.).
tag_only_columns: Tag columns kept as tags only, not data, exactly as in
v0.3; ``True`` for all of them. Keyword-only. ``True`` is the default and
the only value v0.3 accepts: any other value is refused, because
referential tags arrive in v0.4 (tags then become referential by default
and ``tag_only_columns`` keeps a tag plain).

Raises:
ValueError: If the table is not found, has no PK columns and none are
Expand All @@ -81,8 +87,10 @@ def __init__(
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
_query: str | None = None,
) -> None:
check_tag_only_columns(tag_only_columns)
if source_id is None:
source_id = table_name
super().__init__(
Expand Down
10 changes: 10 additions & 0 deletions src/orcapod/core/sources/delta_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from typing_extensions import override

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.reload import same_rows
from orcapod.core.sources.stream_builder import (
Expand Down Expand Up @@ -38,6 +39,11 @@ class DeltaTableSource(RootSource):
version that only compacted files or
changed metadata changes nothing), holds the new rows and moves its modified time.
Record ids keep their rule: the ``record_id_column`` value, or the row's position.

Plain tags: ``tag_only_columns`` (keyword-only) is ``True``, the default, and keeps
every tag column a tag only, not data, as every source does in v0.3. Any other value
is refused: referential tags arrive in v0.4, where tags become referential by default
and ``tag_only_columns`` names the tags kept plain.
"""

def __init__(
Expand All @@ -47,8 +53,11 @@ def __init__(
system_tag_columns: Collection[str] = (),
record_id_column: str | None = None,
source_id: str | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
**kwargs: Any,
) -> None:
check_tag_only_columns(tag_only_columns)
resolved = Path(delta_table_path).resolve()

if source_id is None:
Expand Down Expand Up @@ -156,6 +165,7 @@ def from_config(
return cls(
delta_table_path=config["delta_table_path"],
tag_columns=config.get("tag_columns", ()),
tag_only_columns=config.get("tag_only_columns", True),
system_tag_columns=config.get("system_tag_columns", ()),
record_id_column=config.get("record_id_column"),
source_id=config.get("source_id"),
Expand Down
9 changes: 9 additions & 0 deletions src/orcapod/core/sources/dict_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from collections.abc import Collection, Mapping
from typing import TYPE_CHECKING, Any, ClassVar

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.base import RootSource
from orcapod.core.sources.stream_builder import SourceStreamBuilder
from orcapod.types import DataValue, SchemaLike
Expand All @@ -23,6 +24,11 @@ class DictSource(RootSource):
Each dict becomes one (tag, data) pair in the stream. The dicts are
converted to an Arrow table via the data-context type converter, then
enriched by ``SourceStreamBuilder`` (source-info, schema-hash, system tags).

Plain tags: ``tag_only_columns`` (keyword-only) is ``True``, the default, and keeps
every tag column a tag only, not data, as every source does in v0.3. Any other value
is refused: referential tags arrive in v0.4, where tags become referential by default
and ``tag_only_columns`` names the tags kept plain.
"""

# The config type key; also the kind prefix of the default source id.
Expand All @@ -35,8 +41,11 @@ def __init__(
system_tag_columns: Collection[str] = (),
data_schema: SchemaLike | pa.Schema | None = None,
source_id: str | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
**kwargs: Any,
) -> None:
check_tag_only_columns(tag_only_columns)
super().__init__(source_id=source_id, **kwargs)
tag_columns = _normalize_column_list(tag_columns)

Expand Down
11 changes: 11 additions & 0 deletions src/orcapod/core/sources/postgresql_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

from typing_extensions import override

from orcapod.core.referential_tags import TagOnlyColumns, check_tag_only_columns
from orcapod.core.sources.db_table_source import DBTableSource
from orcapod.databases.postgresql_connector import PostgreSQLConnector

Expand Down Expand Up @@ -57,6 +58,11 @@ class PostgreSQLTableSource(DBTableSource):
label: Human-readable label for this source node.
data_context: Data context governing type conversion and hashing.
config: OrcaPod configuration.
tag_only_columns: Tag columns kept as tags only, not data, exactly as in
v0.3; ``True`` for all of them. Keyword-only. ``True`` is the default and
the only value v0.3 accepts: any other value is refused, because
referential tags arrive in v0.4 (tags then become referential by default
and ``tag_only_columns`` keeps a tag plain).

Raises:
ValueError: If the table is not found, is empty, or has no PK and
Expand All @@ -75,7 +81,10 @@ def __init__(
label: str | None = None,
data_context: str | contexts.DataContext | None = None,
config: OrcapodConfig | None = None,
*,
tag_only_columns: TagOnlyColumns = True,
) -> None:
check_tag_only_columns(tag_only_columns)
self._dsn = dsn # store before try — needed by to_config even if super() raises
connector = PostgreSQLConnector(
dsn
Expand All @@ -91,6 +100,7 @@ def __init__(
label=label,
data_context=data_context,
config=config,
tag_only_columns=tag_only_columns,
)
finally:
try:
Expand Down Expand Up @@ -136,6 +146,7 @@ def from_config(
dsn=config["dsn"],
table_name=config["table_name"],
tag_columns=config.get("tag_columns"),
tag_only_columns=config.get("tag_only_columns", True),
system_tag_columns=config.get("system_tag_columns", ()),
record_id_column=config.get("record_id_column"),
source_id=config.get("source_id"),
Expand Down
Loading
Loading