Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ae88c11
test(spiraldb): scan log in the pyspiral fake and streaming source te…
Nov 12, 2026
042d5c5
docs(specs): ITL-186 D3 key digest definition and D2 measurements
Nov 12, 2026
cbcf806
wip(spiraldb): streaming SpiralDBTableSource built from metadata (ITL…
Nov 12, 2026
2376fed
wip(spiraldb): tests for the streaming source on the fake (ITL-186)
Nov 12, 2026
83fca78
wip(spiraldb): docs, changelog fragment, design issue, live tests (IT…
Nov 12, 2026
012a86e
feat(spiraldb)!: SpiralDBTableSource reads no rows when built and str…
Nov 12, 2026
3e5773e
feat(spiraldb): a displayed SpiralDBTableSource reads a 10-row head (…
Nov 12, 2026
3bca8ec
fix(spiraldb): answer CodeRabbit round 1 on the streaming source (ITL…
Nov 12, 2026
23bb59d
Merge remote-tracking branch 'origin/main' into eywalker/itl-186-add-…
Nov 12, 2026
dba32fa
test(spiraldb): type the bool-size and live asof tests (ITL-186)
Nov 12, 2026
94980c1
refactor(spiraldb): the preview reads a head with its size as data (I…
Nov 12, 2026
bb8e464
fix(spiraldb): count the pinned version for the cap; escape the displ…
Nov 12, 2026
848d90b
docs(spiraldb): docstrings for the Spiral source tests and the fake (…
Nov 12, 2026
68f8fd1
test(spiraldb): failing tests for tags_as_data on SpiralDBTableSource…
Nov 13, 2026
0483972
feat(spiraldb): tags_as_data copies key columns into data as rows are…
Nov 13, 2026
82000e1
docs(spiraldb): key columns as data; changelog fragment and design is…
Nov 13, 2026
48fbda5
Merge origin/main (PR #333 squash-merged) into ITL-996
Nov 13, 2026
a1796df
docs(plan): ITL-996 progress notes
Nov 13, 2026
158a661
fix(spiraldb): tags_as_data is the last parameter; precise DerivedSou…
Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions changelog.d/ITL-996.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
### Added

- `SpiralDBTableSource(..., tags_as_data={"timestamp": "spike_time_us"})` also offers named
key columns as data columns, copied in Arrow as each batch is read, so a function pod can
use a key such as a spike's time (ITL-996). The tag keeps its name; the copy has the key
column's Arrow type and field metadata and is never null. The source still reads no rows
when built, and `limit=`, `where=`, `max_rows` and the notebook preview work as before. The
copies are part of the schema, so the `content_hash` and `pipeline_hash` of a source built
with the option differ from the same source without it; the mapping itself is also part of
the `content_hash`. The `source_id`, record IDs and source-info tokens are unchanged, and
without the option nothing changes. A mapping that names a column that is not a key tag
column, or a new name that is already a column, repeated or reserved, raises
`InputValidationError` when the source is built. `DerivedSource`'s `tags_as_data` errors no
longer say "of the origin". See "Key columns as data" in `docs/concepts/sources.md`.
19 changes: 19 additions & 0 deletions design_issues/ITL-996-SP5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
## `src/orcapod/core/sources/spiraldb_table_source.py` — Spiral tables as a workshop source (ITL-926)

### SP5 — A key a function pod needed could only be copied into data by reading the whole table
**Status:** resolved
**Severity:** medium
**Issue:** ITL-996

In the EDI tables every time series has `timestamp` in its key, so the time of a spike is a
tag, and a function pod never sees tags. Copying it into data took
`DerivedSource(src, tags_as_data=...)`, which reads the whole origin as soon as its schema or
rows are asked for, as building a job does (7 to 8 seconds per source, measured), and is no
longer a Spiral source, so it loses `limit=`, `where=` and the bounded notebook preview.

**Fix:** `SpiralDBTableSource(..., tags_as_data={key column: new name})` copies the named key
columns into data in Arrow as each batch is read. The copy has the key column's type and field
metadata, non-nullable. Validation is `DerivedSource`'s rule (`validate_tags_as_data`) plus
"the tag must be a key column". The copies are in the schema (content and pipeline hash), the
sorted mapping is in the content hash when given, and the source id and record IDs do not
change.
35 changes: 35 additions & 0 deletions docs/concepts/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,41 @@ built, so rows added to the Spiral table afterwards are seen only by a new sourc
the source (and the job around it) again, and the added rows are new keys, the only rows
computed.

**Key columns as data.** A function pod never receives a tag, and in a time-series table the
time is part of the key, so it is a tag. `tags_as_data` also offers named key columns as data
columns, copied as each batch is read (so the source still reads no rows when built, and
`limit=`, `where=`, `max_rows` and the notebook preview work as before). The tag keeps its
name; you name the copy, which comes after the table's data columns, with the key column's
type, never null:

<!-- docs-test: skip -->
```python
spikes = SpiralDBTableSource(
"edi-tutorial-v1-841226", "spike_event", tags_as_data={"timestamp": "spike_time_us"}
)
unit_key = [
"spike_extraction_hash", "subject_id", "session_start_datetime",
"recording_index", "probe_index", "unit_id",
]
per_unit = spikes.group_by(unit_key).drop_tag_columns("timestamp")
print(per_unit.keys())
# (('spike_extraction_hash', 'subject_id', 'session_start_datetime', 'recording_index', 'probe_index', 'unit_id'), ('spike_amplitude_uv', 'spike_time_us'))
```

`per_unit` has one row per unit, and its `spike_time_us` list is in time order: `group_by`
orders the members of a group by the tags outside the group key, and on the full unit key
the only such tag is `timestamp`. `group_by(["unit_id"])` alone sorts by
`spike_extraction_hash`, subject, session, recording, probe, then `timestamp`, so time order
holds only within one session and probe; group on the full unit key.

The copies are data columns, so they are part of the schema, the `content_hash` and the
`pipeline_hash`; the mapping itself is part of the `content_hash` too, because two mappings
can give the same schema while the copy holds different values. The `source_id` and every
row's record id are the same as without the option. Compared with
`DerivedSource(spikes, tags_as_data=...)`, which reads the whole table as soon as its schema
or rows are asked for (building a job asks for the schema), the option keeps a Spiral source,
whose schema is known without a scan.

**Known limitations.**

- Each row's key is hashed with one call, about 0.2 ms per row (about 18 s per 100,000
Expand Down
20 changes: 11 additions & 9 deletions src/orcapod/core/sources/derived_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,20 +58,22 @@ def _derivation_hash(
)


def _validate_tags_as_data(
def validate_tags_as_data(
tags_as_data: Mapping[str, str],
tag_keys: tuple[str, ...],
data_keys: tuple[str, ...],
) -> None:
"""Check a ``tags_as_data`` mapping against the origin's columns.
"""Check a ``tags_as_data`` mapping against the columns it copies from.

The one rule for ``DerivedSource`` and ``SpiralDBTableSource``.

Args:
tags_as_data: Maps a tag column of the origin to the name of its data copy.
tag_keys: The origin's user tag columns.
data_keys: The origin's data columns.
tags_as_data: Maps a tag column to the name of its data copy.
tag_keys: The user tag columns of the stream copied from.
data_keys: The data columns of the stream copied from.

Raises:
InputValidationError: If a key is not a tag column of the origin, or a new
InputValidationError: If a key is not a tag column, or a new
name is already a column, is used twice, or starts with the prefix
reserved for system columns.
"""
Expand All @@ -80,13 +82,13 @@ def _validate_tags_as_data(
if tag not in tag_keys:
kind = "a data column" if tag in data_keys else "not a column"
raise InputValidationError(
f"tags_as_data: {tag!r} is {kind} of the origin, not a tag column. "
f"tags_as_data: {tag!r} is {kind}, not a tag column. "
f"Tag columns: {list(tag_keys)}."
)
if name in tag_keys or name in data_keys:
raise InputValidationError(
f"tags_as_data: cannot copy tag {tag!r} to {name!r}, which is "
"already a column of the origin. Choose a new name."
"already a column. Choose a new name."
)
if name in seen:
raise InputValidationError(
Expand Down Expand Up @@ -233,7 +235,7 @@ def __init__(
else:
self._tags_as_data = dict(tags_as_data)
tag_keys, data_keys = origin.keys()
_validate_tags_as_data(self._tags_as_data, tag_keys, data_keys)
validate_tags_as_data(self._tags_as_data, tag_keys, data_keys)
kwargs.setdefault("data_context", origin.data_context_key)

self._explicit_source_id = source_id
Expand Down
110 changes: 100 additions & 10 deletions src/orcapod/core/sources/spiraldb_table_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@
"my-project-123456", "spike_event", where={"unit_id": 106}
)

# The spike time is a key column (a tag); also offer it as data
source = SpiralDBTableSource(
"my-project-123456", "spike_event", tags_as_data={"timestamp": "spike_time_us"}
)

# Explicit dataset, tag columns overridden
source = SpiralDBTableSource(
"my-project-123456", "spike_event", dataset="prod", tag_columns=["unit_id"]
Expand All @@ -35,10 +40,11 @@
from typing import TYPE_CHECKING, Any, NamedTuple

from orcapod.core.sources.base import RootSource
from orcapod.core.sources.derived_source import validate_tags_as_data
from orcapod.core.sources.stream_builder import SourceStreamBuilder
from orcapod.core.streams.arrow_table_stream import ArrowTableStream
from orcapod.databases.spiraldb_connector import SpiralDBConnector
from orcapod.errors import SourceTooLargeError
from orcapod.errors import InputValidationError, SourceTooLargeError
from orcapod.types import ColumnConfig
from orcapod.utils import arrow_utils
from orcapod.utils.lazy_module import LazyModule
Expand Down Expand Up @@ -165,7 +171,18 @@ class SpiralDBTableSource(RootSource):
the table id is SpiralDB's own: a table dropped and created again under the same
name is a different source, and a renamed table gets a new source id too.
``where``, ``limit`` and ``asof`` choose which rows are read and are not part of
the source's identity.
the source's identity. ``tags_as_data`` is: it decides what a data column holds,
so its sorted items enter the content hash (when it is given), and its copies
are columns of the schema, so they enter both the content and the pipeline hash.
The source id and every row's record ID do not depend on it.

Key columns as data. A function pod never sees tags, so a key it needs (a
spike's time) is offered as data too with ``tags_as_data={"timestamp":
"spike_time_us"}``. The tag keeps its name; the copy is a data column of the
given name, after the table's own data columns in mapping order, with the key
column's Arrow type and field metadata, non-nullable (a key is never null). It is
copied in Arrow as each batch is read, and its source-info value has the form of
every data column's, ``<source id>::<row token>::<copy name>``.

Record identity. A row is identified by its primary key: the row's token is the
data context's Arrow hash of its key columns (in key order, as non-nullable
Expand Down Expand Up @@ -199,11 +216,17 @@ class SpiralDBTableSource(RootSource):
config: OrcaPod configuration.
overrides: Optional pyspiral client config overrides passed through to
``SpiralDBConnector``, e.g. ``{"server.url": "http://api.spiraldb.dev"}``.
tags_as_data: Maps a key column that is a tag column of this source to the
name of a data column that receives a copy of it, as in
``DerivedSource``. ``None`` (default) or an empty mapping copies nothing.

Raises:
ValueError: If the table is not found, has no key schema, a tag column or a
``where`` column is not in the table, or ``limit``, ``max_rows`` or
``batch_rows`` is not a positive integer.
InputValidationError: If ``tags_as_data`` names a column that is not a tag
column or not a key column, or a new name that is already a column, is
given twice, or is reserved for system columns.
"""

# Whether a notebook display reads the first rows. False shows the schema and the
Expand All @@ -228,6 +251,7 @@ def __init__(
data_context: "str | contexts.DataContext | None" = None,
config: "OrcapodConfig | None" = None,
overrides: dict[str, str] | None = None,
tags_as_data: Mapping[str, str] | None = None,
) -> None:
for name, value in (("limit", limit), ("max_rows", max_rows)):
if value is not None and not _is_positive_int(value):
Expand All @@ -251,6 +275,7 @@ def __init__(
self._max_rows = max_rows
self._batch_rows = batch_rows
self._given_asof = asof
self._tags_as_data: dict[str, str] = dict(tags_as_data) if tags_as_data else {}
location = f"SpiralDB project {project_id!r}, dataset {dataset!r}"

with self._connector() as connector:
Expand Down Expand Up @@ -301,13 +326,63 @@ def __init__(
)
except ValueError as exc:
raise ValueError(f"{location}, table {table_name!r}: {exc}") from exc
self._table_schema: pa.Schema = arrow_utils.drop_system_columns(
schema.empty_table()
).schema
if self._tags_as_data:
try:
self._check_tags_as_data(template.stream.keys())
except InputValidationError as exc:
raise InputValidationError(
f"{location}, table {table_name!r}: {exc}"
) from exc
# The copies are columns of the schema: build the template with them.
template = self._builder.build(
self._with_tag_copies(self._table_schema.empty_table()),
tag_columns=template.tag_columns,
source_id=self._source_id,
system_tag_columns=template.system_tag_columns,
)
self._stream = template.stream
self._schema_hash = template.schema_hash
self._tag_columns = template.tag_columns
self._system_tag_columns = template.system_tag_columns
self._table_schema: pa.Schema = arrow_utils.drop_system_columns(
schema.empty_table()
).schema

def _check_tags_as_data(
self, keys: tuple[tuple[str, ...], tuple[str, ...]]
) -> None:
"""Check ``tags_as_data`` against the source's tag and data columns.

Args:
keys: The source's tag and data columns, without the copies.

Raises:
InputValidationError: If the mapping fails ``DerivedSource``'s rule, or a
tag it names is not a key column of the table.
"""
tag_keys, data_keys = keys
validate_tags_as_data(self._tags_as_data, tag_keys, data_keys)
for tag in self._tags_as_data:
if tag not in self._key_columns:
raise InputValidationError(
f"tags_as_data: {tag!r} is not a key column, so it may hold a "
f"null. Key columns, which can be copied: {list(self._key_columns)}."
)

def _with_tag_copies(self, table: "pa.Table") -> "pa.Table":
"""Return ``table`` with each ``tags_as_data`` copy appended, in mapping order.

A copy is the key column's own Arrow column under a field of the new name,
with the key's type and field metadata and declared non-nullable (a key is
never null; a read checks it before copying).
"""
for tag, name in self._tags_as_data.items():
key = table.schema.field(tag)
table = table.append_column(
pa.field(name, key.type, nullable=False, metadata=key.metadata),
table.column(tag),
)
return table

# -------------------------------------------------------------------------
# Identity
Expand All @@ -327,10 +402,20 @@ def _restriction_identity(self) -> None:
return None

def identity_structure(self) -> Any:
"""Identity from the class, the schema and the source id (the table's locator)."""
base = super().identity_structure()
"""Identity from the class, the schema and the source id (the table's locator).

With ``tags_as_data`` given, its sorted items follow: two mappings can give
the same schema (two key columns of one type copied to one name) while the
copy holds different values. Without it the structure is the plain one.
"""
extra: list[Any] = []
restriction = self._restriction_identity()
return base if restriction is None else (*base, restriction)
if restriction is not None:
extra.append(restriction)
if self._tags_as_data:
extra.append(("tags_as_data", tuple(sorted(self._tags_as_data.items()))))
base = super().identity_structure()
return (*base, *extra) if extra else base

# -------------------------------------------------------------------------
# Reading
Expand Down Expand Up @@ -470,13 +555,16 @@ def _read_batches(
seen += rows.num_rows
if max_rows is not None and seen > max_rows:
raise self._too_large("the read returned more rows")
# Tokens first: they refuse a null key before it is copied into
# a non-nullable column.
tokens = self._key_tokens(rows)
yield self._builder.attach_provenance(
rows,
self._with_tag_copies(rows),
tag_columns=self._tag_columns,
system_tag_columns=self._system_tag_columns,
schema_hash=self._schema_hash,
source_id=self.source_id,
provenance_tokens=self._key_tokens(rows),
provenance_tokens=tokens,
)

def iter_data(self) -> Iterator["tuple[TagProtocol, DataProtocol]"]:
Expand Down Expand Up @@ -622,6 +710,7 @@ def to_config(self, db_registry=None) -> dict[str, Any]:
"asof": self._given_asof,
"max_rows": self._max_rows,
"batch_rows": self._batch_rows,
"tags_as_data": dict(self._tags_as_data) or None,
"source_id": self.source_id,
**self._identity_config(),
}
Expand Down Expand Up @@ -654,6 +743,7 @@ def from_config(
asof=config.get("asof"),
max_rows=config.get("max_rows", DEFAULT_MAX_ROWS),
batch_rows=config.get("batch_rows", DEFAULT_BATCH_ROWS),
tags_as_data=config.get("tags_as_data"),
source_id=config.get("source_id"),
label=config.get("label"),
data_context=config.get("data_context"),
Expand Down
Loading
Loading