diff --git a/.zed/rules b/.zed/rules index 770557215..dc1849eb3 100644 --- a/.zed/rules +++ b/.zed/rules @@ -949,6 +949,14 @@ data. The rules: path (_kept_output: a job run, sync or async, and node.run() on its own; ITL-1133), and reads a stored operator cache in row order (_in_row_order_on_read). Never change the order of any other populated table: stored tables depend on it. +- Data-tags come first among the data columns (developer, November 17, 2026, ITL-1205): + a stream lists its data-tags first in its data keys and data schema, in tag order, then + its other data columns in table order. A table built from rows lays a data-tag out among + the tags, so this is every row's order, on every path. ArrowTableStream applies it to + its keys (its table keeps its column order) and every operator's prediction applies it + through data_tags.data_tags_first. No hash of a stream, schema, row or table depends on + the order of a stream's columns, so the order moves none; a library value's own hash (a + polars or pandas DataFrame in a cell) does include that value's column order. - Type strictness (developer, November 10, 2026): Orcapod is type strict. It never expands or converts a type implicitly; the only exception is normalization to large variants. Inputs that disagree on a column's type are an error naming the column and both types, on diff --git a/CLAUDE.md b/CLAUDE.md index 29ed65fd2..1f28e6289 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -991,6 +991,14 @@ data. The rules: path (`_kept_output`: a job run, sync or async, and `node.run()` on its own; ITL-1133), and reads a stored operator cache in row order (`_in_row_order_on_read`). Never change the order of any other populated table: stored tables depend on it. +- **Data-tags come first among the data columns** (developer, November 17, 2026, ITL-1205): + a stream lists its data-tags first in its data keys and data schema, in tag order, then + its other data columns in table order. A table built from rows lays a data-tag out among + the tags, so this is every row's order, on every path. `ArrowTableStream` applies it to + its keys (its table keeps its column order) and every operator's prediction applies it + through `data_tags.data_tags_first`. No hash of a stream, schema, row or table depends on + the order of a stream's columns, so the order moves none; a library value's own hash (a + polars or pandas DataFrame in a cell) does include that value's column order. - **Type strictness** (developer, November 10, 2026): Orcapod is type strict. It never expands or converts a type implicitly; the only exception is normalization to large variants. Inputs that disagree on a column's type are an error naming the column and both types, on diff --git a/changelog.d/ITL-1023.md b/changelog.d/ITL-1023.md new file mode 100644 index 000000000..3b58c7f7e --- /dev/null +++ b/changelog.d/ITL-1023.md @@ -0,0 +1,223 @@ +### Breaking changes and migration + +- A source's declared tag columns are now data-tags by default (ITL-1023, specification + "Orcapod v0.4 — Referential Tags"): each is also a data column of the same name, stored + once, so a function can read it and joins and groups use it as a tag. A script run + unchanged gets new schemas and hashes for such sources. Pass `tag_only_columns=[...]` (or + `tag_only_columns=True`) to keep tags plain, exactly as before; every existing golden hash + is unchanged under that option. +- The term is "data-tag" (developer decision, November 17, 2026: the three column roles + are tag, data and data-tag; on the integration branch it was "referential tag"). Every + name follows it: `ArrowTableStream(data_tags=...)` and `.data_tags`, `stream_data_tags`, + `FallThroughPod` (`.fall_through_tags`), `DataTagMaterializationWarning` and + `data_tags.data_tag_materialization`, the module `orcapod.core.data_tags`. The stored + marker is `orcapod:tag_kind` = `data_tag`, and a saved function node writes + `fall_through_tags`. Stores and saved jobs written by a development build of the + integration branch (marker value `referential`, key `fall_through_references`) are + refused on read and on load, not migrated: clear such stores and save such jobs + again. The identity of a raw `ArrowTableStream` with data-tags and of every + fall-through pod moved with the rename (provisional identities, see Known limitations), + and so did the stored path of a fall-through node and of every node downstream of it. + +### Added + +- `tag_only_columns` on `DictSource`, `ArrowTableSource`, `CSVSource`, `DataFrameSource`, + `DeltaTableSource`, the database sources and `SpiralDBTableSource`. +- `Join` merges the data columns of a data-tag both inputs join on into one column; + plain with data-tag gives a data-tag. `GroupBy` keeps a data-tag key scalar + in both roles. A data-tag non-key loses its tag role, as every non-key tag does + (ITL-1070), but keeps its data column as a list with one element and one source-info + token per member; the members are ordered by the tuple of the input's non-key tags, + the same order for every data list of the group. +- A source-info column has the shape of its data column. A value merged from several equal + elements (a data-tag `GroupBy` key; a `Join` key that is a data-tag on several inputs) has + one merged token instead of a list of every member's token (ITL-1017): the merging + operation's fold (`arrow_utils.fold_system_tag_values`), `group_v1:` over the + members' tokens in member order for a `GroupBy` key, `join_v1:` over the members' + tokens sorted for a `Join` key. No hash or stored id changes. +- `MapTags` renames a data-tag in both roles; `tag_only=True` renames the tag only. + `MapData` is a data-only move. +- Removing the data column of a data-tag (`SelectDataColumns`, `DropDataColumns`) + leaves a plain tag and emits a `DataTagMaterializationWarning` + (`orcapod.core.data_tags.data_tag_materialization = "silent"` hides it). +- A function output named like a tag of its input is an error when the pipeline is built. +- Calling a function pod, `pod(stream)`, lets the input's data-tags fall through: + the output rows carry the function's outputs and the row's own data-tag columns, so + the tags stay data-tags (`FallThroughPod`). The function, its result identity + and its result store are the plain pod's. `pod.process(stream)` is unchanged: its + output data is the function's outputs and every output tag is plain. +- Saved job and pipeline files are at format `0.3.0`; files at `0.1.0` and `0.2.0` load + with every declared source tag plain, as they were written. + +- `Batch` works on data-tags (ITL-1017; developer decision, November 16, 2026: + "Batch always simply takes anything and puts into list"). As for plain tags, every tag + becomes a list; a data-tag is one column, so it is the same list in both roles and + stays a data-tag. Its source info is one token per member, like any data column: nothing + is merged, so no token is folded. System tags fold as before. Plain tags are unchanged. + +- A stored table states which of its tags are data-tags (ITL-1019): a function node's + pipeline table marks each data-tag column with Arrow field metadata + (`orcapod:tag_kind` = `data_tag`, `data_tags.TAG_KIND_METADATA_KEY`); a + plain tag carries no marker. The result table never records data-tag status. The + marker moves no hash and no stored path, and needs no new table layout. +- A function node reads its bindings back from its pipeline table and refuses + (`InputValidationError`, naming the table and both lists) a table whose marker disagrees + with what the node declares: its pod's fall-through tags, or a loaded job's saved record. +- A job or pipeline holding data-tags and fall-through pods can be saved and + loaded; it keeps its hashes and bindings. `FallThroughPod.to_config` and + `from_config` work. +- An operator node stores a stream with data-tags (`LOG`) and replays it (`REPLAY`): its + table marks the data-tags of the operator's own output with the same marker, + never one that came in from upstream. Every read of the table (a replay, the stored + view `get_all_records()`, a loaded job's read) checks the marker against the node (its + operator's prediction, or a loaded job's saved output schema) and refuses a + disagreement (`InputValidationError`, naming the table and both lists). Source info + stays part of each row's record hash; the marker is put on after the hash and never + enters it. +- `DerivedSource` over a stream with data-tags: `node.as_source()` of a function or operator + node presents the bindings its stored table states (checked against the node's keys), + in the plain form and with `tags_as_data`; `DerivedSource(source, tags_as_data=...)` + over a source with data-tags keeps the source's own. A `tags_as_data` copy of a + data-tag is a new plain data column. Nothing new enters the identity for the + binding: the plain form's content hash is the origin node's, and every pipeline hash + hashes the output schema, where a data-tag is in both halves. A saved derived + source writes no new key: its saved schema states the binding. + +- `FieldMetadataKeepingDatabaseProtocol` (`orcapod.protocols.database_protocols`): a + database states with `keeps_field_metadata` whether it keeps Arrow field metadata. + `InMemoryArrowDatabase`, `DeltaTableDatabase` and a `ConnectorArrowDatabase` over + `SpiralDBConnector` state `True`. The SQLite and PostgreSQL connector databases and + `NoOpArrowDatabase` state `False`. `ExtensionAwareDatabase` states what the database + it wraps states, and a connector database what its connector states. A database + without the member is treated as not keeping field metadata. + +- `SelectColumns` and `DropColumns` (`select_columns`, `drop_columns`) handle data-tags + (ITL-1022). A data-tag is one column in both roles, so naming it acts on both: + selecting it keeps it a data-tag, dropping it (or leaving it out of a select) removes the + column from both roles. The roles are never split and nothing is materialized. Before, + both operators raised "not supported with data-tags yet". + +- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as + data-tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each + named column becomes one column in both roles, with its values and source info + unchanged; the new tags follow the stream's tags in the order given, and that order is + part of the operator's identity. System tags, record ids and every data column are + unchanged. A column a function computed may become a tag and then key a join or a + group. Refused, naming the column: a name that is not a data column of the input, a + name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is + one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a + nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered + for saved jobs and pipelines. + +- `AddTags` (`add_tags`, ITL-1196) works on a stream with data-tags: the input's + data-tags stay data-tags, with their values and source info, and the new tags are + plain. A new tag named like any column of the input, a data-tag included, is refused, + as before. Sync and async agree. + +- `MergeJoin` handles data-tags (ITL-1194), following `Join`: a tag that is a data-tag in + both inputs is one scalar data-tag column, not a two-element list, and its source + info is one merged token, the `join_v1` fold of the two inputs' tokens (sorted), as + `Join` gives. A tag that is a data-tag in one input stays a data-tag with its own token when + the other input has it as a plain tag or not at all; a data-only column of that name in + the other input is refused. A shared key that is `T` in one input and `T | None` in the + other is `T` in both roles (ITL-829). Every other colliding data column is merged into a + sorted list, as before; inputs without data-tags merge exactly as before. + +### Changed + +- Data-tags come first among the data columns, in tag order, then the other data columns + in table order (developer decision, November 17, 2026, ITL-1205): in a stream's + `keys()` and data schema, in every operator's predicted schema, and in every row. A + table built from rows lays a data-tag out among the tags, so the sync path, the async + path and a job node's rows now agree with the keys; before, a source with data + `a, x, y` (`x` a data-tag) listed `a, x, y` while the rows of a node over it, and an + operator's async result, gave `x, a, y`. No table's column order, stored table or hash + changes. `TagBy` places the new data-tags there too. + +- The select and drop operators act on a data-tag by role (developer decision, + November 16, 2026, ITL-1195), as documented in "Moving a column between roles": for a + data-tag `x`, dropping the tag (`drop_tag_columns`) removes the tag role only + and leaves `x` plain data with its source info; dropping the data (`drop_data_columns`) + removes the data role only and leaves `x` a plain tag with the same values; dropping + the column (`drop_columns`) removes it from both roles. `select_tag_columns`, + `select_data_columns` and `select_columns` follow the same rule for the names they do + not select. Removing the data role emits `DataTagMaterializationWarning`; removing the + tag role or the column is silent. No second name (such as `untag`) is added. + +- The error for a name that is a tag in one join input and a data column in the other + also offers `tag_by` as a way out (ITL-1175). + +- A node that would store a data-tag refuses every database that does not state + `keeps_field_metadata = True`, before anything is written, so an unknown database + never passes by default. A database that states `True` is not probed. Before, only a + database that implements `validate_records` was checked, with a probe of null-typed + columns. + +- Saved files of format `0.3.0` state every binding: each source config that takes the + option writes `tag_only_columns` (`True` when every tag is plain), and each function + node writes `fall_through_tags` (empty for a plain pod). The format version decides + what a file means, never a missing key: a file at `0.1.0` or `0.2.0` holds only plain + tags, and a `0.3.0` file without these keys, or whose keys disagree with its saved + schemas, is refused on load. + +- Errors that a data-tag causes name it and the way out; no behaviour changes. + - A bind refused because a column is a data-tag on one side and a plain tag or a + data column on the other (listed as a missing or unexpected column) says which side + holds the data-tag, and how to make the two agree: `tag_only_columns` on the + source, or a slot declared with the column in both roles (ITL-1142). The bind stays + strict. + - `pod.process(stream)` and `pod(stream, strict=True)` refusing a data-tag the + function does not read name the data-tags and the two ways out: `pod(stream)`, + or `tag_only_columns`, each only where it can work (not when no data column of the + input is a parameter of the function, except a plain `FunctionPod`'s call). + - "Not supported with data-tags yet" names `tag_only_columns=True`, which keeps + every tag plain as in v0.3, beside its issue. + +### Known limitations + +- `MapColumns` raises "not supported with data-tags yet" + (operator-node storage and `DerivedSource` over a stream with data-tags are supported, + see ITL-1019 above). + `PollingSource` keeps its tags plain. The identities of streams with data-tags may still + change. + +- The SQLite and PostgreSQL databases keep no Arrow field metadata, so a function node + that would store a data-tag there is refused (`InputValidationError`, naming the + database and the two ways out: a database that keeps field metadata, or + `tag_only_columns`) before anything is written. The same holds for an operator node + attached to such a database in `LOG` or `REPLAY` mode. +- SpiralDB's `keeps_field_metadata` is verified against the pyspiral fake only (first + write, append, merge on overlap, reread). The live service is unverified. The + connector database refuses schema evolution, so no merge can drop the marker. The + SpiralDB connector writes a table's Arrow metadata before its rows, so a failed + metadata write stores no row unmarked; a write that stores no row publishes nothing; + a failed write puts the metadata entry back as it was. +- Concurrent writers to one SpiralDB table through separate connectors are not + coordinated: one writer's rollback of a failed write can remove the metadata entry + another writer just published, so rows can read back without their data-tag + marker. A node refuses such a table on read (the marker must match the node), so no + wrong binding is ever read (ITL-1190). +- Writes of differing field metadata to one table are not refused. The in-memory + database and the connector database combine buffered and committed rows with the + first table's field metadata, so a batch whose marker differs from the rows already + there takes theirs. Orcapod's own writers never do this: a table's path derives from + its schema, where a data-tag is in both halves, so one path has one binding. + Only an outside writer can mix them, and a lost marker is refused on read (ITL-1190). +- A fall-through function node with ephemeral results records the hash of the function's + plain output, not of the output with its data-tag columns, so after the ephemeral + result is lost a downstream node's lookup misses (`EphemeralResultMissingError`; + ITL-1199). +- A loaded job keeps the pending type of a data-tag column whose class is not + registered for one role only, so binding a source of that class can be refused + (ITL-1200). +- `Join` raises `ValueError` on a list-valued data-tag key whose source info is one + token per member on one side (after `Batch`) and one token per cell on the other (a + source's list-valued tag) (ITL-1201). +- Reported, not reproduced: a merged float data-tag key could take `0.0` on one path + and `-0.0` on the other (ITL-1202). +- `CachedSource` neither marks nor checks the data-tags of its cache table, so a source with + data-tags and a plain source sharing an explicit `cache_path` can read each other's table + with the wrong binding (ITL-1215). +- On the async compute path, execution observers of a fall-through function node receive + the function's own input and output, without the data-tag columns that the sync path + and cache hits include (ITL-1203). diff --git a/docs/concepts/data-tags.md b/docs/concepts/data-tags.md new file mode 100644 index 000000000..1131a1cd3 --- /dev/null +++ b/docs/concepts/data-tags.md @@ -0,0 +1,401 @@ +# Data-tags + +!!! warning "Integration branch (ITL-1023)" + Data-tags are the first slice of Orcapod v0.4, on an integration branch. The + identities (content and pipeline hashes) of streams with data-tags are **not + frozen yet** and may change before the release. Streams whose tags are all plain + (`tag_only_columns`) keep their v0.3 identities exactly. + +A tag is either **plain** (it carries its own value, as every tag did in v0.3) or +a **data-tag**: a handle to the data column of the **same name**. A data-tag is +one column seen through both roles, so a key column can be joined and grouped on as a tag +and read by a function as data, without a copy. + +## Sources: declared tags are data-tags by default + +Every tag column a source declares is a data-tag unless you keep it plain with +`tag_only_columns`: + +```python +import orcapod as op + +rows = [{"unit": 1, "time": 10, "amp": 0.5}, {"unit": 2, "time": 20, "amp": 0.25}] + +spikes = op.sources.DictSource(rows, tag_columns=["unit", "time"], source_id="spikes") +print(spikes.keys()) +# (('unit', 'time'), ('unit', 'time', 'amp')) + +plain = op.sources.DictSource( + rows, tag_columns=["unit", "time"], tag_only_columns=True, source_id="spikes" +) +print(plain.keys()) +# (('unit', 'time'), ('amp',)) +``` + +`tag_only_columns` takes the names to keep plain, or `True` for all of them. A name that is +not a declared tag is an error. A source built with all its tags plain is exactly the +source v0.3 built: same columns, same hashes. + +The column is stored once: `as_table()` shows it once, and every row's tag and data hold +the same value. + +Data-tags come first among the data columns, in tag order, then the other data columns in +their table order. Every stream, operator result and row follows this order, whatever the +position of the column in the table. + +```python +print(spikes.as_table().column_names) +# ['unit', 'time', 'amp'] +``` + +## Functions read a data-tag column by name + +```python +@op.function_pod(output_keys=["scaled"]) +def scale(time: int, amp: float) -> float: + return time * amp + + +scaled = scale.pod(spikes) +print(scaled.keys()) +# (('unit', 'time'), ('unit', 'time', 'scaled')) +``` + +Calling the pod, `pod(stream)`, lets the input's data-tags **fall through**: each +output row has the function's outputs and the row's own data-tag columns, so the tags +stay data-tags. The function, its result identity and the database of its results are +those of the plain pod; only the data-tag columns fall through, other input data +columns do not. `pod.process(stream)` is the exact form: there the output data is the +function's outputs only, and every tag of the output is plain, with the value it had. + +```python +plain = scale.pod.process(spikes.select_data_columns(["time", "amp"])) +print(plain.keys()) +# (('unit', 'time'), ('scaled',)) +``` + +A function output named like a tag is an error when the pipeline is built. A function +that does not read a data-tag column keeps its result identity: retagging a source +does not make it compute again. + +## Moving a column between roles + +A data-tag is one column in two roles, a tag and a data column. Four steps move a +column between the roles, and none of them changes, copies or creates a value: + +| Step | Data column `x` | Data-tag `x` | Plain tag `x` | +|---|---|---|---| +| `tag_by("x")` | becomes a data-tag | refused: already a tag | refused: not a data column | +| `drop_tag_columns(["x"])` | refused: not a tag | the tag role goes: `x` is plain data | removed | +| `drop_data_columns(["x"])` | removed | the data role goes: `x` is a plain tag, with a `DataTagMaterializationWarning` | refused: not a data column | +| `drop_columns(["x"])` | removed | removed from both roles | removed | + +The select operators follow the same rule by role: `select_tag_columns` removes the tag +role of every tag it does not name, `select_data_columns` the data role of every data +column it does not name, and `select_columns` keeps exactly the columns it names, in +both roles. (The drop and select refusals are those of a missing name: `strict=False` +ignores the name instead.) + +For a data-tag `x`, in plain words: + +- Dropping or selecting away the **data** of `x` leaves a plain tag `x`, and issues a + `DataTagMaterializationWarning`: the tag no longer refers to a data column, so its + values now live only in the tag (see below to turn the warning off). +- Dropping or selecting away the **tag** `x` leaves plain data `x`, silently. +- Dropping the **column** `x` removes it from both roles, silently. + +```python +print(spikes.drop_tag_columns(["time"]).keys()) +# (('unit',), ('unit', 'time', 'amp')) +print(spikes.drop_data_columns(["time"]).keys()) +# (('unit', 'time'), ('unit', 'amp')) +print(spikes.drop_columns(["time"]).keys()) +# (('unit',), ('unit', 'amp')) +``` + +- System tags and record ids never change. +- Source info belongs to the data role: dropping the tag keeps the column's source-info + token, dropping the data removes it (a plain tag carries none). The row's provenance + is still its system tags (its source id and record id). +- No data element comes from a tag value: dropping the tag leaves a data column that was + data before; dropping the data leaves a tag whose values came from a data column. +- A stream keeps at least one data column: dropping the data role of the last one is + refused. +- A stored table marks exactly the data-tags of what it stores, so a column moved + out of one role is stored unmarked. + +### Adding the tag role: `tag_by` + +A source declares its tags, but a value a later step should join, group or sort by is +often computed on the way: a session label, a trial index, a phase. `tag_by` gives +existing data columns the tag role: each becomes a data-tag, one column that is +a tag and the data column it was, with its values and source info unchanged. + +```python +@op.function_pod(output_keys=["phase"]) +def phase_of(time: int) -> str: + return "early" if time < 15 else "late" + + +phased = phase_of.pod(spikes).tag_by("phase") +print(phased.keys()) +# (('unit', 'time', 'phase'), ('unit', 'time', 'phase')) + +by_phase = phased.group_by(["phase"]) +print(by_phase.keys()) +# (('phase',), ('phase', 'unit', 'time')) +``` + +- `s.tag_by("x", "y")` equals `s.tag_by(["x", "y"])`. The new tags follow the stream's + tags in the order given, and that order is part of the operator (`TagBy`): the names + are a list, never a set. +- Nothing else changes: no row is added, removed or reordered, the data columns keep + their names, types and values, every data column keeps its own source-info token, + and the system tags keep their names and values. +- No data element comes from a tag value: the operator adds a tag role to a column + that is already data, and creates or moves no value. Offering a tag's value as data + stays with `DerivedSource(tags_as_data=...)`. +- A computed column may become a tag, and then key a join or a group, as `phase` does + above. + +`tag_by` refuses, naming the column: a name that is not a data column of the stream; a +name that is already a tag (plain or a data-tag); a reserved name (the `__` prefix, a +system column, a name holding `::`); and a column whose type cannot be a tag. A tag is +one value per row of type `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date`. +A nullable column (`T | None`) is refused: drop the rows with a null first, +`s.drop_nulls("x").tag_by("x")`, which narrows it to `T`. Lists, sets, dicts, structs, +your own classes and logical types (`Path`, `UUID`, arrays) are refused. + +## Operators + +| Operator | Data-tag `x` | +|---|---| +| `TagBy` / `tag_by` | Makes the data column `x` a data-tag (above); everything else is unchanged. | +| `Join` | Plain with plain: as before. Plain with data-tag, or data-tag with data-tag: a data-tag and **one** data column `x`. When `x` is a data-tag on several inputs, its source info is one merged token (below). A key that is `T` in one input and `T \| None` in another is `T` in both roles, as for a plain key. Every other data-name collision is still an error. | +| `GroupBy` | A group key stays scalar in both roles, and its source info is one merged token (below). A non-key loses its tag role, as every non-key tag does, but keeps its data column: a list like every other data column, one source-info token per member. The members are ordered by the tuple of the input's non-key tags (then record ids), the same order for every data list of the group, so all of them stay aligned. | +| `SelectDataColumns`, `DropDataColumns` | Removing the data column removes the data role only: `x` stays a plain tag with the same values, without source info, with a `DataTagMaterializationWarning`. | +| `SelectTagColumns`, `DropTagColumns` | Removing the tag removes the tag role only: `x` stays a data column, with its source info. | +| `SelectColumns`, `DropColumns` | `x` is one column, so naming it acts on both roles: selecting it keeps `x` a data-tag, dropping it (or not selecting it) removes the column from both roles. The roles are never split, so nothing is materialized. | +| `MapTags` / `map_tags` | Renames the column in both roles; `tag_only=True` renames the tag only and leaves a plain tag. | +| `MapData` / `map_data` | Always a data-only move: the data column is renamed and `x` stays a plain tag. | +| Filters, `DropNulls`, `SemiJoin`, `Pick`, `Index` | Keep the data-tag. | +| `AddTags` / `add_tags` | Keeps every data-tag; the new tags, constants, are plain tags. A new tag named like any column, a data-tag included, is refused. | +| `Batch` | Keeps every tag, list-valued, as for plain tags: `x` becomes the same list in both roles and stays a data-tag. Its source info is one token per member, like any data column: nothing is merged. | +| `MergeJoin` | Follows `Join`. A data-tag in both inputs: **one** scalar data-tag column `x`, not a two-element list (the join equated the values), with one merged token as its source info, the same fold as `Join`'s (below). A data-tag in one input, and a plain tag or no column of that name in the other: `x` stays a data-tag with its own source info. A data-only column `x` in the other input is refused. A key that is `T` in one input and `T \| None` in the other is `T` in both roles. Every other colliding data column is merged into a sorted list, as before. | + +```python +units = op.sources.DictSource( + [{"unit": 1, "area": "V1"}, {"unit": 2, "area": "V2"}], + tag_columns=["unit"], + source_id="units", +) +joined = spikes.join(units) +print(joined.keys()) +# (('unit', 'time'), ('unit', 'time', 'area', 'amp')) +``` + +`MergeJoin` merges every other colliding data column into a sorted list, and keeps each +data-tag both inputs share as one column: + +```python +more = op.sources.DictSource( + [{"unit": 1, "time": 10, "amp": 0.75}, {"unit": 2, "time": 20, "amp": 0.125}], + tag_columns=["unit", "time"], + source_id="more_spikes", +) +merged = op.operators.MergeJoin()(spikes, more) +print(merged.keys()) +# (('unit', 'time'), ('unit', 'time', 'amp')) +# A join does not promise a row order: sort the rows before showing them. +for tag, data in sorted(merged.iter_data(), key=lambda row: row[0]["time"]): + print(tag["time"], data["time"], data["amp"]) +# 10 10 [0.5, 0.75] +# 20 20 [0.125, 0.25] +``` + +```python +print(spikes.select_columns(["time", "amp"]).keys()) +# (('time',), ('time', 'amp')) +print(spikes.drop_columns(["time"]).keys()) +# (('unit',), ('unit', 'amp')) +``` + +`Batch` lists every column, a data-tag included, so the tag and its data stay one +column: + +```python +batched = spikes.batch() +print(batched.keys()) +# (('unit', 'time'), ('unit', 'time', 'amp')) +for tag, data in batched.iter_data(): + print(tag["time"], data["time"]) +# [10, 20] [10, 20] +``` + +### After `GroupBy`: rename a non-key column with `map_data` + +`GroupBy` keeps only its key tags, and drops every other tag itself: there is nothing left +to drop with `drop_tag_columns` afterwards (that raises, since the tag is gone). A +data-tag non-key column loses its tag role there, but keeps its data column, a list +like every other data column. So after the grouping it is data only, and is renamed with +`map_data`; `map_tags` refuses it, since it is no longer a tag, and the error says to use +`map_data`. + +```python +by_unit = spikes.group_by(["unit"]) +print(by_unit.keys()) +# (('unit',), ('unit', 'time', 'amp')) + +renamed = by_unit.map_data({"time": "times"}) +print(renamed.keys()) +# (('unit',), ('unit', 'times', 'amp')) +``` + +### Source info of a merged value + +A source-info column belongs to its data column and has its shape: a scalar value has one +token, a list one token per element. When a value is the merge of several equal elements +that are different entries (the key of a `GroupBy`, or the key a `Join` or `MergeJoin` +merges from several inputs with data-tags), no single member's token is true for it. Its source info is +then one **merged token**, the fold of the members' tokens by the operation that merged +them (the SHA-256 of their canonical JSON list, after the operation's label), as the +system tags of a group are folded: + +- `GroupBy`: `group_v1:` over the members' tokens in member order (the order of the + group's data lists, which does not depend on row order); +- `Join` and `MergeJoin`: `join_v1:` over the members' tokens sorted, so it does not depend on the + order of the inputs. + +It is a merged token for a group of one member too, and a merged token can itself be a +member of a later merge. A merged token holds no `::`, so it never reads as a row's +`::::`. No content hash, pipeline hash, record id or +function result identity reads source info, so these tokens move no stored result. + +```python +sessions = op.sources.DictSource( + [{"subject": "m1", "t": 1, "v": 0.5}, {"subject": "m1", "t": 2, "v": 0.7}], + tag_columns=["subject", "t"], + source_id="sessions", +) +grouped = sessions.group_by(["subject"]).as_table(columns={"source": True}) +print(grouped.column("_source_subject").to_pylist()) +# ['group_v1:...'] +print(grouped.column("_source_v").to_pylist()) +# [['sessions::row_0::v', 'sessions::row_1::v']] +``` + +The materialization warning can be turned off; it never changes a result or a hash: + +```python +from orcapod.core import data_tags + +data_tags.data_tag_materialization = "silent" +# ... build the pipeline ... +data_tags.data_tag_materialization = "warn" # the default +``` + +## Stored tables and saved jobs + +A stored table states which of its tags are data-tags. A function node stores each +row's tags in its pipeline table, and a data-tag is stored there once, as a tag +column that carries Arrow field metadata: the key `orcapod:tag_kind` with the value +`data_tag` (`data_tags.TAG_KIND_METADATA_KEY`). A plain tag carries no +marker. The function's result table never records data-tag status, so runs of +the same function with plain tags and with data-tags share their results. The marker moves no hash, and +the stores keep it on every write (first write, append, merge, schema evolution; in +memory and on Delta). A database states whether it keeps +field metadata with `keeps_field_metadata` +(`orcapod.protocols.database_protocols.FieldMetadataKeepingDatabaseProtocol`): the +in-memory, Delta and SpiralDB databases do; the SQLite and PostgreSQL databases keep no +Arrow metadata at all. A node that would store a data-tag in a database that does +not state it (a database without the member included) is refused when its database is +attached, before anything is written. Use a database that keeps field metadata, or keep +those tags plain with `tag_only_columns`. + +```python +from orcapod.core.data_tags import stored_data_tags +from orcapod.databases import InMemoryArrowDatabase +from orcapod.pipeline import PipelineJob + +database = InMemoryArrowDatabase() +job = PipelineJob(database=database) +with job: + scale.pod(spikes, label="scale") +job.run() +pipeline_table = next( + database.get_all_records(path) + for path in database.list_table_paths() + if "_result" not in path +) +print(stored_data_tags(pipeline_table.schema)) +# ('unit', 'time') +``` + +A node reads the binding back from its table, and refuses a table whose marker disagrees +with what the node declares (its pod's fall-through tags, or for a loaded job the saved +file's). + +An operator node that stores its output (`LOG`) marks its table the same way: the +data-tags of the operator's own output carry the marker, written from the +operator's binding, never from a marker that came in from upstream. A `REPLAY` read, and +the stored view `get_all_records()`, check the marker against the node (its operator's +prediction, or for a loaded job the saved output schema, where a data-tag is in +both halves). A database that keeps no field metadata is refused when it is attached in +`LOG` or `REPLAY` mode. Source info stays part of each stored row's record hash; the +marker never enters it. + +`node.as_source()` gives the stored rows as a source with the bindings the stored table +states, in the plain form and with `tags_as_data` (a copy of a data-tag is a new, +plain data column). Nothing is added to the derived source's identity for the binding: its +plain form's content hash is the node's, which carries the binding already, and its +pipeline hash is its output schema, where a data-tag is in both halves. + +```python +derived = job.nodes["scale"].as_source() +print(derived.keys()) +# (('unit', 'time'), ('unit', 'time', 'scaled')) +print(derived.as_table().column_names) +# ['unit', 'time', 'scaled'] +``` + +In a stream the binding is the shared name, so the marker stays with the stored table: +the rows of a replay or of a derived source carry none. + +A saved job or pipeline (format `0.3.0`) states every binding: each source config writes +`tag_only_columns` (`True` when every tag is plain), and each function node writes +`fall_through_tags` (empty for a plain pod). The format version, never a missing +key, decides what a file means: a file at `0.1.0` or `0.2.0` holds only plain tags, and a +`0.3.0` file without these keys is refused. A loaded job keeps its hashes and bindings, and +the file is checked against the stored tables when a node reads them. + +## Not supported yet + +`MapColumns` (`map_columns`) raises "not supported with data-tags yet" when a +data-tag reaches it; rename with `map_tags` and `map_data`. +`PollingSource` keeps its tags plain. Each error names its issue and the way out: build the +source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not +data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer +take it from that column: as in v0.3, the value must also be a data column of its own, for +example a copy in the source's data under another name, or +`DerivedSource(origin, tags_as_data={...})`. + +## Errors that name a data-tag + +- `pod.process(stream)` and `pod(stream, strict=True)` are exact: they hand every data + column to the function, so a data-tag the function does not read is refused like + any extra column. The error names the data-tags and the two ways out: call the + pod, `pod(stream)`, which for a plain `FunctionPod` lets them pass through as + data-tags (a wrapper pod such as `CachedFunctionPod` has no fall-through: its + call keeps them as plain tags), or keep them plain with `tag_only_columns`. It offers + only a way out that can work: when no data column of the input is a parameter, a + wrapper pod's call and `tag_only_columns` would leave the function nothing to read, + and the error says so instead. +- Binding a source to a job's input slot is strict: a column that is a data-tag on + one side and a plain tag or a data column on the other is refused (it is listed as a + missing or unexpected column). The error says which side holds it as a data-tag, + and how to make the two agree, in either direction. A data-tag of the source for a + plain slot tag: keep it plain with `tag_only_columns`, or declare the slot with the + column in both roles. A plain source tag (or a data-only source column) for a + data-tag of the slot: make it a declared tag that `tag_only_columns` does not name, or + declare the slot with it as the source has it (a plain tag, or a data column only). `tag_only_columns` lists every plain tag of the source, so the error + names the exact list. diff --git a/docs/concepts/file-types.md b/docs/concepts/file-types.md index a7df34280..a69c97191 100644 --- a/docs/concepts/file-types.md +++ b/docs/concepts/file-types.md @@ -85,7 +85,7 @@ files = DictSource( {"name": "a", "recording": op.File(recordings / "a.txt")}, {"name": "b", "recording": op.File(recordings / "b.txt")}, ], - tag_columns=["name"], + tag_columns=["name"], tag_only_columns=True, source_id="recordings", ) print(files.output_schema()[1]) diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index ebbf1b8ce..f43ec43ae 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -68,7 +68,7 @@ source = DictSource( {"subject_id": "mouse_01", "weight": 25.3, "height": 0.12}, {"subject_id": "mouse_02", "weight": 22.1, "height": 0.10}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) # Apply the function pod to the source stream @@ -88,7 +88,9 @@ for tag, data in result.iter_data(): # {'subject_id': 'mouse_02'} -> {'bmi': 2209.9999999999995} ``` -The function pod preserves tags and replaces data columns with the function's output. +The function pod preserves tags and replaces data columns with the function's output. A +data-tag is the exception: called as `pod(stream)`, the pod lets it fall through, so it stays +both a tag and a data column of the output (see [Data-tags](data-tags.md)). !!! tip Calling a function pod, `compute_bmi.pod(source)`, is the convenient form of @@ -108,7 +110,7 @@ wide = DictSource( {"subject_id": "mouse_01", "weight": 25.3, "height": 0.12, "age": 12}, {"subject_id": "mouse_02", "weight": 22.1, "height": 0.10, "age": 8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) result = compute_bmi.pod(wide) # 'age' is not a parameter: it is dropped @@ -200,7 +202,7 @@ from orcapod.errors import OutputTypeError def halve(weight: float) -> int: return weight / 2 # a float, although the function says it returns an int -weights = DictSource(data=[{"subject_id": "mouse_01", "weight": 25.3}], tag_columns=["subject_id"]) +weights = DictSource(data=[{"subject_id": "mouse_01", "weight": 25.3}], tag_columns=["subject_id"], tag_only_columns=True) try: list(halve.pod(weights).iter_data()) @@ -322,7 +324,7 @@ db = InMemoryArrowDatabase() job = PipelineJob(name="bmi_demo", database=db) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") job.run() @@ -371,7 +373,7 @@ mice.append({"subject_id": "mouse_03", "weight": 27.8, "height": 0.11}) job = PipelineJob(name="bmi_demo", database=db) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") job.run() @@ -437,7 +439,7 @@ from orcapod.pipeline import AsyncPipelineOrchestrator job = PipelineJob(name="bmi_async", database=InMemoryArrowDatabase()) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") job.run(orchestrator=AsyncPipelineOrchestrator()) @@ -476,7 +478,7 @@ import asyncio job = PipelineJob(name="bmi_awaited", database=InMemoryArrowDatabase()) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") @@ -501,7 +503,7 @@ Running the same job again computes nothing new, and the summary says so: ```python job = PipelineJob(name="bmi_demo", database=db) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") summary = job.run() @@ -523,7 +525,7 @@ def careless_bmi(weight: float, height: float) -> float: job = PipelineJob(name="bmi_typo", database=InMemoryArrowDatabase()) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") careless_bmi.pod(source, label="bmi") summary = job.run() @@ -566,7 +568,7 @@ more_mice = mice + [{"subject_id": "mouse_04", "weight": 30.0, "height": 0.12}] load_job = PipelineJob(name="bmi_demo", database=db) with load_job: - source = DictSource(data=more_mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=more_mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") print(load_job.nodes["bmi"].as_table().num_rows) # not run in this process: empty @@ -666,7 +668,7 @@ from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator for orchestrator in (SyncPipelineOrchestrator, AsyncPipelineOrchestrator): job = PipelineJob(name="bmi_typo_fast", database=InMemoryArrowDatabase()) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") careless_bmi.pod(source, label="bmi") try: job.run(orchestrator=orchestrator(error_policy="fail_fast")) @@ -854,7 +856,7 @@ def notify(ctx, weight: float, height: float) -> None: notify_job = PipelineJob(name="notify_demo", database=InMemoryArrowDatabase()) with notify_job: - mice_source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + mice_source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") SideEffectPod(notify)(mice_source, label="notify") notify_job.run() @@ -889,8 +891,11 @@ source was configured. `PipelineJob.load(path)` reads it back, for example in a session, so that you can read the results the job stored. The file also records `orcapod_version`, the Orcapod release that wrote it (the same text as `orcapod.__version__`; a development checkout gives a development version). It is informational: loading never -depends on it. Adding the field moved the file format version from `0.1.0` to `0.2.0`; a -`0.1.0` file, written by v0.2.0, still loads as before, and any other version is rejected. +depends on it. Adding the field moved the file format version from `0.1.0` to `0.2.0`, and +stating every tag binding (data-tags) moved it to `0.3.0`, the version written now. A +`0.1.0` file, written by v0.2.0, and a `0.2.0` file, written by v0.3.0, still load, with every +source tag plain as they were written, within the schema limits described below (a file with +name-only extension type text cannot be loaded); any other version is rejected. The reverse does not hold: v0.2.0 cannot load a file saved by v0.3.0, because hashes differ between the two releases. @@ -906,7 +911,7 @@ workdir = Path(tempfile.mkdtemp()) # any directory you keep job = PipelineJob(name="bmi_demo", database=DeltaTableDatabase(workdir / "store")) with job: - source = DictSource(data=mice, tag_columns=["subject_id"], source_id="mice") + source = DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") compute_bmi.pod(source, label="bmi") job.run() @@ -932,7 +937,7 @@ the results stored for those inputs. Nodes below a source that is not available `node.run(load_only=True)` does the same for one node and what it depends on. ```python -loaded.bind(sources={"mice": DictSource(data=mice, tag_columns=["subject_id"], source_id="mice")}) +loaded.bind(sources={"mice": DictSource(data=mice, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice")}) summary = loaded.run(load_only=True) print(loaded.nodes["bmi"].as_table().num_rows, summary.nodes["bmi"].not_stored) # 3 0 @@ -1123,7 +1128,7 @@ def count_rows(readings: pl.DataFrame) -> int: recordings = DictSource( data=[{"session": "s1", "n_samples": 2}, {"session": "s2", "n_samples": 3}], - tag_columns=["session"], + tag_columns=["session"], tag_only_columns=True, ) readings = load_readings.pod(recordings) for tag, data in readings.iter_data(): @@ -1160,7 +1165,7 @@ ITL-1144. try: DictSource( data=[{"sample": "s1", "labels": [{1, 2}, None]}], - tag_columns=["sample"], + tag_columns=["sample"], tag_only_columns=True, data_schema={"sample": str, "labels": list[set[int] | None]}, ) except ValueError as error: @@ -1169,7 +1174,7 @@ except ValueError as error: labels = DictSource( data=[{"sample": "s1", "counts": [3, None, 1]}], - tag_columns=["sample"], + tag_columns=["sample"], tag_only_columns=True, data_schema={"sample": str, "counts": list[int | None]}, ) for tag, data in labels.iter_data(): @@ -1202,14 +1207,14 @@ weights = DictSource( {"subject_id": "mouse_01", "weight": 25.3}, {"subject_id": "mouse_02", "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) heights = DictSource( data=[ {"subject_id": "mouse_01", "height": 0.12}, {"subject_id": "mouse_02", "height": 0.10}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) result = compute_bmi.pod(weights, heights) @@ -1233,7 +1238,7 @@ Two consequences: ```python chosen = DictSource( data=[{"subject_id": "mouse_02", "cohort": "b"}], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) result = compute_bmi.pod(weights, heights, chosen) diff --git a/docs/concepts/operators.md b/docs/concepts/operators.md index b04fc7a5b..096e60609 100644 --- a/docs/concepts/operators.md +++ b/docs/concepts/operators.md @@ -31,7 +31,9 @@ content. One rule holds for every operator: **no data element comes from a tag value**. An operator may use tags to join, filter, group or order rows, but it never turns a tag column into a data -column, so a function pod never receives tag content. +column, so a function pod never receives the content of a plain tag. A +[data-tag](data-tags.md) is different: it is a tag and a data column at once, so a function +reads its data role. An operator may add a tag, never data. A tag labels rows; it is not a data value, so [`AddTags`](#addtags) can add tags with constant values while every data column passes through @@ -85,7 +87,7 @@ and `SemiJoin`. In the output of `Join` and `MergeJoin` the column is `T` when a input's type. Any other type difference is still an error, and a data column is never reconciled this way. -A name is a tag or a data column, never both: if one input has `x` as a tag column and another has it as a data column, `Join` (and `MergeJoin`) raises `InputValidationError` when the graph is built; rename one with `map_tags` or `map_data`. +A plain tag and a data column never share a name across inputs: if one input has `x` as a plain tag column and another has it as a data column only, `Join` (and `MergeJoin`) raises `InputValidationError` when the graph is built; rename one with `map_tags` or `map_data`. A data-tag is the exception: it is one column in both roles, and `Join` merges a key that is a data-tag on its inputs (or plain on one and a data-tag on another) into a data-tag with one data column (see [Data-tags](data-tags.md)). Column order follows each operator's argument symmetry: commutative operators (`Join`, `MergeJoin`) list tag and data columns by first appearance across their inputs in canonical @@ -102,7 +104,7 @@ subjects = DictSource( {"subject_id": "mouse_01", "age": 12}, {"subject_id": "mouse_02", "age": 8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) measurements = DictSource( @@ -110,7 +112,7 @@ measurements = DictSource( {"subject_id": "mouse_01", "weight": 25.3}, {"subject_id": "mouse_02", "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) join = Join() @@ -136,14 +138,14 @@ scale_a = DictSource( {"subject_id": "mouse_01", "weight": 25.3}, {"subject_id": "mouse_02", "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) scale_b = DictSource( data=[ {"subject_id": "mouse_01", "weight": 24.9}, {"subject_id": "mouse_02", "weight": 22.6}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) merged = MergeJoin()(scale_a, scale_b) @@ -184,14 +186,14 @@ all_subjects = DictSource( {"subject_id": "mouse_02", "age": 8}, {"subject_id": "mouse_03", "age": 15}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) weighed = DictSource( data=[ {"subject_id": "mouse_01", "weight": 25.3}, {"subject_id": "mouse_02", "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) only_weighed = SemiJoin()(all_subjects, weighed) @@ -215,7 +217,7 @@ source = DictSource( {"subject_id": "mouse_01", "age": 12}, {"subject_id": "mouse_02", "age": 8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) batch = Batch() @@ -255,7 +257,7 @@ source = DictSource( {"subject_id": "mouse_01", "probe": "imec1", "spike_count": 980}, {"subject_id": "mouse_02", "probe": "imec0", "spike_count": 1450}, ], - tag_columns=["subject_id", "probe"], + tag_columns=["subject_id", "probe"], tag_only_columns=True, ) grouped = GroupBy(by=["subject_id"])(source) @@ -272,14 +274,17 @@ The output schema follows three rules: - **Group keys stay scalar tag columns** (`subject_id: str`), in the input's tag order. They are the only tag columns of the output. - **A tag that is not a key is dropped** (`probe`). A list of the members' values is not a key - for the group, just as SQL drops a non-key, non-aggregated column after `GROUP BY`. + for the group, just as SQL drops a non-key, non-aggregated column after `GROUP BY`. A + [data-tag](data-tags.md) that is not a key loses its tag role only: its data + column stays, as a list like any other. - **Data columns become `list[T]`** (`spike_count: list[int]`), one element per member. !!! warning "Member tag values are not carried" - `GroupBy` does not keep the members' values of a non-key tag, as a tag or as data. If a - later step needs to know which member each element came from, make the column data - *before* grouping: declare it as data at the source, or derive a source that offers the tag - as data with `DerivedSource(origin, tags_as_data={...})`. + `GroupBy` does not keep the members' values of a plain non-key tag, as a tag or as data. + If a later step needs to know which member each element came from, make the column data + *before* grouping: keep the tag a data-tag at the source (the default without + `tag_only_columns`), declare it as data at the source, or derive a source that offers the + tag as data with `DerivedSource(origin, tags_as_data={...})`. ```python from orcapod.sources import DerivedSource @@ -296,7 +301,8 @@ for tag, data in by_subject.iter_data(): The data columns of the output are exactly the data columns of the input: a function pod that follows `GroupBy` receives `spike_count` (and `probe_name`, when the source offers it), never a -tag. +plain tag. A data-tag group key stays a data column too (scalar, in both roles), so a +following function can read its value (see [Data-tags](data-tags.md)). Ordering is deterministic by construction, because orcapod hashes the emitted lists to build cache keys: members are sorted by the input's non-key tag values (with the internal record id @@ -338,7 +344,7 @@ source = DictSource( {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, {"subject_id": "mouse_03", "age": 15, "weight": 27.8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) select = SelectDataColumns(columns=["weight"]) @@ -366,7 +372,7 @@ sessions = DictSource( {"subject_id": "mouse_01", "session": 1, "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "session": 2, "age": 8, "weight": 22.1}, ], - tag_columns=["subject_id", "session"], + tag_columns=["subject_id", "session"], tag_only_columns=True, ) dropped = sessions.drop_columns("session", "age") @@ -530,7 +536,7 @@ recordings = ArrowTableSource( ] ), ), - tag_columns=["session"], + tag_columns=["session"], tag_only_columns=True, ) print(recordings.output_schema()[1]) # Schema({'start_ms': int | None}) @@ -606,7 +612,7 @@ visits = ArrowTableSource( "notes": pa.array([None, "noisy", None, "ok"], pa.large_string()), } ), - tag_columns=["session"], + tag_columns=["session"], tag_only_columns=True, ) print(visits.output_schema()) # (Schema({'session': str | None}), Schema({'start_ms': int | None, 'notes': str | None})) @@ -684,6 +690,9 @@ except InputValidationError as error: The names and values, in order, are the operator's identity, so two different values are two operators, with different hashes and stored tables. +On a stream with [data-tags](data-tags.md), the data-tags stay data-tags and the new tags are +plain: a constant tag is never a data value. + ### Pick and Index `Pick(column, key)` extracts one entry from a struct- or dict-typed data column, and @@ -755,7 +764,7 @@ def counts(values): "count": pa.array(values, type=pa.int8()), } ) - return ArrowTableSource(table, tag_columns=["subject_id"]) + return ArrowTableSource(table, tag_columns=["subject_id"], tag_only_columns=True) merged = MergeJoin()(counts([1, 2]), counts([3, 4])) print(merged.as_table().schema.field("count").type) diff --git a/docs/concepts/pipeline-job.md b/docs/concepts/pipeline-job.md index 72087d8ab..48afe16f7 100644 --- a/docs/concepts/pipeline-job.md +++ b/docs/concepts/pipeline-job.md @@ -27,7 +27,7 @@ def compute_bmi(weight: float, height: float) -> float: def mice(rows): - return DictSource(data=rows, tag_columns=["subject_id"], source_id="mice") + return DictSource(data=rows, tag_columns=["subject_id"], tag_only_columns=True, source_id="mice") cohort_a = [ diff --git a/docs/concepts/sources.md b/docs/concepts/sources.md index e31961138..ce9c1b0ca 100644 --- a/docs/concepts/sources.md +++ b/docs/concepts/sources.md @@ -44,21 +44,26 @@ Two construction rules apply to every source and are the most common first error - Every name in `tag_columns` must be a column of the data, otherwise the source raises `ValueError: tag_columns not found in table`. -- At least one column must be left over as a data column, otherwise it raises - `ValueError: No data columns found in the table`. - -**Plain tags and `tag_only_columns`.** The sources built from a table (`ArrowTableSource`, -`DictSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource` and the database sources) also -take the keyword-only option `tag_only_columns`. In v0.3 every tag column is plain: it is a tag only, -never also a data column. `tag_only_columns=True` says so explicitly; it is the default and -the only value v0.3 accepts, and it changes no hash, source id, record id or saved config. Any -other value is refused. In v0.4 tags become referential by default (a tag that is also the -data column of the same name), and `tag_only_columns` keeps a tag plain, so code that must -keep today's behaviour passes `tag_only_columns=True` now. +- At least one data column must remain, otherwise the source raises + `ValueError: No data columns found in the table`. A data-tag is also a data column, so + this happens only when every tag is kept plain (`tag_only_columns`) and no other column + is left. + +**Data-tags and plain tags: `tag_only_columns`.** The sources built from a table +(`ArrowTableSource`, `DictSource`, `DataFrameSource`, `CSVSource`, `DeltaTableSource` and the +database sources) take the keyword-only option `tag_only_columns`. By default every declared +tag is a data-tag: it is also the data column of the same name (see +[Data-tags](data-tags.md)). `tag_only_columns` names the tags kept plain, a tag +only and never a data column, as every tag was in v0.3; `tag_only_columns=True` keeps all of +them plain. A source with plain tags and one with data-tags over one table have the same source id and record +ids, but different schemas and so different hashes. ```python from orcapod.sources import DictSource +with_data_tags = DictSource(data=[{"k": "a", "x": 1}], tag_columns=["k"]) +print(with_data_tags.keys()) +# (('k',), ('k', 'x')) plain = DictSource( data=[{"k": "a", "x": 1}], tag_columns=["k"], tag_only_columns=True ) @@ -374,10 +379,13 @@ the system tags of every joined source, and binding refuses it with `SourceSpecM (support is planned, ITL-1127). **Using a tag value as an input to a computation.** Tags identify a row and data is what a -function computes on, so a function pod never receives a tag. Sometimes one value is needed -for both: a session's start time identifies the session, and it is also what you subtract to -get an elapsed time. Pass `tags_as_data` to get a source that keeps the tag and also offers -its value as a data column under a name you choose: +function computes on. A data-tag (the default for a declared tag) is both: a function +reads it through its data role, under the same name (see +[Data-tags](data-tags.md)). A plain tag is a tag only, and a function pod +never receives it. When a plain tag's value is also needed as data (a session's start time +identifies the session, and it is also what you subtract to get an elapsed time), or you want +the value as a data column under another name, pass `tags_as_data` to get a source that keeps +the tag and also offers its value as a data column under a name you choose: ```python from datetime import datetime, timezone @@ -397,7 +405,7 @@ experiments = DictSource( {"subject_id": "m1", "session_start": start_2, "experiment_index": 0, "experiment_start": datetime(2026, 1, 6, 9, 32, tzinfo=timezone.utc)}, ], - tag_columns=["subject_id", "session_start", "experiment_index"], + tag_columns=["subject_id", "session_start", "experiment_index"], tag_only_columns=True, source_id="experiments", ) @@ -570,7 +578,7 @@ from orcapod.sources import DictSource source = DictSource( data=[{"subject_id": "mouse_01", "age": 12, "weight": 25.3}], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, source_id="mice", ) @@ -608,7 +616,7 @@ source = DictSource( {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, {"subject_id": "mouse_03", "age": 15, "weight": 27.8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) # Inspect the schema diff --git a/docs/concepts/streams.md b/docs/concepts/streams.md index 74891b27a..cbbfa1840 100644 --- a/docs/concepts/streams.md +++ b/docs/concepts/streams.md @@ -52,7 +52,7 @@ stream = DictSource( {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) ``` @@ -129,7 +129,7 @@ subjects = DictSource( {"subject_id": f"mouse_{i:02d}", "age": 8 + i, "weight": 20.0 + i} for i in range(1, 26) ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, label="subjects", ) @@ -315,7 +315,7 @@ source = DictSource( {"subject_id": "mouse_01", "age": 12, "weight": 25.3}, {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) # Default: user-facing columns only @@ -347,7 +347,7 @@ source = DictSource( {"subject_id": "mouse_02", "age": 8, "weight": 22.1}, {"subject_id": "mouse_03", "age": 15, "weight": 27.8}, ], - tag_columns=["subject_id"], + tag_columns=["subject_id"], tag_only_columns=True, ) # Schema inspection diff --git a/docs/dev/design_issues/ITL-1017-RT2.md b/docs/dev/design_issues/ITL-1017-RT2.md new file mode 100644 index 000000000..895422723 --- /dev/null +++ b/docs/dev/design_issues/ITL-1017-RT2.md @@ -0,0 +1,25 @@ +## `src/orcapod/core/data_tags.py` — data-tags workshop slice (ITL-1023) + +### RT2 — A nested merged source-info token enters the next merge with its label +**Status:** resolved +**Severity:** low +**Issue:** ITL-1017 + +A value merged from several equal elements carries one source-info token, at first +`merged:source_fold_v1:` (`arrow_utils.fold_source_info_tokens`: SHA-256 of the canonical JSON +array of the sorted member tokens). A member can itself be a merged token (grouping a grouped stream, +a join after a group-by), and it enters the preimage as it is, label included. The workshop slice +accepts this on the precedent of source ids: a token is an identifier, and default source ids built +from a hash (`derived: