Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
fc22de6
test(referential_tags): failing EDI-shaped end-to-end test (ITL-1023)
Nov 13, 2026
02b6235
feat(referential_tags): WIP representation, sources, select/drop, Joi…
Nov 13, 2026
c3c5e99
feat(referential_tags): row-preserving ops, renames, guards, async Jo…
Nov 13, 2026
c42eece
feat(referential_tags): renames per developer decision; EDI test on t…
Nov 13, 2026
880dcd4
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
853da82
test(referential_tags): existing tests pin plain tags where they asse…
Nov 13, 2026
1ec232e
feat(pipeline): saved job and pipeline format 0.3.0; v0.3 files load …
Nov 13, 2026
6120671
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
860045e
test(referential_tags): pin plain tags in main's torch golden case an…
Nov 13, 2026
6a6d72b
feat(function_pod): referential tags fall through a pod call (ITL-1023)
Nov 13, 2026
ed51400
fix(referential_tags): fall-through keeps exact types and returned ob…
Nov 13, 2026
5832772
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
3e12d62
fix(referential_tags): type-check clean; fall-through pod refuses to …
Nov 13, 2026
d8628af
test(pipeline): commit the saved-job fixture's CSV and Parquet files …
Nov 13, 2026
1fd2daa
test(objective): pin plain tags where the objective tests assume v0.3…
Nov 13, 2026
34fb2a0
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4473e2e
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
4c0c687
docs(plan): ITL-1023 status log
Nov 13, 2026
5607af1
docs(plan): ITL-1017 merged source-info token plan
Nov 13, 2026
5e8028e
fix(referential_tags): one merged source-info token for a value merge…
Nov 13, 2026
aaff7da
docs(referential_tags): merged source-info token, changelog, open poi…
Nov 13, 2026
8d7e3c9
Merge remote-tracking branch 'origin/main' into eywalker/itl-1023-wor…
Nov 13, 2026
c78ba80
docs(plan): ITL-1017 status log
Nov 13, 2026
8c7cfb6
Merge origin/main into the referential tags slice (ITL-1023, ITL-1017)
Nov 14, 2026
0bfd72a
Merge origin/main (ITL-1074) into the referential tags slice (ITL-1023)
Nov 14, 2026
634ad40
fix(group_by): a referential non-key tag keeps its data column as a l…
Nov 14, 2026
99b5737
Merge origin/main (4dd015a4) into the referential tags slice (ITL-1023)
Nov 16, 2026
404066e
fix(referential_tags): main's new paths meet referential tags (ITL-1023)
Nov 16, 2026
177d5ad
fix(referential_tags): GroupBy key order, meta stripping and join err…
Nov 16, 2026
61129ae
fix(referential_tags): zero-row heads keep references; pin main's new…
Nov 16, 2026
f359aa0
docs: main's new pages pin plain tags on the referential tags slice (…
Nov 16, 2026
3e0ae5a
feat(join): a referential key that differs only in nullability joins …
Nov 16, 2026
22e7662
test(referential_tags): type-clean key nullability assertion (ITL-1023)
Nov 16, 2026
157632b
test: pin plain tags in main's remaining v0.3-shape suites (ITL-1023)
Nov 16, 2026
84d1312
feat(referential_tags): stored tables and saved jobs keep tag binding…
Nov 16, 2026
199221c
feat(referential_tags): refuse a database that cannot keep the marker…
Nov 16, 2026
8f7d50a
ci(coderabbit): review pull requests that target any base branch, on …
eywalker Nov 16, 2026
7d6c580
chore: merge the slice branch (.coderabbit.yaml, PR #405) into ITL-1019
Nov 16, 2026
fc1db0a
fix(pipeline): a saved-node error of a loaded job names the job file …
Nov 16, 2026
00818bd
fix(pipeline): refuse a saved source config whose bindings contradict…
Nov 16, 2026
ec5984a
fix(pipeline): check an explicitly empty tag_columns against the slot…
Nov 16, 2026
26d981a
feat(referential_tags): operator nodes store and derived sources pres…
Nov 16, 2026
cb3dbc4
refactor(operator_node): the stored-table check returns nothing; one …
Nov 16, 2026
3c51256
test(referential_tags): reach the DerivedSource binding check; log IT…
Nov 16, 2026
cecbaf3
feat(databases): a database states keeps_field_metadata; an undeclare…
Nov 16, 2026
25ff7db
feat(batch): Batch on referential tags (ITL-1017) — into the referent…
eywalker Nov 16, 2026
67435e0
feat(referential_tags): SelectColumns/DropColumns on referential tags…
eywalker Nov 16, 2026
57aaa5c
Merge origin/main (37716c76, ITL-1154) into the referential tags slic…
Nov 16, 2026
c111ca4
feat(referential_tags): stored tables and saved jobs keep tag binding…
eywalker Nov 16, 2026
69776b0
chore: merge the slice branch (PR #402 squash-merged) into ITL-1019 W3
Nov 16, 2026
40c57da
Merge the slice head c111ca47 (#402, ITL-1019) into the ITL-1154 refr…
Nov 16, 2026
7137e72
fix(databases): a SpiralDB retry publishes metadata whose write faile…
Nov 17, 2026
1e9565a
fix(databases): a SpiralDB write that stores no row never replaces st…
Nov 17, 2026
5261e9a
test(referential_tags): type-clean field metadata in the SpiralDB sto…
Nov 17, 2026
ded5305
fix(databases): SpiralDB writes a table's Arrow metadata before its r…
Nov 17, 2026
f030aab
fix(databases): a failed first SpiralDB write removes the metadata it…
Nov 17, 2026
edc98fd
Merge origin/main (62899bda: #408, #409, #410, #411) into the referen…
Nov 17, 2026
4b49640
fix(databases): a failed SpiralDB write puts the metadata entry back …
Nov 17, 2026
a6cf552
fix(databases): read the SpiralDB metadata snapshot before the nullab…
Nov 17, 2026
ac68895
Merge origin/main (7c735b7a, v0.3.0 release commit) into the referent…
Nov 17, 2026
01f3db4
docs(changelog): concurrent SpiralDB writers are not coordinated; fol…
Nov 17, 2026
6274a3c
chore: merge the refreshed slice (main 7c735b7a, v0.3.0) into ITL-101…
Nov 17, 2026
a7f08ed
test(referential_tags): meet main's SpiralDB column refusal and hash-…
Nov 17, 2026
ce32310
fix(operator_node): replayed rows carry the node's data context in th…
Nov 17, 2026
8b73ba7
feat(operators): TagBy / stream.tag_by makes data columns referential…
Nov 17, 2026
6aa2314
docs(referential-tags): describe the three moves by role after tag_by…
Nov 17, 2026
e037e58
Merge pull request #407 from metamorphic-labs/eywalker/itl-1019-w3-de…
eywalker Nov 17, 2026
e91d256
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
43bb0ef
Merge origin/main (23fd8f2a: #414, #415, #416) into the referential t…
Nov 17, 2026
eb32ab1
test(tag_by): a logged TagBy operator node stores its new tag marked …
Nov 17, 2026
9c48452
chore(changelog): fold the slice's fragments into changelog.d/ITL-102…
Nov 17, 2026
a6b4a17
docs(referential-tags): select and drop act on a referential tag by r…
Nov 17, 2026
3ab814e
refactor(referential_tags): remove ReferenceMaterializationWarning (I…
Nov 17, 2026
194254e
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
8cd398f
fix(tag_by): keep the input's column layout and data context (ITL-1175)
Nov 17, 2026
424f731
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
5292a60
revert(referential_tags): keep ReferenceMaterializationWarning (ITL-1…
Nov 17, 2026
1e585a7
Merge origin/main (565201b2: #417) into the referential tags slice (I…
Nov 17, 2026
27fee5a
feat(merge_join): handle referential tags as Join does (ITL-1194)
Nov 17, 2026
83ff606
fix(tag_by): keep each row's data context key; correct the spec on LO…
Nov 17, 2026
35579f8
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
01c3b72
docs(design_issues): log ST5, the two tag orders of ArrowTableStream …
Nov 17, 2026
447597b
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
50b580b
fix(referential_tags): address the CodeRabbit review of the slice (IT…
Nov 17, 2026
016052a
docs(referential_tags): log the deferred review findings (ITL-1199 to…
Nov 17, 2026
fc205da
docs(referential-tags): sort the MergeJoin sample's rows before print…
Nov 17, 2026
5dd0634
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
378eca2
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
d7d1b34
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
38478e0
Merge origin/main (232ee360: #418) into the referential tags slice (I…
Nov 17, 2026
05deee1
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
a373ca6
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
09ffdf7
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
0937242
Merge pull request #423 from metamorphic-labs/eywalker/itl-1194-merge…
eywalker Nov 17, 2026
460ff01
chore(changelog): fold the ITL-1194 fragment (#423) into changelog.d/…
Nov 17, 2026
816d8c8
Merge remote-tracking branch 'origin/eywalker/itl-1023-workshop-slice…
Nov 17, 2026
bd8f020
Merge branch 'eywalker/itl-1175-operator-and-stream-method-to-mark' i…
Nov 17, 2026
e43c609
docs(design_issues): log AB1, the async barrier input's column order …
Nov 17, 2026
c4cee17
Merge pull request #421 from metamorphic-labs/eywalker/itl-1195-opera…
eywalker Nov 17, 2026
b77cb01
refactor(data_tags): rename "referential tag" to "data-tag" (ITL-1023)
Nov 17, 2026
b011809
Merge origin/main (51be5e4e: #422 add_tags, #419) into the data-tags …
Nov 17, 2026
c2aafc4
fix(data_tags): data-tags come first among the data columns (ITL-1205)
Nov 17, 2026
20309a0
feat(add_tags): AddTags works on a stream with data-tags (ITL-1196, I…
Nov 17, 2026
0248fc0
fix(data_tags): async rows keep data-tags first; fall-through prevali…
Nov 17, 2026
8d6a1d9
docs(data_tags): answer the CodeRabbit doc findings on PR #342 (ITL-1…
Nov 17, 2026
3817334
Merge origin/main (7c8c5efb: #425 Starlight docs site) into the data-…
Nov 17, 2026
66d6ed7
fix(side_effects): pass a data-tag only to a parameter that takes a k…
Nov 17, 2026
99487ce
fix(data_tags): refuse an EmptyData hit of a fall-through node; selec…
Nov 17, 2026
ce427c4
Merge origin/main (ad3f2411: #424 pipeline diagram, #430 Graphviz rem…
Nov 17, 2026
b702ce4
fix(pick, index): only a data-tag is refused in place; a plain tag ge…
Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -949,6 +949,14 @@ data. The rules:
path (_kept_output: a job run, sync or async, and node.run() on its own; ITL-1133),
and reads a stored operator cache in row order (_in_row_order_on_read). Never change
the order of any other populated table: stored tables depend on it.
- Data-tags come first among the data columns (developer, November 17, 2026, ITL-1205):
a stream lists its data-tags first in its data keys and data schema, in tag order, then
its other data columns in table order. A table built from rows lays a data-tag out among
the tags, so this is every row's order, on every path. ArrowTableStream applies it to
its keys (its table keeps its column order) and every operator's prediction applies it
through data_tags.data_tags_first. No hash of a stream, schema, row or table depends on
the order of a stream's columns, so the order moves none; a library value's own hash (a
polars or pandas DataFrame in a cell) does include that value's column order.
- Type strictness (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -991,6 +991,14 @@ data. The rules:
path (`_kept_output`: a job run, sync or async, and `node.run()` on its own; ITL-1133),
and reads a stored operator cache in row order (`_in_row_order_on_read`). Never change
the order of any other populated table: stored tables depend on it.
- **Data-tags come first among the data columns** (developer, November 17, 2026, ITL-1205):
a stream lists its data-tags first in its data keys and data schema, in tag order, then
its other data columns in table order. A table built from rows lays a data-tag out among
the tags, so this is every row's order, on every path. `ArrowTableStream` applies it to
its keys (its table keeps its column order) and every operator's prediction applies it
through `data_tags.data_tags_first`. No hash of a stream, schema, row or table depends on
the order of a stream's columns, so the order moves none; a library value's own hash (a
polars or pandas DataFrame in a cell) does include that value's column order.
- **Type strictness** (developer, November 10, 2026): Orcapod is type strict. It never
expands or converts a type implicitly; the only exception is normalization to large variants.
Inputs that disagree on a column's type are an error naming the column and both types, on
Expand Down
223 changes: 223 additions & 0 deletions changelog.d/ITL-1023.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
### Breaking changes and migration

- A source's declared tag columns are now data-tags by default (ITL-1023, specification
"Orcapod v0.4 — Referential Tags"): each is also a data column of the same name, stored
once, so a function can read it and joins and groups use it as a tag. A script run
unchanged gets new schemas and hashes for such sources. Pass `tag_only_columns=[...]` (or
`tag_only_columns=True`) to keep tags plain, exactly as before; every existing golden hash
is unchanged under that option.
- The term is "data-tag" (developer decision, November 17, 2026: the three column roles
are tag, data and data-tag; on the integration branch it was "referential tag"). Every
name follows it: `ArrowTableStream(data_tags=...)` and `.data_tags`, `stream_data_tags`,
`FallThroughPod` (`.fall_through_tags`), `DataTagMaterializationWarning` and
`data_tags.data_tag_materialization`, the module `orcapod.core.data_tags`. The stored
marker is `orcapod:tag_kind` = `data_tag`, and a saved function node writes
`fall_through_tags`. Stores and saved jobs written by a development build of the
integration branch (marker value `referential`, key `fall_through_references`) are
refused on read and on load, not migrated: clear such stores and save such jobs
again. The identity of a raw `ArrowTableStream` with data-tags and of every
fall-through pod moved with the rename (provisional identities, see Known limitations),
and so did the stored path of a fall-through node and of every node downstream of it.

### Added

- `tag_only_columns` on `DictSource`, `ArrowTableSource`, `CSVSource`, `DataFrameSource`,
`DeltaTableSource`, the database sources and `SpiralDBTableSource`.
- `Join` merges the data columns of a data-tag both inputs join on into one column;
plain with data-tag gives a data-tag. `GroupBy` keeps a data-tag key scalar
in both roles. A data-tag non-key loses its tag role, as every non-key tag does
(ITL-1070), but keeps its data column as a list with one element and one source-info
token per member; the members are ordered by the tuple of the input's non-key tags,
the same order for every data list of the group.
- A source-info column has the shape of its data column. A value merged from several equal
elements (a data-tag `GroupBy` key; a `Join` key that is a data-tag on several inputs) has
one merged token instead of a list of every member's token (ITL-1017): the merging
operation's fold (`arrow_utils.fold_system_tag_values`), `group_v1:<hex>` over the
members' tokens in member order for a `GroupBy` key, `join_v1:<hex>` over the members'
tokens sorted for a `Join` key. No hash or stored id changes.
- `MapTags` renames a data-tag in both roles; `tag_only=True` renames the tag only.
`MapData` is a data-only move.
- Removing the data column of a data-tag (`SelectDataColumns`, `DropDataColumns`)
leaves a plain tag and emits a `DataTagMaterializationWarning`
(`orcapod.core.data_tags.data_tag_materialization = "silent"` hides it).
- A function output named like a tag of its input is an error when the pipeline is built.
- Calling a function pod, `pod(stream)`, lets the input's data-tags fall through:
the output rows carry the function's outputs and the row's own data-tag columns, so
the tags stay data-tags (`FallThroughPod`). The function, its result identity
and its result store are the plain pod's. `pod.process(stream)` is unchanged: its
output data is the function's outputs and every output tag is plain.
- Saved job and pipeline files are at format `0.3.0`; files at `0.1.0` and `0.2.0` load
with every declared source tag plain, as they were written.

- `Batch` works on data-tags (ITL-1017; developer decision, November 16, 2026:
"Batch always simply takes anything and puts into list"). As for plain tags, every tag
becomes a list; a data-tag is one column, so it is the same list in both roles and
stays a data-tag. Its source info is one token per member, like any data column: nothing
is merged, so no token is folded. System tags fold as before. Plain tags are unchanged.

- A stored table states which of its tags are data-tags (ITL-1019): a function node's
pipeline table marks each data-tag column with Arrow field metadata
(`orcapod:tag_kind` = `data_tag`, `data_tags.TAG_KIND_METADATA_KEY`); a
plain tag carries no marker. The result table never records data-tag status. The
marker moves no hash and no stored path, and needs no new table layout.
- A function node reads its bindings back from its pipeline table and refuses
(`InputValidationError`, naming the table and both lists) a table whose marker disagrees
with what the node declares: its pod's fall-through tags, or a loaded job's saved record.
- A job or pipeline holding data-tags and fall-through pods can be saved and
loaded; it keeps its hashes and bindings. `FallThroughPod.to_config` and
`from_config` work.
- An operator node stores a stream with data-tags (`LOG`) and replays it (`REPLAY`): its
table marks the data-tags of the operator's own output with the same marker,
never one that came in from upstream. Every read of the table (a replay, the stored
view `get_all_records()`, a loaded job's read) checks the marker against the node (its
operator's prediction, or a loaded job's saved output schema) and refuses a
disagreement (`InputValidationError`, naming the table and both lists). Source info
stays part of each row's record hash; the marker is put on after the hash and never
enters it.
- `DerivedSource` over a stream with data-tags: `node.as_source()` of a function or operator
node presents the bindings its stored table states (checked against the node's keys),
in the plain form and with `tags_as_data`; `DerivedSource(source, tags_as_data=...)`
over a source with data-tags keeps the source's own. A `tags_as_data` copy of a
data-tag is a new plain data column. Nothing new enters the identity for the
binding: the plain form's content hash is the origin node's, and every pipeline hash
hashes the output schema, where a data-tag is in both halves. A saved derived
source writes no new key: its saved schema states the binding.

- `FieldMetadataKeepingDatabaseProtocol` (`orcapod.protocols.database_protocols`): a
database states with `keeps_field_metadata` whether it keeps Arrow field metadata.
`InMemoryArrowDatabase`, `DeltaTableDatabase` and a `ConnectorArrowDatabase` over
`SpiralDBConnector` state `True`. The SQLite and PostgreSQL connector databases and
`NoOpArrowDatabase` state `False`. `ExtensionAwareDatabase` states what the database
it wraps states, and a connector database what its connector states. A database
without the member is treated as not keeping field metadata.

- `SelectColumns` and `DropColumns` (`select_columns`, `drop_columns`) handle data-tags
(ITL-1022). A data-tag is one column in both roles, so naming it acts on both:
selecting it keeps it a data-tag, dropping it (or leaving it out of a select) removes the
column from both roles. The roles are never split and nothing is materialized. Before,
both operators raised "not supported with data-tags yet".

- `TagBy` and the stream method `tag_by` give existing data columns the tag role, as
data-tags (ITL-1175): `stream.tag_by("x", "y")` (or `tag_by(["x", "y"])`). Each
named column becomes one column in both roles, with its values and source info
unchanged; the new tags follow the stream's tags in the order given, and that order is
part of the operator's identity. System tags, record ids and every data column are
unchanged. A column a function computed may become a tag and then key a join or a
group. Refused, naming the column: a name that is not a data column of the input, a
name that is already a tag, a reserved name, and a type that cannot be a tag (a tag is
one `int`, `float`, `bool`, `str`, `bytes`, `datetime` or `date` value per row; a
nullable column is refused, with `drop_nulls` as the way out). `TagBy` is registered
for saved jobs and pipelines.

- `AddTags` (`add_tags`, ITL-1196) works on a stream with data-tags: the input's
data-tags stay data-tags, with their values and source info, and the new tags are
plain. A new tag named like any column of the input, a data-tag included, is refused,
as before. Sync and async agree.

- `MergeJoin` handles data-tags (ITL-1194), following `Join`: a tag that is a data-tag in
both inputs is one scalar data-tag column, not a two-element list, and its source
info is one merged token, the `join_v1` fold of the two inputs' tokens (sorted), as
`Join` gives. A tag that is a data-tag in one input stays a data-tag with its own token when
the other input has it as a plain tag or not at all; a data-only column of that name in
the other input is refused. A shared key that is `T` in one input and `T | None` in the
other is `T` in both roles (ITL-829). Every other colliding data column is merged into a
sorted list, as before; inputs without data-tags merge exactly as before.

### Changed

- Data-tags come first among the data columns, in tag order, then the other data columns
in table order (developer decision, November 17, 2026, ITL-1205): in a stream's
`keys()` and data schema, in every operator's predicted schema, and in every row. A
table built from rows lays a data-tag out among the tags, so the sync path, the async
path and a job node's rows now agree with the keys; before, a source with data
`a, x, y` (`x` a data-tag) listed `a, x, y` while the rows of a node over it, and an
operator's async result, gave `x, a, y`. No table's column order, stored table or hash
changes. `TagBy` places the new data-tags there too.

- The select and drop operators act on a data-tag by role (developer decision,
November 16, 2026, ITL-1195), as documented in "Moving a column between roles": for a
data-tag `x`, dropping the tag (`drop_tag_columns`) removes the tag role only
and leaves `x` plain data with its source info; dropping the data (`drop_data_columns`)
removes the data role only and leaves `x` a plain tag with the same values; dropping
the column (`drop_columns`) removes it from both roles. `select_tag_columns`,
`select_data_columns` and `select_columns` follow the same rule for the names they do
not select. Removing the data role emits `DataTagMaterializationWarning`; removing the
tag role or the column is silent. No second name (such as `untag`) is added.

- The error for a name that is a tag in one join input and a data column in the other
also offers `tag_by` as a way out (ITL-1175).

- A node that would store a data-tag refuses every database that does not state
`keeps_field_metadata = True`, before anything is written, so an unknown database
never passes by default. A database that states `True` is not probed. Before, only a
database that implements `validate_records` was checked, with a probe of null-typed
columns.

- Saved files of format `0.3.0` state every binding: each source config that takes the
option writes `tag_only_columns` (`True` when every tag is plain), and each function
node writes `fall_through_tags` (empty for a plain pod). The format version decides
what a file means, never a missing key: a file at `0.1.0` or `0.2.0` holds only plain
tags, and a `0.3.0` file without these keys, or whose keys disagree with its saved
schemas, is refused on load.

- Errors that a data-tag causes name it and the way out; no behaviour changes.
- A bind refused because a column is a data-tag on one side and a plain tag or a
data column on the other (listed as a missing or unexpected column) says which side
holds the data-tag, and how to make the two agree: `tag_only_columns` on the
source, or a slot declared with the column in both roles (ITL-1142). The bind stays
strict.
- `pod.process(stream)` and `pod(stream, strict=True)` refusing a data-tag the
function does not read name the data-tags and the two ways out: `pod(stream)`,
or `tag_only_columns`, each only where it can work (not when no data column of the
input is a parameter of the function, except a plain `FunctionPod`'s call).
- "Not supported with data-tags yet" names `tag_only_columns=True`, which keeps
every tag plain as in v0.3, beside its issue.

### Known limitations

- `MapColumns` raises "not supported with data-tags yet"
(operator-node storage and `DerivedSource` over a stream with data-tags are supported,
see ITL-1019 above).
`PollingSource` keeps its tags plain. The identities of streams with data-tags may still
change.

- The SQLite and PostgreSQL databases keep no Arrow field metadata, so a function node
that would store a data-tag there is refused (`InputValidationError`, naming the
database and the two ways out: a database that keeps field metadata, or
`tag_only_columns`) before anything is written. The same holds for an operator node
attached to such a database in `LOG` or `REPLAY` mode.
- SpiralDB's `keeps_field_metadata` is verified against the pyspiral fake only (first
write, append, merge on overlap, reread). The live service is unverified. The
connector database refuses schema evolution, so no merge can drop the marker. The
SpiralDB connector writes a table's Arrow metadata before its rows, so a failed
metadata write stores no row unmarked; a write that stores no row publishes nothing;
a failed write puts the metadata entry back as it was.
- Concurrent writers to one SpiralDB table through separate connectors are not
coordinated: one writer's rollback of a failed write can remove the metadata entry
another writer just published, so rows can read back without their data-tag
marker. A node refuses such a table on read (the marker must match the node), so no
wrong binding is ever read (ITL-1190).
- Writes of differing field metadata to one table are not refused. The in-memory
database and the connector database combine buffered and committed rows with the
first table's field metadata, so a batch whose marker differs from the rows already
there takes theirs. Orcapod's own writers never do this: a table's path derives from
its schema, where a data-tag is in both halves, so one path has one binding.
Only an outside writer can mix them, and a lost marker is refused on read (ITL-1190).
- A fall-through function node with ephemeral results records the hash of the function's
plain output, not of the output with its data-tag columns, so after the ephemeral
result is lost a downstream node's lookup misses (`EphemeralResultMissingError`;
ITL-1199).
- A loaded job keeps the pending type of a data-tag column whose class is not
registered for one role only, so binding a source of that class can be refused
(ITL-1200).
- `Join` raises `ValueError` on a list-valued data-tag key whose source info is one
token per member on one side (after `Batch`) and one token per cell on the other (a
source's list-valued tag) (ITL-1201).
- Reported, not reproduced: a merged float data-tag key could take `0.0` on one path
and `-0.0` on the other (ITL-1202).
- `CachedSource` neither marks nor checks the data-tags of its cache table, so a source with
data-tags and a plain source sharing an explicit `cache_path` can read each other's table
with the wrong binding (ITL-1215).
- On the async compute path, execution observers of a fall-through function node receive
the function's own input and output, without the data-tag columns that the sync path
and cache hits include (ITL-1203).
Loading
Loading