Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
84d1312
feat(referential_tags): stored tables and saved jobs keep tag binding…
Nov 16, 2026
199221c
feat(referential_tags): refuse a database that cannot keep the marker…
Nov 16, 2026
7d6c580
chore: merge the slice branch (.coderabbit.yaml, PR #405) into ITL-1019
Nov 16, 2026
fc1db0a
fix(pipeline): a saved-node error of a loaded job names the job file …
Nov 16, 2026
00818bd
fix(pipeline): refuse a saved source config whose bindings contradict…
Nov 16, 2026
ec5984a
fix(pipeline): check an explicitly empty tag_columns against the slot…
Nov 16, 2026
26d981a
feat(referential_tags): operator nodes store and derived sources pres…
Nov 16, 2026
cb3dbc4
refactor(operator_node): the stored-table check returns nothing; one …
Nov 16, 2026
3c51256
test(referential_tags): reach the DerivedSource binding check; log IT…
Nov 16, 2026
cecbaf3
feat(databases): a database states keeps_field_metadata; an undeclare…
Nov 16, 2026
69776b0
chore: merge the slice branch (PR #402 squash-merged) into ITL-1019 W3
Nov 16, 2026
7137e72
fix(databases): a SpiralDB retry publishes metadata whose write faile…
Nov 17, 2026
1e9565a
fix(databases): a SpiralDB write that stores no row never replaces st…
Nov 17, 2026
5261e9a
test(referential_tags): type-clean field metadata in the SpiralDB sto…
Nov 17, 2026
ded5305
fix(databases): SpiralDB writes a table's Arrow metadata before its r…
Nov 17, 2026
f030aab
fix(databases): a failed first SpiralDB write removes the metadata it…
Nov 17, 2026
4b49640
fix(databases): a failed SpiralDB write puts the metadata entry back …
Nov 17, 2026
a6cf552
fix(databases): read the SpiralDB metadata snapshot before the nullab…
Nov 17, 2026
01f3db4
docs(changelog): concurrent SpiralDB writers are not coordinated; fol…
Nov 17, 2026
6274a3c
chore: merge the refreshed slice (main 7c735b7a, v0.3.0) into ITL-101…
Nov 17, 2026
a7f08ed
test(referential_tags): meet main's SpiralDB column refusal and hash-…
Nov 17, 2026
ce32310
fix(operator_node): replayed rows carry the node's data context in th…
Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 49 additions & 1 deletion changelog.d/ITL-1019.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,39 @@
- A job or pipeline holding referential tags and fall-through pods can be saved and
loaded; it keeps its hashes and bindings. `ReferenceFallThroughPod.to_config` and
`from_config` work.
- An operator node stores a referential stream (`LOG`) and replays it (`REPLAY`): its
table marks the referential tags of the operator's own output with the same marker,
never one that came in from upstream. Every read of the table (a replay, the stored
view `get_all_records()`, a loaded job's read) checks the marker against the node (its
operator's prediction, or a loaded job's saved output schema) and refuses a
disagreement (`InputValidationError`, naming the table and both lists). Source info
stays part of each row's record hash; the marker is put on after the hash and never
enters it.
- `DerivedSource` over a referential stream: `node.as_source()` of a function or operator
node presents the bindings its stored table states (checked against the node's keys),
in the plain form and with `tags_as_data`; `DerivedSource(source, tags_as_data=...)`
over a referential source keeps the source's own. A `tags_as_data` copy of a
referential tag is a new plain data column. Nothing new enters the identity for the
binding: the plain form's content hash is the origin node's, and every pipeline hash
hashes the output schema, where a referential tag is in both halves. A saved derived
source writes no new key: its saved schema states the binding.

- `FieldMetadataKeepingDatabaseProtocol` (`orcapod.protocols.database_protocols`): a
database states with `keeps_field_metadata` whether it keeps Arrow field metadata.
`InMemoryArrowDatabase`, `DeltaTableDatabase` and a `ConnectorArrowDatabase` over
`SpiralDBConnector` state `True`. The SQLite and PostgreSQL connector databases and
`NoOpArrowDatabase` state `False`. `ExtensionAwareDatabase` states what the database
it wraps states, and a connector database what its connector states. A database
without the member is treated as not keeping field metadata.

### Changed

- A node that would store a referential tag refuses every database that does not state
`keeps_field_metadata = True`, before anything is written, so an unknown database
never passes by default. A database that states `True` is not probed. Before, only a
database that implements `validate_records` was checked, with a probe of null-typed
columns.

- Saved files of format `0.3.0` state every binding: each source config that takes the
option writes `tag_only_columns` (`True` when every tag is plain), and each function
node writes `fall_through_references` (empty for a plain pod). The format version decides
Expand All @@ -26,4 +56,22 @@
- The SQLite and PostgreSQL databases keep no Arrow field metadata, so a function node
that would store a referential tag there is refused (`InputValidationError`, naming the
database and the two ways out: a database that keeps field metadata, or
`tag_only_columns`) before anything is written.
`tag_only_columns`) before anything is written. The same holds for an operator node
attached to such a database in `LOG` or `REPLAY` mode.
- SpiralDB's `keeps_field_metadata` is verified against the pyspiral fake only (first
write, append, merge on overlap, reread). The live service is unverified. The
connector database refuses schema evolution, so no merge can drop the marker. The
SpiralDB connector writes a table's Arrow metadata before its rows, so a failed
metadata write stores no row unmarked; a write that stores no row publishes nothing;
a failed write puts the metadata entry back as it was.
- Concurrent writers to one SpiralDB table through separate connectors are not
coordinated: one writer's rollback of a failed write can remove the metadata entry
another writer just published, so rows can read back without their referential tag
marker. A node refuses such a table on read (the marker must match the node), so no
wrong binding is ever read (ITL-1190).
- Writes of differing field metadata to one table are not refused. The in-memory
database and the connector database combine buffered and committed rows with the
first table's field metadata, so a batch whose marker differs from the rows already
there takes theirs. Orcapod's own writers never do this: a table's path derives from
its schema, where a referential tag is in both halves, so one path has one binding.
Only an outside writer can mix them, and a lost marker is refused on read (ITL-1190).
42 changes: 35 additions & 7 deletions docs/concepts/referential-tags.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,10 +190,14 @@ column that carries Arrow field metadata: the key `orcapod:tag_kind` with the va
marker. The function's result table never records referential status, so plain and
referential runs of the same function share their results. The marker moves no hash, and
the stores keep it on every write (first write, append, merge, schema evolution; in
memory and on Delta). The SQLite and PostgreSQL databases keep no Arrow
metadata at all: a function node that would store a referential tag there is refused when
its database is attached, before anything is written. Use a database that keeps field
metadata, or keep those tags plain with `tag_only_columns`.
memory and on Delta). A database states whether it keeps
field metadata with `keeps_field_metadata`
(`orcapod.protocols.database_protocols.FieldMetadataKeepingDatabaseProtocol`): the
in-memory, Delta and SpiralDB databases do; the SQLite and PostgreSQL databases keep no
Arrow metadata at all. A node that would store a referential tag in a database that does
not state it (a database without the member included) is refused when its database is
attached, before anything is written. Use a database that keeps field metadata, or keep
those tags plain with `tag_only_columns`.

```python
from orcapod.core.referential_tags import stored_references
Expand All @@ -218,6 +222,32 @@ A node reads the binding back from its table, and refuses a table whose marker d
with what the node declares (its pod's fall-through tags, or for a loaded job the saved
file's).

An operator node that stores its output (`LOG`) marks its table the same way: the
referential tags of the operator's own output carry the marker, written from the
operator's binding, never from a marker that came in from upstream. A `REPLAY` read, and
the stored view `get_all_records()`, check the marker against the node (its operator's
prediction, or for a loaded job the saved output schema, where a referential tag is in
both halves). A database that keeps no field metadata is refused when it is attached in
`LOG` or `REPLAY` mode. Source info stays part of each stored row's record hash; the
marker never enters it.

`node.as_source()` gives the stored rows as a source with the bindings the stored table
states, in the plain form and with `tags_as_data` (a copy of a referential tag is a new,
plain data column). Nothing is added to the derived source's identity for the binding: its
plain form's content hash is the node's, which carries the binding already, and its
pipeline hash is its output schema, where a referential tag is in both halves.

```python
derived = job.nodes["scale"].as_source()
print(derived.keys())
# (('unit', 'time'), ('unit', 'time', 'scaled'))
print(derived.as_table().column_names)
# ['unit', 'time', 'scaled']
```

In a stream the binding is the shared name, so the marker stays with the stored table:
the rows of a replay or of a derived source carry none.

A saved job or pipeline (format `0.3.0`) states every binding: each source config writes
`tag_only_columns` (`True` when every tag is plain), and each function node writes
`fall_through_references` (empty for a plain pod). The format version, never a missing
Expand All @@ -228,9 +258,7 @@ the file is checked against the stored tables when a node reads them.
## Not supported yet

These raise "not supported with referential tags yet" when a referential tag reaches them:
`MergeJoin`, `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`), storing an operator node's results (`LOG` and `REPLAY` cache modes),
and `DerivedSource` over a stream with referential tags (so `node.as_source()` of a node
with referential tags).
`MergeJoin` and `MapColumns` (`map_columns`; rename with `map_tags` and `map_data`).
`PollingSource` keeps its tags plain. Each error names its issue and the way out: build the
source with `tag_only_columns=True` to keep every tag plain, as in v0.3. A plain tag is not
data, so a function that reads a tag's value (as `scale` above reads `time`) can no longer
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/design_issues/ITL-1023-RT1.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@
**Severity:** medium
**Issue:** ITL-1023

The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), storing an operator node's results in `LOG` or `REPLAY` mode, and `DerivedSource` over a referential stream (ITL-1019; saving a pipeline or job and the function node's stored bindings are supported since ITL-1019), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
The workshop slice of referential tags (specification attached to ITL-1004) covers sources, function pods, `Join`, `GroupBy`, `Batch` (since ITL-1017), select and drop, renames, filters, `DropNulls`, `SemiJoin`, `Pick` and `Index`. Everything else raises "not supported with referential tags yet" when a referential tag reaches it, instead of giving a wrong answer: `MergeJoin` (ITL-1018), and any operator that does not declare `REFERENTIAL_TAGS_ISSUE = None` (ITL-1022). Saving a pipeline or job, the function node's stored bindings, an operator node's stored results (`LOG`, `REPLAY`) and `DerivedSource` over a referential stream are supported since ITL-1019. `PollingSource` keeps its tags plain. The identities of referential streams are not frozen yet. The project's issues (ITL-1009 onward) land the full design and close this entry.
8 changes: 8 additions & 0 deletions docs/dev/design_issues/ITL-1163-LJ4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
## `src/orcapod/pipeline/job.py` — reading a loaded `PipelineJob`

### LJ4 — A saved job does not record an operator node's cache mode
**Status:** open
**Severity:** medium
**Issue:** ITL-1163

`OperatorJobNode.from_descriptor` reads `descriptor.get("cache_mode", "off")`, but `PipelineJob.save` never writes `cache_mode` into an operator node's descriptor. Every loaded operator node is therefore `OFF`, whatever mode it ran in. A load-only run of a loaded job runs the operator again over its rebuilt inputs instead of replaying the table a `LOG` or `REPLAY` node stored. The stored rows are still read by `node.get_all_records()` and `node.as_source()`, and by `REPLAY` when the caller attaches that mode again after loading. This is the case in plain mode and with referential tags (found while adding operator-node storage of referential streams, ITL-1019; `tests/test_pipeline/test_saved_job_with_referential_operator_nodes.py` attaches `REPLAY` again). A fix writes the mode into the saved descriptor, which is a saved-file format question.
Loading
Loading