Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
7bfcb54
docs(plan): ITL-944 step 1 plan for data context v1
Nov 12, 2026
26dcbc5
feat(contexts): data context std:v1:default with a frozen-manifest ch…
Nov 12, 2026
87df847
wip(store): __data_context column, new table layouts, read filter, ol…
Nov 12, 2026
02db3f0
wip(store): function version column, tests moved to the new layouts (…
Nov 12, 2026
9b1c2be
feat(store): __data_context on every stored row, new table layouts, o…
Nov 12, 2026
2e619aa
feat(contexts): manifest fields of the versioning design; logical typ…
Nov 12, 2026
a7691dc
feat(store): function pods that take the invocation context fail on a…
Nov 12, 2026
07ccf30
refactor(store)!: remove the v0.1 manifest and the v0 to v1 migration…
Nov 12, 2026
18a110b
feat(store): refuse an input of another data context (ITL-944)
Nov 12, 2026
8b05071
fix(store): address CodeRabbit review of PR #313 (ITL-944)
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,46 @@ hashes, forming a Merkle chain.

Prefixes are computed from SystemConstant in system_constants.py.

Stored tables: data context and layout versions (developer, November 11, 2026, ITL-944)

Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, large_string):
the key of the data context that wrote it (std:v1:default since v0.3). The key names one
generation of hashing; rows of the current data context are the current entries.

Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION)
Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION)
Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION)
Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION)
Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION)

- The layout version is the last component of a table path. A change to the bookkeeping
columns of a table kind moves its layout version, so the table gets a new path. Never add,
rename or retype a stored column without it.
- Every new stored table kind writes __data_context (contexts/stored.with_data_context), and
every read or lookup keeps rows of the current data context only
(keep_rows_of_data_context, or a lookup constraint on the column).
- A table at an older layout is never read as current and never modified. What a node does
about such tables, and about rows of another data context, is decided in one place,
contexts/stored.account_for_stale_tables: count and report, or, for a node that delivers
side effects (a SideEffectPod node, or a function node whose pod takes the invocation
context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of
node goes through that function; never decide it locally.
- A released data context manifest (contexts/data/<version>.json) is frozen: its sha256 is
pinned in contexts/released_manifests.json. A change that moves an existing golden hash
value for a reason other than a logical type's own change needs a new data context version
with its own manifest file, never an edit of a released one.
- In memory the data context column is still _context_key (ColumnConfig context).
- A writer refuses an input of another data context than its own
(contexts/stored.require_same_data_context, InputValidationError naming both keys) before
anything is written; a whole-input condition is checked outside the row error policy. The
eventual behaviour is ITL-951.
- std:v0.1:default is not shipped; saved files that name it load through
contexts.resolve_saved_context. There is no SchemaVersionError, ignore_schema or
orcapod.migrations: do not reintroduce a per-layout migration path without a decision.
- Logical types and the value handlers that hash their Python values are registered in
contexts/logical_types.json, never in a data context manifest: a logical type is
versioned by its own identifier, and that file is not frozen.

### System tag evolution rules

1. Name-preserving — single-stream ops. Column name/value pass through unchanged.
Expand Down
180 changes: 174 additions & 6 deletions CHANGELOG.md

Large diffs are not rendered by default.

42 changes: 42 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,48 @@ The pipeline hash uses a **resolver pattern** — `PipelineElementProtocol` obje
Prefixes are computed from `SystemConstant` in `system_constants.py`. The `constants` singleton
(with no global prefix) is used throughout.

### Stored tables: data context and layout versions

Every row Orcapod stores carries `__data_context` (`constants.DATA_CONTEXT_COL`,
`large_string`): the key of the data context that wrote it (`std:v1:default` since v0.3;
developer, November 11, 2026, ITL-944). The key names one generation of hashing; rows of the
current data context are the current entries.

| Table | Layout (last path component) | Constant |
|---|---|---|
| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` |
| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` |
| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` |
| Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` |
| Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` |

- A change to the bookkeeping columns of a table kind moves its layout version, so the table
gets a new path. Never add, rename or retype a stored column without it.
- Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`),
and every read or lookup keeps rows of the current data context only
(`keep_rows_of_data_context`, or a lookup constraint on the column).
- A table at an older layout is never read as current and never modified. What a node does
about such tables, and about rows of another data context, is decided in one place,
`contexts/stored.account_for_stale_tables`: count and report, or, for a node that delivers
side effects (a `SideEffectPod` node, or a function node whose pod takes the invocation
context, `ctx_arg`), raise `SideEffectLogDataContextError` before any delivery. A new kind
of node goes through that function; never decide it locally.
- A released data context manifest (`contexts/data/<version>.json`) is frozen: its sha256 is
pinned in `contexts/released_manifests.json`. A change that moves an existing golden hash
value for a reason other than a logical type's own change needs a new data context version
with its own manifest file, never an edit of a released one.
- In memory the data context column is still `_context_key` (`ColumnConfig(context=True)`).
- A writer refuses an input of another data context than its own
(`contexts/stored.require_same_data_context`, `InputValidationError` naming both keys)
before anything is written; a whole-input condition is checked outside the row error
policy. The eventual behaviour is ITL-951.
- `std:v0.1:default` is not shipped; saved files that name it load through
`contexts.resolve_saved_context`. There is no `SchemaVersionError`, `ignore_schema` or
`orcapod.migrations`: do not reintroduce a per-layout migration path without a decision.
- Logical types and the value handlers that hash their Python values are registered in
`contexts/logical_types.json`, never in a data context manifest: a logical type is
versioned by its own identifier, and that file is not frozen.

### System tag evolution rules

1. **Name-preserving** — single-stream ops (filter, select, map). Column name and value pass
Expand Down
146 changes: 146 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2442,3 +2442,149 @@ results.
Known compromise: the struct's children are nullable at the Arrow level, because Polars drops
nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null
instead. Tighten under a new storage version once nested nullability survives Polars.

---

## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix)

### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans
**Status:** resolved
**Severity:** high
**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927)

`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost
information for some inputs, so different data reached the same row hash: nested-list lengths
(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are
all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean
column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list`
data, so a source with such a column failed at construction (ITL-927).

**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2`
(developer's label rule: the label moves whenever any table hash can change). Every
Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options
stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with
their extension identity instead of as their storage type.

Stored rows: under the default `table_scope`, table paths do not move (developer, November
11, 2026: only a change of a table's schema goes into its path; a value-only change stays in
the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision
that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the
ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under
`arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place,
unchanged and distinguishable, for a later migration (ITL-913, ITL-914).

Key lookups cannot match a row of another label. Every read of a whole table drops the rows
of another label through one function, `hashing/stored_hash_labels.keep_rows_hashed_under`
(without it: 4 rows for 2 inputs): `FunctionJobNode._fetch_joined_records` (phase 1,
`get_all_records`, `as_table`, `iter_data`, `as_source`, loaded jobs) on
`__pipeline_base_entry_id`; `ResultCache.get_all_records` on `__input_data_hash`;
`OperatorJobNode._read_stored_records` and `CachedSource._read_cached_records` on the starfix
format at the start of the `_record_hash` record id. Each reader logs one warning with the
total number of rows it left out (`report_skipped_rows`; developer: total counts are enough,
eventually old entries should be detected and a migration suggested).
`stored_hash_labels(database, record_path)` counts a table's rows by label.

Side-effect invocation logs are the one exception (developer, same day): they are read by key
only, so deliveries logged under `arrow_v0.1` would not be found and every side effect would
be delivered again. A side-effect node whose log holds such rows raises
`SideEffectLogHashLabelError` before delivering anything. The developer wants a separate,
later discussion of what side-effect logs hold.

Known limits, follow-ups:
- The warning is logged once per reader object (node, result cache, cached source) and
table, not once per process: stored paths are relative to the store, so a process-wide
record of reported tables would silence a second store with the same layout.
- A global hash-version identifier stored with every row is being designed (not decided); it
would also give operator-node and cached-source rows a label.
- Operator-node and cached-source record hashes carry no label, only the starfix format at
the start of the hex digest. That separates `arrow_v0.1` (format 1) from `arrow_v0.2`
(format 2) exactly, but a future label that keeps the starfix format could not be told apart
there. Storing the label with those rows (a schema change) would close that gap.
- The node content hash does not separate the generations: a source with an explicit
`source_id` gives the same node content hash under both labels.
- A `DeltaTableDatabase` object keeps the table version it read first, so a job object that
read a table before a run does not see the rows the run added until the store is reopened
(not specific to hash labels; seen when re-reading a node of the same job after an
incremental run).
- Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an
explicit `source_id`) move to new paths when the content hash moves, so their v0.2 rows are
in another table.

---

## `src/orcapod/contexts/` — data context of stored rows

### DC1 — a stored row did not say which generation of hashing wrote it
**Status:** resolved, with open follow-ups listed below
**Severity:** high
**Issue:** ITL-944

The data context key (`std:v0.1:default`) never changed although its manifest
`contexts/data/v0.1.json` was edited in place 44 times, so the key stored in result rows
(`_context_key`) and pipeline rows (`__input_data_context_key`) told a reader nothing.
Operator-cache, cached-source and side-effect tables stored no key at all. PR #309 told the
rows of v0.2 from the rows of v0.3 by the label inside each hash, which names an algorithm,
not a generation, and is missing from operator-cache and cached-source record ids.

**Fix:** (developer's decisions of November 11, 2026, ITL-938 design document Parts 5.5 to 8)
- The default data context is `std:v1:default`, manifest `contexts/data/v1.json`.
`contexts/released_manifests.json` pins the sha256 of every released manifest (`null` while
a manifest is open); `tests/test_contexts/test_data_context_v1.py` fails when a pinned file
changes, when more than the default manifest is open, or when a final release ships an open
manifest. `v1.json` is pinned once, in the v0.3.0 release-preparation commit.
- Every stored row carries `__data_context` (`large_string`): function pipeline and result
tables, operator-cache tables, cached-source tables, side-effect invocation logs. It
replaces `__input_data_context_key` and the result table's `_context_key`. The value is the
key of the node, cache or source that writes the row.
- Layouts moved: `pdb_v2`, `rdb_v2`, `tdb_v2`, and new components `odb_v1` (operator cache)
and `sdb_v1` (cached source, also after an explicit `cache_path`).
- Reads and lookups keep rows of the current data context only, through
`contexts/stored.keep_rows_of_data_context` or a lookup constraint on the column. A table
at an old layout is counted and reported once per reader object, never read as current,
never written.
- What a node does about stored rows of other generations is decided in one place,
`contexts/stored.account_for_stale_tables`: report and skip, or, for a node that delivers
side effects, raise `SideEffectLogDataContextError` before anything is delivered. Both
kinds of side-effecting node are covered (developer, November 11, 2026): `SideEffectPod`
nodes (invocation log) and function nodes whose pod takes the invocation context
(`ctx_arg`; pipeline and result tables).
- `orcapod.contexts.stored_data_contexts(database, record_path)` lists the data contexts a
table holds. The label filtering of PR #309 (`hashing/stored_hash_labels.py`) is removed.
- The result table also stores `__pf_var_function_version`, the full function version.

Known limits, follow-ups:
- Row keys do not include the data context. A later data context under which a row's key is
unchanged collides with the old row on write (`skip_duplicates`), and the new row is not
stored. The design's answer is that migration relabels unaffected rows; until a migration
exists this must be revisited before `std:v2:default` is introduced.
- The content hash of a stream without a producer covers its `_context_key` column, so it
moves with the data context key.
- Old-layout tables are reported once per reader object, not once per process or job, and
counting their rows reads the whole table (the store protocol has no row count).
- The manifest has the fields of the ITL-938 design document, section 7.4 (`released_in`,
`previous`, `definitions`, `libraries`, `golden`, `changes`), and the logical types and
their value handlers moved to the unfrozen `contexts/logical_types.json`. The names under
`definitions` are labels only: nothing in the code reads them or checks that a definition
changed, and `changes` lists what is true on this branch (starfix format 2); the entries
for ITL-935 and ITL-942 are added when those land. `golden.digest` stays `null` until the
manifest is frozen, because the golden files still move.
- `v0.1.json` is removed (to live in `orcapod-sanctuary`); `std:v0.1:default` raises
`ContextResolutionError`, and a node saved under it loads under the current data context
with a warning (`contexts.resolve_saved_context`).
- The v0 to v1 machinery is removed: `SchemaVersionError`, `NodeConfig(ignore_schema=)`,
`orcapod.migrations`, `orcapod migrate`. A table without a layout component is an old
layout like `pdb_v1`.
- An input of another data context raises `InputValidationError`
(`contexts/stored.require_same_data_context`) before anything is written. The developer
wants the node's data context to win eventually: ITL-951.
- Found while testing that rule, not fixed here: a data context is not carried reliably in
memory. `ArrowTableStream` gives every row the stream object's data context and ignores
the `_context_key` column of its table (the two can disagree); a source's `data_context`
is not given to the rows of its stream; and `PipelineJob` wraps each input in a source
node of the default data context. So inside a job an input of another data context is
not seen as such, and the check only fires for nodes and caches driven directly.
- For the same reason there is no per-row data context check on the cache-hit path of a
function node (CodeRabbit, PR #313): every row of a stream carries the stream object's
data context, so the whole-input check covers hits and misses. To revisit with ITL-951.
- Open: whether the in-memory column `_context_key` (`ColumnConfig(context=True)`) is
renamed to `__data_context` as well. It is unchanged here.
10 changes: 5 additions & 5 deletions docs/concepts/arrays.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ for tag, data in scale.pod(source).iter_data():
!!! warning "v0.3 changed the array format"
Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and
hashed the file. v0.3 replaces both, as described below, and does not migrate older
stores. Under v0.3, every node that handles arrays computes again into new tables, and its
v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes
downstream of an array column get new pipeline and operator tables too; a function pod
among them that takes and returns no array reuses its stored results. Start v0.3
pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is
stores. Every node that handles arrays, and every node downstream of an array column, gets
new tables in v0.3. Because v0.3 also changed every hash of Arrow data, v0.3 reuses no
result of a store written by v0.2: it ignores the v0.2 rows and computes every input
again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)).
The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is
planned after v0.3 (ITL-889).

## What counts as the same array
Expand Down
Loading
Loading