Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 90 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@
#### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883)

**The array format and every array hash changed in v0.3.** A store written by v0.2 that holds
`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays
computes again into new tables, and its v0.2 results are no longer returned. Nothing written by
v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for
v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889).
`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables,
so its v0.2 results are no longer found; and since every Arrow-derived hash changed too
(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and
computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay
readable with v0.2. A migration is planned after v0.3 (ITL-889).

- **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata
`{"version":1}`) over `struct<dtype: large_string, shape: large_list<int64>,
Expand All @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.
change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the
pipeline and operator tables of every node downstream of an array column move as well,
even when it has no array column itself. A pod whose inputs and outputs hold no array
keeps its function result table, so it reuses its v0.2 results without computing again.
Nodes with no array column upstream do not move.
keeps its function result table, and nodes with no array column upstream keep their tables;
but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed
as well (ITL-910, below), so the v0.2 rows in such tables are ignored, the inputs are computed
again, and the new rows are stored next to the old ones.
- **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming
the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes,
NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and
Expand All @@ -47,6 +50,80 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.
the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its
subclasses, whatever the registration order.

#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910)

**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1`
to `arrow_v0.2`.** starfix 0.4 (hash format 1) lost information for some inputs, so different
data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and
`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean
columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash
format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept.

- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default
`source_id`s and the `record_id`s and source content hashes built from them, function
input-data hashes, pipeline base and entry ids, operator and cached-source record hashes,
invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The
hash label in front of each (`arrow_v0.2:000002...`) shows the new format. Pipeline hashes,
schema hashes, system-tag column names and table paths under the default
`table_scope="pipeline_hash"` do not move. Paths that embed a content hash
(`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or
`Batch` orders members by `record_id`, the order of members can change too.
- **Results stored by 0.2.0 are ignored and computed again; they are not reused, changed or
deleted.** Rows a v0.2 store holds are keyed under `arrow_v0.1` and can never match v0.3
inputs. v0.3 does not refuse such a store. A value-only change does not move a table, so a
v0.3 job writes its `arrow_v0.2` rows into the same tables, next to the `arrow_v0.1` rows,
and uses rows of its own label only:
- The first v0.3 run on a v0.2 store **computes every input again**. Nothing stored by v0.2
counts as a cached result, also for functions and data that did not change.
- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, also on a
job loaded from a file, including a file saved by v0.2), `get_all_cached_outputs()` of a
cached function, a stored (`LOG` / `REPLAY`) operator node and a `CachedSource` return
rows hashed under `arrow_v0.2` only. Before the first v0.3 run, such a read on a v0.2
store returns no rows; after it, one row per input, never the v0.2 rows as well.
- The v0.2 rows stay where they are, byte for byte, and stay readable with v0.2. Each
stored key names its label (`arrow_v0.1:000001...` or `arrow_v0.2:000002...`), and
`orcapod.hashing.stored_hash_labels.stored_hash_labels(store, record_path)` counts the
rows of a table by label, for example `{"arrow_v0.1": 3, "arrow_v0.2": 3}`.
- When a read leaves out rows of another label, Orcapod logs one warning per table and
reader with the total, on the logger `orcapod.hashing.stored_hash_labels`:
`Skipped 3 rows of stored table 'my_job/.../pdb_v1': written under another hash label
than the current one. The rows are left in place unchanged and can be migrated later.`
- **Side-effect nodes are the exception: they fail instead of delivering again.** A
side-effect pod finds a delivery it already made by an Arrow-derived record id, so
deliveries logged by v0.2 cannot be found, and running the node would deliver every side
effect a second time. A side effect is an external action, not a value that can be
computed again, so a side-effect node whose invocation log holds deliveries logged under
another label raises `orcapod.errors.SideEffectLogHashLabelError` (a `StoreError`, so it
fails the run whatever the `error_policy`) before it delivers or writes anything. The
error names the log, the label found and the current one. Function, operator and source
nodes of the same job are not refused. Use a fresh store for the side-effect node, or
keep using v0.2 for it, until the log can be migrated.

**Migration:** none is needed to run the function, operator and source nodes of a v0.2
store under v0.3, at the price of computing everything again and of tables that hold both
generations; side-effect nodes need a fresh store. A tool that converts
`arrow_v0.1` rows into `arrow_v0.2` rows is planned (ITL-913, ITL-914); until then the old
rows are kept for it. Operator-node and cached-source tables store their record hashes
without a label; there the rows of v0.2 are told apart by the starfix hash format at the
start of each record hash (`000001` for v0.2, `000002` for v0.3). That is exact for these
two versions, but a later Arrow hash label that keeps starfix hash format 2 could not be
told apart in those two kinds of table. A table that `table_scope="content_hash"` or a
`CachedSource` places under a content hash moves to a new path instead, so its v0.2 rows
are in another table.
- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator
carrying one works, and its type is kept. A `fixed_size_list<T>[n]` hashes differently from a
`large_list<T>` with the same values. pyarrow's canonical extension types (`arrow.uuid`,
`arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid`
column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and
`StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846).
- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed
silently: a non-nullable field holding a null; a zero-row table with a `map` column; two
schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a
column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`,
interval, `string_view` and `binary_view` columns, union and run-end-encoded types are
still refused, now with `ValueError` instead of `NotImplementedError`.
- Column order and struct field order are still not part of an Arrow hash.

#### A changed column type is refused when appending to a Delta table (ITL-884)

Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows'
Expand Down Expand Up @@ -80,6 +157,13 @@ stay queued for the next flush. A flush that fails only for other reasons still
the backend's exception as its cause. (ITL-884 #290)
- **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`;
`except StoreError` handles both. (ITL-884 #290)
- **`orcapod.errors.SideEffectLogHashLabelError`** — a `StoreError` raised by a side-effect
node whose invocation log holds deliveries logged under another Arrow hash label (a log
written by v0.2), before anything is delivered. It has `record_path`, `found_label`,
`current_label` and `node_label`. (ITL-910)
- **`orcapod.hashing.stored_hash_labels.stored_hash_labels(database, record_path)`** — counts
the rows of a stored table by the Arrow hash label they were hashed under, to tell rows
written by v0.2 (`arrow_v0.1`) from rows written by v0.3 (`arrow_v0.2`). (ITL-910)

### Fixed

Expand Down
67 changes: 67 additions & 0 deletions DESIGN_ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -2442,3 +2442,70 @@ results.
Known compromise: the struct's children are nullable at the Arrow level, because Polars drops
nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null
instead. Tighten under a new storage version once nested nullability survives Polars.

---

## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix)

### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans
**Status:** resolved
**Severity:** high
**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927)

`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost
information for some inputs, so different data reached the same row hash: nested-list lengths
(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are
all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean
column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list`
data, so a source with such a column failed at construction (ITL-927).

**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2`
(developer's label rule: the label moves whenever any table hash can change). Every
Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options
stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with
their extension identity instead of as their storage type.

Stored rows: under the default `table_scope`, table paths do not move (developer, November
11, 2026: only a change of a table's schema goes into its path; a value-only change stays in
the same table), so a v0.3 node writes into the same tables as a v0.2 store. The first decision
that day was to refuse such a store (`StoreHashLabelError`); the developer reversed it in the
ITL-938 design session: v0.3 does not refuse a v0.2 store. It ignores rows hashed under
`arrow_v0.1`, computes every input again under `arrow_v0.2`, and leaves the old rows in place,
unchanged and distinguishable, for a later migration (ITL-913, ITL-914).

Key lookups cannot match a row of another label. Every read of a whole table drops the rows
of another label through one function, `hashing/stored_hash_labels.keep_rows_hashed_under`
(without it: 4 rows for 2 inputs): `FunctionJobNode._fetch_joined_records` (phase 1,
`get_all_records`, `as_table`, `iter_data`, `as_source`, loaded jobs) on
`__pipeline_base_entry_id`; `ResultCache.get_all_records` on `__input_data_hash`;
`OperatorJobNode._read_stored_records` and `CachedSource._read_cached_records` on the starfix
format at the start of the `_record_hash` record id. Each reader logs one warning with the
total number of rows it left out (`report_skipped_rows`; developer: total counts are enough,
eventually old entries should be detected and a migration suggested).
`stored_hash_labels(database, record_path)` counts a table's rows by label.

Side-effect invocation logs are the one exception (developer, same day): they are read by key
only, so deliveries logged under `arrow_v0.1` would not be found and every side effect would
be delivered again. A side-effect node whose log holds such rows raises
`SideEffectLogHashLabelError` before delivering anything. The developer wants a separate,
later discussion of what side-effect logs hold.

Known limits, follow-ups:
- The warning is logged once per reader object (node, result cache, cached source) and
table, not once per process: stored paths are relative to the store, so a process-wide
record of reported tables would silence a second store with the same layout.
- A global hash-version identifier stored with every row is being designed (not decided); it
would also give operator-node and cached-source rows a label.
- Operator-node and cached-source record hashes carry no label, only the starfix format at
the start of the hex digest. That separates `arrow_v0.1` (format 1) from `arrow_v0.2`
(format 2) exactly, but a future label that keeps the starfix format could not be told apart
there. Storing the label with those rows (a schema change) would close that gap.
- The node content hash does not separate the generations: a source with an explicit
`source_id` gives the same node content hash under both labels.
- A `DeltaTableDatabase` object keeps the table version it read first, so a job object that
read a table before a run does not see the rows the run added until the store is reopened
(not specific to hash labels; seen when re-reading a node of the same job after an
incremental run).
- Tables placed under a content hash (`table_scope="content_hash"`, `CachedSource` without an
explicit `source_id`) move to new paths when the content hash moves, so their v0.2 rows are
in another table.
10 changes: 5 additions & 5 deletions docs/concepts/arrays.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ for tag, data in scale.pod(source).iter_data():
!!! warning "v0.3 changed the array format"
Orcapod v0.2 stored arrays as `.npy` files under the extension name `numpy.ndarray` and
hashed the file. v0.3 replaces both, as described below, and does not migrate older
stores. Under v0.3, every node that handles arrays computes again into new tables, and its
v0.2 results are no longer returned. They stay on disk, readable with v0.2. Nodes
downstream of an array column get new pipeline and operator tables too; a function pod
among them that takes and returns no array reuses its stored results. Start v0.3
pipelines with a fresh store, or keep using v0.2 for an existing one. A migration is
stores. Every node that handles arrays, and every node downstream of an array column, gets
new tables in v0.3. Because v0.3 also changed every hash of Arrow data, v0.3 reuses no
result of a store written by v0.2: it ignores the v0.2 rows and computes every input
again (see [A store written by an older version](function-pods.md#a-store-written-by-an-older-version)).
The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is
planned after v0.3 (ITL-889).

## What counts as the same array
Expand Down
25 changes: 25 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,31 @@ at DEBUG and does not stop for `"fail_fast"` (ITL-879); this applies to these ro
This is the contract for now. An opt-in mode that tolerates store failures may be added
later.

### A store written by an older version

Stored rows are keyed by hashes of their Arrow data, and each hash carries the label of the
scheme that produced it: `arrow_v0.1` up to Orcapod v0.2, `arrow_v0.2` since v0.3 (see
[Arrow Table Hash](../reference/hashing.md#9-arrow-table-hash-arrow_v02)). Rows keyed under
another label can never match current inputs, so Orcapod **ignores** them: it does not refuse
the store, and it does not reuse those rows.

- The first run of v0.3 on a store written by v0.2 computes every input again and stores the
new rows in the same tables, next to the old ones.
- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the
current label only: nothing before that first run, one row per input after it.
- The old rows are never changed or deleted. They stay readable with the version that wrote
them and are kept for a later migration. To see which labels a table holds, use
`stored_hash_labels` (see
[Stores written under another label](../reference/hashing.md#stores-written-under-another-label)).
- When a read leaves such rows out, Orcapod logs one warning with their number:
`Skipped 3 rows of stored table '...': written under another hash label than the current
one. The rows are left in place unchanged and can be migrated later.`
- Side-effect pods are the exception. A side effect is an external action, and deliveries
logged by v0.2 cannot be matched with current inputs, so running the node would deliver
everything a second time. Instead the node raises `SideEffectLogHashLabelError` (from
`orcapod.errors`, a `StoreError`) before it delivers anything. Use a fresh store for that
node, or keep using the version that wrote the log.

## Multiple input streams

If you pass multiple streams to a function pod, they are automatically joined (using
Expand Down
Loading
Loading