Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
5c34afb
docs(plan): ITL-945 step 1 plan for the hash label rule
Nov 12, 2026
4030f55
docs(plan): ITL-945 tracer results and open points
Nov 12, 2026
2dc0d24
test(hashing): failing tests for the hash label rule (ITL-945)
Nov 12, 2026
8a1f5a1
refactor(hashing)!: rename hash labels to <algorithm>_v1 (ITL-945)
Nov 12, 2026
db94f0f
feat(hashing)!: keep hash labels out of every hash input (ITL-945)
Nov 12, 2026
2c4b1e3
feat(core)!: bare hex digests inside names; stream identity without t…
Nov 12, 2026
4b9e825
feat(store)!: labelled row keys for operator-cache and cached-source …
Nov 12, 2026
de11a36
test(golden): pinned hash values after the hash label rule (ITL-945)
Nov 12, 2026
bc012b3
refactor(hashing): define each hash label once, beside its algorithm …
Nov 12, 2026
50d453e
docs: the hash label rule in the hashing reference, CHANGELOG, DESIGN…
Nov 12, 2026
55ba591
fix(streams): keep tag sorting in the producer-less stream identity; …
Nov 12, 2026
a6adec1
docs: a v0.2 side-effect log is not detected at the v0.3 path (ITL-94…
Nov 12, 2026
7af5cd7
docs(changelog): a fresh store does not prevent repeated side effects…
Nov 12, 2026
40a7102
fix(sources): default derived source ids keep the hash label (ITL-945)
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,34 @@ checked by `tests/test_hashing/golden/test_golden_hashes.py`.
- Never move or rename the fixtures in `tests/test_hashing/golden/cases.py`: function pods,
dataclasses and pydantic models hash with their module path.

## Hash labels

One rule holds for every hash label (developer, November 11, 2026; ITL-945; stated with
examples in `docs/reference/hashing.md`, "The hash label rule").

- A label names the algorithm that produced a digest (the whole procedure: canonical form,
preparation, encoding, digest function) and nothing else. Format `<algorithm>_v<N>`,
integer `N`: `semantic_v1`, `arrow_v1`, `directory_v1`, `ndarray_v1`, `polars_expr_v1`. The
file content hash keeps `sha256`, the only exception. A logical type's implementer picks
its label.
- Change a label only when its algorithm would give an existing input a different digest.
- Define each label in exactly one place, as a constant beside its algorithm. Never write a
label literal anywhere else in code, and never in a data context file.
- Never feed a label into a hash computation. Feed a nested hash's digest only: never put
`to_string()`, `str(hash)` or `to_prefixed_digest()` into an identity structure, a preimage
table or a digest input. Inside a structure, return the `ContentHash` object and let the
structure hasher embed it.
- A hash inside a name (path component, column name, `uri`) is the bare hex digest:
`to_hex()`. Exception (developer, November 11, 2026): a source id is the source's name. A
default source id built from a hash keeps a source-kind prefix and the hash's label
(`derived:<label>:<hex>`, via `to_string()`, never a label literal) and is used as it is in
the `source_id` system tag and in source-info values.
- A hash stored as a value in its own column or field carries its label: `to_string()` as
text, `to_prefixed_digest()` in a binary column.
- A record id is an identifier built by a recipe, not a hash, and carries no label.
- `tests/test_hashing/test_hash_label_rule.py` recomputes every golden value under renamed
labels; a value that moves there means a label reached a hash input.

## Testing conventions

Tests use the **public interface** wherever possible (developer, November 10, 2026): drive an
Expand Down
160 changes: 154 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@
#### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883)

**The array format and every array hash changed in v0.3.** A store written by v0.2 that holds
`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays
computes again into new tables, and its v0.2 results are no longer returned. Nothing written by
v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for
v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889).
`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables,
so its v0.2 results are no longer found; and since every Arrow-derived hash changed too
(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and
computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay
readable with v0.2. A migration is planned after v0.3 (ITL-889).

- **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata
`{"version":1}`) over `struct<dtype: large_string, shape: large_list<int64>,
Expand All @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.
change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the
pipeline and operator tables of every node downstream of an array column move as well,
even when it has no array column itself. A pod whose inputs and outputs hold no array
keeps its function result table, so it reuses its v0.2 results without computing again.
Nodes with no array column upstream do not move.
keeps its function result table, and nodes with no array column upstream keep their tables;
but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed
as well (ITL-910, below), so the v0.2 rows in such tables are ignored, the inputs are computed
again, and the new rows are stored next to the old ones.
- **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming
the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes,
NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and
Expand All @@ -47,6 +50,144 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.
the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its
subclasses, whatever the registration order.

#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910)

**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1`
to `arrow_v1`.** starfix 0.4 (hash format 1) lost information for some inputs, so different
data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and
`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean
columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash
format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept.

- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default
`source_id`s and the `record_id`s and source content hashes built from them, function
input-data hashes, pipeline base and entry ids, operator and cached-source record hashes,
invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The
hash label in front of each (`arrow_v1:000002...`) shows the new format. Pipeline hashes,
schema hashes, system-tag column names and table paths under the default
`table_scope="pipeline_hash"` do not move for this reason; they move in v0.3 because of
the hash label rule (ITL-945, below). Paths that embed a content hash
(`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or
`Batch` orders members by `record_id`, the order of members can change too.
- **Results stored by 0.2.0 are ignored and computed again; they are not reused, changed or
deleted.** Rows a v0.2 store holds are keyed under `arrow_v0.1` and can never match v0.3
inputs. v0.3 does not refuse such a store. A value-only change does not move a table, and v0.3
uses rows of its own label only. (With the hash label rule, ITL-945 below, every v0.3 table
is in fact at a new path, so the rows v0.3 writes are in new tables and the v0.2 tables are
left as they are; the points below describe what a read at a v0.3 path returns.)
- The first v0.3 run on a v0.2 store **computes every input again**. Nothing stored by v0.2
counts as a cached result, also for functions and data that did not change.
- Reading a node (`as_table()`, `iter_data()`, `get_all_records()`, `as_source()`, also on a
job loaded from a file, including a file saved by v0.2), `get_all_cached_outputs()` of a
cached function, a stored (`LOG` / `REPLAY`) operator node and a `CachedSource` return
rows hashed under `arrow_v1` only. Before the first v0.3 run, such a read on a v0.2
store returns no rows; after it, one row per input, never the v0.2 rows as well.
- The v0.2 rows stay where they are, byte for byte, and stay readable with v0.2. Each
stored key names its label (`arrow_v0.1:000001...` or `arrow_v1:000002...`), and
`orcapod.hashing.stored_hash_labels.stored_hash_labels(store, record_path)` counts the
rows of a table by label, for example `{"arrow_v0.1": 3, "arrow_v1": 3}`.
- When a read leaves out rows of another label, Orcapod logs one warning per table and
reader with the total, on the logger `orcapod.hashing.stored_hash_labels`:
`Skipped 3 rows of stored table 'my_job/.../pdb_v1': written under another hash label
than the current one. The rows are left in place unchanged and can be migrated later.`
- **Side-effect nodes are the exception: they fail instead of delivering again, for a log
at their v0.3 path.** A log written by v0.2 is at another path (ITL-945, below), is not
detected, and its side effects are delivered again on the first v0.3 run. A
side-effect pod finds a delivery it already made by an Arrow-derived record id, so
deliveries logged by v0.2 cannot be found, and running the node would deliver every side
effect a second time. A side effect is an external action, not a value that can be
computed again, so a side-effect node whose invocation log holds deliveries logged under
another label raises `orcapod.errors.SideEffectLogHashLabelError` (a `StoreError`, so it
fails the run whatever the `error_policy`) before it delivers or writes anything. The
error names the log, the label found and the current one. Function, operator and source
nodes of the same job are not refused. A fresh store is not a way around this: it has
no invocation log, so the node delivers every side effect again. Run the node on a
fresh store only when a repeated delivery is acceptable, or keep using v0.2 for it
until the log can be migrated.

**Migration:** none is needed to run the function, operator and source nodes of a v0.2
store under v0.3, at the price of computing everything again; a side-effect node delivers again on
its first v0.3 run, whatever the store (see above). A tool that converts
`arrow_v0.1` rows into `arrow_v1` rows is planned (ITL-913, ITL-914); until then the old
rows are kept for it. Operator-node and cached-source tables written by v0.2 store
their record hashes without a label; such rows are told apart by the starfix hash format
at the start of each record hash (`000001` for v0.2). Since ITL-945 (below) v0.3 stores
these record hashes with their label, like every other stored hash. A table that `table_scope="content_hash"` or a
`CachedSource` places under a content hash moves to a new path instead, so its v0.2 rows
are in another table.
- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator
carrying one works, and its type is kept. A `fixed_size_list<T>[n]` hashes differently from a
`large_list<T>` with the same values. pyarrow's canonical extension types (`arrow.uuid`,
`arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid`
column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and
`StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846).
- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed
silently: a non-nullable field holding a null; a zero-row table with a `map` column; two
schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a
column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`,
interval, `string_view` and `binary_view` columns, union and run-end-encoded types are
still refused, now with `ValueError` instead of `NotImplementedError`.
- Column order and struct field order are still not part of an Arrow hash.

#### Hash label rule: labels renamed, out of hash inputs, bare digests inside names (ITL-945)

**Every structure hash, every table path and every key moves in v0.3.** One rule now holds
for every hash label (developer, November 11, 2026; `docs/reference/hashing.md`, "The hash
label rule"):

- **Labels are renamed to `<algorithm>_v1`.** `semantic_v0.1` becomes `semantic_v1`, the Arrow
label is `arrow_v1` (it was `arrow_v0.1` in v0.2), `merkle_sha256` becomes `directory_v1`;
`ndarray_v1` and `polars_expr_v1` keep their names; the file content hash keeps `sha256`.
A label names the algorithm that produced a digest and changes only when that algorithm
would give an existing input a different digest.
- **A label never enters a hash computation.** A nested hash now contributes its digest only:
as `{"__type__": "hash", "digest": "<hex>"}` inside a structure hash, as
`b"<extension name>::" + digest` for an extension-typed cell inside an Arrow hash, and as the
raw digest in the entry-id and operator-node record-hash preimages and in the run snapshot
hash. Renaming a label later changes no digest. Until now `"<label>:<hex>"` was hashed, so
this moves every `content_hash()` and `pipeline_hash()` whose structure holds a nested hash
(types, schemas, pods, streams), every schema hash and so every system-tag column name,
every Arrow hash over a logical-type cell, every pipeline base and entry id, and the run URI.
- **Hashes inside names are bare hex digests.** The function `uri` (its second component), the
path components `schema:`, `instance:` and `node:`, and the slot name of a `SourceNode`
built from a stream no longer hold a label. A source id is the exception: it is the
source's name, so the default `source_id` of a `DerivedSource` (`derived:<label>:<hex>`)
and of `node.as_source()` keeps the hash's label, now `semantic_v1`. A v0.3 function table is at
`<job>/<function>/<64 hex>/v<major>/python.function.v0/schema:<64 hex>/pdb_v1`.
`data_function.uri[1]` and `output_data_schema_hash` return the bare digest.
- **Hashes stored as values carry their label**, now including the record id (`_record_hash`)
of operator-node and cached-source tables: `b"arrow_v1:" + digest` instead of hex text.
- **A producer-less `ArrowTableStream`** no longer hashes its derived `_content_hash` column
into its content hash (the column's values carry the Arrow label).
- **`BasicDirectoryHasher(algorithm=...)`** raises `ValueError` for any algorithm but
`"sha256"`: `directory_v1` names the SHA-256 procedure.
- **Each label is defined once, in code**: `SEMANTIC_HASH_LABEL`, `ARROW_HASH_LABEL`,
`DIRECTORY_HASH_LABEL`, `NumpyArrayHandler.HASH_LABEL`, `POLARS_EXPR_HASH_LABEL`. The data
context file no longer carries `hasher_id`; `hasher_id` remains a constructor argument of
`SemanticAwarePythonHasher` and `StarfixArrowHasher` and defaults to the label.
- **Unchanged:** `ContentHash` and all its methods (`to_string`, `to_hex`,
`to_prefixed_digest`, `from_string`, `from_prefixed_digest`); the digests of plain Arrow
tables and schemas, of flat structures of primitives, and of ndarray, Polars expression,
file and directory values (their label text changes, their digest does not);
`InvocationContext.invocation_hash`, which keeps its `<label>:<hex>::<label>:<hex>` shape
with the new label names; record ids.
- **Every v0.3 table path differs from v0.2's.** For the same node, v0.2 and v0.3 agree on
the job name, the function or operator name, the major version, the wrapper kind and the
table layout suffix, and differ in every hash inside the path: the output-schema hash of a
function (`semantic_v0.1:<hex>` became another, bare, digest), `schema:<pipeline hash>`,
`instance:<content hash>` and `node:<content hash>`. Nothing in v0.3 that looks for older
data at its own table path can find a table written by v0.2. In particular the label filter
and `SideEffectLogHashLabelError` (ITL-910, above) act on rows at a v0.3 path only: a
side-effect log written by v0.2 is at another path, is **not detected**, and its side effects
are delivered again on the first v0.3 run. Detecting tables of an older layout is part of
ITL-944.
- **Migration.** None is needed beyond the v0.3 step already described: a store written by
v0.2 is an older generation whose rows v0.3 does not use. Function, operator and
cached-source tables are at new paths, so their v0.2 tables are left untouched. Code that
compared `uri[1]` or a path component with `content_hash().to_string()` must compare with
`to_hex()`. A job file saved by v0.2 holds the old labels and paths; loaded by
v0.3, its nodes point at no v0.3 table and show no rows.

#### A changed column type is refused when appending to a Delta table (ITL-884)

Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows'
Expand Down Expand Up @@ -80,6 +221,13 @@ stay queued for the next flush. A flush that fails only for other reasons still
the backend's exception as its cause. (ITL-884 #290)
- **`orcapod.errors.StoreError`** — the base of `StoreReadError` and `StoreWriteError`;
`except StoreError` handles both. (ITL-884 #290)
- **`orcapod.errors.SideEffectLogHashLabelError`** — a `StoreError` raised by a side-effect
node whose invocation log holds deliveries logged under another Arrow hash label (a log
written by v0.2), before anything is delivered. It has `record_path`, `found_label`,
`current_label` and `node_label`. (ITL-910)
- **`orcapod.hashing.stored_hash_labels.stored_hash_labels(database, record_path)`** — counts
the rows of a stored table by the Arrow hash label they were hashed under, to tell rows
written by v0.2 (`arrow_v0.1`) from rows written by v0.3 (`arrow_v1`). (ITL-910)

### Fixed

Expand Down
28 changes: 28 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,34 @@ checked by `tests/test_hashing/golden/test_golden_hashes.py`.
- Never move or rename the fixtures in `tests/test_hashing/golden/cases.py`: function pods,
dataclasses and pydantic models hash with their module path.

## Hash labels

One rule holds for every hash label (developer, November 11, 2026; ITL-945; stated with
examples in `docs/reference/hashing.md`, "The hash label rule").

- A label names the algorithm that produced a digest (the whole procedure: canonical form,
preparation, encoding, digest function) and nothing else. Format `<algorithm>_v<N>`,
integer `N`: `semantic_v1`, `arrow_v1`, `directory_v1`, `ndarray_v1`, `polars_expr_v1`. The
file content hash keeps `sha256`, the only exception. A logical type's implementer picks
its label.
- Change a label only when its algorithm would give an existing input a different digest.
- Define each label in exactly one place, as a constant beside its algorithm. Never write a
label literal anywhere else in code, and never in a data context file.
- Never feed a label into a hash computation. Feed a nested hash's digest only: never put
`to_string()`, `str(hash)` or `to_prefixed_digest()` into an identity structure, a preimage
table or a digest input. Inside a structure, return the `ContentHash` object and let the
structure hasher embed it.
- A hash inside a name (path component, column name, `uri`) is the bare hex digest:
`to_hex()`. Exception (developer, November 11, 2026): a source id is the source's name. A
default source id built from a hash keeps a source-kind prefix and the hash's label
(`derived:<label>:<hex>`, via `to_string()`, never a label literal) and is used as it is in
the `source_id` system tag and in source-info values.
- A hash stored as a value in its own column or field carries its label: `to_string()` as
text, `to_prefixed_digest()` in a binary column.
- A record id is an identifier built by a recipe, not a hash, and carries no label.
- `tests/test_hashing/test_hash_label_rule.py` recomputes every golden value under renamed
labels; a value that moves there means a label reached a hash input.

## Testing conventions

Tests use the **public interface** wherever possible (developer, November 10, 2026): drive an
Expand Down
Loading
Loading