Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
4f907c4
docs(plan): ITL-942 step 1 analysis of the wrong-hash findings
Nov 12, 2026
5acdf37
test(hashing): failing regression tests for the ITL-942 wrong-hash fi…
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
69f4b98
Merge remote-tracking branch 'origin/eywalker/itl-910-adopt-starfix-0…
Nov 12, 2026
c30f04c
docs(plan): ITL-942 finding R (fixed by #292) and G re-check
Nov 12, 2026
007ea85
docs(plan): ITL-935 logical type identifiers, step 1 plan
Nov 12, 2026
7bfcb54
docs(plan): ITL-944 step 1 plan for data context v1
Nov 12, 2026
26dcbc5
feat(contexts): data context std:v1:default with a frozen-manifest ch…
Nov 12, 2026
87df847
wip(store): __data_context column, new table layouts, read filter, ol…
Nov 12, 2026
02db3f0
wip(store): function version column, tests moved to the new layouts (…
Nov 12, 2026
5c34afb
docs(plan): ITL-945 step 1 plan for the hash label rule
Nov 12, 2026
4030f55
docs(plan): ITL-945 tracer results and open points
Nov 12, 2026
9b1c2be
feat(store): __data_context on every stored row, new table layouts, o…
Nov 12, 2026
2dc0d24
test(hashing): failing tests for the hash label rule (ITL-945)
Nov 12, 2026
8a1f5a1
refactor(hashing)!: rename hash labels to <algorithm>_v1 (ITL-945)
Nov 12, 2026
db94f0f
feat(hashing)!: keep hash labels out of every hash input (ITL-945)
Nov 12, 2026
2e619aa
feat(contexts): manifest fields of the versioning design; logical typ…
Nov 12, 2026
2c4b1e3
feat(core)!: bare hex digests inside names; stream identity without t…
Nov 12, 2026
a7691dc
feat(store): function pods that take the invocation context fail on a…
Nov 12, 2026
4b9e825
feat(store)!: labelled row keys for operator-cache and cached-source …
Nov 12, 2026
de11a36
test(golden): pinned hash values after the hash label rule (ITL-945)
Nov 12, 2026
bc012b3
refactor(hashing): define each hash label once, beside its algorithm …
Nov 12, 2026
07ccf30
refactor(store)!: remove the v0.1 manifest and the v0 to v1 migration…
Nov 12, 2026
50d453e
docs: the hash label rule in the hashing reference, CHANGELOG, DESIGN…
Nov 12, 2026
18a110b
feat(store): refuse an input of another data context (ITL-944)
Nov 12, 2026
f7a7c87
fix(hashing): sort set items by canonical JSON, not str() (ITL-942 L1)
Nov 12, 2026
45787de
fix(sources): record-id system tag is large_binary everywhere (ITL-94…
Nov 12, 2026
58198d8
fix(function_pod): default function version is v0.0 for every entry p…
Nov 12, 2026
ed81f34
fix(hashing): Arrow hashed type depends on the declared type only (IT…
Nov 12, 2026
ca6fb1c
fix(operators): fold system-tag source ids over the canonical member …
Nov 12, 2026
8b05071
fix(store): address CodeRabbit review of PR #313 (ITL-944)
Nov 12, 2026
53577ff
fix(operators): column selection identity is the set of names (ITL-94…
Nov 12, 2026
55ba591
fix(streams): keep tag sorting in the producer-less stream identity; …
Nov 12, 2026
d54020d
fix(hashing): dict keys keep their type in structure hashes (ITL-942 …
Nov 12, 2026
a690970
test(hashing): pin MergeJoin input identity as intended; document the…
Nov 12, 2026
f3157b0
fix(hashing): only the hasher and type handlers write '__type__' obje…
Nov 12, 2026
a6adec1
docs: a v0.2 side-effect log is not detected at the v0.3 path (ITL-94…
Nov 12, 2026
088502a
test(hashing): regenerate golden values for the ITL-942 fixes
Nov 12, 2026
749778d
docs: CHANGELOG and DESIGN_ISSUES entries for the ITL-942 wrong-hash …
Nov 12, 2026
603e2c3
feat(logical_types): identifiers for built-in types; whole identifier…
Nov 12, 2026
a63896b
style: type the Encoded expansion and the side-effect version test (I…
Nov 12, 2026
4c0a820
docs: logical type identifiers page, changelog, design issue, agent r…
Nov 12, 2026
7b6626b
docs(plan): ITL-954 plan and the store fixture written by Orcapod v0.2.0
Nov 12, 2026
0111b39
feat(databases): list_table_paths lists the tables under a path prefi…
Nov 12, 2026
7af5cd7
docs(changelog): a fresh store does not prevent repeated side effects…
Nov 12, 2026
03b4be2
test(store): pre-v0.3 tables on the real v0.2.0 store, as written and…
Nov 12, 2026
531ab89
fix(hashing): a namedtuple's field order is part of its hash (ITL-942 T)
Nov 12, 2026
930e94b
feat(store): find tables written before v0.3 under a node's name, wha…
Nov 12, 2026
81ccd84
refactor(operators): delete the dead duplicate SelectDataColumns in f…
Nov 12, 2026
654ae9f
docs: record ITL-942 T in CHANGELOG and DESIGN_ISSUES
Nov 12, 2026
40a7102
fix(sources): default derived source ids keep the hash label (ITL-945)
Nov 12, 2026
5bf7935
docs(store): pre-v0.3 tables are found by name; list_table_paths (ITL…
Nov 12, 2026
43f88cc
test(hashing): address CodeRabbit round 1 on PR #315 (ITL-942)
Nov 12, 2026
6addbf0
fix(job): a node without a name path claims no table in the job total…
Nov 12, 2026
b8990fa
feat(logical_types): wrapper identifiers in the schema hash; golden r…
Nov 12, 2026
3a6a075
feat(logical_types): native containers carry no version; fix []/[None…
Nov 12, 2026
ed2653e
docs(plan): ITL-935 decisions and outcome
Nov 12, 2026
a853811
docs(plan): ITL-942 worker report
Nov 12, 2026
8b291c9
docs(changelog): move the ITL-954 entry to a changelog fragment (ITL-…
Nov 12, 2026
c4490d5
docs(changelog): move the ITL-910 entry to a changelog fragment (ITL-…
Nov 12, 2026
a0b3825
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
13111d5
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
2d2c02f
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
3192a8a
docs(changelog): move the ITL-945 entry to a changelog fragment (ITL-…
Nov 12, 2026
6fb74c2
docs(changelog): move the ITL-942 entry to a changelog fragment (ITL-…
Nov 12, 2026
2369d75
docs(changelog): move the ITL-935 entry to a changelog fragment (ITL-…
Nov 12, 2026
81da973
docs(changelog): move the ITL-944 entry to a changelog fragment (ITL-…
Nov 12, 2026
c19cb9f
docs(plans): ITL-967 combined hash-step merge plan
Nov 12, 2026
f2c1fd8
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
07e9bd3
Merge branch 'eywalker/itl-944-data-context-v1-frozen-manifest' (#313…
Nov 12, 2026
c1949b2
Merge remote-tracking branch 'origin/eywalker/itl-954-find-tables-wri…
Nov 12, 2026
c8f1194
Merge branch 'eywalker/itl-945-apply-the-hash-label-rule-one-format-l…
Nov 12, 2026
128b2c5
Merge branch 'eywalker/itl-942-fix-the-wrong-hash-findings-of-the-has…
Nov 12, 2026
6ba6cc3
Merge remote-tracking branch 'origin/eywalker/itl-935-design-make-the…
Nov 12, 2026
b30b6fa
fix(sources): name every column the Arrow hasher refuses, as ValueErr…
Nov 12, 2026
b14a56c
fix(review): CodeRabbit findings on the ITL-910 code, on the combined…
Nov 12, 2026
8c2066a
test(hashing): regenerate the golden hashes once for the v0.3 hash st…
Nov 12, 2026
fdc6362
test(pipeline): type the values read in the combined-branch tests (IT…
Nov 12, 2026
dad1111
docs(plans): ITL-967 list of golden entries that move relative to main
Nov 12, 2026
970ccf0
Merge remote-tracking branch 'origin/main' (#316, ITL-955) into ITL-967
Nov 12, 2026
444bd35
test(hashing): regenerate the golden hashes after merging main (ITL-9…
Nov 12, 2026
c4b1e4a
docs(design-issues): move the ITL-910, ITL-944, ITL-954, ITL-945, ITL…
Nov 12, 2026
617efe9
docs(plans): ITL-967 progress notes
Nov 12, 2026
3c9c583
Merge remote-tracking branch 'origin/main' (#317, ITL-939) into ITL-967
Nov 12, 2026
9743004
test(hashing): regenerate the golden hashes after merging main (ITL-9…
Nov 12, 2026
870852f
fix(hashing): a value-hashed logical type keeps its identifier with o…
Nov 12, 2026
fe4c62a
wip(review): identifier checks on typed columns; Delta listing symlin…
Nov 12, 2026
19d9624
wip(review): wrapper data context; async operator context check (ITL-…
Nov 12, 2026
29a4e39
wip(review): side-effecting function nodes check before every run; cu…
Nov 12, 2026
2b45dd1
wip(review): excluded nodes in the job total; record writer checks th…
Nov 12, 2026
c0f9e0a
wip(review): documentation corrections; changelog and design-issue en…
Nov 12, 2026
9e6bc31
wip(review): docstrings on new definitions; test-strength changes; re…
Nov 12, 2026
cb54a95
Merge remote-tracking branch 'origin/main' (#321, #310, #311) into IT…
Nov 12, 2026
360fc7b
test(hashing): regenerate the golden hashes for the logical type iden…
Nov 12, 2026
a0a3aea
fix(review): type-check the round's changes; golden entry list agains…
Nov 12, 2026
6dc2a51
fix(review): CodeRabbit round 2 on the hash step (ITL-967)
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 104 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,30 @@ commands (pytest on Python 3.11, 3.12 and 3.13; the others on 3.11):
- The Graphviz `dot` binary is optional. The tests that render a graph to an image skip
when `dot` is not on `PATH` and run when it is.

## Logical type identifiers

A logical type's identifier is its Arrow extension name plus its Arrow extension metadata
(developer, November 11, 2026, ITL-935/ITL-938); see `docs/concepts/logical-type-identifiers.md`.

- The identifier is declared in one place, the extension type returned by
`get_arrow_extension_type()`, and read through `logical_types/identifier.py`
(`logical_type_identifier`, `LogicalTypeIdentifier.text`). Never put a logical type's bare
name into a hash input: use the identifier text.
- Change a type's identifier when, and only when, an existing value would be stored or hashed
differently. It is not a package version. Built-in types use `version_metadata(N)`
(`{"version":N}`); a new built-in type must write one, never empty metadata.
- A changed identifier is a schema change: it moves stored paths and golden values of that
type. Treat it like any other hash change (developer approval, migration assessment).
- The native containers `list[T]` / `set[T]` of a logical type (`ListLogicalType`) have no
version of their own, by design: never add a `"version"` key to their metadata. Their
identifier is the container kind plus the element's identifier; the type hash treats them
like any generic alias (origin and arguments). A change to how they are stored or hashed is
a data context (global hash version) change, not a logical type change.
- On read, stored metadata must be one of the type's `readable_extension_metadata()`;
otherwise raise `LogicalTypeIdentifierError`. Never wrap a stored column by name alone and
never fall back to the storage type. List an older value in `also_reads` only when the
current code decodes that layout.

## Golden hash values

Hash values must stay identical across releases (developer, November 10, 2026). Every hash
Expand All @@ -263,6 +287,34 @@ checked by `tests/test_hashing/golden/test_golden_hashes.py`.
- Never move or rename the fixtures in `tests/test_hashing/golden/cases.py`: function pods,
dataclasses and pydantic models hash with their module path.

## Hash labels

One rule holds for every hash label (developer, November 11, 2026; ITL-945; stated with
examples in `docs/reference/hashing.md`, "The hash label rule").

- A label names the algorithm that produced a digest (the whole procedure: canonical form,
preparation, encoding, digest function) and nothing else. Format `<algorithm>_v<N>`,
integer `N`: `semantic_v1`, `arrow_v1`, `directory_v1`, `ndarray_v1`, `polars_expr_v1`. The
file content hash keeps `sha256`, the only exception. A logical type's implementer picks
its label.
- Change a label only when its algorithm would give an existing input a different digest.
- Define each label in exactly one place, as a constant beside its algorithm. Never write a
label literal anywhere else in code, and never in a data context file.
- Never feed a label into a hash computation. Feed a nested hash's digest only: never put
`to_string()`, `str(hash)` or `to_prefixed_digest()` into an identity structure, a preimage
table or a digest input. Inside a structure, return the `ContentHash` object and let the
structure hasher embed it.
- A hash inside a name (path component, column name, `uri`) is the bare hex digest:
`to_hex()`. Exception (developer, November 11, 2026): a source id is the source's name. A
default source id built from a hash keeps a source-kind prefix and the hash's label
(`derived:<label>:<hex>`, via `to_string()`, never a label literal) and is used as it is in
the `source_id` system tag and in source-info values.
- A hash stored as a value in its own column or field carries its label: `to_string()` as
text, `to_prefixed_digest()` in a binary column.
- A record id is an identifier built by a recipe, not a hash, and carries no label.
- `tests/test_hashing/test_hash_label_rule.py` recomputes every golden value under renamed
labels; a value that moves there means a label reached a hash input.

## Testing conventions

Tests use the **public interface** wherever possible (developer, November 10, 2026): drive an
Expand Down Expand Up @@ -449,6 +501,57 @@ hashes, forming a Merkle chain.

Prefixes are computed from SystemConstant in system_constants.py.

Stored tables: data context and layout versions (developer, November 11, 2026, ITL-944)

Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, large_string):
the key of the data context that wrote it (std:v1:default since v0.3). The key names one
generation of hashing; rows of the current data context are the current entries.

Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION)
Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION)
Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION)
Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION)
Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION)

- The layout version is the last component of a table path. A change to the bookkeeping
columns of a table kind moves its layout version, so the table gets a new path. Never add,
rename or retype a stored column without it.
- Every new stored table kind writes __data_context (contexts/stored.with_data_context), and
every read or lookup keeps rows of the current data context only
(keep_rows_of_data_context, or a lookup constraint on the column).
- A table at an older layout is never read as current and never modified. What a node does
about such tables, and about rows of another data context, is decided in one place,
contexts/stored.account_for_stale_tables: count and report, or, for a node that delivers
side effects (a SideEffectPod node, or a function node whose pod takes the invocation
context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of
node goes through that function; never decide it locally.
- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between versions. find_stale_tables(..., name_path=...) lists the
store under the hash-free beginning of the node's path (ArrowDatabaseProtocol.list_table_paths;
<function>, side_effect_function/<function>, side_effects/<pod>, relative to the node's
database) and every table there whose last component is not in
contexts/stored.CURRENT_LAYOUTS is pre-v0.3. A new layout constant must be added to
CURRENT_LAYOUTS, and a superseded one to the OLD_*_SCHEMA_VERSIONS tuples. A store that
cannot list raises TableListingNotSupportedError: side-effecting nodes then warn and
proceed (developer, November 11, 2026). Operator caches and cached sources have no stable
name; PipelineJob.run() reports one total for them. Every database class implements
list_table_paths.
- A released data context manifest (contexts/data/<version>.json) is frozen: its sha256 is
pinned in contexts/released_manifests.json. A change that moves an existing golden hash
value for a reason other than a logical type's own change needs a new data context version
with its own manifest file, never an edit of a released one.
- In memory the data context column is still _context_key (ColumnConfig context).
- A writer refuses an input of another data context than its own
(contexts/stored.require_same_data_context, InputValidationError naming both keys) before
anything is written; a whole-input condition is checked outside the row error policy. The
eventual behaviour is ITL-951.
- std:v0.1:default is not shipped; saved files that name it load through
contexts.resolve_saved_context. There is no SchemaVersionError, ignore_schema or
orcapod.migrations: do not reintroduce a per-layout migration path without a decision.
- Logical types and the value handlers that hash their Python values are registered in
contexts/logical_types.json, never in a data context manifest: a logical type is
versioned by its own identifier, and that file is not frozen.

### System tag evolution rules

1. Name-preserving — single-stream ops. Column name/value pass through unchanged.
Expand All @@ -463,7 +566,7 @@ Prefixes are computed from SystemConstant in system_constants.py.
directly to derive record identity. They fold to a deterministic digest via
arrow_utils.fold_system_tag_values and their column name gains ::{pipeline_hash}.

The fold is SHA-based (uuid5 for record_id, combine_hashes for source_id) and so is stable
The fold is SHA-based (uuid5 for record_id, SHA-256 of the canonical JSON member list for source_id) and so is stable
across processes. Never use hash() or a set-based construction there: the digest becomes a
cache key, so a per-process value would look correct in a single-process test and miss the
cache on every new driver run.
Expand Down
15 changes: 9 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@
#### NumPy arrays: new storage format and new hashes, no automatic migration (ITL-888, ITL-883)

**The array format and every array hash changed in v0.3.** A store written by v0.2 that holds
`numpy.ndarray` columns is **not migrated**. Under v0.3, every node that handles arrays
computes again into new tables, and its v0.2 results are no longer returned. Nothing written by
v0.2 is changed or deleted, and those tables stay readable with v0.2. Use a fresh store for
v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.3 (ITL-889).
`numpy.ndarray` columns is **not migrated**. Every node that handles arrays gets new tables,
so its v0.2 results are no longer found; and since every Arrow-derived hash changed too
(ITL-910, below), no v0.2 result is reused at all: v0.3 ignores the rows a v0.2 store holds and
computes every input again. Nothing written by v0.2 is changed or deleted, and those tables stay
readable with v0.2. A migration is planned after v0.3 (ITL-889).

- **Storage.** An array column is now the Arrow extension type `orcapod.ndarray` (metadata
`{"version":1}`) over `struct<dtype: large_string, shape: large_list<int64>,
Expand All @@ -29,8 +30,10 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0.
change too, and so do the table paths of those nodes. Pipeline hashes are chained, so the
pipeline and operator tables of every node downstream of an array column move as well,
even when it has no array column itself. A pod whose inputs and outputs hold no array
keeps its function result table, so it reuses its v0.2 results without computing again.
Nodes with no array column upstream do not move.
keeps its function result table, and nodes with no array column upstream keep their tables;
but an array-free pod does **not** reuse its v0.2 results: every Arrow-derived hash changed
as well (ITL-910, below) and every table moved to a new layout (ITL-944, below), so the
v0.2 tables are not read and the inputs are computed again.
- **Refused at write time** with `orcapod.errors.UnsupportedValueError` (a `ValueError`), naming
the column and dtype: masked arrays (the mask used to be dropped silently), object dtypes,
NumPy 2 `StringDType` (used to fail with a misleading pickle error), `longdouble` and
Expand Down
Loading
Loading