Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
170 commits
Select commit Hold shift + click to select a range
272054f
docs(specs): v0.3 identity of a user class, implementation spec (ITL-…
Nov 11, 2026
fae9ba7
docs(specs): record P1, registration order and cycle rules for user c…
Nov 11, 2026
394f60f
docs(plans): implementation plan for v0.3 user class identity (ITL-881)
Nov 11, 2026
27d4a9e
feat(types): identity record, string and naming policy for user class…
Nov 11, 2026
5f12c0c
feat(types): identify dataclasses and pydantic models by name, kind a…
Nov 11, 2026
b4104a4
feat(types): register_type API, rebinding and read-side identity chec…
Nov 11, 2026
a7d0044
test(hashing): golden values for v0.3 user class identity (ITL-881)
Nov 11, 2026
285c884
feat(types): full-length identity hash, per-class qualified option, n…
Nov 11, 2026
ac55871
docs: your own data types page, identity section, CHANGELOG and spec …
Nov 11, 2026
532a574
test(docs): register each doc block's source in linecache (ITL-881)
Nov 11, 2026
fabed37
refactor(types): rename register_type's qualified argument to qualifi…
Nov 11, 2026
b0ab6b6
fix(types): review fixes for user class identity reads (ITL-881)
Nov 11, 2026
1aae9d1
docs(changelog): Delta compaction fix for stored user-class values (I…
Nov 11, 2026
b4dadf3
test(types): pin IPC read of non-large nested storage as a known limi…
Nov 11, 2026
5707fb7
docs(types): docstrings for user-class identity helpers (ITL-881)
Nov 11, 2026
5b9693d
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
Nov 11, 2026
39951f0
docs(changelog): note the IPC limitation tracked as ITL-894 (ITL-881)
Nov 11, 2026
b930f62
test(types): docstring and value checks for the ITL-894 xfail (ITL-881)
Nov 11, 2026
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
4f907c4
docs(plan): ITL-942 step 1 analysis of the wrong-hash findings
Nov 12, 2026
5acdf37
test(hashing): failing regression tests for the ITL-942 wrong-hash fi…
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
69f4b98
Merge remote-tracking branch 'origin/eywalker/itl-910-adopt-starfix-0…
Nov 12, 2026
c30f04c
docs(plan): ITL-942 finding R (fixed by #292) and G re-check
Nov 12, 2026
ac71929
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
Nov 12, 2026
e3e8440
docs(plans): plan for field order as identity (ITL-881)
Nov 12, 2026
b54b031
feat(types)!: declared field order is part of a user class's identity…
Nov 12, 2026
d5357eb
test(hashing): golden values for field order as identity (ITL-881)
Nov 12, 2026
ac48ce2
fix(types): testing a read candidate no longer registers its nested c…
Nov 12, 2026
5b26f92
docs(types): declared field order is identity; known boundary for def…
Nov 12, 2026
fe41fa0
test(types): type-safe struct field access in field-order tests (ITL-…
Nov 12, 2026
007ea85
docs(plan): ITL-935 logical type identifiers, step 1 plan
Nov 12, 2026
887b312
fix(pipeline): a stored row that cannot be decoded as its class fails…
Nov 12, 2026
22b4b1e
fix(types): optional nested class None reads back; stored record chec…
Nov 12, 2026
25346ab
test(types): distinct value per field in workshop scenarios; assert d…
Nov 12, 2026
7bfcb54
docs(plan): ITL-944 step 1 plan for data context v1
Nov 12, 2026
e1bd700
test(types): docstrings for the user-type tests; changelog for the re…
Nov 12, 2026
26dcbc5
feat(contexts): data context std:v1:default with a frozen-manifest ch…
Nov 12, 2026
87df847
wip(store): __data_context column, new table layouts, read filter, ol…
Nov 12, 2026
02db3f0
wip(store): function version column, tests moved to the new layouts (…
Nov 12, 2026
5c34afb
docs(plan): ITL-945 step 1 plan for the hash label rule
Nov 12, 2026
4030f55
docs(plan): ITL-945 tracer results and open points
Nov 12, 2026
9b1c2be
feat(store): __data_context on every stored row, new table layouts, o…
Nov 12, 2026
2dc0d24
test(hashing): failing tests for the hash label rule (ITL-945)
Nov 12, 2026
8a1f5a1
refactor(hashing)!: rename hash labels to <algorithm>_v1 (ITL-945)
Nov 12, 2026
db94f0f
feat(hashing)!: keep hash labels out of every hash input (ITL-945)
Nov 12, 2026
38c583a
fix(pipeline): failing observer callbacks do not replace an identity …
Nov 12, 2026
2e619aa
feat(contexts): manifest fields of the versioning design; logical typ…
Nov 12, 2026
2c4b1e3
feat(core)!: bare hex digests inside names; stream identity without t…
Nov 12, 2026
a7691dc
feat(store): function pods that take the invocation context fail on a…
Nov 12, 2026
4b9e825
feat(store)!: labelled row keys for operator-cache and cached-source …
Nov 12, 2026
de11a36
test(golden): pinned hash values after the hash label rule (ITL-945)
Nov 12, 2026
bc012b3
refactor(hashing): define each hash label once, beside its algorithm …
Nov 12, 2026
07ccf30
refactor(store)!: remove the v0.1 manifest and the v0 to v1 migration…
Nov 12, 2026
50d453e
docs: the hash label rule in the hashing reference, CHANGELOG, DESIGN…
Nov 12, 2026
18a110b
feat(store): refuse an input of another data context (ITL-944)
Nov 12, 2026
f7a7c87
fix(hashing): sort set items by canonical JSON, not str() (ITL-942 L1)
Nov 12, 2026
45787de
fix(sources): record-id system tag is large_binary everywhere (ITL-94…
Nov 12, 2026
58198d8
fix(function_pod): default function version is v0.0 for every entry p…
Nov 12, 2026
ed81f34
fix(hashing): Arrow hashed type depends on the declared type only (IT…
Nov 12, 2026
ca6fb1c
fix(operators): fold system-tag source ids over the canonical member …
Nov 12, 2026
8b05071
fix(store): address CodeRabbit review of PR #313 (ITL-944)
Nov 12, 2026
53577ff
fix(operators): column selection identity is the set of names (ITL-94…
Nov 12, 2026
55ba591
fix(streams): keep tag sorting in the producer-less stream identity; …
Nov 12, 2026
d54020d
fix(hashing): dict keys keep their type in structure hashes (ITL-942 …
Nov 12, 2026
a690970
test(hashing): pin MergeJoin input identity as intended; document the…
Nov 12, 2026
f3157b0
fix(hashing): only the hasher and type handlers write '__type__' obje…
Nov 12, 2026
a6adec1
docs: a v0.2 side-effect log is not detected at the v0.3 path (ITL-94…
Nov 12, 2026
088502a
test(hashing): regenerate golden values for the ITL-942 fixes
Nov 12, 2026
749778d
docs: CHANGELOG and DESIGN_ISSUES entries for the ITL-942 wrong-hash …
Nov 12, 2026
603e2c3
feat(logical_types): identifiers for built-in types; whole identifier…
Nov 12, 2026
a63896b
style: type the Encoded expansion and the side-effect version test (I…
Nov 12, 2026
4c0a820
docs: logical type identifiers page, changelog, design issue, agent r…
Nov 12, 2026
7b6626b
docs(plan): ITL-954 plan and the store fixture written by Orcapod v0.2.0
Nov 12, 2026
0111b39
feat(databases): list_table_paths lists the tables under a path prefi…
Nov 12, 2026
7af5cd7
docs(changelog): a fresh store does not prevent repeated side effects…
Nov 12, 2026
03b4be2
test(store): pre-v0.3 tables on the real v0.2.0 store, as written and…
Nov 12, 2026
531ab89
fix(hashing): a namedtuple's field order is part of its hash (ITL-942 T)
Nov 12, 2026
930e94b
feat(store): find tables written before v0.3 under a node's name, wha…
Nov 12, 2026
81ccd84
refactor(operators): delete the dead duplicate SelectDataColumns in f…
Nov 12, 2026
654ae9f
docs: record ITL-942 T in CHANGELOG and DESIGN_ISSUES
Nov 12, 2026
40a7102
fix(sources): default derived source ids keep the hash label (ITL-945)
Nov 12, 2026
5bf7935
docs(store): pre-v0.3 tables are found by name; list_table_paths (ITL…
Nov 12, 2026
43f88cc
test(hashing): address CodeRabbit round 1 on PR #315 (ITL-942)
Nov 12, 2026
6addbf0
fix(job): a node without a name path claims no table in the job total…
Nov 12, 2026
b8990fa
feat(logical_types): wrapper identifiers in the schema hash; golden r…
Nov 12, 2026
3a6a075
feat(logical_types): native containers carry no version; fix []/[None…
Nov 12, 2026
ed2653e
docs(plan): ITL-935 decisions and outcome
Nov 12, 2026
a853811
docs(plan): ITL-942 worker report
Nov 12, 2026
8b291c9
docs(changelog): move the ITL-954 entry to a changelog fragment (ITL-…
Nov 12, 2026
c4490d5
docs(changelog): move the ITL-910 entry to a changelog fragment (ITL-…
Nov 12, 2026
a0b3825
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
13111d5
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
2d2c02f
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
e18f05d
docs(changelog): move the ITL-881 entry to a changelog fragment (ITL-…
Nov 12, 2026
3192a8a
docs(changelog): move the ITL-945 entry to a changelog fragment (ITL-…
Nov 12, 2026
6fb74c2
docs(changelog): move the ITL-942 entry to a changelog fragment (ITL-…
Nov 12, 2026
2369d75
docs(changelog): move the ITL-935 entry to a changelog fragment (ITL-…
Nov 12, 2026
81da973
docs(changelog): move the ITL-944 entry to a changelog fragment (ITL-…
Nov 12, 2026
c19cb9f
docs(plans): ITL-967 combined hash-step merge plan
Nov 12, 2026
f2c1fd8
Merge branch 'eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orca…
Nov 12, 2026
07e9bd3
Merge branch 'eywalker/itl-944-data-context-v1-frozen-manifest' (#313…
Nov 12, 2026
c1949b2
Merge remote-tracking branch 'origin/eywalker/itl-954-find-tables-wri…
Nov 12, 2026
b1646ab
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
Nov 12, 2026
e99a0c2
docs(design-issues): move the ITL-881 entries to design_issues/ files…
Nov 12, 2026
c8f1194
Merge branch 'eywalker/itl-945-apply-the-hash-label-rule-one-format-l…
Nov 12, 2026
128b2c5
Merge branch 'eywalker/itl-942-fix-the-wrong-hash-findings-of-the-has…
Nov 12, 2026
6ba6cc3
Merge remote-tracking branch 'origin/eywalker/itl-935-design-make-the…
Nov 12, 2026
b30b6fa
fix(sources): name every column the Arrow hasher refuses, as ValueErr…
Nov 12, 2026
b14a56c
fix(review): CodeRabbit findings on the ITL-910 code, on the combined…
Nov 12, 2026
c6837a1
fix(types): review round on user class identity: rebound containers, …
Nov 12, 2026
8c2066a
test(hashing): regenerate the golden hashes once for the v0.3 hash st…
Nov 12, 2026
fdc6362
test(pipeline): type the values read in the combined-branch tests (IT…
Nov 12, 2026
dad1111
docs(plans): ITL-967 list of golden entries that move relative to main
Nov 12, 2026
970ccf0
Merge remote-tracking branch 'origin/main' (#316, ITL-955) into ITL-967
Nov 12, 2026
444bd35
test(hashing): regenerate the golden hashes after merging main (ITL-9…
Nov 12, 2026
c4b1e4a
docs(design-issues): move the ITL-910, ITL-944, ITL-954, ITL-945, ITL…
Nov 12, 2026
617efe9
docs(plans): ITL-967 progress notes
Nov 12, 2026
0856e9d
docs(specs): design and plan for the two kinds of user class (ITL-972)
Nov 12, 2026
89b4c61
Merge branch 'eywalker/itl-881-user-class-identity-impl' (PR #292) in…
Nov 12, 2026
f928de5
test(user_types): failing tests for exact classes and the measured ca…
Nov 12, 2026
4adfdc9
feat(types): exact classes and orcapod.dataclass, first cut (ITL-972)
Nov 12, 2026
3c9c583
Merge remote-tracking branch 'origin/main' (#317, ITL-939) into ITL-967
Nov 12, 2026
9743004
test(hashing): regenerate the golden hashes after merging main (ITL-9…
Nov 12, 2026
85a1ff6
feat(types): no automatic conversion, structural equality, tests adap…
Nov 12, 2026
355e2d3
docs: two kinds of user class; changelog fragment and design issue (I…
Nov 12, 2026
85206fa
Merge branch 'eywalker/itl-967-v03-hash-step-combine-the-hash-changin…
Nov 12, 2026
870852f
fix(hashing): a value-hashed logical type keeps its identifier with o…
Nov 12, 2026
fe4c62a
wip(review): identifier checks on typed columns; Delta listing symlin…
Nov 12, 2026
61e534a
refactor(types): structural class helpers in their own module; golden…
Nov 12, 2026
66e4b58
Merge remote-tracking branch 'origin/main' (#321, #310) into ITL-972
Nov 12, 2026
19d9624
wip(review): wrapper data context; async operator context check (ITL-…
Nov 12, 2026
e9cb1f0
Merge remote-tracking branch 'origin/main' (#311, ITL-943) into ITL-972
Nov 12, 2026
29a4e39
wip(review): side-effecting function nodes check before every run; cu…
Nov 12, 2026
85acbc8
test(hashing): regenerate the golden hashes for the two kinds of user…
Nov 12, 2026
2b45dd1
wip(review): excluded nodes in the job total; record writer checks th…
Nov 12, 2026
c0f9e0a
wip(review): documentation corrections; changelog and design-issue en…
Nov 12, 2026
9e6bc31
wip(review): docstrings on new definitions; test-strength changes; re…
Nov 12, 2026
cb54a95
Merge remote-tracking branch 'origin/main' (#321, #310, #311) into IT…
Nov 12, 2026
8bcf15e
fix(function_pod): convert in-memory pod output by its declared types…
Nov 12, 2026
360fc7b
test(hashing): regenerate the golden hashes for the logical type iden…
Nov 12, 2026
bcc25c9
test(user_types): pin subclass return behaviour; docs state how to re…
Nov 12, 2026
a0a3aea
fix(review): type-check the round's changes; golden entry list agains…
Nov 12, 2026
d940ee3
Merge branch 'eywalker/itl-967-v03-hash-step-combine-the-hash-changin…
Nov 12, 2026
7ba9c29
docs(specs): golden comparison against the ITL-967 tip a0a3aeac (ITL-…
Nov 12, 2026
6dc2a51
fix(review): CodeRabbit round 2 on the hash step (ITL-967)
Nov 12, 2026
3ba1525
fix(types): review round 1: variadic tuple token, union member in fro…
Nov 12, 2026
129903f
Merge branch 'eywalker/itl-967-v03-hash-step-combine-the-hash-changin…
Nov 12, 2026
5203848
fix(types): review round 2: union member by field types; one equality…
Nov 12, 2026
4f6e6e0
fix(types): review round 3: from_instance raises for a union value no…
Nov 12, 2026
b58a34c
chore: merge main (hash step #324, squashed) into ITL-972
Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,37 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
never fall back to the storage type. List an older value in `also_reads` only when the
current code decodes that layout.

## User classes: exact and structural

A class a user defines is one of two kinds (developer, November 12, 2026, ITL-972; see
`docs/concepts/your-own-data-types.md`).

- An ordinary `@dataclass` or a pydantic model is an **exact** class: named by module path
and qualified name, and one type only with itself. Never make two different exact classes
equivalent (no declared names, no equivalence by description). Python subclassing still
applies at a pod input: a subclass column is accepted at a base-class parameter, and the
subclass keeps its own identity and fields.
- A class made with `orcapod.dataclass` is **structural**: identified by its name (default
the class name) and its fields in declared order. Two structural classes with equal
identity records are compatible in the schema check and their instances compare equal.
Equality and hashing of a structural class cover exactly its stored fields; never let a
per-field `compare=` or `hash=` choice through, because a class rebuilt from metadata
cannot know it. Pydantic has no structural kind (ITL-973).
- **No automatic conversion.** Orcapod checks compatibility and passes a value as it is. A
function annotated with a structural class may receive another class of the same
identity; a structural class promises its fields, not its methods. `from_instance` is the
only conversion, and Orcapod never calls it.
- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never
store a subclass instance or a look-alike object in another class's column.
- The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}`
with JSON type tokens. The Arrow extension name is `name@<SHA-256 of the record>` and the
record is the extension metadata; together they are the logical type identifier. A nested
class embeds its whole record and a logical-type field the whole identifier text, so a
structural class can be rebuilt from metadata alone. A change to the record or token
format moves every user-class identifier: treat it as a hash change.
- Arrow-backed data carries the identifier, not a class object: the registry binds an
identifier to the newest class object with that identity, and data is read as that class.

## Golden hash values

Hash values must stay identical across releases (developer, November 10, 2026). Every hash
Expand Down
31 changes: 31 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,37 @@ A logical type's identifier is its Arrow extension name plus its Arrow extension
never fall back to the storage type. List an older value in `also_reads` only when the
current code decodes that layout.

## User classes: exact and structural

A class a user defines is one of two kinds (developer, November 12, 2026, ITL-972; see
`docs/concepts/your-own-data-types.md`).

- An ordinary `@dataclass` or a pydantic model is an **exact** class: named by module path
and qualified name, and one type only with itself. Never make two different exact classes
equivalent (no declared names, no equivalence by description). Python subclassing still
applies at a pod input: a subclass column is accepted at a base-class parameter, and the
subclass keeps its own identity and fields.
- A class made with `orcapod.dataclass` is **structural**: identified by its name (default
the class name) and its fields in declared order. Two structural classes with equal
identity records are compatible in the schema check and their instances compare equal.
Equality and hashing of a structural class cover exactly its stored fields; never let a
per-field `compare=` or `hash=` choice through, because a class rebuilt from metadata
cannot know it. Pydantic has no structural kind (ITL-973).
- **No automatic conversion.** Orcapod checks compatibility and passes a value as it is. A
function annotated with a structural class may receive another class of the same
identity; a structural class promises its fields, not its methods. `from_instance` is the
only conversion, and Orcapod never calls it.
- A value is stored only as its own class (`UserClassLogicalType.accepts_instance`); never
store a subclass instance or a look-alike object in another class's column.
- The identity record (`logical_types/user_class_identity.py`) is `{kind, name, fields}`
with JSON type tokens. The Arrow extension name is `name@<SHA-256 of the record>` and the
record is the extension metadata; together they are the logical type identifier. A nested
class embeds its whole record and a logical-type field the whole identifier text, so a
structural class can be rebuilt from metadata alone. A change to the record or token
format moves every user-class identifier: treat it as a hash change.
- Arrow-backed data carries the identifier, not a class object: the registry binds an
identifier to the newest class object with that identity, and data is read as that class.

## Golden hash values

Hash values must stay identical across releases (developer, November 10, 2026). Every hash
Expand Down
98 changes: 98 additions & 0 deletions changelog.d/ITL-972.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
### Breaking changes and migration

#### Your own classes come in two kinds: exact and structural (ITL-972)

Orcapod 0.3 identifies a class you define by its kind, its name and its fields' names and
types, in declared order. The kind is chosen where the class is defined.

- **An ordinary `@dataclass` or a pydantic model is identified by its exact class.** Its
name is the module path and class name (`pkg.models.Point`; `__main__.Point` for a class
defined in a notebook or script). Two different classes are never the same type, even
with the same name and fields: a pod built for a class refuses the class defined again
(a re-run notebook cell), and the error names the column. A column of a subclass is
accepted where its base class is expected, as before.
- **A class made with the new `@op.dataclass` is structural** (see "Added"): identified by
its name and fields, whatever class object or module it is.
- **Hashes change for every pipeline that uses a dataclass or pydantic model.** The
class's logical type name and Arrow extension name become
`<name>@<SHA-256 of kind, name and fields>`, and the extension metadata holds that
description; the stored struct lists fields in declared order, as before. Every schema
hash, pipeline hash, table path, system-tag column name and data hash that involves such
a value moves. The first run under 0.3 computes those pod and operator results again,
into new tables. Tables written by earlier versions are left untouched and are not read
by 0.3 pipelines; reading such a column directly raises `TypeIdentityError`. There is no
migration tool. Results that involve no such value are unaffected.
- **Changing a class's fields makes a new type.** Adding, removing, renaming, retyping or
reordering fields all count. Defaults, methods and pydantic computed fields are not part
of the identity. The new type's pods get new tables; the results of the old definition
stay where they are. Before, a changed class shared the old tables and a run returned 0
rows with only log warnings.
- **A value is stored only as its own class.** Storing a value in a column of an ordinary
dataclass or pydantic model requires an instance of exactly that class; a subclass
instance or an unrelated object with the same attributes raises `TypeError`. Before,
any object with the right attributes was stored, and a subclass instance lost its own
fields. This holds for a pod's output read in memory as well: its columns are
converted as the pod's declared output types, where the types were taken from the
values before.
- **A stored shape that disagrees with the class raises.** Reading a stored column of an
ordinary dataclass or pydantic model finds the class at its module path. If the class
there has other fields, `DeltaTableDatabase` raises the new
`orcapod.errors.TypeIdentityError` naming the column and both sets of fields. It is not
a `StoreReadError`: the table is intact and the class differs. Like a store error, it
fails the run under every `error_policy`, with both orchestrators: a stored row that
cannot be decoded is never skipped as a failed row. If no class is found, reading raises
`ImportError` naming the column, the class and its stored fields.
- **A subclass is its own type.** A dataclass or pydantic subclass was stored as its
parent when the parent was registered first: its extra fields were silently dropped from
stored data, and the pod's hash and table path depended on the registration order. It
now has its own identity and keeps all its fields, in any registration order.
- **Recursive classes** raise `RecursiveTypeError` (a `TypeError`) naming the field path,
for example `Node.next -> Node`.

### Added

#### `op.dataclass` — a dataclass identified by its name and fields (ITL-972)

`@op.dataclass` is used like `dataclasses.dataclass` (bare, or with `name=` and the
dataclass options except `init=False`) and returns a real dataclass. Orcapod identifies it
by its name (default: the class name) and its fields, not by the class object or its
module:

- Two such classes with the same name and fields are one type. A class cell run again in
a notebook, or a class moved from a notebook into a module, keeps its pods and its
stored results.
- A pod annotated with one class accepts a column of another class of the same type and
receives the value as it is. Orcapod never converts a value from one class to another:
a structural class promises its fields, not its methods.
- Instances of two classes of the same type compare equal when their field values are
equal. Equality and hashing always cover all stored fields: `field(compare=False)` and
`field(hash=...)` are refused with `TypeError` where the class is defined.
- Stored values can be read in a process that does not define the class: Orcapod rebuilds
a dataclass from the description stored with the column.
- `P.from_instance(other)` builds an instance from an instance of any other dataclass with
exactly the same field names and types. It is the only conversion, and it is explicit.
- Every class inside an `op.dataclass` must be an `op.dataclass`.

See [Your Own Data Types](docs/concepts/your-own-data-types.md) for the two kinds and when
to use each.

Re-running a notebook cell that defines a dataclass or pydantic model, or reloading its
module, no longer raises `ValueError: … already bound`.

### Fixed

#### A nested class stored as `None` reads back (ITL-881)

A dataclass or pydantic model with an optional nested class (`child: Inner | None`) that
stored `None` raised `TypeError: 'NoneType' object is not subscriptable` when the value
was read back. It now reads back as `None`.

#### Stored dataclass and pydantic values stay readable after Delta compaction (ITL-881)

Delta compaction (`optimize.compact()`) rewrites a `large_string` struct field as
`string`. Reading a stored dataclass or pydantic column with a `str` or `bytes` field then
failed, and a cached run returned 0 rows. Extension columns now accept large and non-large
layouts of the same type at any nesting depth, as they already did at the top level
(ITL-602). Known limitation: such a non-large column read over Arrow IPC or the C stream
interface (not Parquet or Delta) gives invalid arrays, at any depth; Orcapod's own writers
never produce one (ITL-894).
57 changes: 57 additions & 0 deletions design_issues/ITL-972-UCK1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
## `src/orcapod/logical_types/` — identity of user classes (dataclasses, pydantic models)

### UCK1 — A user class was identified by `module.qualname` only; then every class by name, kind and fields
**Status:** resolved
**Severity:** high
**Issue:** ITL-972 (supersedes the model of ITL-881, PR #292)

In v0.2.0 a dataclass or pydantic model was identified by `f"{module}.{qualname}"`. Two
classes with the same name and different fields shared pipeline hashes and tables, so a
changed class read the old table, failed, and the run returned 0 rows with only log
warnings; re-running the notebook cell that defines a class raised
`ValueError: … already bound` until a kernel restart; a subclass was stored as its parent,
dropping its extra fields.

PR #292 (ITL-881, not merged) identified every class by a declared name, kind and ordered
fields, so two different classes with the same description were interchangeable. A
measurement on November 12, 2026 showed the consequence: in memory a pod annotated with one
class received the other class's object unchanged, while through a store it received its
own class.

**Fix:** two explicit kinds (developer, November 12, 2026; spec
`superpowers/specs/2026-11-12-itl-972-user-class-kinds.md`).

- An ordinary dataclass or pydantic model is an **exact** class: named by module path and
qualified name, never equivalent to another class. Python subclassing still applies at a
pod input. A value is stored only as its own class.
- `orcapod.dataclass` makes a **structural** class: named by its declared name (default the
class name); two structural classes with equal identity records are compatible in the
schema check and their instances compare equal. Orcapod converts nothing: a value is
passed as it is. `from_instance` is the explicit conversion.
- Every kind is described by an identity record `{kind, name, fields}` with structured
type tokens; a nested class embeds its whole record and a logical-type field the whole
logical type identifier. The identity string `name@<SHA-256 of the record>` is the logical
type name and Arrow extension name (Arrow's and Polars' registries are keyed by name),
and the record is the extension metadata, so a structural class is rebuilt from metadata
when no class is defined (`orcapod.synthesized`).
- Kept from PR #292: a subclass has its own identity; field order is identity;
`TypeIdentityError` is not a `StoreReadError` and fails the run under every error policy;
the registry binds an identifier to the newest class object with that identity.
Removed: `op.register_type`, declared names for exact classes, equivalence of exact
classes.

Arrow-backed data carries the identifier, not a class object: after an exact class is
defined again and used, tables built before are read as the new class.

Not covered: a structural pydantic model (ITL-973), computed fields, Enum/NamedTuple/
TypedDict.

**Known limitation (ITL-894):** reading a stored column whose storage differs from the
registered one only by layout (`string` vs `large_string`) is accepted at any depth, so
Delta-compacted tables stay readable. `__arrow_ext_deserialize__` then returns the
registered large storage type, which is correct for Parquet and Delta reads (the decoder
converts the data) but not over Arrow IPC or the C stream interface, where Arrow reads
32-bit offsets as 64-bit and the arrays are invalid. Top-level columns had this since
ITL-602; nested struct fields since this fix. Orcapod's own writers and Polars round trips
never produce non-large storage. Pinned by a strict xfail in
`tests/test_user_types/test_read_path.py`.
2 changes: 2 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,3 +453,5 @@ primary user-facing APIs.
- [Operators](operators.md) handle structural transforms (the complement of function pods)
- Function pods participate in the [identity chain](identity.md) -- each pod's hash includes
the function's identity and its upstream pipeline hashes
- Function pods can take and return dataclasses and pydantic models; see
[Your Own Data Types](your-own-data-types.md) for when two classes count as the same type
25 changes: 25 additions & 0 deletions docs/concepts/identity.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,29 @@ print(step.pod(source).pipeline_hash() == original)
When you change what a function computes, bump its major version (or rename it) so that
stale results are not reused.

## Your own classes in the identity

A class enters both chains through its **identity**: its kind, its name and the names and
types of its fields, in declared order. The identity is part of every schema hash, so it
reaches the pipeline hash of each source, pod and node that uses the class, and with it
the database table paths. Each stored value is hashed together with it, so it also
reaches the content hash.

The name depends on the kind of class:

- A class made with `@op.dataclass` is named by its class name (or the `name=` you give).
Two such classes with the same name and fields share results, wherever each is
defined.
- An ordinary `@dataclass` or a pydantic model is named by its module path and class
name. Moving it to another module makes a new type.

In both kinds a class whose fields change or are reordered is a new type: its pods get
new pipeline hashes and new tables, and the results stored with the old definition stay
where they are.

[Your Own Data Types](your-own-data-types.md) describes the two kinds and when to use
each.

## How it connects to other concepts

- [Sources](sources.md) form the base case of the Merkle chain (schema-only for pipeline hash,
Expand All @@ -179,3 +202,5 @@ stale results are not reused.
- [Operators](operators.md) contribute their own identity to the chain
- [Function Pods](function-pods.md) run inside a `PipelineJob` use content hashes for
memoization and pipeline hashes for DB scoping
- [Your Own Data Types](your-own-data-types.md) explains the identity of dataclasses and
pydantic models
Loading
Loading