Repository navigation
feat(types): v0.3 identity of a user class: declared name, kind and ordered fields (ITL-881) #292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
272054f
docs(specs): v0.3 identity of a user class, implementation spec (ITL-…
fae9ba7
docs(specs): record P1, registration order and cycle rules for user c…
394f60f
docs(plans): implementation plan for v0.3 user class identity (ITL-881)
27d4a9e
feat(types): identity record, string and naming policy for user class…
5f12c0c
feat(types): identify dataclasses and pydantic models by name, kind a…
b4104a4
feat(types): register_type API, rebinding and read-side identity chec…
a7d0044
test(hashing): golden values for v0.3 user class identity (ITL-881)
285c884
feat(types): full-length identity hash, per-class qualified option, n…
ac55871
docs: your own data types page, identity section, CHANGELOG and spec …
532a574
test(docs): register each doc block's source in linecache (ITL-881)
fabed37
refactor(types): rename register_type's qualified argument to qualifi…
b0ab6b6
fix(types): review fixes for user class identity reads (ITL-881)
1aae9d1
docs(changelog): Delta compaction fix for stored user-class values (I…
b4dadf3
test(types): pin IPC read of non-large nested storage as a known limi…
5707fb7
docs(types): docstrings for user-class identity helpers (ITL-881)
5b9693d
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
39951f0
docs(changelog): note the IPC limitation tracked as ITL-894 (ITL-881)
b930f62
test(types): docstring and value checks for the ITL-894 xfail (ITL-881)
ac71929
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
e3e8440
docs(plans): plan for field order as identity (ITL-881)
b54b031
feat(types)!: declared field order is part of a user class's identity…
d5357eb
test(hashing): golden values for field order as identity (ITL-881)
ac48ce2
fix(types): testing a read candidate no longer registers its nested c…
5b26f92
docs(types): declared field order is identity; known boundary for def…
fe41fa0
test(types): type-safe struct field access in field-order tests (ITL-…
887b312
fix(pipeline): a stored row that cannot be decoded as its class fails…
22b4b1e
fix(types): optional nested class None reads back; stored record chec…
25346ab
test(types): distinct value per field in workshop scenarios; assert d…
e1bd700
test(types): docstrings for the user-type tests; changelog for the re…
38c583a
fix(pipeline): failing observer callbacks do not replace an identity …
e18f05d
docs(changelog): move the ITL-881 entry to a changelog fragment (ITL-…
b1646ab
Merge remote-tracking branch 'origin/main' into eywalker/itl-881-user…
e99a0c2
docs(design-issues): move the ITL-881 entries to design_issues/ files…
c6837a1
fix(types): review round on user class identity: rebound containers, …
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| ### Breaking changes and migration | ||
|
|
||
| #### Dataclasses and pydantic models are identified by name, kind and fields (ITL-881) | ||
|
|
||
| Orcapod 0.3 identifies a dataclass or pydantic model by its name, its kind (dataclass or | ||
| pydantic) and its fields' names and types, in declared order. The name is the class name for a | ||
| class defined in a notebook or script (`__main__`), the module path and class name for a | ||
| class defined in a module, or the name declared with `op.register_type`. | ||
|
|
||
| - **Hashes change for every pipeline that uses such a class.** The class's logical type | ||
| name and Arrow extension name become `Name@<SHA-256 of name, kind and fields>`; the | ||
| stored struct lists fields in declared order, as before. Every schema hash, pipeline hash, table path, | ||
| system-tag column name and data hash that involves a dataclass or pydantic value moves. | ||
| The first run under 0.3 computes those pod and operator results again, into new tables. | ||
| Tables written by earlier versions are left untouched and are not read by 0.3 pipelines; | ||
| reading such a column directly raises `TypeIdentityError`. There is no migration tool. | ||
| Results that involve no dataclass or pydantic value are unaffected. | ||
| - **Changing a class's fields makes a new type.** Adding, removing, renaming, retyping or | ||
| reordering fields all count: a dataclass compares its fields as an ordered tuple and a | ||
| pydantic model dumps in declared order, so two classes with the same fields in another | ||
| order are different types. Defaults, methods and pydantic computed fields are not part | ||
| of the identity. The new type's pods get new tables; the results of | ||
| the old definition stay where they are. Before, a changed class shared the old tables and | ||
| a run returned 0 rows with only log warnings. | ||
| - **A stored shape that disagrees with the registered class raises.** `DeltaTableDatabase` | ||
| raises the new `orcapod.errors.TypeIdentityError` as is. It is not a `StoreReadError`: | ||
| the table is intact and the class differs. Like a store error, it fails the run under | ||
| every `error_policy`, with both orchestrators: a stored row that cannot be decoded is | ||
| never skipped as a failed row. Reading a column whose class cannot be found raises | ||
| `ImportError` naming the column, the type and the `op.register_type` call to declare it. | ||
| - **A subclass is its own type.** A dataclass or pydantic subclass was stored as its | ||
| parent when the parent was registered first: its extra fields were silently dropped from | ||
| stored data, and the pod's hash and table path depended on the registration order. It | ||
| now has its own identity and keeps all its fields, in any registration order. | ||
| - **Recursive classes** raise `RecursiveTypeError` (a `TypeError`) naming the field path, | ||
| for example `Node.next -> Node`. | ||
| - Pinned golden hash entries changed (approved): `semantic.json` `type.dataclass`, | ||
| `type.pydantic`; `schema.json` `multi_field.logical_mix`, | ||
| `single_field.{dataclass,pydantic}.{arrow_type,schema_hash}`; `datagram.json` | ||
| `{dataclass,pydantic}.{content_hash,pipeline_hash,system_tag_columns}` and the per-row | ||
| `data_hash`, `_tag_record_id::…` and `_tag_source_id::…` entries of both cases. | ||
|
|
||
| ### Added | ||
|
|
||
| #### `op.register_type` — declare the name of your own data types (ITL-881) | ||
|
|
||
| `@op.register_type(name="Point")` declares the name a dataclass or pydantic model is | ||
| identified by, wherever the class is defined. `@op.register_type(qualified_name=False)` names a | ||
| module class by its class name alone. Registration never modifies the class; declaring the | ||
| same class under a second name raises `ValueError`. See | ||
| [Your Own Data Types](docs/concepts/your-own-data-types.md). | ||
|
|
||
| Re-running a notebook cell that defines a dataclass or pydantic model, or reloading its | ||
| module, no longer breaks later pods with `ValueError: … already bound`: a class with the | ||
| same name and the same fields in the same order is the same type, and its stored results | ||
| are found again. | ||
|
|
||
| ### Fixed | ||
|
|
||
| #### A nested class stored as `None` reads back (ITL-881) | ||
|
|
||
| A dataclass or pydantic model with an optional nested class (`child: Inner | None`) that | ||
| stored `None` raised `TypeError: 'NoneType' object is not subscriptable` when the value | ||
| was read back. It now reads back as `None`. | ||
|
|
||
| #### Stored dataclass and pydantic values stay readable after Delta compaction (ITL-881) | ||
|
|
||
| Delta compaction (`optimize.compact()`) rewrites a `large_string` struct field as | ||
| `string`. Reading a stored dataclass or pydantic column with a `str` or `bytes` field then | ||
| failed, and a cached run returned 0 rows. Extension columns now accept large and non-large | ||
| layouts of the same type at any nesting depth, as they already did at the top level | ||
| (ITL-602). Known limitation: such a non-large column read over Arrow IPC or the C stream | ||
| interface (not Parquet or Delta) gives invalid arrays, at any depth; Orcapod's own writers | ||
| never produce one (ITL-894). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| ## `src/orcapod/logical_types/` — identity of user classes (dataclasses, pydantic models) | ||
|
|
||
| ### UCI1 — A user class was identified by `module.qualname` only, without its fields | ||
| **Status:** resolved | ||
| **Severity:** high | ||
| **Issue:** ITL-881 | ||
|
|
||
| A dataclass or pydantic model was identified by `f"{module}.{qualname}"`. The same class | ||
| defined in a notebook (`__main__`) and in a module were different types; two classes with | ||
| the same name and different fields shared pipeline hashes and tables, so a changed class | ||
| read the old table, failed, and the run returned 0 rows with only log warnings; re-running | ||
| the notebook cell that defines a class raised `ValueError: … already bound` until a kernel | ||
| restart; a subclass was stored as its parent, dropping its extra fields. | ||
|
|
||
| **Fix:** a user class is identified by `{kind, name, fields}` (spec | ||
| `superpowers/specs/2026-11-10-user-class-identity-v0.3.md`). The identity string | ||
| `Name@<full SHA-256 of the canonical record>` is the logical type name and Arrow extension | ||
| name, with the record in the extension metadata. `fields` lists `[name, type]` pairs in | ||
| declared order and the stored struct follows it: field order is part of the identity | ||
| (developer, November 11, 2026), so the same fields in another order are another type. | ||
| `op.register_type` declares names without modifying the class; `__main__` classes default | ||
| to their class name. The registry rebinds an unchanged identity to a re-defined class and | ||
| matches user classes by exact class only. The read path finds the class by kind and name, | ||
| checks the identity, and raises `ImportError` or `TypeIdentityError` naming the column; | ||
| `DeltaTableDatabase` raises `TypeIdentityError` as is, never wrapped in `StoreReadError`. | ||
| Testing a read candidate registers neither the class nor its nested classes | ||
| (`probe_record`), so a rejected candidate can still be declared under a name. Resolves the entry PR #288 proposes for | ||
| the same issue. Not covered: computed fields (deferred by the developer), local classes, | ||
| Enum/NamedTuple/TypedDict. | ||
|
|
||
| **Known limitation (ITL-894):** reading a stored column whose storage differs from the | ||
| registered one only by layout (`string` vs `large_string`) is accepted at any depth, so | ||
| Delta-compacted tables stay readable. `__arrow_ext_deserialize__` then returns the | ||
| registered large storage type, which is correct for Parquet and Delta reads (the decoder | ||
| converts the data) but not over Arrow IPC or the C stream interface, where Arrow reads | ||
| 32-bit offsets as 64-bit and the arrays are invalid. Top-level columns had this since | ||
| ITL-602; nested struct fields since this fix. Orcapod's own writers and Polars round trips | ||
| never produce non-large storage. Pinned by a strict xfail in | ||
| `tests/test_user_types/test_read_path.py`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,221 @@ | ||
| # Your Own Data Types | ||
|
|
||
| Function pods can take and return your own classes: Python | ||
| [dataclasses](https://docs.python.org/3/library/dataclasses.html) and | ||
| [pydantic](https://docs.pydantic.dev/) models. Orcapod stores their values in Arrow | ||
| columns and caches results computed from them like any other value. | ||
|
|
||
| Caching needs to know when two classes are **the same type**. This page explains how | ||
| Orcapod decides, and what that means for notebooks, scripts and modules. | ||
|
|
||
| ## What makes two classes the same type | ||
|
|
||
| Orcapod identifies a dataclass or pydantic model by three things: | ||
|
|
||
| | Component | What it is | | ||
| |---|---| | ||
| | **Name** | The class name for a class defined in a notebook or a script, the module path and class name for a class defined in a module, or a name you declare with `op.register_type` | | ||
| | **Kind** | Dataclass or pydantic model. A dataclass and a pydantic model are never the same type, even with the same name and fields | | ||
| | **Fields** | The name and type of every stored field, in the order the class declares them | | ||
|
|
||
| Two classes that agree on all three are the same type: results stored with one are found | ||
| again with the other. | ||
|
|
||
| Field order counts because your code can see it: a dataclass compares, unpacks and | ||
| prints its fields in declared order, and a pydantic model dumps and serializes in | ||
| declared order. Two classes with the same fields in another order are two types. | ||
|
|
||
| Known boundary: defaults, methods, properties, pydantic computed fields and validators, | ||
| and docstrings are **not** part of the identity, although a pod can observe them. | ||
| Changing one of them keeps the type and its stored results. If such a change alters what | ||
| a pod computes, bump the pod's version, as you would for any change to its code. | ||
|
|
||
| Adding, removing, renaming, retyping or reordering fields makes a **new type**. Results computed with the old | ||
| definition stay in the database, and the new type computes its results into new tables, | ||
| so a stored value is never read back with fields it was not written with. | ||
|
|
||
| ## Defining and redefining a class in a notebook | ||
|
|
||
| Define a class, use it in a pod and run a job: | ||
|
|
||
| ```python | ||
| from dataclasses import dataclass | ||
|
|
||
| from orcapod import PipelineJob, function_pod | ||
| from orcapod.databases import InMemoryArrowDatabase | ||
| from orcapod.sources import DictSource | ||
|
|
||
|
|
||
| @dataclass | ||
| class Point: | ||
| x: float | ||
| y: float | ||
|
|
||
|
|
||
| calls = [] | ||
|
|
||
|
|
||
| @function_pod(output_keys="norm") | ||
| def norm(p: Point) -> float: | ||
| calls.append(p) # only here to show when the function really runs | ||
| return (p.x**2 + p.y**2) ** 0.5 | ||
|
|
||
|
|
||
| db = InMemoryArrowDatabase() | ||
|
|
||
|
|
||
| def run(point_class): | ||
| rows = [ | ||
| {"k": "a", "p": point_class(x=3.0, y=4.0)}, | ||
| {"k": "b", "p": point_class(x=6.0, y=8.0)}, | ||
| ] | ||
| job = PipelineJob(name="points", store=db) | ||
| with job: | ||
| source = DictSource(data=rows, tag_columns=["k"], source_id="pts") | ||
| norm.pod(source, label="norm") | ||
| job.run() | ||
| return job.nodes["norm"] | ||
|
|
||
|
|
||
| print(run(Point).as_table().to_pandas()) | ||
| # k norm | ||
| # 0 a 5.0 | ||
| # 1 b 10.0 | ||
| print("function calls:", len(calls)) | ||
| # function calls: 2 | ||
| ``` | ||
|
|
||
| Running the cell that defines `Point` again creates a new class object with the same | ||
| name and fields. It is the same type, so the next run finds the stored results: | ||
|
|
||
| ```python | ||
| @dataclass | ||
| class Point: | ||
| x: float | ||
| y: float | ||
|
|
||
|
|
||
| run(Point) | ||
| print("function calls:", len(calls)) | ||
| # function calls: 2 | ||
| ``` | ||
|
|
||
| Adding, removing, retyping or reordering fields makes a new type. Re-run the pod cell too, so that `norm` takes | ||
| the new `Point`; the run then computes into new tables: | ||
|
|
||
| ```python | ||
| @dataclass | ||
| class Point: | ||
| x: float | ||
| y: float | ||
| label: str = "" | ||
|
|
||
|
|
||
| @function_pod(output_keys="norm") | ||
| def norm(p: Point) -> float: | ||
| calls.append(p) | ||
| return (p.x**2 + p.y**2) ** 0.5 | ||
|
|
||
|
|
||
| run(Point) | ||
| print("function calls:", len(calls)) | ||
| # function calls: 4 | ||
| ``` | ||
|
|
||
| ## How a class is named | ||
|
|
||
| The name is part of the identity, so it decides whether results survive moving code | ||
| around: | ||
|
|
||
| - A class defined in a **notebook or a script** run directly (`__main__`) is named by its | ||
| class name: `Point`. | ||
| - A class defined in a **module** is named by its module path and class name: | ||
| `pkg.models.Point`. | ||
|
|
||
| So results computed with `Point` in a notebook are **not** found by | ||
| `pkg.models.Point` after you move the class into a module, unless you give the module | ||
| class the same name. Declare it with `op.register_type`: | ||
|
|
||
| ```python | ||
| import orcapod as op | ||
|
|
||
|
|
||
| @op.register_type(name="Point") | ||
| @dataclass | ||
| class ModulePoint: | ||
| x: float | ||
| y: float | ||
| label: str = "" | ||
|
|
||
|
|
||
| run(ModulePoint) | ||
| print("function calls:", len(calls)) | ||
| # function calls: 4 | ||
| ``` | ||
|
|
||
| `op.register_type` accepts: | ||
|
|
||
| - `name="..."`: the name to use, the same wherever the class is defined. It must not be | ||
| empty, contain `@` or spaces, or start with `orcapod.`. | ||
| - `qualified_name=False`: name a module class by its class name alone, without retyping it. | ||
| - no arguments: keep the default name. | ||
|
|
||
| It never changes your class; it records the name in Orcapod's registry and returns the | ||
| class. Declare the name before the class is first used in a source or a pod. Declaring | ||
| the same class again with the same name does nothing; declaring it under a different | ||
| name raises `ValueError`. | ||
|
|
||
| You can see a class's identity on any column that holds it. The part before `@` is the | ||
| name; the rest is a hash of the name, the kind and the fields: | ||
|
|
||
| ```python | ||
| source = DictSource(data=[{"k": "a", "p": ModulePoint(1.0, 2.0)}], tag_columns=["k"]) | ||
| print(source.as_table().schema.field("p").type.extension_name) | ||
| # Point@... | ||
| ``` | ||
|
|
||
| The first time a class defined in a notebook or a script is used, Orcapod prints a | ||
| one-time warning saying that it is identified by its class name and fields. | ||
|
|
||
| ## Classes inside classes | ||
|
|
||
| A field whose type is another dataclass or pydantic model carries that class's own | ||
| identity: its name (however that class is named or registered), its kind and its fields. | ||
| Renaming the inner class, registering it under another name, or changing its fields | ||
| therefore makes the outer class a new type too. The order in which you declare the inner | ||
| and outer classes does not matter, as long as both are declared before first use. | ||
|
|
||
| A class that refers back to itself, directly (`next: Node | None`) or through other | ||
| classes, cannot be stored: Arrow has no recursive struct type. Orcapod raises | ||
| `RecursiveTypeError` naming the path, for example `Node.next -> Node`. | ||
|
|
||
| ## Reading stored values back | ||
|
|
||
| Reading stored values needs a class with the stored name, kind and fields. Orcapod | ||
| looks for one among the classes registered or used in the current process, then in the | ||
| script or notebook (`__main__`), then by importing the name as a module path. | ||
|
|
||
| - If no class with that name exists, reading raises `ImportError` naming the column, the | ||
| type and its fields, and showing the `op.register_type` call to declare it. | ||
| - If a class with that name exists but its fields or their order differ, reading raises | ||
| `TypeIdentityError` (from `orcapod.errors`) naming the column and both sets of fields. | ||
|
|
||
| Neither case is reported as missing data, and nothing stored is changed. | ||
|
|
||
| ## Rules and limits | ||
|
|
||
| - Classes defined inside a function cannot be registered: define them at module level or | ||
| at the top of the notebook. | ||
| - Each subclass is its own type with its own fields; it is never stored as its parent. | ||
| - Two different classes registered under the same name with the same fields are the same | ||
| type. Orcapod warns once when that happens across two modules, because stored values are | ||
| then read back as the most recently registered class. | ||
| - Results stored by Orcapod versions before 0.3 identified classes by module and class | ||
| name only. Version 0.3 does not read those columns; pipelines compute their results again | ||
| into new tables and leave the old ones untouched. | ||
|
|
||
| ## How it connects to other concepts | ||
|
|
||
| - [Function Pods](function-pods.md): the pods that take and return these classes. | ||
| - [Identity & Hashing](identity.md): how a type's identity enters pipeline and content | ||
| hashes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use bullet entries under each category heading.
This fragment uses
####subheadings and standalone paragraphs throughout. Convert these entries to bullets under the existing###category headings. Preserve the migration details and hash-change list.As per coding guidelines, the fragment must contain “the same category headings
CHANGELOG.mduses … each followed by its bullet lines.”Source: Coding guidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Declined. The fragment is this PR's changelog entry moved unchanged out of
CHANGELOG.md(ITL-961).####sub-sections with prose under a category heading are whatCHANGELOG.mditself uses, and the fragment passestests/test_changelog_fragments.py.Posted by a Claude Code agent on behalf of @eywalker.