Skip to content

feat(logical_types): logical type identifiers: extension name + metadata, written by every built-in type and hashed in full (ITL-935) - #319

Closed
eywalker wants to merge 28 commits into
mainfrom
eywalker/itl-935-design-make-the-logical-type-version
Closed

eywalker wants to merge 28 commits into
mainfrom
eywalker/itl-935-design-make-the-logical-type-version

Conversation

@eywalker

Copy link
Copy Markdown
Contributor

Stacked on #309

Fixes ITL-935

Summary

A logical type is now identified by its Arrow extension name and its Arrow extension metadata, and the whole identifier is part of the data type (developer decisions, ITL-938 design session, November 11, 2026).

  • Built-in types write an identifier. orcapod.path, orcapod.upath, orcapod.uuid, orcapod.file, orcapod.directory, pandas.dataframe, pandas.series write {"version":1} (they wrote empty metadata); orcapod.ndarray already did.
  • The whole identifier is hashed. The text <extension name>;<lowercase hex of the metadata> (LogicalTypeIdentifier.text, one helper) replaces the bare name in the type hash (TypeObjectHandler), in function signatures (canonical_annotation_str) and at the start of a value-hashed cell token (hashing/visitors.py).
  • Reading. New protocol member readable_extension_metadata() (default in BaseLogicalType); stored metadata must equal one of its values byte for byte, otherwise LogicalTypeIdentifierError names the column, the stored identifier and what is readable. Before, a column read as plain storage was wrapped by name whatever its metadata said. The seven types above read empty metadata as version 1 (same layout).
  • Native containers (list[T] / set[T] of a logical type) carry no version of their own, by design; the type hash keeps treating them like any generic alias (origin and arguments). A change to them is a data context change.
  • Fixed: [], [None] and [None, None] hashed the same in a list/set of a value-hashed type. The cell now holds the element count and, per element, a null marker or a length-prefixed token.
  • make_arrow_extension_type(..., also_reads=...); an extension name may not contain ;.
  • Docs: docs/concepts/logical-type-identifiers.md (for implementers). CHANGELOG, DESIGN_ISSUES, CLAUDE.md and .zed/rules updated. Plan and decisions: superpowers/plans/2026-11-11-itl-935-logical-type-identifiers.md.

What moves

Stored path of a pod shift(p: Path) -> Path (measured):

function hash schema hash
before e7d5af653505598a… 1eddb0e412608797…
after 5b5a49597b652bdd… 2d2936f8871f994f…
plain(id2: int) -> int, before and after db3fd817dc7cf10d… 84f155b4a2ea98ab…

Golden values, against the #309 base (b88f7eb3): 70 of 771 leaves move, every one in a group with a logical-type column; pipeline.json (operators over plain types) is unchanged.

File Moved Entries
semantic.json 8 type.{path, uuid, ndarray, dataframe, list_path, set_uuid, dataclass, pydantic}
schema.json 14 single_field.{path, uuid, ndarray, dataframe, list_path, set_uuid, dataclass, pydantic}.schema_hash; multi_field.logical_mix; compute_schema_hash.{full, chars_16} and source_schema_hash.{schema_hash, source_id_column, record_id_column} (both have a Path column)
arrow.json 4 extension_schema.extension.{path, uuid}, extension_table.extension.{path, uuid}
datagram.json 44 (+32 keys renamed because the key contains a moved hash) per group path, uuid, ndarray, dataframe, list_path, set_uuid: content_hash, pipeline_hash, two system_tag_columns, two row data_hash (6 each); per group dataclass, pydantic: the same without the row data_hash (4 each)
pipeline.json 0

Per type: path/upath row hashes move because starfix hashes the metadata of the stored string column; uuid, file, directory, array and pandas cells move because the token now starts with the whole identifier; ndarray moves although its metadata is unchanged, because the metadata is now part of the type hash and token; dataclass and pydantic move in type, schema and pipeline hashes only (their metadata {"category": ...} is now hashed; their row hashes do not move).

Pinned outside the golden directory: 6 entries of tests/test_hashing/hash_samples/schema_hash_golden.json (the ones with File, Directory, Path) and 3 values in tests/test_hashing/test_canonical_type_names.py.

Merge points

  • ITL-945 (labels out of hash inputs) changes the tail of the scalar cell token in hashing/visitors.py (content_hash.to_prefixed_digest()). This PR changes the prefix line (type_name = arrow_type_identifier(extension_type).text) and adds _encode_container_elements. Final token: utf8("<ext name>;<hex metadata>") + b"::" + raw digest.
  • ITL-944: contexts/data/v0.1.json is not changed by this PR.
  • History: commit b8990fa7 ("wrapper identifiers in the schema hash") put the wrapper identifier into the schema hash; the developer decided against it and commit 3a6a0752 reverts that part (no force-push).

What other open PRs must do

Behaviour changes in existing tests

  • test_database_hooks.py::test_none_metadata_already_registered_noop asserted that a registered type with metadata silently accepts a column stored without metadata. That is now refused; the test is renamed and expects LogicalTypeIdentifierError.
  • Tests that pinned the name-only text (orcapod.file, type:orcapod.path, orcapod:file::) now pin the identifier text.
  • Test stub logical types that do not derive from BaseLogicalType gained readable_extension_metadata().

Known limits

  • When pyarrow itself resolves the extension type while reading a file, the error does not name the column.
  • The effect of a metadata-only change on stored paths is tested at the hash level (type, schema, function signature), not through a pipeline: a DataContext object passed to a source is not used by the source's inner stream, so a custom context cannot drive a pipeline (out of scope, reported).
  • A third-party logical type registered for a generic alias is hashed as origin and arguments in the schema hash, so its own identifier is not in the path.
  • A dataclass output column of a function result table is stored as a plain struct without extension keys (out of scope, reported).

Test plan

  • uv run pytest tests/ on Python 3.11, 3.12 and 3.13, one at a time: 7814 passed, 96 skipped, 18 xfailed on each
  • uv run --with ruff ruff check . and ruff format --check .
  • uv run --with basedpyright basedpyright: exit 0, 0 errors
  • New tests: tests/test_logical_types/test_logical_type_identifiers.py (identifier written by each built-in, Delta round trip, metadata-only change moves type/schema/function/row hashes, plain types unaffected, explicit error on both read channels, empty metadata read as version 1, container rules, [] vs [None])
  • Doc samples and mkdocs build --strict (part of the suite)

🤖 Generated with Claude Code

Edgar Y. Walker and others added 27 commits November 11, 2026 15:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash
pyarrow's canonical extension types and extension-typed schemas with their
extension identity instead of as their storage type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910)

Explicitly approved regeneration (developer decision 3, November 11, 2026):
184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline
68/305, schema 22/92, semantic 1/64); no key added or removed among them.
21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a
chunked Boolean column, canonical extension types and extension schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910)

Adds orcapod.errors.StoreHashLabelError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

A v0.3 node whose pipeline, result, operator, cached-source or side-effect
table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now
raises StoreHashLabelError on read and before its first write, and
PipelineJob.run checks every node before the first one runs, so a refused
store is left exactly as it was. No opt-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910)

Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses
arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove
StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run
pre-flight. Every whole-table read keeps rows of the current label only
(keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910)

Developer decisions of November 11, 2026: a read that leaves out rows of another
hash label logs one warning with the total; a side-effect node whose invocation
log holds deliveries logged under another Arrow label raises
SideEffectLogHashLabelError before delivering anything.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reported, kept)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… in schema hash and cell token (ITL-935, WIP)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ules (ITL-935, WIP)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…egeneration; tests (ITL-935)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…] cell collision; schema hash of generics unchanged (ITL-935)

Reverts the wrapper-identifier-in-schema-hash change of the previous commit
(developer decision, November 11, 2026).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
ITL-935 Design: make the logical type version explicit in stored extension types, so newer Orcapod can read older representations

Overview

Developer, November 11, 2026: "for the extension types, should we make the version of the logical type more prominent? Eventually I'd like to build a system where newer version should be able to (as possible) handle representations from the older versions." To be settled as a design question later this week (Friday November 13 or Saturday November 14, 2026).

Where versions live today (checked on the PR #289 branch, not exhaustively):

  • The array format writes {"version":1} in ARROW:extension:metadata for its extension types (orcapod.ndarray).
  • Value-hash labels carry a version suffix (ndarray_v1, polars_expr_v1, torch_tensor_v1, arrow_v0.1), but those label a hash, not a stored representation.
  • The other logical types (dataclass, pydantic, list, path and so on) appear to store an extension name and type-specific metadata with no representation version.
  • A reader finds the logical type by extension name (get_by_arrow_extension_name); nothing dispatches on a version.

Goals & Success Criteria

  • Decide where a logical type's representation version is recorded in stored data (extension name, extension metadata, or both) and make it uniform across every logical type Orcapod ships.
  • Define the reading rule: a newer Orcapod, given an older version's representation, either converts it through a registered reader for that version or fails with an error naming the type and both versions. Never a silent misread.
  • Define what "no version recorded" means for data written before this design (for example: treated as version 1).
  • Decide how the version relates to the value-hash label of the same type, and whether reading an old representation yields the old hash or the new one.
  • State which stored representations and hashes change, if any.

Scope & Boundaries

In scope:

  • The design, its decisions and the list of code changes.

Out of scope:

  • Path and hash-algorithm migration in general (ITL-914 and the migratability project), and the orcapod-sanctuary package; this design should fit with them.

Dependencies & Risks

Resources & References

Note, November 11, 2026

PR #289 (orcapod.ndarray) was merged without the developer having reviewed it thoroughly. Developer: "All the more reason having some kind of versioning / ability to protect / give us enough basis to change things in the future without causing utter chaos." The design should therefore assume that representations shipped in v0.3 may need correcting later, and give a safe way to do it: every stored representation carries a version, old versions stay readable, and a change is an explicit new version, never an edit of an existing one.

Decisions, November 11, 2026 (ITL-938 design session)

The design question is answered in outline, and this issue becomes its implementation, wanted before the v0.3 tag:

  • A logical type's identifier is its Arrow extension name plus its extension metadata. It is owned by the type's implementer and opaque to Orcapod.
  • The identifier changes only when an existing value's stored layout or hash would change. It is not the package version: a type may add support for new values and keep its identifier, as long as every old value still hashes identically.
  • The whole identifier enters the schema hash (today only the name does).
  • Built-in types (file, path, upath, uuid, directory, pandas dataframe and series, and any other built-in with empty extension metadata) get an identifier before the v0.3 tag, by the same mechanism as external types.
  • A changed identifier is a schema change, so stored paths move.
  • A hash label names only the algorithm that produced a value; it is not a version marker. Logical types are not part of the global hash version.

Review in Linear

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

The included review limit has been reached and this organization has disabled usage-based review continuation. Wait for reviews to reset or ask a billing admin to change After included review limits.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ae3f7285-7c40-4faf-abaa-a3f7f53b93bd
Commits

Reviewing files that changed from the base of the PR and between d61e3e3 and 2369d75.

Files ignored due to path filters: 1

  • uv.lock is excluded by !**/*.lock
Files selected for processing: 76

  • .zed/rules
  • CHANGELOG.md
  • CLAUDE.md
  • DESIGN_ISSUES.md
  • changelog.d/ITL-935.md
  • docs/concepts/arrays.md
  • docs/concepts/function-pods.md
  • docs/concepts/logical-type-identifiers.md
  • docs/reference/hashing.md
  • mkdocs.yml
  • pyproject.toml
  • src/orcapod/contexts/data/schemas/context_schema.json
  • src/orcapod/contexts/data/v0.1.json
  • src/orcapod/core/cached_function_pod.py
  • src/orcapod/core/data_function.py
  • src/orcapod/core/nodes/function_node.py
  • src/orcapod/core/nodes/operator_node.py
  • src/orcapod/core/result_cache.py
  • src/orcapod/core/sources/cached_source.py
  • src/orcapod/errors.py
  • src/orcapod/hashing/arrow_hashers.py
  • src/orcapod/hashing/hash_utils.py
  • src/orcapod/hashing/schema_cleaner.py
  • src/orcapod/hashing/semantic_hashing/builtin_handlers.py
  • src/orcapod/hashing/semantic_hashing/function_info_extractors.py
  • src/orcapod/hashing/stored_hash_labels.py
  • src/orcapod/hashing/versioned_hashers.py
  • src/orcapod/hashing/visitors.py
  • src/orcapod/logical_types/__init__.py
  • src/orcapod/logical_types/base_logical_type.py
  • src/orcapod/logical_types/builtin_logical_types.py
  • src/orcapod/logical_types/database_hooks.py
  • src/orcapod/logical_types/directory_type.py
  • src/orcapod/logical_types/file_type.py
  • src/orcapod/logical_types/identifier.py
  • src/orcapod/logical_types/list_logical_type_factory.py
  • src/orcapod/logical_types/pandas_type.py
  • src/orcapod/logical_types/protocols.py
  • src/orcapod/logical_types/registry.py
  • src/orcapod/semantic_types/universal_converter.py
  • src/orcapod/side_effects.py
  • superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md
  • superpowers/plans/2026-11-11-itl-910-followup-ignore-old-label-rows.md
  • superpowers/plans/2026-11-11-itl-935-logical-type-identifiers.md
  • tests/test_arrays/test_array_v02_store.py
  • tests/test_arrays/test_array_value_format.py
  • tests/test_core/function_pod/test_write_side_registration.py
  • tests/test_core/operators/test_async_sync_equivalence.py
  • tests/test_core/operators/test_group_by.py
  • tests/test_core/operators/test_materialize_source_info.py
  • tests/test_core/test_result_cache.py
  • tests/test_databases/test_extension_aware_database.py
  • tests/test_hashing/golden/cases.py
  • tests/test_hashing/golden/data/arrow.json
  • tests/test_hashing/golden/data/datagram.json
  • tests/test_hashing/golden/data/pipeline.json
  • tests/test_hashing/golden/data/schema.json
  • tests/test_hashing/golden/data/semantic.json
  • tests/test_hashing/golden/generate_golden_hashes.py
  • tests/test_hashing/hash_samples/schema_hash_golden.json
  • tests/test_hashing/test_canonical_type_names.py
  • tests/test_hashing/test_extension_type_hashing.py
  • tests/test_hashing/test_function_info_extractors.py
  • tests/test_hashing/test_hash_utils.py
  • tests/test_hashing/test_pandas_handlers.py
  • tests/test_hashing/test_semantic_hasher.py
  • tests/test_hashing/test_starfix_arrow_hasher.py
  • tests/test_hashing/test_starfix_format2.py
  • tests/test_hashing/test_stored_hash_labels_helpers.py
  • tests/test_logical_types/test_apply_extension_types.py
  • tests/test_logical_types/test_database_hooks.py
  • tests/test_logical_types/test_logical_type_identifiers.py
  • tests/test_logical_types/test_protocols.py
  • tests/test_logical_types/test_registry.py
  • tests/test_pipeline/test_stored_hash_labels.py
  • tests/test_semantic_types/test_struct_layout_identity.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

…961)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@eywalker

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Posted by a Claude Code agent on behalf of @eywalker.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@eywalker

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Posted by a Claude Code agent on behalf of @eywalker.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

eywalker pushed a commit that referenced this pull request Nov 12, 2026
…-logical-type-version' (#319, ITL-935) into ITL-967

Resolutions (merge points b and d):
- hashing/visitors.py scalar extension-cell token is
  utf8('<extension name>;<hex metadata>') + b'::' + raw digest: the prefix of
  ITL-935, the label-free tail of ITL-945. _encode_container_elements kept.
- builtin_handlers.py imports both Encoded and logical_type_identifier.
- Tests pinning the token assert the identifier text and the raw digest.
- DESIGN_ISSUES.md keeps both sides; golden and pinned-value conflicts take
  one side and are regenerated once after all merges.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@eywalker eywalker mentioned this pull request Nov 12, 2026
7 tasks done
@eywalker

Copy link
Copy Markdown
Contributor Author

Closing as superseded: this branch's head is an ancestor of #324's merged head (6dc2a51), and #324 is merged to main as e140862. The findings CodeRabbit raised on this code were handled in #324's review rounds; the few that remain are tracked in ITL-975, ITL-951 and ITL-904.

Posted by a Claude Code agent on behalf of @eywalker.

@eywalker eywalker closed this Nov 12, 2026
@eywalker
eywalker deleted the eywalker/itl-935-design-make-the-logical-type-version branch November 17, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant