Repository navigation
feat(logical_types): logical type identifiers: extension name + metadata, written by every built-in type and hashed in full (ITL-935) - #319
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash pyarrow's canonical extension types and extension-typed schemas with their extension identity instead of as their storage type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910) Explicitly approved regeneration (developer decision 3, November 11, 2026): 184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline 68/305, schema 22/92, semantic 1/64); no key added or removed among them. 21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a chunked Boolean column, canonical extension types and extension schemas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910) Adds orcapod.errors.StoreHashLabelError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) A v0.3 node whose pipeline, result, operator, cached-source or side-effect table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now raises StoreHashLabelError on read and before its first write, and PipelineJob.run checks every node before the first one runs, so a refused store is left exactly as it was. No opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910) Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run pre-flight. Every whole-table read keeps rows of the current label only (keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910) Developer decisions of November 11, 2026: a read that leaves out rows of another hash label logs one warning with the total; a side-effect node whose invocation log holds deliveries logged under another Arrow label raises SideEffectLogHashLabelError before delivering anything. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reported, kept) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… in schema hash and cell token (ITL-935, WIP) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ules (ITL-935, WIP) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…egeneration; tests (ITL-935) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…] cell collision; schema hash of generics unchanged (ITL-935) Reverts the wrapper-identifier-in-schema-hash change of the previous commit (developer decision, November 11, 2026). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ITL-935 Design: make the logical type version explicit in stored extension types, so newer Orcapod can read older representations
OverviewDeveloper, November 11, 2026: "for the extension types, should we make the version of the logical type more prominent? Eventually I'd like to build a system where newer version should be able to (as possible) handle representations from the older versions." To be settled as a design question later this week (Friday November 13 or Saturday November 14, 2026). Where versions live today (checked on the PR #289 branch, not exhaustively):
Goals & Success Criteria
Scope & BoundariesIn scope:
Out of scope:
Dependencies & Risks
Resources & References
Note, November 11, 2026PR #289 ( Decisions, November 11, 2026 (ITL-938 design session)The design question is answered in outline, and this issue becomes its implementation, wanted before the v0.3 tag:
|
|
Warning Review limit reachedThe included review limit has been reached and this organization has disabled usage-based review continuation. Wait for reviews to reset or ask a billing admin to change After included review limits. Next included review available in 3 minutes. View limit details
|
…961) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
…-logical-type-version' (#319, ITL-935) into ITL-967 Resolutions (merge points b and d): - hashing/visitors.py scalar extension-cell token is utf8('<extension name>;<hex metadata>') + b'::' + raw digest: the prefix of ITL-935, the label-free tail of ITL-945. _encode_container_elements kept. - builtin_handlers.py imports both Encoded and logical_type_identifier. - Tests pinning the token assert the identifier text and the raw digest. - DESIGN_ISSUES.md keeps both sides; golden and pinned-value conflicts take one side and are regenerated once after all merges. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Closing as superseded: this branch's head is an ancestor of #324's merged head (6dc2a51), and #324 is merged to Posted by a Claude Code agent on behalf of @eywalker. |
Stacked on #309
Fixes ITL-935
Summary
A logical type is now identified by its Arrow extension name and its Arrow extension metadata, and the whole identifier is part of the data type (developer decisions, ITL-938 design session, November 11, 2026).
orcapod.path,orcapod.upath,orcapod.uuid,orcapod.file,orcapod.directory,pandas.dataframe,pandas.serieswrite{"version":1}(they wrote empty metadata);orcapod.ndarrayalready did.<extension name>;<lowercase hex of the metadata>(LogicalTypeIdentifier.text, one helper) replaces the bare name in the type hash (TypeObjectHandler), in function signatures (canonical_annotation_str) and at the start of a value-hashed cell token (hashing/visitors.py).readable_extension_metadata()(default inBaseLogicalType); stored metadata must equal one of its values byte for byte, otherwiseLogicalTypeIdentifierErrornames the column, the stored identifier and what is readable. Before, a column read as plain storage was wrapped by name whatever its metadata said. The seven types above read empty metadata as version 1 (same layout).list[T]/set[T]of a logical type) carry no version of their own, by design; the type hash keeps treating them like any generic alias (origin and arguments). A change to them is a data context change.[],[None]and[None, None]hashed the same in a list/set of a value-hashed type. The cell now holds the element count and, per element, a null marker or a length-prefixed token.make_arrow_extension_type(..., also_reads=...); an extension name may not contain;.docs/concepts/logical-type-identifiers.md(for implementers). CHANGELOG, DESIGN_ISSUES, CLAUDE.md and.zed/rulesupdated. Plan and decisions:superpowers/plans/2026-11-11-itl-935-logical-type-identifiers.md.What moves
Stored path of a pod
shift(p: Path) -> Path(measured):e7d5af653505598a…1eddb0e412608797…5b5a49597b652bdd…2d2936f8871f994f…plain(id2: int) -> int, before and afterdb3fd817dc7cf10d…84f155b4a2ea98ab…Golden values, against the #309 base (
b88f7eb3): 70 of 771 leaves move, every one in a group with a logical-type column;pipeline.json(operators over plain types) is unchanged.semantic.jsontype.{path, uuid, ndarray, dataframe, list_path, set_uuid, dataclass, pydantic}schema.jsonsingle_field.{path, uuid, ndarray, dataframe, list_path, set_uuid, dataclass, pydantic}.schema_hash;multi_field.logical_mix;compute_schema_hash.{full, chars_16}andsource_schema_hash.{schema_hash, source_id_column, record_id_column}(both have aPathcolumn)arrow.jsonextension_schema.extension.{path, uuid},extension_table.extension.{path, uuid}datagram.jsonpath, uuid, ndarray, dataframe, list_path, set_uuid:content_hash,pipeline_hash, twosystem_tag_columns, two rowdata_hash(6 each); per groupdataclass, pydantic: the same without the rowdata_hash(4 each)pipeline.jsonPer type: path/upath row hashes move because starfix hashes the metadata of the stored string column; uuid, file, directory, array and pandas cells move because the token now starts with the whole identifier;
ndarraymoves although its metadata is unchanged, because the metadata is now part of the type hash and token; dataclass and pydantic move in type, schema and pipeline hashes only (their metadata{"category": ...}is now hashed; their row hashes do not move).Pinned outside the golden directory: 6 entries of
tests/test_hashing/hash_samples/schema_hash_golden.json(the ones withFile,Directory,Path) and 3 values intests/test_hashing/test_canonical_type_names.py.Merge points
hashing/visitors.py(content_hash.to_prefixed_digest()). This PR changes the prefix line (type_name = arrow_type_identifier(extension_type).text) and adds_encode_container_elements. Final token:utf8("<ext name>;<hex metadata>") + b"::" + raw digest.contexts/data/v0.1.jsonis not changed by this PR.b8990fa7("wrapper identifiers in the schema hash") put the wrapper identifier into the schema hash; the developer decided against it and commit3a6a0752reverts that part (no force-push).What other open PRs must do
{"version":1}; rebase on thealso_readssignature, keep the metadata check in its deferred lookup, regenerate its golden entries.version_metadata(1)instead of empty metadata.pa.Tablevalue identity): they change what a pandas value hashes to; before the v0.3 tag that defines version 1, after it they need version 2.name@fingerprintplus its metadata is a complete identifier; its metadata is now hashed, so it must be canonical JSON.Behaviour changes in existing tests
test_database_hooks.py::test_none_metadata_already_registered_noopasserted that a registered type with metadata silently accepts a column stored without metadata. That is now refused; the test is renamed and expectsLogicalTypeIdentifierError.orcapod.file,type:orcapod.path,orcapod:file::) now pin the identifier text.BaseLogicalTypegainedreadable_extension_metadata().Known limits
DataContextobject passed to a source is not used by the source's inner stream, so a custom context cannot drive a pipeline (out of scope, reported).Test plan
uv run pytest tests/on Python 3.11, 3.12 and 3.13, one at a time: 7814 passed, 96 skipped, 18 xfailed on eachuv run --with ruff ruff check .andruff format --check .uv run --with basedpyright basedpyright: exit 0, 0 errorstests/test_logical_types/test_logical_type_identifiers.py(identifier written by each built-in, Delta round trip, metadata-only change moves type/schema/function/row hashes, plain types unaffected, explicit error on both read channels, empty metadata read as version 1, container rules,[]vs[None])mkdocs build --strict(part of the suite)🤖 Generated with Claude Code