From 272054fedb9aa5d6ed630ade099d47b3301b53ca Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:10:44 -0800 Subject: [PATCH 001/147] docs(specs): v0.3 identity of a user class, implementation spec (ITL-881) Defines the identity record (declared name, kind, order-independent field types), where it enters the hashes, the register_type API proposal, the rebinding and read-path rules, and the 25 golden entries that move. Refs ITL-881 Co-Authored-By: Claude Opus 5.5 --- .../2026-11-10-user-class-identity-v0.3.md | 304 ++++++++++++++++++ 1 file changed, 304 insertions(+) create mode 100644 superpowers/specs/2026-11-10-user-class-identity-v0.3.md diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md new file mode 100644 index 000000000..32d87c730 --- /dev/null +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -0,0 +1,304 @@ +# Identity of a user class in v0.3 (ITL-881) + +**Status:** implementation spec, waiting for approval. No product code changes yet. +**Issue:** ITL-881 (v0.3, a public release; tag target November 16, 2026). +**Builds on:** the findings in PR #288 (`superpowers/specs/2026-11-10-user-class-identity.md`, +experiments E01–E08). "W1"–"W15" are the workshop scenarios from §3.4 of that document. + +Each decision is marked **CONFIRMED** (the developer decided it, November 10, 2026) or +**PENDING** (the default this spec plans with; each pending item is a small, local change). + +| ID | Decision | Status | +|---|---|---| +| D-name | Nominal identity with a declared name; `__main__` classes default to the name only | CONFIRMED: developer decision 1 | +| D-fields | Field names and field types are part of the identity; field order is not | CONFIRMED: developer decision 2 | +| D-computed | Computed fields are not part of the identity in v0.3 | CONFIRMED as deferred: developer decision 2 defers it; v0.3 keeps today's stored fields | +| D-kind | Dataclass and pydantic are distinct kinds | CONFIRMED: developer decision 3 | +| D-rebind | Same identity re-registered: rebind silently | CONFIRMED: developer answer to S1 | +| D-raise | Stored shape disagreeing with registered shape raises, in v0.3 | CONFIRMED: developer answer to S2 | +| D-doc | A page documenting identity | CONFIRMED: developer answer to S4 | +| P1 | A field typed by another user class carries that class's full identity | PENDING: developer asked; default stated by the coordinator | +| P2 | Public API and setting names | PENDING: developer to settle the names | +| D-hash | The 25 golden entries in §e move; nothing else | PENDING: needs the developer's approval of the list (golden rule) | +| D-legacy | Reading a pre-0.3 user-class column directly raises a clear error | PENDING: stored-data behaviour, proposed here | +| D-notice | One-time notice when a `__main__` class is first registered (S3) | PENDING: developer leaned towards a warning; first scope cut | + +## a. The identity definition + +A user class's identity is an **identity record** with three components: + +```json +{"fields": {"x": "int", "y": "float"}, "kind": "dataclass", "name": "Point"} +``` + +### Name component — CONFIRMED (decision 1) + +The name is a string chosen once per class, by the first rule that applies: + +1. **Declared name.** The user declared one (§b). Used verbatim. +2. **`__main__` class** (`cls.__module__` is `"__main__"` or `"__mp_main__"`, the name a + `multiprocessing` spawn child gives the main module, W14): `cls.__qualname__` + (`"Point"`, or `"Outer.Point"` for a nested class), under the default setting + `main="name"`. Setting `main="qualified"` gives `"__main__.Point"` (today's string). +3. **Module class:** `f"{module}.{qualname}"` (`"pkg.models.Point"`), under the default + setting `module="qualified"`. Setting `module="name"` gives `"Point"`. The module is + passed through `canonical_module_name`, which leaves user modules unchanged. + +A name must be non-empty, must not contain `@` or whitespace, and must not start with +`orcapod.` (reserved for built-in types such as `orcapod.path`). A class whose +`__qualname__` contains `` stays rejected in v0.3, even with a declared name +(scope, §g). + +### Kind component — CONFIRMED (decision 3) + +`"dataclass"` (`dataclasses.is_dataclass`, which includes `pydantic.dataclasses.dataclass`) +or `"pydantic"` (`pydantic.BaseModel` subclass). A dataclass and a pydantic model with the +same name and fields have different identity records, so they are different types. + +### Field component — CONFIRMED (decision 2), nested classes PENDING (P1) + +`fields` maps each **stored field's name** to its **type token**. It is a JSON object +serialized with sorted keys, so field order never enters. The stored fields are exactly +the ones stored today: dataclass fields with `init=True`, pydantic `model_fields`. + +Type token of an annotation (from `typing.get_type_hints`, which strips `Annotated` +extras): + +| Annotation | Token | Example | +|---|---|---| +| builtin scalar | its name | `int`, `float`, `str`, `bool`, `bytes` | +| other class without a logical type (stdlib) | `canonical_type_name(cls)` | `datetime.datetime`, `decimal.Decimal` | +| type with a built-in logical type | its logical type name | `orcapod.path`, `orcapod.uuid`, `orcapod.file` | +| generic container | `origin[arg, …]`, origin by its canonical name, args in order | `list[int]`, `dict[str, float]`, `tuple[int, str]` (`typing.List[int]` gives `list[int]`) | +| `Optional` / union | member tokens sorted, joined by ` \| ` | `int \| None` for `Optional[int]`, `int \| None` and `None \| int` | +| nested dataclass / pydantic model (**P1 default**) | that class's identity string (below) | `Inner@3c1d…` | +| nested class (**P1 alternative**) | its fields only: `struct{a: int, b: str}` (sorted, no name, no kind) | — | + +Deliberately **not** part of the identity: defaults and default factories, methods and +validators, properties and pydantic `computed_field`s, `init=False` dataclass fields, +`ClassVar`s, private attributes, field order, docstrings, pydantic field aliases and +constraints (`Field(gt=0)`, `Annotated` metadata), `frozen`/`slots`/config flags, base +classes (only the resulting fields count). Computed fields are deferred (D-computed); a +later decision to include them adds a key that is absent when empty, so it would move +only classes that have computed fields. + +The token is built from the Python annotation, not the Arrow type, because the developer +named field *types* and because the Arrow type cannot tell `str` from `orcapod.path` +(both `large_string`). + +### Identity string + +``` +identity string = f"{name}@{fingerprint}" +fingerprint = sha256(canonical_json(record)).hexdigest()[:16] +canonical_json = json.dumps(record, sort_keys=True, separators=(",", ":")) +``` + +`Point(x: int, y: float)` as a dataclass gives `Point@9117beaaf7ebe3ab`; with `y: int`, +`Point@c0a75f0c39b86873`; as a pydantic model with `y: float`, `Point@703bff8af8d6002c`. +SHA-256 of a documented JSON string, not the Orcapod semantic hasher, so the identity can +be reproduced by hand and does not depend on a hasher version. 16 hex digits (64 bits) +only have to separate classes with the same name and kind. + +### Where each component enters + +The identity string replaces today's `f"{module}.{qualname}"` as the class's logical +type name. Every place that used the old string now uses the new one; there is no +second mechanism. + +| Place | What it carries | How | +|---|---|---| +| Logical type name | identity string | `DataclassLogicalType.logical_type_name` / pydantic equivalent | +| Arrow extension name (`ARROW:extension:name`) | identity string | same string; one Arrow/Polars extension registration per identity | +| Arrow extension metadata | category plus the full record: `{"category":"orcapod.dataclass","identity":{…}}`, canonical JSON | read path uses it to find the class and to print the stored shape | +| Arrow storage type | struct of the stored fields | unchanged (derived from the annotations) | +| Schema hash, source `pipeline_hash`, system-tag column names | identity string | `TypeObjectHandler` hashes a registered class as `type:` (unchanged code) | +| Pod signature hash, node `pipeline_hash`, DB table paths | identity string | via the schema/signature hashes | +| Per-row data hash, `content_hash` | identity string + record + values | the extension column is hashed with its extension name and metadata (unchanged code) | +| `list[Point]`, `Point \| None`, nested fields | identity string | container logical names embed the element's logical name; nested fields via the token (P1) | + +**Same name, different fields → different pipeline identity.** `Point(x: int, y: int)` +and `Point(x: int, y: float)` have different fingerprints, so different logical type +names, different schema hashes, different pod and node pipeline hashes, and different +table paths. They never share a table. That removes the "same table, different shape" +failure (W6, W10: today 0 rows plus log warnings): the changed class computes into its own +tables, and the old tables stay as they were. In one process the two shapes have two +extension names, so Arrow's and Polars' global extension registries (keyed by name) hold +both without conflict; that is why the fields must be in the extension *name*, not only +in its metadata. + +## b. Public API — PENDING (P2: names), behaviour proposed + +```python +import orcapod as op + +@op.register_type # name by the setting: __main__ → "Point", module → "pkg.models.Point" +@dataclass +class Point: ... + +@op.register_type(name="Point") # declared name, same everywhere it is defined +@dataclass +class Point: ... + +op.register_type(Point, name="Point") # call form, for classes you do not own + +op.set_type_naming(main="name", module="qualified") # process-wide defaults (shown) +``` + +- `register_type` records the name for the class object (a process-wide weak-keyed table, + so the class is not modified), validates it, builds its logical type in the default + data context (so an unsupported field type fails at the definition, not at first use) + and returns the class unchanged. Classes that are never registered are registered on + first use, as today, with the name from the setting. +- `set_type_naming` changes the default for classes registered after the call. Values: + `"name"` or `"qualified"`. Defaults: `main="name"`, `module="qualified"`. No TOML key + in v0.3 (scope, §g). +- Registering a class object under a second, different name after it is registered (for + example `register_type(Point, name="Geo")` after `Point` was used) raises `ValueError` + naming both names: one class object has one identity per process. + +**Name collisions (same kind, same name, two different class objects):** + +| Case | What happens | What the user sees | +|---|---|---| +| Same fields, same `module.qualname` (cell re-run, `%autoreload`, W1/W3/W4) | Same identity: the identity is rebound to the new class object (§c) | nothing | +| Same fields, different `module.qualname` (two modules declare `"Point"`, or `module="name"`) | Same identity by definition: they are the same type; values read back are built as the most recently registered class | one `UserWarning` per rebinding naming both classes. Warranted: values come back as a different class object than may be expected | +| Different fields (W2 with changed fields, or two modules) | Different identities: two types that happen to share a display name; no clash | nothing at registration. Reading picks the class whose identity matches (§d) | + +## c. Rebinding rule (S1) — CONFIRMED (developer answer to S1) + +- **Same identity string, new class object → rebind silently.** The registry rebinds the + identity (logical name and extension name) to a logical type built for the new class. + Earlier class objects keep their mapping, so instances created before the re-run still + convert. Arrow's global registration for that extension name is reused (same name, + storage type and metadata by construction). This fixes W1, W3, W4. +- **Different identity under the same name → a different type.** Nothing is rebound; + the new class gets its own logical type and extension name (W2: the next run computes + into new tables). +- **Still raises:** + 1. One class object registered under two names (§b). + 2. An invalid declared name (§a). + 3. A local class (``), as today. + 4. A shape disagreement on read (§d, S2). + +## d. Read path — CONFIRMED behaviour (S2), lookup mechanism proposed + +A stored column carries the extension name `Point@9117beaaf7ebe3ab` and the record in its +metadata. To build values, the reader resolves a class: + +1. **Registered identity:** the extension name is registered in the reading context → use it. +2. **Declared or registered by name:** classes recorded under `(kind, name)` in this + process (by `register_type` or by first use); take the one whose identity string + matches. +3. **`__main__` lookup:** walk the name as an attribute path from + `sys.modules["__main__"]` (`"Point"`, `"Outer.Point"`), accepted if its identity matches. + This makes W12 (read notebook results from a script that defines the same `Point`) + work without a declaration. +4. **Import** (dotted name): import by the longest module prefix, as today, accepted if + its identity matches. + +Outcomes: + +- **No candidate found:** `ImportError` (kept), naming the declared name, the kind, the + stored fields, and how to fix it: "define the class and register it with + `op.register_type(Point, name='Point')` before reading". +- **Candidates found, none with a matching identity (S2):** `TypeIdentityError` + (new, a `ValueError` subclass in `orcapod.errors`) naming the column where the read + path knows it, the class name and kind, the stored fields and the registered fields, + for example "column 'p': stored Point (dataclass) has fields {x: int, y: int}; the + registered Point has {x: int, y: float}". +- **Same identity, different Arrow storage** (possible only if two data contexts map one + annotation to different Arrow types): `TypeIdentityError` naming the column, the class + and both struct types. The existing check in the synthesized + `__arrow_ext_deserialize__` raises it. +- These errors propagate. `DeltaTableDatabase` today catches every read error and logs + "Error reading Delta table"; it re-raises `TypeIdentityError`, so a run can no longer + report success with 0 rows. Other read errors keep their current handling (out of + scope). The misleading "data may have been deleted externally" message is untouched + unless the path is reached for this reason (§g lists it as out of scope). +- **Pre-0.3 column** (D-legacy, PENDING): an extension name without `@` and metadata + without `identity`, read directly (pipelines never reach these tables, §e): raise + `TypeIdentityError` saying the column was written by Orcapod before 0.3 under + `pkg.models.Point`, that v0.3 identifies the class by name and fields, and that the + pipeline recomputes into new tables. Not converted, not dropped. + +## e. Intended hash changes — PENDING (developer approval of the list) + +Every pinned value that involves a dataclass or pydantic type moves, because the identity +string changes for every user class (fields are now in it, for everyone). Measured on +`origin/main` (c5ab1e05) with E08 (`--ref origin/main`): changing the identity string of the +golden fixtures `GoldenPoint` and `GoldenModel` moves exactly these 25 entries, and the +`unchanged` control moves 0. Every identity-string change moves the same set (E08's +`__main__` and module-less variants list identical entries), so the v0.3 string moves +these and only these. The implementation re-runs the golden test and the PR lists the +measured diff; any entry outside this list is a bug. + +| File | Entries | +|---|---| +| `semantic.json` (2) | `type.dataclass`, `type.pydantic` | +| `schema.json` (5) | `multi_field.logical_mix`, `single_field.dataclass.arrow_type`, `single_field.dataclass.schema_hash`, `single_field.pydantic.arrow_type`, `single_field.pydantic.schema_hash` | +| `datagram.json` (10 change) | `{dataclass,pydantic}.content_hash`, `.pipeline_hash`, `.system_tag_columns`, `.row[{"k":"r1"}].data_hash`, `.row[{"k":"r2"}].data_hash` | +| `datagram.json` (8 replaced) | `{dataclass,pydantic}.row[{"k":"r1"/"r2"}]._tag_record_id::` and `._tag_source_id::`: the keys embed the moved pipeline hash, so these 8 keys disappear and 8 keys with the new hash appear | + +`arrow.json` and `pipeline.json` have no user-class entries and do not move. Golden +additions (new keys only): a declared-name dataclass, a nested dataclass (P1), an +`Optional` field, a container field, and permuted-field-order twins that must pin the +same value. + +**For the CHANGELOG (stated plainly):** "Orcapod 0.3 identifies a dataclass or pydantic +model by its name, its kind and its fields. Every pipeline whose data includes such a +class gets new pipeline hashes and table paths, so the first run under 0.3 recomputes +those pod and operator results into new tables. Tables written by earlier versions are +left untouched and are not read by 0.3 pipelines. There is no migration tool. Classes +defined in a notebook or script (`__main__`) are now identified by their name only, so +results survive moving the class into a module if it keeps its name and is registered +with `op.register_type(name=...)`." Results that do not involve user classes are +unaffected. + +## f. Interactions with PR #286 and PR #289 + +- **PR #286** (polars types) touches `builtin_handlers.py`, `logical_types/__init__.py`, + `contexts/data/v0.1.json`, `docs/concepts/function-pods.md`, the golden `cases.py` and + `semantic.json`/`schema.json`/`datagram.json`. This work does not touch + `builtin_handlers.py` or `v0.1.json` (the identity reaches hashing through the + logical type name), adds a new docs page instead of editing `function-pods.md` beyond a + link, and edits the golden files only by regeneration. +- **PR #289** (`orcapod.ndarray`) adds `array_format.py` and tests; no file in common. +- **Preferred order:** #286 and #289 merge first. This branch then merges `origin/main`, + re-runs E08 against the new main to confirm the moved set, and regenerates the golden + files; the PR's golden diff must then show only the §e entries plus the additions. If + this PR merges first, #286 regenerates its own additive entries after merging main; + its user-class-free entries do not move. + +## g. Scope cuts, in order, if time runs short + +1. TOML/config-file setting for the naming defaults (`set_type_naming` stays). +2. The one-time `__main__` notice (S3, D-notice). +3. The collision `UserWarning` for same identity from a different `module.qualname` + (behaviour stays: rebind). +4. The `__main__` lookup on read (§d step 3); a declaration is then required to read + `__main__`-named data in another process. +5. `set_type_naming` itself (defaults fixed: `main="name"`, `module="qualified"`). + +Not cut: the identity record and string, rebinding, `register_type`, the S2 raise, the +read lookup by declared name, the docs page and the CHANGELOG entry. + +Out of scope for v0.3: computed fields; local classes with a declared name; Enum, +NamedTuple and TypedDict support (D6 of the findings); the `DeltaTableDatabase` row-count +and shutdown-hang findings (§9 of the findings). + +## Workshop scenarios after v0.3 (predicted, become the Step 2 tests) + +| # | Scenario | v0.3 | +|---|---|---| +| W1/W3 | Re-run the cell, same fields (dataclass / pydantic) | silent rebind; cache hit | +| W2 | Re-run with changed fields | new type; recompute into new tables; old tables untouched | +| W4 | `%autoreload` of a module class | silent rebind | +| W5/W9 | Restart / second notebook, identical class | cache hit | +| W6/W10 | Restart / second notebook, changed fields | clean recompute, new tables; no 0-row success | +| W7 | Script as `__main__` twice | cache hit | +| W8 | `python -m pkg.pipeline` then `import pkg.pipeline` | miss unless declared (`"Point"` vs `"pkg.pipeline.Point"`); hit with `@op.register_type(name="Point")` | +| W11 | Notebook → module | hit if the module class declares the same name (`"Point"`) | +| W12 | Read notebook results from a script | works if the script defines a matching `Point` (top level or declared); otherwise the `ImportError` of §d | +| W14 | spawn child (`__mp_main__`) | same identity as `__main__` | +| W15 | Local class | `ValueError`, as today | From fae9ba720c2afea7dc92dc9a360b201f9259a7b2 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:12:23 -0800 Subject: [PATCH 002/147] docs(specs): record P1, registration order and cycle rules for user class identity (ITL-881) Refs ITL-881 Co-Authored-By: Claude Opus 5.5 --- .../2026-11-10-user-class-identity-v0.3.md | 56 ++++++++++++++----- 1 file changed, 42 insertions(+), 14 deletions(-) diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md index 32d87c730..d5aeef7fb 100644 --- a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -1,6 +1,6 @@ # Identity of a user class in v0.3 (ITL-881) -**Status:** implementation spec, waiting for approval. No product code changes yet. +**Status:** implementation spec, approved by the coordinator November 10, 2026 (P1 decided by the developer; P2 names and the identity-string format provisional). **Issue:** ITL-881 (v0.3, a public release; tag target November 16, 2026). **Builds on:** the findings in PR #288 (`superpowers/specs/2026-11-10-user-class-identity.md`, experiments E01–E08). "W1"–"W15" are the workshop scenarios from §3.4 of that document. @@ -17,11 +17,12 @@ Each decision is marked **CONFIRMED** (the developer decided it, November 10, 20 | D-rebind | Same identity re-registered: rebind silently | CONFIRMED: developer answer to S1 | | D-raise | Stored shape disagreeing with registered shape raises, in v0.3 | CONFIRMED: developer answer to S2 | | D-doc | A page documenting identity | CONFIRMED: developer answer to S4 | -| P1 | A field typed by another user class carries that class's full identity | PENDING: developer asked; default stated by the coordinator | -| P2 | Public API and setting names | PENDING: developer to settle the names | -| D-hash | The 25 golden entries in §e move; nothing else | PENDING: needs the developer's approval of the list (golden rule) | -| D-legacy | Reading a pre-0.3 user-class column directly raises a clear error | PENDING: stored-data behaviour, proposed here | -| D-notice | One-time notice when a `__main__` class is first registered (S3) | PENDING: developer leaned towards a warning; first scope cut | +| P1 | A field typed by another user class carries that class's identity as that class was registered | CONFIRMED: developer decision, November 10, 2026 | +| D-cycle | A self-referential or cyclic class is rejected with a clear `TypeError` | CONFIRMED by the coordinator from this proposal: Arrow cannot store a recursive struct | +| P2 | Public API names (`register_type`, `set_type_naming`) and the identity-string format | PENDING with the developer; implemented provisionally, each a single-point change | +| D-hash | The 25 golden entries in §e move; nothing else | CONFIRMED by the coordinator: follows from decision 2; the PR lists every moved entry | +| D-legacy | Reading a pre-0.3 user-class column directly raises a clear error | CONFIRMED by the coordinator: a consequence of the identity check; nothing extra is built | +| D-notice | One-time notice when a `__main__` class is first registered (S3) | CONFIRMED in scope by the coordinator; last item before the cuts | ## a. The identity definition @@ -55,7 +56,7 @@ A name must be non-empty, must not contain `@` or whitespace, and must not start or `"pydantic"` (`pydantic.BaseModel` subclass). A dataclass and a pydantic model with the same name and fields have different identity records, so they are different types. -### Field component — CONFIRMED (decision 2), nested classes PENDING (P1) +### Field component — CONFIRMED (decision 2; nested classes: P1) `fields` maps each **stored field's name** to its **type token**. It is a JSON object serialized with sorted keys, so field order never enters. The stored fields are exactly @@ -71,8 +72,30 @@ extras): | type with a built-in logical type | its logical type name | `orcapod.path`, `orcapod.uuid`, `orcapod.file` | | generic container | `origin[arg, …]`, origin by its canonical name, args in order | `list[int]`, `dict[str, float]`, `tuple[int, str]` (`typing.List[int]` gives `list[int]`) | | `Optional` / union | member tokens sorted, joined by ` \| ` | `int \| None` for `Optional[int]`, `int \| None` and `None \| int` | -| nested dataclass / pydantic model (**P1 default**) | that class's identity string (below) | `Inner@3c1d…` | -| nested class (**P1 alternative**) | its fields only: `struct{a: int, b: str}` (sorted, no name, no kind) | — | +| nested dataclass / pydantic model (P1) | that class's identity string (below), as that class is registered | `Inner@3c1d…`, `geo.Inner@…` | + +**Nested classes (P1, CONFIRMED).** The token of a field typed by another user class is +that class's own registered identity string: its name component however *that* class was +registered (declared name, name only, or `module.qualname`), its kind, and its field +component, recursively through the fingerprint. It is never re-derived from +`module.qualname` and never reduced to the storage struct. So renaming the inner class, +declaring it under another name, or changing its fields changes the outer identity. + +**Registration order does not matter.** `register_type` only records the declared name +and checks it (§b); the logical type, and so the identity, is built at first use, and the +outer class's build registers the inner class first (implicitly, by the setting, if it +was never declared). Declaring every class before first use therefore gives the same +identities in any order: outer decorated before inner, inner declared later in the same +cell, inner registered implicitly by the outer's first use. Declaring a class after it +was used under a different name raises (§c). An identical notebook re-run of the inner +class rebinds it (§c) and leaves the outer identity unchanged. + +**Cycles (D-cycle).** A class that refers to itself, directly (`next: Node | None`) or +through other classes, is rejected at first use with a `TypeError` naming the cycle +(`Node.next -> Node`). Today's converter already raises `TypeError: Circular type +dependency detected…`; v0.3 keeps the rejection and names the path. Encoding the +back-reference by name would give a well-defined identity, but Arrow cannot store a +recursive struct, so there is nothing to identify. Deliberately **not** part of the identity: defaults and default factories, methods and validators, properties and pydantic `computed_field`s, `init=False` dataclass fields, @@ -146,10 +169,13 @@ op.set_type_naming(main="name", module="qualified") # process-wide defaults (s ``` - `register_type` records the name for the class object (a process-wide weak-keyed table, - so the class is not modified), validates it, builds its logical type in the default - data context (so an unsupported field type fails at the definition, not at first use) - and returns the class unchanged. Classes that are never registered are registered on - first use, as today, with the name from the setting. + so the class is not modified), checks the kind, the name and that the class is not + local, and returns the class unchanged. The logical type is built at first use, as + today, so declarations may come in any order (§a). Classes that are never declared are + registered on first use with the name from the setting. +- Provisional (P2): the public names live in one place (`orcapod/user_types.py`, + re-exported from `orcapod`) and the identity-string format in one function + (`identity_string(record)`), so a rename or a format change is a single-point edit. - `set_type_naming` changes the default for classes registered after the call. Values: `"name"` or `"qualified"`. Defaults: `main="name"`, `module="qualified"`. No TOML key in v0.3 (scope, §g). @@ -176,7 +202,9 @@ op.set_type_naming(main="name", module="qualified") # process-wide defaults (s the new class gets its own logical type and extension name (W2: the next run computes into new tables). - **Still raises:** - 1. One class object registered under two names (§b). + 1. One class object registered under two names (§b), including a declaration after + the class was already used under its default name. + 5. A self-referential or cyclic class (D-cycle). 2. An invalid declared name (§a). 3. A local class (``), as today. 4. A shape disagreement on read (§d, S2). From 394f60f83803ae069521c37c398af66e020bfd91 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:15:05 -0800 Subject: [PATCH 003/147] docs(plans): implementation plan for v0.3 user class identity (ITL-881) Also records in the spec that a subclass never borrows its parent's identity. Refs ITL-881 Co-Authored-By: Claude Opus 5.5 --- ...-11-10-itl-881-user-class-identity-impl.md | 184 ++++++++++++++++++ .../2026-11-10-user-class-identity-v0.3.md | 6 + 2 files changed, 190 insertions(+) create mode 100644 superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md diff --git a/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md new file mode 100644 index 000000000..caf332c3e --- /dev/null +++ b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md @@ -0,0 +1,184 @@ +# User class identity (v0.3) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Identify a dataclass or pydantic model by `{kind, name, fields}` (spec +`superpowers/specs/2026-11-10-user-class-identity-v0.3.md`), with a declared-name API, silent +rebinding of identical re-registrations, and a raise on stored/registered shape mismatch. + +**Architecture:** One new module, `orcapod/logical_types/user_class_identity.py`, owns the +naming policy, declarations, type tokens, the identity record and string (single-point +format), the process-wide `(kind, name)` table and the read-side class lookup. The two +factories share a new base (`user_class_logical_type.py`) that builds the identity string +and uses it as logical type name and Arrow extension name; the metadata carries the record. +`LogicalTypeRegistry` learns to rebind an identical identity to a new class object and stops +matching user classes by subclass. The public API (`register_type`, `set_type_naming`) lives +in `orcapod/user_types.py`. `TypeIdentityError` lives in `orcapod/errors.py`; the Delta read +path re-raises it. + +**Tech Stack:** Python 3.11–3.13, pyarrow, polars, pydantic v2, deltalake, IPython (dev, for +the notebook scenarios), pytest via `uv run`. + +--- + +## File map + +| File | Change | +|---|---| +| `src/orcapod/errors.py` | add `TypeIdentityError(ValueError)` | +| `src/orcapod/logical_types/user_class_identity.py` | **new**: kinds, naming policy, declarations, `type_token`, `identity_record`, `canonical_json`, `identity_string`, known-class table, `find_class`, `describe_fields`, S3 notice | +| `src/orcapod/logical_types/user_class_logical_type.py` | **new**: `UserClassLogicalType`, `UserClassLogicalTypeFactory` (shared write/read path) | +| `src/orcapod/logical_types/dataclass_logical_type_factory.py` | reduce to subclasses: kind, category, `supports_class`, stored fields | +| `src/orcapod/logical_types/pydantic_logical_type_factory.py` | same | +| `src/orcapod/logical_types/registry.py` | rebinding rule; exact-type match for user classes; `TypeIdentityError` from `_deserialize` | +| `src/orcapod/semantic_types/universal_converter.py` | cycle error names the class and says recursive classes cannot be stored | +| `src/orcapod/databases/delta_lake_databases.py` | re-raise `TypeIdentityError` where read errors are logged | +| `src/orcapod/logical_types/database_hooks.py` | add the column name to a `TypeIdentityError` raised while registering a discovered type | +| `src/orcapod/user_types.py` | **new**: `register_type`, `set_type_naming` | +| `src/orcapod/__init__.py` | re-export the two functions | +| tests | `tests/test_user_types/` (new): unit tests of identity, API, registry, read path; `test_workshop_scenarios.py` (subprocess + IPython `InteractiveShell`) | +| existing tests asserting `module.qualname` names | switch to the identity string | +| `tests/test_hashing/golden/cases.py` + `data/*.json` | add new fixture cases; regenerate (25 listed moves + additions) | +| docs | new `docs/concepts/your-own-data-types.md`; section in `docs/concepts/identity.md`; nav in `mkdocs.yml`; link from `function-pods.md` | +| `CHANGELOG.md`, `DESIGN_ISSUES.md` | entry each | + +## Key definitions (used by every task) + +```python +# user_class_identity.py +KIND_DATACLASS = "dataclass" +KIND_PYDANTIC = "pydantic" +_NAMING = {"main": "name", "module": "qualified"} # set_type_naming +_MAIN_MODULES = frozenset({"__main__", "__mp_main__"}) +_DECLARED: weakref.WeakKeyDictionary[type, str] # register_type +_KNOWN: dict[tuple[str, str], list[weakref.ref[type]]] # (kind, name) -> classes + +def validate_name(name: str) -> None: ... # non-empty, no '@', no whitespace, not 'orcapod.' +def class_kind(cls: type) -> str | None: ... +def resolve_name(cls: type) -> str: ... # declared > main policy > module policy +def declare(cls: type, name: str | None) -> None: ... # raises if already built under another name +def set_naming(main: str | None, module: str | None) -> None: ... +def type_token(annotation, converter) -> str: ... +def identity_record(kind, name, fields: dict[str, str]) -> dict: ... +def canonical_json(obj) -> str: ... # sort_keys, (",", ":") +def identity_string(record) -> str: ... # f"{name}@{sha256(canonical_json)[:16]}" (single point) +def describe_fields(fields: dict[str, str]) -> str: ... # "{x: int, y: float}" +def remember(kind, name, cls) -> None: ... # _KNOWN +def candidates(kind, name) -> list[type]: ... # _KNOWN + declared + __main__ walk + import +``` + +Type tokens: builtins bare (`int`); `NoneType` → `None`; other classes without logical type → +`canonical_type_name`; class with logical type → `lt.logical_type_name` (built-ins +`orcapod.path`, user classes their identity string, P1); `list[...]`/`set[...]`/`dict[...]` +→ `origin[args]` with origin by canonical name minus `builtins.`; unions → sorted members +joined by ` | `; `Literal` → `Literal[...]` with sorted `repr`s. + +--- + +### Task 1: Identity primitives (pure functions) + +**Files:** create `src/orcapod/logical_types/user_class_identity.py`, modify +`src/orcapod/errors.py`; test `tests/test_user_types/test_identity_record.py`. + +- [ ] Step 1: failing tests: `identity_string({"kind":"dataclass","name":"Point","fields":{"x":"int","y":"float"}}) == "Point@9117beaaf7ebe3ab"`; field-order independence; `type_token` for `int`, `int | None`, `Optional[int]`, `None | int`, `list[int]`, `typing.List[int]`, `dict[str, float]`, `Path` (`orcapod.path`), `datetime.datetime`, `Literal["b","a"]`; `validate_name` rejects `""`, `"a@b"`, `"a b"`, `"orcapod.x"`; `resolve_name` for a module class (`tests.…Point` qualified), a class with `__module__="__main__"` (`"Point"`), `"__mp_main__"` (same), after `set_naming(main="qualified")` (`"__main__.Point"`), after `set_naming(module="name")`. +- [ ] Step 2: run, expect ImportError (module missing). +- [ ] Step 3: implement the functions above. +- [ ] Step 4: run, pass. Commit `feat(types): identity record and string for user classes (ITL-881)`. + +### Task 2: Shared user-class logical type; factories use the identity + +**Files:** create `src/orcapod/logical_types/user_class_logical_type.py`; reduce both factory +modules to subclasses; test `tests/test_user_types/test_user_class_logical_type.py`. + +`UserClassLogicalType(identity, record, python_type, storage_type, field_annotations, category)`: +logical name = Arrow extension name = identity string; metadata +`canonical_json({"category": category, "identity": record}).encode()`; conversions as today. +Class attribute `exact_python_type_only = True`. + +`UserClassLogicalTypeFactory` (abstract: `kind`, `category`, `supports_class`, `stored_fields(cls, hints)`): +- `create_for_python_type(cls, converter)`: reject ``; name = `resolve_name(cls)`; + for each stored field: `arrow = converter.register_python_class(annotation)` (wrap a + circular `TypeError` as `TypeError(f"{cls.__qualname__}.{field}: recursive classes cannot be stored ({exc})")`); + token = `type_token(annotation, converter)`; record; identity; `remember(kind, name, cls)`; + S3 notice when the name came from the `__main__` policy (once per name per process). +- `reconstruct_from_arrow(ext_name, storage, metadata, converter)`: no `identity` key → + `TypeIdentityError` (pre-0.3 column); else for each candidate class of (kind, name): get its + logical type through `converter.register_python_class(cls)` + + `converter.get_logical_type(cls)`; return the first whose `logical_type_name == ext_name` + (and whose storage equals the stored storage after `normalize_to_large_types`, else + `TypeIdentityError` naming both struct types). No candidate → `ImportError` naming name, + kind, stored fields and `op.register_type(Point, name='Point')`. Candidates but no match → + `TypeIdentityError` naming class, kind, stored fields, registered fields. + +- [ ] Step 1: failing tests: logical name is the identity string; metadata contains the record; two classes with permuted fields share the identity; dataclass vs pydantic same name+fields differ; same name, `y: int` vs `y: float` differ; nested dataclass token is the inner identity string and changes when the inner fields change or the inner is declared under another name (P1); outer built before inner declared gives the same identity as inner-first (order); self-referential class → `TypeError` naming `Node.next`; `` → `ValueError`; subclass with an extra field gets its own identity (exact-type match). +- [ ] Step 2: run, fail. Step 3: implement. Step 4: update the existing factory/roundtrip tests that assert `module.qualname` (listed in the file map) to read the identity from the registry. Step 5: run `uv run pytest tests/test_logical_types tests/test_user_types -q`, pass. Commit. + +### Task 3: Registry rebinding and exact-type match + +**Files:** `src/orcapod/logical_types/registry.py`; test `tests/test_user_types/test_rebinding.py`. + +- `register_logical_type`: when the logical name is bound to a different instance whose + Arrow extension type has the same name, storage type and serialized metadata, and the + Python type differs → rebind: replace the logical-name and Arrow-name bindings, add the new + Python-type binding, keep the old Python-type binding. If the previous class's + `module.qualname` differs from the new one's → one `UserWarning` naming both. Anything + else keeps raising. +- `get_by_python_type`: skip the `issubclass` fallback for logical types with + `exact_python_type_only`. +- `_deserialize`: raise `TypeIdentityError` (subclass of `ValueError`) for storage/metadata mismatch. + +- [ ] Steps: failing tests (rebind same identity silently, `warnings` none; different module.qualname same identity → one `UserWarning`; one class object under two names → `ValueError`), implement, pass, commit. + +### Task 4: Public API + +**Files:** create `src/orcapod/user_types.py`; modify `src/orcapod/__init__.py`; test +`tests/test_user_types/test_register_type.py`. + +```python +def register_type(cls=None, /, *, name=None): # decorator, decorator factory, or call +def set_type_naming(*, main=None, module=None) -> None: +``` + +- [ ] Steps: failing tests through `op.register_type` / `op.set_type_naming` and a `DictSource` + `function_pod` pipeline: declared name gives the identity `Name@…`; `register_type(Point, name="Geo")` after first use under another name → `ValueError`; invalid names; non-dataclass → `TypeError`; returns the class. Implement, pass, commit. + +### Task 5: Read path and S2 guard + +**Files:** `delta_lake_databases.py`, `database_hooks.py`; test +`tests/test_user_types/test_read_path.py` (public interface: pipeline in a subprocess writes +to `DeltaTableDatabase`, a second subprocess reads with a different class). + +- [ ] Steps: failing tests: reader with matching class (declared in another module) reads values; reader without the class → `ImportError` naming `Point` and `register_type`; reader whose `Point` has different fields reading the stored table directly → `TypeIdentityError` naming column, both field sets; re-run of the pipeline with changed fields computes into new tables (calls == rows), old table still readable by the old class. Implement re-raise, pass, commit. + +### Task 6: Workshop scenarios W1–W12 as tests + +**File:** `tests/test_user_types/test_workshop_scenarios.py` (each scenario in a fresh +subprocess; notebook cells through `IPython.core.interactiveshell.InteractiveShell`, as E02; +children end with `os._exit(0)` after printing JSON, as the E-harness does). + +W1 re-run same fields (calls 0 on second run, no error); W2 changed fields (new tables: +calls 2, rows 2, no error); W3 pydantic re-run; W4 `importlib.reload`; W5 restart + re-run +(two processes, cache hit); W6 restart with changed fields (recompute, rows 2); W7 script as +`__main__` twice; W8 `-m` vs import with a declared name (hit); W11 notebook → module with a +declared name: identical pipeline hashes and table paths; W12 read notebook results from a +script defining the same `Point`. Plus: pydantic vs dataclass same name+fields are distinct +(different pod pipeline hashes). + +- [ ] Steps: write, run (must fail before Tasks 1–5 on `main` for W1–W4, W6, W11, W12; record which), pass after, commit. + +### Task 7: Golden values + +- [ ] Add fixture cases to `cases.py` (new names only; nothing moved): declared-name dataclass, nested dataclass, `Optional` field, container field, permuted-order twin pair. +- [ ] Run `uv run pytest tests/test_hashing/golden -q`: exactly the 25 spec entries fail. Record the list. +- [ ] Regenerate; `git diff --stat` and a script comparing old/new JSON: changed keys == the 25, plus only added keys. Commit `test(hashing): golden values for v0.3 user class identity (ITL-881)`. + +### Task 8: Docs, CHANGELOG, DESIGN_ISSUES + +- [ ] `docs/concepts/your-own-data-types.md` (runnable samples: define, declare, re-run, what changes identity), identity section in `docs/concepts/identity.md`, nav entry, link from `function-pods.md`. Run `uv run pytest tests/test_docs -q`. +- [ ] CHANGELOG entry with the hash-change statement from spec §e. +- [ ] DESIGN_ISSUES entry appended at the end (resolved, with Fix note). + +### Task 9: Verification and PR + +- [ ] ruff check/format, basedpyright exit code, full `uv run pytest tests/` on 3.11/3.12/3.13 (`uv run --python 3.1x`). +- [ ] Merge `origin/main` (after #286/#289 land), re-run E08 on the new main, regenerate golden only for the listed entries. +- [ ] PR body: intended hash changes list; mark ready via sensei:create-pr. diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md index d5aeef7fb..96252c9ed 100644 --- a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -56,6 +56,12 @@ A name must be non-empty, must not contain `@` or whitespace, and must not start or `"pydantic"` (`pydantic.BaseModel` subclass). A dataclass and a pydantic model with the same name and fields have different identity records, so they are different types. +**Subclasses.** Each class has its own identity; a subclass never borrows its parent's. +Today the registry matches an unregistered class to a registered *parent* by +`issubclass`, so `class P3(Point)` with an extra field `z` is stored as `Point` and `z` is +silently dropped (reproduced on `main`, November 10, 2026). v0.3 matches dataclass and +pydantic logical types by exact class only. + ### Field component — CONFIRMED (decision 2; nested classes: P1) `fields` maps each **stored field's name** to its **type token**. It is a JSON object From 27d4a9e31262b44719fe512c6db8ed19ecbe7fb1 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:16:51 -0800 Subject: [PATCH 004/147] feat(types): identity record, string and naming policy for user classes (ITL-881) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/errors.py | 11 + .../logical_types/user_class_identity.py | 446 ++++++++++++++++++ tests/test_user_types/__init__.py | 0 tests/test_user_types/test_identity_record.py | 143 ++++++ 4 files changed, 600 insertions(+) create mode 100644 src/orcapod/logical_types/user_class_identity.py create mode 100644 tests/test_user_types/__init__.py create mode 100644 tests/test_user_types/test_identity_record.py diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 81c653627..94305e06a 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -20,6 +20,17 @@ class SchemaInconsistencyError(InputValidationError): """ +class TypeIdentityError(ValueError): + """Raised when stored values and a registered user class disagree on identity. + + A stored dataclass or pydantic column records its class's identity (name, kind and + field types). This error is raised when the class found for that name has a + different identity or a different Arrow storage type, or when the column was + written by an Orcapod version that did not record the identity. The message names + the column where it is known, the class, and both shapes. + """ + + class DuplicateTagError(ValueError): """Raised when duplicate tag values are found and skip_duplicates=False""" diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py new file mode 100644 index 000000000..5f95454ee --- /dev/null +++ b/src/orcapod/logical_types/user_class_identity.py @@ -0,0 +1,446 @@ +"""Identity of user classes (dataclasses and pydantic models). + +A user class is identified by an *identity record* with three components (ITL-881, +spec ``superpowers/specs/2026-11-10-user-class-identity-v0.3.md``): + +- ``kind``: ``"dataclass"`` or ``"pydantic"``. +- ``name``: a declared name, or a name chosen by the naming policy (the qualified name + for ``__main__`` classes is just ``__qualname__`` by default; module classes use + ``module.qualname`` by default). +- ``fields``: a mapping from each stored field's name to its *type token*. It is + serialized with sorted keys, so field order never enters the identity. + +The *identity string* ``f"{name}@{fingerprint}"`` is the class's logical type name and +Arrow extension name. ``identity_string`` is the only place that defines its format. + +This module also holds the process-wide state that belongs to classes rather than to a +data context: declared names, the naming policy and the table of classes seen under each +``(kind, name)``, which the read path uses to find a class for a stored column. +""" + +from __future__ import annotations + +import dataclasses +import hashlib +import importlib +import json +import re +import sys +import typing +import warnings +import weakref +from typing import TYPE_CHECKING, Any, Literal + +from orcapod.utils.canonical_type_names import canonical_type_name + +if TYPE_CHECKING: + from orcapod.logical_types.protocols import TypeConverterProtocol + +KIND_DATACLASS = "dataclass" +KIND_PYDANTIC = "pydantic" + +NamingStyle = Literal["name", "qualified"] + +_NAMING_STYLES: frozenset[str] = frozenset({"name", "qualified"}) +_DEFAULT_NAMING: dict[str, NamingStyle] = {"main": "name", "module": "qualified"} +_naming: dict[str, NamingStyle] = dict(_DEFAULT_NAMING) + +#: Modules whose classes are "defined in a notebook or script". ``__mp_main__`` is the +#: name a ``multiprocessing`` spawn child gives the main module. +MAIN_MODULES: frozenset[str] = frozenset({"__main__", "__mp_main__"}) + +#: Prefix reserved for Orcapod's own types (``orcapod.path``, ``orcapod.uuid``, ...). +RESERVED_PREFIX = "orcapod." + +#: Number of hex digits of the SHA-256 fingerprint in an identity string. +FINGERPRINT_HEX_DIGITS = 16 + +_NAME_FORBIDDEN = re.compile(r"[@\s]") + +# Declared names, keyed by class object (weak, so classes from re-run cells can go away). +_declared: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() +# The name each class object was built under; a class has one name per process. +_built: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() +# Classes seen under each (kind, name), most recent last. +_known: dict[tuple[str, str], list[weakref.ref[type]]] = {} +# Names that already produced the one-time ``__main__`` notice. +_noticed_main_names: set[str] = set() + + +# --------------------------------------------------------------------------- +# Kinds and names +# --------------------------------------------------------------------------- + + +def class_kind(cls: Any) -> str | None: + """Return the kind of a user class, or ``None`` if it is not one. + + Args: + cls: Any object. + + Returns: + ``"dataclass"`` for a dataclass type (including pydantic dataclasses), + ``"pydantic"`` for a ``pydantic.BaseModel`` subclass, else ``None``. + """ + if not isinstance(cls, type): + return None + if dataclasses.is_dataclass(cls): + return KIND_DATACLASS + from pydantic import BaseModel + + if issubclass(cls, BaseModel): + return KIND_PYDANTIC + return None + + +def validate_name(name: str) -> None: + """Check that ``name`` can be used as the name of a user class. + + Args: + name: The proposed name. + + Raises: + TypeError: If ``name`` is not a string. + ValueError: If ``name`` is empty, contains ``@`` or whitespace, or starts with + the reserved ``orcapod.`` prefix. + """ + if not isinstance(name, str): + raise TypeError(f"A type name must be a string, got {name!r}.") + if not name: + raise ValueError("A type name must not be empty.") + if _NAME_FORBIDDEN.search(name): + raise ValueError( + f"Invalid type name {name!r}: a name must not contain '@' or whitespace." + ) + if name.startswith(RESERVED_PREFIX): + raise ValueError( + f"Invalid type name {name!r}: the prefix {RESERVED_PREFIX!r} is reserved " + f"for Orcapod's own types." + ) + + +def get_naming() -> dict[str, NamingStyle]: + """Return the current naming policy as ``{"main": ..., "module": ...}``.""" + return dict(_naming) + + +def set_naming( + main: NamingStyle | None = None, module: NamingStyle | None = None +) -> None: + """Set how classes that declare no name are named. + + Applies to classes registered after the call. + + Args: + main: ``"name"`` (default) or ``"qualified"`` for classes defined in + ``__main__`` (a notebook or a script run directly). + module: ``"qualified"`` (default) or ``"name"`` for classes defined in an + importable module. + + Raises: + ValueError: If a value is not ``"name"`` or ``"qualified"``. + """ + for key, value in (("main", main), ("module", module)): + if value is None: + continue + if value not in _NAMING_STYLES: + raise ValueError( + f"Invalid naming style for {key}: {value!r}. " + f"Use 'name' or 'qualified'." + ) + _naming[key] = value + + +def is_main_class(cls: type) -> bool: + """Return True if ``cls`` was defined in ``__main__`` (or ``__mp_main__``).""" + return cls.__module__ in MAIN_MODULES + + +def resolve_name(cls: type) -> str: + """Return the name component for ``cls``. + + The first rule that applies wins: a declared name; the naming policy for + ``__main__`` classes; the naming policy for module classes. + + Args: + cls: A dataclass or pydantic model class. + + Returns: + The name component of the class's identity. + """ + declared = _declared.get(cls) + if declared is not None: + return declared + style = _naming["main" if is_main_class(cls) else "module"] + if style == "name": + return cls.__qualname__ + module = cls.__module__ + if not is_main_class(cls): + from orcapod.utils.canonical_type_names import canonical_module_name + + module = canonical_module_name(cls) or module + return f"{module}.{cls.__qualname__}" + + +def declare(cls: type, name: str | None) -> None: + """Record the name a user class is identified by. + + Args: + cls: A dataclass or pydantic model class. + name: The declared name, or ``None`` to use the naming policy. + + Raises: + TypeError: If ``cls`` is not a dataclass or pydantic model class. + ValueError: If the class is local, the name is invalid, or the class was + already registered in this process under a different name. + """ + kind = class_kind(cls) + if kind is None: + raise TypeError( + f"Only dataclasses and pydantic models can be registered as types, " + f"got {cls!r}." + ) + reject_local_class(cls) + if name is not None: + validate_name(name) + effective = name if name is not None else resolve_name(cls) + built_as = _built.get(cls) + if built_as is not None and built_as != effective: + raise ValueError( + f"Cannot register {cls.__module__}.{cls.__qualname__} as {effective!r}: it " + f"is already registered as {built_as!r} in this process, because it was " + f"used before this declaration. Declare the name before the class is first " + f"used (for example with the @op.register_type decorator)." + ) + if name is not None: + _declared[cls] = name + remember(kind, effective, cls) + + +def reject_local_class(cls: type) -> None: + """Raise if ``cls`` is defined inside a function. + + Args: + cls: A class. + + Raises: + ValueError: If ``cls.__qualname__`` contains ````. + """ + if "" in cls.__qualname__: + raise ValueError( + f"Cannot register local class {cls!r}: a class defined inside a function " + f"cannot be found again when stored values are read. Define the class at " + f"module level." + ) + + +def mark_built(cls: type, name: str) -> None: + """Record that ``cls`` was built under ``name`` (one name per class per process).""" + _built[cls] = name + + +# --------------------------------------------------------------------------- +# Known classes per (kind, name) +# --------------------------------------------------------------------------- + + +def remember(kind: str, name: str, cls: type) -> None: + """Record ``cls`` as a class seen under ``(kind, name)``. + + Args: + kind: The class kind. + name: The name component. + cls: The class object. + """ + refs = _known.setdefault((kind, name), []) + refs[:] = [r for r in refs if r() is not None and r() is not cls] + refs.append(weakref.ref(cls)) + + +def known_classes(kind: str, name: str) -> list[type]: + """Return the live classes seen under ``(kind, name)``, most recent first.""" + classes = [r() for r in _known.get((kind, name), [])] + return [c for c in reversed(classes) if c is not None] + + +def candidate_classes(kind: str, name: str) -> list[type]: + """Return the classes that may be the class stored under ``(kind, name)``. + + In order: classes registered or declared under that name in this process, a class + reachable from ``__main__`` by that name, and a class importable by that dotted + name. Each candidate still has to match the stored identity. + + Args: + kind: The stored kind. + name: The stored name component. + + Returns: + Distinct candidate classes of the right kind. + """ + found: list[type] = list(known_classes(kind, name)) + for module_name in MAIN_MODULES: + module = sys.modules.get(module_name) + if module is not None: + obj = _walk_attributes(module, name.split(".")) + if obj is not None: + found.append(obj) + obj = _import_dotted(name) + if obj is not None: + found.append(obj) + result: list[type] = [] + for cls in found: + if class_kind(cls) == kind and cls not in result: + result.append(cls) + return result + + +def _walk_attributes(root: Any, parts: list[str]) -> Any: + obj = root + for part in parts: + obj = getattr(obj, part, None) + if obj is None: + return None + return obj + + +def _import_dotted(name: str) -> Any: + parts = name.split(".") + for split in range(len(parts) - 1, 0, -1): + module_name = ".".join(parts[:split]) + try: + module = importlib.import_module(module_name) + except ImportError: + continue + return _walk_attributes(module, parts[split:]) + return None + + +# --------------------------------------------------------------------------- +# Identity record and string +# --------------------------------------------------------------------------- + + +def identity_record(kind: str, name: str, fields: dict[str, str]) -> dict[str, Any]: + """Build the identity record of a user class. + + Args: + kind: ``"dataclass"`` or ``"pydantic"``. + name: The name component. + fields: Stored field name to type token. + + Returns: + ``{"kind": kind, "name": name, "fields": fields}`` with the fields sorted. + """ + return {"kind": kind, "name": name, "fields": dict(sorted(fields.items()))} + + +def canonical_json(obj: Any) -> str: + """Serialize ``obj`` as canonical JSON (sorted keys, no whitespace).""" + return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + + +def identity_string(record: dict[str, Any]) -> str: + """Return the identity string of an identity record. + + The format is ``f"{name}@{fingerprint}"``, where the fingerprint is the first + ``FINGERPRINT_HEX_DIGITS`` hex digits of the SHA-256 of ``canonical_json(record)``. + This is the only definition of the format. + + Args: + record: An identity record from ``identity_record``. + + Returns: + The identity string, e.g. ``"Point@9117beaaf7ebe3ab"``. + """ + digest = hashlib.sha256(canonical_json(record).encode("utf-8")).hexdigest() + return f"{record['name']}@{digest[:FINGERPRINT_HEX_DIGITS]}" + + +def describe_fields(fields: dict[str, str]) -> str: + """Render a field mapping for messages, e.g. ``{x: int, y: float}``.""" + return "{" + ", ".join(f"{k}: {v}" for k, v in sorted(fields.items())) + "}" + + +def describe_record(record: dict[str, Any]) -> str: + """Render an identity record for messages, e.g. ``Point (dataclass) {x: int}``.""" + return f"{record['name']} ({record['kind']}) {describe_fields(record['fields'])}" + + +# --------------------------------------------------------------------------- +# Type tokens +# --------------------------------------------------------------------------- + + +def type_token(annotation: Any, converter: TypeConverterProtocol) -> str: + """Return the type token of a field annotation. + + Builtins are written bare (``int``); classes with a logical type use its logical + type name (``orcapod.path``, or a nested user class's identity string); other + classes use their canonical name; generics are written ``origin[args]``; unions list + their members sorted, with ``None`` last. + + Args: + annotation: A field annotation (``Annotated`` extras already stripped). + converter: The type converter that resolves logical types. + + Returns: + The type token. + + Raises: + TypeError: If the annotation is not a type the token rules cover. + """ + import types as _types + + if annotation is None or annotation is type(None): + return "None" + origin = typing.get_origin(annotation) + args = typing.get_args(annotation) + if origin is typing.Union or origin is _types.UnionType: + members = [type_token(a, converter) for a in args] + non_none = sorted(m for m in members if m != "None") + return " | ".join(non_none + (["None"] if "None" in members else [])) + if origin is Literal: + return f"Literal[{', '.join(sorted(repr(a) for a in args))}]" + if origin is not None: + inner = ", ".join(type_token(a, converter) for a in args) + return f"{_class_token(origin)}[{inner}]" + if isinstance(annotation, type): + converter.register_python_class(annotation) + lt = converter.get_logical_type(annotation) + if lt is not None: + return lt.logical_type_name + return _class_token(annotation) + raise TypeError(f"Cannot build a type token for annotation {annotation!r}.") + + +def _class_token(cls: Any) -> str: + name = canonical_type_name(cls) + return name.removeprefix("builtins.") + + +# --------------------------------------------------------------------------- +# One-time notice for __main__ classes (S3) +# --------------------------------------------------------------------------- + + +def notice_main_class(cls: type, name: str) -> None: + """Warn once per name that a ``__main__`` class is identified by its name only. + + Args: + cls: The class being registered. + name: The name component it was given. + """ + if not is_main_class(cls) or cls in _declared or name in _noticed_main_names: + return + if name != cls.__qualname__: + return + _noticed_main_names.add(name) + warnings.warn( + f"{name} is defined in a notebook or script, so Orcapod identifies it by its " + f"name {name!r} and its fields. Results stored with it are found again by any " + f"class named {name!r} with the same fields, including one moved into a module " + f"and registered with @op.register_type(name={name!r}). Changing its fields " + f"makes a new type and recomputes.", + UserWarning, + stacklevel=2, + ) diff --git a/tests/test_user_types/__init__.py b/tests/test_user_types/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/test_user_types/test_identity_record.py b/tests/test_user_types/test_identity_record.py new file mode 100644 index 000000000..26f98ea0e --- /dev/null +++ b/tests/test_user_types/test_identity_record.py @@ -0,0 +1,143 @@ +"""Identity record, identity string, type tokens and naming policy of user classes (ITL-881). + +These tests target ``orcapod.logical_types.user_class_identity`` directly: they pin the +documented identity format, which is that module's own interface. +""" + +from __future__ import annotations + +import dataclasses +import datetime +import typing +from collections.abc import Iterator +from pathlib import Path +from typing import Literal, Optional + +import pytest + +from orcapod.contexts import get_default_context +from orcapod.logical_types import user_class_identity as uci + + +@pytest.fixture +def converter(): + return get_default_context().type_converter + + +@pytest.fixture(autouse=True) +def _restore_naming() -> Iterator[None]: + before = uci.get_naming() + yield + uci.set_naming(**before) + + +def _make_class(module: str, qualname: str = "Point") -> type: + cls = dataclasses.make_dataclass(qualname, [("x", int)]) + cls.__module__ = module + cls.__qualname__ = qualname + return cls + + +class TestIdentityString: + def test_documented_example(self) -> None: + record = uci.identity_record("dataclass", "Point", {"x": "int", "y": "float"}) + assert uci.identity_string(record) == "Point@9117beaaf7ebe3ab" + + def test_canonical_json_is_sorted_and_compact(self) -> None: + record = uci.identity_record("pydantic", "P", {"b": "int", "a": "str"}) + assert ( + uci.canonical_json(record) + == '{"fields":{"a":"str","b":"int"},"kind":"pydantic","name":"P"}' + ) + + def test_field_order_does_not_matter(self) -> None: + a = uci.identity_record("dataclass", "P", {"x": "int", "y": "str"}) + b = uci.identity_record("dataclass", "P", {"y": "str", "x": "int"}) + assert uci.identity_string(a) == uci.identity_string(b) + + @pytest.mark.parametrize( + "other", + [ + ("pydantic", "Point", {"x": "int", "y": "float"}), + ("dataclass", "Point2", {"x": "int", "y": "float"}), + ("dataclass", "Point", {"x": "int", "y": "int"}), + ("dataclass", "Point", {"x": "int", "z": "float"}), + ("dataclass", "Point", {"x": "int"}), + ], + ) + def test_each_component_changes_the_identity( + self, other: tuple[str, str, dict[str, str]] + ) -> None: + base = uci.identity_record("dataclass", "Point", {"x": "int", "y": "float"}) + assert uci.identity_string(base) != uci.identity_string( + uci.identity_record(*other) + ) + + def test_describe_fields(self) -> None: + assert uci.describe_fields({"y": "float", "x": "int"}) == "{x: int, y: float}" + + +class TestTypeTokens: + @pytest.mark.parametrize( + ("annotation", "token"), + [ + (int, "int"), + (str, "str"), + (bool, "bool"), + (bytes, "bytes"), + (float, "float"), + (datetime.datetime, "datetime.datetime"), + (Path, "orcapod.path"), + (int | None, "int | None"), + (Optional[int], "int | None"), + (None | int, "int | None"), + (list[int], "list[int]"), + (typing.List[int], "list[int]"), # noqa: UP006 - the typing alias is the case under test + (set[str], "set[str]"), + (dict[str, float], "dict[str, float]"), + (list[int | None], "list[int | None]"), + (Literal["b", "a"], "Literal['a', 'b']"), + ], + ) + def test_token(self, converter, annotation: object, token: str) -> None: + assert uci.type_token(annotation, converter) == token + + +class TestNames: + @pytest.mark.parametrize( + "name", ["", "a@b", "a b", "orcapod.thing", "tab\tname", 3] + ) + def test_invalid_names_rejected(self, name: object) -> None: + with pytest.raises((ValueError, TypeError)): + uci.validate_name(name) # pyright: ignore[reportArgumentType] + + @pytest.mark.parametrize("name", ["Point", "geo.Point", "my-types.Point_2"]) + def test_valid_names_accepted(self, name: str) -> None: + uci.validate_name(name) + + def test_module_class_defaults_to_qualified(self) -> None: + assert uci.resolve_name(_make_class("pkg.models")) == "pkg.models.Point" + + @pytest.mark.parametrize("module", ["__main__", "__mp_main__"]) + def test_main_class_defaults_to_name(self, module: str) -> None: + assert uci.resolve_name(_make_class(module)) == "Point" + + def test_nested_main_class_uses_qualname(self) -> None: + assert uci.resolve_name(_make_class("__main__", "Outer.Point")) == "Outer.Point" + + def test_main_qualified_setting(self) -> None: + uci.set_naming(main="qualified") + assert uci.resolve_name(_make_class("__main__")) == "__main__.Point" + + def test_module_name_setting(self) -> None: + uci.set_naming(module="name") + assert uci.resolve_name(_make_class("pkg.models")) == "Point" + + def test_invalid_setting_rejected(self) -> None: + with pytest.raises(ValueError, match="main"): + uci.set_naming(main="short") # pyright: ignore[reportArgumentType] + + def test_declared_name_wins(self) -> None: + cls = _make_class("pkg.models") + uci.declare(cls, "geo.Point") + assert uci.resolve_name(cls) == "geo.Point" From 5f12c0c151d438bbd5262007a7916fb70a045641 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:22:40 -0800 Subject: [PATCH 005/147] feat(types): identify dataclasses and pydantic models by name, kind and fields (ITL-881) The logical type name and Arrow extension name of a user class become its identity string, Name@, with the record in the extension metadata. Both factories share one write/read path; the read path finds the class by kind and name and checks the identity. The registry rebinds an unchanged identity to a re-defined class and no longer matches a user class's subclass to the parent's type. Golden values move as listed in the spec; regenerated in a later commit. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/errors.py | 9 + .../dataclass_logical_type_factory.py | 354 +---------------- .../pydantic_logical_type_factory.py | 368 ++---------------- src/orcapod/logical_types/registry.py | 66 +++- .../logical_types/user_class_identity.py | 24 +- .../logical_types/user_class_logical_type.py | 354 +++++++++++++++++ .../semantic_types/universal_converter.py | 8 +- .../test_logical_types/test_cache_behavior.py | 16 +- .../test_dataclass_logical_type_factory.py | 82 ++-- .../test_default_context_factories.py | 17 +- .../test_pydantic_logical_type_factory.py | 47 ++- tests/test_logical_types/test_roundtrips.py | 39 +- .../test_schema_compatibility.py | 17 +- tests/test_logical_types/test_type_utils.py | 12 - 14 files changed, 649 insertions(+), 764 deletions(-) create mode 100644 src/orcapod/logical_types/user_class_logical_type.py diff --git a/src/orcapod/errors.py b/src/orcapod/errors.py index 94305e06a..aae207d0c 100644 --- a/src/orcapod/errors.py +++ b/src/orcapod/errors.py @@ -31,6 +31,15 @@ class TypeIdentityError(ValueError): """ +class RecursiveTypeError(TypeError): + """Raised when a class refers back to itself through its fields. + + Arrow cannot store a recursive struct, so a dataclass or pydantic model whose + fields reach the class again (``next: Node | None``) cannot be stored. The message + names the field path, e.g. ``Node.next -> Node``. + """ + + class DuplicateTagError(ValueError): """Raised when duplicate tag values are found and skip_duplicates=False""" diff --git a/src/orcapod/logical_types/dataclass_logical_type_factory.py b/src/orcapod/logical_types/dataclass_logical_type_factory.py index 16f268786..a73df8ccf 100644 --- a/src/orcapod/logical_types/dataclass_logical_type_factory.py +++ b/src/orcapod/logical_types/dataclass_logical_type_factory.py @@ -1,18 +1,8 @@ """DataclassLogicalType and DataclassLogicalTypeFactory. -Provides the ``DataclassLogicalType`` logical type implementation and the -``DataclassLogicalTypeFactory`` that synthesises and reconstructs ``DataclassLogicalType`` -instances for Python dataclasses. - -Write path (``create_for_python_type``): - Iterates dataclass fields, delegates field Arrow-type resolution to the converter - via ``register_python_class``, and returns a ``DataclassLogicalType`` backed by - a ``pa.struct`` extension type. - -Read path (``reconstruct_from_arrow``): - Imports the dataclass by fully-qualified class name, resolves field annotations - against the (already bottom-up resolved) storage type, and returns a - ``DataclassLogicalType``. +Dataclasses are user classes: their logical type name and Arrow extension name are the +class's identity string (name, kind ``"dataclass"`` and field types). The shared write +and read paths live in ``user_class_logical_type``. Category tag: ``"orcapod.dataclass"`` """ @@ -20,51 +10,23 @@ from __future__ import annotations import dataclasses -import json -import logging -from typing import TYPE_CHECKING, Any -from orcapod.logical_types.base_logical_type import BaseLogicalType -from orcapod.logical_types.registry import ( - make_arrow_extension_type, - make_polars_extension_type, +from orcapod.logical_types import user_class_identity as uci +from orcapod.logical_types.user_class_logical_type import ( + UserClassLogicalType, + UserClassLogicalTypeFactory, ) -from orcapod.utils.lazy_module import LazyModule - -if TYPE_CHECKING: - import polars as pl - import pyarrow as pa - from orcapod.logical_types.protocols import TypeConverterProtocol -else: - pa = LazyModule("pyarrow") - pl = LazyModule("polars") - -logger = logging.getLogger(__name__) #: Category tag embedded in Arrow extension metadata. Used as the factory dispatch key. DATACLASS_CATEGORY = "orcapod.dataclass" -class DataclassLogicalType(BaseLogicalType): - """Logical type binding a Python dataclass to its Arrow extension type representation. - - Stores the dataclass's fully-qualified class name as the Arrow extension name - and a ``pa.struct`` of the dataclass fields as the storage type. - - No Arrow-type reasoning lives here — all field-type resolution is owned by the - converter and completed before this object is constructed. - - Args: - logical_name: Fully-qualified class name (e.g. ``"mymodule.sub.MyData"``). - Used as both the logical type name and the Arrow extension name. - python_type: The Python dataclass ``type`` object. - storage_type: The Arrow ``pa.StructType`` for the dataclass fields. - field_annotations: Ordered list of ``(field_name, python_annotation)`` pairs - matching the fields in ``storage_type``. +class DataclassLogicalType(UserClassLogicalType): + """Logical type binding a dataclass to its Arrow extension type. Example: >>> lt = DataclassLogicalType( - ... "mymod.Point", Point, + ... "Point@c0a75f0c39b86873", Point, ... pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.int64())]), ... [("x", int), ("y", int)], ... ) @@ -72,133 +34,13 @@ class DataclassLogicalType(BaseLogicalType): {"x": 1, "y": 2} """ - def __init__( - self, - logical_name: str, - python_type: type, - storage_type: pa.StructType, - field_annotations: list[tuple[str, Any]], - ) -> None: - self._logical_name = logical_name - self._python_type = python_type - self._storage_type = storage_type - self._field_annotations = field_annotations - - _metadata = json.dumps({"category": DATACLASS_CATEGORY}).encode("utf-8") - self._arrow_ext_class = make_arrow_extension_type( - logical_name, storage_type, metadata=_metadata - ) - self._arrow_ext: pa.ExtensionType | None = None - # ``storage_type`` must not contain nested extension types (ET1 in DESIGN_ISSUES.md). - # On the write path, ``DataclassLogicalTypeFactory.create_for_python_type`` strips any - # top-level extension type from each field's Arrow type before inserting it into the - # struct. On the read path, ``reconstruct_from_arrow`` receives a ``storage_type`` - # already guaranteed storage-safe by ``register_storage_type``. - self._polars_ext_class = make_polars_extension_type( - logical_name, - storage_type, - metadata=json.dumps({"category": DATACLASS_CATEGORY}), - ) - self._polars_ext: pl.BaseExtension | None = None - - @property - def logical_type_name(self) -> str: - """Fully-qualified class name used as the logical type identifier.""" - return self._logical_name - - @property - def python_type(self) -> type: - """The Python dataclass type this logical type represents.""" - return self._python_type - - def get_arrow_extension_type(self) -> pa.ExtensionType: - """Return the Arrow extension type for this dataclass. - - Returns: - A cached ``pa.ExtensionType`` instance with ``extension_name`` equal to - the fully-qualified class name and ``storage_type`` equal to the struct - of the dataclass fields. - """ - if self._arrow_ext is None: - self._arrow_ext = self._arrow_ext_class() - return self._arrow_ext - - def get_polars_extension_type(self) -> pl.BaseExtension: - """Return the Polars extension type for this dataclass. + category = DATACLASS_CATEGORY - Returns: - A cached ``pl.BaseExtension`` instance. - """ - if self._polars_ext is None: - self._polars_ext = self._polars_ext_class() - return self._polars_ext - def python_to_storage( - self, value: Any, converter: TypeConverterProtocol | None - ) -> dict[str, Any]: - """Convert a dataclass instance to an Arrow-compatible struct dict. +class DataclassLogicalTypeFactory(UserClassLogicalTypeFactory): + """Factory for dataclass logical types. - Iterates ``_field_annotations`` and delegates each field's conversion to - ``converter.python_to_storage``. - - Args: - value: A dataclass instance of type ``python_type``. - converter: The active converter for per-field delegation. Must not be ``None``. - - Returns: - A dict mapping field names to their Arrow storage values. - - Raises: - ValueError: If ``converter`` is ``None``. - """ - if converter is None: - raise ValueError( - "DataclassLogicalType.python_to_storage requires a converter — " - "pass a TypeConverterProtocol instance for field-level conversion." - ) - return { - name: converter.python_to_storage(getattr(value, name), annotation) - for name, annotation in self._field_annotations - } - - def storage_to_python( - self, storage_value: Any, converter: TypeConverterProtocol | None - ) -> Any: - """Reconstruct a dataclass instance from an Arrow struct dict. - - Args: - storage_value: A dict mapping field names to Arrow storage values. - converter: The active converter for per-field delegation. Must not be ``None``. - - Returns: - A dataclass instance of type ``python_type``. - - Raises: - ValueError: If ``converter`` is ``None``. - """ - if converter is None: - raise ValueError( - "DataclassLogicalType.storage_to_python requires a converter — " - "pass a TypeConverterProtocol instance for field-level conversion." - ) - kwargs = { - name: converter.storage_to_python(storage_value[name], annotation) - for name, annotation in self._field_annotations - } - return self._python_type(**kwargs) - - -class DataclassLogicalTypeFactory: - """Stateless factory that synthesises and reconstructs ``DataclassLogicalType`` instances. - - **Write path** (``create_for_python_type``): derives Arrow struct type from the - dataclass fields by delegating to ``converter.register_python_class`` per field. - - **Read path** (``reconstruct_from_arrow``): imports the dataclass by FQCN, matches - fields against the already-resolved ``storage_type``, and returns a - ``DataclassLogicalType``. - - Category tag: ``"orcapod.dataclass"`` + Stored fields are the dataclass fields with ``init=True``. Register with:: @@ -207,168 +49,12 @@ class DataclassLogicalTypeFactory: category="orcapod.dataclass", python_bases=[object], ) - - Example: - >>> factory = DataclassLogicalTypeFactory() - >>> factory.supports_class(MyDataclass) - True - >>> factory.supports_class(str) - False """ - def supports_class(self, python_type: type) -> bool: - """Return True if ``python_type`` is a dataclass. - - Args: - python_type: Any Python type. - - Returns: - True if ``dataclasses.is_dataclass(python_type)`` is True. - """ - return dataclasses.is_dataclass(python_type) and isinstance(python_type, type) - - def create_for_python_type( - self, - python_type: type, - converter: TypeConverterProtocol, - ) -> DataclassLogicalType: - """Synthesise a ``DataclassLogicalType`` for a Python dataclass (write path). - - Derives the FQCN, obtains type hints, and resolves each field's Arrow type - via ``converter.register_python_class``. Rejects local / unnamed classes. - - Args: - python_type: A Python dataclass type. - converter: The active converter for field-type resolution. - - Returns: - A ``DataclassLogicalType`` ready for registration. - - Raises: - ValueError: If ``python_type`` is a local class (``__qualname__`` contains - ``""``). - """ - import typing - - fqcn = f"{python_type.__module__}.{python_type.__qualname__}" - if "" in fqcn: - raise ValueError( - f"Cannot register local class {python_type!r} as a DataclassLogicalType — " - f"local classes have no stable fully-qualified class name and cannot be " - f"reconstructed on read. Define the dataclass at module level." - ) - - try: - hints = typing.get_type_hints(python_type) - except Exception as exc: - raise ValueError( - f"Cannot get type hints for {python_type!r}: {exc}" - ) from exc - - arrow_fields = [] - field_annotations = [] - for field in dataclasses.fields(python_type): - if not field.init: - continue - annotation = hints.get(field.name, Any) - arrow_type = converter.register_python_class(annotation) - # register_python_class returns a storage-safe type: may be extension at the - # top level, but struct fields are always plain. Strip the top-level extension - # type here before inserting into the struct (ET1; see DESIGN_ISSUES.md). - if isinstance(arrow_type, pa.ExtensionType): - arrow_type = arrow_type.storage_type - arrow_fields.append(pa.field(field.name, arrow_type)) - field_annotations.append((field.name, annotation)) - - storage_type = pa.struct(arrow_fields) - logger.debug( - "DataclassLogicalTypeFactory: synthesised %r for %r", fqcn, python_type - ) - return DataclassLogicalType(fqcn, python_type, storage_type, field_annotations) - - def reconstruct_from_arrow( - self, - arrow_extension_name: str, - storage_type: pa.DataType, - metadata: dict[str, Any], - converter: TypeConverterProtocol, - ) -> DataclassLogicalType: - """Reconstruct a ``DataclassLogicalType`` from Arrow schema metadata (read path). - - Imports the dataclass from its FQCN (``arrow_extension_name``), then matches - the dataclass field annotations against the fields in ``storage_type``. - ``storage_type`` is already bottom-up resolved by ``register_storage_type`` - before this method is called. - - Args: - arrow_extension_name: FQCN of the dataclass (Arrow extension name). - storage_type: Already-resolved ``pa.StructType`` for the dataclass fields. - metadata: Full parsed metadata JSON dict (always contains ``"category"``). - converter: The active converter (not needed here but required by protocol). - - Returns: - A ``DataclassLogicalType`` ready for registration. - - Raises: - ImportError: If the class cannot be imported from ``arrow_extension_name``. - ValueError: If ``storage_type`` is not a struct type. - """ - import typing - - if not pa.types.is_struct(storage_type): - raise ValueError( - f"DataclassLogicalTypeFactory.reconstruct_from_arrow: expected a struct " - f"storage type for {arrow_extension_name!r}, got {storage_type!r}." - ) - - # Import class from FQCN using longest-prefix module walk - cls = _import_from_fqcn(arrow_extension_name) - - try: - hints = typing.get_type_hints(cls) - except Exception as exc: - raise ValueError(f"Cannot get type hints for {cls!r}: {exc}") from exc - - field_annotations = [] - for field in dataclasses.fields(cls): - if not field.init: - continue - annotation = hints.get(field.name, Any) - # Register any logical type the field annotation maps to (registration - # completeness invariant: all nested logical types must be registered when - # the outer type is registered). The return value is discarded; only the - # side effect of registration matters here. - converter.register_python_class(annotation) - field_annotations.append((field.name, annotation)) - - logger.debug( - "DataclassLogicalTypeFactory: reconstructed %r from Arrow", - arrow_extension_name, - ) - return DataclassLogicalType( - arrow_extension_name, cls, storage_type, field_annotations - ) - - -def _import_from_fqcn(fqcn: str) -> type: - """Import a dataclass from its fully-qualified class name. - - Delegates the module-prefix walk to ``type_utils._walk_fqcn``, then - validates the resolved object is a dataclass type. - - Args: - fqcn: Fully-qualified class name, e.g. ``"mypackage.sub.MyClass"``. - - Returns: - The imported dataclass type. - - Raises: - ImportError: If no valid module+attribute split can be found, or if the - resolved object is not a dataclass type. - """ - from orcapod.logical_types.type_utils import _walk_fqcn + kind = uci.KIND_DATACLASS + category = DATACLASS_CATEGORY + logical_type_class = DataclassLogicalType - obj: Any = _walk_fqcn(fqcn) - if not dataclasses.is_dataclass(obj) or not isinstance(obj, type): - raise ImportError(f"{fqcn!r} does not resolve to a dataclass type.") - return obj + def stored_field_names(self, python_type: type) -> list[str]: + """Return the names of the ``init=True`` dataclass fields, in class order.""" + return [f.name for f in dataclasses.fields(python_type) if f.init] diff --git a/src/orcapod/logical_types/pydantic_logical_type_factory.py b/src/orcapod/logical_types/pydantic_logical_type_factory.py index e3f37f090..d714b961a 100644 --- a/src/orcapod/logical_types/pydantic_logical_type_factory.py +++ b/src/orcapod/logical_types/pydantic_logical_type_factory.py @@ -1,379 +1,55 @@ """PydanticLogicalType and PydanticLogicalTypeFactory. -Provides the ``PydanticLogicalType`` logical type implementation and the -``PydanticLogicalTypeFactory`` that synthesises and reconstructs -``PydanticLogicalType`` instances for pydantic v2 ``BaseModel`` subclasses. - -Write path (``create_for_python_type``): - Iterates model fields via ``model_fields`` (pydantic v2 API), delegates - field Arrow-type resolution to the converter via ``register_python_class``, - and returns a ``PydanticLogicalType`` backed by a ``pa.struct`` extension - type. - -Read path (``reconstruct_from_arrow``): - Imports the model by fully-qualified class name, resolves field annotations - against the (already bottom-up resolved) storage type, and returns a - ``PydanticLogicalType``. +Pydantic v2 models are user classes: their logical type name and Arrow extension name +are the class's identity string (name, kind ``"pydantic"`` and field types). A pydantic +model and a dataclass with the same name and fields are different types. The shared +write and read paths live in ``user_class_logical_type``. Category tag: ``"orcapod.pydantic"`` """ from __future__ import annotations -import json -import logging -from typing import TYPE_CHECKING, Any - -from orcapod.logical_types.base_logical_type import BaseLogicalType -from orcapod.logical_types.registry import ( - make_arrow_extension_type, - make_polars_extension_type, +from orcapod.logical_types import user_class_identity as uci +from orcapod.logical_types.user_class_logical_type import ( + UserClassLogicalType, + UserClassLogicalTypeFactory, ) -from orcapod.utils.lazy_module import LazyModule - -if TYPE_CHECKING: - import polars as pl - import pyarrow as pa - from orcapod.logical_types.protocols import TypeConverterProtocol -else: - pa = LazyModule("pyarrow") - pl = LazyModule("polars") - -logger = logging.getLogger(__name__) #: Category tag embedded in Arrow extension metadata. Used as the factory dispatch key. PYDANTIC_CATEGORY = "orcapod.pydantic" -class PydanticLogicalType(BaseLogicalType): +class PydanticLogicalType(UserClassLogicalType): """Logical type binding a pydantic ``BaseModel`` subclass to its Arrow extension type. - Stores the model's fully-qualified class name as the Arrow extension name - and a ``pa.struct`` of the model fields as the storage type. - - No Arrow-type reasoning lives here — all field-type resolution is owned by - the converter and completed before this object is constructed. - - Args: - logical_name: Fully-qualified class name (e.g. ``"mymodule.sub.MyModel"``). - Used as both the logical type name and the Arrow extension name. - python_type: The pydantic ``BaseModel`` subclass. - storage_type: The Arrow ``pa.StructType`` for the model fields. - field_annotations: Ordered list of ``(field_name, python_annotation)`` - pairs matching the fields in ``storage_type``. - - Example: - >>> lt = PydanticLogicalType( - ... "mymod.Point", Point, - ... pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.int64())]), - ... [("x", int), ("y", int)], - ... ) - >>> lt.python_to_storage(Point(x=1, y=2), converter) - {"x": 1, "y": 2} + ``storage_to_python`` constructs the model, so pydantic validation runs on read. """ - def __init__( - self, - logical_name: str, - python_type: type, - storage_type: pa.StructType, - field_annotations: list[tuple[str, Any]], - ) -> None: - self._logical_name = logical_name - self._python_type = python_type - self._storage_type = storage_type - self._field_annotations = field_annotations - - _metadata = json.dumps({"category": PYDANTIC_CATEGORY}).encode("utf-8") - self._arrow_ext_class = make_arrow_extension_type( - logical_name, storage_type, metadata=_metadata - ) - self._arrow_ext: pa.ExtensionType | None = None - # ``storage_type`` must not contain nested extension types (ET1 in DESIGN_ISSUES.md). - # On the write path, ``PydanticLogicalTypeFactory.create_for_python_type`` strips any - # top-level extension type from each field's Arrow type before inserting it into the - # struct. On the read path, ``reconstruct_from_arrow`` receives a ``storage_type`` - # already guaranteed storage-safe by ``register_storage_type``. - self._polars_ext_class = make_polars_extension_type( - logical_name, - storage_type, - metadata=json.dumps({"category": PYDANTIC_CATEGORY}), - ) - self._polars_ext: pl.BaseExtension | None = None - - @property - def logical_type_name(self) -> str: - """Fully-qualified class name used as the logical type identifier.""" - return self._logical_name - - @property - def python_type(self) -> type: - """The pydantic ``BaseModel`` subclass this logical type represents.""" - return self._python_type - - def get_arrow_extension_type(self) -> pa.ExtensionType: - """Return the Arrow extension type for this model. - - Returns: - A cached ``pa.ExtensionType`` instance with ``extension_name`` equal to - the fully-qualified class name and ``storage_type`` equal to the struct - of the model fields. - """ - if self._arrow_ext is None: - self._arrow_ext = self._arrow_ext_class() - return self._arrow_ext - - def get_polars_extension_type(self) -> pl.BaseExtension: - """Return the Polars extension type for this model. + category = PYDANTIC_CATEGORY - Returns: - A cached ``pl.BaseExtension`` instance. - """ - if self._polars_ext is None: - self._polars_ext = self._polars_ext_class() - return self._polars_ext - def python_to_storage( - self, value: Any, converter: TypeConverterProtocol | None - ) -> dict[str, Any]: - """Convert a pydantic model instance to an Arrow-compatible struct dict. +class PydanticLogicalTypeFactory(UserClassLogicalTypeFactory): + """Factory for pydantic model logical types. - Iterates ``_field_annotations`` and delegates each field's conversion to - ``converter.python_to_storage``. - - Args: - value: A pydantic model instance of type ``python_type``. - converter: The active converter for per-field delegation. Must not be ``None``. - - Returns: - A dict mapping field names to their Arrow storage values. - - Raises: - ValueError: If ``converter`` is ``None``. - """ - if converter is None: - raise ValueError( - "PydanticLogicalType.python_to_storage requires a converter — " - "pass a TypeConverterProtocol instance for field-level conversion." - ) - return { - name: converter.python_to_storage(getattr(value, name), annotation) - for name, annotation in self._field_annotations - } - - def storage_to_python( - self, storage_value: Any, converter: TypeConverterProtocol | None - ) -> Any: - """Reconstruct a pydantic model instance from an Arrow struct dict. - - Args: - storage_value: A dict mapping field names to Arrow storage values. - converter: The active converter for per-field delegation. Must not be ``None``. - - Returns: - A pydantic model instance of type ``python_type``. Pydantic validation - runs on construction, ensuring the model is always in a valid state. - - Raises: - ValueError: If ``converter`` is ``None``. - """ - if converter is None: - raise ValueError( - "PydanticLogicalType.storage_to_python requires a converter — " - "pass a TypeConverterProtocol instance for field-level conversion." - ) - kwargs = { - name: converter.storage_to_python(storage_value[name], annotation) - for name, annotation in self._field_annotations - } - return self._python_type(**kwargs) - - -class PydanticLogicalTypeFactory: - """Stateless factory that synthesises and reconstructs ``PydanticLogicalType`` instances. - - **Write path** (``create_for_python_type``): derives Arrow struct type from the - model fields by delegating to ``converter.register_python_class`` per field. - Only fields in ``model_fields`` are stored — computed fields and private - attributes are excluded. - - **Read path** (``reconstruct_from_arrow``): imports the model by FQCN, matches - fields against the already-resolved ``storage_type``, and returns a - ``PydanticLogicalType``. - - Category tag: ``"orcapod.pydantic"`` + Stored fields are the model's ``model_fields``; computed fields and private + attributes are not stored. Register with:: from pydantic import BaseModel + converter.register_logical_type_factory( PydanticLogicalTypeFactory(), category="orcapod.pydantic", python_bases=[BaseModel], ) - - Example: - >>> factory = PydanticLogicalTypeFactory() - >>> factory.supports_class(MyModel) - True - >>> factory.supports_class(str) - False """ - def supports_class(self, python_type: type) -> bool: - """Return True if ``python_type`` is a pydantic ``BaseModel`` subclass. - - Args: - python_type: Any Python type. - - Returns: - True if ``python_type`` is a ``BaseModel`` subclass. - """ - from pydantic import BaseModel - - return isinstance(python_type, type) and issubclass(python_type, BaseModel) - - def create_for_python_type( - self, - python_type: type, - converter: TypeConverterProtocol, - ) -> PydanticLogicalType: - """Synthesise a ``PydanticLogicalType`` for a pydantic model (write path). - - Derives the FQCN, obtains type hints, and resolves each field's Arrow type - via ``converter.register_python_class``. Only fields present in - ``model_fields`` are stored — computed fields and private attributes are - excluded. Rejects local / unnamed classes. - - Args: - python_type: A pydantic ``BaseModel`` subclass. - converter: The active converter for field-type resolution. - - Returns: - A ``PydanticLogicalType`` ready for registration. - - Raises: - ValueError: If ``python_type`` is a local class (``__qualname__`` contains - ``""``). - """ - import typing - - fqcn = f"{python_type.__module__}.{python_type.__qualname__}" - if "" in fqcn: - raise ValueError( - f"Cannot register local class {python_type!r} as a PydanticLogicalType — " - f"local classes have no stable fully-qualified class name and cannot be " - f"reconstructed on read. Define the model at module level." - ) - - try: - hints = typing.get_type_hints(python_type) - except Exception as exc: - raise ValueError( - f"Cannot get type hints for {python_type!r}: {exc}" - ) from exc - - arrow_fields = [] - field_annotations = [] - for field_name in python_type.model_fields: - annotation = hints.get(field_name, Any) - arrow_type = converter.register_python_class(annotation) - # Strip top-level extension type before inserting into the struct (ET1; - # see DESIGN_ISSUES.md): Arrow cannot represent extension types inside - # struct field types. - if isinstance(arrow_type, pa.ExtensionType): - arrow_type = arrow_type.storage_type - arrow_fields.append(pa.field(field_name, arrow_type)) - field_annotations.append((field_name, annotation)) - - storage_type = pa.struct(arrow_fields) - logger.debug( - "PydanticLogicalTypeFactory: synthesised %r for %r", fqcn, python_type - ) - return PydanticLogicalType(fqcn, python_type, storage_type, field_annotations) - - def reconstruct_from_arrow( - self, - arrow_extension_name: str, - storage_type: pa.DataType, - metadata: dict[str, Any], - converter: TypeConverterProtocol, - ) -> PydanticLogicalType: - """Reconstruct a ``PydanticLogicalType`` from Arrow schema metadata (read path). - - Imports the model from its FQCN (``arrow_extension_name``), then matches - the model field annotations against the fields in ``storage_type``. - ``storage_type`` is already bottom-up resolved by ``register_storage_type`` - before this method is called. - - Args: - arrow_extension_name: FQCN of the pydantic model (Arrow extension name). - storage_type: Already-resolved ``pa.StructType`` for the model fields. - metadata: Full parsed metadata JSON dict (always contains ``"category"``). - converter: The active converter (used for registration completeness invariant). - - Returns: - A ``PydanticLogicalType`` ready for registration. + kind = uci.KIND_PYDANTIC + category = PYDANTIC_CATEGORY + logical_type_class = PydanticLogicalType - Raises: - ImportError: If the class cannot be imported from ``arrow_extension_name``. - ValueError: If ``storage_type`` is not a struct type. - """ - import typing - - if not pa.types.is_struct(storage_type): - raise ValueError( - f"PydanticLogicalTypeFactory.reconstruct_from_arrow: expected a struct " - f"storage type for {arrow_extension_name!r}, got {storage_type!r}." - ) - - cls = _import_pydantic_model_from_fqcn(arrow_extension_name) - - try: - hints = typing.get_type_hints(cls) - except Exception as exc: - raise ValueError(f"Cannot get type hints for {cls!r}: {exc}") from exc - - field_annotations = [] - for field_name in cls.model_fields: - annotation = hints.get(field_name, Any) - # Register any logical type the field annotation maps to (registration - # completeness invariant: all nested logical types must be registered when - # the outer type is registered). The return value is discarded. - converter.register_python_class(annotation) - field_annotations.append((field_name, annotation)) - - logger.debug( - "PydanticLogicalTypeFactory: reconstructed %r from Arrow", - arrow_extension_name, - ) - return PydanticLogicalType( - arrow_extension_name, cls, storage_type, field_annotations - ) - - -def _import_pydantic_model_from_fqcn(fqcn: str) -> type: - """Import a pydantic ``BaseModel`` subclass from its fully-qualified class name. - - Delegates the module-prefix walk to ``type_utils._walk_fqcn``, then - validates the resolved object is a ``BaseModel`` subclass. - - Args: - fqcn: Fully-qualified class name, e.g. ``"mypackage.sub.MyModel"``. - - Returns: - The imported ``BaseModel`` subclass. - - Raises: - ImportError: If no valid module+attribute split can be found, or if the - resolved object is not a ``BaseModel`` subclass. - """ - from pydantic import BaseModel - from orcapod.logical_types.type_utils import _walk_fqcn - - obj: Any = _walk_fqcn(fqcn) - if not (isinstance(obj, type) and issubclass(obj, BaseModel)): - raise ImportError( - f"{fqcn!r} does not resolve to a pydantic BaseModel subclass." - ) - return obj + def stored_field_names(self, python_type: type) -> list[str]: + """Return the names in ``model_fields``, in class order.""" + return list(python_type.model_fields) # pyright: ignore[reportAttributeAccessIssue] # supports_class guarantees a BaseModel subclass diff --git a/src/orcapod/logical_types/registry.py b/src/orcapod/logical_types/registry.py index a33790422..e3729f647 100644 --- a/src/orcapod/logical_types/registry.py +++ b/src/orcapod/logical_types/registry.py @@ -8,6 +8,7 @@ import logging import re +import warnings from typing import TYPE_CHECKING, Iterable from orcapod.logical_types.protocols import ( @@ -128,13 +129,15 @@ def _deserialize( else: # Non-canonical _storage — require exact match. storage_ok = storage_type == _storage + from orcapod.errors import TypeIdentityError + if not storage_ok: - raise ValueError( + raise TypeIdentityError( f"Arrow extension type '{_name}': expected storage_type " f"{_storage!r} but got {storage_type!r}." ) if serialized != _metadata: - raise ValueError( + raise TypeIdentityError( f"Arrow extension type '{_name}': expected metadata " f"{_metadata!r} but got {serialized!r}." ) @@ -219,6 +222,15 @@ def ext_from_params( ) +def _same_arrow_extension(a: pa.ExtensionType, b: pa.ExtensionType) -> bool: + """Return True if two extension types have the same name, storage and metadata.""" + return ( + a.extension_name == b.extension_name + and a.storage_type == b.storage_type + and a.__arrow_ext_serialize__() == b.__arrow_ext_serialize__() + ) + + class LogicalTypeRegistry: """Registry for ``LogicalType`` instances. @@ -294,6 +306,19 @@ def register_logical_type(self, logical_type: LogicalTypeProtocol) -> None: existing_by_arrow = self._by_arrow_name.get(arrow_ext_name) existing_by_python = self._by_python_type.get(py_type) + if ( + existing_by_python is None + and existing_by_logical is not None + and existing_by_logical is existing_by_arrow + and existing_by_logical is not logical_type + and getattr(logical_type, "exact_python_type_only", False) + and _same_arrow_extension( + existing_by_logical.get_arrow_extension_type(), arrow_ext + ) + ): + self._rebind(existing_by_logical, logical_type) + return + # Triplet conflict check: raise if any key is bound to a different instance. for existing, label, key in [ (existing_by_logical, "logical_type_name", logical_name), @@ -344,6 +369,41 @@ def register_logical_type(self, logical_type: LogicalTypeProtocol) -> None: self._by_arrow_name[arrow_ext_name] = logical_type self._by_python_type[py_type] = logical_type + def _rebind( + self, previous: LogicalTypeProtocol, logical_type: LogicalTypeProtocol + ) -> None: + """Rebind an identity to a logical type built for a new class object. + + Used when a user class with an unchanged identity is defined again (a notebook + cell re-run, a module reload). The logical-name and Arrow-name bindings move to + the new logical type; the previous class object keeps its binding, so instances + created before the re-definition still convert. Arrow's and Polars' global + registrations are reused: name, storage type and metadata are identical. + + Args: + previous: The logical type currently bound to the identity. + logical_type: The logical type for the new class object. + """ + old_cls = previous.python_type + new_cls = logical_type.python_type + old_path = f"{old_cls.__module__}.{old_cls.__qualname__}" + new_path = f"{new_cls.__module__}.{new_cls.__qualname__}" + if old_path != new_path: + warnings.warn( + f"{new_path} has the same identity as {old_path} " + f"({logical_type.logical_type_name}: same name, kind and fields), so " + f"Orcapod treats them as one type. Stored values of this type are now " + f"read back as {new_path}.", + UserWarning, + stacklevel=4, + ) + name = logical_type.logical_type_name + self._by_logical_name[name] = logical_type + self._by_arrow_name[logical_type.get_arrow_extension_type().extension_name] = ( + logical_type + ) + self._by_python_type[new_cls] = logical_type + def get_by_logical_name(self, name: str) -> LogicalTypeProtocol | None: """Return the logical type registered under *name*, or ``None``.""" return self._by_logical_name.get(name) @@ -359,6 +419,8 @@ def get_by_python_type(self, python_type: type) -> LogicalTypeProtocol | None: if result is not None: return result for registered_type, registered_lt in self._by_python_type.items(): + if getattr(registered_lt, "exact_python_type_only", False): + continue try: if issubclass(python_type, registered_type): return registered_lt diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index 5f95454ee..ae8b91f3a 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -22,7 +22,6 @@ import dataclasses import hashlib -import importlib import json import re import sys @@ -145,8 +144,7 @@ def set_naming( continue if value not in _NAMING_STYLES: raise ValueError( - f"Invalid naming style for {key}: {value!r}. " - f"Use 'name' or 'qualified'." + f"Invalid naming style for {key}: {value!r}. Use 'name' or 'qualified'." ) _naming[key] = value @@ -234,6 +232,11 @@ def reject_local_class(cls: type) -> None: ) +def built_name(cls: type) -> str | None: + """Return the name ``cls`` was built under in this process, or ``None``.""" + return _built.get(cls) + + def mark_built(cls: type, name: str) -> None: """Record that ``cls`` was built under ``name`` (one name per class per process).""" _built[cls] = name @@ -304,15 +307,12 @@ def _walk_attributes(root: Any, parts: list[str]) -> Any: def _import_dotted(name: str) -> Any: - parts = name.split(".") - for split in range(len(parts) - 1, 0, -1): - module_name = ".".join(parts[:split]) - try: - module = importlib.import_module(module_name) - except ImportError: - continue - return _walk_attributes(module, parts[split:]) - return None + from orcapod.logical_types.type_utils import _walk_fqcn + + try: + return _walk_fqcn(name) + except ImportError: + return None # --------------------------------------------------------------------------- diff --git a/src/orcapod/logical_types/user_class_logical_type.py b/src/orcapod/logical_types/user_class_logical_type.py new file mode 100644 index 000000000..f7e4bcd10 --- /dev/null +++ b/src/orcapod/logical_types/user_class_logical_type.py @@ -0,0 +1,354 @@ +"""Shared logical type and factory for user classes (dataclasses and pydantic models). + +A user class's logical type name and Arrow extension name are its *identity string* +(``user_class_identity.identity_string``): the name, kind and field types, so two +classes with the same name and different fields are different types with different +tables (ITL-881). The Arrow extension metadata carries the full identity record, which +the read path uses to find the class and to describe a mismatch. + +Write path (``create_for_python_type``): + Resolves the name (declared name or naming policy), resolves each stored field's + Arrow type and type token through the converter, and builds the identity. + +Read path (``reconstruct_from_arrow``): + Reads the identity record from the metadata, finds candidate classes by kind and + name, and accepts the one whose identity string equals the stored extension name. +""" + +from __future__ import annotations + +import logging +import typing +from typing import TYPE_CHECKING, Any + +from orcapod.errors import RecursiveTypeError, TypeIdentityError +from orcapod.logical_types import user_class_identity as uci +from orcapod.logical_types.base_logical_type import BaseLogicalType +from orcapod.logical_types.registry import ( + make_arrow_extension_type, + make_polars_extension_type, +) +from orcapod.utils.lazy_module import LazyModule + +if TYPE_CHECKING: + import polars as pl + import pyarrow as pa + + from orcapod.logical_types.protocols import TypeConverterProtocol +else: + pa = LazyModule("pyarrow") + pl = LazyModule("polars") + +logger = logging.getLogger(__name__) + + +class UserClassLogicalType(BaseLogicalType): + """Logical type binding a user class to its Arrow extension type. + + The logical type name and Arrow extension name are the class's identity string; + the storage type is a ``pa.struct`` of the stored fields. Subclasses set + ``category``. + + Args: + logical_name: The identity string (e.g. ``"Point@9117beaaf7ebe3ab"``). + python_type: The class. + storage_type: The Arrow ``pa.StructType`` of the stored fields. + field_annotations: ``(field_name, python_annotation)`` pairs matching the + fields in ``storage_type``. + identity_record: The identity record, embedded in the Arrow extension metadata. + ``None`` embeds only the category (for low-level tests). + """ + + category: str = "" + + #: A subclass of the class is a different class with its own identity, so the + #: registry must not match it to this logical type by ``issubclass``. + exact_python_type_only: bool = True + + def __init__( + self, + logical_name: str, + python_type: type, + storage_type: pa.StructType, + field_annotations: list[tuple[str, Any]], + identity_record: dict[str, Any] | None = None, + ) -> None: + self._logical_name = logical_name + self._python_type = python_type + self._storage_type = storage_type + self._field_annotations = field_annotations + self._identity_record = identity_record + + metadata: dict[str, Any] = {"category": self.category} + if identity_record is not None: + metadata["identity"] = identity_record + metadata_json = uci.canonical_json(metadata) + # ``storage_type`` never contains nested extension types (ET1 in DESIGN_ISSUES.md): + # the factory strips each field's top-level extension type before building it. + self._arrow_ext_class = make_arrow_extension_type( + logical_name, storage_type, metadata=metadata_json.encode("utf-8") + ) + self._arrow_ext: pa.ExtensionType | None = None + self._polars_ext_class = make_polars_extension_type( + logical_name, storage_type, metadata=metadata_json + ) + self._polars_ext: pl.BaseExtension | None = None + + @property + def logical_type_name(self) -> str: + """The class's identity string.""" + return self._logical_name + + @property + def python_type(self) -> type: + """The class this logical type represents.""" + return self._python_type + + @property + def identity_record(self) -> dict[str, Any] | None: + """The identity record, or ``None`` for a logical type built without one.""" + return self._identity_record + + @property + def storage_type(self) -> pa.StructType: + """The Arrow struct type of the stored fields.""" + return self._storage_type + + def get_arrow_extension_type(self) -> pa.ExtensionType: + """Return the cached Arrow extension type (name = identity string).""" + if self._arrow_ext is None: + self._arrow_ext = self._arrow_ext_class() + return self._arrow_ext + + def get_polars_extension_type(self) -> pl.BaseExtension: + """Return the cached Polars extension type.""" + if self._polars_ext is None: + self._polars_ext = self._polars_ext_class() + return self._polars_ext + + def python_to_storage( + self, value: Any, converter: TypeConverterProtocol | None + ) -> dict[str, Any]: + """Convert an instance to an Arrow-compatible struct dict. + + Args: + value: An instance of ``python_type``. + converter: The active converter for per-field conversion. + + Returns: + A dict mapping field names to their Arrow storage values. + + Raises: + ValueError: If ``converter`` is ``None``. + """ + if converter is None: + raise ValueError( + f"{type(self).__name__}.python_to_storage requires a converter — " + "pass a TypeConverterProtocol instance for field-level conversion." + ) + return { + name: converter.python_to_storage(getattr(value, name), annotation) + for name, annotation in self._field_annotations + } + + def storage_to_python( + self, storage_value: Any, converter: TypeConverterProtocol | None + ) -> Any: + """Build an instance from an Arrow struct dict. + + Args: + storage_value: A dict mapping field names to Arrow storage values. + converter: The active converter for per-field conversion. + + Returns: + An instance of ``python_type``. + + Raises: + ValueError: If ``converter`` is ``None``. + """ + if converter is None: + raise ValueError( + f"{type(self).__name__}.storage_to_python requires a converter — " + "pass a TypeConverterProtocol instance for field-level conversion." + ) + kwargs = { + name: converter.storage_to_python(storage_value[name], annotation) + for name, annotation in self._field_annotations + } + return self._python_type(**kwargs) + + +class UserClassLogicalTypeFactory: + """Base factory for user-class logical types. + + Subclasses set ``kind``, ``category`` and ``logical_type_class`` and implement + ``supports_class`` and ``stored_field_names``. + """ + + kind: str = "" + category: str = "" + logical_type_class: type[UserClassLogicalType] = UserClassLogicalType + + def supports_class(self, python_type: type) -> bool: + """Return True if this factory handles ``python_type``.""" + return uci.class_kind(python_type) == self.kind + + def stored_field_names(self, python_type: type) -> list[str]: + """Return the names of the fields stored for ``python_type``, in class order.""" + raise NotImplementedError + + # ------------------------------------------------------------------ + # Write path + # ------------------------------------------------------------------ + + def create_for_python_type( + self, + python_type: type, + converter: TypeConverterProtocol, + ) -> UserClassLogicalType: + """Build the logical type of a user class (write path). + + Args: + python_type: The class. + converter: The active converter for field-type resolution. + + Returns: + A logical type whose name is the class's identity string. + + Raises: + ValueError: If the class is local or its type hints cannot be resolved. + TypeError: If the class refers back to itself (directly or through other + classes), naming the field path. + """ + uci.reject_local_class(python_type) + name = uci.built_name(python_type) or uci.resolve_name(python_type) + hints = _type_hints(python_type) + + arrow_fields = [] + field_annotations: list[tuple[str, Any]] = [] + tokens: dict[str, str] = {} + for field_name in self.stored_field_names(python_type): + annotation = hints.get(field_name, Any) + try: + arrow_type = converter.register_python_class(annotation) + except TypeError as exc: + if isinstance(exc, RecursiveTypeError): + raise RecursiveTypeError( + f"{python_type.__qualname__}.{field_name} -> {exc}" + ) from exc + raise + # Struct fields cannot hold extension types (ET1; see DESIGN_ISSUES.md). + if isinstance(arrow_type, pa.ExtensionType): + arrow_type = arrow_type.storage_type + arrow_fields.append(pa.field(field_name, arrow_type)) + field_annotations.append((field_name, annotation)) + tokens[field_name] = uci.type_token(annotation, converter) + + record = uci.identity_record(self.kind, name, tokens) + identity = uci.identity_string(record) + uci.mark_built(python_type, name) + uci.remember(self.kind, name, python_type) + uci.notice_main_class(python_type, name) + logger.debug("%s: built %r for %r", type(self).__name__, identity, python_type) + return self.logical_type_class( + identity, python_type, pa.struct(arrow_fields), field_annotations, record + ) + + # ------------------------------------------------------------------ + # Read path + # ------------------------------------------------------------------ + + def reconstruct_from_arrow( + self, + arrow_extension_name: str, + storage_type: pa.DataType, + metadata: dict[str, Any], + converter: TypeConverterProtocol, + ) -> UserClassLogicalType: + """Find the class of a stored column and return its logical type (read path). + + Args: + arrow_extension_name: The stored identity string. + storage_type: The stored struct type (already resolved). + metadata: The parsed extension metadata, with ``category`` and ``identity``. + converter: The active converter. + + Returns: + The logical type of the class whose identity matches the stored one. + + Raises: + ImportError: If no class with the stored kind and name can be found. + TypeIdentityError: If the column has no identity record (written before + Orcapod 0.3), or the classes found have a different identity or a + different storage type. + """ + record = metadata.get("identity") + if not isinstance(record, dict): + raise TypeIdentityError( + f"The stored {self.kind} column type {arrow_extension_name!r} was written " + f"by an Orcapod version before 0.3, which identified classes by module " + f"and qualified name only. Orcapod 0.3 identifies a class by its name, " + f"kind and fields and does not read such columns; re-running the " + f"pipeline computes the results into new tables." + ) + name = record.get("name") + kind = record.get("kind") + if kind != self.kind or not isinstance(name, str): + raise TypeIdentityError( + f"Stored column type {arrow_extension_name!r} has an invalid identity " + f"record for category {self.category!r}: {record!r}." + ) + + candidates = uci.candidate_classes(kind, name) + if not candidates: + raise ImportError( + f"Cannot find the {kind} {name!r} to read stored values of type " + f"{uci.describe_record(record)}. Define the class and register it " + f"under that name before reading, for example " + f"op.register_type({name.rsplit('.', 1)[-1]}, name={name!r})." + ) + + found: list[UserClassLogicalType] = [] + for cls in candidates: + converter.register_python_class(cls) + lt = converter.get_logical_type(cls) + if not isinstance(lt, UserClassLogicalType): + continue + if lt.logical_type_name == arrow_extension_name: + _check_storage(lt, storage_type, record) + return lt + found.append(lt) + + registered = "; ".join( + uci.describe_record(lt.identity_record) + if lt.identity_record is not None + else lt.logical_type_name + for lt in found + ) + raise TypeIdentityError( + f"Stored values of {uci.describe_record(record)} do not match the " + f"registered class: {registered}. A class with the same name but different " + f"fields is a different type; restore the stored fields, or use the class " + f"with the new fields to compute new results." + ) + + +def _type_hints(python_type: type) -> dict[str, Any]: + try: + return typing.get_type_hints(python_type) + except Exception as exc: + raise ValueError(f"Cannot get type hints for {python_type!r}: {exc}") from exc + + +def _check_storage( + lt: UserClassLogicalType, storage_type: pa.DataType, record: dict[str, Any] +) -> None: + from orcapod.utils.arrow_utils import normalize_to_large_types + + if normalize_to_large_types(lt.storage_type) != normalize_to_large_types( + storage_type + ): + raise TypeIdentityError( + f"Stored values of {uci.describe_record(record)} have Arrow storage type " + f"{storage_type}, but the registered class stores {lt.storage_type}." + ) diff --git a/src/orcapod/semantic_types/universal_converter.py b/src/orcapod/semantic_types/universal_converter.py index b68a6f39c..2fae197e7 100644 --- a/src/orcapod/semantic_types/universal_converter.py +++ b/src/orcapod/semantic_types/universal_converter.py @@ -429,9 +429,11 @@ def _register_python_class_impl( # Cycle detection (via the shared ContextVar-backed in_progress set) if annotation in in_progress: - raise TypeError( - f"Circular type dependency detected while synthesising " - f"LogicalType for {annotation!r}." + from orcapod.errors import RecursiveTypeError + + raise RecursiveTypeError( + f"{annotation.__qualname__} (a class that refers back to itself " + f"cannot be stored: Arrow has no recursive struct type)" ) # Factory dispatch via MRO walk diff --git a/tests/test_logical_types/test_cache_behavior.py b/tests/test_logical_types/test_cache_behavior.py index 90bac4471..52416f3e3 100644 --- a/tests/test_logical_types/test_cache_behavior.py +++ b/tests/test_logical_types/test_cache_behavior.py @@ -28,6 +28,15 @@ ) +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + # Module-level dataclass — local classes cannot be reconstructed from FQCN. @@ -74,18 +83,19 @@ def test_cache_populated_after_first_read(tmp_path): parquet_path = _write_parquet(tmp_path, write_converter) read_converter = _fresh_converter() - fqcn = f"{_CachePoint.__module__}.{_CachePoint.__qualname__}" + type_name = _identity(_CachePoint) # Before read: not registered assert ( - read_converter._logical_type_registry.get_by_arrow_extension_name(fqcn) is None + read_converter._logical_type_registry.get_by_arrow_extension_name(type_name) + is None ) read_converter.load_logical_types(pq.read_table(parquet_path)) # After read: registered (cache populated) assert ( - read_converter._logical_type_registry.get_by_arrow_extension_name(fqcn) + read_converter._logical_type_registry.get_by_arrow_extension_name(type_name) is not None ) diff --git a/tests/test_logical_types/test_dataclass_logical_type_factory.py b/tests/test_logical_types/test_dataclass_logical_type_factory.py index 70b098e8f..c300c8bc5 100644 --- a/tests/test_logical_types/test_dataclass_logical_type_factory.py +++ b/tests/test_logical_types/test_dataclass_logical_type_factory.py @@ -9,6 +9,37 @@ import pytest +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + +def _metadata(cls: type) -> dict: + """Return the parsed Arrow extension metadata of a user class (ITL-881).""" + import json + + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) + + +def _metadata(cls: type) -> dict: + """Return the parsed Arrow extension metadata of a user class (ITL-881).""" + import json + + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) + + # ── Helpers ───────────────────────────────────────────────────────────────── @@ -382,16 +413,18 @@ def test_factory_reconstruct_from_arrow(): ) storage = pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.int64())]) - metadata = {"category": "orcapod.dataclass"} - fqcn = f"{_RoundTripPoint.__module__}.{_RoundTripPoint.__qualname__}" + metadata = _metadata(_RoundTripPoint) + type_name = _identity(_RoundTripPoint) factory = DataclassLogicalTypeFactory() converter = _make_full_converter() - lt = factory.reconstruct_from_arrow(fqcn, storage, metadata, converter=converter) + lt = factory.reconstruct_from_arrow( + type_name, storage, metadata, converter=converter + ) assert isinstance(lt, DataclassLogicalType) assert lt.python_type is _RoundTripPoint - assert lt.logical_type_name == fqcn + assert lt.logical_type_name == type_name def test_factory_reconstruct_from_arrow_invalid_fqcn(): @@ -408,7 +441,14 @@ def test_factory_reconstruct_from_arrow_invalid_fqcn(): factory.reconstruct_from_arrow( "nonexistent.module.NoSuchClass", storage, - {"category": "orcapod.dataclass"}, + { + "category": "orcapod.dataclass", + "identity": { + "kind": "dataclass", + "name": "nonexistent.module.NoSuchClass", + "fields": {"x": "int"}, + }, + }, converter, ) @@ -429,7 +469,7 @@ def test_reconstruct_from_arrow_registers_nested_types(): pa.field("label", pa.large_string()), ] ) - outer_fqcn = f"{_OuterForRegistrationTest.__module__}.{_OuterForRegistrationTest.__qualname__}" + outer_type_name = _identity(_OuterForRegistrationTest) factory = DataclassLogicalTypeFactory() converter = _make_full_converter() @@ -442,7 +482,7 @@ def test_reconstruct_from_arrow_registers_nested_types(): # reconstruct_from_arrow for Outer should trigger registration of Inner as a side effect factory.reconstruct_from_arrow( - outer_fqcn, outer_storage, {"category": "orcapod.dataclass"}, converter + outer_type_name, outer_storage, _metadata(_OuterForRegistrationTest), converter ) # Inner must now be registered @@ -499,12 +539,12 @@ def test_dataclass_with_uuid_round_trip(): assert reconstructed.label == "hello" -# ── _import_from_fqcn nested class tests ───────────────────────────────────── +# ── nested class lookup on read ─────────────────────────────────────────────── @dataclasses.dataclass class _OuterForNestedTest: - """Module-level outer class for testing nested-class FQCN import.""" + """Module-level outer class for testing the read-side lookup of a nested class.""" @dataclasses.dataclass class Inner: @@ -512,20 +552,18 @@ class Inner: y: str -def test_import_from_fqcn_nested_class(): - """_import_from_fqcn resolves module-level nested dataclasses via attribute walk.""" - from orcapod.logical_types.dataclass_logical_type_factory import _import_from_fqcn - - # _OuterForNestedTest.Inner lives in this test module; its FQCN uses '.' for nesting - module = _OuterForNestedTest.__module__ - inner_qualname = ( - _OuterForNestedTest.Inner.__qualname__ - ) # e.g. "_OuterForNestedTest.Inner" +def test_reconstruct_finds_class_nested_in_a_class(): + """The read path imports a class nested in another class by its dotted name.""" + from orcapod.logical_types.dataclass_logical_type_factory import ( + DataclassLogicalTypeFactory, + ) - fqcn = f"{module}.{inner_qualname}" - cls = _import_from_fqcn(fqcn) - assert cls is _OuterForNestedTest.Inner - assert dataclasses.is_dataclass(cls) + inner = _OuterForNestedTest.Inner + storage = pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.large_string())]) + lt = DataclassLogicalTypeFactory().reconstruct_from_arrow( + _identity(inner), storage, _metadata(inner), _make_full_converter() + ) + assert lt.python_type is inner def test_python_to_storage_raises_when_converter_none(): diff --git a/tests/test_logical_types/test_default_context_factories.py b/tests/test_logical_types/test_default_context_factories.py index 2c880b16b..550d5c185 100644 --- a/tests/test_logical_types/test_default_context_factories.py +++ b/tests/test_logical_types/test_default_context_factories.py @@ -20,6 +20,15 @@ from orcapod.logical_types.registry import LogicalTypeRegistry +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + # ── Module-level dataclasses (local classes cannot be registered) ──────────── @@ -115,8 +124,8 @@ def test_default_context_dataclass_auto_registered_on_use(): converter = create_registry().get_context().type_converter arrow_type = converter.register_python_class(_SimplePoint) assert isinstance(arrow_type, pa.ExtensionType) - fqcn = f"{_SimplePoint.__module__}.{_SimplePoint.__qualname__}" - assert arrow_type.extension_name == fqcn + type_name = _identity(_SimplePoint) + assert arrow_type.extension_name == type_name def test_default_context_pydantic_auto_registered_on_use(): @@ -124,8 +133,8 @@ def test_default_context_pydantic_auto_registered_on_use(): converter = create_registry().get_context().type_converter arrow_type = converter.register_python_class(_SimpleModel) assert isinstance(arrow_type, pa.ExtensionType) - fqcn = f"{_SimpleModel.__module__}.{_SimpleModel.__qualname__}" - assert arrow_type.extension_name == fqcn + type_name = _identity(_SimpleModel) + assert arrow_type.extension_name == type_name # ── Parquet round-trip tests ───────────────────────────────────────────────── diff --git a/tests/test_logical_types/test_pydantic_logical_type_factory.py b/tests/test_logical_types/test_pydantic_logical_type_factory.py index f7f6d2cdc..f5087722e 100644 --- a/tests/test_logical_types/test_pydantic_logical_type_factory.py +++ b/tests/test_logical_types/test_pydantic_logical_type_factory.py @@ -11,6 +11,26 @@ from pydantic import BaseModel, PrivateAttr, computed_field, field_validator +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + +def _metadata(cls: type) -> dict: + """Return the parsed Arrow extension metadata of a user class (ITL-881).""" + import json + + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) + + # ── Helpers ────────────────────────────────────────────────────────────────── @@ -410,16 +430,18 @@ def test_factory_reconstruct_from_arrow(): ) storage = pa.struct([pa.field("x", pa.int64()), pa.field("y", pa.int64())]) - metadata = {"category": "orcapod.pydantic"} - fqcn = f"{_RoundTripPoint.__module__}.{_RoundTripPoint.__qualname__}" + metadata = _metadata(_RoundTripPoint) + type_name = _identity(_RoundTripPoint) factory = PydanticLogicalTypeFactory() converter = _make_full_converter() - lt = factory.reconstruct_from_arrow(fqcn, storage, metadata, converter=converter) + lt = factory.reconstruct_from_arrow( + type_name, storage, metadata, converter=converter + ) assert isinstance(lt, PydanticLogicalType) assert lt.python_type is _RoundTripPoint - assert lt.logical_type_name == fqcn + assert lt.logical_type_name == type_name def test_factory_reconstruct_from_arrow_invalid_fqcn(): @@ -436,7 +458,14 @@ def test_factory_reconstruct_from_arrow_invalid_fqcn(): factory.reconstruct_from_arrow( "nonexistent.module.NoSuchModel", storage, - {"category": "orcapod.pydantic"}, + { + "category": "orcapod.pydantic", + "identity": { + "kind": "pydantic", + "name": "nonexistent.module.NoSuchClass", + "fields": {"x": "int"}, + }, + }, converter, ) @@ -454,7 +483,7 @@ def test_reconstruct_from_arrow_registers_nested_types(): pa.field("label", pa.large_string()), ] ) - outer_fqcn = f"{_OuterModel.__module__}.{_OuterModel.__qualname__}" + outer_type_name = _identity(_OuterModel) factory = PydanticLogicalTypeFactory() converter = _make_full_converter() @@ -463,7 +492,7 @@ def test_reconstruct_from_arrow_registers_nested_types(): assert converter._logical_type_registry.get_by_python_type(_InnerModel) is None factory.reconstruct_from_arrow( - outer_fqcn, outer_storage, {"category": "orcapod.pydantic"}, converter + outer_type_name, outer_storage, _metadata(_OuterModel), converter ) # Inner must now be registered as a side effect @@ -737,9 +766,9 @@ def test_literal_model_polars_dataframe_round_trip(): df = pl.from_arrow(table) # Polars column must carry the correct extension type - fqcn = f"{_LiteralRoundTripModel.__module__}.{_LiteralRoundTripModel.__qualname__}" + type_name = _identity(_LiteralRoundTripModel) assert isinstance(df.dtypes[0], pl.BaseExtension) - assert df.dtypes[0].ext_name() == fqcn + assert df.dtypes[0].ext_name() == type_name # Convert back to Arrow and reconstruct Python objects table_back = df.to_arrow() diff --git a/tests/test_logical_types/test_roundtrips.py b/tests/test_logical_types/test_roundtrips.py index c576836f9..4fe911fb7 100644 --- a/tests/test_logical_types/test_roundtrips.py +++ b/tests/test_logical_types/test_roundtrips.py @@ -34,6 +34,15 @@ from orcapod.semantic_types.universal_converter import UniversalTypeConverter +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + # ── Module-level dataclasses ────────────────────────────────────────────────── # DataclassLogicalTypeFactory rejects local (in-function) classes because they # have no stable fully-qualified class name for reconstruction from Arrow schema. @@ -338,12 +347,12 @@ def test_simple_dataclass_round_trip( tmp_path, ) - fqcn = f"{_PointA.__module__}.{_PointA.__qualname__}" + type_name = _identity(_PointA) field = result.schema.field("point") assert hasattr(field.type, "extension_name"), ( f"Expected extension type on field 'point', got {field.type!r}" ) - assert field.type.extension_name == fqcn + assert field.type.extension_name == type_name rows = read_converter.arrow_table_to_python_dicts(result) assert len(rows) == 1 @@ -370,13 +379,13 @@ def test_two_dataclasses_same_shape_distinct_extension_names( tmp_path, ) - fqcn_a = f"{_PointA.__module__}.{_PointA.__qualname__}" - fqcn_b = f"{_PointB.__module__}.{_PointB.__qualname__}" + type_name_a = _identity(_PointA) + type_name_b = _identity(_PointB) field = result.schema.field("point") assert hasattr(field.type, "extension_name") - assert field.type.extension_name == fqcn_a - assert field.type.extension_name != fqcn_b # distinct from _PointB + assert field.type.extension_name == type_name_a + assert field.type.extension_name != type_name_b # distinct from _PointB rows = read_converter.arrow_table_to_python_dicts(result) reconstructed = rows[0]["point"] @@ -401,15 +410,19 @@ def test_nested_dataclass_round_trip( tmp_path, ) - fqcn_outer = f"{_Outer.__module__}.{_Outer.__qualname__}" - fqcn_inner = f"{_Inner.__module__}.{_Inner.__qualname__}" + type_name_outer = _identity(_Outer) + type_name_inner = _identity(_Inner) assert ( - read_converter._logical_type_registry.get_by_arrow_extension_name(fqcn_outer) + read_converter._logical_type_registry.get_by_arrow_extension_name( + type_name_outer + ) is not None ), "_Outer should be registered after read" assert ( - read_converter._logical_type_registry.get_by_arrow_extension_name(fqcn_inner) + read_converter._logical_type_registry.get_by_arrow_extension_name( + type_name_inner + ) is not None ), "_Inner should be registered transitively after read" @@ -444,7 +457,7 @@ def test_delta_polars_read_delta(tmp_path: Path) -> None: import polars as pl delta_path = str(tmp_path / "polars_delta") - fqcn = f"{_PointA.__module__}.{_PointA.__qualname__}" + type_name = _identity(_PointA) # Write — registers _PointA in PyArrow + Polars global registries. write_converter = _fresh_converter() @@ -463,8 +476,8 @@ def test_delta_polars_read_delta(tmp_path: Path) -> None: assert col_dtype.is_extension(), ( f"Expected a Polars extension type on column 'point', got {col_dtype!r}" ) - assert col_dtype.ext_name() == fqcn, ( - f"Expected extension name {fqcn!r}, got {col_dtype.ext_name()!r}" + assert col_dtype.ext_name() == type_name, ( + f"Expected extension name {type_name!r}, got {col_dtype.ext_name()!r}" ) diff --git a/tests/test_logical_types/test_schema_compatibility.py b/tests/test_logical_types/test_schema_compatibility.py index 860519348..b7c527010 100644 --- a/tests/test_logical_types/test_schema_compatibility.py +++ b/tests/test_logical_types/test_schema_compatibility.py @@ -28,6 +28,15 @@ from orcapod.utils.schema_utils import check_schema_compatibility +def _identity(cls: type) -> str: + """Return the identity string (logical type name) of a user class (ITL-881).""" + from orcapod.contexts import get_default_context + + lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + assert lt is not None + return lt.logical_type_name + + # Module-level dataclasses — DataclassLogicalTypeFactory rejects local classes # because they have no stable fully-qualified class name for reconstruction. @@ -64,10 +73,10 @@ def test_arrow_schema_distinct_extension_names_for_same_shape(): assert isinstance(type_a, pa.ExtensionType) assert isinstance(type_b, pa.ExtensionType) - fqcn_a = f"{_PointA.__module__}.{_PointA.__qualname__}" - fqcn_b = f"{_PointB.__module__}.{_PointB.__qualname__}" - assert type_a.extension_name == fqcn_a - assert type_b.extension_name == fqcn_b + type_name_a = _identity(_PointA) + type_name_b = _identity(_PointB) + assert type_a.extension_name == type_name_a + assert type_b.extension_name == type_name_b assert type_a.extension_name != type_b.extension_name diff --git a/tests/test_logical_types/test_type_utils.py b/tests/test_logical_types/test_type_utils.py index e8e52864a..c83c61dc5 100644 --- a/tests/test_logical_types/test_type_utils.py +++ b/tests/test_logical_types/test_type_utils.py @@ -167,15 +167,3 @@ def _patched(name: str, *args, **kwargs): with pytest.raises(ModuleNotFoundError, match="'path'"): _walk_fqcn("pathlib.Path") - - -# ── _import_from_fqcn tests ────────────────────────────────────────────────── - - -def test_import_from_fqcn_raises_for_non_dataclass(): - """_import_from_fqcn raises ImportError when FQCN resolves to a non-dataclass.""" - from orcapod.logical_types.dataclass_logical_type_factory import _import_from_fqcn - - # pathlib.Path is importable via _walk_fqcn but is not a dataclass - with pytest.raises(ImportError): - _import_from_fqcn("pathlib.Path") From b4104a455de35caf97ea70158abbd33424de6ba1 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:39:24 -0800 Subject: [PATCH 006/147] feat(types): register_type API, rebinding and read-side identity checks (ITL-881) - op.register_type declares the name a dataclass or pydantic model is identified by; op.set_type_naming sets the defaults (__main__ classes by name, module classes by module path). - Stored fields are kept in name order, so field order changes neither the identity nor the Arrow struct. - A pod defined before a class cell re-run accepts the re-defined class (schema compatibility compares user classes by identity). - DeltaTableDatabase re-raises TypeIdentityError instead of logging a read failure, and read-time identity errors name the column. - Tests: workshop scenarios W1-W12 in fresh processes (IPython InteractiveShell for notebook cells), public API, S2 storage guard. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/__init__.py | 6 + src/orcapod/databases/delta_lake_databases.py | 21 + src/orcapod/logical_types/database_hooks.py | 32 +- .../dataclass_logical_type_factory.py | 2 +- .../pydantic_logical_type_factory.py | 2 +- .../logical_types/user_class_identity.py | 56 ++- .../logical_types/user_class_logical_type.py | 17 +- src/orcapod/user_types.py | 97 +++++ src/orcapod/utils/schema_utils.py | 6 +- .../test_logical_types/test_cache_behavior.py | 5 +- .../test_dataclass_logical_type_factory.py | 23 +- .../test_default_context_factories.py | 5 +- .../test_pydantic_logical_type_factory.py | 12 +- tests/test_logical_types/test_roundtrips.py | 7 +- .../test_schema_compatibility.py | 4 +- tests/test_user_types/child_harness.py | 149 +++++++ tests/test_user_types/test_read_path.py | 52 +++ tests/test_user_types/test_register_type.py | 361 +++++++++++++++++ .../test_workshop_scenarios.py | 365 ++++++++++++++++++ 19 files changed, 1177 insertions(+), 45 deletions(-) create mode 100644 src/orcapod/user_types.py create mode 100644 tests/test_user_types/child_harness.py create mode 100644 tests/test_user_types/test_read_path.py create mode 100644 tests/test_user_types/test_register_type.py create mode 100644 tests/test_user_types/test_workshop_scenarios.py diff --git a/src/orcapod/__init__.py b/src/orcapod/__init__.py index bc229f096..59b3f6571 100644 --- a/src/orcapod/__init__.py +++ b/src/orcapod/__init__.py @@ -60,6 +60,9 @@ # Extension registration API (ITL-473) from .extensions import OrcapodExtension, register_extension +# Your own data types: dataclass / pydantic identity (ITL-881) +from .user_types import register_type, set_type_naming + # Post-run hook types (ITL-523) from .hooks import ( HookConfig, @@ -106,6 +109,9 @@ # Extension registration API (ITL-473) "OrcapodExtension", "register_extension", + # Your own data types (ITL-881) + "register_type", + "set_type_naming", # Post-run hook types (ITL-523) "HookConfig", "InvocationStatus", diff --git a/src/orcapod/databases/delta_lake_databases.py b/src/orcapod/databases/delta_lake_databases.py index c7dec5dcf..fed945012 100644 --- a/src/orcapod/databases/delta_lake_databases.py +++ b/src/orcapod/databases/delta_lake_databases.py @@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Any, Literal, cast from orcapod.databases.utils import coerce_record_id +from orcapod.errors import TypeIdentityError from orcapod.databases.storage_utils import is_cloud_uri, parse_base_path from orcapod.utils import arrow_utils from orcapod.utils.lazy_module import LazyModule @@ -574,6 +575,10 @@ def _handle_delta_schema_compatibility( # Get Delta table schema and convert from arro3 to pyarrow arro3_schema = delta_table.schema().to_arrow() delta_schema = pa.schema(arro3_schema) # type: ignore + except TypeIdentityError: + # Stored and registered user-class shapes disagree: never report + # this as missing data (ITL-881). + raise except Exception as e: logger.warning(f"Could not get Delta table schema: {e}") return table @@ -644,6 +649,10 @@ def get_all_records( ).to_table() if delta_table_data.num_rows > 0: tables_to_combine.append(delta_table_data) + except TypeIdentityError: + # Stored and registered user-class shapes disagree: never report + # this as missing data (ITL-881). + raise except Exception as e: logger.warning(f"Error reading Delta table: {e}") @@ -710,6 +719,10 @@ def get_records_with_column_value( # Handle (remove/rename) the record id column before returning return self._handle_record_id_column(result, record_id_column) + except TypeIdentityError: + # Stored and registered user-class shapes disagree: never report + # this as missing data (ITL-881). + raise except Exception as e: logger.error( f"Error getting record with {column_values} from {'/'.join(record_path)}: {e}" @@ -769,6 +782,10 @@ def get_record_by_id( # Handle (remove/rename) the record id column before returning return self._handle_record_id_column(result, record_id_column) + except TypeIdentityError: + # Stored and registered user-class shapes disagree: never report + # this as missing data (ITL-881). + raise except Exception as e: logger.error( f"Error getting record {record_id} from {'/'.join(record_path)}: {e}" @@ -826,6 +843,10 @@ def get_records_by_ids( # Handle record_id column based on parameter return self._handle_record_id_column(result, record_id_column) + except TypeIdentityError: + # Stored and registered user-class shapes disagree: never report + # this as missing data (ITL-881). + raise except Exception as e: logger.error( f"Error getting records by IDs from {'/'.join(record_path)}: {e}" diff --git a/src/orcapod/logical_types/database_hooks.py b/src/orcapod/logical_types/database_hooks.py index 65bf04065..845b88bd5 100644 --- a/src/orcapod/logical_types/database_hooks.py +++ b/src/orcapod/logical_types/database_hooks.py @@ -24,8 +24,9 @@ import logging from typing import TYPE_CHECKING +from orcapod.errors import TypeIdentityError from orcapod.logical_types.registry import LogicalTypeRegistry -from orcapod.logical_types.schema_walker import walk_schema +from orcapod.logical_types.schema_walker import walk_field, walk_schema if TYPE_CHECKING: import pyarrow as pa @@ -56,6 +57,10 @@ def register_discovered_logical_types( Raises: ValueError: Propagated from the converter if an extension type's metadata has no registered factory or is malformed. + TypeIdentityError: If a stored user-class column does not match the class + found for it; the message starts with the column name. + ImportError: If no class can be found for a stored user-class column; the + message starts with the column name. """ if converter is None: logger.debug( @@ -73,13 +78,24 @@ def register_discovered_logical_types( [info.extension_name for info in found], ) for info in found: - # Bottom-up resolve the storage type first, then register the extension - resolved_storage = converter.register_storage_type(info.storage_type) - converter.register_logical_type_from_arrow_metadata( - info.extension_name, - info.extension_metadata, - resolved_storage, - ) + try: + # Bottom-up resolve the storage type first, then register the extension + resolved_storage = converter.register_storage_type(info.storage_type) + converter.register_logical_type_from_arrow_metadata( + info.extension_name, + info.extension_metadata, + resolved_storage, + ) + except (TypeIdentityError, ImportError) as exc: + # Name the column(s) holding the type: the factory only knows the type. + columns = ", ".join( + repr(field.name) + for field in schema + if any( + i.extension_name == info.extension_name for i in walk_field(field) + ) + ) + raise type(exc)(f"Column {columns}: {exc}") from exc def apply_logical_types( diff --git a/src/orcapod/logical_types/dataclass_logical_type_factory.py b/src/orcapod/logical_types/dataclass_logical_type_factory.py index a73df8ccf..978cad3f4 100644 --- a/src/orcapod/logical_types/dataclass_logical_type_factory.py +++ b/src/orcapod/logical_types/dataclass_logical_type_factory.py @@ -11,7 +11,7 @@ import dataclasses -from orcapod.logical_types import user_class_identity as uci +import orcapod.logical_types.user_class_identity as uci from orcapod.logical_types.user_class_logical_type import ( UserClassLogicalType, UserClassLogicalTypeFactory, diff --git a/src/orcapod/logical_types/pydantic_logical_type_factory.py b/src/orcapod/logical_types/pydantic_logical_type_factory.py index d714b961a..baa66fc93 100644 --- a/src/orcapod/logical_types/pydantic_logical_type_factory.py +++ b/src/orcapod/logical_types/pydantic_logical_type_factory.py @@ -10,7 +10,7 @@ from __future__ import annotations -from orcapod.logical_types import user_class_identity as uci +import orcapod.logical_types.user_class_identity as uci from orcapod.logical_types.user_class_logical_type import ( UserClassLogicalType, UserClassLogicalTypeFactory, diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index ae8b91f3a..56486bfd1 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -60,6 +60,8 @@ _declared: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() # The name each class object was built under; a class has one name per process. _built: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() +# The identity string each class object was built with. +_identity_of: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() # Classes seen under each (kind, name), most recent last. _known: dict[tuple[str, str], list[weakref.ref[type]]] = {} # Names that already produced the one-time ``__main__`` notice. @@ -146,7 +148,7 @@ def set_naming( raise ValueError( f"Invalid naming style for {key}: {value!r}. Use 'name' or 'qualified'." ) - _naming[key] = value + _naming[key] = typing.cast(NamingStyle, value) def is_main_class(cls: type) -> bool: @@ -237,9 +239,53 @@ def built_name(cls: type) -> str | None: return _built.get(cls) -def mark_built(cls: type, name: str) -> None: - """Record that ``cls`` was built under ``name`` (one name per class per process).""" +def mark_built(cls: type, name: str, identity: str) -> None: + """Record that ``cls`` was built under ``name`` with ``identity``. + + A class has one name per process. + + Args: + cls: The class. + name: Its name component. + identity: Its identity string. + """ _built[cls] = name + _identity_of[cls] = identity + + +def equivalent_hints(a: Any, b: Any) -> bool: + """Return True if two type hints are equal up to user-class identity. + + Two user classes are equivalent when they were built with the same identity string, + for example a notebook class and its re-definition from a re-run cell. Generic hints + are compared by origin and arguments; union members in any order. + + Args: + a: A type hint. + b: A type hint. + + Returns: + True if the hints denote the same types. + """ + import types as _types + + if a == b: + return True + if isinstance(a, type) and isinstance(b, type): + ident = _identity_of.get(a) + return ident is not None and ident == _identity_of.get(b) + origin_a, origin_b = typing.get_origin(a), typing.get_origin(b) + if origin_a is None or origin_b is None: + return False + args_a, args_b = typing.get_args(a), typing.get_args(b) + unions = (typing.Union, _types.UnionType) + if origin_a in unions and origin_b in unions: + return all(any(equivalent_hints(x, y) for y in args_b) for x in args_a) and all( + any(equivalent_hints(x, y) for x in args_a) for y in args_b + ) + if origin_a != origin_b or len(args_a) != len(args_b): + return False + return all(equivalent_hints(x, y) for x, y in zip(args_a, args_b)) # --------------------------------------------------------------------------- @@ -405,8 +451,8 @@ def type_token(annotation: Any, converter: TypeConverterProtocol) -> str: inner = ", ".join(type_token(a, converter) for a in args) return f"{_class_token(origin)}[{inner}]" if isinstance(annotation, type): - converter.register_python_class(annotation) - lt = converter.get_logical_type(annotation) + _ = converter.register_python_class(annotation) # raises for unsupported types + lt = converter.get_logical_type_for_python_type(annotation) if lt is not None: return lt.logical_type_name return _class_token(annotation) diff --git a/src/orcapod/logical_types/user_class_logical_type.py b/src/orcapod/logical_types/user_class_logical_type.py index f7e4bcd10..745755bdf 100644 --- a/src/orcapod/logical_types/user_class_logical_type.py +++ b/src/orcapod/logical_types/user_class_logical_type.py @@ -22,7 +22,7 @@ from typing import TYPE_CHECKING, Any from orcapod.errors import RecursiveTypeError, TypeIdentityError -from orcapod.logical_types import user_class_identity as uci +import orcapod.logical_types.user_class_identity as uci from orcapod.logical_types.base_logical_type import BaseLogicalType from orcapod.logical_types.registry import ( make_arrow_extension_type, @@ -117,13 +117,13 @@ def storage_type(self) -> pa.StructType: def get_arrow_extension_type(self) -> pa.ExtensionType: """Return the cached Arrow extension type (name = identity string).""" if self._arrow_ext is None: - self._arrow_ext = self._arrow_ext_class() + self._arrow_ext = self._arrow_ext_class() # pyright: ignore[reportCallIssue] # synthesized class takes no arguments return self._arrow_ext def get_polars_extension_type(self) -> pl.BaseExtension: """Return the cached Polars extension type.""" if self._polars_ext is None: - self._polars_ext = self._polars_ext_class() + self._polars_ext = self._polars_ext_class() # pyright: ignore[reportCallIssue] # synthesized class takes no arguments return self._polars_ext def python_to_storage( @@ -227,7 +227,9 @@ def create_for_python_type( arrow_fields = [] field_annotations: list[tuple[str, Any]] = [] tokens: dict[str, str] = {} - for field_name in self.stored_field_names(python_type): + # Sorted by name: field order is not part of the identity, so it must not + # change the storage type either (same identity, same struct). + for field_name in sorted(self.stored_field_names(python_type)): annotation = hints.get(field_name, Any) try: arrow_type = converter.register_python_class(annotation) @@ -246,7 +248,7 @@ def create_for_python_type( record = uci.identity_record(self.kind, name, tokens) identity = uci.identity_string(record) - uci.mark_built(python_type, name) + uci.mark_built(python_type, name, identity) uci.remember(self.kind, name, python_type) uci.notice_main_class(python_type, name) logger.debug("%s: built %r for %r", type(self).__name__, identity, python_type) @@ -299,6 +301,7 @@ def reconstruct_from_arrow( f"record for category {self.category!r}: {record!r}." ) + kind = self.kind candidates = uci.candidate_classes(kind, name) if not candidates: raise ImportError( @@ -310,8 +313,8 @@ def reconstruct_from_arrow( found: list[UserClassLogicalType] = [] for cls in candidates: - converter.register_python_class(cls) - lt = converter.get_logical_type(cls) + _ = converter.register_python_class(cls) # raises for unsupported classes + lt = converter.get_logical_type_for_python_type(cls) if not isinstance(lt, UserClassLogicalType): continue if lt.logical_type_name == arrow_extension_name: diff --git a/src/orcapod/user_types.py b/src/orcapod/user_types.py new file mode 100644 index 000000000..d9b4cceaf --- /dev/null +++ b/src/orcapod/user_types.py @@ -0,0 +1,97 @@ +"""Registering your own data types (dataclasses and pydantic models). + +Orcapod identifies a dataclass or pydantic model by its **name**, its **kind** +(dataclass or pydantic) and its **fields** (names and types, in any order). Two classes +that agree on all three are the same type: results stored with one are found again with +the other. See ``docs/concepts/your-own-data-types.md``. + +``register_type`` declares the name a class is identified by. ``set_type_naming`` sets +how classes that declare no name are named. +""" + +from __future__ import annotations + +from collections.abc import Callable +from typing import TypeVar, overload + +import orcapod.logical_types.user_class_identity as _uci + +T = TypeVar("T", bound=type) + +__all__ = ["register_type", "set_type_naming"] + + +@overload +def register_type(cls: T, /, *, name: str | None = None) -> T: ... + + +@overload +def register_type(*, name: str | None = None) -> Callable[[T], T]: ... + + +def register_type( + cls: T | None = None, /, *, name: str | None = None +) -> T | Callable[[T], T]: + """Declare the name a dataclass or pydantic model is identified by. + + Use it as a decorator, with or without a name, or call it on a class you do not + own. Without a name, the class is named by the naming setting (see + ``set_type_naming``): a class defined in a notebook or script by its class name, + a class defined in a module by its module path and class name. + + Declare the name before the class is first used in a source or pod. A class + defined again with the same name and fields (a re-run notebook cell) is the same + type; with different fields it is a different type. + + Args: + cls: The class, when called directly. + name: The declared name, e.g. ``"Point"`` or ``"geo.Point"``. It must not be + empty, contain ``@`` or whitespace, or start with ``orcapod.``. + + Returns: + The class unchanged, or a decorator that returns it unchanged. + + Raises: + TypeError: If the class is not a dataclass or pydantic model. + ValueError: If the name is invalid, the class is defined inside a function, or + the class was already used under a different name. + + Example: + >>> import dataclasses + >>> import orcapod as op + >>> @op.register_type(name="Point") + ... @dataclasses.dataclass + ... class Point: + ... x: int + ... y: int + """ + + def decorate(target: T) -> T: + _uci.declare(target, name) + return target + + if cls is None: + return decorate + return decorate(cls) + + +def set_type_naming( + *, main: _uci.NamingStyle | None = None, module: _uci.NamingStyle | None = None +) -> None: + """Set how dataclasses and pydantic models that declare no name are named. + + Applies to classes first used after the call, so call it before defining or using + your classes. + + Args: + main: Classes defined in a notebook or a script run directly (``__main__``): + ``"name"`` (default) uses the class name (``"Point"``); ``"qualified"`` + uses ``"__main__.Point"``. + module: Classes defined in an importable module: ``"qualified"`` (default) + uses the module path and class name (``"pkg.models.Point"``); ``"name"`` + uses the class name only. + + Raises: + ValueError: If a value is not ``"name"`` or ``"qualified"``. + """ + _uci.set_naming(main=main, module=module) diff --git a/src/orcapod/utils/schema_utils.py b/src/orcapod/utils/schema_utils.py index f6146a6b3..43efab3ef 100644 --- a/src/orcapod/utils/schema_utils.py +++ b/src/orcapod/utils/schema_utils.py @@ -41,11 +41,15 @@ def check_schema_compatibility( ) -> bool: from beartype.door import is_subhint + from orcapod.logical_types.user_class_identity import equivalent_hints + for key, type_info in incoming_types.items(): if key not in receiving_types: logger.warning(f"Key '{key}' not found in parameter types.") return False - if not is_subhint(type_info, receiving_types[key]): + if not is_subhint(type_info, receiving_types[key]) and not equivalent_hints( + type_info, receiving_types[key] + ): logger.warning( f"Type mismatch for key '{key}': expected {receiving_types[key]}, got {type_info}." ) diff --git a/tests/test_logical_types/test_cache_behavior.py b/tests/test_logical_types/test_cache_behavior.py index 52416f3e3..92e168ceb 100644 --- a/tests/test_logical_types/test_cache_behavior.py +++ b/tests/test_logical_types/test_cache_behavior.py @@ -31,8 +31,11 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name diff --git a/tests/test_logical_types/test_dataclass_logical_type_factory.py b/tests/test_logical_types/test_dataclass_logical_type_factory.py index c300c8bc5..415103d35 100644 --- a/tests/test_logical_types/test_dataclass_logical_type_factory.py +++ b/tests/test_logical_types/test_dataclass_logical_type_factory.py @@ -12,30 +12,25 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name -def _metadata(cls: type) -> dict: +def _metadata(cls: type) -> dict[str, object]: """Return the parsed Arrow extension metadata of a user class (ITL-881).""" import json from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) - assert lt is not None - return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) - - -def _metadata(cls: type) -> dict: - """Return the parsed Arrow extension metadata of a user class (ITL-881).""" - import json - - from orcapod.contexts import get_default_context - - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) diff --git a/tests/test_logical_types/test_default_context_factories.py b/tests/test_logical_types/test_default_context_factories.py index 550d5c185..48b5a8f8d 100644 --- a/tests/test_logical_types/test_default_context_factories.py +++ b/tests/test_logical_types/test_default_context_factories.py @@ -23,8 +23,11 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name diff --git a/tests/test_logical_types/test_pydantic_logical_type_factory.py b/tests/test_logical_types/test_pydantic_logical_type_factory.py index f5087722e..a3bce9a06 100644 --- a/tests/test_logical_types/test_pydantic_logical_type_factory.py +++ b/tests/test_logical_types/test_pydantic_logical_type_factory.py @@ -14,19 +14,25 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name -def _metadata(cls: type) -> dict: +def _metadata(cls: type) -> dict[str, object]: """Return the parsed Arrow extension metadata of a user class (ITL-881).""" import json from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return json.loads(lt.get_arrow_extension_type().__arrow_ext_serialize__()) diff --git a/tests/test_logical_types/test_roundtrips.py b/tests/test_logical_types/test_roundtrips.py index 4fe911fb7..fafbf21f2 100644 --- a/tests/test_logical_types/test_roundtrips.py +++ b/tests/test_logical_types/test_roundtrips.py @@ -37,8 +37,11 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context + from orcapod.types import Schema - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name @@ -477,7 +480,7 @@ def test_delta_polars_read_delta(tmp_path: Path) -> None: f"Expected a Polars extension type on column 'point', got {col_dtype!r}" ) assert col_dtype.ext_name() == type_name, ( - f"Expected extension name {type_name!r}, got {col_dtype.ext_name()!r}" + f"Expected extension name {type_name!r}, got {col_dtype.ext_name()!r}" # pyright: ignore[reportAttributeAccessIssue] # polars stubs lack ext_name ) diff --git a/tests/test_logical_types/test_schema_compatibility.py b/tests/test_logical_types/test_schema_compatibility.py index b7c527010..6289f6b09 100644 --- a/tests/test_logical_types/test_schema_compatibility.py +++ b/tests/test_logical_types/test_schema_compatibility.py @@ -32,7 +32,9 @@ def _identity(cls: type) -> str: """Return the identity string (logical type name) of a user class (ITL-881).""" from orcapod.contexts import get_default_context - lt = get_default_context().type_converter.get_logical_type_for_python_type(cls) + converter = get_default_context().type_converter + converter.ensure_types_registered_for_schemas(Schema({"value": cls})) + lt = converter.get_logical_type(cls) assert lt is not None return lt.logical_type_name diff --git a/tests/test_user_types/child_harness.py b/tests/test_user_types/child_harness.py new file mode 100644 index 000000000..c28b944a2 --- /dev/null +++ b/tests/test_user_types/child_harness.py @@ -0,0 +1,149 @@ +"""Run workshop scenarios in fresh Python processes (ITL-881 tests). + +A user class's identity depends on process-wide state (where the class is defined, what +is registered, Arrow's global extension registry), so the scenarios run each "kernel" or +"script" in its own interpreter. A child reports by printing one ``JSON:`` line +through ``report``. + +``WSJOB`` is a module the scenarios write next to their scripts: it runs one function pod +over a ``DictSource`` in a ``PipelineJob`` stored in a ``DeltaTableDatabase``. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import textwrap +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +JSON_MARKER = "JSON:" +HARNESS_DIR = Path(__file__).parent + + +def write_files(root: Path, files: Mapping[str, str]) -> None: + """Write ``files`` (relative path to dedented content) under ``root``.""" + for rel, content in files.items(): + path = root / rel + path.parent.mkdir(parents=True, exist_ok=True) + _ = path.write_text(textwrap.dedent(content).lstrip(), encoding="utf-8") + + +def run_child( + argv: Sequence[str], + *, + cwd: Path, + env: Mapping[str, str] | None = None, + timeout: float = 300.0, +) -> Any: + """Run ``python `` in ``cwd`` and return the payload it reported. + + Args: + argv: Arguments after the interpreter, e.g. ``["-c", code]``. + cwd: Working directory; also put on ``PYTHONPATH`` with this harness. + env: Extra environment variables. + timeout: Seconds before the child is killed. + + Returns: + The reported payload. + + Raises: + AssertionError: If the child exits non-zero or reports nothing. + """ + child_env = dict(os.environ) + child_env["PYTHONPATH"] = os.pathsep.join([str(cwd), str(HARNESS_DIR)]) + child_env.update(env or {}) + proc = subprocess.run( + [sys.executable, *argv], + cwd=cwd, + env=child_env, + capture_output=True, + text=True, + check=False, + timeout=timeout, + ) + for line in proc.stdout.splitlines(): + if line.startswith(JSON_MARKER): + return json.loads(line[len(JSON_MARKER) :]) + raise AssertionError( + f"child reported nothing (exit {proc.returncode}):\n{proc.stderr[-4000:]}" + ) + + +def report(payload: Any) -> None: + """Print ``payload`` as the JSON line the parent reads, then exit at once. + + ``os._exit`` skips interpreter shutdown: a process that read Delta tables sometimes + hangs in Arrow's thread-pool destructor at shutdown (seen in the ITL-881 + experiments; out of scope here). + """ + print(JSON_MARKER + json.dumps(payload, default=str)) + sys.stdout.flush() + sys.stderr.flush() + os._exit(0) + + +def error_text(exc: BaseException) -> str: + """Return ``": "`` for an exception.""" + first = str(exc).splitlines()[0] if str(exc) else "" + return f"{type(exc).__name__}: {first}" + + +#: Job module written next to the scenario scripts. +WSJOB = ''' +import os +from pathlib import Path + +from orcapod import PipelineJob, function_pod +from orcapod.databases import DeltaTableDatabase +from orcapod.sources import DictSource + + +def table_paths(root): + return sorted( + str(p.parent.relative_to(root)) for p in Path(root).rglob("_delta_log") if p.is_dir() + ) + + +def make_pod(point_cls): + """Return ``(pod, calls)`` for ``shift(p: point_cls) -> point_cls``.""" + calls = [] + + def shift(p): + calls.append(p) + return p + + shift.__annotations__ = {"p": point_cls, "return": point_cls} + return function_pod(output_keys="q")(shift), calls + + +def run_job(point_cls, pod_and_calls=None, rows=None): + """Run the shift pod over two rows; report calls, rows and identity.""" + pod, calls = pod_and_calls or make_pod(point_cls) + if rows is None: + names = list(getattr(point_cls, "model_fields", None) or point_cls.__dataclass_fields__) + rows = [{n: i for n in names} for i in (1, 2)] + data = [{"k": f"r{i}", "p": point_cls(**row)} for i, row in enumerate(rows, 1)] + before = len(calls) + db_root = os.environ["WS_DB"] + job = PipelineJob(name="ws", store=DeltaTableDatabase(Path(db_root))) + with job: + pod.pod(DictSource(data=data, tag_columns=["k"], source_id="s"), label="shift") + job.run() + node = job.nodes["shift"] + values = [d.as_dict()["q"] for _, d in node.iter_data()] + return { + "calls": len(calls) - before, + "rows": node.as_table().num_rows, + "q_class_is_point": all(type(v) is point_cls for v in values), + "pod_pipeline_hash": pod.pod.pipeline_hash().to_string(), + "node_identity_path": list(node.node_identity_path), + "table_paths": table_paths(db_root), + "extension_name": getattr( + node.as_table().schema.field("q").type, "extension_name", None + ), + } +''' diff --git a/tests/test_user_types/test_read_path.py b/tests/test_user_types/test_read_path.py new file mode 100644 index 000000000..7c085b1f9 --- /dev/null +++ b/tests/test_user_types/test_read_path.py @@ -0,0 +1,52 @@ +"""Stored and registered shapes of a user class must agree on read (ITL-881, S2). + +A stored column whose extension name matches a registered class but whose Arrow storage +differs (possible only if two data contexts map a field differently, or a table was +written by other means) must raise ``TypeIdentityError`` from ``DeltaTableDatabase`` +instead of being logged and read as "no records". +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path +from typing import Any + +import deltalake +import pyarrow as pa +import pytest + +import orcapod as op +from orcapod.databases import DeltaTableDatabase +from orcapod.errors import TypeIdentityError +from orcapod.sources import DictSource + + +@op.register_type(name="itl881.StoredPoint") +@dataclasses.dataclass +class StoredPoint: + x: int + y: int + + +def _extension_field_metadata() -> dict[bytes | str, bytes | str]: + value: Any = StoredPoint(1, 2) + source = DictSource(data=[{"k": "a", "p": value}], tag_columns=["k"]) + ext = source.as_table().schema.field("p").type + return { + b"ARROW:extension:name": ext.extension_name.encode(), + b"ARROW:extension:metadata": ext.__arrow_ext_serialize__(), + } + + +def test_storage_mismatch_raises_instead_of_reading_nothing(tmp_path: Path) -> None: + wrong = pa.struct([pa.field("x", pa.float64()), pa.field("y", pa.int64())]) + field = pa.field("q", wrong).with_metadata(_extension_field_metadata()) + table = pa.table( + {"q": pa.array([{"x": 1.5, "y": 2}], type=wrong)}, + schema=pa.schema([field]), + ) + deltalake.write_deltalake(str(tmp_path / "points"), table) + + with pytest.raises(TypeIdentityError, match="expected storage_type"): + DeltaTableDatabase(tmp_path).get_all_records(("points",)) diff --git a/tests/test_user_types/test_register_type.py b/tests/test_user_types/test_register_type.py new file mode 100644 index 000000000..204c78ac3 --- /dev/null +++ b/tests/test_user_types/test_register_type.py @@ -0,0 +1,361 @@ +"""Public API for user class identity: ``op.register_type`` and ``op.set_type_naming`` (ITL-881). + +Identity is observed through the public interface: the Arrow extension name of a +``DictSource`` column (``as_table``), the source's ``pipeline_hash`` and values read back +with ``iter_data``. Every test declares names under ``itl881.`` that no other test uses, +because declarations are process-wide. +""" + +from __future__ import annotations + +import dataclasses +from collections.abc import Iterator +from typing import Any + +import pydantic +import pytest + +import orcapod as op +from orcapod.errors import RecursiveTypeError +from orcapod.logical_types import ( + user_class_identity as uci, +) # get_naming: restore the policy +from orcapod.sources import DictSource + + +def _source(value: Any) -> DictSource: + return DictSource(data=[{"k": "a", "p": value}], tag_columns=["k"]) + + +def _type_name(value: Any) -> str: + return _source(value).as_table().schema.field("p").type.extension_name + + +def _name_part(value: Any) -> str: + return _type_name(value).split("@")[0] + + +# --------------------------------------------------------------------------- +# Names +# --------------------------------------------------------------------------- + + +@op.register_type(name="itl881.Declared") +@dataclasses.dataclass +class Declared: + x: int + + +@dataclasses.dataclass +class Undeclared: + x: int + + +@op.register_type +@dataclasses.dataclass +class BareDecorated: + x: int + + +@dataclasses.dataclass +class CalledLater: + x: int + + +def test_declared_name_is_the_name_component() -> None: + assert _name_part(Declared(1)) == "itl881.Declared" + + +def test_module_class_defaults_to_module_path() -> None: + assert _name_part(Undeclared(1)) == f"{__name__}.Undeclared" + + +def test_bare_decorator_uses_the_naming_setting() -> None: + assert _name_part(BareDecorated(1)) == f"{__name__}.BareDecorated" + + +def test_call_form_returns_the_class() -> None: + assert op.register_type(CalledLater, name="itl881.CalledLater") is CalledLater + assert _name_part(CalledLater(1)) == "itl881.CalledLater" + + +@pytest.mark.parametrize("name", ["", "a@b", "a b", "orcapod.Point"]) +def test_invalid_names_raise(name: str) -> None: + with pytest.raises(ValueError): + op.register_type(name=name)(Undeclared) + + +def test_non_dataclass_raises_type_error() -> None: + class Plain: + x: int = 0 + + with pytest.raises(TypeError, match="dataclasses and pydantic models"): + op.register_type(Plain, name="itl881.Plain") + + +def test_local_class_raises() -> None: + @dataclasses.dataclass + class Local: + x: int + + with pytest.raises(ValueError, match="inside a function"): + op.register_type(Local, name="itl881.Local") + + +@dataclasses.dataclass +class UsedFirst: + x: int + + +def test_declaring_after_use_under_another_name_raises() -> None: + _type_name(UsedFirst(1)) + with pytest.raises(ValueError, match="already registered as"): + op.register_type(UsedFirst, name="itl881.UsedFirst") + + +@dataclasses.dataclass +class RenamedByPolicy: + x: int + + +def test_set_type_naming_module_name() -> None: + before = uci.get_naming() + op.set_type_naming(module="name") + try: + assert _name_part(RenamedByPolicy(1)) == "RenamedByPolicy" + finally: + op.set_type_naming(**before) + + +def test_set_type_naming_rejects_unknown_style() -> None: + with pytest.raises(ValueError, match="'name' or 'qualified'"): + op.set_type_naming(main="short") # pyright: ignore[reportArgumentType] + + +# --------------------------------------------------------------------------- +# Fields +# --------------------------------------------------------------------------- + + +@op.register_type(name="itl881.Shape") +@dataclasses.dataclass +class ShapeInt: + x: int + y: int + + +@op.register_type(name="itl881.Shape") +@dataclasses.dataclass +class ShapeFloat: + x: int + y: float + + +@op.register_type(name="itl881.Shape") +class ShapeModel(pydantic.BaseModel): + x: int + y: int + + +def test_same_name_different_field_type_is_a_different_type() -> None: + a, b = _source(ShapeInt(1, 2)), _source(ShapeFloat(1, 2.0)) + assert _type_name(ShapeInt(1, 2)) != _type_name(ShapeFloat(1, 2.0)) + assert a.pipeline_hash() != b.pipeline_hash() + + +def test_dataclass_and_pydantic_model_are_different_types() -> None: + assert _type_name(ShapeInt(1, 2)) != _type_name(ShapeModel(x=1, y=2)) + + +@op.register_type(name="itl881.Defaults") +@dataclasses.dataclass +class DefaultsA: + x: int = 0 + + def method(self) -> int: + return self.x + + @property + def doubled(self) -> int: + return 2 * self.x + + +@op.register_type(name="itl881.Defaults") +@dataclasses.dataclass +class DefaultsB: + """Same fields as DefaultsA; different default, no methods.""" + + x: int = 5 + + +def test_defaults_methods_and_properties_are_not_identity() -> None: + first = _type_name(DefaultsA(1)) + with pytest.warns(UserWarning, match="same identity"): + second = _type_name(DefaultsB(1)) + assert first == second + + +@op.register_type(name="itl881.Computed") +class ComputedA(pydantic.BaseModel): + x: int + + @pydantic.computed_field + @property + def doubled(self) -> int: + return 2 * self.x + + +@op.register_type(name="itl881.Computed") +class ComputedB(pydantic.BaseModel): + x: int + + +def test_pydantic_computed_fields_are_not_identity() -> None: + first = _type_name(ComputedA(x=1)) + with pytest.warns(UserWarning, match="same identity"): + second = _type_name(ComputedB(x=1)) + assert first == second + + +# --------------------------------------------------------------------------- +# Nested classes (P1) and registration order +# --------------------------------------------------------------------------- + + +@op.register_type(name="itl881.InnerA") +@dataclasses.dataclass +class InnerA: + value: int + + +@op.register_type(name="itl881.InnerB") +@dataclasses.dataclass +class InnerB: + value: int + + +@op.register_type(name="itl881.InnerA") +@dataclasses.dataclass +class InnerAFloat: + value: float + + +@op.register_type(name="itl881.Outer") +@dataclasses.dataclass +class OuterA: + inner: InnerA + + +@op.register_type(name="itl881.Outer") +@dataclasses.dataclass +class OuterB: + inner: InnerB + + +@op.register_type(name="itl881.Outer") +@dataclasses.dataclass +class OuterAFloat: + inner: InnerAFloat + + +def test_nested_class_contributes_its_name() -> None: + assert _type_name(OuterA(InnerA(1))) != _type_name(OuterB(InnerB(1))) + + +def test_nested_class_contributes_its_fields() -> None: + assert _type_name(OuterA(InnerA(1))) != _type_name(OuterAFloat(InnerAFloat(1.0))) + + +@dataclasses.dataclass +class LateInner: + value: int + + +@op.register_type(name="itl881.OrderOuter") +@dataclasses.dataclass +class EarlyOuter: + inner: LateInner + + +# Declared after the outer class was decorated, before either is used. +op.register_type(LateInner, name="itl881.OrderInner") + + +@op.register_type(name="itl881.OrderInner") +@dataclasses.dataclass +class FirstInner: + value: int + + +@op.register_type(name="itl881.OrderOuter") +@dataclasses.dataclass +class SecondOuter: + inner: FirstInner + + +def test_registration_order_does_not_matter() -> None: + late_order = _type_name(EarlyOuter(LateInner(1))) + with pytest.warns(UserWarning, match="same identity"): + natural_order = _type_name(SecondOuter(FirstInner(1))) + assert late_order == natural_order + + +def test_nested_values_round_trip() -> None: + _, data = next(iter(_source(OuterA(InnerA(3))).iter_data())) + value = data.as_dict()["p"] + assert value == OuterA(InnerA(3)) + assert type(value.inner) is InnerA + + +@dataclasses.dataclass +class Node: + value: int + next: Node | None = None + + +@dataclasses.dataclass +class CycleA: + b: CycleB | None = None + + +@dataclasses.dataclass +class CycleB: + a: CycleA | None = None + + +def test_self_referential_class_is_rejected_naming_the_field() -> None: + with pytest.raises(RecursiveTypeError, match=r"Node\.next -> Node"): + _source(Node(1)) + + +def test_cyclic_classes_are_rejected_naming_the_path() -> None: + with pytest.raises(RecursiveTypeError, match=r"CycleA\.b -> CycleB\.a -> CycleA"): + _source(CycleA()) + + +# --------------------------------------------------------------------------- +# Subclasses +# --------------------------------------------------------------------------- + + +@dataclasses.dataclass +class Base: + x: int + + +@dataclasses.dataclass +class Child(Base): + z: int = 0 + + +def test_subclass_has_its_own_identity_and_keeps_its_fields() -> None: + assert _type_name(Base(1)) != _type_name(Child(1, 2)) + _, data = next(iter(_source(Child(1, 2)).iter_data())) + assert data.as_dict()["p"] == Child(1, 2) + + +@pytest.fixture(autouse=True) +def _no_policy_leak() -> Iterator[None]: + before = uci.get_naming() + yield + assert uci.get_naming() == before diff --git a/tests/test_user_types/test_workshop_scenarios.py b/tests/test_user_types/test_workshop_scenarios.py new file mode 100644 index 000000000..98231c023 --- /dev/null +++ b/tests/test_user_types/test_workshop_scenarios.py @@ -0,0 +1,365 @@ +"""Workshop scenarios W1–W12 for user class identity (ITL-881). + +Each scenario runs notebook cells (IPython ``InteractiveShell``, as a Jupyter kernel runs +them) or scripts in fresh processes, against one ``DeltaTableDatabase``, and drives the +pipeline only through the public API (``DictSource``, ``function_pod``, ``PipelineJob``, +``ExtensionAwareDatabase``). Scenario numbers follow the findings in PR #288. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +from tests.test_user_types.child_harness import WSJOB, run_child, write_files + +pytestmark = pytest.mark.integration + +KERNEL = r""" +import warnings +from IPython.core.interactiveshell import InteractiveShell +from child_harness import error_text, report + +shell = InteractiveShell.instance() +CELLS = { + "setup": "import wsjob\nfrom dataclasses import dataclass\nimport orcapod as op\n", + "define": "@dataclass\nclass Point:\n x: int\n y: int\n", + "define_reordered": "@dataclass\nclass Point:\n y: int\n x: int\n", + "define_float": "@dataclass\nclass Point:\n x: int\n y: float\n", + "define_model": "import pydantic\nclass Point(pydantic.BaseModel):\n x: int\n y: int\n", + "import_mod": "from wsmodels import Point\n", + "reload_mod": "import importlib, wsmodels\nimportlib.reload(wsmodels)\nfrom wsmodels import Point\n", + "pod": "shift = wsjob.make_pod(Point)\n", + "run": "_result = wsjob.run_job(Point)\n", + "run_pod": "_result = wsjob.run_job(Point, shift)\n", +} +log = [] +for step in SCENARIO: + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = shell.run_cell(CELLS[step], store_history=False, silent=True) + entry = { + "cell": step, + "warnings": [str(w.message) for w in caught if issubclass(w.category, UserWarning)], + } + err = res.error_in_exec or res.error_before_exec + if err is not None: + entry["error"] = error_text(err) + elif step.startswith("run"): + entry.update(shell.user_ns["_result"]) + log.append(entry) +report(log) +""" + +WSMODELS = """ +from dataclasses import dataclass + + +@dataclass +class Point: + x: int + y: int +""" + + +@pytest.fixture +def workdir(tmp_path: Path) -> Path: + write_files(tmp_path, {"wsjob.py": WSJOB, "wsmodels.py": WSMODELS}) + (tmp_path / "db").mkdir() + return tmp_path + + +def _kernel(workdir: Path, cells: list[str]) -> list[dict[str, Any]]: + code = f"SCENARIO = {cells!r}\n" + KERNEL + return run_child(["-c", code], cwd=workdir, env={"WS_DB": str(workdir / "db")}) + + +def _runs(log: list[dict[str, Any]]) -> list[dict[str, Any]]: + errors = [e for e in log if "error" in e] + assert not errors, errors + return [e for e in log if e["cell"].startswith("run")] + + +def _script(workdir: Path, code: str, **env: str) -> Any: + return run_child( + ["-c", code], cwd=workdir, env={"WS_DB": str(workdir / "db"), **env} + ) + + +# --------------------------------------------------------------------------- +# Notebook cell re-runs in one kernel +# --------------------------------------------------------------------------- + + +def test_w1_rerun_define_cell_same_fields_rebinds(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define", "run", "define", "run"]) + first, second = _runs(log) + assert (first["calls"], first["rows"]) == (2, 2) + assert (second["calls"], second["rows"]) == (0, 2) + assert second["q_class_is_point"] + assert first["pod_pipeline_hash"] == second["pod_pipeline_hash"] + + +def test_w1_old_pod_cell_with_redefined_class(workdir: Path) -> None: + """Re-running only the class cell keeps the earlier pod cell usable.""" + log = _kernel(workdir, ["setup", "define", "pod", "run_pod", "define", "run_pod"]) + first, second = _runs(log) + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + + +def test_w1_run_all_twice(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define", "pod", "run_pod"] * 2) + first, second = _runs(log) + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + + +def test_field_order_does_not_change_identity(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define", "run", "define_reordered", "run"]) + first, second = _runs(log) + assert second["calls"] == 0 + assert first["extension_name"] == second["extension_name"] + + +def test_w2_rerun_with_changed_fields_is_a_new_type(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define", "run", "define_float", "run"]) + first, second = _runs(log) + assert (second["calls"], second["rows"]) == (2, 2) + assert second["q_class_is_point"] + assert first["pod_pipeline_hash"] != second["pod_pipeline_hash"] + assert first["extension_name"].split("@")[0] == "Point" + assert second["extension_name"].split("@")[0] == "Point" + assert first["extension_name"] != second["extension_name"] + assert set(first["table_paths"]) < set(second["table_paths"]) + + +def test_w3_pydantic_rerun_rebinds(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define_model", "run", "define_model", "run"]) + first, second = _runs(log) + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + + +def test_dataclass_and_pydantic_with_same_name_and_fields_are_distinct( + workdir: Path, +) -> None: + log = _kernel(workdir, ["setup", "define", "run", "define_model", "run"]) + first, second = _runs(log) + assert second["calls"] == 2 + assert first["pod_pipeline_hash"] != second["pod_pipeline_hash"] + assert first["extension_name"] != second["extension_name"] + + +def test_w4_module_reload_rebinds(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "import_mod", "run", "reload_mod", "run"]) + first, second = _runs(log) + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + assert first["extension_name"].startswith("wsmodels.Point@") + + +def test_main_class_notice_once(workdir: Path) -> None: + log = _kernel(workdir, ["setup", "define", "run", "define", "run"]) + notices = [w for e in log for w in e["warnings"] if "notebook or script" in w] + assert len(notices) == 1 + assert "'Point'" in notices[0] + + +# --------------------------------------------------------------------------- +# Across processes +# --------------------------------------------------------------------------- + + +def test_w5_restart_same_class_hits_cache(workdir: Path) -> None: + first = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + second = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + + +def test_w6_restart_with_changed_fields_recomputes(workdir: Path) -> None: + """Changed fields after a restart compute into new tables instead of 0 rows.""" + first = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + changed = _runs(_kernel(workdir, ["setup", "define_float", "run"]))[0] + back = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + assert (changed["calls"], changed["rows"]) == (2, 2) + assert first["node_identity_path"] != changed["node_identity_path"] + assert (back["calls"], back["rows"]) == (0, 2) + + +def test_w7_script_as_main_twice(workdir: Path) -> None: + write_files( + workdir, + { + "pkg/__init__.py": "", + "pkg/pipeline.py": """ + from dataclasses import dataclass + import wsjob + from child_harness import report + + @dataclass + class Point: + x: int + y: int + + def main(): + report(wsjob.run_job(Point)) + + if __name__ == "__main__": + main() + """, + }, + ) + env = {"WS_DB": str(workdir / "db")} + first = run_child(["pkg/pipeline.py"], cwd=workdir, env=env) + second = run_child(["-m", "pkg.pipeline"], cwd=workdir, env=env) + assert (first["calls"], second["calls"]) == (2, 0) + + +def test_w8_script_then_import_hits_with_declared_name(workdir: Path) -> None: + write_files( + workdir, + { + "pkg/__init__.py": "", + "pkg/pipeline.py": """ + from dataclasses import dataclass + import orcapod as op + import wsjob + from child_harness import report + + @op.register_type(name="Point") + @dataclass + class Point: + x: int + y: int + + def main(): + report(wsjob.run_job(Point)) + + if __name__ == "__main__": + main() + """, + }, + ) + env = {"WS_DB": str(workdir / "db")} + first = run_child(["-m", "pkg.pipeline"], cwd=workdir, env=env) + second = run_child( + ["-c", "import pkg.pipeline; pkg.pipeline.main()"], cwd=workdir, env=env + ) + assert (first["calls"], second["calls"], second["rows"]) == (2, 0, 2) + assert first["node_identity_path"] == second["node_identity_path"] + + +def test_w11_notebook_to_module_with_declared_name(workdir: Path) -> None: + """A class moved from a notebook into a module keeps its results under its name.""" + write_files( + workdir, + { + "pkg/__init__.py": "", + "pkg/models.py": """ + from dataclasses import dataclass + import orcapod as op + + @op.register_type(name="Point") + @dataclass + class Point: + y: int + x: int + """, + }, + ) + notebook = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + module = _script( + workdir, + "import wsjob\nfrom child_harness import report\nfrom pkg.models import Point\n" + "report(wsjob.run_job(Point))\n", + ) + assert (module["calls"], module["rows"]) == (0, 2) + assert module["pod_pipeline_hash"] == notebook["pod_pipeline_hash"] + assert module["node_identity_path"] == notebook["node_identity_path"] + assert module["table_paths"] == notebook["table_paths"] + assert module["extension_name"] == notebook["extension_name"] + + +def test_w11_module_without_declared_name_is_a_different_type(workdir: Path) -> None: + notebook = _runs(_kernel(workdir, ["setup", "define", "run"]))[0] + module = _runs(_kernel(workdir, ["setup", "import_mod", "run"]))[0] + assert module["calls"] == 2 + assert module["pod_pipeline_hash"] != notebook["pod_pipeline_hash"] + + +# --------------------------------------------------------------------------- +# Reading stored values +# --------------------------------------------------------------------------- + +READER = """ +import os +from pathlib import Path +from child_harness import error_text, report +from orcapod.contexts import get_default_context +from orcapod.databases import DeltaTableDatabase, ExtensionAwareDatabase +import wsjob + +{define} + +root = Path(os.environ["WS_DB"]) +path = tuple(next(p for p in wsjob.table_paths(root) if "/_result/" in p).split("/")) +converter = get_default_context().type_converter +try: + table = ExtensionAwareDatabase(DeltaTableDatabase(root), converter=converter).get_all_records(path) + rows = converter.arrow_table_to_python_dicts(table.select(["q"])) + report({{"values": sorted(repr(r["q"]) for r in rows), "classes": sorted({{type(r["q"]).__module__ + "." + type(r["q"]).__qualname__ for r in rows}})}}) +except Exception as exc: + report({{"error": error_text(exc), "message": str(exc)}}) +""" + + +def _store_from_notebook(workdir: Path) -> None: + _runs(_kernel(workdir, ["setup", "define", "run"])) + + +def test_w12_read_notebook_results_from_script_with_same_class(workdir: Path) -> None: + _store_from_notebook(workdir) + define = "from dataclasses import dataclass\n@dataclass\nclass Point:\n x: int\n y: int\n" + out = _script(workdir, READER.format(define=define)) + assert out.get("error") is None, out + assert out["values"] == ["Point(x=1, y=1)", "Point(x=2, y=2)"] + assert out["classes"] == ["__main__.Point"] + + +def test_read_with_declared_class_from_a_module(workdir: Path) -> None: + _store_from_notebook(workdir) + write_files( + workdir, + { + "geo.py": """ + from dataclasses import dataclass + import orcapod as op + + @op.register_type(name="Point") + @dataclass + class Point: + x: int + y: int + """ + }, + ) + out = _script(workdir, READER.format(define="import geo")) + assert out.get("error") is None, out + assert out["classes"] == ["geo.Point"] + + +def test_read_without_the_class_names_it_and_how_to_register(workdir: Path) -> None: + _store_from_notebook(workdir) + out = _script(workdir, READER.format(define="")) + assert out["error"].startswith("ImportError: Column 'q'") + assert "'Point'" in out["message"] + assert "register_type" in out["message"] + assert "{x: int, y: int}" in out["message"] + + +def test_read_with_different_fields_raises_naming_both_shapes(workdir: Path) -> None: + _store_from_notebook(workdir) + define = "from dataclasses import dataclass\n@dataclass\nclass Point:\n x: int\n y: float\n" + out = _script(workdir, READER.format(define=define)) + assert out["error"].startswith("TypeIdentityError: Column 'q'"), out + assert "{x: int, y: int}" in out["message"] + assert "{x: int, y: float}" in out["message"] From a7d0044170054581263d80b095b835ba567e997f Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:41:01 -0800 Subject: [PATCH 007/147] test(hashing): golden values for v0.3 user class identity (ITL-881) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Intended hash changes (approved): exactly the 25 entries listed in the spec move — 17 values change and 8 per-row system-tag keys are replaced because they embed the moved pipeline hash. arrow.json and pipeline.json are unchanged. New fixtures pin a declared-name dataclass and pydantic model (same name and fields, distinct types), a nested dataclass with a container field, list[GoldenPoint], and field-order twins that pin equal identity values. Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/golden/cases.py | 97 ++++++++++++++++++ tests/test_hashing/golden/data/datagram.json | 100 +++++++++++++++---- tests/test_hashing/golden/data/schema.json | 22 +++- tests/test_hashing/golden/data/semantic.json | 10 +- 4 files changed, 202 insertions(+), 27 deletions(-) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index a2e12d808..c7bb02029 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -197,11 +197,82 @@ class GoldenModel(pydantic.BaseModel): count: int +@dataclasses.dataclass +class GoldenDeclared: + """Dataclass declared under a name (ITL-881), with an optional field.""" + + label: str + score: float | None + + +class GoldenDeclaredModel(pydantic.BaseModel): + """Pydantic model declared under the same name and fields as ``GoldenDeclared``.""" + + label: str + score: float | None + + +@dataclasses.dataclass +class GoldenInner: + """Dataclass nested in ``GoldenOuter``; its identity enters the outer one.""" + + value: int + + +@dataclasses.dataclass +class GoldenOuter: + """Dataclass with a nested dataclass field and a container field.""" + + inner: GoldenInner + tags: list[str] + + +@dataclasses.dataclass +class GoldenTwinAB: + """Declared ``golden.Twin``; same fields as ``GoldenTwinBA`` in another order.""" + + a: int + b: str + + +@dataclasses.dataclass +class GoldenTwinBA: + """Declared ``golden.Twin``; must pin the same values as ``GoldenTwinAB``.""" + + b: str + a: int + + +def _declare_golden_names() -> None: + """Declare the names of the ITL-881 fixtures (releases before v0.3 have no names).""" + try: + from orcapod import register_type + except ImportError: + return + register_type(GoldenDeclared, name="golden.Declared") + register_type(GoldenDeclaredModel, name="golden.Declared") + register_type(GoldenInner, name="golden.Inner") + register_type(GoldenOuter, name="golden.Outer") + register_type(GoldenTwinAB, name="golden.Twin") + register_type(GoldenTwinBA, name="golden.Twin") + + +_declare_golden_names() + + def register_logical_classes() -> None: """Register the dataclass and pydantic fixtures with the default type converter.""" tc = _ctx().type_converter tc.register_python_class(GoldenPoint) tc.register_python_class(GoldenModel) + for cls in ( + GoldenDeclared, + GoldenDeclaredModel, + GoldenOuter, + GoldenTwinAB, + GoldenTwinBA, + ): + tc.register_python_class(cls) PATH_A = Path("/golden/data/a.txt") @@ -280,6 +351,12 @@ def _semantic_types() -> dict[str, Any]: "set_uuid": set[uuid.UUID], "dataclass": GoldenPoint, "pydantic": GoldenModel, + "dataclass_declared": GoldenDeclared, + "pydantic_declared": GoldenDeclaredModel, + "dataclass_nested": GoldenOuter, + "dataclass_twin_ab": GoldenTwinAB, + "dataclass_twin_ba": GoldenTwinBA, + "list_dataclass": list[GoldenPoint], } @@ -586,6 +663,26 @@ def _datagram_rows() -> dict[str, list[dict[str, Any]]]: ], "list_path": [{"k": "r1", "c": [PATH_A, PATH_B]}, {"k": "r2", "c": []}], "set_uuid": [{"k": "r1", "c": {UUID_A, UUID_B}}, {"k": "r2", "c": {UUID_A}}], + "dataclass_declared": [ + {"k": "r1", "c": GoldenDeclared("a", 0.5)}, + {"k": "r2", "c": GoldenDeclared("b", None)}, + ], + "pydantic_declared": [ + {"k": "r1", "c": GoldenDeclaredModel(label="a", score=0.5)}, + {"k": "r2", "c": GoldenDeclaredModel(label="b", score=None)}, + ], + "dataclass_nested": [ + {"k": "r1", "c": GoldenOuter(GoldenInner(1), ["x", "y"])}, + {"k": "r2", "c": GoldenOuter(GoldenInner(2), [])}, + ], + "dataclass_twin_ab": [ + {"k": "r1", "c": GoldenTwinAB(1, "x")}, + {"k": "r2", "c": GoldenTwinAB(2, "y")}, + ], + "dataclass_twin_ba": [ + {"k": "r1", "c": GoldenTwinBA("x", 1)}, + {"k": "r2", "c": GoldenTwinBA("y", 2)}, + ], } diff --git a/tests/test_hashing/golden/data/datagram.json b/tests/test_hashing/golden/data/datagram.json index e9bf5b544..4a93fe769 100644 --- a/tests/test_hashing/golden/data/datagram.json +++ b/tests/test_hashing/golden/data/datagram.json @@ -89,17 +89,65 @@ "_tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "_tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e" ], - "dataclass.content_hash": "semantic_v0.1:018c841dd6f3d3df7f15689df0ac43adf3b8086655f315311fd53fd98abae218", - "dataclass.pipeline_hash": "semantic_v0.1:c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c", - "dataclass.row[{\"k\":\"r1\"}]._tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "26db93a5eab45b0485c2699148b0600d", - "dataclass.row[{\"k\":\"r1\"}]._tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "golden_dataclass", - "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000015e306dd402af59e45e6f5a9113e71e691f96006cb8cf63246504800071050f55", - "dataclass.row[{\"k\":\"r2\"}]._tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "71d5c6ce2f7b5b5b9dfdc7a61a7df366", - "dataclass.row[{\"k\":\"r2\"}]._tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "golden_dataclass", - "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b088edbc3232e6d836e6036a75d5c0a0d83864166d21a16ad113608e21afe445", + "dataclass.content_hash": "semantic_v0.1:30806932620b0715e691bb1eaa74f58e587f5e6eeef08833cff14c9d305411f4", + "dataclass.pipeline_hash": "semantic_v0.1:32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635", + "dataclass.row[{\"k\":\"r1\"}]._tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "26db93a5eab45b0485c2699148b0600d", + "dataclass.row[{\"k\":\"r1\"}]._tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "golden_dataclass", + "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000135da7ac200e567d06cc849537f7726196f576b8498fc7e46bf0849740f67c692", + "dataclass.row[{\"k\":\"r2\"}]._tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "71d5c6ce2f7b5b5b9dfdc7a61a7df366", + "dataclass.row[{\"k\":\"r2\"}]._tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "golden_dataclass", + "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000106c5a4a7b5a8d235e66b3aad694a92cdc826c5b416f3148ba93e22767407f564", "dataclass.system_tag_columns": [ - "_tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c", - "_tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c" + "_tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635", + "_tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635" + ], + "dataclass_declared.content_hash": "semantic_v0.1:7e97c35e68612f04276d8de55d9a5232c377aec9ff1ae10e5fa2dd3bc5507f16", + "dataclass_declared.pipeline_hash": "semantic_v0.1:6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c", + "dataclass_declared.row[{\"k\":\"r1\"}]._tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "570e538241085842a7994cad88c5a26b", + "dataclass_declared.row[{\"k\":\"r1\"}]._tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "golden_dataclass_declared", + "dataclass_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001c4cb62165dfd97cc7f0c1949354ba602436ffa8e2f0517fd922e87588cde5782", + "dataclass_declared.row[{\"k\":\"r2\"}]._tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "0e35aec1a9605d428b2741d9f13012d5", + "dataclass_declared.row[{\"k\":\"r2\"}]._tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "golden_dataclass_declared", + "dataclass_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b2cf415f684673e3fa593084372b3b8b0bb165bc82619fde3ba44e7e78ef2001", + "dataclass_declared.system_tag_columns": [ + "_tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c", + "_tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c" + ], + "dataclass_nested.content_hash": "semantic_v0.1:8887fd4cfaf912c84ccbf49786814c789b8ea972774056c132dadf0106016a22", + "dataclass_nested.pipeline_hash": "semantic_v0.1:839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665", + "dataclass_nested.row[{\"k\":\"r1\"}]._tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "3b11b0ab021d59f28c8efaf02ad69aee", + "dataclass_nested.row[{\"k\":\"r1\"}]._tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "golden_dataclass_nested", + "dataclass_nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000014be5231f5cdac2ab183a8616ac8e7390385a21c5ee2efdcc951f4feb81c05ddd", + "dataclass_nested.row[{\"k\":\"r2\"}]._tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "178f8da0e9ed5520827e648f5a890193", + "dataclass_nested.row[{\"k\":\"r2\"}]._tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "golden_dataclass_nested", + "dataclass_nested.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000111562233d9e99e74ad15355aafd40b2e377e4a0799db72330416a082490552d9", + "dataclass_nested.system_tag_columns": [ + "_tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665", + "_tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665" + ], + "dataclass_twin_ab.content_hash": "semantic_v0.1:313370fb33d9bea1af19c4fb2c8206a97a3e26186f686930e51bd1514bb63616", + "dataclass_twin_ab.pipeline_hash": "semantic_v0.1:95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", + "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "f35be6f60b275f2c81fbc49366c2db67", + "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ab", + "dataclass_twin_ab.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001a4f0170a2d724ce2395c6ab6595053a9482d58e05eac5b3311d9e6b56e4f80f4", + "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "7307d67233695c9ab4952175e9735bc9", + "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ab", + "dataclass_twin_ab.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001721c4768261afb271b69fef8680f8df75e5ab8d3494bc8da048ec1322a18fd25", + "dataclass_twin_ab.system_tag_columns": [ + "_tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", + "_tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e" + ], + "dataclass_twin_ba.content_hash": "semantic_v0.1:9dd09b862ed11da8904ae2df1d7224394adea497e46773b2aa5778448583aeda", + "dataclass_twin_ba.pipeline_hash": "semantic_v0.1:95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", + "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "65d4fb8402a75d059b3fbfa686a22420", + "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ba", + "dataclass_twin_ba.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001a4f0170a2d724ce2395c6ab6595053a9482d58e05eac5b3311d9e6b56e4f80f4", + "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "3d1c61df9063598ca8178d55c4a47ff9", + "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ba", + "dataclass_twin_ba.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001721c4768261afb271b69fef8680f8df75e5ab8d3494bc8da048ec1322a18fd25", + "dataclass_twin_ba.system_tag_columns": [ + "_tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", + "_tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e" ], "dataframe.content_hash": "semantic_v0.1:f18422d163082bd23e621e586c3d484ab3a25b6c1ad959b08daa4d63ef461e68", "dataframe.pipeline_hash": "semantic_v0.1:38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14", @@ -173,17 +221,29 @@ "_tag_source_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09", "_tag_record_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09" ], - "pydantic.content_hash": "semantic_v0.1:8e5ec65a6c6e45a672c6abf5dbf1e3332e8efebef128be193704364769780612", - "pydantic.pipeline_hash": "semantic_v0.1:aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967", - "pydantic.row[{\"k\":\"r1\"}]._tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "cb9bbf1a138d5c848e86f99bab7c0acb", - "pydantic.row[{\"k\":\"r1\"}]._tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "golden_pydantic", - "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000012280b233c56771fa4f901eff64090067273d718ac8d36b881b78d79d4128ab15", - "pydantic.row[{\"k\":\"r2\"}]._tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "10c1dbaba95a5c5fa7c14661231224eb", - "pydantic.row[{\"k\":\"r2\"}]._tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "golden_pydantic", - "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000138d77ff36d7b2f3c326a128b325d8219e31f55be9264a21abf51638993059c7e", + "pydantic.content_hash": "semantic_v0.1:cdcdbc5ee86513eb5cb5a5ac65597283624cba0fdcaff90160ba2944e1bf1ca2", + "pydantic.pipeline_hash": "semantic_v0.1:1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad", + "pydantic.row[{\"k\":\"r1\"}]._tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "cb9bbf1a138d5c848e86f99bab7c0acb", + "pydantic.row[{\"k\":\"r1\"}]._tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "golden_pydantic", + "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000017138b5a4d4d4ebc1cfafd17ad95a997c673a0ec614a1fb453b0b0adca1a698c9", + "pydantic.row[{\"k\":\"r2\"}]._tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "10c1dbaba95a5c5fa7c14661231224eb", + "pydantic.row[{\"k\":\"r2\"}]._tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "golden_pydantic", + "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b32cd5f30c4ee4b7a38bce31ac7d94419a9212d33ad07e5d08757ce8e94e9118", "pydantic.system_tag_columns": [ - "_tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967", - "_tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967" + "_tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad", + "_tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad" + ], + "pydantic_declared.content_hash": "semantic_v0.1:ec413e1781f85a762bb5fe2a55794d766e3390954c7606b9d6a1e3e47f71b1af", + "pydantic_declared.pipeline_hash": "semantic_v0.1:922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13", + "pydantic_declared.row[{\"k\":\"r1\"}]._tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "fec6f63bdc695be9aeddfdd100112d42", + "pydantic_declared.row[{\"k\":\"r1\"}]._tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "golden_pydantic_declared", + "pydantic_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001b2d14e386eb560940d8d9f54e03045b5f20f2e8cf560e22fd724fad309d233b5", + "pydantic_declared.row[{\"k\":\"r2\"}]._tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "89983c8f835d51c3aaf2bbca0d7f3d55", + "pydantic_declared.row[{\"k\":\"r2\"}]._tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "golden_pydantic_declared", + "pydantic_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001f215ba71a9109b04f48d2627489608d2d83b599f0a4dd97ddc8862f298a9a0a0", + "pydantic_declared.system_tag_columns": [ + "_tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13", + "_tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13" ], "scalars.content_hash": "semantic_v0.1:6b123e343edb881e71967e118f1e90b2dc46d20fb60e43b9b7991e3ef822a1d8", "scalars.pipeline_hash": "semantic_v0.1:1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07", diff --git a/tests/test_hashing/golden/data/schema.json b/tests/test_hashing/golden/data/schema.json index a33e88f3e..c2502b47d 100644 --- a/tests/test_hashing/golden/data/schema.json +++ b/tests/test_hashing/golden/data/schema.json @@ -2,7 +2,7 @@ "compute_schema_hash.chars_16": "a3e81ce4fbd701a0", "compute_schema_hash.full": "a3e81ce4fbd701a0fb97cae134f236953f7f66a8e7b7f4741339b45fe9f485ba", "multi_field.empty": "semantic_v0.1:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", - "multi_field.logical_mix": "semantic_v0.1:29372233b68192c5ff9e48524aaf6073c904eac64479d1c43a6f37e3819a6f1b", + "multi_field.logical_mix": "semantic_v0.1:db02d466882c5127e1e6ab0e264865450102d30773579e666514eba27a72a1ca", "multi_field.nullable_type": "semantic_v0.1:560844c33bb686eddb98de8d5b608f38d845ee5bf64e6a094f9a8326679aafe9", "multi_field.optional_field": "semantic_v0.1:bc030299c9a955ea2f18a2c746121180b5ed0d1fd6265c95ef243088e392b843", "multi_field.two_scalars": "semantic_v0.1:bc030299c9a955ea2f18a2c746121180b5ed0d1fd6265c95ef243088e392b843", @@ -11,8 +11,16 @@ "single_field.bool.schema_hash": "semantic_v0.1:d0845939265975cb588b4821498b4844a6ae18d9613e6bb9bd8c274db521eab0", "single_field.bytes.arrow_type": "large_binary", "single_field.bytes.schema_hash": "semantic_v0.1:c691ab0e3a0ce67c3f8d7b9065d60e2a14afcdce983b362c62de85c9dcf19a52", - "single_field.dataclass.arrow_type": "extension[struct]", - "single_field.dataclass.schema_hash": "semantic_v0.1:a65f5d99a90ffeb981faabef6dbdb0fa03a547287f882e5330b449294580e4e7", + "single_field.dataclass.arrow_type": "extension[struct]", + "single_field.dataclass.schema_hash": "semantic_v0.1:b57fab5e732720af0d9263f2e2885591790c4760d35675acaf084670ef2c4545", + "single_field.dataclass_declared.arrow_type": "extension[struct]", + "single_field.dataclass_declared.schema_hash": "semantic_v0.1:ec33ba184eb852e3b75c7c9e7541b8695f259aec588f541ac6a119d3198b1215", + "single_field.dataclass_nested.arrow_type": "extension[struct, tags: large_list>]", + "single_field.dataclass_nested.schema_hash": "semantic_v0.1:1aee0c1efd5f33da7984bfa837a50b7706ea65a5dc68ad05ba844efffda262d5", + "single_field.dataclass_twin_ab.arrow_type": "extension[struct]", + "single_field.dataclass_twin_ab.schema_hash": "semantic_v0.1:38f6e2b2c6dc3d14de182f2938128a353f056222e184caa202413e3b42edde1e", + "single_field.dataclass_twin_ba.arrow_type": "extension[struct]", + "single_field.dataclass_twin_ba.schema_hash": "semantic_v0.1:38f6e2b2c6dc3d14de182f2938128a353f056222e184caa202413e3b42edde1e", "single_field.dataframe.arrow_type": "extension[large_binary]", "single_field.dataframe.schema_hash": "semantic_v0.1:2544430b3979b1e0ad8cecb9ab45c93b91dda546d5a02a59b5587c73aa04ecd8", "single_field.dict_str_int.arrow_type": "large_list>", @@ -21,6 +29,8 @@ "single_field.float.schema_hash": "semantic_v0.1:1d165501d30136c6e1309c58db930165e0cb51aae8469323283a95297a880191", "single_field.int.arrow_type": "int64", "single_field.int.schema_hash": "semantic_v0.1:2faa1297b96c1248edfafb2422fded3963672852acbe2973155351859eab83d2", + "single_field.list_dataclass.arrow_type": "extension[large_list>]", + "single_field.list_dataclass.schema_hash": "semantic_v0.1:d078161a5775aafe2afde86369e2cc57dcdf00ed010292015a60147b8eee1c47", "single_field.list_int.arrow_type": "large_list", "single_field.list_int.schema_hash": "semantic_v0.1:906c18997d1191d374db82ea5cc763d3d04dc3ceccb4f3fde82cf6d53d0f8dc7", "single_field.list_list_int.arrow_type": "large_list>", @@ -33,8 +43,10 @@ "single_field.optional_int.schema_hash": "semantic_v0.1:16d9f93d8da8f8c5a582352f7022b1191ab1e4593722bbd37fdc1238d34cdc13", "single_field.path.arrow_type": "extension[large_string]", "single_field.path.schema_hash": "semantic_v0.1:84e095c4212af644cc68357daa75a3b9094b953c7fb9030fb4af86f8310b30c3", - "single_field.pydantic.arrow_type": "extension[struct]", - "single_field.pydantic.schema_hash": "semantic_v0.1:a62df3085a2efc4667d0e216148c6898991129df452b12c12cff79900503ce72", + "single_field.pydantic.arrow_type": "extension[struct]", + "single_field.pydantic.schema_hash": "semantic_v0.1:259566a29cf776cf97fd5c6a0201a8026b3d41644a24c2cb1d3b7544507be4ad", + "single_field.pydantic_declared.arrow_type": "extension[struct]", + "single_field.pydantic_declared.schema_hash": "semantic_v0.1:d05e37ea7d328efbd2833d604e9cff59e1b1ca5f9b09133bb37e9ba524c94896", "single_field.set_uuid.arrow_type": "extension[large_list]", "single_field.set_uuid.schema_hash": "semantic_v0.1:73501f29f5a260ab092f67ea9d88c8c3d5d8d5c90bba8d9aea88b3b5400c6272", "single_field.str.arrow_type": "large_string", diff --git a/tests/test_hashing/golden/data/semantic.json b/tests/test_hashing/golden/data/semantic.json index 7e29628ae..ea3156065 100644 --- a/tests/test_hashing/golden/data/semantic.json +++ b/tests/test_hashing/golden/data/semantic.json @@ -1,18 +1,24 @@ { "type.bool": "semantic_v0.1:cd8f971312b872f0e9a3ef40aa58d6bb537258e72a9a142e0cf712f0a35020e3", "type.bytes": "semantic_v0.1:f65cea8d1f29ca5b96d67caf1ced2f630b509e32574a7b78b9071958818a1c59", - "type.dataclass": "semantic_v0.1:84b8714a03d2a6784736ae38435d9ffbe87c397ea64af275d91e8bdd8281adbc", + "type.dataclass": "semantic_v0.1:814071362ffe7481bde7e4173907a59d5bc6df940f81c9838764d0f83973abbe", + "type.dataclass_declared": "semantic_v0.1:fee6ecce526f9b6791a66e6bb638a4ba0934e79beff9fe247bf71807224fc786", + "type.dataclass_nested": "semantic_v0.1:214904a85c3d6015ee0b2e83dceba5852d3196dbb295b0f4550198a707c23ad1", + "type.dataclass_twin_ab": "semantic_v0.1:921bd905da2fce576b9a8d0274b9716e5cba49edefd123b955c8ebcffe24835c", + "type.dataclass_twin_ba": "semantic_v0.1:921bd905da2fce576b9a8d0274b9716e5cba49edefd123b955c8ebcffe24835c", "type.dataframe": "semantic_v0.1:51e3c34453121e01086e882959cd12746b3236ff7b259a40a42d43493bf26dff", "type.dict_str_int": "semantic_v0.1:f0d36b085fa65f3368d1d98687ffcb6c80099a418419e45a77d434b37e4393dd", "type.float": "semantic_v0.1:0059c08fc7dfe19dc8d095b7e95d56e8d377bbe49ac30376a00105eca2e7c75f", "type.int": "semantic_v0.1:24be397c58c33686e96b4f85a9f443fd2483564ec7dee1c14b96da3e60ec16ac", + "type.list_dataclass": "semantic_v0.1:9568f2ffe6c7dd3c492522b02c06d59b6f57a6888e0db1d59db41dcac1bcf363", "type.list_int": "semantic_v0.1:5c376e29b0cfebd85277c71c07c9996afaf06fab35f81681b82f705c9feafa3e", "type.list_list_int": "semantic_v0.1:ba62756844df6c82b48a739fadd6bb568a3507896dc5dba192dd3e0285635c48", "type.list_path": "semantic_v0.1:e2029c1032fad111f798363e1f718ecde715573c690c3cc78f33824a246a82ce", "type.ndarray": "semantic_v0.1:b6f37b2016cf3ee3062c3e715bab25fa02b68e92931bce243684300ba1dda5b6", "type.optional_int": "semantic_v0.1:91bc2bd8f4f083d6cbf6157350e693dd52b0bc06c57b7719db8581712098cd73", "type.path": "semantic_v0.1:02a12969616ecd278958ffca715be2865fe4a32c27523059df37245c5048a880", - "type.pydantic": "semantic_v0.1:01ca376f8a57dd68d9645036cb5fe247b410f6dfc6ef3cc09661a9e71d2eb4d6", + "type.pydantic": "semantic_v0.1:76a4fd9c0acc945dfba19bb621c91b9eb9f6361df50e6f17a38a3ae7b0b947bc", + "type.pydantic_declared": "semantic_v0.1:a18f664d19b35c92b7a12ac983973d9aedb5e8c86f23a168207f3eee3225cc24", "type.set_uuid": "semantic_v0.1:0ac1b6672ab001f6ee64f406e6d1af3a21024d6f78ef0c7414014cdfad96eaf6", "type.str": "semantic_v0.1:309d607de6633b959ae5118171f079b2830f4b1340064fac065652e305b038ad", "type.tuple_int_str": "semantic_v0.1:65ac71aa155fbfa28072adba04ed55d8d99d89e9c87a0d5ed45ccd15fcafc099", From 285c884c979e01ad02dbadeac32dd564bdebe3ab Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:45:59 -0800 Subject: [PATCH 008/147] feat(types): full-length identity hash, per-class qualified option, no naming switch (ITL-881) Developer decisions (Nov 10, 2026): the identity string carries the full 64-hex SHA-256; the process-wide naming setter is dropped in favour of the fixed location rule (__main__ -> class name, module -> module path) plus register_type(qualified=False) per class. Registration never modifies the class, is idempotent for the same name and raises for a different one. Golden values regenerated: the same 25 listed entries move, plus additions. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/__init__.py | 3 +- .../logical_types/user_class_identity.py | 115 +++++++------- src/orcapod/user_types.py | 69 ++++----- tests/test_hashing/golden/data/datagram.json | 140 +++++++++--------- tests/test_hashing/golden/data/schema.json | 34 ++--- tests/test_hashing/golden/data/semantic.json | 16 +- tests/test_user_types/test_identity_record.py | 27 +--- tests/test_user_types/test_register_type.py | 57 ++++--- 8 files changed, 220 insertions(+), 241 deletions(-) diff --git a/src/orcapod/__init__.py b/src/orcapod/__init__.py index 59b3f6571..95fa89776 100644 --- a/src/orcapod/__init__.py +++ b/src/orcapod/__init__.py @@ -61,7 +61,7 @@ from .extensions import OrcapodExtension, register_extension # Your own data types: dataclass / pydantic identity (ITL-881) -from .user_types import register_type, set_type_naming +from .user_types import register_type # Post-run hook types (ITL-523) from .hooks import ( @@ -111,7 +111,6 @@ "register_extension", # Your own data types (ITL-881) "register_type", - "set_type_naming", # Post-run hook types (ITL-523) "HookConfig", "InvocationStatus", diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index 56486bfd1..aa6fefcca 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -4,9 +4,8 @@ spec ``superpowers/specs/2026-11-10-user-class-identity-v0.3.md``): - ``kind``: ``"dataclass"`` or ``"pydantic"``. -- ``name``: a declared name, or a name chosen by the naming policy (the qualified name - for ``__main__`` classes is just ``__qualname__`` by default; module classes use - ``module.qualname`` by default). +- ``name``: a declared name; otherwise ``__qualname__`` for a class defined in + ``__main__`` (a notebook or script) and ``module.qualname`` for a module class. - ``fields``: a mapping from each stored field's name to its *type token*. It is serialized with sorted keys, so field order never enters the identity. @@ -14,7 +13,7 @@ Arrow extension name. ``identity_string`` is the only place that defines its format. This module also holds the process-wide state that belongs to classes rather than to a -data context: declared names, the naming policy and the table of classes seen under each +data context: declared names and the table of classes seen under each ``(kind, name)``, which the read path uses to find a class for a stored column. """ @@ -38,12 +37,6 @@ KIND_DATACLASS = "dataclass" KIND_PYDANTIC = "pydantic" -NamingStyle = Literal["name", "qualified"] - -_NAMING_STYLES: frozenset[str] = frozenset({"name", "qualified"}) -_DEFAULT_NAMING: dict[str, NamingStyle] = {"main": "name", "module": "qualified"} -_naming: dict[str, NamingStyle] = dict(_DEFAULT_NAMING) - #: Modules whose classes are "defined in a notebook or script". ``__mp_main__`` is the #: name a ``multiprocessing`` spawn child gives the main module. MAIN_MODULES: frozenset[str] = frozenset({"__main__", "__mp_main__"}) @@ -51,8 +44,8 @@ #: Prefix reserved for Orcapod's own types (``orcapod.path``, ``orcapod.uuid``, ...). RESERVED_PREFIX = "orcapod." -#: Number of hex digits of the SHA-256 fingerprint in an identity string. -FINGERPRINT_HEX_DIGITS = 16 +#: Number of hex digits of the SHA-256 fingerprint in an identity string (the full digest). +FINGERPRINT_HEX_DIGITS = 64 _NAME_FORBIDDEN = re.compile(r"[@\s]") @@ -120,47 +113,38 @@ def validate_name(name: str) -> None: ) -def get_naming() -> dict[str, NamingStyle]: - """Return the current naming policy as ``{"main": ..., "module": ...}``.""" - return dict(_naming) - +def is_main_class(cls: type) -> bool: + """Return True if ``cls`` was defined in ``__main__`` (or ``__mp_main__``).""" + return cls.__module__ in MAIN_MODULES -def set_naming( - main: NamingStyle | None = None, module: NamingStyle | None = None -) -> None: - """Set how classes that declare no name are named. - Applies to classes registered after the call. +def default_name(cls: type, *, qualified: bool | None = None) -> str: + """Return the name a class gets without a declared name. Args: - main: ``"name"`` (default) or ``"qualified"`` for classes defined in - ``__main__`` (a notebook or a script run directly). - module: ``"qualified"`` (default) or ``"name"`` for classes defined in an - importable module. + cls: A dataclass or pydantic model class. + qualified: ``None`` applies the location rule: ``__qualname__`` for a class + defined in ``__main__`` (or ``__mp_main__``), ``module.qualname`` for a + module class. ``False`` always gives ``__qualname__``; ``True`` always gives + ``module.qualname``. - Raises: - ValueError: If a value is not ``"name"`` or ``"qualified"``. + Returns: + The name component. """ - for key, value in (("main", main), ("module", module)): - if value is None: - continue - if value not in _NAMING_STYLES: - raise ValueError( - f"Invalid naming style for {key}: {value!r}. Use 'name' or 'qualified'." - ) - _naming[key] = typing.cast(NamingStyle, value) - + if qualified is None: + qualified = not is_main_class(cls) + if not qualified: + return cls.__qualname__ + module = cls.__module__ + if not is_main_class(cls): + from orcapod.utils.canonical_type_names import canonical_module_name -def is_main_class(cls: type) -> bool: - """Return True if ``cls`` was defined in ``__main__`` (or ``__mp_main__``).""" - return cls.__module__ in MAIN_MODULES + module = canonical_module_name(cls) or module + return f"{module}.{cls.__qualname__}" def resolve_name(cls: type) -> str: - """Return the name component for ``cls``. - - The first rule that applies wins: a declared name; the naming policy for - ``__main__`` classes; the naming policy for module classes. + """Return the name component for ``cls``: its declared name or ``default_name``. Args: cls: A dataclass or pydantic model class. @@ -171,28 +155,27 @@ def resolve_name(cls: type) -> str: declared = _declared.get(cls) if declared is not None: return declared - style = _naming["main" if is_main_class(cls) else "module"] - if style == "name": - return cls.__qualname__ - module = cls.__module__ - if not is_main_class(cls): - from orcapod.utils.canonical_type_names import canonical_module_name + return default_name(cls) - module = canonical_module_name(cls) or module - return f"{module}.{cls.__qualname__}" - -def declare(cls: type, name: str | None) -> None: +def declare(cls: type, name: str | None, *, qualified: bool | None = None) -> None: """Record the name a user class is identified by. + Registration never modifies the class: the name is kept in a process-wide table + keyed by the class object. Declaring the same class again with the same name is a + no-op; any other name raises. + Args: cls: A dataclass or pydantic model class. - name: The declared name, or ``None`` to use the naming policy. + name: The declared name, or ``None`` for ``default_name(cls, qualified=...)``. + qualified: Only without ``name``: ``False`` for the class name alone, + ``True`` for ``module.qualname``, ``None`` for the location rule. Raises: TypeError: If ``cls`` is not a dataclass or pydantic model class. - ValueError: If the class is local, the name is invalid, or the class was - already registered in this process under a different name. + ValueError: If both ``name`` and ``qualified`` are given, the class is local, + the name is invalid, or the class is already registered (declared or used) + under a different name. """ kind = class_kind(cls) if kind is None: @@ -200,20 +183,24 @@ def declare(cls: type, name: str | None) -> None: f"Only dataclasses and pydantic models can be registered as types, " f"got {cls!r}." ) + if name is not None and qualified is not None: + raise ValueError( + "Pass either name=... or qualified=..., not both: qualified only chooses " + "between the class name and its module path when no name is given." + ) reject_local_class(cls) if name is not None: validate_name(name) - effective = name if name is not None else resolve_name(cls) - built_as = _built.get(cls) - if built_as is not None and built_as != effective: + effective = name if name is not None else default_name(cls, qualified=qualified) + previous = _built.get(cls) or _declared.get(cls) + if previous is not None and previous != effective: raise ValueError( f"Cannot register {cls.__module__}.{cls.__qualname__} as {effective!r}: it " - f"is already registered as {built_as!r} in this process, because it was " - f"used before this declaration. Declare the name before the class is first " - f"used (for example with the @op.register_type decorator)." + f"is already registered as {previous!r} in this process, and a class has " + f"one name. Declare the name once, before the class is first used (for " + f"example with the @op.register_type decorator)." ) - if name is not None: - _declared[cls] = name + _declared[cls] = effective remember(kind, effective, cls) diff --git a/src/orcapod/user_types.py b/src/orcapod/user_types.py index d9b4cceaf..ab8069aa2 100644 --- a/src/orcapod/user_types.py +++ b/src/orcapod/user_types.py @@ -5,8 +5,9 @@ that agree on all three are the same type: results stored with one are found again with the other. See ``docs/concepts/your-own-data-types.md``. -``register_type`` declares the name a class is identified by. ``set_type_naming`` sets -how classes that declare no name are named. +``register_type`` declares the name a class is identified by. Without it, a class +defined in a notebook or script (``__main__``) is named by its class name and a class +defined in a module by its module path and class name. """ from __future__ import annotations @@ -18,43 +19,57 @@ T = TypeVar("T", bound=type) -__all__ = ["register_type", "set_type_naming"] +__all__ = ["register_type"] @overload -def register_type(cls: T, /, *, name: str | None = None) -> T: ... +def register_type( + cls: T, /, *, name: str | None = None, qualified: bool | None = None +) -> T: ... @overload -def register_type(*, name: str | None = None) -> Callable[[T], T]: ... +def register_type( + *, name: str | None = None, qualified: bool | None = None +) -> Callable[[T], T]: ... def register_type( - cls: T | None = None, /, *, name: str | None = None + cls: T | None = None, + /, + *, + name: str | None = None, + qualified: bool | None = None, ) -> T | Callable[[T], T]: """Declare the name a dataclass or pydantic model is identified by. - Use it as a decorator, with or without a name, or call it on a class you do not - own. Without a name, the class is named by the naming setting (see - ``set_type_naming``): a class defined in a notebook or script by its class name, - a class defined in a module by its module path and class name. + Use it as a decorator, with or without arguments, or call it on a class you do + not own. The class is never modified: the name is recorded in Orcapod's registry + and the same class object is returned. + + Without arguments the class keeps its default name: its class name if it is + defined in a notebook or script, its module path and class name if it is defined + in a module. ``qualified=False`` names a module class by its class name alone, + without retyping it. - Declare the name before the class is first used in a source or pod. A class - defined again with the same name and fields (a re-run notebook cell) is the same - type; with different fields it is a different type. + Declare the name before the class is first used in a source or pod. Declaring the + same class again with the same name does nothing; a different name raises. Args: cls: The class, when called directly. name: The declared name, e.g. ``"Point"`` or ``"geo.Point"``. It must not be empty, contain ``@`` or whitespace, or start with ``orcapod.``. + qualified: Only without ``name``. ``False``: the class name (``"Point"``). + ``True``: the module path and class name (``"pkg.models.Point"``). Returns: The class unchanged, or a decorator that returns it unchanged. Raises: TypeError: If the class is not a dataclass or pydantic model. - ValueError: If the name is invalid, the class is defined inside a function, or - the class was already used under a different name. + ValueError: If both ``name`` and ``qualified`` are given, the name is invalid, + the class is defined inside a function, or the class is already + registered under a different name. Example: >>> import dataclasses @@ -67,31 +82,9 @@ def register_type( """ def decorate(target: T) -> T: - _uci.declare(target, name) + _uci.declare(target, name, qualified=qualified) return target if cls is None: return decorate return decorate(cls) - - -def set_type_naming( - *, main: _uci.NamingStyle | None = None, module: _uci.NamingStyle | None = None -) -> None: - """Set how dataclasses and pydantic models that declare no name are named. - - Applies to classes first used after the call, so call it before defining or using - your classes. - - Args: - main: Classes defined in a notebook or a script run directly (``__main__``): - ``"name"`` (default) uses the class name (``"Point"``); ``"qualified"`` - uses ``"__main__.Point"``. - module: Classes defined in an importable module: ``"qualified"`` (default) - uses the module path and class name (``"pkg.models.Point"``); ``"name"`` - uses the class name only. - - Raises: - ValueError: If a value is not ``"name"`` or ``"qualified"``. - """ - _uci.set_naming(main=main, module=module) diff --git a/tests/test_hashing/golden/data/datagram.json b/tests/test_hashing/golden/data/datagram.json index 4a93fe769..641fb51cb 100644 --- a/tests/test_hashing/golden/data/datagram.json +++ b/tests/test_hashing/golden/data/datagram.json @@ -89,65 +89,65 @@ "_tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "_tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e" ], - "dataclass.content_hash": "semantic_v0.1:30806932620b0715e691bb1eaa74f58e587f5e6eeef08833cff14c9d305411f4", - "dataclass.pipeline_hash": "semantic_v0.1:32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635", - "dataclass.row[{\"k\":\"r1\"}]._tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "26db93a5eab45b0485c2699148b0600d", - "dataclass.row[{\"k\":\"r1\"}]._tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "golden_dataclass", - "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000135da7ac200e567d06cc849537f7726196f576b8498fc7e46bf0849740f67c692", - "dataclass.row[{\"k\":\"r2\"}]._tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "71d5c6ce2f7b5b5b9dfdc7a61a7df366", - "dataclass.row[{\"k\":\"r2\"}]._tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635": "golden_dataclass", - "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000106c5a4a7b5a8d235e66b3aad694a92cdc826c5b416f3148ba93e22767407f564", + "dataclass.content_hash": "semantic_v0.1:c5a9e1b35bfc3292f482d92c6da779430f90266b4969471c3c9dda92babb38e4", + "dataclass.pipeline_hash": "semantic_v0.1:7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e", + "dataclass.row[{\"k\":\"r1\"}]._tag_record_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e": "26db93a5eab45b0485c2699148b0600d", + "dataclass.row[{\"k\":\"r1\"}]._tag_source_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e": "golden_dataclass", + "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001bf6f92863abf8af4b0aac7124cdef0ddd5389e4d8cf951ac5ec452517f8a073b", + "dataclass.row[{\"k\":\"r2\"}]._tag_record_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e": "71d5c6ce2f7b5b5b9dfdc7a61a7df366", + "dataclass.row[{\"k\":\"r2\"}]._tag_source_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e": "golden_dataclass", + "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b0d45b9c6372b65c8a2be29561177153c1711e2ed1aab8323d83cfc084d52564", "dataclass.system_tag_columns": [ - "_tag_source_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635", - "_tag_record_id::32fb95c409827fcc6d4c6adea25bb26ec7a68fb56139f8c2315c9a8be40d7635" + "_tag_source_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e", + "_tag_record_id::7ec5213aa2a100ec89a8e49a6b8983e529d90154d6b6a061c8743c776f921b3e" ], - "dataclass_declared.content_hash": "semantic_v0.1:7e97c35e68612f04276d8de55d9a5232c377aec9ff1ae10e5fa2dd3bc5507f16", - "dataclass_declared.pipeline_hash": "semantic_v0.1:6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c", - "dataclass_declared.row[{\"k\":\"r1\"}]._tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "570e538241085842a7994cad88c5a26b", - "dataclass_declared.row[{\"k\":\"r1\"}]._tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "golden_dataclass_declared", - "dataclass_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001c4cb62165dfd97cc7f0c1949354ba602436ffa8e2f0517fd922e87588cde5782", - "dataclass_declared.row[{\"k\":\"r2\"}]._tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "0e35aec1a9605d428b2741d9f13012d5", - "dataclass_declared.row[{\"k\":\"r2\"}]._tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c": "golden_dataclass_declared", - "dataclass_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b2cf415f684673e3fa593084372b3b8b0bb165bc82619fde3ba44e7e78ef2001", + "dataclass_declared.content_hash": "semantic_v0.1:e6245cec578022a262899f3887df76aa88814cd15febaf5d0e1da03c21fef5d7", + "dataclass_declared.pipeline_hash": "semantic_v0.1:7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf", + "dataclass_declared.row[{\"k\":\"r1\"}]._tag_record_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf": "570e538241085842a7994cad88c5a26b", + "dataclass_declared.row[{\"k\":\"r1\"}]._tag_source_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf": "golden_dataclass_declared", + "dataclass_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000100c60f00122ee9d0689aa5f61af45feeaeaa37c636cd91f711ccaf5eeab6ff7d", + "dataclass_declared.row[{\"k\":\"r2\"}]._tag_record_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf": "0e35aec1a9605d428b2741d9f13012d5", + "dataclass_declared.row[{\"k\":\"r2\"}]._tag_source_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf": "golden_dataclass_declared", + "dataclass_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000016146a4d84d42a55151513f89988bc964b7c813f2384484a2ddaa0e8249e348d5", "dataclass_declared.system_tag_columns": [ - "_tag_source_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c", - "_tag_record_id::6d8c984d8b961f08f37441d936724b86e1b441a42daf7b106570630327b99b0c" + "_tag_source_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf", + "_tag_record_id::7173cbcdb0a61df572dce8cd566fb2fdee7efa6ba1bebc2470f9c55178389baf" ], - "dataclass_nested.content_hash": "semantic_v0.1:8887fd4cfaf912c84ccbf49786814c789b8ea972774056c132dadf0106016a22", - "dataclass_nested.pipeline_hash": "semantic_v0.1:839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665", - "dataclass_nested.row[{\"k\":\"r1\"}]._tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "3b11b0ab021d59f28c8efaf02ad69aee", - "dataclass_nested.row[{\"k\":\"r1\"}]._tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "golden_dataclass_nested", - "dataclass_nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000014be5231f5cdac2ab183a8616ac8e7390385a21c5ee2efdcc951f4feb81c05ddd", - "dataclass_nested.row[{\"k\":\"r2\"}]._tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "178f8da0e9ed5520827e648f5a890193", - "dataclass_nested.row[{\"k\":\"r2\"}]._tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665": "golden_dataclass_nested", - "dataclass_nested.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000111562233d9e99e74ad15355aafd40b2e377e4a0799db72330416a082490552d9", + "dataclass_nested.content_hash": "semantic_v0.1:6980d157081d777cdab23f84ed44901566e0f0b07977b980a4cc2c4af8fc327e", + "dataclass_nested.pipeline_hash": "semantic_v0.1:ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6", + "dataclass_nested.row[{\"k\":\"r1\"}]._tag_record_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6": "3b11b0ab021d59f28c8efaf02ad69aee", + "dataclass_nested.row[{\"k\":\"r1\"}]._tag_source_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6": "golden_dataclass_nested", + "dataclass_nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000013e4d70f87128e09feac88ca6a1473bb286b74369f99e120b7eef26465e0b83e2", + "dataclass_nested.row[{\"k\":\"r2\"}]._tag_record_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6": "178f8da0e9ed5520827e648f5a890193", + "dataclass_nested.row[{\"k\":\"r2\"}]._tag_source_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6": "golden_dataclass_nested", + "dataclass_nested.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001a4947a9cee130966d37ef41907b9c4704a87bc4e9521356953b3e6629ce00e2b", "dataclass_nested.system_tag_columns": [ - "_tag_source_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665", - "_tag_record_id::839d02c4c871eea07cd2b8fbe59963241153d2a5ea2eabb57562fa46961c9665" + "_tag_source_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6", + "_tag_record_id::ed1742fde2b22c4bf603b673363a85adf2c84a747caeda527e74575f96465be6" ], - "dataclass_twin_ab.content_hash": "semantic_v0.1:313370fb33d9bea1af19c4fb2c8206a97a3e26186f686930e51bd1514bb63616", - "dataclass_twin_ab.pipeline_hash": "semantic_v0.1:95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", - "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "f35be6f60b275f2c81fbc49366c2db67", - "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ab", - "dataclass_twin_ab.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001a4f0170a2d724ce2395c6ab6595053a9482d58e05eac5b3311d9e6b56e4f80f4", - "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "7307d67233695c9ab4952175e9735bc9", - "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ab", - "dataclass_twin_ab.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001721c4768261afb271b69fef8680f8df75e5ab8d3494bc8da048ec1322a18fd25", + "dataclass_twin_ab.content_hash": "semantic_v0.1:bfe47ccef93a7e2eb6f9becbe4325d4467422070c4a325c73a94a64e3194250a", + "dataclass_twin_ab.pipeline_hash": "semantic_v0.1:9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc", + "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "f35be6f60b275f2c81fbc49366c2db67", + "dataclass_twin_ab.row[{\"k\":\"r1\"}]._tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "golden_dataclass_twin_ab", + "dataclass_twin_ab.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000125e527402024a34d9d53489982ae2634a022bc6f9a0b8249606df28a66aa118c", + "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "7307d67233695c9ab4952175e9735bc9", + "dataclass_twin_ab.row[{\"k\":\"r2\"}]._tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "golden_dataclass_twin_ab", + "dataclass_twin_ab.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001f0b425f53eb41258940eaf8c37d0c18b1f3be4ecbde6969cba18ee8929013a63", "dataclass_twin_ab.system_tag_columns": [ - "_tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", - "_tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e" + "_tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc", + "_tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc" ], - "dataclass_twin_ba.content_hash": "semantic_v0.1:9dd09b862ed11da8904ae2df1d7224394adea497e46773b2aa5778448583aeda", - "dataclass_twin_ba.pipeline_hash": "semantic_v0.1:95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", - "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "65d4fb8402a75d059b3fbfa686a22420", - "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ba", - "dataclass_twin_ba.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001a4f0170a2d724ce2395c6ab6595053a9482d58e05eac5b3311d9e6b56e4f80f4", - "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "3d1c61df9063598ca8178d55c4a47ff9", - "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e": "golden_dataclass_twin_ba", - "dataclass_twin_ba.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001721c4768261afb271b69fef8680f8df75e5ab8d3494bc8da048ec1322a18fd25", + "dataclass_twin_ba.content_hash": "semantic_v0.1:124abac0b10fc20bec6865c930effbc767f96a2ae8387c4974b3bddedb72caa3", + "dataclass_twin_ba.pipeline_hash": "semantic_v0.1:9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc", + "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "65d4fb8402a75d059b3fbfa686a22420", + "dataclass_twin_ba.row[{\"k\":\"r1\"}]._tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "golden_dataclass_twin_ba", + "dataclass_twin_ba.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000125e527402024a34d9d53489982ae2634a022bc6f9a0b8249606df28a66aa118c", + "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "3d1c61df9063598ca8178d55c4a47ff9", + "dataclass_twin_ba.row[{\"k\":\"r2\"}]._tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc": "golden_dataclass_twin_ba", + "dataclass_twin_ba.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001f0b425f53eb41258940eaf8c37d0c18b1f3be4ecbde6969cba18ee8929013a63", "dataclass_twin_ba.system_tag_columns": [ - "_tag_source_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e", - "_tag_record_id::95a4adeb53a615cd4f439952de84c10081597823834f46ae700b2a6f0177c77e" + "_tag_source_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc", + "_tag_record_id::9e3acf5de3bee62961bbc669bde6a0513bb0552d09a8ce6c96efb030fe6d9acc" ], "dataframe.content_hash": "semantic_v0.1:f18422d163082bd23e621e586c3d484ab3a25b6c1ad959b08daa4d63ef461e68", "dataframe.pipeline_hash": "semantic_v0.1:38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14", @@ -221,29 +221,29 @@ "_tag_source_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09", "_tag_record_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09" ], - "pydantic.content_hash": "semantic_v0.1:cdcdbc5ee86513eb5cb5a5ac65597283624cba0fdcaff90160ba2944e1bf1ca2", - "pydantic.pipeline_hash": "semantic_v0.1:1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad", - "pydantic.row[{\"k\":\"r1\"}]._tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "cb9bbf1a138d5c848e86f99bab7c0acb", - "pydantic.row[{\"k\":\"r1\"}]._tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "golden_pydantic", - "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000017138b5a4d4d4ebc1cfafd17ad95a997c673a0ec614a1fb453b0b0adca1a698c9", - "pydantic.row[{\"k\":\"r2\"}]._tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "10c1dbaba95a5c5fa7c14661231224eb", - "pydantic.row[{\"k\":\"r2\"}]._tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad": "golden_pydantic", - "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b32cd5f30c4ee4b7a38bce31ac7d94419a9212d33ad07e5d08757ce8e94e9118", + "pydantic.content_hash": "semantic_v0.1:69cacd7109204aa8adf23d7e4b52c392d08b1ea58410c6b3edd2443627eb698b", + "pydantic.pipeline_hash": "semantic_v0.1:0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb", + "pydantic.row[{\"k\":\"r1\"}]._tag_record_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb": "cb9bbf1a138d5c848e86f99bab7c0acb", + "pydantic.row[{\"k\":\"r1\"}]._tag_source_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb": "golden_pydantic", + "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000141edb65540d977e17ea78d6ac46147f1a4e27de391d8e3ae733375ec0f68b0cc", + "pydantic.row[{\"k\":\"r2\"}]._tag_record_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb": "10c1dbaba95a5c5fa7c14661231224eb", + "pydantic.row[{\"k\":\"r2\"}]._tag_source_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb": "golden_pydantic", + "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001aa87be5aef99410d9d42fe7a2feaa5c7fbe85086f5f22d99a3989183b5c691d9", "pydantic.system_tag_columns": [ - "_tag_source_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad", - "_tag_record_id::1a84b3bbf2e3fba5441f0488550fac320cf4f04057035d608e3d462552db68ad" + "_tag_source_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb", + "_tag_record_id::0b31d2b41a234110192f6ec86c12f91796c8993e1b0cdc92670027ae511fc2cb" ], - "pydantic_declared.content_hash": "semantic_v0.1:ec413e1781f85a762bb5fe2a55794d766e3390954c7606b9d6a1e3e47f71b1af", - "pydantic_declared.pipeline_hash": "semantic_v0.1:922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13", - "pydantic_declared.row[{\"k\":\"r1\"}]._tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "fec6f63bdc695be9aeddfdd100112d42", - "pydantic_declared.row[{\"k\":\"r1\"}]._tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "golden_pydantic_declared", - "pydantic_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001b2d14e386eb560940d8d9f54e03045b5f20f2e8cf560e22fd724fad309d233b5", - "pydantic_declared.row[{\"k\":\"r2\"}]._tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "89983c8f835d51c3aaf2bbca0d7f3d55", - "pydantic_declared.row[{\"k\":\"r2\"}]._tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13": "golden_pydantic_declared", - "pydantic_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001f215ba71a9109b04f48d2627489608d2d83b599f0a4dd97ddc8862f298a9a0a0", + "pydantic_declared.content_hash": "semantic_v0.1:7fd221eb0acc8f9728479fd94e3cfde8f46d18909139d0fb37fdf6925736e084", + "pydantic_declared.pipeline_hash": "semantic_v0.1:fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007", + "pydantic_declared.row[{\"k\":\"r1\"}]._tag_record_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007": "fec6f63bdc695be9aeddfdd100112d42", + "pydantic_declared.row[{\"k\":\"r1\"}]._tag_source_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007": "golden_pydantic_declared", + "pydantic_declared.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001215e94d60e5fa74938aaab850d2ae94c7b8fd2e2f685af5cb476d0c85a035fef", + "pydantic_declared.row[{\"k\":\"r2\"}]._tag_record_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007": "89983c8f835d51c3aaf2bbca0d7f3d55", + "pydantic_declared.row[{\"k\":\"r2\"}]._tag_source_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007": "golden_pydantic_declared", + "pydantic_declared.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b4ff9032628c25bdce2bb7eb4408d46f20bbb8b69add7f19f2599c682976cf02", "pydantic_declared.system_tag_columns": [ - "_tag_source_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13", - "_tag_record_id::922544613bc92fc665c6522c877b5c589c28c6abd859a12205a7c5ddf8329f13" + "_tag_source_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007", + "_tag_record_id::fcc47a35b179eaca586c8b0c9877454ab9a9bbe7380c73e94531a9d4f4d90007" ], "scalars.content_hash": "semantic_v0.1:6b123e343edb881e71967e118f1e90b2dc46d20fb60e43b9b7991e3ef822a1d8", "scalars.pipeline_hash": "semantic_v0.1:1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07", diff --git a/tests/test_hashing/golden/data/schema.json b/tests/test_hashing/golden/data/schema.json index c2502b47d..e52408ed8 100644 --- a/tests/test_hashing/golden/data/schema.json +++ b/tests/test_hashing/golden/data/schema.json @@ -2,7 +2,7 @@ "compute_schema_hash.chars_16": "a3e81ce4fbd701a0", "compute_schema_hash.full": "a3e81ce4fbd701a0fb97cae134f236953f7f66a8e7b7f4741339b45fe9f485ba", "multi_field.empty": "semantic_v0.1:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", - "multi_field.logical_mix": "semantic_v0.1:db02d466882c5127e1e6ab0e264865450102d30773579e666514eba27a72a1ca", + "multi_field.logical_mix": "semantic_v0.1:ab36d5d6db29f829c3aa875245fca70d17d25054bd44088fe18d900ae95d8e19", "multi_field.nullable_type": "semantic_v0.1:560844c33bb686eddb98de8d5b608f38d845ee5bf64e6a094f9a8326679aafe9", "multi_field.optional_field": "semantic_v0.1:bc030299c9a955ea2f18a2c746121180b5ed0d1fd6265c95ef243088e392b843", "multi_field.two_scalars": "semantic_v0.1:bc030299c9a955ea2f18a2c746121180b5ed0d1fd6265c95ef243088e392b843", @@ -11,16 +11,16 @@ "single_field.bool.schema_hash": "semantic_v0.1:d0845939265975cb588b4821498b4844a6ae18d9613e6bb9bd8c274db521eab0", "single_field.bytes.arrow_type": "large_binary", "single_field.bytes.schema_hash": "semantic_v0.1:c691ab0e3a0ce67c3f8d7b9065d60e2a14afcdce983b362c62de85c9dcf19a52", - "single_field.dataclass.arrow_type": "extension[struct]", - "single_field.dataclass.schema_hash": "semantic_v0.1:b57fab5e732720af0d9263f2e2885591790c4760d35675acaf084670ef2c4545", - "single_field.dataclass_declared.arrow_type": "extension[struct]", - "single_field.dataclass_declared.schema_hash": "semantic_v0.1:ec33ba184eb852e3b75c7c9e7541b8695f259aec588f541ac6a119d3198b1215", - "single_field.dataclass_nested.arrow_type": "extension[struct, tags: large_list>]", - "single_field.dataclass_nested.schema_hash": "semantic_v0.1:1aee0c1efd5f33da7984bfa837a50b7706ea65a5dc68ad05ba844efffda262d5", - "single_field.dataclass_twin_ab.arrow_type": "extension[struct]", - "single_field.dataclass_twin_ab.schema_hash": "semantic_v0.1:38f6e2b2c6dc3d14de182f2938128a353f056222e184caa202413e3b42edde1e", - "single_field.dataclass_twin_ba.arrow_type": "extension[struct]", - "single_field.dataclass_twin_ba.schema_hash": "semantic_v0.1:38f6e2b2c6dc3d14de182f2938128a353f056222e184caa202413e3b42edde1e", + "single_field.dataclass.arrow_type": "extension[struct]", + "single_field.dataclass.schema_hash": "semantic_v0.1:85e2c0a314e1cf2427c86f632cae5db5f8f04b681160b2b0408edec4c670d081", + "single_field.dataclass_declared.arrow_type": "extension[struct]", + "single_field.dataclass_declared.schema_hash": "semantic_v0.1:7c2cd6ddbdece7ff6754374e3a1a3d414e35c46ebcb0abf1600a06f4e9becec9", + "single_field.dataclass_nested.arrow_type": "extension[struct, tags: large_list>]", + "single_field.dataclass_nested.schema_hash": "semantic_v0.1:60298722254093098ecebefb67f409caf5abb8478e361c34c3e3223a4d2d8fc3", + "single_field.dataclass_twin_ab.arrow_type": "extension[struct]", + "single_field.dataclass_twin_ab.schema_hash": "semantic_v0.1:3258fb49892b288c4f9155d6405bda39c5d9e5d0c366f20bfe718beb2ff314ee", + "single_field.dataclass_twin_ba.arrow_type": "extension[struct]", + "single_field.dataclass_twin_ba.schema_hash": "semantic_v0.1:3258fb49892b288c4f9155d6405bda39c5d9e5d0c366f20bfe718beb2ff314ee", "single_field.dataframe.arrow_type": "extension[large_binary]", "single_field.dataframe.schema_hash": "semantic_v0.1:2544430b3979b1e0ad8cecb9ab45c93b91dda546d5a02a59b5587c73aa04ecd8", "single_field.dict_str_int.arrow_type": "large_list>", @@ -29,8 +29,8 @@ "single_field.float.schema_hash": "semantic_v0.1:1d165501d30136c6e1309c58db930165e0cb51aae8469323283a95297a880191", "single_field.int.arrow_type": "int64", "single_field.int.schema_hash": "semantic_v0.1:2faa1297b96c1248edfafb2422fded3963672852acbe2973155351859eab83d2", - "single_field.list_dataclass.arrow_type": "extension[large_list>]", - "single_field.list_dataclass.schema_hash": "semantic_v0.1:d078161a5775aafe2afde86369e2cc57dcdf00ed010292015a60147b8eee1c47", + "single_field.list_dataclass.arrow_type": "extension[large_list>]", + "single_field.list_dataclass.schema_hash": "semantic_v0.1:29d0eaf9ac1816d31044a9cacc2115d49848602457a8ca84c556b2203638767d", "single_field.list_int.arrow_type": "large_list", "single_field.list_int.schema_hash": "semantic_v0.1:906c18997d1191d374db82ea5cc763d3d04dc3ceccb4f3fde82cf6d53d0f8dc7", "single_field.list_list_int.arrow_type": "large_list>", @@ -43,10 +43,10 @@ "single_field.optional_int.schema_hash": "semantic_v0.1:16d9f93d8da8f8c5a582352f7022b1191ab1e4593722bbd37fdc1238d34cdc13", "single_field.path.arrow_type": "extension[large_string]", "single_field.path.schema_hash": "semantic_v0.1:84e095c4212af644cc68357daa75a3b9094b953c7fb9030fb4af86f8310b30c3", - "single_field.pydantic.arrow_type": "extension[struct]", - "single_field.pydantic.schema_hash": "semantic_v0.1:259566a29cf776cf97fd5c6a0201a8026b3d41644a24c2cb1d3b7544507be4ad", - "single_field.pydantic_declared.arrow_type": "extension[struct]", - "single_field.pydantic_declared.schema_hash": "semantic_v0.1:d05e37ea7d328efbd2833d604e9cff59e1b1ca5f9b09133bb37e9ba524c94896", + "single_field.pydantic.arrow_type": "extension[struct]", + "single_field.pydantic.schema_hash": "semantic_v0.1:05c0f155ab1bc6cffbcc73d225a9b7413cebca8521e5f4f5a821fcdf60b3e07a", + "single_field.pydantic_declared.arrow_type": "extension[struct]", + "single_field.pydantic_declared.schema_hash": "semantic_v0.1:ab6ecbf3038e3d789a57e5032fad7432e8accedde4a5c2c0a52cecab7ecf53bb", "single_field.set_uuid.arrow_type": "extension[large_list]", "single_field.set_uuid.schema_hash": "semantic_v0.1:73501f29f5a260ab092f67ea9d88c8c3d5d8d5c90bba8d9aea88b3b5400c6272", "single_field.str.arrow_type": "large_string", diff --git a/tests/test_hashing/golden/data/semantic.json b/tests/test_hashing/golden/data/semantic.json index ea3156065..8388fde1c 100644 --- a/tests/test_hashing/golden/data/semantic.json +++ b/tests/test_hashing/golden/data/semantic.json @@ -1,24 +1,24 @@ { "type.bool": "semantic_v0.1:cd8f971312b872f0e9a3ef40aa58d6bb537258e72a9a142e0cf712f0a35020e3", "type.bytes": "semantic_v0.1:f65cea8d1f29ca5b96d67caf1ced2f630b509e32574a7b78b9071958818a1c59", - "type.dataclass": "semantic_v0.1:814071362ffe7481bde7e4173907a59d5bc6df940f81c9838764d0f83973abbe", - "type.dataclass_declared": "semantic_v0.1:fee6ecce526f9b6791a66e6bb638a4ba0934e79beff9fe247bf71807224fc786", - "type.dataclass_nested": "semantic_v0.1:214904a85c3d6015ee0b2e83dceba5852d3196dbb295b0f4550198a707c23ad1", - "type.dataclass_twin_ab": "semantic_v0.1:921bd905da2fce576b9a8d0274b9716e5cba49edefd123b955c8ebcffe24835c", - "type.dataclass_twin_ba": "semantic_v0.1:921bd905da2fce576b9a8d0274b9716e5cba49edefd123b955c8ebcffe24835c", + "type.dataclass": "semantic_v0.1:a3986cabd66cbb48db7e99acc9b24ac00b7683707f8b3276e90b6b4f5a08405e", + "type.dataclass_declared": "semantic_v0.1:1358372ae18c5582131bd9db1da741d3fa41630e6d49095d1a1c8c5856b1feae", + "type.dataclass_nested": "semantic_v0.1:83aca3a4d53a66d3fb444f38b90b2296f3b516dd889c756a2550beb21c4d8cb4", + "type.dataclass_twin_ab": "semantic_v0.1:31e10e6217c02aadbec8d61e73caf4b27453757745b1a81ce282ad22b5bda539", + "type.dataclass_twin_ba": "semantic_v0.1:31e10e6217c02aadbec8d61e73caf4b27453757745b1a81ce282ad22b5bda539", "type.dataframe": "semantic_v0.1:51e3c34453121e01086e882959cd12746b3236ff7b259a40a42d43493bf26dff", "type.dict_str_int": "semantic_v0.1:f0d36b085fa65f3368d1d98687ffcb6c80099a418419e45a77d434b37e4393dd", "type.float": "semantic_v0.1:0059c08fc7dfe19dc8d095b7e95d56e8d377bbe49ac30376a00105eca2e7c75f", "type.int": "semantic_v0.1:24be397c58c33686e96b4f85a9f443fd2483564ec7dee1c14b96da3e60ec16ac", - "type.list_dataclass": "semantic_v0.1:9568f2ffe6c7dd3c492522b02c06d59b6f57a6888e0db1d59db41dcac1bcf363", + "type.list_dataclass": "semantic_v0.1:a223a6db21a8c2edd5e052457b483251e104e92fd9d8523ab12850a9cffc9f11", "type.list_int": "semantic_v0.1:5c376e29b0cfebd85277c71c07c9996afaf06fab35f81681b82f705c9feafa3e", "type.list_list_int": "semantic_v0.1:ba62756844df6c82b48a739fadd6bb568a3507896dc5dba192dd3e0285635c48", "type.list_path": "semantic_v0.1:e2029c1032fad111f798363e1f718ecde715573c690c3cc78f33824a246a82ce", "type.ndarray": "semantic_v0.1:b6f37b2016cf3ee3062c3e715bab25fa02b68e92931bce243684300ba1dda5b6", "type.optional_int": "semantic_v0.1:91bc2bd8f4f083d6cbf6157350e693dd52b0bc06c57b7719db8581712098cd73", "type.path": "semantic_v0.1:02a12969616ecd278958ffca715be2865fe4a32c27523059df37245c5048a880", - "type.pydantic": "semantic_v0.1:76a4fd9c0acc945dfba19bb621c91b9eb9f6361df50e6f17a38a3ae7b0b947bc", - "type.pydantic_declared": "semantic_v0.1:a18f664d19b35c92b7a12ac983973d9aedb5e8c86f23a168207f3eee3225cc24", + "type.pydantic": "semantic_v0.1:0a50adf8aebe578185ea609609b24c84aebe09ad403f0d1241e2604af85bdbc8", + "type.pydantic_declared": "semantic_v0.1:7e15bd38029dd3d24767d4a5a53443fe624f39292c0eddaf98477c1a23be5511", "type.set_uuid": "semantic_v0.1:0ac1b6672ab001f6ee64f406e6d1af3a21024d6f78ef0c7414014cdfad96eaf6", "type.str": "semantic_v0.1:309d607de6633b959ae5118171f079b2830f4b1340064fac065652e305b038ad", "type.tuple_int_str": "semantic_v0.1:65ac71aa155fbfa28072adba04ed55d8d99d89e9c87a0d5ed45ccd15fcafc099", diff --git a/tests/test_user_types/test_identity_record.py b/tests/test_user_types/test_identity_record.py index 26f98ea0e..90722d343 100644 --- a/tests/test_user_types/test_identity_record.py +++ b/tests/test_user_types/test_identity_record.py @@ -9,7 +9,6 @@ import dataclasses import datetime import typing -from collections.abc import Iterator from pathlib import Path from typing import Literal, Optional @@ -24,13 +23,6 @@ def converter(): return get_default_context().type_converter -@pytest.fixture(autouse=True) -def _restore_naming() -> Iterator[None]: - before = uci.get_naming() - yield - uci.set_naming(**before) - - def _make_class(module: str, qualname: str = "Point") -> type: cls = dataclasses.make_dataclass(qualname, [("x", int)]) cls.__module__ = module @@ -41,7 +33,9 @@ def _make_class(module: str, qualname: str = "Point") -> type: class TestIdentityString: def test_documented_example(self) -> None: record = uci.identity_record("dataclass", "Point", {"x": "int", "y": "float"}) - assert uci.identity_string(record) == "Point@9117beaaf7ebe3ab" + assert uci.identity_string(record) == ( + "Point@9117beaaf7ebe3ab0bd25de9448c92eb64aa070a58611517bd1e470a544584d8" + ) def test_canonical_json_is_sorted_and_compact(self) -> None: record = uci.identity_record("pydantic", "P", {"b": "int", "a": "str"}) @@ -125,17 +119,12 @@ def test_main_class_defaults_to_name(self, module: str) -> None: def test_nested_main_class_uses_qualname(self) -> None: assert uci.resolve_name(_make_class("__main__", "Outer.Point")) == "Outer.Point" - def test_main_qualified_setting(self) -> None: - uci.set_naming(main="qualified") - assert uci.resolve_name(_make_class("__main__")) == "__main__.Point" - - def test_module_name_setting(self) -> None: - uci.set_naming(module="name") - assert uci.resolve_name(_make_class("pkg.models")) == "Point" + def test_default_name_qualified_false(self) -> None: + assert uci.default_name(_make_class("pkg.models"), qualified=False) == "Point" - def test_invalid_setting_rejected(self) -> None: - with pytest.raises(ValueError, match="main"): - uci.set_naming(main="short") # pyright: ignore[reportArgumentType] + def test_default_name_qualified_true_for_main(self) -> None: + cls = _make_class("__main__") + assert uci.default_name(cls, qualified=True) == "__main__.Point" def test_declared_name_wins(self) -> None: cls = _make_class("pkg.models") diff --git a/tests/test_user_types/test_register_type.py b/tests/test_user_types/test_register_type.py index 204c78ac3..ab9daf1d2 100644 --- a/tests/test_user_types/test_register_type.py +++ b/tests/test_user_types/test_register_type.py @@ -1,4 +1,4 @@ -"""Public API for user class identity: ``op.register_type`` and ``op.set_type_naming`` (ITL-881). +"""Public API for user class identity: ``op.register_type`` (ITL-881). Identity is observed through the public interface: the Arrow extension name of a ``DictSource`` column (``as_table``), the source's ``pipeline_hash`` and values read back @@ -9,7 +9,6 @@ from __future__ import annotations import dataclasses -from collections.abc import Iterator from typing import Any import pydantic @@ -17,9 +16,6 @@ import orcapod as op from orcapod.errors import RecursiveTypeError -from orcapod.logical_types import ( - user_class_identity as uci, -) # get_naming: restore the policy from orcapod.sources import DictSource @@ -113,23 +109,45 @@ def test_declaring_after_use_under_another_name_raises() -> None: op.register_type(UsedFirst, name="itl881.UsedFirst") +@op.register_type(qualified=False) @dataclasses.dataclass -class RenamedByPolicy: +class ShortNamed: x: int -def test_set_type_naming_module_name() -> None: - before = uci.get_naming() - op.set_type_naming(module="name") - try: - assert _name_part(RenamedByPolicy(1)) == "RenamedByPolicy" - finally: - op.set_type_naming(**before) +def test_qualified_false_names_a_module_class_by_its_class_name() -> None: + assert _name_part(ShortNamed(1)) == "ShortNamed" -def test_set_type_naming_rejects_unknown_style() -> None: - with pytest.raises(ValueError, match="'name' or 'qualified'"): - op.set_type_naming(main="short") # pyright: ignore[reportArgumentType] +def test_name_and_qualified_together_raise() -> None: + with pytest.raises(ValueError, match="either name"): + op.register_type(Undeclared, name="itl881.X", qualified=False) + + +@dataclasses.dataclass +class DeclaredTwice: + x: int + + +def test_redeclaring_with_the_same_name_is_a_no_op_and_another_name_raises() -> None: + op.register_type(DeclaredTwice, name="itl881.Twice") + assert op.register_type(DeclaredTwice, name="itl881.Twice") is DeclaredTwice + with pytest.raises(ValueError, match="already registered as 'itl881.Twice'"): + op.register_type(DeclaredTwice, name="itl881.Other") + + +@dataclasses.dataclass +class NotMutated: + x: int + + +def test_registration_does_not_modify_the_class() -> None: + before = dict(vars(NotMutated)) + params = NotMutated.__dataclass_params__ # pyright: ignore[reportAttributeAccessIssue] + op.register_type(NotMutated, name="itl881.NotMutated") + _type_name(NotMutated(1)) + assert dict(vars(NotMutated)) == before + assert NotMutated.__dataclass_params__ is params # pyright: ignore[reportAttributeAccessIssue] # --------------------------------------------------------------------------- @@ -352,10 +370,3 @@ def test_subclass_has_its_own_identity_and_keeps_its_fields() -> None: assert _type_name(Base(1)) != _type_name(Child(1, 2)) _, data = next(iter(_source(Child(1, 2)).iter_data())) assert data.as_dict()["p"] == Child(1, 2) - - -@pytest.fixture(autouse=True) -def _no_policy_leak() -> Iterator[None]: - before = uci.get_naming() - yield - assert uci.get_naming() == before From ac55871691636aae436ef7a23d137bfc4a81e37d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:49:07 -0800 Subject: [PATCH 009/147] docs: your own data types page, identity section, CHANGELOG and spec update (ITL-881) Adds docs/concepts/your-own-data-types.md (runnable samples: cell re-run, field order, changed fields, declared name), an identity section in identity.md, links and nav. The spec records the developer decisions (full hash, no naming switch, no class mutation), the measured golden diff and an appendix on the alternative not pursued. CHANGELOG lists the intended hash changes and what existing stores do under v0.3. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 44 ++++ DESIGN_ISSUES.md | 29 +++ docs/concepts/function-pods.md | 2 + docs/concepts/identity.md | 20 ++ docs/concepts/your-own-data-types.md | 214 ++++++++++++++++++ mkdocs.yml | 1 + ...-11-10-itl-881-user-class-identity-impl.md | 5 + .../2026-11-10-user-class-identity-v0.3.md | 190 ++++++++++------ 8 files changed, 437 insertions(+), 68 deletions(-) create mode 100644 docs/concepts/your-own-data-types.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 3a71b50c0..a3af187f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,18 @@ ### New +#### `op.register_type` — declare the name of your own data types (ITL-881) + +`@op.register_type(name="Point")` declares the name a dataclass or pydantic model is +identified by, wherever the class is defined. `@op.register_type(qualified=False)` names a +module class by its class name alone. Registration never modifies the class; declaring the +same class under a second name raises `ValueError`. See +[Your Own Data Types](docs/concepts/your-own-data-types.md). + +Re-running a notebook cell that defines a dataclass or pydantic model, or reloading its +module, no longer breaks later pods with `ValueError: … already bound`: a class with the +same name and fields is the same type, and its stored results are found again. + #### `orcapod.File` — content-identified file type (ITL-450) `orcapod.File` is a new first-class type for materialized files. It subclasses @@ -115,6 +127,38 @@ replaced by `_`. Correcting stored values is tracked in ITL-853. ### Breaking Changes +#### Dataclasses and pydantic models are identified by name, kind and fields (ITL-881) + +Orcapod 0.3 identifies a dataclass or pydantic model by its name, its kind (dataclass or +pydantic) and its fields' names and types, in any order. The name is the class name for a +class defined in a notebook or script (`__main__`), the module path and class name for a +class defined in a module, or the name declared with `op.register_type`. + +- **Hashes change for every pipeline that uses such a class.** The class's logical type + name and Arrow extension name become `Name@`, and the + stored struct lists fields in name order. Every schema hash, pipeline hash, table path, + system-tag column name and data hash that involves a dataclass or pydantic value moves. + The first run under 0.3 computes those pod and operator results again, into new tables. + Tables written by earlier versions are left untouched and are not read by 0.3 pipelines; + reading such a column directly raises `TypeIdentityError`. There is no migration tool. + Results that involve no dataclass or pydantic value are unaffected. +- **Changing a class's fields makes a new type.** Its pods get new tables; the results of + the old definition stay where they are. Before, a changed class shared the old tables and + a run returned 0 rows with only log warnings. +- **A stored shape that disagrees with the registered class raises.** `DeltaTableDatabase` + re-raises the new `orcapod.errors.TypeIdentityError` instead of logging the read failure + and returning no records. Reading a column whose class cannot be found raises + `ImportError` naming the column, the type and the `op.register_type` call to declare it. +- **A subclass is its own type.** A dataclass or pydantic subclass was stored as its + registered parent, silently dropping its extra fields; it now has its own identity. +- **Recursive classes** raise `RecursiveTypeError` (a `TypeError`) naming the field path, + for example `Node.next -> Node`. +- Pinned golden hash entries changed (approved): `semantic.json` `type.dataclass`, + `type.pydantic`; `schema.json` `multi_field.logical_mix`, + `single_field.{dataclass,pydantic}.{arrow_type,schema_hash}`; `datagram.json` + `{dataclass,pydantic}.{content_hash,pipeline_hash,system_tag_columns}` and the per-row + `data_hash`, `_tag_record_id::…` and `_tag_source_id::…` entries of both cases. + #### Operator outputs keep their inputs' Arrow types, normalized to large variants (ITL-832, ITL-813) `MergeJoin` no longer infers Arrow types from Python values, and it no longer fails on a diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 5b167d8de..a748e7c57 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2208,3 +2208,32 @@ subprocesses with different `PYTHONHASHSEED` values and were verified identical 3.11, 3.12 and 3.13. The generator's header says when regeneration is legitimate: a changed value needs a migration assessment and the developer's approval. Not pinned yet: the `StarfixArrowHasher` gaps tracked in ITL-848. + +--- + +## `src/orcapod/logical_types/` — identity of user classes (dataclasses, pydantic models) + +### UCI1 — A user class was identified by `module.qualname` only, without its fields +**Status:** resolved +**Severity:** high +**Issue:** ITL-881 + +A dataclass or pydantic model was identified by `f"{module}.{qualname}"`. The same class +defined in a notebook (`__main__`) and in a module were different types; two classes with +the same name and different fields shared pipeline hashes and tables, so a changed class +read the old table, failed, and the run returned 0 rows with only log warnings; re-running +the notebook cell that defines a class raised `ValueError: … already bound` until a kernel +restart; a subclass was stored as its parent, dropping its extra fields. + +**Fix:** a user class is identified by `{kind, name, fields}` (spec +`superpowers/specs/2026-11-10-user-class-identity-v0.3.md`). The identity string +`Name@` is the logical type name and Arrow extension +name, with the record in the extension metadata; stored structs list fields in name order. +`op.register_type` declares names without modifying the class; `__main__` classes default +to their class name. The registry rebinds an unchanged identity to a re-defined class and +matches user classes by exact class only. The read path finds the class by kind and name, +checks the identity, and raises `ImportError` or `TypeIdentityError` naming the column; +`DeltaTableDatabase` re-raises `TypeIdentityError`. Resolves the entry PR #288 proposes for +the same issue. Not covered: computed fields (deferred by the developer), local classes, +Enum/NamedTuple/TypedDict. + diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index b8618052c..28928b3d9 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -276,3 +276,5 @@ primary user-facing APIs. - [Operators](operators.md) handle structural transforms (the complement of function pods) - Function pods participate in the [identity chain](identity.md) -- each pod's hash includes the function's identity and its upstream pipeline hashes +- Function pods can take and return dataclasses and pydantic models; see + [Your Own Data Types](your-own-data-types.md) for when two classes count as the same type diff --git a/docs/concepts/identity.md b/docs/concepts/identity.md index 6bf7a2b5a..d845c146d 100644 --- a/docs/concepts/identity.md +++ b/docs/concepts/identity.md @@ -171,6 +171,24 @@ print(step.pod(source).pipeline_hash() == original) When you change what a function computes, bump its major version (or rename it) so that stale results are not reused. +## Your own classes in the identity + +A dataclass or pydantic model enters both chains through its **identity**: its name, its +kind (dataclass or pydantic) and the names and types of its fields, in any order. The +identity is part of every schema hash, so it reaches the pipeline hash of each source, +pod and node that uses the class, and with it the database table paths. Each stored value +is hashed together with it, so it also reaches the content hash. + +Two consequences: + +- Two classes with the same name, kind and fields share results, wherever each is + defined. A class re-defined in a notebook cell with unchanged fields keeps its cache. +- A class whose fields change is a new type: its pods get new pipeline hashes and new + tables, and the results stored with the old definition stay where they are. + +How a class gets its name, and how to keep results when moving a class from a notebook +into a module, is described in [Your Own Data Types](your-own-data-types.md). + ## How it connects to other concepts - [Sources](sources.md) form the base case of the Merkle chain (schema-only for pipeline hash, @@ -179,3 +197,5 @@ stale results are not reused. - [Operators](operators.md) contribute their own identity to the chain - [Function Pods](function-pods.md) run inside a `PipelineJob` use content hashes for memoization and pipeline hashes for DB scoping +- [Your Own Data Types](your-own-data-types.md) explains the identity of dataclasses and + pydantic models diff --git a/docs/concepts/your-own-data-types.md b/docs/concepts/your-own-data-types.md new file mode 100644 index 000000000..a56ca5d8b --- /dev/null +++ b/docs/concepts/your-own-data-types.md @@ -0,0 +1,214 @@ +# Your Own Data Types + +Function pods can take and return your own classes: Python +[dataclasses](https://docs.python.org/3/library/dataclasses.html) and +[pydantic](https://docs.pydantic.dev/) models. Orcapod stores their values in Arrow +columns and caches results computed from them like any other value. + +Caching needs to know when two classes are **the same type**. This page explains how +Orcapod decides, and what that means for notebooks, scripts and modules. + +## What makes two classes the same type + +Orcapod identifies a dataclass or pydantic model by three things: + +| Component | What it is | +|---|---| +| **Name** | The class name for a class defined in a notebook or a script, the module path and class name for a class defined in a module, or a name you declare with `op.register_type` | +| **Kind** | Dataclass or pydantic model. A dataclass and a pydantic model are never the same type, even with the same name and fields | +| **Fields** | The name and type of every stored field, in any order | + +Two classes that agree on all three are the same type: results stored with one are found +again with the other. Anything else about the class does not count: field order, +defaults, methods, properties, pydantic computed fields and validators, docstrings. + +Changing a field's name or type makes a **new type**. Results computed with the old +definition stay in the database, and the new type computes its results into new tables, +so a stored value is never read back with fields it was not written with. + +## Defining and redefining a class in a notebook + +Define a class, use it in a pod and run a job: + +```python +from dataclasses import dataclass + +from orcapod import PipelineJob, function_pod +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@dataclass +class Point: + x: float + y: float + + +calls = [] + + +@function_pod(output_keys="norm") +def norm(p: Point) -> float: + calls.append(p) # only here to show when the function really runs + return (p.x**2 + p.y**2) ** 0.5 + + +db = InMemoryArrowDatabase() + + +def run(point_class): + rows = [ + {"k": "a", "p": point_class(x=3.0, y=4.0)}, + {"k": "b", "p": point_class(x=6.0, y=8.0)}, + ] + job = PipelineJob(name="points", store=db) + with job: + source = DictSource(data=rows, tag_columns=["k"], source_id="pts") + norm.pod(source, label="norm") + job.run() + return job.nodes["norm"] + + +print(run(Point).as_table().to_pandas()) +# k norm +# 0 a 5.0 +# 1 b 10.0 +print("function calls:", len(calls)) +# function calls: 2 +``` + +Running the cell that defines `Point` again creates a new class object with the same +name and fields. It is the same type, so the next run finds the stored results. The +same holds when the fields are listed in another order: + +```python +@dataclass +class Point: + y: float + x: float + + +run(Point) +print("function calls:", len(calls)) +# function calls: 2 +``` + +Adding, removing or retyping a field makes a new type. Re-run the pod cell too, so that `norm` takes +the new `Point`; the run then computes into new tables: + +```python +@dataclass +class Point: + x: float + y: float + label: str = "" + + +@function_pod(output_keys="norm") +def norm(p: Point) -> float: + calls.append(p) + return (p.x**2 + p.y**2) ** 0.5 + + +run(Point) +print("function calls:", len(calls)) +# function calls: 4 +``` + +## How a class is named + +The name is part of the identity, so it decides whether results survive moving code +around: + +- A class defined in a **notebook or a script** run directly (`__main__`) is named by its + class name: `Point`. +- A class defined in a **module** is named by its module path and class name: + `pkg.models.Point`. + +So results computed with `Point` in a notebook are **not** found by +`pkg.models.Point` after you move the class into a module, unless you give the module +class the same name. Declare it with `op.register_type`: + +```python +import orcapod as op + + +@op.register_type(name="Point") +@dataclass +class ModulePoint: + x: float + y: float + label: str = "" + + +run(ModulePoint) +print("function calls:", len(calls)) +# function calls: 4 +``` + +`op.register_type` accepts: + +- `name="..."`: the name to use, the same wherever the class is defined. It must not be + empty, contain `@` or spaces, or start with `orcapod.`. +- `qualified=False`: name a module class by its class name alone, without retyping it. +- no arguments: keep the default name. + +It never changes your class; it records the name in Orcapod's registry and returns the +class. Declare the name before the class is first used in a source or a pod. Declaring +the same class again with the same name does nothing; declaring it under a different +name raises `ValueError`. + +You can see a class's identity on any column that holds it. The part before `@` is the +name; the rest is a hash of the name, the kind and the fields: + +```python +source = DictSource(data=[{"k": "a", "p": ModulePoint(1.0, 2.0)}], tag_columns=["k"]) +print(source.as_table().schema.field("p").type.extension_name) +# Point@... +``` + +The first time a class defined in a notebook or a script is used, Orcapod prints a +one-time warning saying that it is identified by its class name and fields. + +## Classes inside classes + +A field whose type is another dataclass or pydantic model carries that class's own +identity: its name (however that class is named or registered), its kind and its fields. +Renaming the inner class, registering it under another name, or changing its fields +therefore makes the outer class a new type too. The order in which you declare the inner +and outer classes does not matter, as long as both are declared before first use. + +A class that refers back to itself, directly (`next: Node | None`) or through other +classes, cannot be stored: Arrow has no recursive struct type. Orcapod raises +`RecursiveTypeError` naming the path, for example `Node.next -> Node`. + +## Reading stored values back + +Reading stored values needs a class with the stored name, kind and fields. Orcapod +looks for one among the classes registered or used in the current process, then in the +script or notebook (`__main__`), then by importing the name as a module path. + +- If no class with that name exists, reading raises `ImportError` naming the column, the + type and its fields, and showing the `op.register_type` call to declare it. +- If a class with that name exists but its fields differ, reading raises + `TypeIdentityError` (from `orcapod.errors`) naming the column and both sets of fields. + +Neither case is reported as missing data, and nothing stored is changed. + +## Rules and limits + +- Classes defined inside a function cannot be registered: define them at module level or + at the top of the notebook. +- Each subclass is its own type with its own fields; it is never stored as its parent. +- Two different classes registered under the same name with the same fields are the same + type. Orcapod warns once when that happens across two modules, because stored values are + then read back as the most recently registered class. +- Results stored by Orcapod versions before 0.3 identified classes by module and class + name only. Version 0.3 does not read those columns; pipelines compute their results again + into new tables and leave the old ones untouched. + +## How it connects to other concepts + +- [Function Pods](function-pods.md): the pods that take and return these classes. +- [Identity & Hashing](identity.md): how a type's identity enters pipeline and content + hashes. diff --git a/mkdocs.yml b/mkdocs.yml index dd10922a2..7e7eec44e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -38,6 +38,7 @@ nav: - Streams: concepts/streams.md - Operators: concepts/operators.md - Function Pods: concepts/function-pods.md + - Your Own Data Types: concepts/your-own-data-types.md - Identity & Hashing: concepts/identity.md - File Hash Caching: concepts/file-hash-caching.md - API Reference: diff --git a/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md index caf332c3e..ca4ae5671 100644 --- a/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md +++ b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md @@ -19,6 +19,11 @@ path re-raises it. **Tech Stack:** Python 3.11–3.13, pyarrow, polars, pydantic v2, deltalake, IPython (dev, for the notebook scenarios), pytest via `uv run`. +**Changes after the plan was written (developer decisions, November 10, 2026):** +`set_type_naming` was dropped (fixed location rule plus `register_type(qualified=False)`), +the identity string uses the full 64-hex digest, and stored structs list fields in name +order. The spec is authoritative where this plan differs. + --- ## File map diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md index 96252c9ed..140d0f4b8 100644 --- a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -1,6 +1,8 @@ # Identity of a user class in v0.3 (ITL-881) -**Status:** implementation spec, approved by the coordinator November 10, 2026 (P1 decided by the developer; P2 names and the identity-string format provisional). +**Status:** implemented on this branch (PR #292). Approved by the coordinator November 10, +2026; P1, the identity-string format (fold, full-length hash) and the dropped naming switch +decided by the developer the same day. **Issue:** ITL-881 (v0.3, a public release; tag target November 16, 2026). **Builds on:** the findings in PR #288 (`superpowers/specs/2026-11-10-user-class-identity.md`, experiments E01–E08). "W1"–"W15" are the workshop scenarios from §3.4 of that document. @@ -19,7 +21,10 @@ Each decision is marked **CONFIRMED** (the developer decided it, November 10, 20 | D-doc | A page documenting identity | CONFIRMED: developer answer to S4 | | P1 | A field typed by another user class carries that class's identity as that class was registered | CONFIRMED: developer decision, November 10, 2026 | | D-cycle | A self-referential or cyclic class is rejected with a clear `TypeError` | CONFIRMED by the coordinator from this proposal: Arrow cannot store a recursive struct | -| P2 | Public API names (`register_type`, `set_type_naming`) and the identity-string format | PENDING with the developer; implemented provisionally, each a single-point change | +| D-format | Identity string = name + `@` + the full 64-hex SHA-256 of the record (the "fold") | CONFIRMED: developer, November 10, 2026; alternative B (fields only in metadata) not pursued, appendix A | +| D-naming | No process-wide naming switch: fixed location rule, per-class `register_type(name=...)` or `register_type(qualified=False)` | CONFIRMED: developer dropped `set_type_naming`; a project-wide choice would live in the data context, not in v0.3 | +| D-mutation | `register_type` never modifies the class; same name again is a no-op; another name raises | CONFIRMED: developer | +| P2 | Public API name `register_type` and argument `qualified` | `register_type` accepted by the developer; `qualified=False` proposed and implemented provisionally | | D-hash | The 25 golden entries in §e move; nothing else | CONFIRMED by the coordinator: follows from decision 2; the PR lists every moved entry | | D-legacy | Reading a pre-0.3 user-class column directly raises a clear error | CONFIRMED by the coordinator: a consequence of the identity check; nothing extra is built | | D-notice | One-time notice when a `__main__` class is first registered (S3) | CONFIRMED in scope by the coordinator; last item before the cuts | @@ -37,14 +42,17 @@ A user class's identity is an **identity record** with three components: The name is a string chosen once per class, by the first rule that applies: 1. **Declared name.** The user declared one (§b). Used verbatim. -2. **`__main__` class** (`cls.__module__` is `"__main__"` or `"__mp_main__"`, the name a +2. **Per-class choice without a name:** `register_type(qualified=False)` gives + `cls.__qualname__`; `register_type(qualified=True)` gives `module.qualname`. +3. **`__main__` class** (`cls.__module__` is `"__main__"` or `"__mp_main__"`, the name a `multiprocessing` spawn child gives the main module, W14): `cls.__qualname__` - (`"Point"`, or `"Outer.Point"` for a nested class), under the default setting - `main="name"`. Setting `main="qualified"` gives `"__main__.Point"` (today's string). -3. **Module class:** `f"{module}.{qualname}"` (`"pkg.models.Point"`), under the default - setting `module="qualified"`. Setting `module="name"` gives `"Point"`. The module is + (`"Point"`, or `"Outer.Point"` for a nested class). +4. **Module class:** `f"{module}.{qualname}"` (`"pkg.models.Point"`). The module is passed through `canonical_module_name`, which leaves user modules unchanged. +Rules 3 and 4 are fixed (D-naming): no process-wide switch changes them, so a class's +identity never depends on whether or when some setting was called. + A name must be non-empty, must not contain `@` or whitespace, and must not start with `orcapod.` (reserved for built-in types such as `orcapod.path`). A class whose `__qualname__` contains `` stays rejected in v0.3, even with a declared name @@ -66,7 +74,10 @@ pydantic logical types by exact class only. `fields` maps each **stored field's name** to its **type token**. It is a JSON object serialized with sorted keys, so field order never enters. The stored fields are exactly -the ones stored today: dataclass fields with `init=True`, pydantic `model_fields`. +the ones stored today: dataclass fields with `init=True`, pydantic `model_fields`. The +Arrow storage struct also lists the fields in name order, so two classes with the same +identity always have the same storage type (found by the W11 test: with class order the +same identity had two structs, and the Delta read failed into 0 rows). Type token of an annotation (from `typing.get_type_hints`, which strips `Annotated` extras): @@ -77,8 +88,9 @@ extras): | other class without a logical type (stdlib) | `canonical_type_name(cls)` | `datetime.datetime`, `decimal.Decimal` | | type with a built-in logical type | its logical type name | `orcapod.path`, `orcapod.uuid`, `orcapod.file` | | generic container | `origin[arg, …]`, origin by its canonical name, args in order | `list[int]`, `dict[str, float]`, `tuple[int, str]` (`typing.List[int]` gives `list[int]`) | -| `Optional` / union | member tokens sorted, joined by ` \| ` | `int \| None` for `Optional[int]`, `int \| None` and `None \| int` | -| nested dataclass / pydantic model (P1) | that class's identity string (below), as that class is registered | `Inner@3c1d…`, `geo.Inner@…` | +| `Optional` / union | member tokens sorted, `None` last, joined by ` \| ` | `int \| None` for `Optional[int]`, `int \| None` and `None \| int` | +| `Literal` | `Literal[...]` with the members' `repr`s sorted | `Literal['a', 'b']` | +| nested dataclass / pydantic model (P1) | that class's identity string (below), as that class is registered | `Inner@3c1d…` (full digest), `geo.Inner@…` | **Nested classes (P1, CONFIRMED).** The token of a field typed by another user class is that class's own registered identity string: its name component however *that* class was @@ -89,17 +101,18 @@ declaring it under another name, or changing its fields changes the outer identi **Registration order does not matter.** `register_type` only records the declared name and checks it (§b); the logical type, and so the identity, is built at first use, and the -outer class's build registers the inner class first (implicitly, by the setting, if it -was never declared). Declaring every class before first use therefore gives the same +outer class's build registers the inner class first (implicitly, by the location rule, if +it was never declared). Declaring every class before first use therefore gives the same identities in any order: outer decorated before inner, inner declared later in the same cell, inner registered implicitly by the outer's first use. Declaring a class after it was used under a different name raises (§c). An identical notebook re-run of the inner class rebinds it (§c) and leaves the outer identity unchanged. **Cycles (D-cycle).** A class that refers to itself, directly (`next: Node | None`) or -through other classes, is rejected at first use with a `TypeError` naming the cycle -(`Node.next -> Node`). Today's converter already raises `TypeError: Circular type -dependency detected…`; v0.3 keeps the rejection and names the path. Encoding the +through other classes, is rejected at first use with `RecursiveTypeError` (a `TypeError` subclass) naming the +path (`Node.next -> Node`, `CycleA.b -> CycleB.a -> CycleA`). Today's converter already +raises `TypeError: Circular type dependency detected…`; v0.3 keeps the rejection and names +the path. Encoding the back-reference by name would give a well-defined identity, but Arrow cannot store a recursive struct, so there is nothing to identify. @@ -119,15 +132,25 @@ named field *types* and because the Arrow type cannot tell `str` from `orcapod.p ``` identity string = f"{name}@{fingerprint}" -fingerprint = sha256(canonical_json(record)).hexdigest()[:16] +fingerprint = sha256(canonical_json(record)).hexdigest() # full 64 hex digits canonical_json = json.dumps(record, sort_keys=True, separators=(",", ":")) ``` -`Point(x: int, y: float)` as a dataclass gives `Point@9117beaaf7ebe3ab`; with `y: int`, -`Point@c0a75f0c39b86873`; as a pydantic model with `y: float`, `Point@703bff8af8d6002c`. -SHA-256 of a documented JSON string, not the Orcapod semantic hasher, so the identity can -be reproduced by hand and does not depend on a hasher version. 16 hex digits (64 bits) -only have to separate classes with the same name and kind. +`Point(x: int, y: float)` as a dataclass gives +`Point@9117beaaf7ebe3ab0bd25de9448c92eb64aa070a58611517bd1e470a544584d8`; with `y: int`, +`Point@c0a75f0c…`; as a pydantic model with `y: float`, `Point@703bff8a…`. SHA-256 of a +documented JSON string, not the Orcapod semantic hasher, so the identity can be reproduced +by hand and does not depend on a hasher version. The full digest is used (developer): +there is no truncation rule to explain, and the name is not meant for people to type. +The length lives in one constant (`FINGERPRINT_HEX_DIGITS`) and the format in one function +(`identity_string`). + +Where the long name shows: the Arrow field metadata of stored columns, `extension<...>` in +Arrow type reprs, and Polars extension dtypes. It does not appear in table paths (those +use pipeline hashes) or in Orcapod's error messages, which describe a class as +`Point (dataclass) {x: int, y: float}`. A prefix-matching lookup for abbreviated names was +considered and not built: no API takes a logical type name, and the read path compares +full strings from stored metadata. ### Where each component enters @@ -140,7 +163,7 @@ second mechanism. | Logical type name | identity string | `DataclassLogicalType.logical_type_name` / pydantic equivalent | | Arrow extension name (`ARROW:extension:name`) | identity string | same string; one Arrow/Polars extension registration per identity | | Arrow extension metadata | category plus the full record: `{"category":"orcapod.dataclass","identity":{…}}`, canonical JSON | read path uses it to find the class and to print the stored shape | -| Arrow storage type | struct of the stored fields | unchanged (derived from the annotations) | +| Arrow storage type | struct of the stored fields, in name order | derived from the annotations | | Schema hash, source `pipeline_hash`, system-tag column names | identity string | `TypeObjectHandler` hashes a registered class as `type:` (unchanged code) | | Pod signature hash, node `pipeline_hash`, DB table paths | identity string | via the schema/signature hashes | | Per-row data hash, `content_hash` | identity string + record + values | the extension column is hashed with its extension name and metadata (unchanged code) | @@ -156,45 +179,45 @@ extension names, so Arrow's and Polars' global extension registries (keyed by na both without conflict; that is why the fields must be in the extension *name*, not only in its metadata. -## b. Public API — PENDING (P2: names), behaviour proposed +## b. Public API — `register_type` (D-naming, D-mutation) ```python import orcapod as op -@op.register_type # name by the setting: __main__ → "Point", module → "pkg.models.Point" +@op.register_type # default name: __main__ → "Point", module → "pkg.models.Point" @dataclass class Point: ... -@op.register_type(name="Point") # declared name, same everywhere it is defined +@op.register_type(name="Point") # declared name, the same wherever the class is defined @dataclass class Point: ... -op.register_type(Point, name="Point") # call form, for classes you do not own +@op.register_type(qualified=False) # a module class named by its class name: "Point" +@dataclass +class Point: ... -op.set_type_naming(main="name", module="qualified") # process-wide defaults (shown) +op.register_type(Point, name="Point") # call form, for classes you do not own ``` -- `register_type` records the name for the class object (a process-wide weak-keyed table, - so the class is not modified), checks the kind, the name and that the class is not - local, and returns the class unchanged. The logical type is built at first use, as - today, so declarations may come in any order (§a). Classes that are never declared are - registered on first use with the name from the setting. -- Provisional (P2): the public names live in one place (`orcapod/user_types.py`, - re-exported from `orcapod`) and the identity-string format in one function - (`identity_string(record)`), so a rename or a format change is a single-point edit. -- `set_type_naming` changes the default for classes registered after the call. Values: - `"name"` or `"qualified"`. Defaults: `main="name"`, `module="qualified"`. No TOML key - in v0.3 (scope, §g). -- Registering a class object under a second, different name after it is registered (for - example `register_type(Point, name="Geo")` after `Point` was used) raises `ValueError` - naming both names: one class object has one identity per process. +- `register_type` records the name for the class object in a process-wide weak-keyed + table. It never modifies the class (no attributes, no dataclass parameters, no pydantic + config) and returns the same object. It checks the kind, the name, `name` and + `qualified` not both given, and that the class is not local. The logical type is built + at first use, so declarations may come in any order (§a). Classes never declared are + registered on first use under the location rule. +- Registering the same class again with the same name is a no-op. A different name raises + `ValueError` naming both, including a declaration after the class was already used + under its default name: one class object has one name per process, and a registration + is never changed later. +- The public name lives in one place (`orcapod/user_types.py`, re-exported from + `orcapod`). **Name collisions (same kind, same name, two different class objects):** | Case | What happens | What the user sees | |---|---|---| | Same fields, same `module.qualname` (cell re-run, `%autoreload`, W1/W3/W4) | Same identity: the identity is rebound to the new class object (§c) | nothing | -| Same fields, different `module.qualname` (two modules declare `"Point"`, or `module="name"`) | Same identity by definition: they are the same type; values read back are built as the most recently registered class | one `UserWarning` per rebinding naming both classes. Warranted: values come back as a different class object than may be expected | +| Same fields, different `module.qualname` (two modules declare `"Point"`, or use `qualified=False`) | Same identity by definition: they are the same type; values read back are built as the most recently registered class | one `UserWarning` per rebinding naming both classes. Warranted: values come back as a different class object than may be expected | | Different fields (W2 with changed fields, or two modules) | Different identities: two types that happen to share a display name; no clash | nothing at registration. Reading picks the class whose identity matches (§d) | ## c. Rebinding rule (S1) — CONFIRMED (developer answer to S1) @@ -207,18 +230,21 @@ op.set_type_naming(main="name", module="qualified") # process-wide defaults (s - **Different identity under the same name → a different type.** Nothing is rebound; the new class gets its own logical type and extension name (W2: the next run computes into new tables). +- A pod defined before the class cell was re-run still accepts the re-defined class: + schema compatibility treats two user classes with the same identity string as the same + type (`user_class_identity.equivalent_hints`, used by `check_schema_compatibility`). - **Still raises:** 1. One class object registered under two names (§b), including a declaration after the class was already used under its default name. - 5. A self-referential or cyclic class (D-cycle). - 2. An invalid declared name (§a). + 2. An invalid declared name, or `name` together with `qualified` (§a, §b). 3. A local class (``), as today. 4. A shape disagreement on read (§d, S2). + 5. A self-referential or cyclic class (D-cycle). ## d. Read path — CONFIRMED behaviour (S2), lookup mechanism proposed -A stored column carries the extension name `Point@9117beaaf7ebe3ab` and the record in its -metadata. To build values, the reader resolves a class: +A stored column carries the extension name `Point@9117beaa…` (full digest) and the record +in its metadata. To build values, the reader resolves a class: 1. **Registered identity:** the extension name is registered in the reading context → use it. 2. **Declared or registered by name:** classes recorded under `(kind, name)` in this @@ -237,26 +263,28 @@ Outcomes: stored fields, and how to fix it: "define the class and register it with `op.register_type(Point, name='Point')` before reading". - **Candidates found, none with a matching identity (S2):** `TypeIdentityError` - (new, a `ValueError` subclass in `orcapod.errors`) naming the column where the read - path knows it, the class name and kind, the stored fields and the registered fields, - for example "column 'p': stored Point (dataclass) has fields {x: int, y: int}; the - registered Point has {x: int, y: float}". + (new, a `ValueError` subclass in `orcapod.errors`) naming the column, the class name and + kind, the stored fields and the registered fields, for example "Column 'q': Stored + values of Point (dataclass) {x: int, y: int} do not match the registered class: Point + (dataclass) {x: int, y: float}. …". `ExtensionAwareDatabase` reads add the column name + (`register_discovered_logical_types`); the `ImportError` gets it too. - **Same identity, different Arrow storage** (possible only if two data contexts map one annotation to different Arrow types): `TypeIdentityError` naming the column, the class and both struct types. The existing check in the synthesized `__arrow_ext_deserialize__` raises it. - These errors propagate. `DeltaTableDatabase` today catches every read error and logs - "Error reading Delta table"; it re-raises `TypeIdentityError`, so a run can no longer - report success with 0 rows. Other read errors keep their current handling (out of + "Error reading Delta table"; it re-raises `TypeIdentityError` in all five read paths + (`get_all_records`, the record lookups, the schema read), so a run can no longer report + success with 0 rows. Other read errors keep their current handling (out of scope). The misleading "data may have been deleted externally" message is untouched unless the path is reached for this reason (§g lists it as out of scope). -- **Pre-0.3 column** (D-legacy, PENDING): an extension name without `@` and metadata +- **Pre-0.3 column** (D-legacy): an extension name without `@` and metadata without `identity`, read directly (pipelines never reach these tables, §e): raise `TypeIdentityError` saying the column was written by Orcapod before 0.3 under `pkg.models.Point`, that v0.3 identifies the class by name and fields, and that the pipeline recomputes into new tables. Not converted, not dropped. -## e. Intended hash changes — PENDING (developer approval of the list) +## e. Intended hash changes — CONFIRMED list, measured Every pinned value that involves a dataclass or pydantic type moves, because the identity string changes for every user class (fields are now in it, for everyone). Measured on @@ -264,8 +292,13 @@ string changes for every user class (fields are now in it, for everyone). Measur golden fixtures `GoldenPoint` and `GoldenModel` moves exactly these 25 entries, and the `unchanged` control moves 0. Every identity-string change moves the same set (E08's `__main__` and module-less variants list identical entries), so the v0.3 string moves -these and only these. The implementation re-runs the golden test and the PR lists the -measured diff; any entry outside this list is a bug. +these and only these. + +**Measured after implementation** (golden files regenerated, compared with `origin/main` +c5ab1e05): 17 values change and 8 keys are replaced, exactly the entries below; 71 keys +are added (the 8 replacements and the new fixtures); `arrow.json` and `pipeline.json` are +byte-identical to `main`. Before regenerating, the golden test failed on exactly these 25 +entries plus the unpinned additions. | File | Entries | |---|---| @@ -275,9 +308,11 @@ measured diff; any entry outside this list is a bug. | `datagram.json` (8 replaced) | `{dataclass,pydantic}.row[{"k":"r1"/"r2"}]._tag_record_id::` and `._tag_source_id::`: the keys embed the moved pipeline hash, so these 8 keys disappear and 8 keys with the new hash appear | `arrow.json` and `pipeline.json` have no user-class entries and do not move. Golden -additions (new keys only): a declared-name dataclass, a nested dataclass (P1), an -`Optional` field, a container field, and permuted-field-order twins that must pin the -same value. +additions (new keys only): a declared-name dataclass and a pydantic model with the same +name and fields (distinct values), a nested dataclass with a container field (P1), +`list[GoldenPoint]`, an `Optional` field, and field-order twins (`GoldenTwinAB`, +`GoldenTwinBA`, both `golden.Twin`) that pin equal type, schema, data and pipeline hashes +(only their `source_id`-derived entries differ). **For the CHANGELOG (stated plainly):** "Orcapod 0.3 identifies a dataclass or pydantic model by its name, its kind and its fields. Every pipeline whose data includes such a @@ -306,22 +341,24 @@ unaffected. ## g. Scope cuts, in order, if time runs short -1. TOML/config-file setting for the naming defaults (`set_type_naming` stays). -2. The one-time `__main__` notice (S3, D-notice). -3. The collision `UserWarning` for same identity from a different `module.qualname` - (behaviour stays: rebind). -4. The `__main__` lookup on read (§d step 3); a declaration is then required to read - `__main__`-named data in another process. -5. `set_type_naming` itself (defaults fixed: `main="name"`, `module="qualified"`). +None were needed: everything below is implemented. The order that applied: + +1. The one-time `__main__` notice (S3, D-notice). +2. The collision `UserWarning` for same identity from a different `module.qualname`. +3. The `__main__` lookup on read (§d step 3). -Not cut: the identity record and string, rebinding, `register_type`, the S2 raise, the +(`set_type_naming` was dropped by decision, not cut.) Not cut: the identity record and string, rebinding, `register_type`, the S2 raise, the read lookup by declared name, the docs page and the CHANGELOG entry. Out of scope for v0.3: computed fields; local classes with a declared name; Enum, NamedTuple and TypedDict support (D6 of the findings); the `DeltaTableDatabase` row-count and shutdown-hang findings (§9 of the findings). -## Workshop scenarios after v0.3 (predicted, become the Step 2 tests) +## Workshop scenarios after v0.3 (tested: `tests/test_user_types/test_workshop_scenarios.py`) + +On `origin/main` 15 of the 19 scenario tests fail; all pass on this branch. The four that +pass on `main` pin behaviour v0.3 keeps (W5, W7, W12 with the same class, and a module +class without a declared name being a different type from the notebook class). | # | Scenario | v0.3 | |---|---|---| @@ -336,3 +373,20 @@ and shutdown-hang findings (§9 of the findings). | W12 | Read notebook results from a script | works if the script defines a matching `Point` (top level or declared); otherwise the `ImportError` of §d | | W14 | spawn child (`__mp_main__`) | same identity as `__main__` | | W15 | Local class | `ValueError`, as today | + +## Appendix A. Alternative B (fields only in the extension metadata) — not pursued + +The developer asked whether the fields must be folded into the name, or whether the name +could stay `Point` with kind and fields in `ARROW:extension:metadata` and a type hash +composed from name, kind and fields. The developer then confirmed the fold (November 10, +2026) and the investigation stopped. Evidence gathered before it stopped: + +- Arrow's and Polars' extension registries are keyed by name only. Orcapod's registry + swallows `ArrowKeyError` on a second registration under the same name + (`registry.py`), so a second shape under one name would be read through the first + shape's `__arrow_ext_deserialize__`, which raises on the storage mismatch. B would need + a parametric extension type that accepts every shape and dispatches on the metadata; + not built or tested. +- Whether every Polars round trip keeps extension metadata as it keeps the name was not + verified. +- B would change the semantic hasher's type handler; the fold needs no hashing change. From 532a574e3386c9726605d4c4379d386af0b52d5d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:56:57 -0800 Subject: [PATCH 010/147] test(docs): register each doc block's source in linecache (ITL-881) inspect.getsource on a function pod defined in a doc block read the runner's own file (linecache fell back to the __main__ loader), so on Python 3.13 the new Your Own Data Types page failed with a TokenError. Other pages passed only because the wrong lines happened to tokenize. Co-Authored-By: Claude Opus 5.5 --- tests/test_docs/_doc_samples.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/test_docs/_doc_samples.py b/tests/test_docs/_doc_samples.py index 334fcc93e..e9044d6de 100644 --- a/tests/test_docs/_doc_samples.py +++ b/tests/test_docs/_doc_samples.py @@ -27,6 +27,7 @@ import contextlib import io import json +import linecache import re import sys import textwrap @@ -216,8 +217,18 @@ def run_blocks(blocks: list[Block], name: str = "") -> list[dict[str, Any]] results.append(result) continue captured = io.StringIO() + filename = f"{name}:{block.lineno}" + # Register the block's source so ``inspect.getsource`` (used to hash function + # pods) reads the block itself. Without it, ``linecache`` falls back to the + # ``__main__`` module's loader, i.e. this runner's own file (seen on 3.13). + linecache.cache[filename] = ( + len(block.source), + None, + block.source.splitlines(keepends=True), + filename, + ) try: - code = compile(block.source, f"{name}:{block.lineno}", "exec") + code = compile(block.source, filename, "exec") with contextlib.redirect_stdout(captured): exec(code, namespace) # noqa: S102 - running our own documentation except BaseException: # noqa: BLE001 - report SystemExit etc. as failures too From fabed37dd123c98b2e57ce0f85743f278023b98b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 21:58:17 -0800 Subject: [PATCH 011/147] refactor(types): rename register_type's qualified argument to qualified_name (ITL-881) Developer decision, November 10, 2026. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- docs/concepts/your-own-data-types.md | 2 +- .../logical_types/user_class_identity.py | 24 ++++++++++--------- src/orcapod/user_types.py | 14 +++++------ ...-11-10-itl-881-user-class-identity-impl.md | 2 +- .../2026-11-10-user-class-identity-v0.3.md | 16 ++++++------- tests/test_user_types/test_identity_record.py | 10 ++++---- tests/test_user_types/test_register_type.py | 8 +++---- 8 files changed, 41 insertions(+), 37 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a3af187f5..6ea2baed3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ #### `op.register_type` — declare the name of your own data types (ITL-881) `@op.register_type(name="Point")` declares the name a dataclass or pydantic model is -identified by, wherever the class is defined. `@op.register_type(qualified=False)` names a +identified by, wherever the class is defined. `@op.register_type(qualified_name=False)` names a module class by its class name alone. Registration never modifies the class; declaring the same class under a second name raises `ValueError`. See [Your Own Data Types](docs/concepts/your-own-data-types.md). diff --git a/docs/concepts/your-own-data-types.md b/docs/concepts/your-own-data-types.md index a56ca5d8b..a74c8ee3d 100644 --- a/docs/concepts/your-own-data-types.md +++ b/docs/concepts/your-own-data-types.md @@ -150,7 +150,7 @@ print("function calls:", len(calls)) - `name="..."`: the name to use, the same wherever the class is defined. It must not be empty, contain `@` or spaces, or start with `orcapod.`. -- `qualified=False`: name a module class by its class name alone, without retyping it. +- `qualified_name=False`: name a module class by its class name alone, without retyping it. - no arguments: keep the default name. It never changes your class; it records the name in Orcapod's registry and returns the diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index aa6fefcca..52670b5ad 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -118,7 +118,7 @@ def is_main_class(cls: type) -> bool: return cls.__module__ in MAIN_MODULES -def default_name(cls: type, *, qualified: bool | None = None) -> str: +def default_name(cls: type, *, qualified_name: bool | None = None) -> str: """Return the name a class gets without a declared name. Args: @@ -131,9 +131,9 @@ def default_name(cls: type, *, qualified: bool | None = None) -> str: Returns: The name component. """ - if qualified is None: - qualified = not is_main_class(cls) - if not qualified: + if qualified_name is None: + qualified_name = not is_main_class(cls) + if not qualified_name: return cls.__qualname__ module = cls.__module__ if not is_main_class(cls): @@ -158,7 +158,7 @@ def resolve_name(cls: type) -> str: return default_name(cls) -def declare(cls: type, name: str | None, *, qualified: bool | None = None) -> None: +def declare(cls: type, name: str | None, *, qualified_name: bool | None = None) -> None: """Record the name a user class is identified by. Registration never modifies the class: the name is kept in a process-wide table @@ -167,13 +167,13 @@ def declare(cls: type, name: str | None, *, qualified: bool | None = None) -> No Args: cls: A dataclass or pydantic model class. - name: The declared name, or ``None`` for ``default_name(cls, qualified=...)``. - qualified: Only without ``name``: ``False`` for the class name alone, + name: The declared name, or ``None`` for ``default_name(cls, qualified_name=...)``. + qualified_name: Only without ``name``: ``False`` for the class name alone, ``True`` for ``module.qualname``, ``None`` for the location rule. Raises: TypeError: If ``cls`` is not a dataclass or pydantic model class. - ValueError: If both ``name`` and ``qualified`` are given, the class is local, + ValueError: If both ``name`` and ``qualified_name`` are given, the class is local, the name is invalid, or the class is already registered (declared or used) under a different name. """ @@ -183,15 +183,17 @@ def declare(cls: type, name: str | None, *, qualified: bool | None = None) -> No f"Only dataclasses and pydantic models can be registered as types, " f"got {cls!r}." ) - if name is not None and qualified is not None: + if name is not None and qualified_name is not None: raise ValueError( - "Pass either name=... or qualified=..., not both: qualified only chooses " + "Pass either name=... or qualified_name=..., not both: qualified_name only chooses " "between the class name and its module path when no name is given." ) reject_local_class(cls) if name is not None: validate_name(name) - effective = name if name is not None else default_name(cls, qualified=qualified) + effective = ( + name if name is not None else default_name(cls, qualified_name=qualified_name) + ) previous = _built.get(cls) or _declared.get(cls) if previous is not None and previous != effective: raise ValueError( diff --git a/src/orcapod/user_types.py b/src/orcapod/user_types.py index ab8069aa2..1db0d9baa 100644 --- a/src/orcapod/user_types.py +++ b/src/orcapod/user_types.py @@ -24,13 +24,13 @@ @overload def register_type( - cls: T, /, *, name: str | None = None, qualified: bool | None = None + cls: T, /, *, name: str | None = None, qualified_name: bool | None = None ) -> T: ... @overload def register_type( - *, name: str | None = None, qualified: bool | None = None + *, name: str | None = None, qualified_name: bool | None = None ) -> Callable[[T], T]: ... @@ -39,7 +39,7 @@ def register_type( /, *, name: str | None = None, - qualified: bool | None = None, + qualified_name: bool | None = None, ) -> T | Callable[[T], T]: """Declare the name a dataclass or pydantic model is identified by. @@ -49,7 +49,7 @@ def register_type( Without arguments the class keeps its default name: its class name if it is defined in a notebook or script, its module path and class name if it is defined - in a module. ``qualified=False`` names a module class by its class name alone, + in a module. ``qualified_name=False`` names a module class by its class name alone, without retyping it. Declare the name before the class is first used in a source or pod. Declaring the @@ -59,7 +59,7 @@ def register_type( cls: The class, when called directly. name: The declared name, e.g. ``"Point"`` or ``"geo.Point"``. It must not be empty, contain ``@`` or whitespace, or start with ``orcapod.``. - qualified: Only without ``name``. ``False``: the class name (``"Point"``). + qualified_name: Only without ``name``. ``False``: the class name (``"Point"``). ``True``: the module path and class name (``"pkg.models.Point"``). Returns: @@ -67,7 +67,7 @@ def register_type( Raises: TypeError: If the class is not a dataclass or pydantic model. - ValueError: If both ``name`` and ``qualified`` are given, the name is invalid, + ValueError: If both ``name`` and ``qualified_name`` are given, the name is invalid, the class is defined inside a function, or the class is already registered under a different name. @@ -82,7 +82,7 @@ def register_type( """ def decorate(target: T) -> T: - _uci.declare(target, name, qualified=qualified) + _uci.declare(target, name, qualified_name=qualified_name) return target if cls is None: diff --git a/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md index ca4ae5671..54b3aaa6c 100644 --- a/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md +++ b/superpowers/plans/2026-11-10-itl-881-user-class-identity-impl.md @@ -20,7 +20,7 @@ path re-raises it. the notebook scenarios), pytest via `uv run`. **Changes after the plan was written (developer decisions, November 10, 2026):** -`set_type_naming` was dropped (fixed location rule plus `register_type(qualified=False)`), +`set_type_naming` was dropped (fixed location rule plus `register_type(qualified_name=False)`), the identity string uses the full 64-hex digest, and stored structs list fields in name order. The spec is authoritative where this plan differs. diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md index 140d0f4b8..15b053f8d 100644 --- a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -22,9 +22,9 @@ Each decision is marked **CONFIRMED** (the developer decided it, November 10, 20 | P1 | A field typed by another user class carries that class's identity as that class was registered | CONFIRMED: developer decision, November 10, 2026 | | D-cycle | A self-referential or cyclic class is rejected with a clear `TypeError` | CONFIRMED by the coordinator from this proposal: Arrow cannot store a recursive struct | | D-format | Identity string = name + `@` + the full 64-hex SHA-256 of the record (the "fold") | CONFIRMED: developer, November 10, 2026; alternative B (fields only in metadata) not pursued, appendix A | -| D-naming | No process-wide naming switch: fixed location rule, per-class `register_type(name=...)` or `register_type(qualified=False)` | CONFIRMED: developer dropped `set_type_naming`; a project-wide choice would live in the data context, not in v0.3 | +| D-naming | No process-wide naming switch: fixed location rule, per-class `register_type(name=...)` or `register_type(qualified_name=False)` | CONFIRMED: developer dropped `set_type_naming`; a project-wide choice would live in the data context, not in v0.3 | | D-mutation | `register_type` never modifies the class; same name again is a no-op; another name raises | CONFIRMED: developer | -| P2 | Public API name `register_type` and argument `qualified` | `register_type` accepted by the developer; `qualified=False` proposed and implemented provisionally | +| P2 | Public API name `register_type` and argument `qualified_name` | CONFIRMED: developer, November 10, 2026 | | D-hash | The 25 golden entries in §e move; nothing else | CONFIRMED by the coordinator: follows from decision 2; the PR lists every moved entry | | D-legacy | Reading a pre-0.3 user-class column directly raises a clear error | CONFIRMED by the coordinator: a consequence of the identity check; nothing extra is built | | D-notice | One-time notice when a `__main__` class is first registered (S3) | CONFIRMED in scope by the coordinator; last item before the cuts | @@ -42,8 +42,8 @@ A user class's identity is an **identity record** with three components: The name is a string chosen once per class, by the first rule that applies: 1. **Declared name.** The user declared one (§b). Used verbatim. -2. **Per-class choice without a name:** `register_type(qualified=False)` gives - `cls.__qualname__`; `register_type(qualified=True)` gives `module.qualname`. +2. **Per-class choice without a name:** `register_type(qualified_name=False)` gives + `cls.__qualname__`; `register_type(qualified_name=True)` gives `module.qualname`. 3. **`__main__` class** (`cls.__module__` is `"__main__"` or `"__mp_main__"`, the name a `multiprocessing` spawn child gives the main module, W14): `cls.__qualname__` (`"Point"`, or `"Outer.Point"` for a nested class). @@ -192,7 +192,7 @@ class Point: ... @dataclass class Point: ... -@op.register_type(qualified=False) # a module class named by its class name: "Point" +@op.register_type(qualified_name=False) # a module class named by its class name: "Point" @dataclass class Point: ... @@ -202,7 +202,7 @@ op.register_type(Point, name="Point") # call form, for classes you do not own - `register_type` records the name for the class object in a process-wide weak-keyed table. It never modifies the class (no attributes, no dataclass parameters, no pydantic config) and returns the same object. It checks the kind, the name, `name` and - `qualified` not both given, and that the class is not local. The logical type is built + `qualified_name` not both given, and that the class is not local. The logical type is built at first use, so declarations may come in any order (§a). Classes never declared are registered on first use under the location rule. - Registering the same class again with the same name is a no-op. A different name raises @@ -217,7 +217,7 @@ op.register_type(Point, name="Point") # call form, for classes you do not own | Case | What happens | What the user sees | |---|---|---| | Same fields, same `module.qualname` (cell re-run, `%autoreload`, W1/W3/W4) | Same identity: the identity is rebound to the new class object (§c) | nothing | -| Same fields, different `module.qualname` (two modules declare `"Point"`, or use `qualified=False`) | Same identity by definition: they are the same type; values read back are built as the most recently registered class | one `UserWarning` per rebinding naming both classes. Warranted: values come back as a different class object than may be expected | +| Same fields, different `module.qualname` (two modules declare `"Point"`, or use `qualified_name=False`) | Same identity by definition: they are the same type; values read back are built as the most recently registered class | one `UserWarning` per rebinding naming both classes. Warranted: values come back as a different class object than may be expected | | Different fields (W2 with changed fields, or two modules) | Different identities: two types that happen to share a display name; no clash | nothing at registration. Reading picks the class whose identity matches (§d) | ## c. Rebinding rule (S1) — CONFIRMED (developer answer to S1) @@ -236,7 +236,7 @@ op.register_type(Point, name="Point") # call form, for classes you do not own - **Still raises:** 1. One class object registered under two names (§b), including a declaration after the class was already used under its default name. - 2. An invalid declared name, or `name` together with `qualified` (§a, §b). + 2. An invalid declared name, or `name` together with `qualified_name` (§a, §b). 3. A local class (``), as today. 4. A shape disagreement on read (§d, S2). 5. A self-referential or cyclic class (D-cycle). diff --git a/tests/test_user_types/test_identity_record.py b/tests/test_user_types/test_identity_record.py index 90722d343..e9fc1e423 100644 --- a/tests/test_user_types/test_identity_record.py +++ b/tests/test_user_types/test_identity_record.py @@ -119,12 +119,14 @@ def test_main_class_defaults_to_name(self, module: str) -> None: def test_nested_main_class_uses_qualname(self) -> None: assert uci.resolve_name(_make_class("__main__", "Outer.Point")) == "Outer.Point" - def test_default_name_qualified_false(self) -> None: - assert uci.default_name(_make_class("pkg.models"), qualified=False) == "Point" + def test_default_name_qualified_name_false(self) -> None: + assert ( + uci.default_name(_make_class("pkg.models"), qualified_name=False) == "Point" + ) - def test_default_name_qualified_true_for_main(self) -> None: + def test_default_name_qualified_name_true_for_main(self) -> None: cls = _make_class("__main__") - assert uci.default_name(cls, qualified=True) == "__main__.Point" + assert uci.default_name(cls, qualified_name=True) == "__main__.Point" def test_declared_name_wins(self) -> None: cls = _make_class("pkg.models") diff --git a/tests/test_user_types/test_register_type.py b/tests/test_user_types/test_register_type.py index ab9daf1d2..1046ad229 100644 --- a/tests/test_user_types/test_register_type.py +++ b/tests/test_user_types/test_register_type.py @@ -109,19 +109,19 @@ def test_declaring_after_use_under_another_name_raises() -> None: op.register_type(UsedFirst, name="itl881.UsedFirst") -@op.register_type(qualified=False) +@op.register_type(qualified_name=False) @dataclasses.dataclass class ShortNamed: x: int -def test_qualified_false_names_a_module_class_by_its_class_name() -> None: +def test_qualified_name_false_names_a_module_class_by_its_class_name() -> None: assert _name_part(ShortNamed(1)) == "ShortNamed" -def test_name_and_qualified_together_raise() -> None: +def test_name_and_qualified_name_together_raise() -> None: with pytest.raises(ValueError, match="either name"): - op.register_type(Undeclared, name="itl881.X", qualified=False) + op.register_type(Undeclared, name="itl881.X", qualified_name=False) @dataclasses.dataclass From b0ab6b6194a4f02cee43db0fa33dddcd1e744309 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:05:36 -0800 Subject: [PATCH 012/147] fix(types): review fixes for user class identity reads (ITL-881) - Accept large/non-large storage differences at any depth when reading an extension column: Delta compaction rewrites a struct field large_string as string, which made every stored class with a str field unreadable. - Test read candidates without registering them, so a failed read never fixes a class's name; the error says when a class with the stored fields exists under another name. - With qualified_name=True a __main__ class is __main__. in a spawn child too; declared names are NFC-normalized. - Remove the now unused _canonical_storage helper. - Spec: canonical JSON uses ensure_ascii=False; logical type names (e.g. a future orcapod.ndarray) are part of field tokens; metadata bytes are already hashed. Co-Authored-By: Claude Opus 5.5 --- src/orcapod/logical_types/registry.py | 31 ++---- .../logical_types/user_class_identity.py | 34 +++++- .../logical_types/user_class_logical_type.py | 104 ++++++++++++++---- .../2026-11-10-user-class-identity-v0.3.md | 30 ++++- tests/test_logical_types/test_registry.py | 40 +------ tests/test_user_types/test_identity_record.py | 12 +- tests/test_user_types/test_read_path.py | 35 ++++++ .../test_workshop_scenarios.py | 53 +++++++++ 8 files changed, 241 insertions(+), 98 deletions(-) diff --git a/src/orcapod/logical_types/registry.py b/src/orcapod/logical_types/registry.py index e3729f647..9aeb6e4b5 100644 --- a/src/orcapod/logical_types/registry.py +++ b/src/orcapod/logical_types/registry.py @@ -36,27 +36,6 @@ def _sanitize(name: str) -> str: return re.sub(r"[^A-Za-z0-9]", "_", name) -def _canonical_storage(t: pa.DataType) -> pa.DataType: - """Return the canonical large-variant for string/binary families; identity elsewhere. - - Maps: - ``string`` / ``string_view`` → ``large_string`` - ``binary`` / ``binary_view`` → ``large_binary`` - ``large_string`` / ``large_binary`` → unchanged (already canonical) - anything else → unchanged - - Used in ``__arrow_ext_deserialize__`` to accept physically-equivalent but - layout-different string/binary variants that Delta Lake or other compaction - tools may write (e.g. delta-rs ``optimize.compact()`` normalises - ``large_string`` columns to plain ``string`` in the rewritten Parquet files). - """ - if pa.types.is_string(t) or pa.types.is_string_view(t): - return pa.large_string() - if pa.types.is_binary(t) or pa.types.is_binary_view(t): - return pa.large_binary() - return t - - def make_arrow_extension_type( extension_name: str, storage_type: pa.DataType, @@ -123,9 +102,13 @@ def _deserialize( # # Cross-family mismatches (string where large_binary was registered, or # vice-versa) are genuine semantic errors and always raise. - if _canonical_storage(_storage) == _storage: - # _storage is the canonical large form — accept any same-family variant. - storage_ok = _canonical_storage(storage_type) == _storage + from orcapod.utils.arrow_utils import normalize_to_large_types + + if normalize_to_large_types(_storage) == _storage: + # _storage is the canonical large form — accept any same-family variant, + # at any nesting depth (a struct field rewritten from large_string to + # string by Delta compaction, ITL-881). + storage_ok = normalize_to_large_types(storage_type) == _storage else: # Non-canonical _storage — require exact match. storage_ok = storage_type == _storage diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index 52670b5ad..4f96014e2 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -25,6 +25,7 @@ import re import sys import typing +import unicodedata import warnings import weakref from typing import TYPE_CHECKING, Any, Literal @@ -53,8 +54,11 @@ _declared: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() # The name each class object was built under; a class has one name per process. _built: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() -# The identity string each class object was built with. +# The identity string and record each class object was built with. _identity_of: weakref.WeakKeyDictionary[type, str] = weakref.WeakKeyDictionary() +_record_of: weakref.WeakKeyDictionary[type, dict[str, Any]] = ( + weakref.WeakKeyDictionary() +) # Classes seen under each (kind, name), most recent last. _known: dict[tuple[str, str], list[weakref.ref[type]]] = {} # Names that already produced the one-time ``__main__`` notice. @@ -135,11 +139,12 @@ def default_name(cls: type, *, qualified_name: bool | None = None) -> str: qualified_name = not is_main_class(cls) if not qualified_name: return cls.__qualname__ - module = cls.__module__ - if not is_main_class(cls): - from orcapod.utils.canonical_type_names import canonical_module_name + if is_main_class(cls): + # A spawn child names the main module "__mp_main__"; both are "__main__". + return f"__main__.{cls.__qualname__}" + from orcapod.utils.canonical_type_names import canonical_module_name - module = canonical_module_name(cls) or module + module = canonical_module_name(cls) or cls.__module__ return f"{module}.{cls.__qualname__}" @@ -191,6 +196,8 @@ def declare(cls: type, name: str | None, *, qualified_name: bool | None = None) reject_local_class(cls) if name is not None: validate_name(name) + # One spelling per name: the same name typed in two Unicode forms is one name. + name = unicodedata.normalize("NFC", name) effective = ( name if name is not None else default_name(cls, qualified_name=qualified_name) ) @@ -228,7 +235,19 @@ def built_name(cls: type) -> str | None: return _built.get(cls) -def mark_built(cls: type, name: str, identity: str) -> None: +def identity_of(cls: type) -> str | None: + """Return the identity string ``cls`` was built with, or ``None`` if not built.""" + return _identity_of.get(cls) + + +def built_record(cls: type) -> dict[str, Any] | None: + """Return the identity record ``cls`` was built with, or ``None`` if not built.""" + return _record_of.get(cls) + + +def mark_built( + cls: type, name: str, identity: str, record: dict[str, Any] | None = None +) -> None: """Record that ``cls`` was built under ``name`` with ``identity``. A class has one name per process. @@ -237,9 +256,12 @@ def mark_built(cls: type, name: str, identity: str) -> None: cls: The class. name: Its name component. identity: Its identity string. + record: Its identity record. """ _built[cls] = name _identity_of[cls] = identity + if record is not None: + _record_of[cls] = record def equivalent_hints(a: Any, b: Any) -> bool: diff --git a/src/orcapod/logical_types/user_class_logical_type.py b/src/orcapod/logical_types/user_class_logical_type.py index 745755bdf..76b969f31 100644 --- a/src/orcapod/logical_types/user_class_logical_type.py +++ b/src/orcapod/logical_types/user_class_logical_type.py @@ -222,6 +222,39 @@ def create_for_python_type( """ uci.reject_local_class(python_type) name = uci.built_name(python_type) or uci.resolve_name(python_type) + record, storage_type, field_annotations = self._describe( + python_type, name, converter + ) + identity = uci.identity_string(record) + uci.mark_built(python_type, name, identity, record) + uci.remember(self.kind, name, python_type) + uci.notice_main_class(python_type, name) + logger.debug("%s: built %r for %r", type(self).__name__, identity, python_type) + return self.logical_type_class( + identity, python_type, storage_type, field_annotations, record + ) + + def _describe( + self, python_type: type, name: str, converter: TypeConverterProtocol + ) -> tuple[dict[str, Any], pa.StructType, list[tuple[str, Any]]]: + """Return the identity record, storage struct and field annotations of a class. + + Does not register ``python_type`` itself (its field types are registered, as + their Arrow types are needed), so the read path can test a candidate class + without fixing its name. + + Args: + python_type: The class. + name: The name component to use. + converter: The active converter for field-type resolution. + + Returns: + ``(record, storage_type, field_annotations)``. + + Raises: + ValueError: If the type hints cannot be resolved. + RecursiveTypeError: If the class refers back to itself, naming the path. + """ hints = _type_hints(python_type) arrow_fields = [] @@ -247,14 +280,7 @@ def create_for_python_type( tokens[field_name] = uci.type_token(annotation, converter) record = uci.identity_record(self.kind, name, tokens) - identity = uci.identity_string(record) - uci.mark_built(python_type, name, identity) - uci.remember(self.kind, name, python_type) - uci.notice_main_class(python_type, name) - logger.debug("%s: built %r for %r", type(self).__name__, identity, python_type) - return self.logical_type_class( - identity, python_type, pa.struct(arrow_fields), field_annotations, record - ) + return record, pa.struct(arrow_fields), field_annotations # ------------------------------------------------------------------ # Read path @@ -311,31 +337,63 @@ def reconstruct_from_arrow( f"op.register_type({name.rsplit('.', 1)[-1]}, name={name!r})." ) - found: list[UserClassLogicalType] = [] + mismatches: list[str] = [] + renamable = False # a candidate has the stored fields under another name for cls in candidates: - _ = converter.register_python_class(cls) # raises for unsupported classes - lt = converter.get_logical_type_for_python_type(cls) - if not isinstance(lt, UserClassLogicalType): + built = uci.identity_of(cls) + if built is not None: + if built != arrow_extension_name: + built_record = uci.built_record(cls) + mismatches.append(_describe_candidate(cls, built_record)) + continue + lt = converter.get_logical_type_for_python_type(cls) + if isinstance(lt, UserClassLogicalType): + _check_storage(lt, storage_type, record) + return lt + continue + # Not used yet in this process: test it without registering it, so a + # non-matching candidate keeps its freedom to be declared under a name. + try: + candidate_record, _, _ = self._describe( + cls, uci.resolve_name(cls), converter + ) + except (TypeError, ValueError) as exc: + mismatches.append( + f"{cls.__module__}.{cls.__qualname__} (cannot be stored: {exc})" + ) + continue + if uci.identity_string(candidate_record) != arrow_extension_name: + mismatches.append(_describe_candidate(cls, candidate_record)) + renamable = renamable or candidate_record["fields"] == record["fields"] continue - if lt.logical_type_name == arrow_extension_name: + _ = converter.register_python_class(cls) + lt = converter.get_logical_type_for_python_type(cls) + if isinstance(lt, UserClassLogicalType): _check_storage(lt, storage_type, record) return lt - found.append(lt) - registered = "; ".join( - uci.describe_record(lt.identity_record) - if lt.identity_record is not None - else lt.logical_type_name - for lt in found + hint = ( + f" A class with the stored fields exists under another name: register it " + f"as {name!r} before it is first used, e.g. " + f"op.register_type(, name={name!r})." + if renamable + else " A class with the same name but different fields is a different " + "type; restore the stored fields, or use the class with the new fields " + "to compute new results." ) raise TypeIdentityError( - f"Stored values of {uci.describe_record(record)} do not match the " - f"registered class: {registered}. A class with the same name but different " - f"fields is a different type; restore the stored fields, or use the class " - f"with the new fields to compute new results." + f"Stored values of {uci.describe_record(record)} do not match the classes " + f"found: {'; '.join(mismatches)}.{hint}" ) +def _describe_candidate(cls: type, record: dict[str, Any] | None) -> str: + where = f"{cls.__module__}.{cls.__qualname__}" + if record is None: + return where + return f"{where} as {uci.describe_record(record)}" + + def _type_hints(python_type: type) -> dict[str, Any]: try: return typing.get_type_hints(python_type) diff --git a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md index 15b053f8d..663f6b524 100644 --- a/superpowers/specs/2026-11-10-user-class-identity-v0.3.md +++ b/superpowers/specs/2026-11-10-user-class-identity-v0.3.md @@ -54,7 +54,10 @@ Rules 3 and 4 are fixed (D-naming): no process-wide switch changes them, so a cl identity never depends on whether or when some setting was called. A name must be non-empty, must not contain `@` or whitespace, and must not start with -`orcapod.` (reserved for built-in types such as `orcapod.path`). A class whose +`orcapod.` (reserved for built-in types such as `orcapod.path`). A declared name is +NFC-normalized, so one name typed in two Unicode forms is one name (Python already +normalizes class and field identifiers). With `qualified_name=True`, a `__main__` class is +named `__main__.` in a `multiprocessing` spawn child too. A class whose `__qualname__` contains `` stays rejected in v0.3, even with a declared name (scope, §g). @@ -124,6 +127,13 @@ classes (only the resulting fields count). Computed fields are deferred (D-compu later decision to include them adds a key that is absent when empty, so it would move only classes that have computed fields. +A type with a logical type contributes its logical type *name*, so those names are part of +user-class identities: adding or renaming a built-in logical type (for example a new +`orcapod.ndarray` from PR #289, where today an `np.ndarray` field gives the token +`numpy.ndarray`) moves the identity of every class with such a field. Built-in logical +type names are already a stability commitment (`orcapod.path`); after #289 lands this +branch adds a golden fixture with an ndarray field. + The token is built from the Python annotation, not the Arrow type, because the developer named field *types* and because the Arrow type cannot tell `str` from `orcapod.path` (both `large_string`). @@ -133,7 +143,8 @@ named field *types* and because the Arrow type cannot tell `str` from `orcapod.p ``` identity string = f"{name}@{fingerprint}" fingerprint = sha256(canonical_json(record)).hexdigest() # full 64 hex digits -canonical_json = json.dumps(record, sort_keys=True, separators=(",", ":")) +canonical_json = json.dumps(record, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + # hashed as UTF-8 ``` `Point(x: int, y: float)` as a dataclass gives @@ -271,7 +282,16 @@ Outcomes: - **Same identity, different Arrow storage** (possible only if two data contexts map one annotation to different Arrow types): `TypeIdentityError` naming the column, the class and both struct types. The existing check in the synthesized - `__arrow_ext_deserialize__` raises it. + `__arrow_ext_deserialize__` raises it. Storage that differs only in large versus + non-large layouts, at any depth, is accepted: Delta compaction rewrites a struct field + `large_string` as `string` (found in review; on `main` this read failed into 0 rows). +- **Candidates are tested without registering them.** A class found by name that is not + yet used in the process is described without fixing its name, so a failed read never + stops the user from then declaring the class under the stored name. When a candidate + has the stored fields under another name, the error says to declare it. +- **Extension metadata is compared byte for byte**, like the extension name. It is part + of the per-row data hash already (`ARROW:extension:*` keys are hashed), so any change to + it is a hash change and needs the same approval; it is not a separate format risk. - These errors propagate. `DeltaTableDatabase` today catches every read error and logs "Error reading Delta table"; it re-raises `TypeIdentityError` in all five read paths (`get_all_records`, the record lookups, the schema read), so a run can no longer report @@ -332,7 +352,9 @@ unaffected. `builtin_handlers.py` or `v0.1.json` (the identity reaches hashing through the logical type name), adds a new docs page instead of editing `function-pods.md` beyond a link, and edits the golden files only by regeneration. -- **PR #289** (`orcapod.ndarray`) adds `array_format.py` and tests; no file in common. +- **PR #289** (`orcapod.ndarray`) adds `array_format.py` and tests; no file in common, but + a field typed `np.ndarray` contributes its logical type name to the identity (§a), so + after #289 lands the ndarray token is re-checked and pinned with a golden fixture. - **Preferred order:** #286 and #289 merge first. This branch then merges `origin/main`, re-runs E08 against the new main to confirm the moved set, and regenerates the golden files; the PR's golden diff must then show only the §e entries plus the additions. If diff --git a/tests/test_logical_types/test_registry.py b/tests/test_logical_types/test_registry.py index 6876b13c6..8df15212f 100644 --- a/tests/test_logical_types/test_registry.py +++ b/tests/test_logical_types/test_registry.py @@ -20,7 +20,6 @@ LogicalTypeRegistry, make_arrow_extension_type, make_polars_extension_type, - _canonical_storage, ) @@ -678,43 +677,6 @@ def test_registry_does_not_expose_ensure_methods(): assert not hasattr(registry, "ensure_extension_type") -# --------------------------------------------------------------------------- -# _canonical_storage helper -# --------------------------------------------------------------------------- - - -class TestCanonicalStorage: - """Unit tests for the _canonical_storage helper.""" - - def test_string_maps_to_large_string(self): - assert _canonical_storage(pa.string()) == pa.large_string() - - def test_utf8_maps_to_large_string(self): - # pa.utf8() is an alias for pa.string() - assert _canonical_storage(pa.utf8()) == pa.large_string() - - def test_string_view_maps_to_large_string(self): - assert _canonical_storage(pa.string_view()) == pa.large_string() - - def test_large_string_is_identity(self): - assert _canonical_storage(pa.large_string()) == pa.large_string() - - def test_binary_maps_to_large_binary(self): - assert _canonical_storage(pa.binary()) == pa.large_binary() - - def test_binary_view_maps_to_large_binary(self): - assert _canonical_storage(pa.binary_view()) == pa.large_binary() - - def test_large_binary_is_identity(self): - assert _canonical_storage(pa.large_binary()) == pa.large_binary() - - def test_int64_is_identity(self): - assert _canonical_storage(pa.int64()) == pa.int64() - - def test_fixed_binary_is_identity(self): - assert _canonical_storage(pa.binary(16)) == pa.binary(16) - - # --------------------------------------------------------------------------- # __arrow_ext_deserialize__ storage-family tolerance (ITL-602) # --------------------------------------------------------------------------- @@ -801,7 +763,7 @@ def test_non_canonical_storage_accepts_exact_match(self): """Extension registered with plain string (non-canonical) still reads its own data. Regression test for the Copilot review finding: the previous check - _canonical_storage(storage_type) != _storage would map string → large_string and + normalize_to_large_types(storage_type) != _storage would map string → large_string and then compare large_string != string, incorrectly raising for a type that should read back cleanly. """ diff --git a/tests/test_user_types/test_identity_record.py b/tests/test_user_types/test_identity_record.py index e9fc1e423..5aa94891e 100644 --- a/tests/test_user_types/test_identity_record.py +++ b/tests/test_user_types/test_identity_record.py @@ -124,10 +124,18 @@ def test_default_name_qualified_name_false(self) -> None: uci.default_name(_make_class("pkg.models"), qualified_name=False) == "Point" ) - def test_default_name_qualified_name_true_for_main(self) -> None: - cls = _make_class("__main__") + @pytest.mark.parametrize("module", ["__main__", "__mp_main__"]) + def test_default_name_qualified_name_true_for_main(self, module: str) -> None: + cls = _make_class(module) assert uci.default_name(cls, qualified_name=True) == "__main__.Point" + def test_declared_names_are_nfc_normalized(self) -> None: + composed, decomposed = "Caf\u00e9", "Cafe\u0301" + first, second = _make_class("pkg.a"), _make_class("pkg.b") + uci.declare(first, composed) + uci.declare(second, decomposed) + assert uci.resolve_name(first) == uci.resolve_name(second) == composed + def test_declared_name_wins(self) -> None: cls = _make_class("pkg.models") uci.declare(cls, "geo.Point") diff --git a/tests/test_user_types/test_read_path.py b/tests/test_user_types/test_read_path.py index 7c085b1f9..b8f3aa885 100644 --- a/tests/test_user_types/test_read_path.py +++ b/tests/test_user_types/test_read_path.py @@ -50,3 +50,38 @@ def test_storage_mismatch_raises_instead_of_reading_nothing(tmp_path: Path) -> N with pytest.raises(TypeIdentityError, match="expected storage_type"): DeltaTableDatabase(tmp_path).get_all_records(("points",)) + + +@op.register_type(name="itl881.Labelled") +@dataclasses.dataclass +class Labelled: + label: str + n: int + + +def _run_labelled(db_root: Path) -> tuple[int, list[Labelled]]: + from orcapod import PipelineJob, function_pod + + calls: list[Labelled] = [] + + @function_pod(output_keys="q") + def echo(p: Labelled) -> Labelled: + calls.append(p) + return p + + rows: list[Any] = [{"k": f"r{i}", "p": Labelled(f"x{i}", i)} for i in (1, 2)] + job = PipelineJob(name="compact", store=DeltaTableDatabase(db_root)) + with job: + echo.pod(DictSource(data=rows, tag_columns=["k"], source_id="s"), label="echo") + job.run() + values = [data.as_dict()["q"] for _, data in job.nodes["echo"].iter_data()] + return len(calls), sorted(values, key=lambda v: v.n) + + +def test_stored_values_read_after_delta_compaction(tmp_path: Path) -> None: + """Delta compaction rewrites large_string struct fields as string; still readable.""" + expected = [Labelled("x1", 1), Labelled("x2", 2)] + assert _run_labelled(tmp_path) == (2, expected) + for log in tmp_path.rglob("_delta_log"): + deltalake.DeltaTable(str(log.parent)).optimize.compact() + assert _run_labelled(tmp_path) == (0, expected) diff --git a/tests/test_user_types/test_workshop_scenarios.py b/tests/test_user_types/test_workshop_scenarios.py index 98231c023..1937888ee 100644 --- a/tests/test_user_types/test_workshop_scenarios.py +++ b/tests/test_user_types/test_workshop_scenarios.py @@ -363,3 +363,56 @@ def test_read_with_different_fields_raises_naming_both_shapes(workdir: Path) -> assert out["error"].startswith("TypeIdentityError: Column 'q'"), out assert "{x: int, y: int}" in out["message"] assert "{x: int, y: float}" in out["message"] + + +def test_failed_read_leaves_the_class_free_to_be_declared(workdir: Path) -> None: + """A read that rejects a candidate class must not fix that class's name.""" + _store_from_notebook(workdir) + write_files( + workdir, + { + "pkg/__init__.py": "", + "pkg/models.py": """ + from dataclasses import dataclass + + @dataclass + class Point: + x: int + y: int + """, + }, + ) + code = """ +import os +from pathlib import Path +import orcapod as op +from child_harness import error_text, report +from orcapod.contexts import get_default_context +from orcapod.databases import DeltaTableDatabase, ExtensionAwareDatabase +import wsjob +from pkg.models import Point + +root = Path(os.environ["WS_DB"]) +path = tuple(next(p for p in wsjob.table_paths(root) if "/_result/" in p).split("/")) +converter = get_default_context().type_converter + +def read(): + db = ExtensionAwareDatabase(DeltaTableDatabase(root), converter=converter) + table = db.get_all_records(path) + return [repr(r["q"]) for r in converter.arrow_table_to_python_dicts(table.select(["q"]))] + +out = {} +try: + read() +except Exception as exc: + out["first"] = error_text(exc) + out["message"] = str(exc) +op.register_type(Point, name="Point") +out["second"] = sorted(read()) +report(out) +""" + out = _script(workdir, code) + assert out["first"].startswith("TypeIdentityError"), out + assert "pkg.models.Point" in out["message"] + assert "op.register_type(, name='Point')" in out["message"] + assert out["second"] == ["Point(x=1, y=1)", "Point(x=2, y=2)"] From 1aae9d1b85191357d927366b07c53b4124978f0a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:05:47 -0800 Subject: [PATCH 013/147] docs(changelog): Delta compaction fix for stored user-class values (ITL-881) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ea2baed3..da68435cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,6 +95,14 @@ info. Hashes of pipelines that already worked are unchanged. ### Fixed +#### Stored dataclass and pydantic values stay readable after Delta compaction (ITL-881) + +Delta compaction (`optimize.compact()`) rewrites a `large_string` struct field as +`string`. Reading a stored dataclass or pydantic column with a `str` or `bytes` field then +failed, and a cached run returned 0 rows. Extension columns now accept large and non-large +layouts of the same type at any nesting depth, as they already did at the top level +(ITL-602). + #### Materialized rows keep their own source info (ITL-845) When rows were collected and turned back into a stream, every row got the source-info From b4dadf3642cbedd907b73fbd4ffc9a3346f5dd00 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:29:07 -0800 Subject: [PATCH 014/147] test(types): pin IPC read of non-large nested storage as a known limitation (ITL-881, ITL-894) Strict xfail: over Arrow IPC a stored column whose storage differs from the registered type only by layout reads as invalid arrays. Parquet and Delta reads are correct. Fix tracked in ITL-894; noted in DESIGN_ISSUES. Co-Authored-By: Claude Opus 5.5 --- DESIGN_ISSUES.md | 10 +++++++ tests/test_user_types/test_read_path.py | 35 +++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index a748e7c57..242a1d35d 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2237,3 +2237,13 @@ checks the identity, and raises `ImportError` or `TypeIdentityError` naming the the same issue. Not covered: computed fields (deferred by the developer), local classes, Enum/NamedTuple/TypedDict. +**Known limitation (ITL-894):** reading a stored column whose storage differs from the +registered one only by layout (`string` vs `large_string`) is accepted at any depth, so +Delta-compacted tables stay readable. `__arrow_ext_deserialize__` then returns the +registered large storage type, which is correct for Parquet and Delta reads (the decoder +converts the data) but not over Arrow IPC or the C stream interface, where Arrow reads +32-bit offsets as 64-bit and the arrays are invalid. Top-level columns had this since +ITL-602; nested struct fields since this fix. Orcapod's own writers and Polars round trips +never produce non-large storage. Pinned by a strict xfail in +`tests/test_user_types/test_read_path.py`. + diff --git a/tests/test_user_types/test_read_path.py b/tests/test_user_types/test_read_path.py index b8f3aa885..04cb8c138 100644 --- a/tests/test_user_types/test_read_path.py +++ b/tests/test_user_types/test_read_path.py @@ -85,3 +85,38 @@ def test_stored_values_read_after_delta_compaction(tmp_path: Path) -> None: for log in tmp_path.rglob("_delta_log"): deltalake.DeltaTable(str(log.parent)).optimize.compact() assert _run_labelled(tmp_path) == (0, expected) + + +@pytest.mark.xfail( + strict=True, + raises=pa.ArrowInvalid, + reason=( + "ITL-894: _deserialize accepts a non-large storage variant but returns the " + "large storage type, so IPC reinterprets 32-bit offsets as 64-bit. Parquet " + "and Delta reads convert the data and are correct." + ), +) +def test_ipc_read_of_non_large_nested_storage_is_valid() -> None: + import io + + narrow = pa.struct([pa.field("label", pa.string()), pa.field("n", pa.int64())]) + value: Any = Labelled("x1", 1) + ext = ( + DictSource(data=[{"k": "a", "p": value}], tag_columns=["k"]) + .as_table() + .schema.field("p") + .type + ) + metadata: dict[bytes | str, bytes | str] = { + b"ARROW:extension:name": ext.extension_name.encode(), + b"ARROW:extension:metadata": ext.__arrow_ext_serialize__(), + } + table = pa.table( + {"p": pa.array([{"label": "hello", "n": 1}], type=narrow)}, + schema=pa.schema([pa.field("p", narrow).with_metadata(metadata)]), + ) + sink = io.BytesIO() + with pa.ipc.new_stream(sink, table.schema) as writer: + writer.write_table(table) + read = pa.ipc.open_stream(sink.getvalue()).read_all() + read.validate(full=True) From 5707fb7c5a93bdf1d96148753b6e1c18530c5939 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:30:04 -0800 Subject: [PATCH 015/147] docs(types): docstrings for user-class identity helpers (ITL-881) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/logical_types/user_class_identity.py | 3 +++ src/orcapod/logical_types/user_class_logical_type.py | 3 +++ 2 files changed, 6 insertions(+) diff --git a/src/orcapod/logical_types/user_class_identity.py b/src/orcapod/logical_types/user_class_identity.py index 4f96014e2..ee005cda6 100644 --- a/src/orcapod/logical_types/user_class_identity.py +++ b/src/orcapod/logical_types/user_class_identity.py @@ -355,6 +355,7 @@ def candidate_classes(kind: str, name: str) -> list[type]: def _walk_attributes(root: Any, parts: list[str]) -> Any: + """Follow ``parts`` as attributes from ``root``; return ``None`` if one is missing.""" obj = root for part in parts: obj = getattr(obj, part, None) @@ -364,6 +365,7 @@ def _walk_attributes(root: Any, parts: list[str]) -> Any: def _import_dotted(name: str) -> Any: + """Import the object a dotted name refers to, or return ``None`` if there is none.""" from orcapod.logical_types.type_utils import _walk_fqcn try: @@ -471,6 +473,7 @@ def type_token(annotation: Any, converter: TypeConverterProtocol) -> str: def _class_token(cls: Any) -> str: + """Return the type token of a class: its canonical name, builtins written bare.""" name = canonical_type_name(cls) return name.removeprefix("builtins.") diff --git a/src/orcapod/logical_types/user_class_logical_type.py b/src/orcapod/logical_types/user_class_logical_type.py index 76b969f31..6d4954720 100644 --- a/src/orcapod/logical_types/user_class_logical_type.py +++ b/src/orcapod/logical_types/user_class_logical_type.py @@ -388,6 +388,7 @@ def reconstruct_from_arrow( def _describe_candidate(cls: type, record: dict[str, Any] | None) -> str: + """Render a candidate class for a read error: its path and, if known, its identity.""" where = f"{cls.__module__}.{cls.__qualname__}" if record is None: return where @@ -395,6 +396,7 @@ def _describe_candidate(cls: type, record: dict[str, Any] | None) -> str: def _type_hints(python_type: type) -> dict[str, Any]: + """Return the resolved type hints of a class, as ``ValueError`` on failure.""" try: return typing.get_type_hints(python_type) except Exception as exc: @@ -404,6 +406,7 @@ def _type_hints(python_type: type) -> dict[str, Any]: def _check_storage( lt: UserClassLogicalType, storage_type: pa.DataType, record: dict[str, Any] ) -> None: + """Raise ``TypeIdentityError`` if stored and registered storage differ beyond layout.""" from orcapod.utils.arrow_utils import normalize_to_large_types if normalize_to_large_types(lt.storage_type) != normalize_to_large_types( From 39951f01eb2d24d0fbec21abb75864cd49a7f62e Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:33:44 -0800 Subject: [PATCH 016/147] docs(changelog): note the IPC limitation tracked as ITL-894 (ITL-881) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d63f80c35..51b5be066 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,7 +58,9 @@ Delta compaction (`optimize.compact()`) rewrites a `large_string` struct field a `string`. Reading a stored dataclass or pydantic column with a `str` or `bytes` field then failed, and a cached run returned 0 rows. Extension columns now accept large and non-large layouts of the same type at any nesting depth, as they already did at the top level -(ITL-602). +(ITL-602). Known limitation: such a non-large column read over Arrow IPC or the C stream +interface (not Parquet or Delta) gives invalid arrays, at any depth; Orcapod's own writers +never produce one (ITL-894). ## [0.2.0] — 2026-11-10 From b930f6293c2a6499e6856be3dd1ed21cd03131bc Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Tue, 10 Nov 2026 22:43:23 -0800 Subject: [PATCH 017/147] test(types): docstring and value checks for the ITL-894 xfail (ITL-881) CodeRabbit review: an XPASS should mean a correct read, so the test also checks the decoded value, the extension name and its metadata. Co-Authored-By: Claude Opus 5.5 --- tests/test_user_types/test_read_path.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/test_user_types/test_read_path.py b/tests/test_user_types/test_read_path.py index 04cb8c138..a4d3cf91d 100644 --- a/tests/test_user_types/test_read_path.py +++ b/tests/test_user_types/test_read_path.py @@ -97,6 +97,7 @@ def test_stored_values_read_after_delta_compaction(tmp_path: Path) -> None: ), ) def test_ipc_read_of_non_large_nested_storage_is_valid() -> None: + """A user-class column with non-large nested storage reads correctly over IPC.""" import io narrow = pa.struct([pa.field("label", pa.string()), pa.field("n", pa.int64())]) @@ -120,3 +121,7 @@ def test_ipc_read_of_non_large_nested_storage_is_valid() -> None: writer.write_table(table) read = pa.ipc.open_stream(sink.getvalue()).read_all() read.validate(full=True) + read_type = read.schema.field("p").type + assert read_type.extension_name == ext.extension_name + assert read_type.__arrow_ext_serialize__() == ext.__arrow_ext_serialize__() + assert read.column("p").to_pylist() == [{"label": "hello", "n": 1}] From 0886ffdb7aa355d5a470ec1d9ce64474f24020d9 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:31:48 -0800 Subject: [PATCH 018/147] docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-910-adopt-starfix-05.md | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md diff --git a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md new file mode 100644 index 000000000..20c23f90e --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md @@ -0,0 +1,149 @@ +# ITL-910 — Adopt starfix 0.5.0 (hash format 2) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:subagent-driven-development (recommended) or sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Move Orcapod's Arrow hashing onto starfix 0.5.0 (hash format 2, prefix `00 00 02`), bump the +Arrow hash label `arrow_v0.1` → `arrow_v0.2`, and regenerate the golden hashes once, explicitly. + +**Architecture:** `StarfixArrowHasher` (`src/orcapod/hashing/arrow_hashers.py`) keeps calling +`ArrowDigester.hash_table` / `hash_schema` with the default (order-independent) options; only the +dependency, the label and the tests/docs/golden data change, plus whatever the new starfix rejections +require. No `ordered_columns` / `ordered_struct_fields` (developer decision 5). No legacy 0.4 mode. + +**Tech Stack:** starfix 0.5.0 (pure Python, `pyarrow>=19`), pyarrow 23.0.1 (locked; Orcapod floor already +`>=20`, so no pyarrow floor change), pytest, uv. + +Branch: `eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod` (Linear gitBranchName). + +--- + +## Status log (for a successor) + +- 2026-11-11: investigation done (below). Plan sent to coordinator via `ask` — WAIT for the answer + before Task 4 (golden regeneration) and before any migration-related code. + +## Investigation results + +### starfix 0.5.0 API (installed wheel, README, `arrow_digester.py`) + +- Same entry points as 0.4: `ArrowDigester(schema, ...)`, `update`, `finalize`, `hash_schema`, + `hash_record_batch`, `hash_table`, `hash_array`. New keyword-only options `ordered_columns`, + `ordered_struct_fields` (not used here). `include_metadata` unchanged. +- Every hash starts `00 00 02`; all values differ from 0.4. +- `ValueError` (was `NotImplementedError` in 0.4) for data of Null/Map/Interval/Utf8View/BinaryView, + any row count; `ValueError` for ListView/Union/RunEndEncoded; new `ValueError`s: ambiguous node + paths (duplicate names, `"a/b"` aliasing struct `a` child `b`), non-nullable field with a visible null, + duplicate metadata keys with `include_metadata`. `TypeError` for wrong container type (Orcapod + converts RecordBatch → Table first, unaffected). +- FixedSizeList data now hashes (any depth), distinct from `List`/`LargeList` of the same values. +- Canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`) now hash. + +### Orcapod callers + +`grep -rn starfix src`: only `src/orcapod/hashing/arrow_hashers.py` (`hash_schema`, `hash_table`). +Everything else (sources' default `source_id`, record ids, `__input_data_hash`, base/pipeline entry +ids, OperatorNode record hashes, `Data`/`Tag` content hashes, `ArrowTableHandler` for `pa.Table`, +pandas DataFrame/Series handlers) goes through `StarfixArrowHasher`. + +### Behaviour changes visible to an Orcapod user (probe: `ArrowTableSource(t, tag_columns=["k"])`, +`hasher.hash_table`, `hasher.hash_schema`, 0.4 vs 0.5) + +| Input | 0.4 | 0.5 | +|---|---|---| +| `fixed_size_list` column (top level, in struct, in list, with null slot) | `NotImplementedError` (source fails) | hashes; source constructs | +| homogeneous `tuple[T, T]` pod output (maps to `fixed_size_list`) | fails to hash | works | +| `arrow.uuid`, `arrow.fixed_shape_tensor` extension columns | `NotImplementedError` | hashes | +| `hash_schema` with null / view / interval types | `NotImplementedError` | hashes | +| **zero-row** table with a `map` column | hashes | `ValueError` | +| non-nullable field with a visible null | hashes silently | `ValueError: Non-nullable field 'x' contains nulls` | +| duplicate column names (raw hasher, e.g. a `pa.Table` value) | hashes | `ValueError: Ambiguous node path` (sources already failed with `KeyError`) | +| column `"a/b"` next to struct `a{b}` | hashes | `ValueError: Ambiguous node path 'a/b'` | +| `map` (rows), `null`, view, interval data | `NotImplementedError` | `ValueError` (same rejection, new class) | +| `CachedSource._build_merged_stream` | `except NotImplementedError` silently served from cache when live ingestion hit an unhashable type | now raises (the hashing error is no longer mistaken for "source cannot provide data") | + +ITL-848 collisions through the public interface (`ArrowTableSource.iter_data()` row content hashes): +nested list lengths `[[[1]],[[2],[3]]]` vs `[[[1],[2]],[[3]]]` collide on 0.4, differ on 0.5; null struct vs +`{"a": None}` collide on 0.4, differ on 0.5; Bool column hash depends on chunking on 0.4, not on 0.5; +non-nullable `[True]` vs `[True, False, False]` collide on 0.4, differ on 0.5. Slice offsets are canonical +on both. + +### Where the label lives + +- `src/orcapod/hashing/versioned_hashers.py:40` `_CURRENT_ARROW_HASHER_ID` +- `src/orcapod/contexts/data/v0.1.json` `arrow_hasher._config.hasher_id` (+ changelog list) +- `src/orcapod/contexts/data/schemas/context_schema.json:178` (example value) +- tests: `tests/test_hashing/test_starfix_arrow_hasher.py` (label + three pinned format-1 digests), + `tests/test_hashing/test_pandas_handlers.py` (label), `tests/test_core/operators/test_materialize_source_info.py` + (three pinned format-1 digests), `tests/test_core/operators/test_async_sync_equivalence.py`, + `tests/test_hashing/golden/cases.py` (docstring), golden data files. +- docs: `docs/reference/hashing.md` §3/§4 worked-example tables (regenerate with + `superpowers/scripts/hash_audit_example.py`); `notebooks/tutorials/01_introduction_to_orcapod.ipynb` + (outputs). Historical plans/specs under `superpowers/` are left as they are. +- Stored names: no table path embeds an Arrow hash under the default `table_scope="pipeline_hash"`. + With `table_scope="content_hash"`, the path embeds the node content hash, which moves when a source's + content hash moves. + +### Golden estimate (simulated: starfix 0.5 overlay + label patched to `arrow_v0.2`) + +| File | Entries | Changed | Unchanged | Keys added/removed | +|---|---|---|---|---| +| arrow.json | 53 | 53 | 0 | 0 | +| datagram.json | 171 | 40 | 131 | 0 | +| pipeline.json | 305 | 68 | 237 | 0 | +| schema.json | 92 | 22 | 70 | 0 | +| semantic.json | 64 | 1 (`value.dataframe.basic`, pandas via StarfixArrowHasher) | 63 | 0 | + +Every changed entry is an Arrow-derived value (row data hashes, record/entry ids, source content hashes +and default source ids, `pa.Table`/DataFrame hashes). No pipeline hash, schema hash, system-tag column +name or table path moved. + +### Migration finding (needs a decision) + +Default `table_scope` paths do not move, so a v0.3 job pointed at a v0.2 store writes into the same +tables. Observed with `DeltaTableDatabase` and the docs' `bmi_demo` job: v0.3 `run()` recomputes both rows +(old record ids miss), and a later read without `run()` returns **4 rows for 2 inputs** (v0.2 + v0.3 mixed); +before the first v0.3 run, a read returns the v0.2 rows. Stored rows carry the label +(`__input_data_hash`, `__pipeline_base_entry_id` start `b"arrow_v0.1:"` vs `b"arrow_v0.2:"`). OperatorNode +REPLAY reads all rows matching the node content hash, so a v0.3 LOG-mode write next to v0.2 rows can +duplicate too. + +## Tasks (after approval) + +### Task 1: failing tests first +- [ ] New `tests/test_hashing/test_starfix_format2.py` (public interface: `ArrowTableSource`, `iter_data`, + `content_hash`; the hasher itself via `resolve_context(None).arrow_hasher` for table-level checks): + label `arrow_v0.2` and digest prefix `00 00 02`; the four ITL-848 cases above; Bool chunking; slice/offset + and `combine_chunks` canonicality; `fixed_size_list` column hashes and differs from the same values as + `large_list`; `ArrowTableSource` with `fixed_size_list` columns constructs, keeps the type, and runs through + a function pod; pod returning `tuple[float, float]`; rejections with `ValueError` (zero-row map, + non-nullable null, duplicate names via `pa.Table` value hash, `"a/b"` alias); `arrow.uuid` column hashes. +- [ ] Update `test_starfix_arrow_hasher.py` / `test_pandas_handlers.py` label assertions to `arrow_v0.2`. +- [ ] Run, confirm failures on starfix 0.4. WIP commit. + +### Task 2: bump +- [ ] `pyproject.toml`: `starfix>=0.5.0,<0.6` (decision 6 floor); `uv lock`; report what moved in `uv.lock`. +- [ ] Label: `versioned_hashers.py`, `v0.1.json` (+ changelog line: "Hard cut (v0.3): arrow_v0.2 = starfix + hash format 2 (starfix 0.5) …"), `context_schema.json` example. +- [ ] `StarfixArrowHasher` docstring: format 2, prefix `00 00 02`, rejections. +- [ ] Re-pin the format-1 digests in `test_starfix_arrow_hasher.py` and `test_materialize_source_info.py` + (list old/new in the PR). +- [ ] Remove the generator header's "Known gaps (ITL-848)" items that are now hashable (fixed-size list, + `arrow.uuid`); add golden cases for them (adding entries is allowed). + +### Task 3: migration handling — per coordinator decision (see ask) + +### Task 4: regenerate golden (approved regeneration) +- [ ] `uv run python tests/test_hashing/golden/generate_golden_hashes.py`; compare with + `scratchpad/scripts/cmp.py` against main's files; every changed value must be Arrow-derived. + +### Task 5: docs +- [ ] CHANGELOG `[0.3.0] — unreleased` → Breaking changes and migration: starfix format 2 entry. +- [ ] `docs/reference/hashing.md` worked-example tables, prefix note; notebook outputs. +- [ ] DESIGN_ISSUES: one appended entry (resolved, Fix note). + +### Task 6: verification +- [ ] Full suite on 3.11, 3.12, 3.13, one at a time; ruff check/format; basedpyright exit code. +- [ ] Perf: 100k-row table hash, 0.4 vs 0.5; golden-test wall time before/after. + +### Task 7: PR +- [ ] Merge origin/main, push, PR (`Fixes ITL-910`, mentions ITL-848, ITL-927), CodeRabbit rounds. From 1b57f9b8094dadebd6518f8fad0162b7bef9945a Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:36:37 -0800 Subject: [PATCH 019/147] docs(plan): ITL-910 suite findings under starfix 0.5 (WIP) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-910-adopt-starfix-05.md | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md index 20c23f90e..96bb966f7 100644 --- a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md +++ b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md @@ -74,7 +74,8 @@ on both. - `src/orcapod/contexts/data/schemas/context_schema.json:178` (example value) - tests: `tests/test_hashing/test_starfix_arrow_hasher.py` (label + three pinned format-1 digests), `tests/test_hashing/test_pandas_handlers.py` (label), `tests/test_core/operators/test_materialize_source_info.py` - (three pinned format-1 digests), `tests/test_core/operators/test_async_sync_equivalence.py`, + (three pinned format-1 digests), `tests/test_core/operators/test_async_sync_equivalence.py` + (`PINNED_ROW_HASHES`, `PINNED_STREAM_CONTENT_HASH`), `tests/test_hashing/golden/cases.py` (docstring), golden data files. - docs: `docs/reference/hashing.md` §3/§4 worked-example tables (regenerate with `superpowers/scripts/hash_audit_example.py`); `notebooks/tutorials/01_introduction_to_orcapod.ipynb` @@ -107,6 +108,24 @@ before the first v0.3 run, a read returns the v0.2 rows. Stored rows carry the l REPLAY reads all rows matching the node content hash, so a v0.3 LOG-mode write next to v0.2 rows can duplicate too. +### Full suite under starfix 0.5 overlay (label unchanged): 201 failed / 7400 passed / 96 skipped + +- 186 golden entries (all Arrow-derived, see above). +- Pinned format-1 digests: `test_starfix_arrow_hasher.py` (5 tests), `test_materialize_source_info.py` (1), + `test_async_sync_equivalence.py::test_issue_example_types_and_pinned_hashes`, + `test_struct_layout_identity.py::TestStructHashesUnchanged` (4 params). +- `test_group_by.py::test_falls_back_to_record_id_when_key_covers_all_tags`: member order comes from + record ids (uuid5 of the Arrow-hashed default `source_id`), so `["a","b"]` becomes `["b","a"]`. Arrow-derived. +- `test_array_v02_store.py` (2): the ITL-888 claim "an array-free pod reuses its v0.2 results" stops being + true: every function node recomputes (input data hashes move) into the same tables (mixing finding). +- `test_async_sync_equivalence.py::test_arrow_hasher_hash_schema_with_extension_type` (strict xfail, ITL-846) + now XPASSes: starfix 0.5 accepts a live extension type in `hash_schema`, but hashes it as its storage + type, so `hash_schema(Path column) == hash_schema(large_string column)`. +- Same identity loss in `hash_table` for pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`): they are `pa.BaseExtensionType`, not `pa.ExtensionType`, so + `normalize_extension_columns` skips them, `include_metadata` stays off, and an `arrow.uuid` table hashes + equal to a `fixed_size_binary(16)` table. (On 0.4 they raised.) + ## Tasks (after approval) ### Task 1: failing tests first From b9f446548537018b5206f3602500c8ee6765a456 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:39:01 -0800 Subject: [PATCH 020/147] test(hashing): failing tests for starfix hash format 2 under arrow_v0.2 (ITL-910) Co-Authored-By: Claude Opus 5.5 --- .../operators/test_async_sync_equivalence.py | 22 +- tests/test_hashing/test_starfix_format2.py | 245 ++++++++++++++++++ 2 files changed, 256 insertions(+), 11 deletions(-) create mode 100644 tests/test_hashing/test_starfix_format2.py diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 809af1813..916509dd0 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -362,18 +362,18 @@ def test_semijoin_with_nested_columns() -> None: assert_equivalent(sync_rows(SemiJoin(), *sources), async_rows(SemiJoin(), *sources)) -@pytest.mark.xfail( - strict=True, - raises=TypeError, - reason=( - "ITL-846: ArrowHasher.hash_schema passes live extension types to starfix, " - "which cannot hash them (hash_table normalizes them first)." - ), -) def test_arrow_hasher_hash_schema_with_extension_type() -> None: - """``hash_schema`` hashes a schema with a logical (extension) column.""" - schema = pa.schema([pa.field("p", _logical_column([Path("/a")], Path).type)]) - _ = get_default_context().arrow_hasher.hash_schema(schema) # pyright: ignore[reportAttributeAccessIssue] # concrete hasher method, not on the protocol + """``hash_schema`` hashes a logical (extension) column with its identity (ITL-846). + + The extension name and metadata are part of the hash, so the schema does not hash like + one holding the extension's storage type. + """ + ext_type = _logical_column([Path("/a")], Path).type + assert isinstance(ext_type, pa.ExtensionType) + hasher = get_default_context().arrow_hasher + ext_hash = hasher.hash_schema(pa.schema([pa.field("p", ext_type)])) # pyright: ignore[reportAttributeAccessIssue] # concrete hasher method, not on the protocol + storage_hash = hasher.hash_schema(pa.schema([pa.field("p", ext_type.storage_type)])) # pyright: ignore[reportAttributeAccessIssue] # concrete hasher method, not on the protocol + assert ext_hash != storage_hash # ── Data meta columns (ITL-854) ───────────────────────────────────────────── diff --git a/tests/test_hashing/test_starfix_format2.py b/tests/test_hashing/test_starfix_format2.py new file mode 100644 index 000000000..ec21528ae --- /dev/null +++ b/tests/test_hashing/test_starfix_format2.py @@ -0,0 +1,245 @@ +"""Arrow hashing on starfix hash format 2 under the label ``arrow_v0.2`` (ITL-910). + +starfix 0.5 replaced hash format 1 (prefix ``00 00 01``) with the lossless format 2 (prefix +``00 00 02``). These tests pin what that fixes in Orcapod (the collisions of ITL-848), what it +newly accepts (``fixed_size_list`` data, ITL-927; canonical extension types) and what it newly +rejects. +""" + +from __future__ import annotations + +import pyarrow as pa +import pytest + +from orcapod.contexts import get_default_context +from orcapod.core.operators import Join +from orcapod.hashing.versioned_hashers import _CURRENT_ARROW_HASHER_ID +from orcapod.sources import ArrowTableSource +from orcapod.types import ContentHash + +FORMAT_2_PREFIX = b"\x00\x00\x02" + + +def _hash_table(table: pa.Table) -> ContentHash: + """Hash ``table`` with the default context's Arrow hasher.""" + return get_default_context().arrow_hasher.hash_table(table) + + +def _row_hashes(column: pa.Array) -> list[str]: + """Content hashes of the data of a one-column source keyed by a row tag.""" + table = pa.table( + {"k": pa.array([f"r{i}" for i in range(len(column))], pa.large_string()), "x": column} + ) + source = ArrowTableSource(table, tag_columns=["k"]) + return [data.content_hash().to_string() for _, data in source.iter_data()] + + +def _non_nullable(name: str, column: pa.Array) -> pa.Table: + """A one-column table whose field is declared non-nullable.""" + return pa.Table.from_arrays( + [column], schema=pa.schema([pa.field(name, column.type, nullable=False)]) + ) + + +# --------------------------------------------------------------------------- +# Label and format +# --------------------------------------------------------------------------- + + +class TestLabel: + def test_current_arrow_label_is_arrow_v0_2(self): + assert _CURRENT_ARROW_HASHER_ID == "arrow_v0.2" + + def test_default_context_uses_arrow_v0_2(self): + assert get_default_context().arrow_hasher.hasher_id == "arrow_v0.2" + + def test_table_hash_is_format_2(self): + result = _hash_table(pa.table({"x": pa.array([1, 2], pa.int64())})) + assert result.method == "arrow_v0.2" + assert result.digest[:3] == FORMAT_2_PREFIX + + def test_row_data_hash_carries_new_label(self): + hashes = _row_hashes(pa.array([1], pa.int64())) + assert hashes[0].startswith("arrow_v0.2:000002") + + +# --------------------------------------------------------------------------- +# ITL-848 collisions are gone +# --------------------------------------------------------------------------- + + +class TestFormat1CollisionsFixed: + def test_nested_list_lengths_are_part_of_the_row_hash(self): + list3 = pa.large_list(pa.large_list(pa.large_list(pa.int64()))) + first = _row_hashes(pa.array([[[[1]], [[2], [3]]]], list3)) + second = _row_hashes(pa.array([[[[1], [2]], [[3]]]], list3)) + assert first != second + + def test_null_struct_differs_from_struct_of_nulls(self): + struct = pa.struct([("a", pa.int64())]) + null_struct = _row_hashes(pa.array([None], struct)) + struct_of_nulls = _row_hashes(pa.array([{"a": None}], struct)) + assert null_struct != struct_of_nulls + + def test_boolean_hash_does_not_depend_on_chunking(self): + table = pa.table({"x": pa.array([True, False, True])}) + rechunked = pa.Table.from_batches(table.to_batches(max_chunksize=1)) + assert rechunked.column("x").num_chunks == 3 + assert _hash_table(table) == _hash_table(rechunked) + + def test_boolean_row_count_is_part_of_the_hash(self): + one = _non_nullable("x", pa.array([True])) + three = _non_nullable("x", pa.array([True, False, False])) + assert _hash_table(one) != _hash_table(three) + + def test_slice_offset_does_not_change_the_hash(self): + sliced = pa.table({"x": pa.array([1, 2, 3, 4], pa.int64())}).slice(1, 2) + fresh = pa.table({"x": pa.array([2, 3], pa.int64())}) + assert _hash_table(sliced) == _hash_table(fresh) + + def test_chunk_layout_does_not_change_the_hash(self): + table = pa.table({"x": pa.array(["a", None, "ccc", "dd"], pa.large_string())}) + chunked = pa.Table.from_batches(table.to_batches(max_chunksize=3)) + assert chunked.column("x").num_chunks == 2 + assert _hash_table(chunked) == _hash_table(chunked.combine_chunks()) + assert _hash_table(chunked) == _hash_table(table) + + +# --------------------------------------------------------------------------- +# fixed_size_list (ITL-927) +# --------------------------------------------------------------------------- + + +def _fsl_table() -> pa.Table: + return pa.table( + { + "k": pa.array(["a", "b"], pa.large_string()), + "v": pa.array([[1.0, 2.0], [3.0, 4.0]], pa.list_(pa.float32(), 2)), + } + ) + + +class TestFixedSizeList: + def test_fixed_size_list_column_hashes(self): + assert _hash_table(_fsl_table()).digest[:3] == FORMAT_2_PREFIX + + def test_fixed_size_list_differs_from_large_list_of_same_values(self): + fsl = pa.table({"v": pa.array([[1.0, 2.0]], pa.list_(pa.float32(), 2))}) + large = pa.table({"v": pa.array([[1.0, 2.0]], pa.large_list(pa.float32()))}) + assert _hash_table(fsl) != _hash_table(large) + + def test_nested_fixed_size_list_hashes(self): + nested = pa.table( + { + "s": pa.array( + [{"v": [1, 2]}], pa.struct([("v", pa.list_(pa.int64(), 2))]) + ), + "l": pa.array([[[1, 2]]], pa.large_list(pa.list_(pa.int64(), 2))), + } + ) + assert _hash_table(nested).digest[:3] == FORMAT_2_PREFIX + + def test_source_with_fixed_size_list_column_keeps_its_type(self): + source = ArrowTableSource(_fsl_table(), tag_columns=["k"]) + assert source.as_table().schema.field("v").type == pa.list_(pa.float32(), 2) + hashes = [data.content_hash() for _, data in source.iter_data()] + assert len(set(hashes)) == 2 + + def test_fixed_size_list_column_runs_through_a_join(self): + source = ArrowTableSource(_fsl_table(), tag_columns=["k"], infer_nullable=True) + other = ArrowTableSource( + pa.table( + { + "k": pa.array(["a", "b"], pa.large_string()), + "w": pa.array([1, 2], pa.int64()), + } + ), + tag_columns=["k"], + infer_nullable=True, + ) + joined = Join()(source, other) + table = joined.as_table() + assert table.schema.field("v").type == pa.list_(pa.float32(), 2) + assert table.column("v").to_pylist() == [[1.0, 2.0], [3.0, 4.0]] + assert len({data.content_hash() for _, data in joined.iter_data()}) == 2 + + +# --------------------------------------------------------------------------- +# Canonical Arrow extension types keep their identity +# --------------------------------------------------------------------------- + + +class TestCanonicalExtensionTypes: + def test_uuid_column_differs_from_its_storage(self): + storage = pa.array([b"0" * 16, b"1" * 16], pa.binary(16)) + uuid_table = pa.table({"u": pa.ExtensionArray.from_storage(pa.uuid(), storage)}) + storage_table = pa.table({"u": storage}) + assert _hash_table(uuid_table) != _hash_table(storage_table) + + def test_fixed_shape_tensor_differs_from_its_storage(self): + tensor_type = pa.fixed_shape_tensor(pa.float32(), [2]) + storage = pa.array([[1.0, 2.0]], pa.list_(pa.float32(), 2)) + tensor_table = pa.table( + {"t": pa.ExtensionArray.from_storage(tensor_type, storage)} + ) + assert _hash_table(tensor_table) != _hash_table(pa.table({"t": storage})) + + def test_uuid_schema_differs_from_its_storage(self): + hasher = get_default_context().arrow_hasher + uuid_schema = pa.schema([pa.field("u", pa.uuid())]) + storage_schema = pa.schema([pa.field("u", pa.binary(16))]) + assert hasher.hash_schema(uuid_schema) != hasher.hash_schema(storage_schema) # pyright: ignore[reportAttributeAccessIssue] # concrete hasher method, not on the protocol + + +# --------------------------------------------------------------------------- +# Inputs starfix 0.5 rejects +# --------------------------------------------------------------------------- + + +class TestRejections: + def test_zero_row_map_column_is_rejected(self): + table = pa.table( + {"m": pa.array([], pa.map_(pa.large_string(), pa.int64()))} + ) + with pytest.raises(ValueError, match="map"): + _hash_table(table) + + def test_non_nullable_column_holding_a_null_is_rejected(self): + table = pa.Table.from_arrays( + [pa.array(["a", "b"], pa.large_string()), pa.array([1, None], pa.int64())], + schema=pa.schema( + [ + pa.field("k", pa.large_string()), + pa.field("x", pa.int64(), nullable=False), + ] + ), + ) + with pytest.raises(ValueError, match="Non-nullable field 'x'"): + ArrowTableSource(table, tag_columns=["k"]) + + def test_duplicate_column_names_are_rejected(self): + table = pa.Table.from_arrays( + [pa.array([1]), pa.array([2])], names=["x", "x"] + ) + with pytest.raises(ValueError, match="Ambiguous node path 'x'"): + get_default_context().semantic_hasher.hash_object(table) + + def test_slash_name_aliasing_a_struct_child_is_rejected(self): + table = pa.table( + { + "k": pa.array(["a"], pa.large_string()), + "a/b": pa.array([1], pa.int64()), + "a": pa.array([{"b": 2}], pa.struct([("b", pa.int64())])), + } + ) + with pytest.raises(ValueError, match="Ambiguous node path 'a/b'"): + ArrowTableSource(table, tag_columns=["k"]) + + def test_slash_name_aliasing_nothing_is_accepted(self): + table = pa.table( + { + "k": pa.array(["a"], pa.large_string()), + "km/h": pa.array([1], pa.int64()), + } + ) + ArrowTableSource(table, tag_columns=["k"]) From ca5d8cd55ccb20668c8c6fa8c4ee68f897bad6f0 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:40:08 -0800 Subject: [PATCH 021/147] feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0.2 (ITL-910) Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash pyarrow's canonical extension types and extension-typed schemas with their extension identity instead of as their storage type. Co-Authored-By: Claude Opus 5.5 --- pyproject.toml | 2 +- .../contexts/data/schemas/context_schema.json | 2 +- src/orcapod/contexts/data/v0.1.json | 5 ++- src/orcapod/hashing/arrow_hashers.py | 33 ++++++++++++--- src/orcapod/hashing/schema_cleaner.py | 41 +++++++++++++++++++ src/orcapod/hashing/versioned_hashers.py | 6 ++- uv.lock | 8 ++-- 7 files changed, 82 insertions(+), 15 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index efd70d255..8f0e26bbf 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,7 +19,7 @@ dependencies = [ "graphviz>=0.21", "gitpython>=3.1.45", "universal-pathlib>=0.3.8", - "starfix~=0.4.0", + "starfix>=0.5.0,<0.6", "tzdata>=2024.1", "uuid-utils>=0.11.1", "s3fs>=2025.12.0", diff --git a/src/orcapod/contexts/data/schemas/context_schema.json b/src/orcapod/contexts/data/schemas/context_schema.json index 38c377a8f..c5f0c0f38 100644 --- a/src/orcapod/contexts/data/schemas/context_schema.json +++ b/src/orcapod/contexts/data/schemas/context_schema.json @@ -175,7 +175,7 @@ "arrow_hasher": { "_class": "orcapod.hashing.arrow_hashers.StarfixArrowHasher", "_config": { - "hasher_id": "arrow_v0.1", + "hasher_id": "arrow_v0.2", "type_converter": {"_ref": "type_converter"}, "semantic_hasher": {"_ref": "semantic_hasher"} } diff --git a/src/orcapod/contexts/data/v0.1.json b/src/orcapod/contexts/data/v0.1.json index a830d71cd..66017eb30 100644 --- a/src/orcapod/contexts/data/v0.1.json +++ b/src/orcapod/contexts/data/v0.1.json @@ -138,7 +138,7 @@ "arrow_hasher": { "_class": "orcapod.hashing.arrow_hashers.StarfixArrowHasher", "_config": { - "hasher_id": "arrow_v0.1", + "hasher_id": "arrow_v0.2", "type_converter": {"_ref": "type_converter"}, "semantic_hasher": {"_ref": "semantic_hasher"} } @@ -165,7 +165,8 @@ "Moved SpikeInterface types (LogicalSIRecording, LogicalSISorting, LogicalSIMotion, LogicalSISortingAnalyzer) and their handlers to the standalone orcapod-extension-spikeinterface package; removed all SI _optional entries from this spec and SI deps from pyproject.toml; introduced OrcapodExtension protocol and op.register_extension() for normalized extension registration (ITL-473)", "Registered SchemaHandler for orcapod.types.Schema in the context registry; moved handler dispatch before _is_structure check in hash_object so SchemaHandler takes priority over _expand_mapping for Schema objects (ITL-639)", "Registered PolarsExprHandler for polars.Expr: hashes an expression as ContentHash('polars_expr_v1', sha256 of the canonical JSON of an Orcapod-owned expression tree parsed strictly from Expr.meta.serialize(format='json')); expressions outside the supported subset (Python functions, multi-column selectors, aggregations, alias, when/then, non-finite float literals, pl.Series literals) raise ValueError naming the expression (ITL-828)", - "Hard cut (v0.3): numpy.ndarray moves to the value-based array format. LogicalNumpyArray stores extension orcapod.ndarray over struct (Orcapod dtype tag, shape, little-endian C-order packed bytes; extension metadata {\"version\":1}) and NumpyArrayHandler hashes by value with label ndarray_v1 (array_digest_v1 over the stored row). The old extension name numpy.ndarray (.npy bytes, label sha256) is no longer registered or written; v0.2 stores holding arrays are not migrated (ITL-888)" + "Hard cut (v0.3): numpy.ndarray moves to the value-based array format. LogicalNumpyArray stores extension orcapod.ndarray over struct (Orcapod dtype tag, shape, little-endian C-order packed bytes; extension metadata {\"version\":1}) and NumpyArrayHandler hashes by value with label ndarray_v1 (array_digest_v1 over the stored row). The old extension name numpy.ndarray (.npy bytes, label sha256) is no longer registered or written; v0.2 stores holding arrays are not migrated (ITL-888)", + "Hard cut (v0.3): the Arrow hasher label moves from arrow_v0.1 (starfix 0.4, hash format 1, prefix 000001) to arrow_v0.2 (starfix 0.5, hash format 2, prefix 000002). Format 2 is lossless (nested list lengths, null structs, Boolean row counts, chunking-independent) and hashes fixed-size lists; every Arrow-derived hash changes. Order options stay off; pyarrow canonical extension types (arrow.uuid, arrow.fixed_shape_tensor) and hash_schema on extension columns hash with their extension identity (ITL-910)" ] } } diff --git a/src/orcapod/hashing/arrow_hashers.py b/src/orcapod/hashing/arrow_hashers.py index 54c27a528..521fb969b 100644 --- a/src/orcapod/hashing/arrow_hashers.py +++ b/src/orcapod/hashing/arrow_hashers.py @@ -8,6 +8,7 @@ from orcapod.hashing.schema_cleaner import ( clean_schema_for_hashing, has_extension_metadata, + has_extension_type, ) from orcapod.hashing.visitors import SemanticHashingVisitor from orcapod.utils.arrow_utils import normalize_extension_columns @@ -30,8 +31,22 @@ class StarfixArrowHasher: Extension-typed columns without a registered hasher pass through with their full extension metadata intact. 2. **Starfix hashing** — ``ArrowDigester.hash_table`` produces a 35-byte - versioned SHA-256 digest that is byte-for-byte identical to the Rust - ``starfix`` crate output. + digest: the starfix hash format number ``00 00 02`` (hash format 2, + starfix 0.5) followed by a SHA-256 digest, byte-for-byte identical to + the Rust ``starfix`` crate output. The order of columns and of struct + fields is not part of the hash (starfix's order options stay off); row + order and list element order are. Extension identity + (``ARROW:extension:name`` and ``ARROW:extension:metadata``) is hashed + through starfix's ``include_metadata`` whenever the table has an + extension column, including pyarrow's built-in canonical extension types + such as ``arrow.uuid``. + + starfix raises ``ValueError`` for inputs it cannot hash: data of ``null``, + ``map``, interval and view types (whatever the row count); union and + run-end-encoded types; two schema nodes sharing a path (duplicate column + names, or a ``/`` in a name aliasing a struct child); and a non-nullable + field holding a null. Those errors propagate unchanged; they name the + column path and type. Parameters ---------- @@ -127,8 +142,14 @@ def _process_table_columns(self, table: "pa.Table | pa.RecordBatch") -> "pa.Tabl return normalize_extension_columns(intermediate) def hash_schema(self, schema: "pa.Schema") -> ContentHash: - """Hash an Arrow schema using the starfix canonical algorithm.""" - include_meta = has_extension_metadata(schema) + """Hash an Arrow schema using the starfix canonical algorithm. + + Extension-typed columns are normalized to their storage type with the + extension identity in field metadata, as ``hash_table`` does, so the + extension name and metadata are part of the hash. + """ + schema = normalize_extension_columns(schema.empty_table()).schema + include_meta = has_extension_metadata(schema) or has_extension_type(schema) if include_meta: schema = clean_schema_for_hashing(schema) digest = ArrowDigester.hash_schema(schema, include_metadata=include_meta) @@ -140,7 +161,9 @@ def hash_table(self, table: "pa.Table | pa.RecordBatch") -> ContentHash: table = pa.Table.from_batches([table]) processed_table = self._process_table_columns(table) - include_meta = has_extension_metadata(processed_table.schema) + include_meta = has_extension_metadata( + processed_table.schema + ) or has_extension_type(processed_table.schema) if include_meta: clean_schema = clean_schema_for_hashing(processed_table.schema) clean_table = pa.Table.from_arrays( diff --git a/src/orcapod/hashing/schema_cleaner.py b/src/orcapod/hashing/schema_cleaner.py index 09a55768f..2fa55086f 100644 --- a/src/orcapod/hashing/schema_cleaner.py +++ b/src/orcapod/hashing/schema_cleaner.py @@ -162,6 +162,47 @@ def clean_schema_for_hashing(schema: pa.Schema) -> pa.Schema: ) +def _contains_extension_type(arrow_type: pa.DataType) -> bool: + """Return True if ``arrow_type`` is or contains an Arrow extension type.""" + if isinstance(arrow_type, pa.BaseExtensionType): + return True + if pa.types.is_struct(arrow_type): + return any( + _contains_extension_type(arrow_type.field(i).type) + for i in range(arrow_type.num_fields) + ) + if ( + pa.types.is_list(arrow_type) + or pa.types.is_large_list(arrow_type) + or pa.types.is_fixed_size_list(arrow_type) + ): + return _contains_extension_type(arrow_type.value_type) + if pa.types.is_map(arrow_type): + return _contains_extension_type(arrow_type.key_type) or _contains_extension_type( + arrow_type.item_type + ) + if pa.types.is_dictionary(arrow_type): + return _contains_extension_type(arrow_type.value_type) + return False + + +def has_extension_type(schema: pa.Schema) -> bool: + """Return True if any field of ``schema``, at any depth, has an extension type. + + Covers Python-defined ``pa.ExtensionType`` subclasses and pyarrow's built-in canonical + extension types (``arrow.uuid``, ``arrow.fixed_shape_tensor``, ...), which derive from + ``pa.BaseExtensionType`` only. starfix hashes an extension type as its storage type and + adds its ``ARROW:extension:*`` identity only when metadata is included. + + Args: + schema: The ``pa.Schema`` to inspect. + + Returns: + True if an extension type occurs anywhere in ``schema``; False otherwise. + """ + return any(_contains_extension_type(field.type) for field in schema) + + def has_extension_metadata(schema: pa.Schema) -> bool: """Return True if ``schema`` has any ``ARROW:extension:name`` key at any depth. diff --git a/src/orcapod/hashing/versioned_hashers.py b/src/orcapod/hashing/versioned_hashers.py index c75cf7ed5..1aa1c84f4 100644 --- a/src/orcapod/hashing/versioned_hashers.py +++ b/src/orcapod/hashing/versioned_hashers.py @@ -36,8 +36,10 @@ # hashes can be distinguished from newly-computed ones. _CURRENT_SEMANTIC_HASHER_ID = "semantic_v0.1" -# The hasher_id for the Arrow hasher. -_CURRENT_ARROW_HASHER_ID = "arrow_v0.1" +# The hasher_id for the Arrow hasher. Bumped whenever the hash of any table can change, +# whatever the cause; its number is not kept equal to the starfix hash format number. +# arrow_v0.1: starfix 0.4 (hash format 1). arrow_v0.2: starfix 0.5 (hash format 2, ITL-910). +_CURRENT_ARROW_HASHER_ID = "arrow_v0.2" # --------------------------------------------------------------------------- diff --git a/uv.lock b/uv.lock index 7ea17ee90..0ab5dbb3f 100644 --- a/uv.lock +++ b/uv.lock @@ -2388,7 +2388,7 @@ requires-dist = [ { name = "redis", marker = "extra == 'all'", specifier = ">=6.2.0" }, { name = "redis", marker = "extra == 'redis'", specifier = ">=6.2.0" }, { name = "s3fs", specifier = ">=2025.12.0" }, - { name = "starfix", specifier = "~=0.4.0" }, + { name = "starfix", specifier = ">=0.5.0,<0.6" }, { name = "typer", specifier = ">=0.12" }, { name = "tzdata", specifier = ">=2024.1" }, { name = "universal-pathlib", specifier = ">=0.3.8" }, @@ -3932,14 +3932,14 @@ wheels = [ [[package]] name = "starfix" -version = "0.4.0" +version = "0.5.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pyarrow" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/fc/c1/f3269cca708e13d47bf86c5459f9575e436e99d88cd29fec9e5cfd1d6003/starfix-0.4.0.tar.gz", hash = "sha256:e63711756640d4618d34c58f9d80ed7de9722f50ff806ec0ceecdcf151f26e48", size = 71807, upload-time = "2026-06-29T00:14:18.687Z" } +sdist = { url = "https://files.pythonhosted.org/packages/cb/15/f431d84042cab83680b3e857455c133bc0493c629b93fc23f28cd85b07ee/starfix-0.5.0.tar.gz", hash = "sha256:677f001fae5f0173c52ffaaf761fb96becf856ff679ab5d201208f453396a9fb", size = 225505, upload-time = "2026-11-11T23:21:04.713Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/1f/a2/b08c4c9e3e95e6e4224ad1fe857a2eaef1e473d9db6063d8151f5e8ac21f/starfix-0.4.0-py3-none-any.whl", hash = "sha256:3ec6c423eb131bfde1471a03219eea46389603753c30d90bdf3bfbe2bf8863c3", size = 10809, upload-time = "2026-06-29T00:14:17.748Z" }, + { url = "https://files.pythonhosted.org/packages/4e/f6/8b9042044b9bf520c5bac38d0630c411e66ccc47635e8c0293bd9a7d5038/starfix-0.5.0-py3-none-any.whl", hash = "sha256:6982fa4598f99e20428b76041378e3203f2894e6542f6e3ba75746bdbfea0ad2", size = 28736, upload-time = "2026-11-11T23:21:03.711Z" }, ] [[package]] From bd252f28ab2d0e688a017bc8f838779cff390904 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:44:46 -0800 Subject: [PATCH 022/147] test(golden): regenerate golden hashes for starfix hash format 2 / arrow_v0.2 (ITL-910) Explicitly approved regeneration (developer decision 3, November 11, 2026): 184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline 68/305, schema 22/92, semantic 1/64); no key added or removed among them. 21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a chunked Boolean column, canonical extension types and extension schemas. Co-Authored-By: Claude Opus 5.5 --- tests/test_hashing/golden/cases.py | 52 ++++++- tests/test_hashing/golden/data/arrow.json | 127 +++++++++------- tests/test_hashing/golden/data/datagram.json | 80 +++++----- tests/test_hashing/golden/data/pipeline.json | 140 +++++++++--------- tests/test_hashing/golden/data/schema.json | 44 +++--- tests/test_hashing/golden/data/semantic.json | 2 +- .../golden/generate_golden_hashes.py | 7 +- 7 files changed, 261 insertions(+), 191 deletions(-) diff --git a/tests/test_hashing/golden/cases.py b/tests/test_hashing/golden/cases.py index ada9870a7..2813051e7 100644 --- a/tests/test_hashing/golden/cases.py +++ b/tests/test_hashing/golden/cases.py @@ -497,6 +497,30 @@ def _arrow_tables() -> dict[str, pa.Table]: } ) tables["empty.zero_rows"] = pa.table({"a": pa.array([], pa.int64())}) + # Added with starfix hash format 2 (ITL-910): inputs format 1 could not hash or + # hashed alike (ITL-848, ITL-927). + tables["fixed_size_list.float32_2"] = pa.table( + {"a": pa.array([[0.5, 1.5], [2.5, 3.5]], pa.list_(pa.float32(), 2))} + ) + tables["fixed_size_list.null_slot"] = pa.table( + {"a": pa.array([[1, 2], None], pa.list_(pa.int64(), 2))} + ) + list3 = pa.large_list(pa.large_list(pa.large_list(pa.int64()))) + tables["collision.list_lengths_a"] = pa.table( + {"a": pa.array([[[[1]], [[2], [3]]]], list3)} + ) + tables["collision.list_lengths_b"] = pa.table( + {"a": pa.array([[[[1], [2]], [[3]]]], list3)} + ) + tables["collision.null_struct"] = pa.table( + {"a": pa.array([None], pa.struct([("x", pa.int64())]))} + ) + tables["collision.struct_of_nulls"] = pa.table( + {"a": pa.array([{"x": None}], pa.struct([("x", pa.int64())]))} + ) + tables["bool.chunked"] = pa.Table.from_batches( + tables["bool"].to_batches(max_chunksize=1) + ) return tables @@ -525,6 +549,22 @@ def _extension_tables() -> dict[str, pa.Table]: tables["extension.unregistered_passthrough"] = pa.table( {"c": pa.ExtensionArray.from_storage(GoldenOpaqueType(), storage)} ) + # pyarrow's canonical extension types, hashable with their identity since ITL-910. + tables["extension.canonical_uuid"] = pa.table( + { + "c": pa.ExtensionArray.from_storage( + pa.uuid(), pa.array([UUID_A.bytes, UUID_B.bytes], pa.binary(16)) + ) + } + ) + tables["extension.canonical_fixed_shape_tensor"] = pa.table( + { + "c": pa.ExtensionArray.from_storage( + pa.fixed_shape_tensor(pa.float32(), [2]), + pa.array([[0.5, 1.5], [2.5, 3.5]], pa.list_(pa.float32(), 2)), + ) + } + ) return tables @@ -542,14 +582,22 @@ def arrow_extension_table_hashes() -> dict[str, Any]: @case("arrow", "schema") def arrow_schema_hashes() -> dict[str, Any]: - # hash_schema is not reachable for extension-typed schemas yet (ITL-848), so only - # extension-free schemas are pinned here. hasher = _ctx().arrow_hasher return { k: hasher.hash_schema(t.schema).to_string() for k, t in _arrow_tables().items() } +@case("arrow", "extension_schema") +def arrow_extension_schema_hashes() -> dict[str, Any]: + # hash_schema hashes extension columns with their identity since ITL-910 (ITL-846). + hasher = _ctx().arrow_hasher + return { + k: hasher.hash_schema(t.schema).to_string() + for k, t in _extension_tables().items() + } + + # --------------------------------------------------------------------------- # Group: datagram — public sources carrying each logical type # --------------------------------------------------------------------------- diff --git a/tests/test_hashing/golden/data/arrow.json b/tests/test_hashing/golden/data/arrow.json index ff71bb164..5c636b5ec 100644 --- a/tests/test_hashing/golden/data/arrow.json +++ b/tests/test_hashing/golden/data/arrow.json @@ -1,55 +1,76 @@ { - "extension_table.extension.path": "arrow_v0.1:000001e32d9b28a94c4f47e33a647ed137a7316d4cf80d5bd216cc74e2ea4e46bc8d48", - "extension_table.extension.unregistered_passthrough": "arrow_v0.1:00000197e4832ff6efd7436451ab12fbbcde498ae4871ab74b13c9b80c32a4ba9ee44a", - "extension_table.extension.uuid": "arrow_v0.1:0000016efd6e3b9a4c03e2c78bc410a5e04bd5e4bfd7800e3a4054b67805c22fd6436f", - "schema.bool": "arrow_v0.1:000001e1a1ab1078fd92ded5be471806dd8fc4f7d4654dfdb7d9644942269de543e0ee", - "schema.empty.zero_rows": "arrow_v0.1:000001f31bd23e3b2489f68d11a6d3087d09fb8df563c2fb84b3084a6ddf042fe6546a", - "schema.float.float32": "arrow_v0.1:000001fcd0edfcd85a395e7a0d04b384f3cba908f87c571f7fffb6b1ac5a86958d058f", - "schema.float.float64": "arrow_v0.1:000001f68c38ed991e7d525d27dda0d5c79ad04c0a7d6ffbc61099167ac6377b475852", - "schema.layout.binary": "arrow_v0.1:0000011b8b0793ca3db4779c18d400924a60b3871633eedc7c247a97c706ad47180b9c", - "schema.layout.large_binary": "arrow_v0.1:0000011b8b0793ca3db4779c18d400924a60b3871633eedc7c247a97c706ad47180b9c", - "schema.layout.large_list": "arrow_v0.1:000001cec41debef42a67a59f89ebf943757a06619b738e0a945db6134bce4f846c8a1", - "schema.layout.large_string": "arrow_v0.1:00000191f7f5c26abcceb89a6fb399fd0dc333c743ad38a5f2a4da8b274b7e96acfe18", - "schema.layout.list": "arrow_v0.1:000001cec41debef42a67a59f89ebf943757a06619b738e0a945db6134bce4f846c8a1", - "schema.layout.string": "arrow_v0.1:00000191f7f5c26abcceb89a6fb399fd0dc333c743ad38a5f2a4da8b274b7e96acfe18", - "schema.nested.list_list": "arrow_v0.1:0000014387f6de75a16b23c9a7e9d90788dab5f2cf9bf4e891a71d5b14dd193de90424", - "schema.nested.list_struct": "arrow_v0.1:000001a257fe950b239e9d3a4da720c75744f10a405c7ad6824b744e0644a461c4da5a", - "schema.nested.struct": "arrow_v0.1:000001402dbf363e9e7ebaf5df601c1684ebab94eea5090394c38ab3c0076113f02d35", - "schema.nullability.non_nullable": "arrow_v0.1:0000013487bfb8486c6973de2ead68dde548f34da47db81157410996575a8f36791e85", - "schema.nullability.nullable": "arrow_v0.1:000001f31bd23e3b2489f68d11a6d3087d09fb8df563c2fb84b3084a6ddf042fe6546a", - "schema.nulls.int64": "arrow_v0.1:000001f31bd23e3b2489f68d11a6d3087d09fb8df563c2fb84b3084a6ddf042fe6546a", - "schema.nulls.string": "arrow_v0.1:00000191f7f5c26abcceb89a6fb399fd0dc333c743ad38a5f2a4da8b274b7e96acfe18", - "schema.order.ab": "arrow_v0.1:00000100ad4d558238e956277266aaa4960beca8b50b1bc6fb5d367fa90d01febeaecf", - "schema.order.ba": "arrow_v0.1:00000100ad4d558238e956277266aaa4960beca8b50b1bc6fb5d367fa90d01febeaecf", - "schema.order.rows_reversed": "arrow_v0.1:00000100ad4d558238e956277266aaa4960beca8b50b1bc6fb5d367fa90d01febeaecf", - "schema.width.int16": "arrow_v0.1:00000178c6959f782c8a48a2d5b2d3f29e66bf5c705bbd9a5f5e64254b6fa6ffd34a52", - "schema.width.int32": "arrow_v0.1:0000011844a51e73c25585ae05f082dabb80bfebaa16200fc06da0d6b76552db358063", - "schema.width.int64": "arrow_v0.1:000001f31bd23e3b2489f68d11a6d3087d09fb8df563c2fb84b3084a6ddf042fe6546a", - "schema.width.int8": "arrow_v0.1:000001d22fd4d64d8554ce4ab154133f31f73b2de03b55c681899ce92aeea83a708f7a", - "schema.width.uint8": "arrow_v0.1:000001c63af839f862f03281ac8fc3054dd41f42080dc323c1296f0662125f19385a49", - "table.bool": "arrow_v0.1:0000012fafed9df99235be951e30505fabd673fd8ca4e3ef8ed48fce0c5ef9ff8aacef", - "table.empty.zero_rows": "arrow_v0.1:000001d93c4003276f674c8c0cdc018db38a9cc545d7aec3a1a71f672e78facea059d7", - "table.float.float32": "arrow_v0.1:00000177212619946903d4b8c103832d9b0daad142e51e81ee334b5c00040cab8254ed", - "table.float.float64": "arrow_v0.1:000001c1d762b8d385549ea148addb9929818adb75d575f60a64156dfb0bce3b5eb19c", - "table.layout.binary": "arrow_v0.1:000001b19bcc9f2f94795bfd70390c3a62ee22a97e33db3e6d0c5b5902df8f59fd0650", - "table.layout.large_binary": "arrow_v0.1:000001b19bcc9f2f94795bfd70390c3a62ee22a97e33db3e6d0c5b5902df8f59fd0650", - "table.layout.large_list": "arrow_v0.1:00000189f2c4331bbc824253ae959dd9d76b9c1a0775f389cf087997d51c00f0f98d97", - "table.layout.large_string": "arrow_v0.1:000001590b6b8a65a49348746f5b11ad476c65d25f106becdf95bb807b1fdca8d8e88b", - "table.layout.list": "arrow_v0.1:00000189f2c4331bbc824253ae959dd9d76b9c1a0775f389cf087997d51c00f0f98d97", - "table.layout.string": "arrow_v0.1:000001590b6b8a65a49348746f5b11ad476c65d25f106becdf95bb807b1fdca8d8e88b", - "table.nested.list_list": "arrow_v0.1:000001639751855e6618fdfba13f9d4bf5305716443012b812bf81fd652a43e89a671d", - "table.nested.list_struct": "arrow_v0.1:0000013c0c348a94fa935da0e77e9a47c26c79ff726c3383ad0206da391735348af38d", - "table.nested.struct": "arrow_v0.1:0000012af9c2ad25c501bbfa488ea24088eca78480303b8b413968c33836a133458dfb", - "table.nullability.non_nullable": "arrow_v0.1:000001a55bac289fa4cb83a654a76adc0eaf9cb5a953d14c90c7afb9a3b64c1f5f51e8", - "table.nullability.nullable": "arrow_v0.1:000001d469c2c16d7e2c80bed9daf4d1bd919eaa8523edd905623eccb5fe8abed7c430", - "table.nulls.int64": "arrow_v0.1:000001604458bd26489ed611491258c0a8a7e1336772fe6b603d1948ec16bce5a7f6da", - "table.nulls.string": "arrow_v0.1:00000189124cc15f2fbfc1d2056ae127123c1408307fbf8ba05b6018ea62b49eef4d39", - "table.order.ab": "arrow_v0.1:000001fb52c5ca8baed03ac9c12f5dfd0fc7289280a4bd147200eb7cf901a1cee2dfae", - "table.order.ba": "arrow_v0.1:000001fb52c5ca8baed03ac9c12f5dfd0fc7289280a4bd147200eb7cf901a1cee2dfae", - "table.order.rows_reversed": "arrow_v0.1:00000151aa84ec1e72539ee16ed86379c9681bc6c0aee94edf56caa3f302d3d2a5dc13", - "table.width.int16": "arrow_v0.1:0000013e000f3e0b4beeba1e5ed6246ab460a93ffc370591e07e8350c16bffdf5dedd6", - "table.width.int32": "arrow_v0.1:000001d831e96457f732f785fb0963eb56a8723e18778d90b6ad83cfc6ac3062cb51ef", - "table.width.int64": "arrow_v0.1:000001d469c2c16d7e2c80bed9daf4d1bd919eaa8523edd905623eccb5fe8abed7c430", - "table.width.int8": "arrow_v0.1:00000160707f6988db37fafe9ef4506de6855cddcacba6a75f9f30875b3067da949b8d", - "table.width.uint8": "arrow_v0.1:000001dded35c6cee950b7a706dc347563e65ce3525b910ae89a111dfea033483609ac" + "extension_schema.extension.canonical_fixed_shape_tensor": "arrow_v0.2:0000022df2fb43d6457645dea99103a5ee4ca34c1728ef387685dbb73c58a43fc6a002", + "extension_schema.extension.canonical_uuid": "arrow_v0.2:00000236be584e0cd3a2668a6a580119beecc6e63496abf7dc098302bb597de10190e9", + "extension_schema.extension.path": "arrow_v0.2:0000024887bbb622b764650a2d4346f7fcd84cdc17f4a6a8a1b9a33cf0e78a782d8420", + "extension_schema.extension.unregistered_passthrough": "arrow_v0.2:000002853ab951d35ce3198dd79a53acdb002b47b213f4d7a7b7bd3f9e81e76535873f", + "extension_schema.extension.uuid": "arrow_v0.2:000002bfbbdea0a2cf9f0941aad6fbb6510bbc8b5d46f4e1bc473ddb42ba2e817dbf76", + "extension_table.extension.canonical_fixed_shape_tensor": "arrow_v0.2:0000029cdbbb5f9c731c669dceb6661878b8f0d1cccc5c6609f7e18d2f0b8691c63bf6", + "extension_table.extension.canonical_uuid": "arrow_v0.2:0000024cf6272f99fa3465a975a572a4cfa5aaaf07d1d063fda398cbbaf1dafbe8913d", + "extension_table.extension.path": "arrow_v0.2:000002f558ddf91c1c5f706827496773171d001d541cc96945e2f58420d939653efcfb", + "extension_table.extension.unregistered_passthrough": "arrow_v0.2:000002b9c5f64eb011d128c21fa0570cab2bf34915277103a843c3d2174bb14c4ad158", + "extension_table.extension.uuid": "arrow_v0.2:000002062ec7b363fa63cfbfac9085e5fbc271720d1bcfbd66018d30996915f6be39af", + "schema.bool": "arrow_v0.2:00000292f4b85e8651fe4c797b8057205fb5aa89dd8420bb6ecef56e70f91c23b06718", + "schema.bool.chunked": "arrow_v0.2:00000292f4b85e8651fe4c797b8057205fb5aa89dd8420bb6ecef56e70f91c23b06718", + "schema.collision.list_lengths_a": "arrow_v0.2:00000217b4f541f59698108a9542f81fb4551d5ed9a636f7d7e9c91e44b9ffacb1a244", + "schema.collision.list_lengths_b": "arrow_v0.2:00000217b4f541f59698108a9542f81fb4551d5ed9a636f7d7e9c91e44b9ffacb1a244", + "schema.collision.null_struct": "arrow_v0.2:000002377ba0fc7c623d903020da3c502ec41ae878327623355ec9d059e46cef008230", + "schema.collision.struct_of_nulls": "arrow_v0.2:000002377ba0fc7c623d903020da3c502ec41ae878327623355ec9d059e46cef008230", + "schema.empty.zero_rows": "arrow_v0.2:000002756a354ce60282443c464797dfb1fe280a84608545b20376d17bb62cba7e282e", + "schema.fixed_size_list.float32_2": "arrow_v0.2:00000256387ebbc501cb6fcad40accf4a1586d4cd4c97cc22a6c9f99b3103575e5c2c6", + "schema.fixed_size_list.null_slot": "arrow_v0.2:000002a91c8dfd080d7012a5495e6a913c69c3bade25caeb2cbdcb1174d7871af4b431", + "schema.float.float32": "arrow_v0.2:0000026a27b2c2276546fc46218d22a66cc44d468aa7fe5b478f6c1afe2ba648ca86ea", + "schema.float.float64": "arrow_v0.2:000002a8bb0c99c2d60fd20d091bf861eff2c889fbbd9d8f08e0a8ad4afd4a6ebb009d", + "schema.layout.binary": "arrow_v0.2:000002346ed5761cafbd7a622360039d70f05235102ebe07cdbaaa94dcc7fb99a91f65", + "schema.layout.large_binary": "arrow_v0.2:000002346ed5761cafbd7a622360039d70f05235102ebe07cdbaaa94dcc7fb99a91f65", + "schema.layout.large_list": "arrow_v0.2:000002409130499175385ea85061c49e23cc7d91be7cff0a9b4bb3eadab2dcd65e4aa6", + "schema.layout.large_string": "arrow_v0.2:000002d9023abe34e3bde2cb36f499d4eecc685252bb1e06713f0f8e04af77ef580e3b", + "schema.layout.list": "arrow_v0.2:000002409130499175385ea85061c49e23cc7d91be7cff0a9b4bb3eadab2dcd65e4aa6", + "schema.layout.string": "arrow_v0.2:000002d9023abe34e3bde2cb36f499d4eecc685252bb1e06713f0f8e04af77ef580e3b", + "schema.nested.list_list": "arrow_v0.2:0000029753856b34b9c54c1cfa002906119f22e1255fa9b944feb46c8856f395af8fdd", + "schema.nested.list_struct": "arrow_v0.2:000002cd3954fa9d6958fc1a13b0e9534b62cf76fd31dc45b3ee26ad0c054664a73b35", + "schema.nested.struct": "arrow_v0.2:0000021c6ab7531323b6ed5c4f8bceebe1a7351205355481e40a82213076e969ba8381", + "schema.nullability.non_nullable": "arrow_v0.2:000002389fdc8277c962a8352a2529d0925a0c4d915257d6a1b12d7e052574be1b89e1", + "schema.nullability.nullable": "arrow_v0.2:000002756a354ce60282443c464797dfb1fe280a84608545b20376d17bb62cba7e282e", + "schema.nulls.int64": "arrow_v0.2:000002756a354ce60282443c464797dfb1fe280a84608545b20376d17bb62cba7e282e", + "schema.nulls.string": "arrow_v0.2:000002d9023abe34e3bde2cb36f499d4eecc685252bb1e06713f0f8e04af77ef580e3b", + "schema.order.ab": "arrow_v0.2:000002b8f677fa5b62affbb33424741912b558b4465167c45a3598287211060412a743", + "schema.order.ba": "arrow_v0.2:000002b8f677fa5b62affbb33424741912b558b4465167c45a3598287211060412a743", + "schema.order.rows_reversed": "arrow_v0.2:000002b8f677fa5b62affbb33424741912b558b4465167c45a3598287211060412a743", + "schema.width.int16": "arrow_v0.2:000002c9ea066d6e1b441ad66b24a5107953dbdda78c7d25d4f4bde8fb685c4748245f", + "schema.width.int32": "arrow_v0.2:000002447a113da124fde874ccfa83a281d7053ae1606ac6fac1f90b6d0a41bf5e8845", + "schema.width.int64": "arrow_v0.2:000002756a354ce60282443c464797dfb1fe280a84608545b20376d17bb62cba7e282e", + "schema.width.int8": "arrow_v0.2:00000233ec0dccf2b00dbbd7b75e5bb06152b5a53acee7101617fbff06641e7965cb9b", + "schema.width.uint8": "arrow_v0.2:00000252e16232ff906ecf17b849d7398396d2fe973ff9ca378ffae61b93de5846d7a7", + "table.bool": "arrow_v0.2:000002dee7bb2ff6dce8ddcd5037265ab54ba5f847433b9fc2718b1e7b8c7acad1a44a", + "table.bool.chunked": "arrow_v0.2:000002dee7bb2ff6dce8ddcd5037265ab54ba5f847433b9fc2718b1e7b8c7acad1a44a", + "table.collision.list_lengths_a": "arrow_v0.2:000002295a4cad9a00a66f4c0b07e5b24145f0962f0633ca6b6995ded658cf71196827", + "table.collision.list_lengths_b": "arrow_v0.2:0000026806409bd5df9d075d56b3bbfdb6655d46d1f396e0926052cdb56cfd93edd357", + "table.collision.null_struct": "arrow_v0.2:00000238e6e122820c9902bc72be26e7c542fbc0e8f13ade44142728ac66b0b14f6a70", + "table.collision.struct_of_nulls": "arrow_v0.2:0000024ef1bfba9ecf0d650296647c90a102c4d5699ec82be53e9dab27d62e436c3cd4", + "table.empty.zero_rows": "arrow_v0.2:000002d655beec05f54e13a35a4b07c5cb5b0fae4f406e22261858cb72a6f96e3aa33f", + "table.fixed_size_list.float32_2": "arrow_v0.2:000002a963ce04c642d126d49f7babad7e389aaa077d7a937cdf38ec9ee4590a6929a7", + "table.fixed_size_list.null_slot": "arrow_v0.2:00000223564917b29cabbfdfabd08e1ebb26cad4b4c1fd1af05b62875105fbff10c5fc", + "table.float.float32": "arrow_v0.2:00000287ee09555fc05dfbac367f42c6d50de2d3e7a458444920677ff1bd62ad04da40", + "table.float.float64": "arrow_v0.2:000002d83df46319a27052e5272f72f2025c85cef0b3c8139c36940094ccc91db262e0", + "table.layout.binary": "arrow_v0.2:0000021977fb0b4aac38946863766589aa73c09e47a59bd46d2fe0bf5f8292b6e2b64a", + "table.layout.large_binary": "arrow_v0.2:0000021977fb0b4aac38946863766589aa73c09e47a59bd46d2fe0bf5f8292b6e2b64a", + "table.layout.large_list": "arrow_v0.2:0000027db606bdc3327cd8b8bb45c17ce27f5235498e5f1aca008d18e0e876218405af", + "table.layout.large_string": "arrow_v0.2:0000021aedf65c94c24afaaec8660667c76e8fc296459255d74925110c5d66612f25d9", + "table.layout.list": "arrow_v0.2:0000027db606bdc3327cd8b8bb45c17ce27f5235498e5f1aca008d18e0e876218405af", + "table.layout.string": "arrow_v0.2:0000021aedf65c94c24afaaec8660667c76e8fc296459255d74925110c5d66612f25d9", + "table.nested.list_list": "arrow_v0.2:000002d0d198475a44dc9ab4694adc8348990fca5a0e0365aaf75317a1be70479178a4", + "table.nested.list_struct": "arrow_v0.2:0000029919cc15883d26c0304782ccfeb1ac7a543a9260d9b401a4cfd56ae59f33bac3", + "table.nested.struct": "arrow_v0.2:0000022efc170a4f2578623b5525861024eb528aa2d167cf0006d7694ca971dc3fe370", + "table.nullability.non_nullable": "arrow_v0.2:000002b37da33f664f9dc4acb7851c5a8431abd457651c43aa5c2094093cc97cd33b40", + "table.nullability.nullable": "arrow_v0.2:00000236be7e40075274babcbf4ef0082a8b76252a91c085ca04e41c6ae1a718eb64a7", + "table.nulls.int64": "arrow_v0.2:0000023ba22c1f5257d8cf501925b2db2486687b2180a5b2882c8022ce3d45a56f1efd", + "table.nulls.string": "arrow_v0.2:0000026885780c872492587fc493be66f47184080b87ccbec1327e64fbe52c864d24a4", + "table.order.ab": "arrow_v0.2:000002d29b71de1ff0980cd38422829174ff8a702f4db64ec704e9d41e966bdceaeeaa", + "table.order.ba": "arrow_v0.2:000002d29b71de1ff0980cd38422829174ff8a702f4db64ec704e9d41e966bdceaeeaa", + "table.order.rows_reversed": "arrow_v0.2:000002edcc170ff123dccde69febaae0fd8a87420ac5b4d408047a18cf270223cb7e39", + "table.width.int16": "arrow_v0.2:000002c53e3f002b007a80a664f8fda9e8e11b852fe6d89f6e4883d0e7aba2499f4b7a", + "table.width.int32": "arrow_v0.2:000002ee36b64c06c9157f548f5b89d21935b40cbfa6bd06cdf283561c92523c75a8a5", + "table.width.int64": "arrow_v0.2:00000236be7e40075274babcbf4ef0082a8b76252a91c085ca04e41c6ae1a718eb64a7", + "table.width.int8": "arrow_v0.2:000002f195acdfc1ca1214df9812ef4d5d7c4dbe2ce78d2b4a6e2f6aa6f4f0cd8593d6", + "table.width.uint8": "arrow_v0.2:00000254a73782eb935f0145074c560cd638d035964249d5d2f5b3245c91f9132b7232" } diff --git a/tests/test_hashing/golden/data/datagram.json b/tests/test_hashing/golden/data/datagram.json index 5d6fa4e81..92a75dbe0 100644 --- a/tests/test_hashing/golden/data/datagram.json +++ b/tests/test_hashing/golden/data/datagram.json @@ -3,13 +3,13 @@ "arrow_source.int32.pipeline_hash": "semantic_v0.1:1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "arrow_source.int32.row[{\"k\":\"r1\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.int32.row[{\"k\":\"r1\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int32.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001a01cd0f32e44a4839ee25caa5b0fb526dd1926b6cd9a43d717ec6a3c92364d29", + "arrow_source.int32.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002d92f4d29531f412586881fd7891dc20e057fbb83dedc711af0f9bd83b2a56d35", "arrow_source.int32.row[{\"k\":\"r2\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.int32.row[{\"k\":\"r2\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int32.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001cbdc2fbc671d4157fa251c1772062b3c0a655fa8a5b21cbdc9b64de0d564c859", + "arrow_source.int32.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:00000243c87e6d05d3588edc8eb3eecc6d82902681548386747578da1a3f7953efe35b", "arrow_source.int32.row[{\"k\":\"r3\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.int32.row[{\"k\":\"r3\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int32.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:0000012f75fdca845f528ca916bbfc14e1a3a6dba421d54f016c9e64f045efda579918", + "arrow_source.int32.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:000002825e1c06b5514e562a8961019373bb97d951aac4526cd6d2695d96aa2509b326", "arrow_source.int32.system_tag_columns": [ "_tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "_tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90" @@ -18,13 +18,13 @@ "arrow_source.int64.pipeline_hash": "semantic_v0.1:1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "arrow_source.int64.row[{\"k\":\"r1\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.int64.row[{\"k\":\"r1\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int64.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001c028af694082af2f2d902ddc2b8b21363644f3914428cb5363760cbebe521908", + "arrow_source.int64.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002784ccdd2b780948d2512ca81004840de8c31b16f446898f4e30ce7dcfbbcd4e8", "arrow_source.int64.row[{\"k\":\"r2\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.int64.row[{\"k\":\"r2\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int64.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000017e332b258dbc5fe1f459b9d3dd514afa4dfeaae1e9c816a4f6fd4c545d691f11", + "arrow_source.int64.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:0000021d3e14bd65d5ca2633843f2ab29ecdf7df5626e980824695bd21aa24d86cd2fd", "arrow_source.int64.row[{\"k\":\"r3\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.int64.row[{\"k\":\"r3\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int64.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:0000015c299c665cda513e3eda946aab5ae38ed0b637e47f036462361c91804656da01", + "arrow_source.int64.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:0000023846c7aef08fc14007f95079971a9c8fc62255fa724097ba05594d335f1cd806", "arrow_source.int64.system_tag_columns": [ "_tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "_tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90" @@ -33,13 +33,13 @@ "arrow_source.int64_nullable.pipeline_hash": "semantic_v0.1:686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94", "arrow_source.int64_nullable.row[{\"k\":\"r1\"}]._tag_record_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.int64_nullable.row[{\"k\":\"r1\"}]._tag_source_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "golden_arrow", - "arrow_source.int64_nullable.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000012081a40ec7eba71155774352b7e33349f231cd7a1ad4b4b0c1c85897ef6617c5", + "arrow_source.int64_nullable.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002521ff3d4701e70a5a30a4bb915ab2c1e06d4d4fe7b43e8b542f63c2cb2302c26", "arrow_source.int64_nullable.row[{\"k\":\"r2\"}]._tag_record_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.int64_nullable.row[{\"k\":\"r2\"}]._tag_source_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "golden_arrow", - "arrow_source.int64_nullable.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001ebc7fce25941def04ca935645b0dec2d066ab0f9a902df12fa0dc7f18c84a009", + "arrow_source.int64_nullable.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002601fc387de4781312ad355ab3e8ca0f1285daeda7f5ba831bc7c9a167cf6daa8", "arrow_source.int64_nullable.row[{\"k\":\"r3\"}]._tag_record_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.int64_nullable.row[{\"k\":\"r3\"}]._tag_source_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94": "golden_arrow", - "arrow_source.int64_nullable.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:000001951dc05c8c79ecd4aa8521ae9845c34c8d0b0e343c14a97bdba79157e46bb51e", + "arrow_source.int64_nullable.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:000002565243c66b8ae12ded84d33ef6be188fda9250f3e100812d65efacd3144deeef", "arrow_source.int64_nullable.system_tag_columns": [ "_tag_source_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94", "_tag_record_id::686bec5f94c7ca900d74321adabd05ed62e7562ffcee48ff6c906a4b44174a94" @@ -48,13 +48,13 @@ "arrow_source.int8.pipeline_hash": "semantic_v0.1:1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "arrow_source.int8.row[{\"k\":\"r1\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.int8.row[{\"k\":\"r1\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int8.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001760d63148f60206156a8c6224b0f78ded56f5b7982e6211196b4a330a07b311d", + "arrow_source.int8.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002b17bad7455bf81d473552afdb3ad02a6c6dda0b630b3ba0347c717c419fb489f", "arrow_source.int8.row[{\"k\":\"r2\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.int8.row[{\"k\":\"r2\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int8.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000012622e7bf473c8c58db20b86b66fbc8a070cf4e7731339a0b2103f9d68a00f643", + "arrow_source.int8.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002ff7e5abaafd6df855a3d15271da4f3c4d63e10e3aa190237ff8dc76925762cdd", "arrow_source.int8.row[{\"k\":\"r3\"}]._tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.int8.row[{\"k\":\"r3\"}]._tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90": "golden_arrow", - "arrow_source.int8.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:000001c502c37a181f7c3e19bf6486dc69d141b3b35d14c742c854ac39c8f8f9964fee", + "arrow_source.int8.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:000002fa70cfa37eaab0c4e54cc5fe972b6001d9e2db1814160b335fb1d78df7e1169b", "arrow_source.int8.system_tag_columns": [ "_tag_source_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90", "_tag_record_id::1d303b36f5b4f916fa158b7b557192579008f0984e6127d8832c5731797bae90" @@ -63,13 +63,13 @@ "arrow_source.large_string.pipeline_hash": "semantic_v0.1:b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "arrow_source.large_string.row[{\"k\":\"r1\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.large_string.row[{\"k\":\"r1\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.large_string.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001b4407c40d9132743421fec4649ee47bd485dd294d5e5d0c178345df08cecb387", + "arrow_source.large_string.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:0000027dc69926b790513bbe98295d34f942eb71ed1a12b4aab259220fd0489bf4fb6c", "arrow_source.large_string.row[{\"k\":\"r2\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.large_string.row[{\"k\":\"r2\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.large_string.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001cd356426fd64cb49847fc1e14449cc9893d210b4388f3ccfa1d6052fef5f5ddc", + "arrow_source.large_string.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002e2668873da695349ec66d49132a1443f22972c7abb108643145169a8e45a7dc1", "arrow_source.large_string.row[{\"k\":\"r3\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.large_string.row[{\"k\":\"r3\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.large_string.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:000001e0c4e88cf1c63c94e17cdcfca663841a9c103ca9ef471c2108002d0b4c690d61", + "arrow_source.large_string.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:000002b08595f56b1cd4a87cb0bb7658c9defbae43f2a9c02c37b5b781bb2d69bf0d89", "arrow_source.large_string.system_tag_columns": [ "_tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "_tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e" @@ -78,13 +78,13 @@ "arrow_source.string.pipeline_hash": "semantic_v0.1:b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "arrow_source.string.row[{\"k\":\"r1\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "6b49f11e03d55f29b32b06c106ded2da", "arrow_source.string.row[{\"k\":\"r1\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.string.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001b4407c40d9132743421fec4649ee47bd485dd294d5e5d0c178345df08cecb387", + "arrow_source.string.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:0000027dc69926b790513bbe98295d34f942eb71ed1a12b4aab259220fd0489bf4fb6c", "arrow_source.string.row[{\"k\":\"r2\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "c4c6f49b43e25f92a3cf173c6ceebb4d", "arrow_source.string.row[{\"k\":\"r2\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.string.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001cd356426fd64cb49847fc1e14449cc9893d210b4388f3ccfa1d6052fef5f5ddc", + "arrow_source.string.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002e2668873da695349ec66d49132a1443f22972c7abb108643145169a8e45a7dc1", "arrow_source.string.row[{\"k\":\"r3\"}]._tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "8401f08964385da2a8c873cc2aa3fbe5", "arrow_source.string.row[{\"k\":\"r3\"}]._tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e": "golden_arrow", - "arrow_source.string.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.1:000001e0c4e88cf1c63c94e17cdcfca663841a9c103ca9ef471c2108002d0b4c690d61", + "arrow_source.string.row[{\"k\":\"r3\"}].data_hash": "arrow_v0.2:000002b08595f56b1cd4a87cb0bb7658c9defbae43f2a9c02c37b5b781bb2d69bf0d89", "arrow_source.string.system_tag_columns": [ "_tag_source_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e", "_tag_record_id::b58c50c5ff232cdb71eb0272547485074c9784600f5a34218434f7c3de419e8e" @@ -93,10 +93,10 @@ "dataclass.pipeline_hash": "semantic_v0.1:c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c", "dataclass.row[{\"k\":\"r1\"}]._tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "26db93a5eab45b0485c2699148b0600d", "dataclass.row[{\"k\":\"r1\"}]._tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "golden_dataclass", - "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000015e306dd402af59e45e6f5a9113e71e691f96006cb8cf63246504800071050f55", + "dataclass.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:0000027b15d8db313a05d7a3f0af1cfa788fd730bb3d47eacb201de61f4c24b9a2f0bd", "dataclass.row[{\"k\":\"r2\"}]._tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "71d5c6ce2f7b5b5b9dfdc7a61a7df366", "dataclass.row[{\"k\":\"r2\"}]._tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c": "golden_dataclass", - "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001b088edbc3232e6d836e6036a75d5c0a0d83864166d21a16ad113608e21afe445", + "dataclass.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002abd8873e34ca7ee3f967b31d5fc3e56013d151495ae59d9a53f856f81deacc85", "dataclass.system_tag_columns": [ "_tag_source_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c", "_tag_record_id::c16022c03e2749f6a4f1d7ac2b255e87d1ccce6895c5328f2489103558b2b14c" @@ -105,10 +105,10 @@ "dataframe.pipeline_hash": "semantic_v0.1:38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14", "dataframe.row[{\"k\":\"r1\"}]._tag_record_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14": "3b7c66de1eb65a05a236c53ce3a3e292", "dataframe.row[{\"k\":\"r1\"}]._tag_source_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14": "golden_dataframe", - "dataframe.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000108d551603b5a5df12a0c1ac3f7d5b3caa1108525058b0e450c02ae93e94e358f", + "dataframe.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:00000229c5394be73a9c86795235a5f408835ded8167d2b05e00a114ab89760aebbee2", "dataframe.row[{\"k\":\"r2\"}]._tag_record_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14": "a075d0a40203532b92170507912ff6e6", "dataframe.row[{\"k\":\"r2\"}]._tag_source_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14": "golden_dataframe", - "dataframe.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000134950bb185ada91e996752020cd29a1dff584382c019be51e4f0ef15560e868b", + "dataframe.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:0000024dc4bbb7a657d7ba9af692ed778b268a1ed8a3ebf4c7243362a5c6072af4a451", "dataframe.system_tag_columns": [ "_tag_source_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14", "_tag_record_id::38f74931045e006f752450c2d26dbf9096c5bff65fbe952591dec83b04296a14" @@ -117,10 +117,10 @@ "list_path.pipeline_hash": "semantic_v0.1:3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020", "list_path.row[{\"k\":\"r1\"}]._tag_record_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020": "059c77251e845bafb5a3839bf547a2f5", "list_path.row[{\"k\":\"r1\"}]._tag_source_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020": "golden_list_path", - "list_path.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000135fc611ef9ad245d4906a4d4d8da286dafc50f184443aaecefdf4fbd63e65433", + "list_path.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002cc2ece0ae8ac648486e63890a8d4b354e1276bd85ca6575ce7ec8d8085928dd3", "list_path.row[{\"k\":\"r2\"}]._tag_record_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020": "35b02f96d67b500db2ae69b3138302b8", "list_path.row[{\"k\":\"r2\"}]._tag_source_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020": "golden_list_path", - "list_path.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000018efe66b052bc46996b623c89fbc73f8e2ba005fa27d2388989fbfdbd053624f8", + "list_path.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002cdaddd554ca8557405f6b0dc07b8a1b706b116ac336d86d6da402fa99ddea137", "list_path.system_tag_columns": [ "_tag_source_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020", "_tag_record_id::3d12eb0f29844205335f9ffd7843afc13c308dece8bae64b9f94d980c690e020" @@ -129,10 +129,10 @@ "ndarray.pipeline_hash": "semantic_v0.1:be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5", "ndarray.row[{\"k\":\"r1\"}]._tag_record_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5": "9caca45075165a87bf8320fef620e949", "ndarray.row[{\"k\":\"r1\"}]._tag_source_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5": "golden_ndarray", - "ndarray.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001c80cacf9821f15fd18dfc7f6649268fcffefeaad1449c81cdfa716a8738f8d6e", + "ndarray.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002e79fea6f79a5a7e617db9cc2a749f627de86be7128c084616e397c5a04d28d49", "ndarray.row[{\"k\":\"r2\"}]._tag_record_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5": "992e0c0ae0e957df944a73440a6f6451", "ndarray.row[{\"k\":\"r2\"}]._tag_source_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5": "golden_ndarray", - "ndarray.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001ba0bc856108ff6198b30e7d1f5fc78bdbca5f1790c108b0a35f1d735b5e1b896", + "ndarray.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002922f3a9172b4d736387286cf2dd4d1121d85238403d9c8c350785a03809cbb3b", "ndarray.system_tag_columns": [ "_tag_source_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5", "_tag_record_id::be37e32a759adc87f88657caf69a48238722f512a9c80892ab023739e4ebfda5" @@ -141,10 +141,10 @@ "nested.pipeline_hash": "semantic_v0.1:938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09", "nested.row[{\"k\":\"r1\"}]._tag_record_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09": "22fdf98f207f580e80aa48ea45e7d81b", "nested.row[{\"k\":\"r1\"}]._tag_source_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09": "golden_nested", - "nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001c31188a44c589ce3daf646ca787cf70383531e48ea6864e540be884a15d180bf", + "nested.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:00000261f569412bb12da248eaef81f93ee80ab2c93978f737fa604cc11309276f6f40", "nested.row[{\"k\":\"r2\"}]._tag_record_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09": "20a95fcca8955b96b501e71105cfcd7c", "nested.row[{\"k\":\"r2\"}]._tag_source_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09": "golden_nested", - "nested.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001a7658c283bcd7a1c6d933a5dba77b96c6457e8384a5a8d58a151a6acec619469", + "nested.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002f4312647b9ca61759ae335c7b0639d667709ae8f797a7564e31bdf9a454bc5a1", "nested.system_tag_columns": [ "_tag_source_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09", "_tag_record_id::938fabe22ed56031d43337723a8c8478ae6010fdb3594e9dc8c14d7d98b6df09" @@ -153,10 +153,10 @@ "optional.pipeline_hash": "semantic_v0.1:2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0", "optional.row[{\"k\":\"r1\"}]._tag_record_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0": "4e41e4e151be5a88b2f275893f537118", "optional.row[{\"k\":\"r1\"}]._tag_source_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0": "golden_optional", - "optional.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001f910e5b3279111cf4734db9a4669b90b16aa1cecba678ef8364d7b6b28615487", + "optional.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002f774b1bc717e2ddf7bacc9707ccb64b05ef3935a68fcb064980cb7134e672c1e", "optional.row[{\"k\":\"r2\"}]._tag_record_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0": "5b3fa9a72962574690a8e28a8f04224a", "optional.row[{\"k\":\"r2\"}]._tag_source_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0": "golden_optional", - "optional.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000010d55a89a30404439bfd8b5fb204fda1e0aafeaefc0e9c625b524221d032f0bca", + "optional.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:0000027714fb655c044c9c124dfcb5b6ee5895e2ff00b3edd051990817bdb01e6bea34", "optional.system_tag_columns": [ "_tag_source_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0", "_tag_record_id::2b2731c4ac73505b88c0146d11eb7c77d6e6c701f08da676440e28d9f6d845a0" @@ -165,10 +165,10 @@ "path.pipeline_hash": "semantic_v0.1:bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09", "path.row[{\"k\":\"r1\"}]._tag_record_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09": "8995b9653892518d975b74e051ddff38", "path.row[{\"k\":\"r1\"}]._tag_source_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09": "golden_path", - "path.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000135bf11f7d6830036c99e7445af70c2d668909f9000362d8ede899e435e18a8fc", + "path.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002a230080e1f03f22543bcd61b9956710b80b51a14fc6b74c0bdc994e3e82b4cd1", "path.row[{\"k\":\"r2\"}]._tag_record_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09": "e2adbc7a08965c6db075bab848287b62", "path.row[{\"k\":\"r2\"}]._tag_source_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09": "golden_path", - "path.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000015b4f8156dc62851f96532b141710b02eb0d2d9e558578002c8daa84669c00339", + "path.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:00000234b6166fa805040e49bada1a6fad4b8c406173b1df7d3aa99db1de764196de36", "path.system_tag_columns": [ "_tag_source_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09", "_tag_record_id::bf2528f33e2c72825ecca9151c2302d77df0e6e7544e955e56101468f9b2bd09" @@ -177,10 +177,10 @@ "pydantic.pipeline_hash": "semantic_v0.1:aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967", "pydantic.row[{\"k\":\"r1\"}]._tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "cb9bbf1a138d5c848e86f99bab7c0acb", "pydantic.row[{\"k\":\"r1\"}]._tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "golden_pydantic", - "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000012280b233c56771fa4f901eff64090067273d718ac8d36b881b78d79d4128ab15", + "pydantic.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002459c1d7ed12f91d7f292eafe1514160775ff885d92151dcf0da2ecf256dbbf1a", "pydantic.row[{\"k\":\"r2\"}]._tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "10c1dbaba95a5c5fa7c14661231224eb", "pydantic.row[{\"k\":\"r2\"}]._tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967": "golden_pydantic", - "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:00000138d77ff36d7b2f3c326a128b325d8219e31f55be9264a21abf51638993059c7e", + "pydantic.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002ce57007b0730b902132be534f05b27828f588bea325de4d54876721c6c5c7463", "pydantic.system_tag_columns": [ "_tag_source_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967", "_tag_record_id::aeb2c7a55a50242b1c5916876b4004e77108a32ac2b97bd974184ad4091ed967" @@ -189,10 +189,10 @@ "scalars.pipeline_hash": "semantic_v0.1:1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07", "scalars.row[{\"k\":\"r1\"}]._tag_record_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07": "dd4b595c8e6554f7ae4ca854e411dfa8", "scalars.row[{\"k\":\"r1\"}]._tag_source_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07": "golden_scalars", - "scalars.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:00000162a09a0c2dd7d8e7c09355e4c4d3b1992e26b0d6a53b706da57ea9117183eb7c", + "scalars.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002c8161def16592dc28c2dfa2cff5084c6d1bf397667bf401453b85f1636b29896", "scalars.row[{\"k\":\"r2\"}]._tag_record_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07": "ffc8945f5a4053d79c69860fd298d6e0", "scalars.row[{\"k\":\"r2\"}]._tag_source_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07": "golden_scalars", - "scalars.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001d49094978c1867f7432b53ac342c9bf81f9e5cb636a8a7e8c5090027eb9b6f70", + "scalars.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:0000020528b8069a7f49fd5905ecc79c4ff589d0ec471e0274bffdd5a51ae7756655f2", "scalars.system_tag_columns": [ "_tag_source_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07", "_tag_record_id::1fdf145196d8e5dd7b38887cc8798325147df1f5c6f2af80952488a76f9aec07" @@ -201,10 +201,10 @@ "set_uuid.pipeline_hash": "semantic_v0.1:060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae", "set_uuid.row[{\"k\":\"r1\"}]._tag_record_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae": "2d3990a5b84f5cf0a5c846c448a25267", "set_uuid.row[{\"k\":\"r1\"}]._tag_source_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae": "golden_set_uuid", - "set_uuid.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:0000011eb69965d474d220990b2895cc181cf2938ebbc5cc8ae96f5eead708129c4198", + "set_uuid.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:000002648eddf1e15a4ed2e1ef619eb2d072950ba44f29ad23c935f8241e845cea0e81", "set_uuid.row[{\"k\":\"r2\"}]._tag_record_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae": "6ce8daa1eb235f2cb041e30468ff396a", "set_uuid.row[{\"k\":\"r2\"}]._tag_source_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae": "golden_set_uuid", - "set_uuid.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:000001fc38bc3b27de6a06637c17cf1c24119d2d13782bc2a78080596dfae657ff5d0b", + "set_uuid.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:000002ec023f0196ddaaa81bf359b83f34e17b39554ede462359bb9f030977c4a005f8", "set_uuid.system_tag_columns": [ "_tag_source_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae", "_tag_record_id::060b8a8e22b65b3cf6cde2ee5fc3ae4eedd840b377a2e071ec5e99cbb8a913ae" @@ -213,10 +213,10 @@ "uuid.pipeline_hash": "semantic_v0.1:33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581", "uuid.row[{\"k\":\"r1\"}]._tag_record_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581": "c17eccb33f355b1e9d6878f55b081026", "uuid.row[{\"k\":\"r1\"}]._tag_source_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581": "golden_uuid", - "uuid.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.1:000001e05de561e03537d1d2a793c43b38a9a7eae680f223a0396113a4137917faa8ca", + "uuid.row[{\"k\":\"r1\"}].data_hash": "arrow_v0.2:00000228d36dcdde1b6e7ff9ea8439b7d31c857f2df240443250ddec8cab124a896222", "uuid.row[{\"k\":\"r2\"}]._tag_record_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581": "adfcc16e72e55d30884f9feaf02caef2", "uuid.row[{\"k\":\"r2\"}]._tag_source_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581": "golden_uuid", - "uuid.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.1:0000014c189ef879ddf91cfe2ee92808d90ebf835de50dcb6a84c468545d5fe61c4ff8", + "uuid.row[{\"k\":\"r2\"}].data_hash": "arrow_v0.2:00000236a95b5f9e34143f912dcf0b7da9f7531c7fca0120c93237b053897f3e11ca0f", "uuid.system_tag_columns": [ "_tag_source_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581", "_tag_record_id::33b9e5f9945a83a66c7e77d79a188ec3dc9bbb2c64820c88ab47ffc361296581" diff --git a/tests/test_hashing/golden/data/pipeline.json b/tests/test_hashing/golden/data/pipeline.json index d5b5257ce..5b36b2577 100644 --- a/tests/test_hashing/golden/data/pipeline.json +++ b/tests/test_hashing/golden/data/pipeline.json @@ -3,10 +3,10 @@ "batch.pipeline_hash": "semantic_v0.1:60c337773086050f28d338eb45ebda97ab278426b7b491c996b951c31d71a078", "batch.row[{\"subject\":[\"s1\",\"s2\"]}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "0843c16c912c561eb8920aa92fc21c85", "batch.row[{\"subject\":[\"s1\",\"s2\"]}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "5f85d65fa9a90d1a4266f0b684b4a2c2a040cf7835bd14dfd37cde20fc635efb", - "batch.row[{\"subject\":[\"s1\",\"s2\"]}].data_hash": "arrow_v0.1:0000010819e767e277e136c495f98e9bc5da5d252b5a2b83cdbdec869994e5ba033508", + "batch.row[{\"subject\":[\"s1\",\"s2\"]}].data_hash": "arrow_v0.2:0000020b638d3ccc763f2a88e2ade9b759bb8641c55073185d4839047a3fc91276dc7d", "batch.row[{\"subject\":[\"s3\"]}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "47519d093cfc58878d21a35188b10c78", "batch.row[{\"subject\":[\"s3\"]}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e223d8ec1e6450dae88d4eb042479e5fa6104e6fd30b15add076db936b290367", - "batch.row[{\"subject\":[\"s3\"]}].data_hash": "arrow_v0.1:0000014957b9ee25e0ea51c695f25e5fc85b74946b3b9c4e391f88bb8d4a4734694a4e", + "batch.row[{\"subject\":[\"s3\"]}].data_hash": "arrow_v0.2:000002928bbf5d31fadf74ea3118af85c0d2273f77d53274f492e63a955520e4fac208", "batch.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -15,33 +15,33 @@ "column_selection.drop_data.pipeline_hash": "semantic_v0.1:d4de6c0b8a694ef32b07cba1fcc682e22f0b73c27dd7f3a4f6b398188bd5255c", "column_selection.drop_data.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "column_selection.drop_data.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.drop_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ee27bd12d66af74db02ecec14119254c263796aa8551e3d3249dc0025f22be17", + "column_selection.drop_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c734791b7452bb2affc60031491e7541bb53fb971cf3ef8b9af919eb0bf1f56e", "column_selection.drop_data.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "column_selection.drop_data.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.drop_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "column_selection.drop_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "column_selection.drop_data.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "column_selection.drop_data.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.drop_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001966c5e58c797ab3329316dffd36c3dcf09a3e0a400fc4511114e1e99f10939ea", + "column_selection.drop_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000025f33a2f5bf3e956edfb7ca628b6871f52b7bbf788d5a3774ddffe300d7d4aff5", "column_selection.drop_data.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" ], "column_selection.drop_tag.content_hash": "semantic_v0.1:adc2bbbc40dfe518e580cc332638c7d7cdad2d170a392649ea72c38a97d21db0", "column_selection.drop_tag.pipeline_hash": "semantic_v0.1:ddfedb2b384d0d6d5e5236f60aaedc02182660a9518c3e6d93f2535cd02c7c30", - "column_selection.drop_tag.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", - "column_selection.drop_tag.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.1:000001966c5e58c797ab3329316dffd36c3dcf09a3e0a400fc4511114e1e99f10939ea", + "column_selection.drop_tag.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", + "column_selection.drop_tag.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.2:0000025f33a2f5bf3e956edfb7ca628b6871f52b7bbf788d5a3774ddffe300d7d4aff5", "column_selection.drop_tag.system_tag_columns": [], "column_selection.select_data.content_hash": "semantic_v0.1:4d5b0e7c758f28ad95439e0910a4f0fcc8d6916953ad4516bd33f4bdeebf45e0", "column_selection.select_data.pipeline_hash": "semantic_v0.1:390c5e93d5a1cc266a2f62461a5c7e706a801d5cb9d3ab045176540b47db04ad", "column_selection.select_data.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "column_selection.select_data.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.select_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:0000017cb817a7c2cfd6c716987b09fea2cc10609b9c4b2c854116bdc7a3ede1e625ea", + "column_selection.select_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:00000281f4d4a4e7742a2cae9d733cb30453503892bf97d5d5695d5b491df60c9af327", "column_selection.select_data.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "column_selection.select_data.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.select_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000124d0ecb4bf9d093b60b0947eaff30ac930b9b3b429f06d1b8848185b7e0fb1f8", + "column_selection.select_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000022420dc43cfa6c4d3682f044a90a1a222554a5d2fd5bc7ebf015f830d3e18869f", "column_selection.select_data.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "column_selection.select_data.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "column_selection.select_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001810e6ae31eebcd4f2bfadeb2472fce6a5d52d063b10cf286713e2ecd2fdec977", + "column_selection.select_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:000002805d579510905b591a2d3748d388531edbef8cec4da1cef5f18abe1a77b21238", "column_selection.select_data.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -50,10 +50,10 @@ "column_selection.select_tag.pipeline_hash": "semantic_v0.1:7ee4cf0256ecc2c9150b67a01a71d0a6a1368f452b872bede59b60ae6fcd8073", "column_selection.select_tag.row[{\"cohort\":\"a\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "37d45a37e6255cd78bb62c4c9d047ca5", "column_selection.select_tag.row[{\"cohort\":\"a\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "column_selection.select_tag.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "column_selection.select_tag.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "column_selection.select_tag.row[{\"cohort\":\"b\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "d0cc03b6594d51788075066f5c81deb0", "column_selection.select_tag.row[{\"cohort\":\"b\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "column_selection.select_tag.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.1:000001966c5e58c797ab3329316dffd36c3dcf09a3e0a400fc4511114e1e99f10939ea", + "column_selection.select_tag.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.2:0000025f33a2f5bf3e956edfb7ca628b6871f52b7bbf788d5a3774ddffe300d7d4aff5", "column_selection.select_tag.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -62,10 +62,10 @@ "group_by.pipeline_hash": "semantic_v0.1:5ab1cb5d5ce9b641d19045bae0d94bbb9e15703bebfe37ec57fd13b141d2a0f0", "group_by.row[{\"cohort\":\"a\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "7d212ee5c358519c9e4729f4a0898aad", "group_by.row[{\"cohort\":\"a\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "bab61585da6894512c5947710f0c0d4bb46e3f0fcd511a088a4c76780f1ca5ae", - "group_by.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.1:0000018910e185c56ec7890dcba74a0553f06e355191c27a97a423dbbc32bea2441149", + "group_by.row[{\"cohort\":\"a\"}].data_hash": "arrow_v0.2:0000022b0f148a444e75daddc97429afe196d896fc41169dfe6fe9408438fde809c3ac", "group_by.row[{\"cohort\":\"b\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "b7f2f8f7412a54fdb03cd96b6b0b1643", "group_by.row[{\"cohort\":\"b\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "90ffe0efe9214e2770ce632ae3146e4bb33e9d28568b55b7dd67f07862836010", - "group_by.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.1:000001ba6adaea3c56bf0be427961819e0491e678c6dc6486999ab5025d9074ec9bbd4", + "group_by.row[{\"cohort\":\"b\"}].data_hash": "arrow_v0.2:0000028b04ecd69f72beab1905bf77d8f6300e916e6a6ce3a3db61b8dd15d641a213f0", "group_by.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -74,10 +74,10 @@ "index_pick.index.pipeline_hash": "semantic_v0.1:375b6916dd1d0c46e5553b471809ef66120dc092a19192a1f7b052e2481299c1", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "dd65b531e3d957a0ae3f2fba11e021d5", "index_pick.index.row[{\"subject\":\"s1\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000189f1942fbc40737ff78f6428f7be73a5dc531b1426c075ae3d5bc790b4ef8fcd", + "index_pick.index.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:0000025f5739a1fc427e583ba103ab5a7c1a5b868aadb554a722a1c0bd1d2081e4a34f", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "00c12388e8c4522893b029b39a8ec7aa", "index_pick.index.row[{\"subject\":\"s2\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000164b528c22b7183231910091256b6cb1d63bef2ded631aa330114a2fef997ed8c", + "index_pick.index.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002d9f79b11b0d6928476289f10dc1ffce27ef6af09affa82b079560b96435f62eb", "index_pick.index.system_tag_columns": [ "_tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a", "_tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a" @@ -86,10 +86,10 @@ "index_pick.pick.pipeline_hash": "semantic_v0.1:054e63991c65cb241b7ad88d011c4ef14ab3062b8f08642c0c39ddc7466d90a5", "index_pick.pick.row[{\"subject\":\"s1\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "dd65b531e3d957a0ae3f2fba11e021d5", "index_pick.pick.row[{\"subject\":\"s1\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.pick.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001380f07142b3132f93c7dfd5ef23532dbc35e83a7ddf3bd28ffbdc3405eb14045", + "index_pick.pick.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002d3aa3df31c715ed9534811e3b3ded71a017ea8132b501c69ed3ad5b78411d423", "index_pick.pick.row[{\"subject\":\"s2\"}]._tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "00c12388e8c4522893b029b39a8ec7aa", "index_pick.pick.row[{\"subject\":\"s2\"}]._tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a": "lists", - "index_pick.pick.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000157b70714f9f88f91649e6d135d91610267c66aef90f7552a03354fe3a8ffbdb9", + "index_pick.pick.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000218fa0f5bdf561a92ce32cecef15970f979890cf7c44fd9b8a0c8766e7c730dc2", "index_pick.pick.system_tag_columns": [ "_tag_source_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a", "_tag_record_id::2ae63653af6a57129ca7c6edfa8aa02c0511f59a8ee4acb678e223bb6302e22a" @@ -100,12 +100,12 @@ "job.node[adjust].row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "e8e02084627f5e1aad83839e421538e8", "job.node[adjust].row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "job.node[adjust].row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "job.node[adjust].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:0000014a3871996b0dc2563b8f3979740245bde22aac53376bba5592849cebc035b864", + "job.node[adjust].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002df3a95f3b9eddaf078096dee1388e410637a1a4ce3dd8028296b1a274c1292f2", "job.node[adjust].row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "42d239bc04015d1aaded030d9e02850f", "job.node[adjust].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "144bb37f3c295e16919d770641bb8c65", "job.node[adjust].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "job.node[adjust].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "job.node[adjust].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:0000014c738e65c53a408dfd0e017a32dd96504847f5ce08b13213b96168262a97e264", + "job.node[adjust].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000023565098d1a88ecd75dbc8ddc4900dc8304e066dc0c889114a2a41e2fe495fc30", "job.node[adjust].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -118,12 +118,12 @@ "job.node[joined].row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "e8e02084627f5e1aad83839e421538e8", "job.node[joined].row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "job.node[joined].row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "job.node[joined].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000130ab270ba209296b0933d7df77e7712b75b0e1cc5f9344775e30a29c9f4cc882", + "job.node[joined].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c945ad08138f51298940c60333852ef5191eb16077d96105386d44840c0ccfdf", "job.node[joined].row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "42d239bc04015d1aaded030d9e02850f", "job.node[joined].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "144bb37f3c295e16919d770641bb8c65", "job.node[joined].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "job.node[joined].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "job.node[joined].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000186afa42ffc71c45174cd8b17e5a938cdb4d20d29a99d653af922ed86c4cf0e4d", + "job.node[joined].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000026b8369e60be10e43dd72a0be4e8d93a992f827cd6baab33909331dcdaeea8b64", "job.node[joined].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -134,10 +134,10 @@ "job.node[scans].pipeline_hash": "semantic_v0.1:2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "job.node[scans].row[{\"subject\":\"s1\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "45c5e6aa24d25f21a15c4d1016b857d3", "job.node[scans].row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "job.node[scans].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ac375e92d49c253f99d8cdbb55487634eeb95fec0d05ef5e9515b59ac74b78d1", + "job.node[scans].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002ef2b37a1979bd734849c241456c50e96644db0ba5a7b25ecb84cee278e6686f2", "job.node[scans].row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "42d239bc04015d1aaded030d9e02850f", "job.node[scans].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "job.node[scans].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001f7fa6c7bb04912cca05f5a979e9bdecab132fd134dba8f18bf379836955c857b", + "job.node[scans].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000210b14e22145ce1d5f71e190ade4ca03ba523ad98d8749ea7beef3c15dd749c20", "job.node[scans].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec" @@ -146,24 +146,24 @@ "job.node[subjects].pipeline_hash": "semantic_v0.1:f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "job.node[subjects].row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "job.node[subjects].row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "job.node[subjects].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000137de028aa69769a5d29c9003e923af9a0b081ff4067b2eb08af4ef943e4939b6", + "job.node[subjects].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002d1c5b49de38ef6efc552c2a47b8ab59dd267f8eb55d91814edee99b1b52acc65", "job.node[subjects].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "job.node[subjects].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "job.node[subjects].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001584cc7c090776151311c6768ba98c02c0235250c6685493d3207138ef8d8b484", + "job.node[subjects].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002338ce8710f5ecb156a72c8446651adb7b0c21635d0aa92d298f09da56faed0f8", "job.node[subjects].row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "job.node[subjects].row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "job.node[subjects].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001562e4c70b60df10ed1243d0d9129beff719b867cb16e18d654f375d361e64c51", + "job.node[subjects].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000023401f1ce40c8cfdd6e4f343d7ffdcd51c1f0521f44cf6030e27e1ca0da154616", "job.node[subjects].system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" ], "job.table[golden_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes": [ - "arrow_v0.1:00000130ab270ba209296b0933d7df77e7712b75b0e1cc5f9344775e30a29c9f4cc882", - "arrow_v0.1:00000186afa42ffc71c45174cd8b17e5a938cdb4d20d29a99d653af922ed86c4cf0e4d" + "arrow_v0.2:0000026b8369e60be10e43dd72a0be4e8d93a992f827cd6baab33909331dcdaeea8b64", + "arrow_v0.2:000002c945ad08138f51298940c60333852ef5191eb16077d96105386d44840c0ccfdf" ], "job.table[golden_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:b61a42486c29af296f5c2f1d8129f20b61e023d492747e397a50756cb153aff8/pdb_v1].record_ids": [ - "arrow_v0.1:00000137cf61ccacb0c63080e74a9bc00a90d5c415a8f89f295323068742ef3088c7b9", - "arrow_v0.1:000001cb65d6834af1be3e6bc180d9ffa57241031847c38afe6decfb454e9cf6004e30" + "arrow_v0.2:000002118147ab4ef2d7e048cc8bb2deba63027eefaeee0a65d4d2d4873ba3a60a9739", + "arrow_v0.2:000002c5733660533bd081d5c22a0138c066d7e5869cb0acad1b30edf3b39a9b701e01" ], "job.table_paths": [ "golden_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1", @@ -175,12 +175,12 @@ "join_pod.join.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "e8e02084627f5e1aad83839e421538e8", "join_pod.join.row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "join_pod.join.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "join_pod.join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000130ab270ba209296b0933d7df77e7712b75b0e1cc5f9344775e30a29c9f4cc882", + "join_pod.join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c945ad08138f51298940c60333852ef5191eb16077d96105386d44840c0ccfdf", "join_pod.join.row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "42d239bc04015d1aaded030d9e02850f", "join_pod.join.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "144bb37f3c295e16919d770641bb8c65", "join_pod.join.row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "join_pod.join.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "join_pod.join.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000186afa42ffc71c45174cd8b17e5a938cdb4d20d29a99d653af922ed86c4cf0e4d", + "join_pod.join.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000026b8369e60be10e43dd72a0be4e8d93a992f827cd6baab33909331dcdaeea8b64", "join_pod.join.system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -193,12 +193,12 @@ "join_pod.pod.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "e8e02084627f5e1aad83839e421538e8", "join_pod.pod.row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "join_pod.pod.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "join_pod.pod.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:0000014a3871996b0dc2563b8f3979740245bde22aac53376bba5592849cebc035b864", + "join_pod.pod.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002df3a95f3b9eddaf078096dee1388e410637a1a4ce3dd8028296b1a274c1292f2", "join_pod.pod.row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "42d239bc04015d1aaded030d9e02850f", "join_pod.pod.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "144bb37f3c295e16919d770641bb8c65", "join_pod.pod.row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "join_pod.pod.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956:1": "subjects", - "join_pod.pod.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:0000014c738e65c53a408dfd0e017a32dd96504847f5ce08b13213b96168262a97e264", + "join_pod.pod.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000023565098d1a88ecd75dbc8ddc4900dc8304e066dc0c889114a2a41e2fe495fc30", "join_pod.pod.system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -209,13 +209,13 @@ "map_data.pipeline_hash": "semantic_v0.1:968f2ba68b526db0d6a5f0ad80f746f5cb57d2fe24588d9c374cf17d229d83e1", "map_data.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "map_data.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "map_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:0000012b54015317016ea374e1b6f4b14a7ba6454a4a320bdd1a66d56081d9b4e1e327", + "map_data.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:0000022ad1363967c26697bb148943f0b193f748f21da43e4ba5f3763197a04bf53289", "map_data.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "map_data.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "map_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001bf869237358aa3c91a042d4c6171c7eff1b6089ec739096d20313b4c32f8edfc", + "map_data.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002a61fcaac02c3fd261c22c4fab3c89182aef1912fb4a19ae1c01af9cf69b4b6b6", "map_data.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "map_data.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "map_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001e87c392dee12edd21fe5080442060fcd8d9a50627e70b90b25865ff7361bf893", + "map_data.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:000002ef21c37efbb54c93fff65d16c5222fed1364ebf8ab9a49758b064f3edd83f3a1", "map_data.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -226,12 +226,12 @@ "merge_join.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::eba863719d1f29e059571a7c159a0aeaeed1858760612e5b471011b3f7843d7e:1": "e8e02084627f5e1aad83839e421538e8", "merge_join.row[{\"subject\":\"s1\"}]._tag_source_id::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b:0": "reweigh", "merge_join.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::eba863719d1f29e059571a7c159a0aeaeed1858760612e5b471011b3f7843d7e:1": "subjects", - "merge_join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ec09b4a15ec5e06d5d5be991cb2865b48ead20dac75c66d16a4d8472de7cb4e3", + "merge_join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002654a49c9ec7661bbbb88c7695033d0082d20abafa31fcfa9112e39834de7cc5d", "merge_join.row[{\"subject\":\"s3\"}]._tag_record_id::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b:0": "e7fcc4debbb656f5b186d21f53897b41", "merge_join.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::eba863719d1f29e059571a7c159a0aeaeed1858760612e5b471011b3f7843d7e:1": "d9391c2dcea750c496e0895ad301b30d", "merge_join.row[{\"subject\":\"s3\"}]._tag_source_id::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b:0": "reweigh", "merge_join.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::eba863719d1f29e059571a7c159a0aeaeed1858760612e5b471011b3f7843d7e:1": "subjects", - "merge_join.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:0000013aeb0e14ea94b020f971939368a6b7948b5f9713b0f2cf2c1ddbe43a57ce6e53", + "merge_join.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:000002b78038f0da00d8b6e29ff96251d4d12632f5c978e5fa58dc118e53e582f90c96", "merge_join.system_tag_columns": [ "_tag_source_id::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b:0", "_tag_record_id::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b::ceea3411fcbfdce34ac8306ac3b5a8d234839abc6addf6bcd2fe28f1c974081b:0", @@ -242,13 +242,13 @@ "pod.pipeline_hash": "semantic_v0.1:f6e833fa312653a09ad67ca8f8d447630badf1ad1cfe6f298d822fbc231243d2", "pod.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "pod.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "pod.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000155d85a85806e3601d4eea978d885d32a45e3fcf6d018d37cbbcf140d077fe21a", + "pod.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:0000024da91c65d3ebacee016a9080299e31887b2aabe961f00c475443c6930693e22a", "pod.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "pod.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "pod.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:0000017db75343362e61ae6e3a5dcce30fa838e5be6c908cf1c18629a6905bedc10587", + "pod.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000021afa3e9774d8a28522ff03f4981ecb0b290e3cefc0f602fe11edf4d3430d103e", "pod.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "pod.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "pod.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001a3b408003eb5481eec125ba73ba0190c72fee13f147953cc4e850afbafc36c1c", + "pod.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:000002c2a1324b1334c4c49e87c175c8ad33ea00a325fea75e441edb4108c611994179", "pod.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -257,10 +257,10 @@ "polars_filter.pipeline_hash": "semantic_v0.1:c3495e32f466d812eefc8a1af06ffb3a94e4de51a0bee266c03fc1f1539f4fb3", "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "b47fb712e72a55f4920c3fc7db2f75f5", "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ee27bd12d66af74db02ecec14119254c263796aa8551e3d3249dc0025f22be17", + "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c734791b7452bb2affc60031491e7541bb53fb971cf3ef8b9af919eb0bf1f56e", "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "37d45a37e6255cd78bb62c4c9d047ca5", "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "polars_filter.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "polars_filter.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -269,10 +269,10 @@ "polars_filter_expr.expr.pipeline_hash": "semantic_v0.1:365702de4f28d94a1d119f12fd83b6b15b09ad7ed9aaf3e412ec9ca2c8392fcc", "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "b47fb712e72a55f4920c3fc7db2f75f5", "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ee27bd12d66af74db02ecec14119254c263796aa8551e3d3249dc0025f22be17", + "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c734791b7452bb2affc60031491e7541bb53fb971cf3ef8b9af919eb0bf1f56e", "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "37d45a37e6255cd78bb62c4c9d047ca5", "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "polars_filter_expr.expr.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "polars_filter_expr.expr.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -281,10 +281,10 @@ "polars_filter_expr.mixed.pipeline_hash": "semantic_v0.1:145c27c659a765361d8fbe7b8831928621e8aac954dbda45bbcbd0251f36e27c", "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "b47fb712e72a55f4920c3fc7db2f75f5", "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ee27bd12d66af74db02ecec14119254c263796aa8551e3d3249dc0025f22be17", + "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c734791b7452bb2affc60031491e7541bb53fb971cf3ef8b9af919eb0bf1f56e", "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "37d45a37e6255cd78bb62c4c9d047ca5", "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "polars_filter_expr.mixed.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "polars_filter_expr.mixed.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -295,7 +295,7 @@ "polars_filter_expr_job.node[adjust].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::53252e3314d5992ea18c9a7886b9673a191d0933c83da9b1d8926e09ab8962fd:1": "144bb37f3c295e16919d770641bb8c65", "polars_filter_expr_job.node[adjust].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "polars_filter_expr_job.node[adjust].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::53252e3314d5992ea18c9a7886b9673a191d0933c83da9b1d8926e09ab8962fd:1": "subjects", - "polars_filter_expr_job.node[adjust].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:0000014c738e65c53a408dfd0e017a32dd96504847f5ce08b13213b96168262a97e264", + "polars_filter_expr_job.node[adjust].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000023565098d1a88ecd75dbc8ddc4900dc8304e066dc0c889114a2a41e2fe495fc30", "polars_filter_expr_job.node[adjust].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -308,7 +308,7 @@ "polars_filter_expr_job.node[joined].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::53252e3314d5992ea18c9a7886b9673a191d0933c83da9b1d8926e09ab8962fd:1": "144bb37f3c295e16919d770641bb8c65", "polars_filter_expr_job.node[joined].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0": "scans", "polars_filter_expr_job.node[joined].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956::53252e3314d5992ea18c9a7886b9673a191d0933c83da9b1d8926e09ab8962fd:1": "subjects", - "polars_filter_expr_job.node[joined].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000186afa42ffc71c45174cd8b17e5a938cdb4d20d29a99d653af922ed86c4cf0e4d", + "polars_filter_expr_job.node[joined].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000026b8369e60be10e43dd72a0be4e8d93a992f827cd6baab33909331dcdaeea8b64", "polars_filter_expr_job.node[joined].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec:0", @@ -319,10 +319,10 @@ "polars_filter_expr_job.node[older].pipeline_hash": "semantic_v0.1:53252e3314d5992ea18c9a7886b9673a191d0933c83da9b1d8926e09ab8962fd", "polars_filter_expr_job.node[older].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "polars_filter_expr_job.node[older].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "polars_filter_expr_job.node[older].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001584cc7c090776151311c6768ba98c02c0235250c6685493d3207138ef8d8b484", + "polars_filter_expr_job.node[older].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002338ce8710f5ecb156a72c8446651adb7b0c21635d0aa92d298f09da56faed0f8", "polars_filter_expr_job.node[older].row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "polars_filter_expr_job.node[older].row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "polars_filter_expr_job.node[older].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001562e4c70b60df10ed1243d0d9129beff719b867cb16e18d654f375d361e64c51", + "polars_filter_expr_job.node[older].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000023401f1ce40c8cfdd6e4f343d7ffdcd51c1f0521f44cf6030e27e1ca0da154616", "polars_filter_expr_job.node[older].system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -331,10 +331,10 @@ "polars_filter_expr_job.node[scans].pipeline_hash": "semantic_v0.1:2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "polars_filter_expr_job.node[scans].row[{\"subject\":\"s1\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "45c5e6aa24d25f21a15c4d1016b857d3", "polars_filter_expr_job.node[scans].row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "polars_filter_expr_job.node[scans].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ac375e92d49c253f99d8cdbb55487634eeb95fec0d05ef5e9515b59ac74b78d1", + "polars_filter_expr_job.node[scans].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002ef2b37a1979bd734849c241456c50e96644db0ba5a7b25ecb84cee278e6686f2", "polars_filter_expr_job.node[scans].row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "42d239bc04015d1aaded030d9e02850f", "polars_filter_expr_job.node[scans].row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "polars_filter_expr_job.node[scans].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001f7fa6c7bb04912cca05f5a979e9bdecab132fd134dba8f18bf379836955c857b", + "polars_filter_expr_job.node[scans].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000210b14e22145ce1d5f71e190ade4ca03ba523ad98d8749ea7beef3c15dd749c20", "polars_filter_expr_job.node[scans].system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec" @@ -343,22 +343,22 @@ "polars_filter_expr_job.node[subjects].pipeline_hash": "semantic_v0.1:f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000137de028aa69769a5d29c9003e923af9a0b081ff4067b2eb08af4ef943e4939b6", + "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002d1c5b49de38ef6efc552c2a47b8ab59dd267f8eb55d91814edee99b1b52acc65", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001584cc7c090776151311c6768ba98c02c0235250c6685493d3207138ef8d8b484", + "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002338ce8710f5ecb156a72c8446651adb7b0c21635d0aa92d298f09da56faed0f8", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001562e4c70b60df10ed1243d0d9129beff719b867cb16e18d654f375d361e64c51", + "polars_filter_expr_job.node[subjects].row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000023401f1ce40c8cfdd6e4f343d7ffdcd51c1f0521f44cf6030e27e1ca0da154616", "polars_filter_expr_job.node[subjects].system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" ], "polars_filter_expr_job.table[golden_filter_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes": [ - "arrow_v0.1:00000186afa42ffc71c45174cd8b17e5a938cdb4d20d29a99d653af922ed86c4cf0e4d" + "arrow_v0.2:0000026b8369e60be10e43dd72a0be4e8d93a992f827cd6baab33909331dcdaeea8b64" ], "polars_filter_expr_job.table[golden_filter_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:1bf083d50625083700e69f699b90f9f01c9c1e93d56e1a56feb4ad1506005e93/pdb_v1].record_ids": [ - "arrow_v0.1:000001ea2218fa4e752319925270feb64dd91210fd24cc94e1fe0faed39c8345774c8c" + "arrow_v0.2:0000022ad2e7cf689d57072ff3ed2627943d06158df2dfff7c59a978eefcfc4dd4300c" ], "polars_filter_expr_job.table_paths": [ "golden_filter_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1", @@ -368,10 +368,10 @@ "semi_join.pipeline_hash": "semantic_v0.1:822ca17d6b35230006b80ecd32d046192ed22378cd995365b9b0ac6e49f7224a", "semi_join.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "semi_join.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "semi_join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000137de028aa69769a5d29c9003e923af9a0b081ff4067b2eb08af4ef943e4939b6", + "semi_join.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002d1c5b49de38ef6efc552c2a47b8ab59dd267f8eb55d91814edee99b1b52acc65", "semi_join.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "semi_join.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "semi_join.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001584cc7c090776151311c6768ba98c02c0235250c6685493d3207138ef8d8b484", + "semi_join.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002338ce8710f5ecb156a72c8446651adb7b0c21635d0aa92d298f09da56faed0f8", "semi_join.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" @@ -380,13 +380,13 @@ "sources.cohorts.pipeline_hash": "semantic_v0.1:13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "b47fb712e72a55f4920c3fc7db2f75f5", "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s1\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ee27bd12d66af74db02ecec14119254c263796aa8551e3d3249dc0025f22be17", + "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002c734791b7452bb2affc60031491e7541bb53fb971cf3ef8b9af919eb0bf1f56e", "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "37d45a37e6255cd78bb62c4c9d047ca5", "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s2\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.1:00000133552dc58a31ff558a0bf035031e2228ec7981bb85c39ab22ae0aaad86efdb5f", + "sources.cohorts.row[{\"cohort\":\"a\",\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000252e4f32a133cd6a8813f816e08b6bde1cdb467baa95d026fc41c5672b6b98110", "sources.cohorts.row[{\"cohort\":\"b\",\"subject\":\"s3\"}]._tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "d0cc03b6594d51788075066f5c81deb0", "sources.cohorts.row[{\"cohort\":\"b\",\"subject\":\"s3\"}]._tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4": "cohorts", - "sources.cohorts.row[{\"cohort\":\"b\",\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001966c5e58c797ab3329316dffd36c3dcf09a3e0a400fc4511114e1e99f10939ea", + "sources.cohorts.row[{\"cohort\":\"b\",\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000025f33a2f5bf3e956edfb7ca628b6871f52b7bbf788d5a3774ddffe300d7d4aff5", "sources.cohorts.system_tag_columns": [ "_tag_source_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4", "_tag_record_id::13d8efe7433022d8f328bbd9ea2cf66397ef44167932c3b08fb43153bc1be1d4" @@ -395,10 +395,10 @@ "sources.scans.pipeline_hash": "semantic_v0.1:2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "sources.scans.row[{\"subject\":\"s1\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "45c5e6aa24d25f21a15c4d1016b857d3", "sources.scans.row[{\"subject\":\"s1\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "sources.scans.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:000001ac375e92d49c253f99d8cdbb55487634eeb95fec0d05ef5e9515b59ac74b78d1", + "sources.scans.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002ef2b37a1979bd734849c241456c50e96644db0ba5a7b25ecb84cee278e6686f2", "sources.scans.row[{\"subject\":\"s2\"}]._tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "42d239bc04015d1aaded030d9e02850f", "sources.scans.row[{\"subject\":\"s2\"}]._tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec": "scans", - "sources.scans.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001f7fa6c7bb04912cca05f5a979e9bdecab132fd134dba8f18bf379836955c857b", + "sources.scans.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:00000210b14e22145ce1d5f71e190ade4ca03ba523ad98d8749ea7beef3c15dd749c20", "sources.scans.system_tag_columns": [ "_tag_source_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec", "_tag_record_id::2f6020c8bd0f792547e5060b0fcbf2f2629eaaf14ffb4be67a050bd30bb414ec" @@ -407,13 +407,13 @@ "sources.subjects.pipeline_hash": "semantic_v0.1:f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "sources.subjects.row[{\"subject\":\"s1\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "e8e02084627f5e1aad83839e421538e8", "sources.subjects.row[{\"subject\":\"s1\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "sources.subjects.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000137de028aa69769a5d29c9003e923af9a0b081ff4067b2eb08af4ef943e4939b6", + "sources.subjects.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:000002d1c5b49de38ef6efc552c2a47b8ab59dd267f8eb55d91814edee99b1b52acc65", "sources.subjects.row[{\"subject\":\"s2\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "144bb37f3c295e16919d770641bb8c65", "sources.subjects.row[{\"subject\":\"s2\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "sources.subjects.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001584cc7c090776151311c6768ba98c02c0235250c6685493d3207138ef8d8b484", + "sources.subjects.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:000002338ce8710f5ecb156a72c8446651adb7b0c21635d0aa92d298f09da56faed0f8", "sources.subjects.row[{\"subject\":\"s3\"}]._tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "d9391c2dcea750c496e0895ad301b30d", "sources.subjects.row[{\"subject\":\"s3\"}]._tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956": "subjects", - "sources.subjects.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:000001562e4c70b60df10ed1243d0d9129beff719b867cb16e18d654f375d361e64c51", + "sources.subjects.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000023401f1ce40c8cfdd6e4f343d7ffdcd51c1f0521f44cf6030e27e1ca0da154616", "sources.subjects.system_tag_columns": [ "_tag_source_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956", "_tag_record_id::f004fc93e6b885270cb7be1a7e8e368a81c824e524081a3bc1bd3a23cb102956" diff --git a/tests/test_hashing/golden/data/schema.json b/tests/test_hashing/golden/data/schema.json index b0fba5589..007896af3 100644 --- a/tests/test_hashing/golden/data/schema.json +++ b/tests/test_hashing/golden/data/schema.json @@ -46,58 +46,58 @@ "source_schema_hash.record_id_column": "_tag_record_id::a3e81ce4fbd701a0fb97cae134f236953f7f66a8e7b7f4741339b45fe9f485ba", "source_schema_hash.schema_hash": "a3e81ce4fbd701a0fb97cae134f236953f7f66a8e7b7f4741339b45fe9f485ba", "source_schema_hash.source_id_column": "_tag_source_id::a3e81ce4fbd701a0fb97cae134f236953f7f66a8e7b7f4741339b45fe9f485ba", - "source_stream_builder.default_source_id.content_hash": "semantic_v0.1:7eafdaffcb2ac49d760748516b005f13cdab23008421690ade60c691ed7199ea", + "source_stream_builder.default_source_id.content_hash": "semantic_v0.1:167c1d63d67f2fa1998069d2470f6051338e187b615e6c85f79e636dd588b2e2", "source_stream_builder.default_source_id.pipeline_hash": "semantic_v0.1:f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", - "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "71c09013eb5f5f5582ad41da30acbcf7", - "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", - "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000128b75b53ced7f9752b31affee90b8cd666b24b0802f3de19dff3f742d6636bb8", - "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "e2960b661cf05e14874582345bdde8e8", - "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", - "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001a39a4d7b517d56c4c323c744f782e70521e4d759af0f4ff4aedc806c1a191297", - "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "5d4deb7fe5675941942e2b590adaf240", - "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", - "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:0000010364967ccec6827a7b2dd8ef76352ce8aed48ccd414bae0ddf1e5b46b0dfc40c", + "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "2d25e22e5ed45f2ebcd9ba0e574ce4ab", + "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", + "source_stream_builder.default_source_id.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:00000289f61a7619845a37a72864e1da3c74a3aff25d66aa6c1d4b6d3874e84ae12e85", + "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "7a85d0e335bf539d990068a86fb4409e", + "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", + "source_stream_builder.default_source_id.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000029f0b00a74de3d3f453d7f4dec0ddf7b3dfa05a6cbedec40a31a038f7d764adcb", + "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "174dcf5eb0c05b08967518af5d71cd7b", + "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", + "source_stream_builder.default_source_id.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000026b8c4f7e93665f79d39dce50db9d7f494ee1e49a34cd03f226b604f2a6e6427f", "source_stream_builder.default_source_id.schema_hash": "f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", - "source_stream_builder.default_source_id.source_id": "000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", + "source_stream_builder.default_source_id.source_id": "000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", "source_stream_builder.default_source_id.system_tag_columns": [ "_tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "_tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367" ], - "source_stream_builder.default_source_id.table_hash": "arrow_v0.1:000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", - "source_stream_builder.named_source_id.content_hash": "semantic_v0.1:f424ce4643b9e4dd37573e30c019354c66d7361265828482115a6a18cb74f7c7", + "source_stream_builder.default_source_id.table_hash": "arrow_v0.2:000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", + "source_stream_builder.named_source_id.content_hash": "semantic_v0.1:69b4b3098babd295b560941edfc51ecc92cb1a49d68c7614a3172449f09cab3e", "source_stream_builder.named_source_id.pipeline_hash": "semantic_v0.1:f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "source_stream_builder.named_source_id.row[{\"subject\":\"s1\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "3657ba768edb5ccda83740fcb80aa959", "source_stream_builder.named_source_id.row[{\"subject\":\"s1\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.named_source_id.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000128b75b53ced7f9752b31affee90b8cd666b24b0802f3de19dff3f742d6636bb8", + "source_stream_builder.named_source_id.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:00000289f61a7619845a37a72864e1da3c74a3aff25d66aa6c1d4b6d3874e84ae12e85", "source_stream_builder.named_source_id.row[{\"subject\":\"s2\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "bce9b135c1675095ac3a1e7099490014", "source_stream_builder.named_source_id.row[{\"subject\":\"s2\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.named_source_id.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001a39a4d7b517d56c4c323c744f782e70521e4d759af0f4ff4aedc806c1a191297", + "source_stream_builder.named_source_id.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000029f0b00a74de3d3f453d7f4dec0ddf7b3dfa05a6cbedec40a31a038f7d764adcb", "source_stream_builder.named_source_id.row[{\"subject\":\"s3\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "9ac1e81621a25135b681f367e7a3b98a", "source_stream_builder.named_source_id.row[{\"subject\":\"s3\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.named_source_id.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:0000010364967ccec6827a7b2dd8ef76352ce8aed48ccd414bae0ddf1e5b46b0dfc40c", + "source_stream_builder.named_source_id.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000026b8c4f7e93665f79d39dce50db9d7f494ee1e49a34cd03f226b604f2a6e6427f", "source_stream_builder.named_source_id.schema_hash": "f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "source_stream_builder.named_source_id.source_id": "golden_source", "source_stream_builder.named_source_id.system_tag_columns": [ "_tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "_tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367" ], - "source_stream_builder.named_source_id.table_hash": "arrow_v0.1:000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a", - "source_stream_builder.record_id_column.content_hash": "semantic_v0.1:6983755521b8c4ca1771dab9fd20de2d92ce24a15438cca78378fb098d57d87d", + "source_stream_builder.named_source_id.table_hash": "arrow_v0.2:000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3", + "source_stream_builder.record_id_column.content_hash": "semantic_v0.1:0bbf3af4ed09b723aa345c2de91cbf5845771a5860d319c775388aae5751d482", "source_stream_builder.record_id_column.pipeline_hash": "semantic_v0.1:f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "source_stream_builder.record_id_column.row[{\"subject\":\"s1\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "2c42f741f6ab5ba29ec12fddfed561f8", "source_stream_builder.record_id_column.row[{\"subject\":\"s1\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.record_id_column.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.1:00000128b75b53ced7f9752b31affee90b8cd666b24b0802f3de19dff3f742d6636bb8", + "source_stream_builder.record_id_column.row[{\"subject\":\"s1\"}].data_hash": "arrow_v0.2:00000289f61a7619845a37a72864e1da3c74a3aff25d66aa6c1d4b6d3874e84ae12e85", "source_stream_builder.record_id_column.row[{\"subject\":\"s2\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "8df4a32981b850a9915c8a5715376d1b", "source_stream_builder.record_id_column.row[{\"subject\":\"s2\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.record_id_column.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.1:000001a39a4d7b517d56c4c323c744f782e70521e4d759af0f4ff4aedc806c1a191297", + "source_stream_builder.record_id_column.row[{\"subject\":\"s2\"}].data_hash": "arrow_v0.2:0000029f0b00a74de3d3f453d7f4dec0ddf7b3dfa05a6cbedec40a31a038f7d764adcb", "source_stream_builder.record_id_column.row[{\"subject\":\"s3\"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "f015aeed0c0e5c14a3280d0f3e828564", "source_stream_builder.record_id_column.row[{\"subject\":\"s3\"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367": "golden_source", - "source_stream_builder.record_id_column.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.1:0000010364967ccec6827a7b2dd8ef76352ce8aed48ccd414bae0ddf1e5b46b0dfc40c", + "source_stream_builder.record_id_column.row[{\"subject\":\"s3\"}].data_hash": "arrow_v0.2:0000026b8c4f7e93665f79d39dce50db9d7f494ee1e49a34cd03f226b604f2a6e6427f", "source_stream_builder.record_id_column.schema_hash": "f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "source_stream_builder.record_id_column.source_id": "golden_source", "source_stream_builder.record_id_column.system_tag_columns": [ "_tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367", "_tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367" ], - "source_stream_builder.record_id_column.table_hash": "arrow_v0.1:000001630ee60cef9dfe3cce7a6782fa30ab89d235ba2de5934430582a04969396ab0a" + "source_stream_builder.record_id_column.table_hash": "arrow_v0.2:000002e12eafa1dc72203ca9da1ca5d064031683cf089f47bad40744ecd2c923d0fde3" } diff --git a/tests/test_hashing/golden/data/semantic.json b/tests/test_hashing/golden/data/semantic.json index 4180e3648..dc68a6bff 100644 --- a/tests/test_hashing/golden/data/semantic.json +++ b/tests/test_hashing/golden/data/semantic.json @@ -32,7 +32,7 @@ "value.bytearray.binary": "semantic_v0.1:d891b13714f4aec76114361fe22ebfb829f952b88c9ad49c4afa16896612ee75", "value.bytes.binary": "semantic_v0.1:d891b13714f4aec76114361fe22ebfb829f952b88c9ad49c4afa16896612ee75", "value.bytes.empty": "semantic_v0.1:12ae32cb1ec02d01eda3581b127c1fee3b0dc53572ed6baf239721a03d82e126", - "value.dataframe.basic": "arrow_v0.1:000001b0e17dcec7cd73f09d83119c75f39a9638089533cbf9cc478de4a79e3082cdb9", + "value.dataframe.basic": "arrow_v0.2:000002bac72e84776fc6d62d3928f88ec11672873c07bca9de0cc5a7689cd9675800dc", "value.dict.ab": "semantic_v0.1:43258cff783fe7036d8a43033f830adfc60ec037382473548ac742b888292777", "value.dict.ba": "semantic_v0.1:43258cff783fe7036d8a43033f830adfc60ec037382473548ac742b888292777", "value.dict.empty": "semantic_v0.1:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", diff --git a/tests/test_hashing/golden/generate_golden_hashes.py b/tests/test_hashing/golden/generate_golden_hashes.py index 0aa7446a5..9e86f5c24 100644 --- a/tests/test_hashing/golden/generate_golden_hashes.py +++ b/tests/test_hashing/golden/generate_golden_hashes.py @@ -29,9 +29,10 @@ process, whatever ``PYTHONHASHSEED`` is. A value that differs by interpreter is a defect to fix, never something to pin per version. -Known gaps (ITL-848), not pinned until fixed: ``hash_schema`` on a schema with an extension -type, ``null``-typed columns, the ``arrow.uuid`` canonical extension type, and map and -fixed-size-list values. +Not pinned, because starfix rejects their data (ITL-848): ``null``-typed and map columns. +Since starfix 0.5 (hash format 2, label ``arrow_v0.2``, ITL-910), ``hash_schema`` on extension +columns, the canonical ``arrow.uuid`` and ``arrow.fixed_shape_tensor`` types, and +fixed-size-list values are hashable and pinned. Comparing releases ------------------ From a5ceb0ab7cdcc91da9168d27bf0b9cc45564a26d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:44:46 -0800 Subject: [PATCH 023/147] test: re-pin format-1 digests and record-id order in starfix format 2 (ITL-910) Co-Authored-By: Claude Opus 5.5 --- .../operators/test_async_sync_equivalence.py | 11 +++++---- tests/test_core/operators/test_group_by.py | 5 ++-- .../operators/test_materialize_source_info.py | 15 ++++++++---- tests/test_hashing/test_pandas_handlers.py | 4 ++-- .../test_hashing/test_starfix_arrow_hasher.py | 24 +++++++++---------- .../test_struct_layout_identity.py | 21 ++++++++-------- 6 files changed, 44 insertions(+), 36 deletions(-) diff --git a/tests/test_core/operators/test_async_sync_equivalence.py b/tests/test_core/operators/test_async_sync_equivalence.py index 916509dd0..5bec5adda 100644 --- a/tests/test_core/operators/test_async_sync_equivalence.py +++ b/tests/test_core/operators/test_async_sync_equivalence.py @@ -546,15 +546,16 @@ def _narrow_source(column: str, source_id: str) -> ArrowTableSource: # Recorded from the sync path on main (ab544982) before the ITL-837 fix. The sync -# path must not change; the async path must now produce the same values. +# path must not change; the async path must now produce the same values. Row hashes +# re-recorded in starfix hash format 2 (arrow_v0.2, ITL-910); the stream hashes did not move. PINNED_ROW_HASHES = { 1: ( - "arrow_v0.1:000001515f1eb3674534e0226bbe2aada98adb5137a0f8829ee815fdb994809ac001fa", - "arrow_v0.1:00000119d2a6630f08dd30ef49c4a298862b7d129d2169d84963ea4f801eb8248e99e4", + "arrow_v0.2:0000027e1a4693c531d4411fafa68a1ac623a0ed3f04a6402de0fb8f040f0aba717ae8", + "arrow_v0.2:0000025c4232f42a6012b104827b2c4731e143af6044e010e9a37ae5ffa82fe3ae7153", ), 2: ( - "arrow_v0.1:000001ea3d721d986f7382996ef41ded3d1a43baf41a543fde471b20c71190b45dbf98", - "arrow_v0.1:000001fa7bf20e72872ffc6860a21d75ad93c38bd54753fa9eb325502acc783b693d43", + "arrow_v0.2:000002e3b4cd9f773e0263ba270d251b90058702fa282e129feebdbc428d832080752e", + "arrow_v0.2:00000218de95b54e8987999a339fe1726957da66219b529c4699be853334833e71f60e", ), } PINNED_STREAM_CONTENT_HASH = ( diff --git a/tests/test_core/operators/test_group_by.py b/tests/test_core/operators/test_group_by.py index 5519c94ef..bdf1ec73d 100644 --- a/tests/test_core/operators/test_group_by.py +++ b/tests/test_core/operators/test_group_by.py @@ -122,13 +122,14 @@ def test_falls_back_to_record_id_when_key_covers_all_tags(self): record_id is a uuid5 of source content, so the order it imposes is not human-predictable but IS stable across processes. Asserted exactly -- sorting the result before comparing would make this blind to the very - ordering it exists to pin down. + ordering it exists to pin down. The default source_id is an Arrow table + hash, so this order moved with starfix hash format 2 (ITL-910). """ table = pa.table({"subject": ["G", "G"], "path": ["b", "a"]}) src = ArrowTableSource(table, tag_columns=["subject"], infer_nullable=True) out = GroupBy(by=["subject"]).process(src) assert len(out.as_table()) == 1 - assert out.as_table().column("path").to_pylist()[0] == ["a", "b"] + assert out.as_table().column("path").to_pylist()[0] == ["b", "a"] def test_record_id_breaks_ties_between_duplicate_tags(self): """Duplicate tag tuples must not fall back to raw emission order.""" diff --git a/tests/test_core/operators/test_materialize_source_info.py b/tests/test_core/operators/test_materialize_source_info.py index 2ed6791b2..c9acb337a 100644 --- a/tests/test_core/operators/test_materialize_source_info.py +++ b/tests/test_core/operators/test_materialize_source_info.py @@ -564,6 +564,9 @@ def _record_ids(node: OperatorJobNode) -> dict[str, bytes]: class TestIdentityPinned: """Values recorded on ``main`` at ab544982, before the fix. + Re-recorded under starfix hash format 2 (``arrow_v0.2``, ITL-910): the node content + hash and the record hashes are Arrow-derived; the pipeline hash is not. + The node's identity, the output data and the row-0 record hash (whose source-info token was already right) are unchanged; only rows whose stored source-info token was wrong get a new record hash. @@ -584,7 +587,7 @@ def test_operator_node_identity_and_row_zero_record_hash(self): "semantic_v0.1:1f533cfa7293bad96debd5cda622e04bc39d72970ce6827dcd6e890a7428823c" ) assert node.content_hash().to_string() == ( - "semantic_v0.1:6dc7f9c42cf4d0351d8a577bb2c72d69aadbca97e1c31d0cfc62d6a8087f3038" + "semantic_v0.1:36a4b61a5ed690ffa95ff31ba0c19a43e31b5b2ac51e1e5848d4b184e4b9ccf7" ) ids = _record_ids(node) assert ids["a"] == ROW_ZERO_RECORD_ID @@ -605,16 +608,18 @@ def test_operator_node_identity_and_row_zero_record_hash(self): } -# ``_record_hash`` of rows a, b, c stored by the pinned node on ab544982 (pre-fix). +# ``_record_hash`` of rows a, b, c stored by the pinned node before the fix (ab544982). # Row a's source-info token was already right, so the fix leaves its record hash. +# Re-recorded in starfix hash format 2 (ITL-910) by reproducing the pre-fix broadcast +# with ``_broadcast_row_zero``. ROW_ZERO_RECORD_ID = ( - b"000001688856047a9f37112172e612834209fb07f2cf4be5db30f0fa3b7059db85905e" + b"000002016f74f387a07874a088c8c44f1b85ff70a395680158b6a87520085c95439ec4" ) PRE_FIX_RECORD_IDS = frozenset( { ROW_ZERO_RECORD_ID, - b"0000016f23c92ad43ef01bb80e438f16da66d02354998351430b3237d938a340943039", - b"00000138599ee77c1f3dc62bf02a117150b32db27c6582db885cda2d28b7cef3964657", + b"000002de01a6a62bba3ec3ec7cb9c64a8bf4a7da8f0ab959fca01dcfd2d162f91e9cf6", + b"00000270703c99569d9ffb213ac4a8825f880e6ffe25eb1b30019da118752a73996acc", } ) diff --git a/tests/test_hashing/test_pandas_handlers.py b/tests/test_hashing/test_pandas_handlers.py index f6af7faf0..cf6417236 100644 --- a/tests/test_hashing/test_pandas_handlers.py +++ b/tests/test_hashing/test_pandas_handlers.py @@ -24,7 +24,7 @@ def test_returns_content_hash(self, arrow_hasher): df = pd.DataFrame({"a": [1.0, 2.0], "b": [3, 4]}) result = PandasDataFrameHandler(arrow_hasher).handle(df, hasher=None) assert isinstance(result, ContentHash) - assert result.method == "arrow_v0.1" + assert result.method == "arrow_v0.2" def test_same_content_same_hash(self, arrow_hasher): from orcapod.hashing.semantic_hashing.builtin_handlers import ( @@ -95,7 +95,7 @@ def test_returns_content_hash(self, arrow_hasher): s = pd.Series([1.0, 2.0, 3.0], name="x") result = PandasSeriesHandler(arrow_hasher).handle(s, hasher=None) assert isinstance(result, ContentHash) - assert result.method == "arrow_v0.1" + assert result.method == "arrow_v0.2" def test_same_content_same_hash(self, arrow_hasher): from orcapod.hashing.semantic_hashing.builtin_handlers import ( diff --git a/tests/test_hashing/test_starfix_arrow_hasher.py b/tests/test_hashing/test_starfix_arrow_hasher.py index 330a7c85b..8b046816e 100644 --- a/tests/test_hashing/test_starfix_arrow_hasher.py +++ b/tests/test_hashing/test_starfix_arrow_hasher.py @@ -38,7 +38,7 @@ # Helpers # --------------------------------------------------------------------------- -HASHER_ID = "arrow_v0.1" +HASHER_ID = "arrow_v0.2" _DIGEST_LEN = 35 # 3 version bytes + 32 SHA-256 bytes @@ -107,7 +107,7 @@ def test_golden_value(self): h = _make_hasher().hash_schema(schema) assert ( h.digest.hex() - == "000001d676ef0263a8e0e7500b1c97033993dbe445172ca0f9e7577b3994bfa6224b4c" + == "000002e9eb1c825a8a882b565ef45a2ec832c10e31ad067736d2a6cb8a9403ff91c33f" ), ( f"Schema golden digest changed — was the starfix algorithm updated? " f"Got: {h.digest.hex()}" @@ -126,10 +126,10 @@ def test_nullability_golden_values(self): s_nullable = pa.schema([pa.field("x", pa.int32(), nullable=True)]) s_non_nullable = pa.schema([pa.field("x", pa.int32(), nullable=False)]) assert _make_hasher().hash_schema(s_nullable).digest.hex() == ( - "000001b8005339e69df64cda60f9ff9c98caa264092a7b666c3f7f85a2bfed20bae3db" + "000002fa30e1a356fea8d3a5602a6fa1d49a43349a840f65ead55463f5ead55ea64e9a" ) assert _make_hasher().hash_schema(s_non_nullable).digest.hex() == ( - "00000179353568a71430411e8108ee02d425800f6d5054d9b2baa871cad90f3e06422a" + "0000022571a7dcd462c26761f5b5353ccbd0c8d80655b462c3bb622560ec0d590a5662" ) @@ -254,7 +254,7 @@ def test_golden_value_table(self): h = _make_hasher().hash_table(table) assert ( h.digest.hex() - == "0000010cd7fe5462420b84f03a06925374e528817a3b72319e679a17e7380964878791" + == "00000277215cd88b3019240dec61d8c4bed8bf6fd05e48d5684e1e7d3acc957f54102c" ), ( f"Table golden digest changed — was the starfix algorithm updated? " f"Got: {h.digest.hex()}" @@ -306,9 +306,9 @@ def test_hasher_id_matches_current_constant(self): hasher = get_versioned_semantic_arrow_hasher() assert hasher.hasher_id == _CURRENT_ARROW_HASHER_ID - def test_current_hasher_id_is_arrow_v0_1(self): + def test_current_hasher_id_is_arrow_v0_2(self): """Pin the current version constant to detect accidental version drifts.""" - assert _CURRENT_ARROW_HASHER_ID == "arrow_v0.1" + assert _CURRENT_ARROW_HASHER_ID == "arrow_v0.2" # --------------------------------------------------------------------------- @@ -327,7 +327,7 @@ def test_v01_context_hasher_id(self): from orcapod.contexts import resolve_context ctx = resolve_context("v0.1") - assert ctx.arrow_hasher.hasher_id == "arrow_v0.1" + assert ctx.arrow_hasher.hasher_id == "arrow_v0.2" def test_context_hash_table_functional(self): from orcapod.contexts import resolve_context @@ -350,7 +350,7 @@ def test_context_hash_table_functional(self): class TestHashSchemaExtensionAware: def test_extension_free_golden_value_unchanged(self): """Stability invariant: extension-free schema hash is byte-identical to - the pre-v0.3.0 golden value (include_metadata=False is the default).""" + the starfix format-2 (arrow_v0.2) golden value (include_metadata=False is the default).""" schema = pa.schema( [ pa.field("id", pa.int32(), nullable=False), @@ -360,7 +360,7 @@ def test_extension_free_golden_value_unchanged(self): h = _make_hasher().hash_schema(schema) assert ( h.digest.hex() - == "000001d676ef0263a8e0e7500b1c97033993dbe445172ca0f9e7577b3994bfa6224b4c" + == "000002e9eb1c825a8a882b565ef45a2ec832c10e31ad067736d2a6cb8a9403ff91c33f" ), ( f"Stability invariant broken — extension-free golden digest changed. Got: {h.digest.hex()}" ) @@ -442,7 +442,7 @@ def test_unrelated_metadata_does_not_affect_hash(self): class TestHashTableExtensionAware: def test_extension_free_table_golden_value_unchanged(self): """Stability invariant: extension-free table hash is byte-identical to - the pre-v0.3.0 golden value.""" + the starfix format-2 (arrow_v0.2) golden value.""" table = pa.table( { "id": pa.array([1, 2, 3], type=pa.int32()), @@ -453,7 +453,7 @@ def test_extension_free_table_golden_value_unchanged(self): h = _make_hasher().hash_table(table) assert ( h.digest.hex() - == "0000010cd7fe5462420b84f03a06925374e528817a3b72319e679a17e7380964878791" + == "00000277215cd88b3019240dec61d8c4bed8bf6fd05e48d5684e1e7d3acc957f54102c" ), ( f"Stability invariant broken — extension-free table golden digest changed. Got: {h.digest.hex()}" ) diff --git a/tests/test_semantic_types/test_struct_layout_identity.py b/tests/test_semantic_types/test_struct_layout_identity.py index 45fc9133f..1dcf7e3be 100644 --- a/tests/test_semantic_types/test_struct_layout_identity.py +++ b/tests/test_semantic_types/test_struct_layout_identity.py @@ -50,38 +50,39 @@ def _tables() -> dict[str, pa.Table]: _FLAT_HASHES: dict[str, object] = { - "stream_content": "90dab8a6e5b7b03c83ef20fbfdd4040410f05f9e20d8ca2f64d6f2ef374b8e99", + "stream_content": "1362a2951a4ce959504f5a33d325766ef6c35e021d166be54b2f358b8e47e97d", "stream_pipeline": "34209011cc4048b6425d8b01c9c11e20dfc27a6f45b40a9948389bced49a76e4", "source_content": "7263fb2ff4346ac025631fd34c7204971478d83165084f9a748e61acc5e9f919", "source_pipeline": "34209011cc4048b6425d8b01c9c11e20dfc27a6f45b40a9948389bced49a76e4", "data_rows": [ - "0000010aa7ec467ed05cf6156b4c537186edb548c2e0e61f09d66dde96d9f644aee318", - "000001df354be2d86566d498a075294b6cbd403e4566dac064c6306787245d7ecf8004", + "000002a3b465dfc79ea3f32b39f55417f4c2372d7a3cc7ad63aef00aa9108b21b76b9c", + "000002e540dc3b59a01d14edc26eebff96453fb1de9018c351676cef8d1f67ee3346b2", ], } -# Hashes on main (365a5f74), before the converter change. +# Hashes on main (365a5f74), before the converter change; the stream content and per-row +# data hashes re-recorded in starfix hash format 2 (arrow_v0.2, ITL-910). MAIN_HASHES: dict[str, dict[str, object]] = { "small": _FLAT_HASHES, "large": _FLAT_HASHES, "nested": { - "stream_content": "9e7d62a473a0e47a8ca74ad950245a3256248a7ed8994645198d82e4d6b33902", + "stream_content": "d88d09bb6c288a29cd0edbf62826bf8d9afb6b2c6b9d320aa248c4496b7d63f3", "stream_pipeline": "f8745264f2422bf76eaec06120aa00b510cafa0ee3abdfac3d5396a614afb3a1", "source_content": "2ee725a65a6050703c4164f901ac06e2137a0e2c71f7cf7b636423a1e1ac2a9b", "source_pipeline": "f8745264f2422bf76eaec06120aa00b510cafa0ee3abdfac3d5396a614afb3a1", "data_rows": [ - "0000010f2cedeebe0599b55b059f918c148c3c0056dd9848d7fdecfc1eba992c23962f", - "000001618d4b5959d543202f49171b6c2b89bc5fef30b9bd157c03a13458b3ae882c8c", + "000002b734209953036fae7ee935c232f7949f4874dc22dca2af4ecfa2882e5ddaf457", + "0000024d11adcf6077042679df69776a748d71f2818fc47042913d83fa1fbebca67ad9", ], }, "list_of_struct": { - "stream_content": "276d8ec2a45a4e62be7496c9ce33687558647c5369c0828671f51f357a4fb6ea", + "stream_content": "407a24e60bf74b666560f868d193dc35efcaf531d8d357bd7e414b861436336d", "stream_pipeline": "85b8d45d79762e42d1edf56c3853379193c21d5879a30166fcf3bc3c8e56a733", "source_content": "19a5d5780cd84af633704022bf8f7a80ba72555533390ee5e69c487f4ec253cf", "source_pipeline": "85b8d45d79762e42d1edf56c3853379193c21d5879a30166fcf3bc3c8e56a733", "data_rows": [ - "00000151b3c44eaf49ed64174b9b47f89a779cd478c6874cbcac5a7df049d33ae1a7f9", - "000001dbb51ff7212e3b1dbe16f90d8d3133c5b8743d76d823fd9c903f6c9d2538eca5", + "000002c262cfe3995378fe521abbc19b8cd14f10f626b5e5b1259a20f6d4277677fb87", + "000002188277589a87f65b49271c1a2cf461ebed48ddd2fd3cb943198f420fced6862a", ], }, } From f5433ffd6f2e675a63c592b2158784d0c12bee56 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:46:27 -0800 Subject: [PATCH 024/147] docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing reference and DESIGN_ISSUES (ITL-910, WIP) Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 33 ++++++++++++++++++++ DESIGN_ISSUES.md | 23 ++++++++++++++ docs/reference/hashing.md | 64 +++++++++++++++++++++++++++++++++------ 3 files changed, 111 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dfdc26d3b..eec006848 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,39 @@ v0.3, or keep using v0.2 for an existing store. A migration is planned after v0. the MRO, so a registered `np.ndarray` subclass wins over `np.ndarray` for itself and its subclasses, whatever the registration order. +#### Arrow hashing moves to starfix hash format 2: every Arrow-derived hash changes (ITL-910) + +**Orcapod now requires `starfix>=0.5.0,<0.6`, and the Arrow hash label moves from `arrow_v0.1` +to `arrow_v0.2`.** starfix 0.4 (hash format 1) lost information for some inputs, so different +data could hash the same (ITL-848): the lengths of nested lists (`[[[1]], [[2], [3]]]` and +`[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are all null, and Boolean +columns, whose hash also depended on how the table was split into chunks. starfix 0.5 (hash +format 2, prefix `00 00 02`) is lossless; this is a fix, so the old values are not kept. + +- **What moves.** Every hash computed from an Arrow table: per-row data hashes, default + `source_id`s and the `record_id`s and source content hashes built from them, function + input-data hashes, pipeline base and entry ids, operator and cached-source record hashes, + invocation hashes, and the hashes of `pa.Table`, pandas `DataFrame` and `Series` values. The + hash label in front of each (`arrow_v0.2:000002...`) shows the new format. Pipeline hashes, + schema hashes, system-tag column names and table paths under the default + `table_scope="pipeline_hash"` do not move. Paths that embed a content hash + (`table_scope="content_hash"`, `CachedSource` tables) move with it. Where a `GroupBy` or + `Batch` orders members by `record_id`, the order of members can change too. +- **Results stored by 0.2.0.** STORED-DATA-PARAGRAPH (ITL-910 D1, pending) +- **Now hashable.** `fixed_size_list` columns, at any depth (ITL-927): a source or operator + carrying one works, and its type is kept. A `fixed_size_list[n]` hashes differently from a + `large_list` with the same values. pyarrow's canonical extension types (`arrow.uuid`, + `arrow.fixed_shape_tensor`) hash with their extension name and metadata, so an `arrow.uuid` + column never hashes like the `fixed_size_binary(16)` column holding the same bytes, and + `StarfixArrowHasher.hash_schema` keeps the identity of extension columns (ITL-846). +- **Now refused** with `ValueError` naming the column path and type, where starfix 0.4 hashed + silently: a non-nullable field holding a null; a zero-row table with a `map` column; two + schema nodes sharing a path (duplicate column names, for example in a `pa.Table` value, or a + column `"a/b"` next to a struct column `a` with a child `b`). Data of `null`, `map`, + interval, `string_view` and `binary_view` columns, union and run-end-encoded types are + still refused, now with `ValueError` instead of `NotImplementedError`. +- Column order and struct field order are still not part of an Arrow hash. + #### A changed column type is refused when appending to a Delta table (ITL-884) Before writing to an existing `DeltaTableDatabase` table, the flush now compares the new rows' diff --git a/DESIGN_ISSUES.md b/DESIGN_ISSUES.md index 695708a20..910936f32 100644 --- a/DESIGN_ISSUES.md +++ b/DESIGN_ISSUES.md @@ -2442,3 +2442,26 @@ results. Known compromise: the struct's children are nullable at the Arrow level, because Polars drops nested non-null flags on its way back to Arrow (ITL-842). The logical type enforces non-null instead. Tighten under a new storage version once nested nullability survives Polars. + +--- + +## `src/orcapod/hashing/arrow_hashers.py` — Arrow hash format (starfix) + +### HS2 — starfix hash format 1 collided on nested lists, null structs and Booleans +**Status:** in progress +**Severity:** high +**Issue:** ITL-910 (fixes the starfix side of ITL-848; ITL-927) + +`StarfixArrowHasher` used starfix 0.4 (hash format 1, label `arrow_v0.1`), which lost +information for some inputs, so different data reached the same row hash: nested-list lengths +(`[[[1]], [[2], [3]]]` and `[[[1], [2]], [[3]]]`), a null struct and a struct whose fields are +all null, and non-nullable Boolean columns (`[True]` and `[True, False, False]`); a Boolean +column's hash also depended on chunking. starfix 0.4 also could not hash `fixed_size_list` +data, so a source with such a column failed at construction (ITL-927). + +**Fix:** starfix 0.5 (hash format 2, prefix `00 00 02`) under the new label `arrow_v0.2` +(developer's label rule: the label moves whenever any table hash can change). Every +Arrow-derived hash moves; the golden files were regenerated once, explicitly. Order options +stay off. pyarrow's canonical extension types and `hash_schema` on extension columns hash with +their extension identity instead of as their storage type. Open: what a v0.3 node does with +rows a v0.2 store holds under `arrow_v0.1` in the same table (ITL-910 decision D1). diff --git a/docs/reference/hashing.md b/docs/reference/hashing.md index 13ddcf73a..8cdb2c3a5 100644 --- a/docs/reference/hashing.md +++ b/docs/reference/hashing.md @@ -132,24 +132,24 @@ each suffix is the stream's `pipeline_hash()` (sites 1b and 6). | Site | Row | Value (hex) | |------|-----|-------------| -| 7 | `alice` `base_entry_id` | `6172726f775f76302e313a00000113f967f1187ac2982e65b6afbf324d264d50a6efe3357a35f09b8a2b32454d61` | -| 7 | `bob` `base_entry_id` | `6172726f775f76302e313a000001b6162a6faa76b5e065e89849acc09f41edfb5bb1a7c7ea626fd2920de6046586` | -| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e313a000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 7 | `alice` `base_entry_id` | `6172726f775f76302e323a000002d147e6dfdc31690391034ed592bb75d4c4407034504165d168717c0779b64d3f` | +| 7 | `bob` `base_entry_id` | `6172726f775f76302e323a000002250edea78c481fa87d262c5938c9288a5538a17a10636b54a6ea83a18c77eb82` | +| 8 | `alice` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 8 | `bob` `pipeline_entry_id` (idx=0) | `6172726f775f76302e323a0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | -> The hex prefix `6172726f775f76302e31` decodes to the ASCII string `arrow_v0.1` — the +> The hex prefix `6172726f775f76302e32` decodes to the ASCII string `arrow_v0.2` — the > `method` portion of the `b"{method}:{digest}"` format described in §3. #### §4 — Side-Effect Record ID & Invocation Hash | Site | Row | Value | |------|-----|-------| -| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:0000016d7853d1b6b3dc62063d3982911c3f18b8a719e63cfd41429fff970137c52d9c` | -| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.1:000001244dc0d59c23ec5d4a19b972a2a98f410054da09f839115841cefdf7f4851f63` | +| 10 | `alice` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000028f3588b0c592a1ef02e782927fd355fa4e7887597c75d350a428647f9fc81cd9` | +| 10 | `bob` `invocation_hash` | `semantic_v0.1:1f71386f47538f7b5ca97554104469650b8ec5103a824cbe811a7b776913da91::arrow_v0.2:0000026122df3637ef917d0f6fb996572dfab4c8b92a6539f97430054c0b2271363acc` | > Both rows share the same `semantic_v0.1:1f71386f...` component — this is the > `SideEffectPodStream.pipeline_hash()`, which depends only on the pod and its upstream -> topology, not on row content. The `arrow_v0.1:...` component is the per-row `record_id` +> topology, not on row content. The `arrow_v0.2:...` component is the per-row `record_id` > from site 9, which differs per row. #### §5 — Data Function URI Hash @@ -334,7 +334,7 @@ topology suffix, encoding the full join lineage into each column name. | Field | Value | |---|---| | **Inputs** | Full raw Arrow table (all columns, before any system tag injection) | -| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — versioned SHA-256 via the `starfix` Rust crate | +| **Algorithm** | `StarfixArrowHasher.hash_table(table).to_hex(path_n_char)` — starfix hash format 2: the 3-byte format number `00 00 02` followed by a SHA-256 digest (`starfix` 0.5, byte-identical to the Rust crate) | | **Output format** | Truncated hex `str` (length = `path_n_char`) | | **Uniqueness guarantee** | Unique per raw table content; changes if any cell value changes | | **Known exclusions** | Used **only** as a fallback when no explicit `source_id` is provided. A user-supplied `source_id` bypasses this computation entirely. | @@ -618,3 +618,49 @@ print(hasher.hash_object(np.arange(6, dtype=np.float32).reshape(2, 3)).to_string print(hasher.hash_object(np.array(2.5)).to_string()) # ndarray_v1:47f182d862f5dd597d88836deddaed609ed94490827302e75e8364e877633cae ``` + +## 9. Arrow Table Hash (`arrow_v0.2`) + +Every Arrow-derived hash in the sites above (row data hashes, `record_id`s built from a +default `source_id`, base and pipeline entry ids, operator and cached-source record hashes, +`pa.Table` and pandas values) comes from `StarfixArrowHasher.hash_table`, which returns +`ContentHash(method="arrow_v0.2", digest=D)`. `D` is 35 bytes: the starfix hash format +number `00 00 02` followed by a SHA-256 digest, computed by `starfix` 0.5 (hash format 2) and +byte-identical to the Rust `starfix` crate. The byte layout is specified in starfix's +`docs/byte-layout-spec.md`. + +**Label rule.** The label `arrow_vX.Y` moves whenever the hash of any table can change, +whatever the cause. Its number is not kept equal to the starfix format number; the data +context's changelog records which starfix format each label uses. + +| Label | starfix | Format prefix | Orcapod | +|---|---|---|---| +| `arrow_v0.1` | 0.4 | `00 00 01` | up to v0.2 | +| `arrow_v0.2` | 0.5 | `00 00 02` | v0.3 | + +**What is part of the hash.** Column names, Arrow types (with large and non-large layouts of +strings, binaries and lists hashing alike), nullability, every value, row order, list element +order, list lengths at every depth, null structs (distinct from structs whose fields are all +null) and the row count. Column order and struct field order are not part of the hash. How a +table is split into chunks or slices is not part of it either. A `fixed_size_list[n]` +column hashes differently from a `large_list` column with the same values. Extension +columns are hashed as their storage type plus their extension name and metadata, including +pyarrow's canonical extension types (`arrow.uuid`, `arrow.fixed_shape_tensor`); only +`ARROW:extension:*` metadata is hashed, all other metadata is ignored. + +**What cannot be hashed.** starfix raises `ValueError`, naming the column path and type, for: +data of `null`, `map`, interval, `string_view` and `binary_view` columns (whatever the row +count; their schemas hash); union and run-end-encoded types; two schema nodes sharing a path +(duplicate column names, or a name containing `/` that aliases a struct child, such as a +column `"a/b"` next to a struct column `a` with a child `b`); and a non-nullable field holding +a null. A source whose table has such a column fails when it is constructed. + +```python +import pyarrow as pa +from orcapod.contexts import get_default_context + +hasher = get_default_context().arrow_hasher +digest = hasher.hash_table(pa.table({"x": pa.array([1, 2, 3], pa.int64())})) +print(digest.method, digest.digest[:3].hex()) +# arrow_v0.2 000002 +``` From bf8784fa2010e7188129c515c2467e9bc5884e8d Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:47:48 -0800 Subject: [PATCH 025/147] style: ruff format and type the format-2 tests (ITL-910) Co-Authored-By: Claude Opus 5.5 --- src/orcapod/hashing/schema_cleaner.py | 6 +++--- tests/test_hashing/test_starfix_format2.py | 21 ++++++++++++--------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/src/orcapod/hashing/schema_cleaner.py b/src/orcapod/hashing/schema_cleaner.py index 2fa55086f..a3021c811 100644 --- a/src/orcapod/hashing/schema_cleaner.py +++ b/src/orcapod/hashing/schema_cleaner.py @@ -178,9 +178,9 @@ def _contains_extension_type(arrow_type: pa.DataType) -> bool: ): return _contains_extension_type(arrow_type.value_type) if pa.types.is_map(arrow_type): - return _contains_extension_type(arrow_type.key_type) or _contains_extension_type( - arrow_type.item_type - ) + return _contains_extension_type( + arrow_type.key_type + ) or _contains_extension_type(arrow_type.item_type) if pa.types.is_dictionary(arrow_type): return _contains_extension_type(arrow_type.value_type) return False diff --git a/tests/test_hashing/test_starfix_format2.py b/tests/test_hashing/test_starfix_format2.py index ec21528ae..44bba9489 100644 --- a/tests/test_hashing/test_starfix_format2.py +++ b/tests/test_hashing/test_starfix_format2.py @@ -8,11 +8,15 @@ from __future__ import annotations +from typing import Any + import pyarrow as pa import pytest from orcapod.contexts import get_default_context from orcapod.core.operators import Join + +# Non-public on purpose: the label constant itself is under test. from orcapod.hashing.versioned_hashers import _CURRENT_ARROW_HASHER_ID from orcapod.sources import ArrowTableSource from orcapod.types import ContentHash @@ -25,16 +29,19 @@ def _hash_table(table: pa.Table) -> ContentHash: return get_default_context().arrow_hasher.hash_table(table) -def _row_hashes(column: pa.Array) -> list[str]: +def _row_hashes(column: pa.Array[Any]) -> list[str]: """Content hashes of the data of a one-column source keyed by a row tag.""" table = pa.table( - {"k": pa.array([f"r{i}" for i in range(len(column))], pa.large_string()), "x": column} + { + "k": pa.array([f"r{i}" for i in range(len(column))], pa.large_string()), + "x": column, + } ) source = ArrowTableSource(table, tag_columns=["k"]) return [data.content_hash().to_string() for _, data in source.iter_data()] -def _non_nullable(name: str, column: pa.Array) -> pa.Table: +def _non_nullable(name: str, column: pa.Array[Any]) -> pa.Table: """A one-column table whose field is declared non-nullable.""" return pa.Table.from_arrays( [column], schema=pa.schema([pa.field(name, column.type, nullable=False)]) @@ -198,9 +205,7 @@ def test_uuid_schema_differs_from_its_storage(self): class TestRejections: def test_zero_row_map_column_is_rejected(self): - table = pa.table( - {"m": pa.array([], pa.map_(pa.large_string(), pa.int64()))} - ) + table = pa.table({"m": pa.array([], pa.map_(pa.large_string(), pa.int64()))}) with pytest.raises(ValueError, match="map"): _hash_table(table) @@ -218,9 +223,7 @@ def test_non_nullable_column_holding_a_null_is_rejected(self): ArrowTableSource(table, tag_columns=["k"]) def test_duplicate_column_names_are_rejected(self): - table = pa.Table.from_arrays( - [pa.array([1]), pa.array([2])], names=["x", "x"] - ) + table = pa.Table.from_arrays([pa.array([1]), pa.array([2])], names=["x", "x"]) with pytest.raises(ValueError, match="Ambiguous node path 'x'"): get_default_context().semantic_hasher.hash_object(table) From 6dfb456c2cb2c687f61ff13a456a00476ed9fbfe Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 15:53:39 -0800 Subject: [PATCH 026/147] docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels) Co-Authored-By: Claude Opus 5.5 --- .../2026-11-11-itl-910-adopt-starfix-05.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md index 96bb966f7..97b835e5f 100644 --- a/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md +++ b/superpowers/plans/2026-11-11-itl-910-adopt-starfix-05.md @@ -166,3 +166,26 @@ duplicate too. ### Task 7: PR - [ ] Merge origin/main, push, PR (`Fixes ITL-910`, mentions ITL-848, ITL-927), CodeRabbit rounds. + +## D1 = D (developer, 2026-11-11): fail explicitly on rows hashed under another label + +Design (tests first): +- New public `orcapod.errors.StoreHashLabelError(StoreError)` (always fails the run, whatever + `error_policy`): attributes `record_path`, `found_label`, `current_label`; message names the table, + both labels and the remedy (fresh store, or keep the version that wrote it; migration tool planned). + No opt-out (`ignore_schema` does not disable it). +- Helpers in `src/orcapod/hashing/stored_hash_labels.py` return what they find (no raising): + `foreign_arrow_label(values, current_label)` for `b"