Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,11 @@ Developer, November 12, 2026 (ITL-981); see `docs/concepts/arrays.md` and
`allow_pickle=False` on write and read; never raise `max_header_size`). Options:
`memory_order`, `byte_order`, `struct_layout`. Where stored arrays must be ordered, order
them by value hash (`storage_sort_key`), never by stored bytes.
- Library value types are provisional until the v0.4 audit (developer, November 13, 2026,
ITL-1068): their extension names are listed in `provisional` of
`contexts/logical_types.json`, and using one issues `ProvisionalHashWarning` once per type
per process. A new library value type must be added to that list. The list never enters a
hash, identifier or stored path.

## User classes: exact and structural

Expand Down
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -331,6 +331,11 @@ Developer, November 12, 2026 (ITL-981); see `docs/concepts/arrays.md` and
`allow_pickle=False` on write and read; never raise `max_header_size`). Options:
`memory_order`, `byte_order`, `struct_layout`. Where stored arrays must be ordered, order
them by value hash (`storage_sort_key`), never by stored bytes.
- Library value types are provisional until the v0.4 audit (developer, November 13, 2026,
ITL-1068): their extension names are listed in `provisional` of
`contexts/logical_types.json`, and using one issues `ProvisionalHashWarning` once per type
per process. A new library value type must be added to that list. The list never enters a
hash, identifier or stored path.

## User classes: exact and structural

Expand Down
17 changes: 17 additions & 0 deletions changelog.d/ITL-1068.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
### Added

- `orcapod.errors.ProvisionalHashWarning`, a `UserWarning` issued once per type per process
when a library value type is first used (ITL-1068). Silence it with
`warnings.filterwarnings("ignore", category=ProvisionalHashWarning)`.

### Known limitations

- The value hashes and logical type identifiers of the library value types are provisional:
`numpy.ndarray`, `torch.Tensor`, and pandas and polars `DataFrame` and `Series`. They will
be reviewed for v0.4 and may change then; results computed from such values under v0.3 may
then be computed again (nothing stored is lost or wrong). Using one of these types issues
`ProvisionalHashWarning` once per type per process: a value hashed, a source column, a
function pod parameter or output, or stored results read back. The types are listed in the
`provisional` list of `orcapod/contexts/logical_types.json`; the list changes no hash,
identifier or stored path. See "Provisional value hashes" in `docs/concepts/arrays.md`
(ITL-1068).
20 changes: 20 additions & 0 deletions design_issues/ITL-1068-PH1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
## `src/orcapod/logical_types/`

### PH1 — Library value type hashes are provisional until the v0.4 audit
**Status:** in progress
**Severity:** medium
**Issue:** ITL-1068

The value hashes and logical type identifiers of the library value types
(`orcapod.numpy.ndarray`, `orcapod.torch.tensor`, `orcapod.pandas.dataframe`,
`orcapod.pandas.series`, `orcapod.polars.dataframe`, `orcapod.polars.series`) are not yet
guaranteed stable across releases. ITL-1069 audits them for v0.4 against the principle "for
anything maintained or yielded by another system, respect every distinction it has; make no
assumption about what matters semantically". Until then v0.3 warns whenever one is used.

**Fix (interim, ITL-1068):** the types are listed by extension name in `provisional` of
`contexts/logical_types.json`. The logical type registry (`get_by_python_type`,
`get_by_arrow_extension_name`) and the value handler registry (`get_handler`) of each data
context hold that list and issue `orcapod.errors.ProvisionalHashWarning` once per type per
process (`logical_types/provisional.py`). The list enters no hash, identifier or stored path.
The entry is resolved when ITL-1069 settles the hashes and removes the names from the list.
30 changes: 30 additions & 0 deletions docs/concepts/arrays.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,36 @@ for tag, data in scale.pod(source).iter_data():
The v0.2 tables and rows stay on disk unchanged, readable with v0.2. A migration is
planned after v0.3 (ITL-889).

## Provisional value hashes

The value hashes and stored identities of the library value types are provisional in v0.3:
`numpy.ndarray`, `torch.Tensor`, and the pandas and polars `DataFrame` and `Series`. They
will be reviewed for v0.4 and may change then. Every other type, and the hashing of everything
else in a pipeline, is unaffected.

Using one of these types issues `orcapod.errors.ProvisionalHashWarning` once per type per
process, on whichever use comes first: a value hashed, a source column of the type, a function
pod with a parameter or output of the type, or stored results of the type read back. It names
the type, for example `orcapod.numpy.ndarray`.

In practice nothing needs to change. Use these types as described on this page. If v0.4
changes how one of them is hashed, a result Orcapod computed under v0.3 from such a value gets
a different identity, so v0.4 computes it again instead of reusing it. Nothing stored is lost
or becomes wrong: the v0.3 rows stay where they are. Plan for that recomputation if a
pipeline is expensive. To silence the warning once you have taken note of it, filter its class
after importing Orcapod:

```python
import warnings

from orcapod.errors import ProvisionalHashWarning

warnings.filterwarnings("ignore", category=ProvisionalHashWarning)
```

Python's `-W` option and `PYTHONWARNINGS` cannot name this class: they are read before
installed packages can be imported.

## What counts as the same array

Two arrays are the same value when everything below that counts is equal.
Expand Down
13 changes: 13 additions & 0 deletions docs/reference/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -485,6 +485,8 @@ moves no path.

### 1e. Polars DataFrame and Series values

This hash is provisional in v0.3 (see [section 8](#8-array-value-hash)).

A `polars.DataFrame` or `polars.Series` is a value-hashed logical type
(`PolarsDataFrameHandler`, `PolarsSeriesHandler`; the algorithm is defined in
`orcapod.logical_types.polars_type`). Inside a table its cell token is, like every
Expand Down Expand Up @@ -914,6 +916,15 @@ in any data-level preimage; they serve logging, observability, and result inspec

## 8. Array Value Hash

**Provisional.** The value hashes of the library value types, `numpy.ndarray` (this section),
`torch.Tensor` ([below](#torch-tensor-value-hash)), polars `DataFrame` and `Series`
([1e](#1e-polars-dataframe-and-series-values)) and pandas `DataFrame` and `Series`, and their
logical type identifiers, are provisional in v0.3 and may change in v0.4. Using one issues
`ProvisionalHashWarning` once per type per process (see
[Provisional value hashes](../concepts/arrays.md#provisional-value-hashes)). The types are
declared in the `provisional` list of `orcapod/contexts/logical_types.json`, by extension name;
that list enters no hash, identifier or stored path.

**Value hash labels.** A type whose values are hashed by a dedicated handler labels its hash
`<type>_v<N>`, for example `ndarray_v1` or `torch_tensor_v1`. The label is the `ContentHash.method`; the type's
implementer chooses it. The bytes produced under an existing label never change. A new
Expand Down Expand Up @@ -1376,6 +1387,8 @@ print(digest.method, digest.digest[:3].hex())

## 10. pandas Value Hash (`pandas_dataframe_v1`, `pandas_series_v1`)

This hash is provisional in v0.3 (see [section 8](#8-array-value-hash)).

A `pandas.DataFrame` or `pandas.Series` value (a pod input or output, a value in a source
column, or any object given to the semantic hasher) is hashed by
`pandas_frame_content_hash` / `pandas_series_content_hash` in
Expand Down
5 changes: 5 additions & 0 deletions pytest.ini
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,8 @@ markers =
postgres: mark test as requiring a live PostgreSQL instance
integration: mark test as an integration test (may be slower)
minio: tests requiring a running MinIO instance (via testcontainers or MINIO_ENDPOINT env var)
# The library value types warn that their hashes are provisional (ITL-1068), once per
# type per process, so whichever test uses one first would show it. The tests of the
# warning record it explicitly.
filterwarnings =
ignore::orcapod.errors.ProvisionalHashWarning
15 changes: 13 additions & 2 deletions src/orcapod/contexts/logical_types.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"description": "Logical types registered in every data context, with the value handlers that hash their Python values. This file is not frozen: a logical type is versioned by its own identifier (Arrow extension name and metadata), not by the data context version, so adding one edits this file and no released data context manifest. hash_ignore lists, per extension name, the property names the value hash of a library value type ignores (empty: every property the storage keeps is part of the hash); the list is written into the type's extension metadata, so changing it changes the type's identifier. The logical type entry and the value handler entry of such a type both refer to their list as hash_ignore:<extension name>.",
"description": "Logical types registered in every data context, with the value handlers that hash their Python values. This file is not frozen: a logical type is versioned by its own identifier (Arrow extension name and metadata), not by the data context version, so adding one edits this file and no released data context manifest. hash_ignore lists, per extension name, the property names the value hash of a library value type ignores (empty: every property the storage keeps is part of the hash); the list is written into the type's extension metadata, so changing it changes the type's identifier. The logical type entry and the value handler entry of such a type both refer to their list as hash_ignore:<extension name>. provisional lists, by extension name, the types whose value hash and stored identity may still change (until the v0.4 audit): using one issues ProvisionalHashWarning once per type per process. Deleting a name stops its warning and changes nothing else; the list never enters an identifier, a hash or a stored path. A new library value type is declared provisional here.",
"hash_ignore": {
"orcapod.numpy.ndarray": [],
"orcapod.pandas.dataframe": [],
Expand All @@ -8,6 +8,14 @@
"orcapod.polars.series": [],
"orcapod.torch.tensor": []
},
"provisional": [
"orcapod.numpy.ndarray",
"orcapod.pandas.dataframe",
"orcapod.pandas.series",
"orcapod.polars.dataframe",
"orcapod.polars.series",
"orcapod.torch.tensor"
],
"logical_type_registry": {
"_class": "orcapod.logical_types.registry.LogicalTypeRegistry",
"_config": {
Expand Down Expand Up @@ -124,7 +132,10 @@
},
"category": "set"
}
]
],
"provisional": {
"_ref": "provisional"
}
}
},
"value_handlers": [
Expand Down
21 changes: 20 additions & 1 deletion src/orcapod/contexts/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,10 @@ def get_context(self, context_string: str | None = None) -> DataContext:
# ``hash_ignore`` object is referred to as ``hash_ignore:<extension name>``.
_HASH_IGNORE_REF_PREFIX = "hash_ignore:"

# The registration file's ``provisional`` list (extension names of the types
# whose value hash is provisional, ITL-1068) is referred to by this name.
_PROVISIONAL_REF = "provisional"

def _uses_logical_type_registration(self, spec: dict[str, Any]) -> bool:
"""Whether ``spec`` takes its logical types from the registration file.

Expand All @@ -395,7 +399,8 @@ def _load_logical_type_registration(self) -> dict[str, Any]:
of ``[type, handler]`` object-spec pairs and, when present, its
``deferred_value_handlers`` list of handlers loaded on first use and
its ``hash_ignore`` object: per extension name, the property names the
value hash of that type ignores.
value hash of that type ignores; and its ``provisional`` list: the
extension names of the types whose value hash is provisional.
"""
return _read_json_file(self.logical_types_file)

Expand Down Expand Up @@ -433,6 +438,19 @@ def _create_context_from_spec(self, spec: dict[str, Any]) -> DataContext:
f"names, got {ignore!r}"
)
ref_lut[f"{self._HASH_IGNORE_REF_PREFIX}{name}"] = list(ignore)
# The types whose value hash is provisional are declared once, by
# extension name; the logical type registry takes the list as
# ``{"_ref": "provisional"}`` and the value handler registry below
# is given it. It never enters an identifier or a hash (ITL-1068).
provisional = registration.get(self._PROVISIONAL_REF, [])
if not isinstance(provisional, list) or not all(
isinstance(item, str) for item in provisional
):
raise ValueError(
f"provisional must be a list of extension names, got "
f"{provisional!r}"
)
ref_lut[self._PROVISIONAL_REF] = list(provisional)
ref_lut[self._LOGICAL_TYPE_REGISTRY_REF] = parse_objectspec(
registration["logical_type_registry"], ref_lut=ref_lut
)
Expand Down Expand Up @@ -462,6 +480,7 @@ def _create_context_from_spec(self, spec: dict[str, Any]) -> DataContext:
entry["handler"],
parse_objectspec(entry.get("_config", {}), ref_lut=ref_lut),
)
handler_registry.declare_provisional(ref_lut[self._PROVISIONAL_REF])

return DataContext(
context_key=context_key,
Expand Down
19 changes: 19 additions & 0 deletions src/orcapod/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,25 @@ class StoreContentWarning(UserWarning):
"""


class ProvisionalHashWarning(UserWarning):
"""A value type was used whose value hash and stored identity are provisional.

The library value types (``numpy.ndarray``, ``torch.Tensor``, pandas and polars
``DataFrame`` and ``Series``) are hashed and stored with formats that may still
change in v0.4. A result computed now from such a value may then get another hash,
so it is computed again after the upgrade: nothing stored is lost or wrong. The
warning is issued once per type per process, on whichever use comes first: a value
hashed, a column converted to or from Arrow, a function pod annotated with the type,
a stored column read. Which types are provisional is declared in the ``provisional``
list of ``orcapod/contexts/logical_types.json``.

To silence it, call ``warnings.filterwarnings("ignore",
category=ProvisionalHashWarning)`` after importing Orcapod. Python's ``-W`` option
and ``PYTHONWARNINGS`` cannot name this class: they are read before installed
packages can be imported.
"""


class EmptyDataAccessError(Exception):
"""Raised when a payload-access method is called on an ``EmptyData`` instance.

Expand Down
15 changes: 14 additions & 1 deletion src/orcapod/hashing/semantic_hashing/builtin_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,17 @@
)
from orcapod.hashing.semantic_hashing.encoded import Encoded
from orcapod.hashing.semantic_hashing.function_info_extractors import ParameterDefault
from orcapod.logical_types.array_format import NDARRAY_OPTIONS
from orcapod.logical_types.array_format import NDARRAY_EXTENSION_NAME, NDARRAY_OPTIONS
from orcapod.logical_types.identifier import logical_type_identifier
from orcapod.logical_types.ignore_list import IgnoreListHandler
from orcapod.logical_types.pandas_type import (
LogicalPandasDataFrame,
LogicalPandasSeries,
)
from orcapod.logical_types.polars_type import (
POLARS_DATAFRAME_EXTENSION_NAME,
POLARS_SERIES_EXTENSION_NAME,
)
from orcapod.types import ContentHash, Schema
from orcapod.utils.canonical_type_names import canonical_type_name

Expand Down Expand Up @@ -429,6 +437,7 @@ class NumpyArrayHandler(IgnoreListHandler):
array is hashed (the data context's list for the type).
"""

EXTENSION_NAME = NDARRAY_EXTENSION_NAME
OPTIONS = NDARRAY_OPTIONS
HASH_LABEL = "ndarray_v1"
"""``ContentHash.method`` of an ndarray hash."""
Expand Down Expand Up @@ -471,6 +480,7 @@ class PandasDataFrameHandler(IgnoreListHandler):
the registry were constructed before the arrow hasher).
"""

EXTENSION_NAME = LogicalPandasDataFrame.EXTENSION_NAME
OPTIONS = PANDAS_DATAFRAME_OPTIONS
HASH_LABEL = PANDAS_DATAFRAME_HASH_LABEL

Expand Down Expand Up @@ -539,6 +549,7 @@ class PandasSeriesHandler(IgnoreListHandler):
the registry were constructed before the arrow hasher).
"""

EXTENSION_NAME = LogicalPandasSeries.EXTENSION_NAME
OPTIONS = PANDAS_SERIES_OPTIONS
HASH_LABEL = PANDAS_SERIES_HASH_LABEL

Expand Down Expand Up @@ -636,6 +647,7 @@ class PolarsDataFrameHandler(IgnoreListHandler):
``struct_field_order``.
"""

EXTENSION_NAME = POLARS_DATAFRAME_EXTENSION_NAME
OPTIONS = frozenset({"column_order", "struct_field_order"})
HASH_LABEL = "polars_dataframe_v1"
"""``ContentHash.method`` of a ``pl.DataFrame`` hash."""
Expand Down Expand Up @@ -677,6 +689,7 @@ class PolarsSeriesHandler(IgnoreListHandler):
``struct_field_order``.
"""

EXTENSION_NAME = POLARS_SERIES_EXTENSION_NAME
OPTIONS = frozenset({"series_name", "struct_field_order"})
HASH_LABEL = "polars_series_v1"
"""``ContentHash.method`` of a ``pl.Series`` hash."""
Expand Down
2 changes: 2 additions & 0 deletions src/orcapod/hashing/semantic_hashing/torch_handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from orcapod.logical_types.ignore_list import IgnoreListHandler
from orcapod.logical_types.torch_type import (
OPTIONS,
TORCH_TENSOR_EXTENSION_NAME,
TORCH_TENSOR_HASH_LABEL,
tensor_digest,
)
Expand All @@ -29,6 +30,7 @@ class TorchTensorHandler(IgnoreListHandler):
elements, also inside an untyped Python structure.
"""

EXTENSION_NAME = TORCH_TENSOR_EXTENSION_NAME
OPTIONS = OPTIONS
HASH_LABEL = TORCH_TENSOR_HASH_LABEL

Expand Down
29 changes: 27 additions & 2 deletions src/orcapod/hashing/semantic_hashing/type_handler_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@

import logging
import threading
from collections.abc import Mapping
from collections.abc import Iterable, Mapping
from typing import TYPE_CHECKING, Any

from orcapod.logical_types.provisional import warn_provisional
from orcapod.utils.deferred_types import load_object, qualified_name

if TYPE_CHECKING:
Expand Down Expand Up @@ -48,6 +49,8 @@ def __init__(
self._deferred: dict[str, dict[str, Any]] = {}
self._handlers: dict[type, "PythonTypeHandlerProtocol"] = {}
self._lock = threading.RLock()
# Extension names of the types whose value hash is provisional (ITL-1068).
self._provisional: frozenset[str] = frozenset()
if handlers:
for entry in handlers:
# Skip empty or incomplete pairs. When a handler pair in the context
Expand Down Expand Up @@ -124,16 +127,38 @@ def unregister(self, target_type: type) -> bool:
return True
return False

def declare_provisional(self, extension_names: Iterable[str]) -> None:
"""Declare the types whose value hash is provisional (ITL-1068).

A handler whose ``EXTENSION_NAME`` is declared issues
``ProvisionalHashWarning`` the first time in the process that ``get_handler``
returns it for a value. The data context loader passes the ``provisional``
list of ``contexts/logical_types.json``; the declaration never enters a hash.

Args:
extension_names: The extension names declared provisional. They replace
any earlier declaration.
"""
with self._lock:
self._provisional = frozenset(extension_names)

def get_handler(self, obj: Any) -> "PythonTypeHandlerProtocol | None":
"""Look up the handler for *obj* using MRO-aware resolution.

A handler of a type declared provisional (``declare_provisional``) issues
``ProvisionalHashWarning`` on its first use in the process.

Args:
obj: The object for which a handler is needed.

Returns:
The registered ``PythonTypeHandlerProtocol``, or None.
"""
return self.get_handler_for_type(type(obj))
handler = self.get_handler_for_type(type(obj))
extension_name = getattr(handler, "EXTENSION_NAME", None)
if extension_name is not None and extension_name in self._provisional:
warn_provisional(extension_name)
return handler

def get_handler_for_type(
self, target_type: type
Expand Down
Loading
Loading