From c76abdaaed1cd6dd6d9b1696767a67e9f60c191b Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 11 Nov 2026 16:08:56 -0800 Subject: [PATCH 01/40] docs(specs): ITL-938 versioning design, inventory of hashes, labels, versions and paths Part of ITL-938 Co-Authored-By: Claude Fable 5.2 --- ...-11-11-itl-938-versioning-system-design.md | 186 ++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 superpowers/specs/2026-11-11-itl-938-versioning-system-design.md diff --git a/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md b/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md new file mode 100644 index 000000000..136139510 --- /dev/null +++ b/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md @@ -0,0 +1,186 @@ +# ITL-938: one coherent versioning system for hashes, hash labels and stored paths + +Status: **draft, discussion in progress** (inventory complete; the system, the path decision, +the v0.3 cut list and the follow-up issues are filled in as the developer decides them). + +Linear: [ITL-938](https://linear.app/metamorphic/issue/ITL-938). Design only; no library code. + +Everything in section 1 was read from the code on `origin/main` at `d61e3e39` and checked by +running it (November 11, 2026). The ITL-910 comparisons ran a snapshot of +`origin/eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod` at `6df680ce` +(starfix 0.5.0). Items read from code but not exercised by a run are marked "(code only)". + +Terms used here: + +- **Label**: the text in front of a digest that names the algorithm, as in + `arrow_v0.1:000001d0e4…`. In code it is `ContentHash.method`. +- **Labelled / bare**: a stored hash is *labelled* when the label is stored with the digest, + and *bare* when only the hex or raw bytes are stored. +- **Preimage**: the exact bytes or structure fed into a hash function. + +--- + +## 1. Inventory + +### 1.1 The hashers + +| Label | What it hashes | Algorithm | Where the label is set | +|---|---|---|---| +| `semantic_v0.1` | Python objects and structures: schemas, function identity, node identity, every `content_hash()` and `pipeline_hash()` | SHA-256 of canonical JSON of the expanded structure. A nested hash is embedded as its labelled string, so **labels are part of the preimage** of every enclosing hash | `contexts/data/v0.1.json` (`semantic_hasher.hasher_id`); duplicated in `hashing/versioned_hashers.py` (no callers) and `contexts/data/schemas/context_schema.json` | +| `arrow_v0.1` (ITL-910: `arrow_v0.2`) | Arrow tables and schemas: every row's data hash, row keys, pandas values | starfix `ArrowDigester`. The digest is 35 bytes: starfix's own 3-byte format number (`000001` for starfix 0.4, `000002` for 0.5) followed by SHA-256 | same three places | +| `sha256` | file content (`orcapod.File`) | SHA-256 of bytes; the label is the algorithm name, with no version | `hashing/hash_utils.py` | +| `merkle_sha256` | directory content (`orcapod.Directory`) | Merkle tree over entries; child file labels are dropped from the preimage | `hashing/directory_hashers.py` | +| `ndarray_v1` | `numpy.ndarray` values | `array_digest_v1` over dtype tag, shape and bytes | `hashing/semantic_hashing/builtin_handlers.py`, `logical_types/array_format.py` | +| `polars_expr_v1` | `polars.Expr` (filter predicates) | SHA-256 of canonical JSON of an Orcapod-owned expression tree | `hashing/semantic_hashing/polars_expr.py` | +| not on main yet | `torch_tensor_v1` (PR #300); `polars_dataframe_v1`, `polars_series_v1` (PR #286); `pandas_dataframe_v1`, `pandas_series_v1`, `arrow_table_v1` (ITL-897 branches, no PR) | per-type handlers | — | +| no label at all | PR #292 class identity `Point@`; the two `uuid5` namespaces for record ids; `combine_hashes` (SHA-256 of concatenation) for folded system tags | fixed algorithms with no name | — | + +An extension-typed cell inside an Arrow table is hashed as +`::