diff --git a/.zed/rules b/.zed/rules index 0c226b8ac..16aac7af9 100644 --- a/.zed/rules +++ b/.zed/rules @@ -355,7 +355,19 @@ Function Pods: inspect data content (never tags), synthesize new values, cached Every pipeline element has two parallel hashes: -1. content_hash() — data-inclusive. Changes when data changes. Used for deduplication. +1. content_hash() — the pipeline bound to its named sources: schema, topology and source + identity. For a source it is a stable address, not a digest of its rows (developer, + November 11, 2026: "sources content hash is meant to be a stable address and is not + necessarily going to reflect the very data content"). A root source hashes + (class name, output schema, source_id); given a source_id, changed, added or reordered + rows leave it unchanged. Row content enters only where a source puts it into its own + identity: the default source_id of the in-memory sources (a hash of the table, used when + none is given) and the elements of a ListSource. A downstream stream or node hashes its + pod's identity plus its upstreams' content hashes; a function pod's identity is its uri, + not its body. Only a datagram (one row's tags or data) has a content_hash() that is a + digest of its values; that row hash, not the node's content hash, is what results are + memoized by. The node's content hash tells apart, inside a shared table, the rows of + pipelines bound to different sources (__node_content_hash). 2. pipeline_hash() — schema + topology only. Ignores data content. Used for DB path scoping so that different sources with identical schemas share database tables. diff --git a/CLAUDE.md b/CLAUDE.md index 1908ec144..cd32f3a13 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -379,8 +379,19 @@ every emitted element came from an input row — and it keys only on tags, never Every pipeline element has two parallel hashes: -1. **`content_hash()`** — data-inclusive. Changes when data changes. Used for deduplication - and memoization. +1. **`content_hash()`** — the pipeline bound to its named sources: schema, topology and source + identity. For a source it is a stable *address*, not a digest of its rows (developer, + November 11, 2026: "sources content hash is meant to be a stable address and is not + necessarily going to reflect the very data content"). A root source hashes + `(class name, output schema, source_id)`; given a `source_id`, changed, added or reordered + rows leave it unchanged. Row content enters only where a source puts it into its own + identity: the default `source_id` of the in-memory sources (a hash of the table, used when + none is given) and the elements of a `ListSource`. A downstream stream or node hashes its + pod's identity plus its upstreams' content hashes; a function pod's identity is its uri, + not its body. Only a datagram (one row's tags or data) has a `content_hash()` that is a + digest of its values; that row hash, not the node's content hash, is what results are + memoized by. The node's content hash tells apart, inside a shared table, the rows of + pipelines bound to different sources (`__node_content_hash`). 2. **`pipeline_hash()`** — schema + topology only. Ignores data content. Used for DB path scoping so that different sources with identical schemas share database tables. diff --git a/superpowers/plans/2026-11-11-itl-938-post-session-decisions.md b/superpowers/plans/2026-11-11-itl-938-post-session-decisions.md new file mode 100644 index 000000000..1e6ea4102 --- /dev/null +++ b/superpowers/plans/2026-11-11-itl-938-post-session-decisions.md @@ -0,0 +1,25 @@ +# ITL-938: bring the design document in line with the post-session decisions + +Docs only. File: `superpowers/specs/2026-11-11-itl-938-versioning-system-design.md` (PR #312). +Source of truth for each statement: PR bodies #313 to #318 (read November 11, 2026); the +coordinator's brief where no PR exists (ITL-935, ITL-953, ITL-957). + +CodeRabbit on #312: review skipped (file over the size limit); no reviews, no inline comments. +Nothing to plan for. + +## Edits, by decision + +1. **MergeJoin (E) withdrawn; ITL-942 as in PR #315.** 5.2 row 8; 5.3 (drop the E row); 5.4 + diagram (fold wording); Part 6 (what moves paths: F and N only); 7.2 row 2; Part 8 items 5, 7; + A.9 rows E, L; Part 9. +2. **Source ids.** 4.1 intro; 5.6 rule 5 + new 5a; 7.1 (label row, source identity row); 7.2 + row 6a + new rows; 7.3 default-id row; Part 8 items 1a, 11; Part 9. +3. **Native containers carry no version.** 5.5 table; 7.1 logical type row; 7.2 row 4; Part 8 + item 4; Part 9. +4. **Old data: prefix scan, warn when a store cannot list (PR #318).** Part 6 paths block and + "old layout" paragraph; 6.1 resolutions; 7.1; 7.2 row 3; 7.3; 7.3a rows 4, 5; Part 8 item 1; + Part 9. +5. **GroupBy / operator rule (PR #316, #317).** 5.2 new rows; 7.2 new rows; Part 9. +6. **Issue and PR numbers.** 7.2 State column; Part 8 items; header. + +Then: one docs commit, push, update PR body, check CI and CodeRabbit, `worker_done`. diff --git a/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md b/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md new file mode 100644 index 000000000..b09c6d8a7 --- /dev/null +++ b/superpowers/specs/2026-11-11-itl-938-versioning-system-design.md @@ -0,0 +1,1862 @@ +# ITL-938: one coherent versioning system for hashes, hash labels and stored paths + +Status: **design discussion closed on November 11, 2026.** Every decision is the developer's +and is listed, dated and quoted, in Part 9. Where an earlier part shows an intermediate view, +the later part governs; **Part 7.1 is the system on one page.** Decisions the developer made +later the same day, while the implementation issues were in progress, are listed at the end of +Part 9 and applied throughout. + +| Part | Content | +|---|---| +| 1 | What exists today: hashers, where each hash is stored, the store on disk, every version marker, measured failure modes | +| 2, 3 | The axes along which Orcapod's behaviour can change, and what each one affects | +| 4 | The path-or-column rule | +| 5 | The system: schema and subject; every change from the old version to v0.3; the data context as the global hash version; logical types; labels | +| 6 | Paths before and after v0.3; side-effecting nodes | +| 7 | The system on one page; what lands before the v0.3 tag and what follows; what ITL-910 must change; the data context file | +| 8 | Follow-up issues to file, with issue and pull request numbers where they exist | +| 9 | Decisions | +| Appendix A | Every hash computation site and what affects its result | + +Linear: [ITL-938](https://linear.app/metamorphic/issue/ITL-938). Design only; no library code. + +**How the facts here were obtained.** Everything was read from the code on `origin/main` at +`d61e3e39` and checked by running it on November 11, 2026. Comparisons with ITL-910 ran a +snapshot of `origin/eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod` at `6df680ce` +(starfix 0.5.0). Every hash string quoted is a real value from those runs. Items read from +code but not exercised by a run are marked "(code only)". + +**The probe pipeline** used for all examples: + +```python +s1 = DictSource([{"id": 0, "x": 5}, {"id": 1, "x": 10}, {"id": 2, "x": 15}], tag_columns=["id"], source_id="source1") +s2 = DictSource([{"id": 0, "y": 3}, {"id": 2, "y": 6}, {"id": 4, "y": 9}], tag_columns=["id"], source_id="source2") + +@function_pod(output_keys="doubled") +def double(x: int) -> int: return 2 * x + +@function_pod(output_keys="sum") +def take_sum(doubled: int, y: int) -> int: return doubled + y + +job = PipelineJob(name="my_pipeline", store=DeltaTableDatabase("./store")) +with job: + take_sum.pod(double.pod(s1, label="double").join(s2, label="join"), label="sum") +job.run() +``` + +**Terms.** + +| Term | Meaning | +|---|---| +| Label | The text in front of a digest that names the algorithm, as in `arrow_v0.1:000001d3da…`. In code: `ContentHash.method`. | +| Labelled / bare | A stored hash is *labelled* when the label is stored with the digest, *bare* when only hex or raw bytes are stored. | +| Preimage | The exact bytes or structure fed into a hash function. | +| Key column | A column whose value is used to find a row (a lookup or row key). | +| Axis of change | One independent way in which Orcapod's behaviour can change between two versions of the code. | + +--- + +## Part 1. What exists today + +### 1.1 Where hashing happens + +```mermaid +flowchart LR + subgraph VALUE["Per-type value hashers"] + V1["file bytes
sha256:2cf24dba…"] + V2["directory
merkle_sha256:…"] + V3["numpy array
ndarray_v1:8462766b…"] + V4["polars expression
polars_expr_v1:…"] + end + subgraph SEM["Structure hasher semantic_v0.1"] + S1["schema hash
semantic_v0.1:15816996…"] + S2["function uri hash
semantic_v0.1:17de3328…"] + S3["node pipeline_hash
semantic_v0.1:6fe7280a…"] + S4["node content_hash
semantic_v0.1:5da63a88…"] + end + subgraph ARW["Arrow hasher arrow_v0.1"] + A1["row data hash
arrow_v0.1:000001d3da…"] + A2["pipeline entry id / row key
arrow_v0.1:00000189…"] + A3["default source_id
00000188… (bare)"] + end + subgraph FIXED["Fixed, unnamed algorithms"] + F1["source record id
uuid5, 16 bytes"] + F2["folded system tags
uuid5 / sha256"] + end + VALUE -->|"value inside a structure"| SEM + VALUE -->|"extension cell inside a row"| ARW + S1 --> S2 --> S3 + S3 --> S4 + A3 -->|"when the user gives no source_id"| S4 + F1 --> A2 + A1 --> A2 +``` + +| Label | What it hashes | Algorithm | Where the label is set | +|---|---|---|---| +| `semantic_v0.1` | Python structures: schemas, function identity, node identity, every `content_hash()` and `pipeline_hash()` | SHA-256 of canonical JSON of the expanded structure. A nested hash is embedded as its labelled string, so **labels are part of the preimage** of every enclosing hash | `contexts/data/v0.1.json`; duplicated in `hashing/versioned_hashers.py` (no callers) and `contexts/data/schemas/context_schema.json` | +| `arrow_v0.1` (ITL-910: `arrow_v0.2`) | Arrow tables and schemas: each row's data hash, row keys | starfix `ArrowDigester`. The digest is 35 bytes: starfix's own 3-byte format number (`000001` for starfix 0.4, `000002` for 0.5), then SHA-256 | same three places | +| `sha256` | file content | SHA-256 of the bytes. The label is the algorithm name; it has no version | `hashing/hash_utils.py` | +| `merkle_sha256` | directory content | Merkle tree over entries; child file labels are dropped from the preimage | `hashing/directory_hashers.py` | +| `ndarray_v1` | `numpy.ndarray` values | `array_digest_v1` over dtype tag, shape, bytes | `builtin_handlers.py`, `logical_types/array_format.py` | +| `polars_expr_v1` | `polars.Expr` filter predicates | SHA-256 of canonical JSON of an Orcapod-owned expression tree | `semantic_hashing/polars_expr.py` | +| not on main yet | `torch_tensor_v1` (PR #300); `polars_dataframe_v1`, `polars_series_v1` (PR #286); `pandas_dataframe_v1`, `pandas_series_v1`, `arrow_table_v1` (ITL-897 branches) | per-type handlers | — | +| no label | PR #292 class identity `Point@9117beaa…` (SHA-256 of a JSON description); the two `uuid5` namespaces for record ids; `combine_hashes` for folded system tags | fixed algorithms with no name | — | + +An extension-typed cell inside an Arrow table is hashed as +`::