Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
0bf5e72
test(operators): several columns, names, mappings and inputs, in ever…
Nov 18, 2026
03f4a26
test(sources, function_pod, pipeline): several tags, keys, inputs and…
Nov 18, 2026
e8ae621
test(operators): async rows equal sync rows in every several-input ca…
Nov 18, 2026
f40e1af
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
8129cfa
test(operators): issue ids for the defects found (ITL-1243, ITL-1247)…
Nov 18, 2026
86a99fc
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
ec9c140
test(operators): type the results helper by Mapping (ITL-1236)
Nov 18, 2026
7217f5c
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
56213e8
test(data_tags, streams, sources): several nullable data-tags, list-v…
Nov 18, 2026
aa7fa71
test(operators): column order of sync rows, async rows and keys, per …
Nov 18, 2026
b66ee60
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
ea4814f
test(sources, pipeline): compare key orders with each other; check bo…
Nov 18, 2026
86ed772
Merge branch 'eywalker/itl-1236-part2-sources-pods-bind' into eywalke…
Nov 18, 2026
d896a18
test(operators): ITL-1249 for the column order xfails; an operator no…
Nov 18, 2026
fb75b54
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
74cef31
Merge branch 'eywalker/itl-1236-part2-sources-pods-bind' into eywalke…
Nov 18, 2026
e1800eb
test(operators): one column-order test per arrangement, xfail only wh…
Nov 18, 2026
4bf0e18
Merge branch 'eywalker/itl-1236-tests-audit-every-argument-that-takes…
Nov 18, 2026
3702e65
test(function_pod, sources, pipeline): weighted outputs, async arrang…
Nov 18, 2026
adf6650
Merge branch 'eywalker/itl-1236-part2-sources-pods-bind' into eywalke…
Nov 18, 2026
88f810b
test(data_tags, streams): narrowing checked against the computed sche…
Nov 18, 2026
41def2c
test(function_pod): pin the output schema as a set of named, typed ou…
Nov 18, 2026
6fd7584
test(function_pod): the ITL-1253 xfail names ITL-1256 and expects onl…
Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -582,6 +582,18 @@ unless the test is about the default (developer, November 16, 2026, ITL-1159). T
sources that take the option (the table, file and database sources); `ListSource`,
`DerivedSource`, `CachedSource` and `PollingSource` do not take it.

Every argument or input that can take several items (column names, a mapping, keywords,
input streams, sources to bind, output keys) is tested (developer, November 17, 2026,
ITL-1236): with two or more items; in more than one order, asserting what the order does or
must not change (rows, column order, content hash, pipeline hash, system-tag column names);
in each role it accepts (plain tag, data, data-tag: build the role you test, whatever the
`tag_only_columns` default above says); on inputs with two or more tag columns AND two or
more data columns, in more than one arrangement (tag order, table layout); and on the sync
and the async path. A test whose streams have one tag and one data column does not test an
operator. The shared family of such inputs is `tests/test_core/operators/several_items.py`;
the inventory is `docs/dev/specs/itl-1236-several-item-test-audit.md`. The docs show the
several-item form of such an argument, not only the one-item form.

---

## Project layout
Expand Down Expand Up @@ -792,7 +804,11 @@ Every pipeline element has two parallel hashes:
identity: the default source_id of the in-memory sources
(<source kind>:<algorithm label>:<hex> of the table hash, used when none is given) and the elements of a ListSource. A downstream stream or node hashes its
pod's identity plus its upstreams' content hashes; a function pod's identity is its uri,
not its body. Only a datagram (one row's tags or data) has a content_hash() that is a
not its body. The uri holds the hash of the output schema, a set of named, typed outputs
in no order (developer, November 17, 2026, ITL-1244): reordering output_keys changes it
only when a key's type changes; a swap among same-typed outputs changes what the function
computes, like a body edit, and needs a version bump (no check on a cache hit yet:
ITL-1253). The input parameters (names and order) are in no part of the uri. Only a datagram (one row's tags or data) has a content_hash() that is a
digest of its values; that row hash, not the node's content hash, is what results are
memoized by. The node's content hash tells apart, inside a shared table, the rows of
pipelines bound to different sources (__node_content_hash).
Expand Down
18 changes: 17 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,18 @@ unless the test is about the default (developer, November 16, 2026, ITL-1159). T
sources that take the option (the table, file and database sources); `ListSource`,
`DerivedSource`, `CachedSource` and `PollingSource` do not take it.

Every argument or input that can take several items (column names, a mapping, keywords,
input streams, sources to bind, output keys) is tested (developer, November 17, 2026,
ITL-1236): with two or more items; in more than one order, asserting what the order does or
must not change (rows, column order, content hash, pipeline hash, system-tag column names);
in each role it accepts (plain tag, data, data-tag: build the role you test, whatever the
`tag_only_columns` default above says); on inputs with two or more tag columns AND two or
more data columns, in more than one arrangement (tag order, table layout); and on the sync
and the async path. A test whose streams have one tag and one data column does not test an
operator. The shared family of such inputs is `tests/test_core/operators/several_items.py`;
the inventory is `docs/dev/specs/itl-1236-several-item-test-audit.md`. The docs show the
several-item form of such an argument, not only the one-item form.

---

## Project layout
Expand Down Expand Up @@ -812,7 +824,11 @@ Every pipeline element has two parallel hashes:
identity: the default `source_id` of the in-memory sources
(`<source kind>:<algorithm label>:<hex>` of the table hash, used when none is given) and the elements of a `ListSource`. A downstream stream or node hashes its
pod's identity plus its upstreams' content hashes; a function pod's identity is its uri,
not its body. Only a datagram (one row's tags or data) has a `content_hash()` that is a
not its body. The uri holds the hash of the output schema, a set of named, typed outputs
in no order (developer, November 17, 2026, ITL-1244): reordering `output_keys` changes it
only when a key's type changes; a swap among same-typed outputs changes what the function
computes, like a body edit, and needs a version bump (no check on a cache hit yet:
ITL-1253). The input parameters (names and order) are in no part of the uri. Only a datagram (one row's tags or data) has a `content_hash()` that is a
digest of its values; that row hash, not the node's content hash, is what results are
memoized by. The node's content hash tells apart, inside a shared table, the rows of
pipelines bound to different sources (`__node_content_hash`).
Expand Down
3 changes: 3 additions & 0 deletions docs/concepts/function-pods.md
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,9 @@ print("function calls:", len(calls))
values. To force recomputation after changing what a function does, bump the major
version, e.g. `@function_pod(output_keys="bmi", version="v1.0")` (the default is
`"v0.0"`; a minor bump such as `"v0.1"` is not enough), or rename the function.
The same holds when you reorder `output_keys` among outputs of the same type: the
output schema is a set of named, typed outputs, so its identity does not change (see
*Change detection* in [Identity & Hashing](identity.md#change-detection)).

!!! note "When the database cannot be read"
A table that does not exist yet reads as empty, and its rows are computed. In a
Expand Down
37 changes: 37 additions & 0 deletions docs/concepts/identity.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,43 @@ print(step.pod(source).pipeline_hash() == original)
When you change what a function computes, bump its major version (or rename it) so that
stale results are not reused.

The output schema is a set of named, typed outputs: each output key with its type, in no
order, like every other schema. `output_keys` also says which position of the returned
tuple each name takes, and that mapping belongs with the body. So reordering the output keys
changes the identity only when a key's type changes. Swapping two keys of the same type
keeps the identity, as a body edit does, and it changes what the function computes: bump
the major version. Until stored results are checked against the current function on a cache
hit (ITL-1253), a run serves the earlier results without a warning. The function's input
parameters (their names and their order) are not part of the identity either.

```python
@function_pod(output_keys=["total", "doubled"])
def pair(age: int) -> tuple[int, int]:
return age + 1, age * 2

original = pair.pod(source).pipeline_hash()

@function_pod(output_keys=["doubled", "total"]) # same types: same identity
def pair(age: int) -> tuple[int, int]:
return age + 1, age * 2

print(pair.pod(source).pipeline_hash() == original)
# True

@function_pod(output_keys=["label", "next_age"])
def describe(age: int) -> tuple[str, int]:
return f"age {age}", age + 1

original = describe.pod(source).pipeline_hash()

@function_pod(output_keys=["next_age", "label"]) # next_age is now a str: new identity
def describe(age: int) -> tuple[str, int]:
return f"age {age}", age + 1

print(describe.pod(source).pipeline_hash() == original)
# False
```

## Your own classes in the identity

A class enters both chains through its **identity**: its kind, its name and the names and
Expand Down
25 changes: 25 additions & 0 deletions docs/dev/design_issues/ITL-1244-OK1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
## `src/orcapod/core/data_function.py`

### OK1 — The order of `output_keys` is in no identity
**Status:** resolved (decided: the identity is right as it is; the cache check is ITL-1253)
**Severity:** high
**Issue:** ITL-1244

Two pods of one function whose `output_keys` differ only in order (`['sum', 'product']`
against `['product', 'sum']`) have one uri, content hash and pipeline hash: the uri holds
the hash of the output schema, and a schema hash sorts its field names. A job with the
second order on the database of a job with the first is served the first one's stored rows,
so the values appear under the other names. Found by the several-item test audit
(ITL-1236).

**Fix:** Decided by the developer (November 17, 2026): the outputs of a function are a set of
named, typed values, so the uri holds the order-free output schema, as before. A swap of
differently typed keys changes the schema and so the uri; a swap of same-typed keys does not.
The positional mapping belongs with the body: a swap among same-typed outputs changes what
the function computes and needs a version bump, as a body edit does. No code changed. The
behaviour is pinned in `tests/test_core/function_pod/test_output_keys_identity.py` (two and
three keys in every order, sync and async, in memory and in a job on an in-memory and a
Delta database, plain tags and data-tags; input parameters pinned as outside the uri) and
documented in `docs/concepts/identity.md` (*Change detection*). A cache hit whose stored
function variation data differs from the current function is still served silently: that
check is ITL-1253 (a strict xfail test states it).
Loading
Loading