Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,20 @@ Orcapod stores or embeds in a name is pinned in the golden folder of the current
generator writes the new value. `--release 0.4.0` freezes v1. Any version published
before it, a pre-release included, would freeze v1 there: the Release workflow requires
the release step for every version. A withdrawn version is never released again.
- Provisional operators are outside the freeze (developer, November 18, 2026, ITL-1266):
`orcapod.operators.PROVISIONAL_OPERATORS` (`core/operators/provisional.py`) is the one
list (name to the issue deciding its identity; today `PolarsFilter`), and each listed class
sets `PROVISIONAL = True` (a test checks they agree). Building one issues
`ProvisionalHashWarning` once per operator per process. A golden case that depends on one
declares it (`@case(..., provisional=("PolarsFilter",))`; `compute_group_kinds` refuses a
case that builds an undeclared one); its entries live in
`data/<folder>/provisional/<group>.json`, listed in `golden.provisional` of the manifest,
outside the history, so no release freezes them. They are checked like every other value
and change only with the developer's approval of the exact list, without a new data
context. `generate_golden_hashes.py --relocate` moves entries, values unchanged, between
frozen and provisional files; an operator leaving the group has its entries relocated into
a frozen file before the release step. Never put a value that depends on a provisional
operator in a frozen file.
- The unit of immutability is the golden file. The manifest's `golden.history`
(`src/orcapod/contexts/data/<version>.json`) lists every file with the sha256 of its bytes;
a release appends a revision with `added: {}`, pinned in
Expand Down
14 changes: 14 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,20 @@ Orcapod stores or embeds in a name is pinned in the golden folder of the current
generator writes the new value. `--release 0.4.0` freezes v1. Any version published
before it, a pre-release included, would freeze v1 there: the Release workflow requires
the release step for every version. A withdrawn version is never released again.
- Provisional operators are outside the freeze (developer, November 18, 2026, ITL-1266):
`orcapod.operators.PROVISIONAL_OPERATORS` (`core/operators/provisional.py`) is the one
list (name to the issue deciding its identity; today `PolarsFilter`), and each listed class
sets `PROVISIONAL = True` (a test checks they agree). Building one issues
`ProvisionalHashWarning` once per operator per process. A golden case that depends on one
declares it (`@case(..., provisional=("PolarsFilter",))`; `compute_group_kinds` refuses a
case that builds an undeclared one); its entries live in
`data/<folder>/provisional/<group>.json`, listed in `golden.provisional` of the manifest,
outside the history, so no release freezes them. They are checked like every other value
and change only with the developer's approval of the exact list, without a new data
context. `generate_golden_hashes.py --relocate` moves entries, values unchanged, between
frozen and provisional files; an operator leaving the group has its entries relocated into
a frozen file before the release step. Never put a value that depends on a provisional
operator in a frozen file.
- The unit of immutability is the golden file. The manifest's `golden.history`
(`src/orcapod/contexts/data/<version>.json`) lists every file with the sha256 of its bytes;
a release appends a revision with `added: {}`, pinned in
Expand Down
7 changes: 7 additions & 0 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,13 @@ against are in `docs/reference/golden-hashes.md`.
step of `0.4.0` freezes v1. Every published version needs the release step, a pre-release
included (the gate below refuses to tag one without it), so **publishing `0.4.0rc1` would
freeze v1 at the release candidate**: plan the release around it.
- Provisional operators (`orcapod.operators.PROVISIONAL_OPERATORS`, ITL-1266) are outside
the freeze: the golden values of the cases that depend on one are in
`tests/test_hashing/golden/data/v1/provisional/`, which no release step lists or pins. An
operator that stops being provisional in a release leaves the list (with its `PROVISIONAL`
marker and its golden cases' declarations) before the release step, and
`generate_golden_hashes.py --relocate` moves its golden values, unchanged, into a frozen
file, so that the release step freezes them.
- Release tags are protected on `metamorphic-labs/orcapod-python` (and on
`nauticalab/orcapod-python`, the release repository up to `0.3.0`): the repository ruleset
`protect-release-tags` (target `refs/tags/v*`; rules deletion, update and non-fast-forward;
Expand Down
11 changes: 11 additions & 0 deletions changelog.d/ITL-1266.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
### Added

- Provisional operators: `orcapod.operators.PROVISIONAL_OPERATORS` lists the operators whose
identity is not frozen with data context v1 (ITL-1266). Today it holds only `PolarsFilter`
(its identity is decided under ITL-1247). Building a provisional operator, directly or
through a stream method such as `polars_filter`, issues `ProvisionalHashWarning` once per
operator per process. A later release may change its identity, and with it the stored
paths and record ids of its results and of every step downstream of it; those results are
then computed again (nothing stored is lost or wrong). The hash stability that data context
v1 promises from 0.4.0 holds for a pipeline only when it contains no provisional operator.
See "Provisional operators" in `docs/concepts/operators.md`.
43 changes: 43 additions & 0 deletions docs/concepts/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,11 @@ A new name already taken by a column that is not renamed is refused too.

### PolarsFilter

!!! warning "Provisional"
`PolarsFilter` is a [provisional operator](#provisional-operators): its identity is not
frozen with data context v1 and may change in a later release (ITL-1247). Building one
issues `ProvisionalHashWarning` once per process.

Filter rows using Polars expressions:

```python
Expand Down Expand Up @@ -714,6 +719,44 @@ large variants as described in [Arrow types of operator outputs](#arrow-types-of
that is `None` raises `ValueError` naming the operator, the column and the row's tag, whatever
`fail_on_miss` is.

## Provisional operators

Data context v1 freezes the hashes of Orcapod at the 0.4.0 release: from then on a pipeline
keeps its stored paths, record ids and default source ids from one release to the next. A
provisional operator is outside that freeze. Its identity is still being decided, so a later
release may change it, and with it the content and pipeline hashes, stored paths and record
ids of its results and of every step downstream of it. A pipeline that contains one keeps its
hashes across releases only until such a change: after it, those steps compute their results
again instead of reusing them. Nothing stored is lost or wrong; the earlier rows stay where
they are.

The provisional operators are listed in one place:

```python
from orcapod.operators import PROVISIONAL_OPERATORS

print(sorted(PROVISIONAL_OPERATORS))
# ['PolarsFilter']
```

Each name maps to the issue that decides the operator's identity (`PolarsFilter`: whether
the order of its predicates belongs to it). Building a provisional operator, directly or
through a stream method such as `polars_filter`, issues `orcapod.errors.ProvisionalHashWarning`
once per operator per process. To silence it once you have taken note, filter its class after
importing Orcapod:

```python
import warnings

from orcapod.errors import ProvisionalHashWarning

warnings.filterwarnings("ignore", category=ProvisionalHashWarning)
```

An operator leaves the group at a release: from that release on its identity is frozen like
every other operator's. Its golden values then move into the frozen golden files (see
[Golden hashes](../reference/golden-hashes.md#provisional-files)).

## Arrow types of operator outputs

`output_schema()` reports Python types, but every column also has an Arrow type, and
Expand Down
78 changes: 78 additions & 0 deletions docs/dev/scripts/itl_1266_moved_entries.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
"""Prove that ITL-1266 only moved golden entries between files, values unchanged.

Usage::

uv run python docs/dev/scripts/itl_1266_moved_entries.py <base revision>

Reads every golden file of data context v1 (frozen and provisional) at the base revision
(git) and in the working tree, and compares: the same entry names, the same values, and
the list of entries whose file changed. Exits 1 on any other difference.
"""

from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path

FOLDER = "tests/test_hashing/golden/data/v1"


def _at(revision: str) -> dict[str, tuple[str, object]]:
names = subprocess.run(
["git", "ls-tree", "-r", "--name-only", revision, FOLDER],
check=True,
capture_output=True,
text=True,
).stdout.split()
out: dict[str, tuple[str, object]] = {}
for name in names:
text = subprocess.run(
["git", "show", f"{revision}:{name}"],
check=True,
capture_output=True,
text=True,
).stdout
for entry, value in json.loads(text).items():
assert entry not in out, entry
out[entry] = (name.removeprefix(FOLDER + "/"), value)
return out


def _tree() -> dict[str, tuple[str, object]]:
out: dict[str, tuple[str, object]] = {}
for path in sorted(Path(FOLDER).rglob("*.json")):
for entry, value in json.loads(path.read_text()).items():
assert entry not in out, entry
out[entry] = (path.relative_to(FOLDER).as_posix(), value)
return out


def main() -> int:
before, after = _at(sys.argv[1]), _tree()
failures = 0
if set(before) != set(after):
print("entry names differ:", sorted(set(before) ^ set(after)))
failures += 1
changed_value = sorted(
e for e in set(before) & set(after) if before[e][1] != after[e][1]
)
if changed_value:
print("values differ:", changed_value)
failures += 1
moved: dict[tuple[str, str], list[str]] = {}
for entry in sorted(set(before) & set(after)):
if before[entry][0] != after[entry][0]:
moved.setdefault((before[entry][0], after[entry][0]), []).append(entry)
print(f"entries before: {len(before)}, after: {len(after)}")
print(f"values changed: {len(changed_value)}")
for (source, target), entries in sorted(moved.items()):
cases = sorted({e.split(".", 1)[0] for e in entries})
print(f"moved {len(entries)}: {source} -> {target}; cases {cases}")
print(f"moved in all: {sum(len(e) for e in moved.values())}")
return 1 if failures else 0


if __name__ == "__main__":
raise SystemExit(main())
80 changes: 75 additions & 5 deletions docs/reference/golden-hashes.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,16 @@ wanted on purpose starts a new data context version. It never edits a released v
entries that move, each explained. 0.4.0 freezes `v1`. See
[Withdrawing a release](#withdrawing-a-release).

!!! note "Provisional operators are outside the freeze"
A provisional operator (`orcapod.operators.PROVISIONAL_OPERATORS`, today only
`PolarsFilter`) has an identity that no release freezes yet (ITL-1266). The golden values
of every case that depends on one, its own hashes and those of every step downstream of
it, are in the provisional files, `data/v1/provisional/<group>.json`. They are computed
and checked on every run like every other value, but no release lists them: they can
change in a later release without a new data context, with the developer's approval of
the exact list. So the promise on this page holds for a pipeline only when it contains
no provisional operator. See [Provisional files](#provisional-files).

## The pieces at a glance

The mechanism has five layers. Each one is checked against the layer below it, and the last
Expand Down Expand Up @@ -68,12 +78,16 @@ tests/test_hashing/golden/data/
semantic-01.json
torch-01.json
semantic-02.json # added after 0.4.0: new semantic entries only
provisional/ # never frozen: see "Provisional files"
pipeline.json
semantic.json
```

- A file is named after the group of cases it pins plus a two-digit counter:
`<group>-NN.json`, where the group is one of `cases.GROUPS` (`semantic`, `schema`,
`arrow`, `datagram`, `pipeline`, `torch`). Names carry no date, issue or release; that
information lives in the manifest. The folder holds nothing else.
information lives in the manifest. The folder holds nothing else but the `provisional`
subfolder ([Provisional files](#provisional-files)).
- A file maps entry names to values. An entry name appears in exactly one file of the
folder, whatever its group.
- A file listed by a released revision is never edited. New values go into the group's next
Expand All @@ -99,7 +113,8 @@ tests/test_hashing/golden/data/
"previous": "d002...", "digest": "d003..."},
{"date": "2026-12-09", "release": "0.4.1", "added": {},
"previous": "d003...", "digest": "d004..."}
]
],
"provisional": {"pipeline.json": "c86b...", "semantic.json": "2f1d..."}
}
```

Expand All @@ -112,6 +127,8 @@ tests/test_hashing/golden/data/
file they list, are fixed.
- `date` is the date the revision was last changed. It is informational: nothing can verify
it.
- `provisional` lists the provisional files, each with the sha256 of its bytes, outside the
history: no revision lists them, so no release freezes them.

### The pins: `src/orcapod/contexts/released_manifests.json`

Expand All @@ -130,6 +147,37 @@ tests/test_hashing/golden/data/
the `manifest` pin and release revision digest (`revision`) that release shipped. A
withdrawn release has no `releases` line and no release revision, so it freezes nothing.

### Provisional files

`tests/test_hashing/golden/data/v1/provisional/<group>.json` holds the values of the cases
that depend on a provisional operator (ITL-1266), one file per group and no counter: such a
file is never frozen, and the generator rewrites it in place.

- **Which entries.** A case declares the provisional operators it depends on:
`@case("pipeline", "polars_filter", provisional=("PolarsFilter",))`. The generator runs
every case while Orcapod records which provisional operators it builds
(`orcapod.utils.provisional.recording_provisional_uses`), and refuses a case that builds
one without declaring it, or that declares a name `PROVISIONAL_OPERATORS` does not hold.
The golden test checks that every entry is in the kind of file its case declares. So a
value that depends on a provisional operator cannot be pinned in a frozen file: neither
through a stream method that builds it (`stream.polars_filter`) nor in a step downstream
of it.
- **Today:** the `pipeline` cases `polars_filter`, `polars_filter_expr` and
`polars_filter_expr_job`, and the `semantic` case `polars_expr`: the `polars_expr_v1`
hashes, which only `PolarsFilter`'s identity feeds in the library. The 86 entries moved
out of `pipeline-01.json` and `semantic-01.json`, values unchanged, before `v1` was frozen.
- **What holds.** Every provisional value is computed and compared on every CI run, so a
change nobody intended fails. An intended change needs the developer's approval of the
exact list of entries that move, each explained (`--remove`, then `--add`), but no new
data context, before or after a release. `--release` neither lists nor pins a provisional
file, the manifest pin leaves out `golden`, and the tag check does not compare them.
- **Leaving the group.** When an operator stops being provisional, at a release, it leaves
`PROVISIONAL_OPERATORS`, its `PROVISIONAL` marker and its cases' declarations go, and
`generate_golden_hashes.py --relocate` moves its entries, values unchanged, into the open
frozen file of their group. The `--release` step then freezes them.
- **Joining the group.** Only an operator whose values no release lists can become
provisional: `--relocate` refuses to move an entry out of a file a release lists.

### Digest definitions

| Value | Computed as |
Expand Down Expand Up @@ -161,7 +209,12 @@ writes go through `tests/test_hashing/golden/generate_golden_hashes.py`.
existing value, and it writes nothing when the tree already breaks a rule.
3. The diff shows only added entries or a new file, and the edit to the open revision.
4. `--check` computes every case and reports moved, unpinned and no longer produced entries
without writing anything.
without writing anything, and entries pinned in the wrong kind of file (a provisional
case in a frozen file, or the other way round).

An entry of a case that declares a provisional operator goes to the provisional file of its
group instead ([Provisional files](#provisional-files)). `--relocate` moves pinned entries
between the frozen and the provisional files, values unchanged, as their cases declare.

Before a release, a value that no release lists can change only with the developer's
approval and a written migration assessment. `--remove <entry>` (repeatable) takes the entry
Expand All @@ -188,6 +241,9 @@ edit and the generator refuses to write over it.
4. The tag and the PyPI wheel and sdist, which carry `v1.json` and `released_manifests.json`,
are records of what shipped, outside the repository.

An operator that stops being provisional in this release has its entries moved into a frozen
file (`--relocate`) before the release step, so that the release freezes them.

Every published version needs the release step, a pre-release included: the release gate
refuses to tag a version without it. So the first version of a data context published on
PyPI freezes it; for `v1`, publishing `0.4.0rc1` would freeze `v1` at the release candidate.
Expand Down Expand Up @@ -384,8 +440,19 @@ so the bytes that are hashed are the bytes that were committed (4).

`tests/test_hashing/golden/test_golden_hashes.py` and the generator

- **Checks:** the folder holds only `<group>-NN.json` files of known groups, and an entry name
appears in only one of them (8).
- **Checks:** the folder holds only `<group>-NN.json` files of known groups and the
`provisional` subfolder of `<group>.json` files, and an entry name appears in only one of
all of them (8).

### Placement check

`tests/test_hashing/golden/cases.py` (`compute_group_kinds`) and
`tests/test_hashing/golden/test_golden_hashes.py`

- **Checks:** a case that builds a provisional operator declares it; every entry of a case
that declares one is in a provisional file, and every other entry in a frozen file.
- **Protects against:** a value that depends on a provisional operator frozen by a release,
and a frozen value moved out of the freeze without a declaration.

## Residual risks and assumptions

Expand All @@ -398,13 +465,16 @@ so the bytes that are hashed are the bytes that were committed (4).
| Values on `main` after the last release are open | An unreleased commit has no guarantee | Use a released version for anything that must stay stable |
| Optional groups skip without their package | `torch` values are unchecked where torch is missing | CI installs every extra |
| The `date` field cannot be verified | Informational | Order and integrity come from the chain and the tags |
| Provisional values are outside the freeze | A pipeline with a provisional operator may compute its results again after an upgrade | `ProvisionalHashWarning` when the operator is built; the list in `PROVISIONAL_OPERATORS` and on the operators page |

A deliberate attacker with merge rights who also rewrites history and tags is beyond what any
check inside the repository can stop; the PyPI artifacts remain the record of what shipped.

## Rules

- A released golden file never changes, and is never removed or renamed.
- A value of a case that depends on a provisional operator is pinned in a provisional file,
never in a frozen one; no release freezes a provisional file.
- `v1` is open until 0.4.0 (0.3.0 was withdrawn on 2026-11-17): until then, a change to a
`v1` value needs the developer's approval of the exact list of entries that move, each
explained. Never edit or regenerate a golden file to make a test pass.
Expand Down
Loading
Loading