Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 46 additions & 19 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -818,19 +818,45 @@ Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, lar
the key of the data context that wrote it (std:v1:default since v0.3). The key names one
generation of hashing; rows of the current data context are the current entries.

Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION)
Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION)
Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION)
Function pipeline — pdb_v1 (PIPELINE_DB_SCHEMA_VERSION)
Function result — rdb_v1, after the type's revision rev1 (RESULT_DB_SCHEMA_VERSION)
Side-effect log — tdb_v1 (TRACKING_DB_SCHEMA_VERSION)
Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION)
Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION)

- The layout version is the last component of a table path. A change to the bookkeeping
columns of a table kind moves its layout version, so the table gets a new path. Never add,
rename or retype a stored column without it.
- The v0.3 layouts (pdb_v2, rdb_v2, tdb_v2, odb_v1, sdb_v1) were redefined before
their first release (ITL-1154): the record-id system tag became text (large_string) without a
layout bump, since no release shipped them. A store written by a development build of v0.3 is
discarded, not migrated (its side-effect logs would deliver again).
- The layout version is the last component of a table path. Every layout marker starts at v1
for the first release, 0.4.0 (developer, November 18, 2026, ITL-1259; 0.3.0 was withdrawn,
so nothing released uses v2). A change to the common bookkeeping columns of a table kind
moves its layout version, so the table gets a new path. Never add, rename or retype a stored
column without it.
- A function result table is at
<function>/<output schema hash>/v<function major>/python.function.v1/rev1/rdb_v1, three
components with three owners (docs/reference/data-function-types.md):
- python.function.v1, the data function type and its MAJOR (what a function means and how it
is run): the last part of the uri, so in every pod and node hash. A new major moves hashes.
- rev1, the REVISION of the type's own record (the pf_var_* / pf_exec_* columns), declared by
the type: path only. It never enters a uri, content or pipeline hash, source-info token or
record id; a new revision is a new result table and moves no hash.
- rdb_v1, the common result columns (input data hash, outputs, their source-info tokens,
__pod_ts, __data_context, record id; result_cache.COMMON_RESULT_COLUMNS), global: path only.
- The test between a new major and the other two: is a row stored before the change still a
correct result for the same input? No: a new major. Yes: a new revision (the type's
columns) or a new rdb version (the common columns). Components are delimited, never parsed
out of one string; the major is not repeated.
- Pipeline tables carry no revision (.../python.function.v1/schema:<pipeline hash>/pdb_v1). A
saved job names the type and major only (data_function_type_id); the revision comes from
the installed library's declaration (declaration_for(type id)).
- A data function type's declaration (PYTHON_FUNCTION, beside its class in core/data_function.py;
the classes in core/data_function_types.py, a leaf module) is the
source of truth: type name, major, revision, identity recipe, run rules, and its record
(every column with name, Python and Arrow type, nullability, meaning, and for a hash its
label CONSTANT and what it covers). The data function builds its stored columns from it;
never a second list. Each declaration is written out as a snapshot
(src/orcapod/data_function_types/<type>/v<major>-rev<revision>.json, pinned in
released.json; scripts/data_function_types.py --check, --write, and --release, run by
generate_golden_hashes.py --release). An open snapshot must equal the code; a released one
never changes: a change to what a type stores is a new revision, to what its functions mean
a new major.
- Every new stored table kind writes __data_context (contexts/stored.with_data_context), and
every read or lookup keeps rows of the current data context only
(keep_rows_of_data_context, or a lookup constraint on the column).
Expand All @@ -840,17 +866,18 @@ generation of hashing; rows of the current data context are the current entries.
side effects (a SideEffectPod node, or a function node whose pod takes the invocation
context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of
node goes through that function; never decide it locally.
- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between versions. find_stale_tables(..., name_path=...) lists the
- A table of another layout is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between builds. find_stale_tables(..., name_path=...) lists the
store under the hash-free beginning of the node's path (ArrowDatabaseProtocol.list_table_paths;
<function>, side_effect_function/<function>, side_effects/<pod>, relative to the node's
database) and every table there whose last component is not in
contexts/stored.CURRENT_LAYOUTS is pre-v0.3. A new layout constant must be added to
CURRENT_LAYOUTS, and a superseded one to the OLD_*_SCHEMA_VERSIONS tuples. A store that
cannot list raises TableListingNotSupportedError: side-effecting nodes then warn and
proceed (developer, November 11, 2026). Operator caches and cached sources have no stable
name; PipelineJob.run() reports one total for them. Every database class implements
list_table_paths.
database; <prefix>/source for a cached source) and every table there whose last component
is not in contexts/stored.CURRENT_LAYOUTS is of another layout (tables_of_another_layout).
A new layout constant must be added to CURRENT_LAYOUTS. A store that cannot list raises
TableListingNotSupportedError: side-effecting nodes then warn and proceed (developer,
November 11, 2026). Every database class implements list_table_paths. Stores of the
versions before 0.4 are not looked for otherwise (no old-layout names, no layout-less
paths, no job-wide total): nobody has one, and their layouts had the names the first
release uses (ITL-1259).
- A released data context manifest (contexts/data/<version>.json) is frozen: the digest of
its canonical JSON without golden is pinned in contexts/released_manifests.json
(manifest), with one releases line per library release (ITL-946). Only its golden
Expand Down
65 changes: 47 additions & 18 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -847,18 +847,46 @@ current data context are the current entries.

| Table | Layout (last path component) | Constant |
|---|---|---|
| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` |
| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` |
| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` |
| Function pipeline | `pdb_v1` | `PIPELINE_DB_SCHEMA_VERSION` |
| Function result | `rdb_v1` (after the type's revision, `rev1`) | `RESULT_DB_SCHEMA_VERSION` |
| Side-effect log | `tdb_v1` | `TRACKING_DB_SCHEMA_VERSION` |
| Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` |
| Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` |

- A change to the bookkeeping columns of a table kind moves its layout version, so the table
gets a new path. Never add, rename or retype a stored column without it.
- The v0.3 layouts (`pdb_v2`, `rdb_v2`, `tdb_v2`, `odb_v1`, `sdb_v1`) were redefined before
their first release (ITL-1154): the record-id system tag became text (`large_string`) without a
layout bump, since no release shipped them. A store written by a development build of v0.3 is
discarded, not migrated (its side-effect logs would deliver again).
- Every layout marker starts at `v1` for the first release, 0.4.0 (developer, November 18,
2026, ITL-1259; 0.3.0 was withdrawn, so nothing released uses `v2`). A change to the common
bookkeeping columns of a table kind moves its layout version, so the table gets a new path.
Never add, rename or retype a stored column without it.
- A function result table is at
`<function>/<output schema hash>/v<function major>/python.function.v1/rev1/rdb_v1`, three
components with three owners (`docs/reference/data-function-types.md`):
- `python.function.v1`, the data function type and its MAJOR (what a function means and how
it is run): the last part of the uri, so in every pod and node hash. A new major moves
hashes.
- `rev1`, the REVISION of the type's own record (the `pf_var_*` / `pf_exec_*` columns),
declared by the type: path only. It never enters a uri, content or pipeline hash,
source-info token or record id; a new revision is a new result table and moves no hash.
- `rdb_v1`, the common result columns (input data hash, outputs, their source-info
tokens, `__pod_ts`, `__data_context`, record id; `result_cache.COMMON_RESULT_COLUMNS`),
global: path only.
- The test between a new major and the other two: is a row stored before the change still a
correct result for the same input? No: a new major. Yes: a new revision (the type's
columns) or a new `rdb` version (the common columns). Components are delimited, never
parsed out of one string; the major is not repeated.
- Pipeline tables carry no revision (`.../python.function.v1/schema:<pipeline hash>/pdb_v1`).
A saved job names the type and major only (`data_function_type_id`); the revision comes
from the installed library's declaration (`declaration_for(type id)`).
- A data function type's declaration (`PYTHON_FUNCTION`, beside its class in
`core/data_function.py`; the classes in `core/data_function_types.py`, a leaf module) is
the source of truth: type name, major, revision, identity recipe, run rules, and its record
(every column with name, Python and Arrow type, nullability, meaning, and for a hash its
label CONSTANT and what it covers). The data function builds its stored columns from it;
never a second list. Each declaration is written out as a snapshot
(`src/orcapod/data_function_types/<type>/v<major>-rev<revision>.json`, pinned in
`released.json`; `scripts/data_function_types.py --check`, `--write`, and `--release`, run
by `generate_golden_hashes.py --release`). An open snapshot must equal the code; a released
one never changes: a change to what a type stores is a new revision, to what its functions
mean a new major.
- Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`),
and every read or lookup keeps rows of the current data context only
(`keep_rows_of_data_context`, or a lookup constraint on the column).
Expand All @@ -868,17 +896,18 @@ current data context are the current entries.
side effects (a `SideEffectPod` node, or a function node whose pod takes the invocation
context, `ctx_arg`), raise `SideEffectLogDataContextError` before any delivery. A new kind
of node goes through that function; never decide it locally.
- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between versions. `find_stale_tables(..., name_path=...)` lists the
- A table of another layout is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between builds. `find_stale_tables(..., name_path=...)` lists the
store under the hash-free beginning of the node's path (`ArrowDatabaseProtocol.list_table_paths`;
`<function>`, `side_effect_function/<function>`, `side_effects/<pod>`, relative to the
node's database) and every table there whose last component is not in
`contexts/stored.CURRENT_LAYOUTS` is pre-v0.3. A new layout constant must be added to
`CURRENT_LAYOUTS`, and a superseded one to the `OLD_*_SCHEMA_VERSIONS` tuples. A store that
cannot list raises `TableListingNotSupportedError`: side-effecting nodes then warn and
proceed (developer, November 11, 2026). Operator caches and cached sources have no stable
name; `PipelineJob.run()` reports one total for them. Every database class implements
`list_table_paths`.
node's database; `<prefix>/source` for a cached source) and every table there whose last
component is not in `contexts/stored.CURRENT_LAYOUTS` is of another layout
(`tables_of_another_layout`). A new layout constant must be added to `CURRENT_LAYOUTS`. A
store that cannot list raises `TableListingNotSupportedError`: side-effecting nodes then
warn and proceed (developer, November 11, 2026). Every database class implements
`list_table_paths`. Stores of the versions before 0.4 are not looked for otherwise (no
old-layout names, no layout-less paths, no job-wide total): nobody has one, and their
layouts had the names the first release uses (ITL-1259).
- A released data context manifest (`contexts/data/<version>.json`) is frozen: the digest of
its canonical JSON without `golden` is pinned in `contexts/released_manifests.json`
(`manifest`), with one `releases` line per library release (ITL-946). Only its `golden`
Expand Down
6 changes: 4 additions & 2 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,10 @@ Do these on `main` before the Release workflow runs:
the `CHANGELOG.md` heading. It appends the release revision to the `golden` history of
the current data context's manifest, writes its digest under `releases` in
`src/orcapod/contexts/released_manifests.json`, and at the data context's first release
sets `released_in` and the `manifest` pin. It computes no hash and refuses a version
already released. Commit both files with the release pull request; the release is tagged
sets `released_in` and the `manifest` pin. It also pins every open data function type
snapshot (`scripts/data_function_types.py --release`: the sha256 and the version go into
`src/orcapod/data_function_types/released.json`; ITL-1259). It computes no hash and
refuses a version already released. Commit these files with the release pull request; the release is tagged
on a commit that contains them. The Release workflow's build job checks this before it
creates the tag (`scripts/check_golden_tags.py --release-version X.Y.Z`) and stops if the
release is not recorded, or if a golden file was added after the release step (the
Expand Down
23 changes: 23 additions & 0 deletions changelog.d/ITL-1259.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
### Breaking changes and migration

- The Python function type is `python.function.v1`, and every stored table's layout marker
starts at `v1` (`pdb_v1`, `rdb_v1`, `tdb_v1`, `odb_v1`, `sdb_v1`): the first release's
names (ITL-1259). Every function pod has a new identity and its results are computed once
again. A saved job that names `python.function.v0` loads read only and its functions
cannot be bound: rebuild the pipeline to run it.

### Added

- A data function type declares what a function of the type means and what its result
table stores (`orcapod.core.data_function_types`, `PYTHON_FUNCTION`; ITL-1259). A function
result table is at `.../python.function.v1/rev1/rdb_v1`: the type and its major (in the
identity), the revision of the type's record and the layout of the common columns (both in
the path only). Each released declaration is pinned as a snapshot. See
`docs/reference/data-function-types.md`.

### Changed

- Stores written by Orcapod before 0.4 are no longer looked for: no old layout names, no
layout-less paths, no job-wide total of old tables (ITL-1259; nobody has such a store).
A node still reports, and a side-effecting node still refuses, tables of another layout
under its own name, and rows of another data context.
Loading
Loading