diff --git a/.zed/rules b/.zed/rules index 915a08c69..3fe851042 100644 --- a/.zed/rules +++ b/.zed/rules @@ -818,19 +818,45 @@ Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, lar the key of the data context that wrote it (std:v1:default since v0.3). The key names one generation of hashing; rows of the current data context are the current entries. - Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION) - Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION) - Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION) + Function pipeline — pdb_v1 (PIPELINE_DB_SCHEMA_VERSION) + Function result — rdb_v1, after the type's revision rev1 (RESULT_DB_SCHEMA_VERSION) + Side-effect log — tdb_v1 (TRACKING_DB_SCHEMA_VERSION) Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION) Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION) -- The layout version is the last component of a table path. A change to the bookkeeping - columns of a table kind moves its layout version, so the table gets a new path. Never add, - rename or retype a stored column without it. -- The v0.3 layouts (pdb_v2, rdb_v2, tdb_v2, odb_v1, sdb_v1) were redefined before - their first release (ITL-1154): the record-id system tag became text (large_string) without a - layout bump, since no release shipped them. A store written by a development build of v0.3 is - discarded, not migrated (its side-effect logs would deliver again). +- The layout version is the last component of a table path. Every layout marker starts at v1 + for the first release, 0.4.0 (developer, November 18, 2026, ITL-1259; 0.3.0 was withdrawn, + so nothing released uses v2). A change to the common bookkeeping columns of a table kind + moves its layout version, so the table gets a new path. Never add, rename or retype a stored + column without it. +- A function result table is at + //v/python.function.v1/rev1/rdb_v1, three + components with three owners (docs/reference/data-function-types.md): + - python.function.v1, the data function type and its MAJOR (what a function means and how it + is run): the last part of the uri, so in every pod and node hash. A new major moves hashes. + - rev1, the REVISION of the type's own record (the pf_var_* / pf_exec_* columns), declared by + the type: path only. It never enters a uri, content or pipeline hash, source-info token or + record id; a new revision is a new result table and moves no hash. + - rdb_v1, the common result columns (input data hash, outputs, their source-info tokens, + __pod_ts, __data_context, record id; result_cache.COMMON_RESULT_COLUMNS), global: path only. + - The test between a new major and the other two: is a row stored before the change still a + correct result for the same input? No: a new major. Yes: a new revision (the type's + columns) or a new rdb version (the common columns). Components are delimited, never parsed + out of one string; the major is not repeated. + - Pipeline tables carry no revision (.../python.function.v1/schema:/pdb_v1). A + saved job names the type and major only (data_function_type_id); the revision comes from + the installed library's declaration (declaration_for(type id)). +- A data function type's declaration (PYTHON_FUNCTION, beside its class in core/data_function.py; + the classes in core/data_function_types.py, a leaf module) is the + source of truth: type name, major, revision, identity recipe, run rules, and its record + (every column with name, Python and Arrow type, nullability, meaning, and for a hash its + label CONSTANT and what it covers). The data function builds its stored columns from it; + never a second list. Each declaration is written out as a snapshot + (src/orcapod/data_function_types//v-rev.json, pinned in + released.json; scripts/data_function_types.py --check, --write, and --release, run by + generate_golden_hashes.py --release). An open snapshot must equal the code; a released one + never changes: a change to what a type stores is a new revision, to what its functions mean + a new major. - Every new stored table kind writes __data_context (contexts/stored.with_data_context), and every read or lookup keeps rows of the current data context only (keep_rows_of_data_context, or a lookup constraint on the column). @@ -840,17 +866,18 @@ generation of hashing; rows of the current data context are the current entries. side effects (a SideEffectPod node, or a function node whose pod takes the invocation context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of node goes through that function; never decide it locally. -- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the - hashes in a path differ between versions. find_stale_tables(..., name_path=...) lists the +- A table of another layout is found by NAME, never by a computed hash (ITL-954): the + hashes in a path differ between builds. find_stale_tables(..., name_path=...) lists the store under the hash-free beginning of the node's path (ArrowDatabaseProtocol.list_table_paths; , side_effect_function/, side_effects/, relative to the node's - database) and every table there whose last component is not in - contexts/stored.CURRENT_LAYOUTS is pre-v0.3. A new layout constant must be added to - CURRENT_LAYOUTS, and a superseded one to the OLD_*_SCHEMA_VERSIONS tuples. A store that - cannot list raises TableListingNotSupportedError: side-effecting nodes then warn and - proceed (developer, November 11, 2026). Operator caches and cached sources have no stable - name; PipelineJob.run() reports one total for them. Every database class implements - list_table_paths. + database; /source for a cached source) and every table there whose last component + is not in contexts/stored.CURRENT_LAYOUTS is of another layout (tables_of_another_layout). + A new layout constant must be added to CURRENT_LAYOUTS. A store that cannot list raises + TableListingNotSupportedError: side-effecting nodes then warn and proceed (developer, + November 11, 2026). Every database class implements list_table_paths. Stores of the + versions before 0.4 are not looked for otherwise (no old-layout names, no layout-less + paths, no job-wide total): nobody has one, and their layouts had the names the first + release uses (ITL-1259). - A released data context manifest (contexts/data/.json) is frozen: the digest of its canonical JSON without golden is pinned in contexts/released_manifests.json (manifest), with one releases line per library release (ITL-946). Only its golden diff --git a/CLAUDE.md b/CLAUDE.md index a06981d07..6e9160ad1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -847,18 +847,46 @@ current data context are the current entries. | Table | Layout (last path component) | Constant | |---|---|---| -| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` | -| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` | -| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` | +| Function pipeline | `pdb_v1` | `PIPELINE_DB_SCHEMA_VERSION` | +| Function result | `rdb_v1` (after the type's revision, `rev1`) | `RESULT_DB_SCHEMA_VERSION` | +| Side-effect log | `tdb_v1` | `TRACKING_DB_SCHEMA_VERSION` | | Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` | | Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` | -- A change to the bookkeeping columns of a table kind moves its layout version, so the table - gets a new path. Never add, rename or retype a stored column without it. -- The v0.3 layouts (`pdb_v2`, `rdb_v2`, `tdb_v2`, `odb_v1`, `sdb_v1`) were redefined before - their first release (ITL-1154): the record-id system tag became text (`large_string`) without a - layout bump, since no release shipped them. A store written by a development build of v0.3 is - discarded, not migrated (its side-effect logs would deliver again). +- Every layout marker starts at `v1` for the first release, 0.4.0 (developer, November 18, + 2026, ITL-1259; 0.3.0 was withdrawn, so nothing released uses `v2`). A change to the common + bookkeeping columns of a table kind moves its layout version, so the table gets a new path. + Never add, rename or retype a stored column without it. +- A function result table is at + `//v/python.function.v1/rev1/rdb_v1`, three + components with three owners (`docs/reference/data-function-types.md`): + - `python.function.v1`, the data function type and its MAJOR (what a function means and how + it is run): the last part of the uri, so in every pod and node hash. A new major moves + hashes. + - `rev1`, the REVISION of the type's own record (the `pf_var_*` / `pf_exec_*` columns), + declared by the type: path only. It never enters a uri, content or pipeline hash, + source-info token or record id; a new revision is a new result table and moves no hash. + - `rdb_v1`, the common result columns (input data hash, outputs, their source-info + tokens, `__pod_ts`, `__data_context`, record id; `result_cache.COMMON_RESULT_COLUMNS`), + global: path only. + - The test between a new major and the other two: is a row stored before the change still a + correct result for the same input? No: a new major. Yes: a new revision (the type's + columns) or a new `rdb` version (the common columns). Components are delimited, never + parsed out of one string; the major is not repeated. + - Pipeline tables carry no revision (`.../python.function.v1/schema:/pdb_v1`). + A saved job names the type and major only (`data_function_type_id`); the revision comes + from the installed library's declaration (`declaration_for(type id)`). +- A data function type's declaration (`PYTHON_FUNCTION`, beside its class in + `core/data_function.py`; the classes in `core/data_function_types.py`, a leaf module) is + the source of truth: type name, major, revision, identity recipe, run rules, and its record + (every column with name, Python and Arrow type, nullability, meaning, and for a hash its + label CONSTANT and what it covers). The data function builds its stored columns from it; + never a second list. Each declaration is written out as a snapshot + (`src/orcapod/data_function_types//v-rev.json`, pinned in + `released.json`; `scripts/data_function_types.py --check`, `--write`, and `--release`, run + by `generate_golden_hashes.py --release`). An open snapshot must equal the code; a released + one never changes: a change to what a type stores is a new revision, to what its functions + mean a new major. - Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`), and every read or lookup keeps rows of the current data context only (`keep_rows_of_data_context`, or a lookup constraint on the column). @@ -868,17 +896,18 @@ current data context are the current entries. side effects (a `SideEffectPod` node, or a function node whose pod takes the invocation context, `ctx_arg`), raise `SideEffectLogDataContextError` before any delivery. A new kind of node goes through that function; never decide it locally. -- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the - hashes in a path differ between versions. `find_stale_tables(..., name_path=...)` lists the +- A table of another layout is found by NAME, never by a computed hash (ITL-954): the + hashes in a path differ between builds. `find_stale_tables(..., name_path=...)` lists the store under the hash-free beginning of the node's path (`ArrowDatabaseProtocol.list_table_paths`; ``, `side_effect_function/`, `side_effects/`, relative to the - node's database) and every table there whose last component is not in - `contexts/stored.CURRENT_LAYOUTS` is pre-v0.3. A new layout constant must be added to - `CURRENT_LAYOUTS`, and a superseded one to the `OLD_*_SCHEMA_VERSIONS` tuples. A store that - cannot list raises `TableListingNotSupportedError`: side-effecting nodes then warn and - proceed (developer, November 11, 2026). Operator caches and cached sources have no stable - name; `PipelineJob.run()` reports one total for them. Every database class implements - `list_table_paths`. + node's database; `/source` for a cached source) and every table there whose last + component is not in `contexts/stored.CURRENT_LAYOUTS` is of another layout + (`tables_of_another_layout`). A new layout constant must be added to `CURRENT_LAYOUTS`. A + store that cannot list raises `TableListingNotSupportedError`: side-effecting nodes then + warn and proceed (developer, November 11, 2026). Every database class implements + `list_table_paths`. Stores of the versions before 0.4 are not looked for otherwise (no + old-layout names, no layout-less paths, no job-wide total): nobody has one, and their + layouts had the names the first release uses (ITL-1259). - A released data context manifest (`contexts/data/.json`) is frozen: the digest of its canonical JSON without `golden` is pinned in `contexts/released_manifests.json` (`manifest`), with one `releases` line per library release (ITL-946). Only its `golden` diff --git a/RELEASING.md b/RELEASING.md index d40c0d97a..4a746186b 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -70,8 +70,10 @@ Do these on `main` before the Release workflow runs: the `CHANGELOG.md` heading. It appends the release revision to the `golden` history of the current data context's manifest, writes its digest under `releases` in `src/orcapod/contexts/released_manifests.json`, and at the data context's first release - sets `released_in` and the `manifest` pin. It computes no hash and refuses a version - already released. Commit both files with the release pull request; the release is tagged + sets `released_in` and the `manifest` pin. It also pins every open data function type + snapshot (`scripts/data_function_types.py --release`: the sha256 and the version go into + `src/orcapod/data_function_types/released.json`; ITL-1259). It computes no hash and + refuses a version already released. Commit these files with the release pull request; the release is tagged on a commit that contains them. The Release workflow's build job checks this before it creates the tag (`scripts/check_golden_tags.py --release-version X.Y.Z`) and stops if the release is not recorded, or if a golden file was added after the release step (the diff --git a/changelog.d/ITL-1259.md b/changelog.d/ITL-1259.md new file mode 100644 index 000000000..5c44c4d2e --- /dev/null +++ b/changelog.d/ITL-1259.md @@ -0,0 +1,23 @@ +### Breaking changes and migration + +- The Python function type is `python.function.v1`, and every stored table's layout marker + starts at `v1` (`pdb_v1`, `rdb_v1`, `tdb_v1`, `odb_v1`, `sdb_v1`): the first release's + names (ITL-1259). Every function pod has a new identity and its results are computed once + again. A saved job that names `python.function.v0` loads read only and its functions + cannot be bound: rebuild the pipeline to run it. + +### Added + +- A data function type declares what a function of the type means and what its result + table stores (`orcapod.core.data_function_types`, `PYTHON_FUNCTION`; ITL-1259). A function + result table is at `.../python.function.v1/rev1/rdb_v1`: the type and its major (in the + identity), the revision of the type's record and the layout of the common columns (both in + the path only). Each released declaration is pinned as a snapshot. See + `docs/reference/data-function-types.md`. + +### Changed + +- Stores written by Orcapod before 0.4 are no longer looked for: no old layout names, no + layout-less paths, no job-wide total of old tables (ITL-1259; nobody has such a store). + A node still reports, and a side-effecting node still refuses, tables of another layout + under its own name, and rows of another data context. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index b6e1f475b..6e1d5f1e7 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -896,6 +896,10 @@ stating every tag binding (data-tags) moved it to `0.3.0`, the version written n `0.1.0` file, written by v0.2.0, and a `0.2.0` file, written by v0.3.0, still load, with every source tag plain as they were written, within the schema limits described below (a file with name-only extension type text cannot be loaded); any other version is rejected. +A job saved before 0.4 names its functions `python.function.v0`; it loads read only (its +blueprint, with its tables of the older layouts reported and not read), and its functions +cannot be bound, because 0.4 runs them as `python.function.v1` +([Data function types](../reference/data-function-types.md)): rebuild the pipeline to run it. The reverse does not hold: v0.2.0 cannot load a file saved by v0.3.0, because hashes differ between the two releases. @@ -1032,35 +1036,28 @@ Every row Orcapod stores names the **data context** that wrote it, in the column `__data_context`: `std:v1:default` since v0.3. The data context is one key for the whole of Orcapod's hashing, so rows that share it were keyed in exactly the same way (see [The data context of a stored row](../reference/hashing.md#the-data-context-of-a-stored-row)). -Tables written before v0.3 have no such column and are stored at an older table layout -(`pdb_v1`, `rdb_v1`). Orcapod **does not read them as results**: it does not refuse the +A table at another layout than this build's (another last path component; see +[Data function types](../reference/data-function-types.md) for the result table path) was +written by another build. Orcapod **does not read it as results**: it does not refuse the store, and it does not reuse those rows. -- The first run of v0.3 on a database written by v0.2 computes every input again and stores the - rows in its own tables (`pdb_v2`, `rdb_v2`). +- A run on such a store computes every input again and stores the rows in its own tables. - Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the - current data context only: nothing before that first run, one row per input after it. -- The old tables are never changed or deleted. They stay readable with the version that - wrote them and are kept for a later migration. To see which data contexts a table holds, + current data context only. +- The other tables are never changed or deleted. To see which data contexts a table holds, use `orcapod.contexts.stored_data_contexts`. -- When Orcapod finds a table at an older layout, it logs one warning with the number of - rows: `Found 3 rows in stored table '...' written by an earlier version of Orcapod (older - table layout, data context std:v0.1:default). They are not used and are left unchanged; - they can be migrated later.` -- The hashes in a table path can differ between versions, so older tables are looked for - by name: Orcapod lists the tables stored under the job and function name and treats every - table that is not at a current layout as written by an earlier version. Tables found this - way are reported once per node, without a number of rows; a table of another function - with the same name in the same job is included. A job run also reports one total of the - older tables that belong to none of its function nodes (operator caches, cached sources, - nodes that left the job). +- The hashes in a table path can differ between builds, so such tables are looked for by + name: Orcapod lists the tables stored under the job and function name and reports, once + per node, every table there that is not at a current layout; a table of another function + with the same name in the same job is included. Stores of the versions before 0.4 are not + looked for otherwise: nobody has one (ITL-1259). - Nodes that perform side effects are the exception: side-effect pods, and function pods that take the invocation context (`ctx_arg`). A side effect is an external action, and - completions stored by v0.2 cannot be matched with current inputs, so running the node - would do everything a second time. Instead the node raises + completions stored at another layout cannot be matched with current inputs, so running the + node would do everything a second time. Instead the node raises `SideEffectLogDataContextError` (from `orcapod.errors`, a `OrcapodDatabaseError`), naming the - tables, before it does anything. Every older table under the node's function or pod name - in the job counts. Keep using the version that wrote them. A fresh database is not a safe + tables, before it does anything. Every such table under the node's function or pod name + in the job counts. Keep using the build that wrote them. A fresh database is not a safe way around the error: it holds no completions, so on it the node performs every side effect again; use one only when a repeated delivery is acceptable. If the database cannot list its tables, the node warns that it could not check, and runs. diff --git a/docs/dev/design_issues/ITL-954-DC2.md b/docs/dev/design_issues/ITL-954-DC2.md index c263cf227..061bd7a3c 100644 --- a/docs/dev/design_issues/ITL-954-DC2.md +++ b/docs/dev/design_issues/ITL-954-DC2.md @@ -53,3 +53,11 @@ Known limits, follow-ups: - To verify again once the hash-moving branches are combined: `uv run pytest tests/test_pipeline/test_pre_v03_tables.py tests/test_databases/test_list_table_paths.py -v` (the `as_written` cases then exercise real moved hashes). + +**Superseded in part (ITL-1259, developer, November 18, 2026):** the first release, 0.4.0, +names its layouts `pdb_v1`, `rdb_v1`, `tdb_v1` again, so a v0.2 store's tables look current +by their last path component. The detection of stores written before 0.4 (the old-layout +names, the layout-less paths, the job-wide total) and its tests +(`tests/test_pipeline/test_pre_v03_tables.py`) are removed: nobody has such a store. The +listing under a node's name stays, for tables of another layout (`tables_of_another_layout`), +with the side-effect refusal. diff --git a/docs/reference/data-function-types.md b/docs/reference/data-function-types.md new file mode 100644 index 000000000..e812f39f0 --- /dev/null +++ b/docs/reference/data-function-types.md @@ -0,0 +1,161 @@ +# Data Function Types + +A function pod runs its function through a **data function type**: today the only one is +the Python function type, `python.function.v1`. The type says what a function of the type +means and how it is run, and what its result table stores about each row. This page states +both, for the type at its current major and revision, and how a change to either is +recorded (ITL-1259). + +## The result table path + +A function stores each result in its result table, at + +```text +/_result///v/python.function.v1/rev1/rdb_v1 +``` + +The first four components after `_result` are the function's uri. The last three each +have one owner: + +| Component | What it controls | Owner | In a hash | +|---|---|---|---| +| `python.function.v1` | The type and its **major**: what a function of the type means and how it is run | The data function type | Yes: the last part of every function's uri, so of every pod and node hash | +| `rev1` | The **revision** of the type's own record: the columns it stores about the function and its execution (`pf_var_*`, `pf_exec_*`) | The data function type | No: the path only | +| `rdb_v1` | The **layout** of the columns every result table has, whatever its type | The result cache, for all types | No: the path only | + +The test between a new major and the other two: **is a row stored before the change still a +correct result for the same input?** If not, the change is a new major: the function's +identity moves and every result is computed again. If it is, the change is a new revision +(when it changes what the type records) or a new `rdb` version (when it changes the common +columns): a new result table, and no hash moves. + +The function's pipeline table carries no revision: + +```text +///v/python.function.v1/schema:/pdb_v1 +``` + +A saved job names the type and its major only (`"data_function_type_id": +"python.function.v1"`); the revision is the one the installed version of Orcapod declares. + +```python +from orcapod import PipelineJob, function_pod +from orcapod.core.data_function import PYTHON_FUNCTION +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@function_pod(output_keys=["total", "label"]) +def describe_pair(x: int, y: int) -> tuple[int, str]: + return x + y, f"{x}+{y}" + + +types_store = InMemoryArrowDatabase() +types_job = PipelineJob(name="types_demo", database=types_store) +with types_job: + pairs = DictSource( + data=[{"k": "a", "x": 1, "y": 2}], tag_columns=["k"], tag_only_columns=True + ) + _ = describe_pair.pod(pairs, label="describe") +types_job.run() + +print(PYTHON_FUNCTION.type_id, PYTHON_FUNCTION.revision_component) +# python.function.v1 rev1 +print(describe_pair.pod.uri[-1]) +# python.function.v1 +for path in sorted(types_store.list_sources()): + print(path[-4:]) +# ('v0', 'python.function.v1', 'rev1', 'rdb_v1') +# ('v0', 'python.function.v1', 'schema:...', 'pdb_v1') +``` + +## The Python function type, `python.function.v1` + +### Identity + +A function's uri has four parts, in order: + +1. the function's name: its `__name__`, or the `function_name` given; +2. the hash of the output schema: each output key with its type, in no order; +3. `v` of the function's own version (the part before the first dot); +4. the type and its major: `python.function.v1`. + +### How a function is run (what the major stands for) + +- Inputs are bound to the function's parameters by name, as keyword arguments. +- A parameter with a default may be left out of the inputs. +- The returned value is mapped to the output keys by position: one key takes the whole + value, several keys take the items of a returned sequence. +- A function with no output keys and no return annotation has no outputs. +- A returned value must conform to its declared output type. +- Variadic (`*args`, `**kwargs`) parameters are refused. + +### Its record, revision 1 + +Each result row holds these columns about the function (`__pf_var_` prefix): + +| Column | Python type | Arrow type | Nullable | Meaning | +|---|---|---|---|---| +| `function_name` | `str` | `large_string` | yes | the function's name, as in the uri | +| `function_version` | `str` | `large_string` | yes | the full version string the function was declared with (`v3.7`) | +| `function_signature_hash` | `bytes` | `large_binary` | yes | the hash of the function used as a value | +| `function_content_hash` | `bytes` | `large_binary` | yes | the hash of the function's code | +| `git_hash` | `str` | `large_string` | yes | the git commit of the function's file or working directory, with `-dirty`, `-untracked` and `-` suffixes; `unknown` outside git | + +and these about how it was computed (`__pf_exec_` prefix): + +| Column | Python type | Arrow type | Nullable | Meaning | +|---|---|---|---|---| +| `executor_type` | `str` | `large_string` | yes | the executor that ran the function (`local`, `ray`, `none`) | +| `executor_info` | `dict[str, str]` | `large_list>` | yes | the executor's own settings, as text | +| `python_version` | `str` | `large_string` | yes | the Python version that ran the function (`3.12.3`) | +| `extra_info` | `dict[str, str]` | `large_list>` | yes | further facts about the run, as text; empty today | + +The two hashes, each stored as `