From bbbef8c586cc0b3173bab1796ba6c88142887c91 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Wed, 18 Nov 2026 01:36:36 -0800 Subject: [PATCH 1/4] feat(data_function): declare the data function type; python.function.v1, rev1, rdb_v1 (ITL-1259) A data function type now declares what a function of the type means and what its result table stores (developer, November 18, 2026): - core/data_function_types.py (a leaf module): StoredColumn and DataFunctionTypeDeclaration (type name, major, revision, identity recipe, run rules, the record: every column with name, Python and Arrow type, nullability, meaning, and for a hash its label constant and what it covers), snapshot helpers. - PYTHON_FUNCTION beside PythonDataFunction: python.function.v1, rev1, the columns stored today. The data function builds its variation and execution schemas from it; DataFunctionBase.declaration looks it up by type id (DECLARATIONS, declaration_for), so wrappers, proxies and a loaded job's stub node get the installed revision. The common result columns are declared once, result_cache.COMMON_RESULT_COLUMNS. - Result tables: /rev1/rdb_v1 (ResultCache takes the revision); the revision is in the path only. Pipeline tables carry no revision. A saved job names the type and its major only. - Every layout marker starts at v1 (pdb_v1, rdb_v1, tdb_v1; odb_v1, sdb_v1 unchanged). Forced: the old-layout tuples now list the v2 names of the withdrawn 0.3.0, and the tests built on a real v0.2 store (test_pre_v03_tables.py), whose v1 tables now look current, are removed. - Snapshot src/orcapod/data_function_types/python.function/v1-rev1.json, pinned in released.json (open until 0.4.0); scripts/data_function_types.py --check/--write/--release, run by generate_golden_hashes.py --release. - docs/reference/data-function-types.md (executed samples; a test checks its tables against the declaration), CLAUDE.md and .zed/rules, release docs, changelog. Golden: exactly the 22 measured entries move (uri, pod/node hashes, table paths and keys); nothing is regenerated before #446 and the developer's approval. Test literals that pin a function pod hash or uri are updated. Co-Authored-By: Claude Opus 5.5 --- .zed/rules | 46 +- CLAUDE.md | 46 +- RELEASING.md | 6 +- changelog.d/ITL-1259.md | 15 + docs/concepts/function-pods.md | 4 +- docs/reference/data-function-types.md | 162 ++++++ docs/reference/golden-hashes.md | 5 +- docs/reference/hashing.md | 26 +- docs/tutorial/05-pipeline-job.md | 8 +- docs/tutorial/08-inspecting-and-tracing.md | 6 +- scripts/data_function_types.py | 199 +++++++ site/astro.config.mjs | 1 + src/orcapod/core/cached_function_pod.py | 1 + src/orcapod/core/data_function.py | 181 +++++- src/orcapod/core/data_function_types.py | 146 +++++ src/orcapod/core/executors/ray.py | 5 +- src/orcapod/core/nodes/function_node.py | 16 +- src/orcapod/core/result_cache.py | 57 +- .../python.function/v1-rev1.json | 92 +++ src/orcapod/data_function_types/released.json | 6 + src/orcapod/pipeline/serialization.py | 3 +- .../protocols/core_protocols/data_function.py | 12 +- src/orcapod/system_constants.py | 24 +- tests/test_arrays/test_array_v02_store.py | 2 +- .../test_cached_data_function.py | 8 +- .../data_function/test_data_function.py | 4 +- .../test_data_function_config.py | 4 +- .../test_core/data_function/test_executor.py | 16 +- .../function_pod/test_cached_function_pod.py | 7 +- .../test_function_pod_auto_select.py | 8 +- .../function_pod/test_function_pod_config.py | 2 +- .../function_pod/test_function_pod_node.py | 2 +- .../test_node_content_hash_redundancy.py | 8 +- .../test_pandas_dataframe_order.py | 4 +- .../test_side_effect_function_pod.py | 4 +- tests/test_core/test_result_cache.py | 14 +- tests/test_data_function_types.py | 446 +++++++++++++++ .../test_delta_list_binary_commits.py | 2 +- tests/test_databases/test_list_table_paths.py | 56 +- .../golden/generate_golden_hashes.py | 26 + .../test_hashing/test_canonical_type_names.py | 2 +- .../test_type_hash_registration.py | 2 +- .../loaded_job/test_loaded_job_readback.py | 2 +- .../test_bind_narrower_source.py | 2 +- .../test_pipeline/test_hash_step_combined.py | 10 +- .../test_observability_reader.py | 4 +- tests/test_pipeline/test_pre_v03_tables.py | 529 ------------------ .../test_saved_job_before_data_tags.py | 7 +- .../test_serialization_helpers.py | 16 +- .../test_pipeline/test_store_write_failure.py | 4 +- .../test_pipeline/test_stored_data_context.py | 55 +- .../test_tag_table_source_info.py | 6 +- 52 files changed, 1587 insertions(+), 732 deletions(-) create mode 100644 changelog.d/ITL-1259.md create mode 100644 docs/reference/data-function-types.md create mode 100644 scripts/data_function_types.py create mode 100644 src/orcapod/core/data_function_types.py create mode 100644 src/orcapod/data_function_types/python.function/v1-rev1.json create mode 100644 src/orcapod/data_function_types/released.json create mode 100644 tests/test_data_function_types.py delete mode 100644 tests/test_pipeline/test_pre_v03_tables.py diff --git a/.zed/rules b/.zed/rules index 915a08c69..6764ade47 100644 --- a/.zed/rules +++ b/.zed/rules @@ -818,19 +818,45 @@ Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, lar the key of the data context that wrote it (std:v1:default since v0.3). The key names one generation of hashing; rows of the current data context are the current entries. - Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION) - Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION) - Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION) + Function pipeline — pdb_v1 (PIPELINE_DB_SCHEMA_VERSION) + Function result — rdb_v1, after the type's revision rev1 (RESULT_DB_SCHEMA_VERSION) + Side-effect log — tdb_v1 (TRACKING_DB_SCHEMA_VERSION) Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION) Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION) -- The layout version is the last component of a table path. A change to the bookkeeping - columns of a table kind moves its layout version, so the table gets a new path. Never add, - rename or retype a stored column without it. -- The v0.3 layouts (pdb_v2, rdb_v2, tdb_v2, odb_v1, sdb_v1) were redefined before - their first release (ITL-1154): the record-id system tag became text (large_string) without a - layout bump, since no release shipped them. A store written by a development build of v0.3 is - discarded, not migrated (its side-effect logs would deliver again). +- The layout version is the last component of a table path. Every layout marker starts at v1 + for the first release, 0.4.0 (developer, November 18, 2026, ITL-1259; 0.3.0 was withdrawn, + so nothing released uses v2). A change to the common bookkeeping columns of a table kind + moves its layout version, so the table gets a new path. Never add, rename or retype a stored + column without it. +- A function result table is at + //v/python.function.v1/rev1/rdb_v1, three + components with three owners (docs/reference/data-function-types.md): + - python.function.v1, the data function type and its MAJOR (what a function means and how it + is run): the last part of the uri, so in every pod and node hash. A new major moves hashes. + - rev1, the REVISION of the type's own record (the pf_var_* / pf_exec_* columns), declared by + the type: path only. It never enters a uri, content or pipeline hash, source-info token or + record id; a new revision is a new result table and moves no hash. + - rdb_v1, the common result columns (input data hash, outputs, their source-info tokens, + __pod_ts, __data_context, record id; result_cache.COMMON_RESULT_COLUMNS), global: path only. + - The test between a new major and the other two: is a row stored before the change still a + correct result for the same input? No: a new major. Yes: a new revision (the type's + columns) or a new rdb version (the common columns). Components are delimited, never parsed + out of one string; the major is not repeated. + - Pipeline tables carry no revision (.../python.function.v1/schema:/pdb_v1). A + saved job names the type and major only (data_function_type_id); the revision comes from + the installed library's declaration (declaration_for(type id)). +- A data function type's declaration (PYTHON_FUNCTION, beside its class in core/data_function.py; + the classes in core/data_function_types.py, a leaf module) is the + source of truth: type name, major, revision, identity recipe, run rules, and its record + (every column with name, Python and Arrow type, nullability, meaning, and for a hash its + label CONSTANT and what it covers). The data function builds its stored columns from it; + never a second list. Each declaration is written out as a snapshot + (src/orcapod/data_function_types//v-rev.json, pinned in + released.json; scripts/data_function_types.py --check, --write, and --release, run by + generate_golden_hashes.py --release). An open snapshot must equal the code; a released one + never changes: a change to what a type stores is a new revision, to what its functions mean + a new major. - Every new stored table kind writes __data_context (contexts/stored.with_data_context), and every read or lookup keeps rows of the current data context only (keep_rows_of_data_context, or a lookup constraint on the column). diff --git a/CLAUDE.md b/CLAUDE.md index a06981d07..7686a4e7a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -847,18 +847,46 @@ current data context are the current entries. | Table | Layout (last path component) | Constant | |---|---|---| -| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` | -| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` | -| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` | +| Function pipeline | `pdb_v1` | `PIPELINE_DB_SCHEMA_VERSION` | +| Function result | `rdb_v1` (after the type's revision, `rev1`) | `RESULT_DB_SCHEMA_VERSION` | +| Side-effect log | `tdb_v1` | `TRACKING_DB_SCHEMA_VERSION` | | Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` | | Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` | -- A change to the bookkeeping columns of a table kind moves its layout version, so the table - gets a new path. Never add, rename or retype a stored column without it. -- The v0.3 layouts (`pdb_v2`, `rdb_v2`, `tdb_v2`, `odb_v1`, `sdb_v1`) were redefined before - their first release (ITL-1154): the record-id system tag became text (`large_string`) without a - layout bump, since no release shipped them. A store written by a development build of v0.3 is - discarded, not migrated (its side-effect logs would deliver again). +- Every layout marker starts at `v1` for the first release, 0.4.0 (developer, November 18, + 2026, ITL-1259; 0.3.0 was withdrawn, so nothing released uses `v2`). A change to the common + bookkeeping columns of a table kind moves its layout version, so the table gets a new path. + Never add, rename or retype a stored column without it. +- A function result table is at + `//v/python.function.v1/rev1/rdb_v1`, three + components with three owners (`docs/reference/data-function-types.md`): + - `python.function.v1`, the data function type and its MAJOR (what a function means and how + it is run): the last part of the uri, so in every pod and node hash. A new major moves + hashes. + - `rev1`, the REVISION of the type's own record (the `pf_var_*` / `pf_exec_*` columns), + declared by the type: path only. It never enters a uri, content or pipeline hash, + source-info token or record id; a new revision is a new result table and moves no hash. + - `rdb_v1`, the common result columns (input data hash, outputs, their source-info + tokens, `__pod_ts`, `__data_context`, record id; `result_cache.COMMON_RESULT_COLUMNS`), + global: path only. + - The test between a new major and the other two: is a row stored before the change still a + correct result for the same input? No: a new major. Yes: a new revision (the type's + columns) or a new `rdb` version (the common columns). Components are delimited, never + parsed out of one string; the major is not repeated. + - Pipeline tables carry no revision (`.../python.function.v1/schema:/pdb_v1`). + A saved job names the type and major only (`data_function_type_id`); the revision comes + from the installed library's declaration (`declaration_for(type id)`). +- A data function type's declaration (`PYTHON_FUNCTION`, beside its class in + `core/data_function.py`; the classes in `core/data_function_types.py`, a leaf module) is + the source of truth: type name, major, revision, identity recipe, run rules, and its record + (every column with name, Python and Arrow type, nullability, meaning, and for a hash its + label CONSTANT and what it covers). The data function builds its stored columns from it; + never a second list. Each declaration is written out as a snapshot + (`src/orcapod/data_function_types//v-rev.json`, pinned in + `released.json`; `scripts/data_function_types.py --check`, `--write`, and `--release`, run + by `generate_golden_hashes.py --release`). An open snapshot must equal the code; a released + one never changes: a change to what a type stores is a new revision, to what its functions + mean a new major. - Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`), and every read or lookup keeps rows of the current data context only (`keep_rows_of_data_context`, or a lookup constraint on the column). diff --git a/RELEASING.md b/RELEASING.md index d40c0d97a..4a746186b 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -70,8 +70,10 @@ Do these on `main` before the Release workflow runs: the `CHANGELOG.md` heading. It appends the release revision to the `golden` history of the current data context's manifest, writes its digest under `releases` in `src/orcapod/contexts/released_manifests.json`, and at the data context's first release - sets `released_in` and the `manifest` pin. It computes no hash and refuses a version - already released. Commit both files with the release pull request; the release is tagged + sets `released_in` and the `manifest` pin. It also pins every open data function type + snapshot (`scripts/data_function_types.py --release`: the sha256 and the version go into + `src/orcapod/data_function_types/released.json`; ITL-1259). It computes no hash and + refuses a version already released. Commit these files with the release pull request; the release is tagged on a commit that contains them. The Release workflow's build job checks this before it creates the tag (`scripts/check_golden_tags.py --release-version X.Y.Z`) and stops if the release is not recorded, or if a golden file was added after the release step (the diff --git a/changelog.d/ITL-1259.md b/changelog.d/ITL-1259.md new file mode 100644 index 000000000..ec3537603 --- /dev/null +++ b/changelog.d/ITL-1259.md @@ -0,0 +1,15 @@ +### Breaking changes and migration + +- The Python function type is `python.function.v1`, and every stored table's layout marker + starts at `v1` (`pdb_v1`, `rdb_v1`, `tdb_v1`, `odb_v1`, `sdb_v1`): the first release's + names (ITL-1259). Every function pod has a new identity and its results are computed once + again. A saved job that names `python.function.v0` no longer loads its function. + +### Added + +- A data function type declares what a function of the type means and what its result + table stores (`orcapod.core.data_function_types`, `PYTHON_FUNCTION`; ITL-1259). A function + result table is at `.../python.function.v1/rev1/rdb_v1`: the type and its major (in the + identity), the revision of the type's record and the layout of the common columns (both in + the path only). Each released declaration is pinned as a snapshot. See + `docs/reference/data-function-types.md`. diff --git a/docs/concepts/function-pods.md b/docs/concepts/function-pods.md index b6e1f475b..6be6eb119 100644 --- a/docs/concepts/function-pods.md +++ b/docs/concepts/function-pods.md @@ -1033,11 +1033,11 @@ Every row Orcapod stores names the **data context** that wrote it, in the column Orcapod's hashing, so rows that share it were keyed in exactly the same way (see [The data context of a stored row](../reference/hashing.md#the-data-context-of-a-stored-row)). Tables written before v0.3 have no such column and are stored at an older table layout -(`pdb_v1`, `rdb_v1`). Orcapod **does not read them as results**: it does not refuse the +(`pdb_v2`, `rdb_v2`). Orcapod **does not read them as results**: it does not refuse the store, and it does not reuse those rows. - The first run of v0.3 on a database written by v0.2 computes every input again and stores the - rows in its own tables (`pdb_v2`, `rdb_v2`). + rows in its own tables (`pdb_v1`, `rdb_v1`). - Reading a node (`as_table()`, `iter_data()`, `get_all_records()`) returns the rows of the current data context only: nothing before that first run, one row per input after it. - The old tables are never changed or deleted. They stay readable with the version that diff --git a/docs/reference/data-function-types.md b/docs/reference/data-function-types.md new file mode 100644 index 000000000..eb9ab87cc --- /dev/null +++ b/docs/reference/data-function-types.md @@ -0,0 +1,162 @@ +# Data Function Types + +A function pod runs its function through a **data function type**: today the only one is +the Python function type, `python.function.v1`. The type says what a function of the type +means and how it is run, and what its result table stores about each row. This page states +both, for the type at its current major and revision, and how a change to either is +recorded (ITL-1259). + +## The result table path + +A function stores each result in its result table, at + +```text +/_result///v/python.function.v1/rev1/rdb_v1 +``` + +The first four components after `_result` are the function's uri. The last three each +have one owner: + +| Component | What it controls | Owner | In a hash | +|---|---|---|---| +| `python.function.v1` | The type and its **major**: what a function of the type means and how it is run | The data function type | Yes: the last part of every function's uri, so of every pod and node hash | +| `rev1` | The **revision** of the type's own record: the columns it stores about the function and its execution (`pf_var_*`, `pf_exec_*`) | The data function type | No: the path only | +| `rdb_v1` | The **layout** of the columns every result table has, whatever its type | The result cache, for all types | No: the path only | + +The test between a new major and the other two: **is a row stored before the change still a +correct result for the same input?** If not, the change is a new major: the function's +identity moves and every result is computed again. If it is, the change is a new revision +(when it changes what the type records) or a new `rdb` version (when it changes the common +columns): a new result table, and no hash moves. + +The function's pipeline table carries no revision: + +```text +///v/python.function.v1/schema:/pdb_v1 +``` + +A saved job names the type and its major only (`"data_function_type_id": +"python.function.v1"`); the revision is the one the installed version of Orcapod declares. + +```python +from orcapod import PipelineJob, function_pod +from orcapod.core.data_function import PYTHON_FUNCTION +from orcapod.databases import InMemoryArrowDatabase +from orcapod.sources import DictSource + + +@function_pod(output_keys=["total", "label"]) +def describe_pair(x: int, y: int) -> tuple[int, str]: + return x + y, f"{x}+{y}" + + +types_store = InMemoryArrowDatabase() +types_job = PipelineJob(name="types_demo", database=types_store) +with types_job: + pairs = DictSource( + data=[{"k": "a", "x": 1, "y": 2}], tag_columns=["k"], tag_only_columns=True + ) + _ = describe_pair.pod(pairs, label="describe") +types_job.run() + +print(PYTHON_FUNCTION.type_id, PYTHON_FUNCTION.revision_component) +# python.function.v1 rev1 +print(describe_pair.pod.uri[-1]) +# python.function.v1 +for path in sorted(types_store.list_sources()): + print(path[-4:]) +# ('v0', 'python.function.v1', 'rev1', 'rdb_v1') +# ('v0', 'python.function.v1', 'schema:...', 'pdb_v1') +``` + +## The Python function type, `python.function.v1` + +### Identity + +A function's uri has four parts, in order: + +1. the function's name: its `__name__`, or the `function_name` given; +2. the hash of the output schema: each output key with its type, in no order; +3. `v` of the function's own version (the part before the first dot); +4. the type and its major: `python.function.v1`. + +### How a function is run (what the major stands for) + +- Inputs are bound to the function's parameters by name, as keyword arguments. +- A parameter with a default may be left out of the inputs. +- The returned value is mapped to the output keys by position: one key takes the whole + value, several keys take the items of a returned sequence. +- A function with no output keys (`-> None`, or no return annotation) has no outputs; a + returned value other than `None` is refused. +- A returned value must conform to its declared output type. +- Variadic (`*args`, `**kwargs`) and positional-only parameters are refused. + +### Its record, revision 1 + +Each result row holds these columns about the function (`__pf_var_` prefix): + +| Column | Python type | Arrow type | Nullable | Meaning | +|---|---|---|---|---| +| `function_name` | `str` | `large_string` | yes | the function's name, as in the uri | +| `function_version` | `str` | `large_string` | yes | the full version string the function was declared with (`v3.7`) | +| `function_signature_hash` | `bytes` | `large_binary` | yes | the hash of the function used as a value | +| `function_content_hash` | `bytes` | `large_binary` | yes | the hash of the function's code | +| `git_hash` | `str` | `large_string` | yes | the git commit of the function's file or working directory, with `-dirty`, `-untracked` and `-` suffixes; `unknown` outside git | + +and these about how it was computed (`__pf_exec_` prefix): + +| Column | Python type | Arrow type | Nullable | Meaning | +|---|---|---|---|---| +| `executor_type` | `str` | `large_string` | yes | the executor that ran the function (`local`, `ray`, `none`) | +| `executor_info` | `dict[str, str]` | `large_list>` | yes | the executor's own settings, as text | +| `python_version` | `str` | `large_string` | yes | the Python version that ran the function (`3.12.3`) | +| `extra_info` | `dict[str, str]` | `large_list>` | yes | further facts about the run, as text; empty today | + +The two hashes, each stored as `