Repository navigation
Conversation
…v1, rev1, rdb_v1 (ITL-1259) A data function type now declares what a function of the type means and what its result table stores (developer, November 18, 2026): - core/data_function_types.py (a leaf module): StoredColumn and DataFunctionTypeDeclaration (type name, major, revision, identity recipe, run rules, the record: every column with name, Python and Arrow type, nullability, meaning, and for a hash its label constant and what it covers), snapshot helpers. - PYTHON_FUNCTION beside PythonDataFunction: python.function.v1, rev1, the columns stored today. The data function builds its variation and execution schemas from it; DataFunctionBase.declaration looks it up by type id (DECLARATIONS, declaration_for), so wrappers, proxies and a loaded job's stub node get the installed revision. The common result columns are declared once, result_cache.COMMON_RESULT_COLUMNS. - Result tables: <uri>/rev1/rdb_v1 (ResultCache takes the revision); the revision is in the path only. Pipeline tables carry no revision. A saved job names the type and its major only. - Every layout marker starts at v1 (pdb_v1, rdb_v1, tdb_v1; odb_v1, sdb_v1 unchanged). Forced: the old-layout tuples now list the v2 names of the withdrawn 0.3.0, and the tests built on a real v0.2 store (test_pre_v03_tables.py), whose v1 tables now look current, are removed. - Snapshot src/orcapod/data_function_types/python.function/v1-rev1.json, pinned in released.json (open until 0.4.0); scripts/data_function_types.py --check/--write/--release, run by generate_golden_hashes.py --release. - docs/reference/data-function-types.md (executed samples; a test checks its tables against the declaration), CLAUDE.md and .zed/rules, release docs, changelog. Golden: exactly the 22 measured entries move (uri, pod/node hashes, table paths and keys); nothing is regenerated before #446 and the developer's approval. Test literals that pin a function pod hash or uri are updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-1259) Developer: nobody has a store written before 0.4; its layouts had the v1 names the first release uses again, so it cannot be told apart anyway. Removed: the OLD_*_SCHEMA_VERSIONS tuples and _OLD_LAYOUTS; the explicit old-path look-ups (an older layout at the node's own path, and v0.1's layout-less path) in ResultCache, the function node, the side-effect node, the operator node (its whole check) and the cached source; marked_/unclaimed_tables_written_before_v03, report_unclaimed_tables and PipelineJob's job-wide total; the tests of those (two in TestOldLayout, the job-total test). Kept: the per-node listing under the node's name of tables of another layout (tables_of_another_layout, ITL-954) with the side-effect refusal, find_stale_tables / account_for_stale_tables, and all data-context row accounting. Docs, CLAUDE.md, .zed/rules and design issue ITL-954-DC2 follow; the cached-source table-name test now checks that nothing is created or written before the refusal (the read-only listing runs first). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…module, the declared interface, the source (ITL-1256)
The developer's settled design (2026-11-18) as the declaration's content,
still rev1 (nothing is released):
- function_module: the function's __module__ (new text column).
- function_signature_hash, label function_signature_v1: one hash over
{"inputs": {<name>: {"type", "default"?}}, "outputs": [{"key", "type"}]}
from the effective types; inputs by name in no order, outputs in declared
order; no name, module, parameter kind or order, version or body. Two
pods whose output keys differ only in order now record different values.
- function_content_hash, label function_content_v1: {"source": text} from
the def line on (ast cut; no decorator lines; a lambda's enclosing line);
null when the source cannot be read (plain REPL, python -c, exec).
Both algorithms and labels beside PythonDataFunction in core/data_function.py
(function_signature_hash, function_source, function_content_hash). Reuse is
unchanged (identity only). Snapshot rewritten; reference page, hashing.md
(labels, inventory, 1d), CLAUDE.md / .zed/rules labels, changelog. Tests:
every key order (same and mixed types), each construction form, overrides,
defaults, parameter order and kind, ctx, decorator and lambda sources, an
exec function in a job (null content hash), swapped keys in one result
table (Delta; the in-memory case is ITL-1263), two hash seeds. Pinned test
values of the stored hashes updated. Golden: function.stored_signature.ratio
and .scaled move (label and digest); not regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reached
This review includes 64 billable files and costs up to $16.00.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 59 minutes for your next included review. View limit details
|
There was a problem hiding this comment.
🟡 Changes recommended
Incompatible result revisions can bypass stale-table checks and repeat external side effects, while late snapshot validation can leave a partially recorded release that blocks retries.
2 open issues
What changed in this PR
Extends Orcapod’s stored Python-function records with module metadata and separate hashes for the declared interface and source. It builds on #448’s versioned data-function declarations and revision-specific result tables.
Changes:
- Records effective input types, defaults and ordered outputs separately from source text, allowing a null content hash when source is unavailable.
- Adds declaration snapshots and release tooling, revision-aware result paths, and the stacked table-layout cleanup.
- Updates regression tests, reference documentation and executable tutorial examples.
| File | Description |
|---|---|
tests/test_pipeline/test_tag_table_source_info.py |
Changes layout names in docstrings only. |
tests/test_pipeline/test_stored_data_context.py |
Updates layout fixtures and removes obsolete detection tests. |
tests/test_pipeline/test_store_write_failure.py |
Updates result and pipeline table selectors. |
tests/test_pipeline/test_serialization_helpers.py |
Updates serialized function-type expectations. |
tests/test_pipeline/test_saved_job_before_data_tags.py |
Expects old-layout stored results to remain unread. |
tests/test_pipeline/test_observability_reader.py |
Updates function-type paths in fixtures. |
tests/test_pipeline/test_hash_step_combined.py |
Updates layout checks and removes job-wide reporting coverage. |
tests/test_pipeline/test_bind_narrower_source.py |
Updates the pipeline-table selector. |
tests/test_pipeline/loaded_job/test_loaded_job_readback.py |
Updates the stored-table selector. |
tests/test_hashing/test_type_hash_registration.py |
Updates pipeline-table path expectations. |
tests/test_hashing/test_function_default_hashing.py |
Distinguishes stored signatures from function-value hashes. |
tests/test_hashing/test_canonical_type_names.py |
Updates stored hash and function-type expectations. |
tests/test_hashing/golden/generate_golden_hashes.py |
Adds declaration-snapshot pinning to releases. |
tests/test_databases/test_unstorable_columns.py |
Allows read-only listing before storage refusal. |
tests/test_databases/test_list_table_paths.py |
Updates layout and function-type fixtures. |
tests/test_databases/test_delta_list_binary_commits.py |
Updates result-table discovery. |
tests/test_data_function_types.py |
Tests declarations, storage paths, snapshots and documentation. |
tests/test_core/test_result_cache.py |
Supplies explicit record revisions to caches. |
tests/test_core/side_effect_function/test_side_effect_function_pod.py |
Updates side-effect function URI expectations. |
tests/test_core/function_pod/test_pandas_dataframe_order.py |
Updates the older-layout fixture path. |
tests/test_core/function_pod/test_node_content_hash_redundancy.py |
Changes layout wording in comments and docstrings only. |
tests/test_core/function_pod/test_function_pod_node.py |
Expects revision-specific result paths. |
tests/test_core/function_pod/test_function_pod_config.py |
Updates serialized function-type expectations. |
tests/test_core/function_pod/test_function_pod_auto_select.py |
Updates hashes affected by the function-type change. |
tests/test_core/function_pod/test_cached_function_pod.py |
Checks revision components in cache paths. |
tests/test_core/data_function/test_function_record.py |
Tests signature ordering, source availability and stored records. |
tests/test_core/data_function/test_executor.py |
Updates executor function-type fixtures. |
tests/test_core/data_function/test_data_function.py |
Checks module metadata and nullable content hashes. |
tests/test_core/data_function/test_data_function_config.py |
Updates configuration type identifiers. |
tests/test_core/data_function/test_cached_data_function.py |
Checks revision-specific cache paths. |
tests/test_arrays/test_array_v02_store.py |
Changes historical layout wording in the docstring only. |
src/orcapod/system_constants.py |
Resets layout markers and removes old-layout constants. |
src/orcapod/side_effects.py |
Removes explicit old-layout lookups. |
src/orcapod/protocols/core_protocols/data_function.py |
Adds the declaration property to the protocol. |
src/orcapod/pipeline/serialization.py |
Derives the registered type identifier from its declaration. |
src/orcapod/pipeline/job.py |
Removes job-wide pre-0.3 table reporting. |
src/orcapod/data_function_types/released.json |
Tracks the open declaration snapshot. |
src/orcapod/data_function_types/python.function/v1-rev1.json |
Captures the Python-function declaration and record. |
src/orcapod/core/sources/cached_source.py |
Retains name-based stale-layout discovery. |
src/orcapod/core/result_cache.py |
Declares common columns and adds revision-specific paths. |
src/orcapod/core/nodes/operator_node.py |
Removes obsolete old-layout accounting. |
src/orcapod/core/nodes/function_node.py |
Resolves loaded result paths through type declarations. |
src/orcapod/core/executors/ray.py |
Uses the declared Python-function type identifier. |
src/orcapod/core/data_function.py |
Implements declarations, interface hashes and optional source hashes. |
src/orcapod/core/data_function_types.py |
Defines declaration models and snapshot serialization. |
src/orcapod/core/cached_function_pod.py |
Passes the declared revision to result caching. |
src/orcapod/contexts/stored.py |
Simplifies stale-table detection to layouts and contexts. |
site/astro.config.mjs |
Adds navigation to the data-function reference. |
scripts/data_function_types.py |
Adds snapshot writing, validation and release pinning. |
RELEASING.md |
Documents declaration-snapshot release pinning. |
docs/tutorial/08-inspecting-and-tracing.md |
Updates lineage output and stored metadata examples. |
docs/tutorial/05-pipeline-job.md |
Updates generated operator labels in sample output. |
docs/reference/hashing.md |
Documents new hashes, paths and stale-table behavior. |
docs/reference/golden-hashes.md |
Documents coordinated snapshot pinning. |
docs/reference/data-function-types.md |
Explains type identity, revisions, records and storage. |
docs/dev/design_issues/ITL-954-DC2.md |
Records superseded legacy-store detection behavior. |
docs/concepts/function-pods.md |
Updates stored-layout and side-effect guidance. |
CLAUDE.md |
Updates hash-label and storage conventions. |
changelog.d/ITL-1259.md |
Records declaration and storage-layout changes. |
changelog.d/ITL-1256.md |
Records the new function metadata and hash behavior. |
.zed/rules |
Mirrors the updated hash and storage conventions. |
Review effort: Balanced
- A result cache counts a result table of another record revision under its function's name as another layout (tables_of_another_layout, result_revision): after a revision move a side-effecting pod refuses instead of repeating its actions, a plain pod reports and recomputes (tests, sync and async). - DataFunctionWrapper.declaration delegates to the wrapped function. - generate_golden_hashes.py --release checks the type snapshots before it writes anything (no half-done release). - check_golden_tags.py: a snapshot a release tag pinned must be unchanged at HEAD, with its pin (check_type_snapshots; tests). - The snapshot tests use an explicitly open copy, independent of what the repository has released. - The rev1 run rules state only what this base enforces (the no-output and positional-only refusals come with ITL-1257/ITL-1258). - Docs and changelog: a job saved before 0.4 loads read only and its functions cannot be bound (test); the sink test notes the approved removal of pre-0.4 detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eywalker/itl-1256-function-pods-the-stored-function
There was a problem hiding this comment.
…ot pin (ITL-1259) check_golden_tags.py --release-version (run before tagging) now also requires every data function type snapshot at HEAD to be pinned in released.json with its sha256 (unpinned_type_snapshots), so a snapshot added after the release step never ships open. Regression tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eywalker/itl-1256-function-pods-the-stored-function


Closes ITL-1256. Related: ITL-1259, ITL-1253, ITL-1244, ITL-1263.
What this does: the developer's settled design (2026-11-18), as the content of
rev1The Python function type's declaration (
PYTHON_FUNCTION, #448) changes. It is stillrev1: nothing is released. The record now holds:function_name,function_versionfunction_module(new)__module__function_signature_hashfunction_signature_v1function_content_hashfunction_content_v1, nullablegit_hash, executor columns,python_versionThe signature hash is one hash over:
input_schema=/output_schema=, else the annotations.ctxparameter stays in it, as the data function's input schema has it.The content hash is
{"source": <text>}, where the text is:defline on, as written (docstring, comments and whitespace included, decorator lines excluded, found with anastcut);python -c,exec). Pods defined in a REPL keep working.The algorithms and their label constants sit beside
PythonDataFunctionincore/data_function.py(function_signature_hash,function_source,function_content_hash). Reuse is unchanged: it goes on identity only, and ITL-1253 will make comparing an aspect opt-in.Docs:
docs/reference/data-function-types.md(record and hash tables, checked by a test against the declaration);docs/reference/hashing.md(labels table, hash-site inventory, §1d); the label lists inCLAUDE.mdand.zed/rules;changelog.d/ITL-1256.md.Tests:
tests/test_core/data_function/test_function_record.py, 27 passing and 2 xfail. They cover:PythonDataFunction,FunctionPod.from_fn,@function_pod);'ratio'and['ratio']giving one signature;input_schema=/output_schema=overrides, defaults by value, parameter order and kind not counting, name and module kept out of the signature,ctxkept in it;defline, the decorator not counting, a lambda's enclosing line;exec-defined function storing a null content hash and its module, and working in a job, sync and async;xfail(strict, ITL-1263):InMemoryArrowDatabaserefuses rows whose columns are in another order than the stored table. That defect predates this PR.For the developer's approval: golden values
Measured with
generate_golden_hashes.py --checkon this branch: 30 problems, which are #448's 28 plus these two. Both move by label and digest:semantic-01.jsonfunction.stored_signature.ratiosemantic_v1:7ce8ba814ee77feeca9950b3e641353aa5037695f0db5d75aa7a8e5030742051function_signature_v1:dc0c9f09974aab9a5abfb9c87262ca88a698424455ddb3cbba0792d72962dbbcsemantic-01.jsonfunction.stored_signature.scaledsemantic_v1:4ab074762136b2b8fc08bbe27fde9bfb0d53342c475232b60b965ae072c3da64function_signature_v1:973a435b2428508ae6bf684a18243e3923ef78fcec4332d745db78088cea929aNo content-hash value is pinned anywhere. Nothing else moves; the uri, the pod and node hashes, the paths and the record ids are #448's.
Entries to add at the regeneration step (not yet written, so no golden file changes before approval):
ratio,scaled) and absent (anexecfunction);input_schema=override;PythonDataFunction,from_fnand@function_pod.Pinned test values updated, outside the golden files:
test_canonical_type_names.py::TestFunctionPodGolden(the stored signature and content hashes of a real data function).Verification
ruff check,ruff format --checkandbasedpyrightall exit 0.scripts/data_function_types.py --checkis OK.🤖 Generated with Claude Code