Skip to content

feat(data_function): the record of python.function.v1 rev1: module, declared interface, source (ITL-1256) - #449

Open
eywalker wants to merge 7 commits into
mainfrom
eywalker/itl-1256-function-pods-the-stored-function
Open

eywalker wants to merge 7 commits into
mainfrom
eywalker/itl-1256-function-pods-the-stored-function

Conversation

@eywalker

Copy link
Copy Markdown
Contributor

Stacked on #448 (ITL-1259), which merges after #446. Until #448 is merged this PR also shows its two commits; its own change is the last commit. Golden entries are not regenerated (see below). Until #446 and #448 are on main and the developer approves the list, the golden test fails on #448's 22 entries plus the 2 entries listed here.

Closes ITL-1256. Related: ITL-1259, ITL-1253, ITL-1244, ITL-1263.

What this does: the developer's settled design (2026-11-18), as the content of rev1

The Python function type's declaration (PYTHON_FUNCTION, #448) changes. It is still rev1: nothing is released. The record now holds:

Column Form Says
function_name, function_version text as before
function_module (new) text the function's __module__
function_signature_hash hash, label function_signature_v1 the declared interface
function_content_hash hash, label function_content_v1, nullable the function as written
git_hash, executor columns, python_version as before

The signature hash is one hash over:

{"inputs": {<name>: {"type": <type hash>, "default": <value hash, only when present>}},
 "outputs": [{"key": <output key>, "type": <type hash>}, ...]}
  • It uses the effective types: input_schema= / output_schema=, else the annotations.
  • Inputs are keyed by name, in no order. Outputs are in declared order.
  • It does not hold the function's name or module, the parameters' kind or order, the version, or the body.
  • A ctx parameter stays in it, as the data function's input schema has it.
  • A default the hasher cannot hash still fails the pod at construction, naming the parameter.

The content hash is {"source": <text>}, where the text is:

  • the source from the def line on, as written (docstring, comments and whitespace included, decorator lines excluded, found with an ast cut);
  • for a lambda, the enclosing line;
  • absent (null) when the source cannot be read (a plain REPL, python -c, exec). Pods defined in a REPL keep working.

The algorithms and their label constants sit beside PythonDataFunction in core/data_function.py (function_signature_hash, function_source, function_content_hash). Reuse is unchanged: it goes on identity only, and ITL-1253 will make comparing an aspect opt-in.

Docs: docs/reference/data-function-types.md (record and hash tables, checked by a test against the declaration); docs/reference/hashing.md (labels table, hash-site inventory, §1d); the label lists in CLAUDE.md and .zed/rules; changelog.d/ITL-1256.md.

Tests: tests/test_core/data_function/test_function_record.py, 27 passing and 2 xfail. They cover:

  • every order of two and three output keys, same and mixed types, in all three construction forms (PythonDataFunction, FunctionPod.from_fn, @function_pod);
  • one declaration giving one record in every form; 'ratio' and ['ratio'] giving one signature;
  • input_schema= / output_schema= overrides, defaults by value, parameter order and kind not counting, name and module kept out of the signature, ctx kept in it;
  • the source starting at the def line, the decorator not counting, a lambda's enclosing line;
  • an exec-defined function storing a null content hash and its module, and working in a job, sync and async;
  • swapped same-typed keys storing two different signature hashes in one result table. The Delta case passes; the in-memory case is xfail(strict, ITL-1263): InMemoryArrowDatabase refuses rows whose columns are in another order than the stored table. That defect predates this PR.
  • the record being identical under two hash seeds, in separate processes.

For the developer's approval: golden values

Measured with generate_golden_hashes.py --check on this branch: 30 problems, which are #448's 28 plus these two. Both move by label and digest:

Entry Pinned Live
semantic-01.json function.stored_signature.ratio semantic_v1:7ce8ba814ee77feeca9950b3e641353aa5037695f0db5d75aa7a8e5030742051 function_signature_v1:dc0c9f09974aab9a5abfb9c87262ca88a698424455ddb3cbba0792d72962dbbc
semantic-01.json function.stored_signature.scaled semantic_v1:4ab074762136b2b8fc08bbe27fde9bfb0d53342c475232b60b965ae072c3da64 function_signature_v1:973a435b2428508ae6bf684a18243e3923ef78fcec4332d745db78088cea929a

No content-hash value is pinned anywhere. Nothing else moves; the uri, the pod and node hashes, the paths and the record ids are #448's.

Entries to add at the regeneration step (not yet written, so no golden file changes before approval):

  • the stored content hash, present (ratio, scaled) and absent (an exec function);
  • the stored signature hash for a function with 2 and with 3 output keys, each in two orders;
  • the stored signature hash with an input_schema= override;
  • the same declaration through PythonDataFunction, from_fn and @function_pod.

Pinned test values updated, outside the golden files: test_canonical_type_names.py::TestFunctionPodGolden (the stored signature and content hashes of a real data function).

Verification

🤖 Generated with Claude Code

Edgar Y. Walker and others added 3 commits November 18, 2026 01:36
…v1, rev1, rdb_v1 (ITL-1259)

A data function type now declares what a function of the type means and
what its result table stores (developer, November 18, 2026):

- core/data_function_types.py (a leaf module): StoredColumn and
  DataFunctionTypeDeclaration (type name, major, revision, identity recipe,
  run rules, the record: every column with name, Python and Arrow type,
  nullability, meaning, and for a hash its label constant and what it
  covers), snapshot helpers.
- PYTHON_FUNCTION beside PythonDataFunction: python.function.v1, rev1, the
  columns stored today. The data function builds its variation and execution
  schemas from it; DataFunctionBase.declaration looks it up by type id
  (DECLARATIONS, declaration_for), so wrappers, proxies and a loaded job's
  stub node get the installed revision. The common result columns are
  declared once, result_cache.COMMON_RESULT_COLUMNS.
- Result tables: <uri>/rev1/rdb_v1 (ResultCache takes the revision); the
  revision is in the path only. Pipeline tables carry no revision. A saved
  job names the type and its major only.
- Every layout marker starts at v1 (pdb_v1, rdb_v1, tdb_v1; odb_v1, sdb_v1
  unchanged). Forced: the old-layout tuples now list the v2 names of the
  withdrawn 0.3.0, and the tests built on a real v0.2 store
  (test_pre_v03_tables.py), whose v1 tables now look current, are removed.
- Snapshot src/orcapod/data_function_types/python.function/v1-rev1.json,
  pinned in released.json (open until 0.4.0); scripts/data_function_types.py
  --check/--write/--release, run by generate_golden_hashes.py --release.
- docs/reference/data-function-types.md (executed samples; a test checks its
  tables against the declaration), CLAUDE.md and .zed/rules, release docs,
  changelog.

Golden: exactly the 22 measured entries move (uri, pod/node hashes, table
paths and keys); nothing is regenerated before #446 and the developer's
approval. Test literals that pin a function pod hash or uri are updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-1259)

Developer: nobody has a store written before 0.4; its layouts had the v1
names the first release uses again, so it cannot be told apart anyway.
Removed: the OLD_*_SCHEMA_VERSIONS tuples and _OLD_LAYOUTS; the explicit
old-path look-ups (an older layout at the node's own path, and v0.1's
layout-less path) in ResultCache, the function node, the side-effect node,
the operator node (its whole check) and the cached source;
marked_/unclaimed_tables_written_before_v03, report_unclaimed_tables and
PipelineJob's job-wide total; the tests of those (two in TestOldLayout,
the job-total test).

Kept: the per-node listing under the node's name of tables of another
layout (tables_of_another_layout, ITL-954) with the side-effect refusal,
find_stale_tables / account_for_stale_tables, and all data-context row
accounting. Docs, CLAUDE.md, .zed/rules and design issue ITL-954-DC2
follow; the cached-source table-name test now checks that nothing is
created or written before the refusal (the read-only listing runs first).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…module, the declared interface, the source (ITL-1256)

The developer's settled design (2026-11-18) as the declaration's content,
still rev1 (nothing is released):

- function_module: the function's __module__ (new text column).
- function_signature_hash, label function_signature_v1: one hash over
  {"inputs": {<name>: {"type", "default"?}}, "outputs": [{"key", "type"}]}
  from the effective types; inputs by name in no order, outputs in declared
  order; no name, module, parameter kind or order, version or body. Two
  pods whose output keys differ only in order now record different values.
- function_content_hash, label function_content_v1: {"source": text} from
  the def line on (ast cut; no decorator lines; a lambda's enclosing line);
  null when the source cannot be read (plain REPL, python -c, exec).

Both algorithms and labels beside PythonDataFunction in core/data_function.py
(function_signature_hash, function_source, function_content_hash). Reuse is
unchanged (identity only). Snapshot rewritten; reference page, hashing.md
(labels, inventory, 1d), CLAUDE.md / .zed/rules labels, changelog. Tests:
every key order (same and mixed types), each construction form, overrides,
defaults, parameter order and kind, ctx, decorator and lambda sources, an
exec function in a job (null content hash), swapped keys in one result
table (Delta; the in-memory case is ITL-1263), two hash seeds. Pinned test
values of the stored hashes updated. Golden: function.stored_signature.ratio
and .scaled move (label and digest); not regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 64 billable files and costs up to $16.00.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 59 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 119 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 69f7597 and e3c30fa.

Files selected for processing: 64

  • .zed/rules
  • CLAUDE.md
  • RELEASING.md
  • changelog.d/ITL-1256.md
  • changelog.d/ITL-1259.md
  • docs/concepts/function-pods.md
  • docs/dev/design_issues/ITL-954-DC2.md
  • docs/reference/data-function-types.md
  • docs/reference/golden-hashes.md
  • docs/reference/hashing.md
  • docs/tutorial/05-pipeline-job.md
  • docs/tutorial/08-inspecting-and-tracing.md
  • scripts/check_golden_tags.py
  • scripts/data_function_types.py
  • site/astro.config.mjs
  • src/orcapod/contexts/stored.py
  • src/orcapod/core/cached_function_pod.py
  • src/orcapod/core/data_function.py
  • src/orcapod/core/data_function_types.py
  • src/orcapod/core/executors/ray.py
  • src/orcapod/core/nodes/function_node.py
  • src/orcapod/core/nodes/operator_node.py
  • src/orcapod/core/result_cache.py
  • src/orcapod/core/sources/cached_source.py
  • src/orcapod/data_function_types/python.function/v1-rev1.json
  • src/orcapod/data_function_types/released.json
  • src/orcapod/pipeline/job.py
  • src/orcapod/pipeline/serialization.py
  • src/orcapod/protocols/core_protocols/data_function.py
  • src/orcapod/side_effects.py
  • src/orcapod/system_constants.py
  • tests/test_arrays/test_array_v02_store.py
  • tests/test_check_golden_tags.py
  • tests/test_core/data_function/test_cached_data_function.py
  • tests/test_core/data_function/test_data_function.py
  • tests/test_core/data_function/test_data_function_config.py
  • tests/test_core/data_function/test_executor.py
  • tests/test_core/data_function/test_function_record.py
  • tests/test_core/function_pod/test_cached_function_pod.py
  • tests/test_core/function_pod/test_function_pod_auto_select.py
  • tests/test_core/function_pod/test_function_pod_config.py
  • tests/test_core/function_pod/test_function_pod_node.py
  • tests/test_core/function_pod/test_node_content_hash_redundancy.py
  • tests/test_core/function_pod/test_pandas_dataframe_order.py
  • tests/test_core/side_effect_function/test_side_effect_function_pod.py
  • tests/test_core/test_result_cache.py
  • tests/test_data_function_types.py
  • tests/test_databases/test_delta_list_binary_commits.py
  • tests/test_databases/test_list_table_paths.py
  • tests/test_databases/test_unstorable_columns.py
  • tests/test_hashing/golden/generate_golden_hashes.py
  • tests/test_hashing/test_canonical_type_names.py
  • tests/test_hashing/test_function_default_hashing.py
  • tests/test_hashing/test_type_hash_registration.py
  • tests/test_pipeline/loaded_job/test_loaded_job_readback.py
  • tests/test_pipeline/test_bind_narrower_source.py
  • tests/test_pipeline/test_hash_step_combined.py
  • tests/test_pipeline/test_observability_reader.py
  • tests/test_pipeline/test_pre_v03_tables.py
  • tests/test_pipeline/test_saved_job_before_data_tags.py
  • tests/test_pipeline/test_serialization_helpers.py
  • tests/test_pipeline/test_store_write_failure.py
  • tests/test_pipeline/test_stored_data_context.py
  • tests/test_pipeline/test_tag_table_source_info.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Incompatible result revisions can bypass stale-table checks and repeat external side effects, while late snapshot validation can leave a partially recorded release that blocks retries.

2 open issues
What changed in this PR

Extends Orcapod’s stored Python-function records with module metadata and separate hashes for the declared interface and source. It builds on #448’s versioned data-function declarations and revision-specific result tables.

Changes:

  • Records effective input types, defaults and ordered outputs separately from source text, allowing a null content hash when source is unavailable.
  • Adds declaration snapshots and release tooling, revision-aware result paths, and the stacked table-layout cleanup.
  • Updates regression tests, reference documentation and executable tutorial examples.
File Description
tests/​test_pipeline/​test_tag_table_source_info.py Changes layout names in docstrings only.
tests/​test_pipeline/​test_stored_data_context.py Updates layout fixtures and removes obsolete detection tests.
tests/​test_pipeline/​test_store_write_failure.py Updates result and pipeline table selectors.
tests/​test_pipeline/​test_serialization_helpers.py Updates serialized function-type expectations.
tests/​test_pipeline/​test_saved_job_before_data_tags.py Expects old-layout stored results to remain unread.
tests/​test_pipeline/​test_observability_reader.py Updates function-type paths in fixtures.
tests/​test_pipeline/​test_hash_step_combined.py Updates layout checks and removes job-wide reporting coverage.
tests/​test_pipeline/​test_bind_narrower_source.py Updates the pipeline-table selector.
tests/​test_pipeline/​loaded_job/​test_loaded_job_readback.py Updates the stored-table selector.
tests/​test_hashing/​test_type_hash_registration.py Updates pipeline-table path expectations.
tests/​test_hashing/​test_function_default_hashing.py Distinguishes stored signatures from function-value hashes.
tests/​test_hashing/​test_canonical_type_names.py Updates stored hash and function-type expectations.
tests/​test_hashing/​golden/​generate_golden_hashes.py Adds declaration-snapshot pinning to releases.
tests/​test_databases/​test_unstorable_columns.py Allows read-only listing before storage refusal.
tests/​test_databases/​test_list_table_paths.py Updates layout and function-type fixtures.
tests/​test_databases/​test_delta_list_binary_commits.py Updates result-table discovery.
tests/​test_data_function_types.py Tests declarations, storage paths, snapshots and documentation.
tests/​test_core/​test_result_cache.py Supplies explicit record revisions to caches.
tests/​test_core/​side_effect_function/​test_side_effect_function_pod.py Updates side-effect function URI expectations.
tests/​test_core/​function_pod/​test_pandas_dataframe_order.py Updates the older-layout fixture path.
tests/​test_core/​function_pod/​test_node_content_hash_redundancy.py Changes layout wording in comments and docstrings only.
tests/​test_core/​function_pod/​test_function_pod_node.py Expects revision-specific result paths.
tests/​test_core/​function_pod/​test_function_pod_config.py Updates serialized function-type expectations.
tests/​test_core/​function_pod/​test_function_pod_auto_select.py Updates hashes affected by the function-type change.
tests/​test_core/​function_pod/​test_cached_function_pod.py Checks revision components in cache paths.
tests/​test_core/​data_function/​test_function_record.py Tests signature ordering, source availability and stored records.
tests/​test_core/​data_function/​test_executor.py Updates executor function-type fixtures.
tests/​test_core/​data_function/​test_data_function.py Checks module metadata and nullable content hashes.
tests/​test_core/​data_function/​test_data_function_config.py Updates configuration type identifiers.
tests/​test_core/​data_function/​test_cached_data_function.py Checks revision-specific cache paths.
tests/​test_arrays/​test_array_v02_store.py Changes historical layout wording in the docstring only.
src/​orcapod/​system_constants.py Resets layout markers and removes old-layout constants.
src/​orcapod/​side_effects.py Removes explicit old-layout lookups.
src/​orcapod/​protocols/​core_protocols/​data_function.py Adds the declaration property to the protocol.
src/​orcapod/​pipeline/​serialization.py Derives the registered type identifier from its declaration.
src/​orcapod/​pipeline/​job.py Removes job-wide pre-0.3 table reporting.
src/​orcapod/​data_function_types/​released.json Tracks the open declaration snapshot.
src/​orcapod/​data_function_types/​python.function/​v1-rev1.json Captures the Python-function declaration and record.
src/​orcapod/​core/​sources/​cached_source.py Retains name-based stale-layout discovery.
src/​orcapod/​core/​result_cache.py Declares common columns and adds revision-specific paths.
src/​orcapod/​core/​nodes/​operator_node.py Removes obsolete old-layout accounting.
src/​orcapod/​core/​nodes/​function_node.py Resolves loaded result paths through type declarations.
src/​orcapod/​core/​executors/​ray.py Uses the declared Python-function type identifier.
src/​orcapod/​core/​data_function.py Implements declarations, interface hashes and optional source hashes.
src/​orcapod/​core/​data_function_types.py Defines declaration models and snapshot serialization.
src/​orcapod/​core/​cached_function_pod.py Passes the declared revision to result caching.
src/​orcapod/​contexts/​stored.py Simplifies stale-table detection to layouts and contexts.
site/​astro.config.mjs Adds navigation to the data-function reference.
scripts/​data_function_types.py Adds snapshot writing, validation and release pinning.
RELEASING.md Documents declaration-snapshot release pinning.
docs/​tutorial/​08-inspecting-and-tracing.md Updates lineage output and stored metadata examples.
docs/​tutorial/​05-pipeline-job.md Updates generated operator labels in sample output.
docs/​reference/​hashing.md Documents new hashes, paths and stale-table behavior.
docs/​reference/​golden-hashes.md Documents coordinated snapshot pinning.
docs/​reference/​data-function-types.md Explains type identity, revisions, records and storage.
docs/​dev/​design_issues/​ITL-954-DC2.md Records superseded legacy-store detection behavior.
docs/​concepts/​function-pods.md Updates stored-layout and side-effect guidance.
CLAUDE.md Updates hash-label and storage conventions.
changelog.d/​ITL-1259.md Records declaration and storage-layout changes.
changelog.d/​ITL-1256.md Records the new function metadata and hash behavior.
.zed/​rules Mirrors the updated hash and storage conventions.

Review effort: Balanced

Comment thread src/orcapod/core/result_cache.py Resolved
Edgar Y. Walker and others added 2 commits November 18, 2026 02:42
- A result cache counts a result table of another record revision under its
  function's name as another layout (tables_of_another_layout,
  result_revision): after a revision move a side-effecting pod refuses
  instead of repeating its actions, a plain pod reports and recomputes
  (tests, sync and async).
- DataFunctionWrapper.declaration delegates to the wrapped function.
- generate_golden_hashes.py --release checks the type snapshots before it
  writes anything (no half-done release).
- check_golden_tags.py: a snapshot a release tag pinned must be unchanged at
  HEAD, with its pin (check_type_snapshots; tests).
- The snapshot tests use an explicitly open copy, independent of what the
  repository has released.
- The rev1 run rules state only what this base enforces (the no-output and
  positional-only refusals come with ITL-1257/ITL-1258).
- Docs and changelog: a job saved before 0.4 loads read only and its
  functions cannot be bound (test); the sink test notes the approved
  removal of pre-0.4 detection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eywalker/itl-1256-function-pods-the-stored-function

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Intentional signature and pipeline hash changes require human sign-off on the migration and final moved-value list, while golden checks remain failing pending approval and #446/#448 integration.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Edgar Y. Walker and others added 2 commits November 18, 2026 04:34
…ot pin (ITL-1259)

check_golden_tags.py --release-version (run before tagging) now also
requires every data function type snapshot at HEAD to be pinned in
released.json with its sha256 (unpinned_type_snapshots), so a snapshot
added after the release step never ships open. Regression tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o eywalker/itl-1256-function-pods-the-stored-function

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Changes to function identities, stored paths, and signature hashes require human migration approval, with golden updates and prerequisite merges still pending.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants