Skip to content

test(function_pod): pin the output schema as a set of named, typed outputs (ITL-1244) - #445

Open
eywalker wants to merge 23 commits into
mainfrom
eywalker/itl-1244-function-pods-the-order-of-output_keys-is
Open

eywalker wants to merge 23 commits into
mainfrom
eywalker/itl-1244-function-pods-the-order-of-output_keys-is

Conversation

@eywalker

@eywalker eywalker commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Merge after #444. This branch is based on the head of #444 (the several-item test audit: #442 → #443 → #444, tests only), so until those are merged this PR also shows their commits. Once they are merged, origin/main is merged into this branch (a merge commit, no rebase).

Closes ITL-1244. Related: ITL-1253, ITL-1254, ITL-1236.

Decision

The developer decided on November 17, 2026: "what we were hashing for the function was NOT the output signature but we were hashing the output schema of the function, which should be order independent."

A function's outputs are a set of named, typed values. The uri keeps the hash of the output schema: each output key with its type, in no order. No code changes, no hash moves.

  • Swapping two output keys of different types changes the schema, so the function gets another uri, content hash, pipeline hash and stored tables.
  • Swapping two output keys of the same type keeps the uri. Like a body edit, it changes what the function computes, so it needs a version bump.
  • A cache hit whose stored function variation data differs from the current function is still served silently. Adding that check is ITL-1253.

What this PR does

  • Tests (tests/test_core/function_pod/test_output_keys_identity.py): the audit's strict xfail for ITL-1244 is replaced by tests that pin the decision.
    • Two and three output keys in every order, for same-typed, differently typed, and partly same-typed outputs.
    • The uri equals the output-schema hash.
    • Renaming an output key gives another uri.
    • The input parameters (names and order) are in no part of the uri.
    • A one-output function keeps the golden uri.
    • @function_pod, FunctionPod.from_fn and PythonDataFunction give one uri.
    • to_config / from_config and a saved and loaded job keep the declared order and the uri.
    • In memory, each order names its own values (plain tags, data-tags, mixed roles).
    • A CachedDataFunction and jobs on one database (InMemoryArrowDatabase and DeltaTableDatabase, sync and async, plain tags and data-tags) store each differently typed order apart; a second pass reuses each order's own rows.
  • Strict xfails:
    • ITL-1253: a same-typed swap must never be served the other order's rows silently.
    • ITL-1254: a dict returned where a tuple is declared must be refused. Today it is iterated by its keys, so with str outputs the key names are stored as the values.
  • The audit test in test_several_items.py now pins both cases: a same-typed swap gives the same uri, a differently typed swap gives another.
  • Docs: docs/concepts/identity.md (Change detection, with executed samples), docs/concepts/function-pods.md and tutorial part 6 now say that the output is a set of named, typed values and that a same-typed reorder needs a version bump.
  • Agent rules: CLAUDE.md and .zed/rules state the same rule.
  • Design issue: docs/dev/design_issues/ITL-1244-OK1.md records the decision as resolved.
  • Changelog: no fragment. Behaviour does not change, and the docs now state the existing rule.

For the developer's review

Before the decision, I measured whether an order-keeping form could leave one-output functions unchanged. It cannot. A Schema hashes as a canonical JSON object, whose keys are sorted, so any ordered structure is another preimage, even for one key. For Schema({'scaled': int}):

form digest
schema (JSON object, today) 8e613efd57cb8a7d7a8ec2267ff78bf987fe0e76720d9074eb4971cf4edf55ad
[['scaled', <type hash>]] (list) 87c644ba0a661a980f9c8c0a2e679327352f95358756a49b33f5e05b9e645e5e
(('scaled', <type hash>),) (tuple) 53b0b64601efde0bbf0ec98d63218a178c8fcfaec59289536e239b517e5b18e2

Verification

  • generate_golden_hashes.py --check: "golden values of semantic, schema, arrow, datagram, pipeline, torch match". src/ is unchanged.
  • Targeted tests in one process: the new file, test_several_items.py, doc samples, site content, test_data_function.py, golden hashes, design issues, hash-site inventory. Result: 1935 passed, 5 xfailed.
  • The two strict xfails fail on their own assertions today: the ITL-1253 test is served the other order's values with no warning, and the ITL-1254 test raises nothing.
  • ruff check, ruff format --check and basedpyright all exit 0.

🤖 Generated with Claude Code

Edgar Y. Walker and others added 22 commits November 17, 2026 21:12
…y order, role and arrangement (ITL-1236)

A shared family of inputs with three tags and three data columns, in several tag
orders, table layouts and role settings (plain, data-tags, mixed), and tests of every
operator over it, sync and async: the result does not depend on the arrangement of the
input; the order of a set argument (names, rename mappings, constraints) changes
nothing; Join over 2, 3 and 4 inputs and MergeJoin are commutative; SemiJoin matches
on every shared tag. Defects found are xfail(strict=True): ITL-1241 (MergeJoin ties),
ITL-1242 (MapTags swap of data-tags), MergeJoin null crash (issue pending).

The inventory of every several-item argument, with the evidence and the ranked gaps,
is docs/dev/specs/itl-1236-several-item-test-audit.md. CLAUDE.md and .zed/rules state
the rule in "Testing conventions".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bound sources, in every order and role (ITL-1236)

Sources: tag order and layout enter no hash; tag_only_columns order changes nothing;
record ids follow the key order (content hash moves, pipeline hash and source id do
not); tags_as_data order is the new columns' order only; CachedSource reads back every
role and arrangement. Function pods: every order of three inputs gives the same rows,
hashes and columns, and a job over another order finds every result stored, sync and
async; parameters match by name; a function reads several data-tags in any
arrangement. Bind: two sources in one call, one refused leaves both unbound; a source in
another arrangement binds without moving anything.

Defects found are xfail(strict=True) with the issue pending: the fall-through tags'
order in a function pod's identity (defect 6), swapped output_keys keep the uri
(defect 4), a mixed-role source in another order breaks the bound node's table
(defect 5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se; MergeJoin and the restricted SemiJoin in every layout (ITL-1236)

Addresses the Copilot review of #442.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind
…; predicates in every order and role, sync and async (ITL-1236)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind

# Conflicts:
#	docs/dev/specs/itl-1236-several-item-test-audit.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind
…alued data-tags, stream methods and column groups (ITL-1236)

DropNulls naming plain tags, data-tags holding nulls and data in every order; Index
with out= over a list-valued data-tag; a DictSource schema in another field order;
every stream method given several items equals its operator (polars_filter keywords
win over constraint_map); every combination of ColumnConfig groups keeps the all_info
column order. Sync and async compared where an async path exists. All as documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arrangement; defect 7 as xfail (ITL-1236)

Addresses the second Copilot review of #442: sync and async rows are compared with
their column order, system-tag order and Arrow schemas within each arrangement, and
a result's keys with its rows' keys. Both fail on inputs whose table layout differs
from the tag order (reported, issue pending); those combinations are
xfail(strict=True), the others are asserted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind

# Conflicts:
#	docs/dev/specs/itl-1236-several-item-test-audit.md
#	tests/test_core/operators/several_items.py
…th bound sources (ITL-1236)

Addresses the Copilot review of #443.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de stores one column order on both orchestrators (ITL-1236)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind

# Conflicts:
#	docs/dev/specs/itl-1236-several-item-test-audit.md
…ere ITL-1249 reproduces (ITL-1236)

Addresses the third Copilot review of #442.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into eywalker/itl-1236-part2-sources-pods-bind
…ements, cache-only readback, bind on both orchestrators (ITL-1236)

Addresses the second Copilot review of #443.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ma; Index keeps the data-tag's lists; column groups checked against their own columns (ITL-1236)

Addresses the Copilot review of #444.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tputs (ITL-1244)

The developer decided (November 17, 2026) that a function's outputs are a
set of named, typed values: the uri holds the order-free output schema hash,
as before. No code changes. The audit's strict xfail for ITL-1244 becomes
tests that pin the decision:

- a swap of differently typed output keys gives another uri, content hash,
  pipeline hash and stored tables; a swap of same-typed keys gives the same
  uri (two and three keys, every order);
- input parameters (names and order) are in no part of the uri;
- @function_pod, FunctionPod.from_fn and PythonDataFunction give one uri;
  to_config/from_config and a saved and loaded job keep the declared order;
- jobs with differently typed orders on one database (in-memory and Delta,
  sync and async, plain tags and data-tags) store their own rows;
- strict xfails for ITL-1253 (a cache hit with other function variation
  data is served silently) and ITL-1254 (a dict returned for a tuple
  annotation is iterated by its keys).

Docs (identity, function pods, tutorial part 6), CLAUDE.md and .zed/rules
state the rule; design issue ITL-1244-OK1 records the decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 15 billable files and costs up to $3.75.

  • Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 1 minute for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 116 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 69f7597 and 6fd7584.

Files selected for processing: 15

  • .zed/rules
  • CLAUDE.md
  • docs/concepts/function-pods.md
  • docs/concepts/identity.md
  • docs/dev/design_issues/ITL-1244-OK1.md
  • docs/dev/specs/itl-1236-several-item-test-audit.md
  • docs/tutorial/06-reruns-and-caching.md
  • tests/test_core/data_tags/test_several_items.py
  • tests/test_core/function_pod/test_output_keys_identity.py
  • tests/test_core/function_pod/test_several_items.py
  • tests/test_core/operators/several_items.py
  • tests/test_core/operators/test_several_items.py
  • tests/test_core/sources/test_several_items.py
  • tests/test_core/streams/test_several_items_methods.py
  • tests/test_pipeline/test_bind_several_items.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test uses identical variation records and mishandles valid mismatch exceptions under an unrestricted xfail, so it cannot reliably detect when the issue is fixed.

2 open issues
What changed in this PR

Pins Orcapod’s existing rule that function outputs are an unordered set of named, typed values, without changing library code or hashes. The diff also includes the stacked several-item test audit from #442–#444.

Changes:

  • Test output-key permutations, identity, cache separation, and configuration/job round trips.
  • Document that reordering same-typed outputs requires a major version bump.
  • Expand coverage across column roles, layouts, multiple inputs, and sync/async execution, with strict expected failures for deferred issues.
File Description
tests/​test_pipeline/​test_bind_several_items.py Tests multi-source binding, atomic rejection, and rearranged columns.
tests/​test_core/​streams/​test_several_items_methods.py Tests stream-method equivalence and column-group combinations.
tests/​test_core/​sources/​test_several_items.py Tests source ordering, record keys, roles, and cached readback.
tests/​test_core/​operators/​test_several_items.py Audits operator permutations, layouts, roles, and execution paths.
tests/​test_core/​operators/​several_items.py Adds shared fixtures and result-comparison helpers.
tests/​test_core/​function_pod/​test_several_items.py Tests multiple inputs, data-tags, cache reuse, and output identity.
tests/​test_core/​function_pod/​test_output_keys_identity.py Pins output-schema identity, value assignment, caching, and serialization.
tests/​test_core/​data_tags/​test_several_items.py Tests nullable and list-valued data-tags.
docs/​tutorial/​06-reruns-and-caching.md Clarifies when output reordering reuses stored results.
docs/​dev/​specs/​itl-1236-several-item-test-audit.md Records audit coverage, decisions, and remaining gaps.
docs/​dev/​design_issues/​ITL-1244-OK1.md Records the resolved output-identity decision.
docs/​concepts/​identity.md Explains output-schema identity with executable examples.
docs/​concepts/​function-pods.md Adds versioning guidance for same-typed output reordering.
CLAUDE.md Updates identity and several-item testing guidance.
.zed/​rules Mirrors the identity and testing guidance.

Review effort: Balanced

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…y its assertion to fail (ITL-1244)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the test and documentation updates clarifying output-schema identity and cache reuse.

1 open issue
1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants