Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash pyarrow's canonical extension types and extension-typed schemas with their extension identity instead of as their storage type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910) Explicitly approved regeneration (developer decision 3, November 11, 2026): 184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline 68/305, schema 22/92, semantic 1/64); no key added or removed among them. 21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a chunked Boolean column, canonical extension types and extension schemas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910) Adds orcapod.errors.StoreHashLabelError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) A v0.3 node whose pipeline, result, operator, cached-source or side-effect table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now raises StoreHashLabelError on read and before its first write, and PipelineJob.run checks every node before the first one runs, so a refused store is left exactly as it was. No opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910) Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run pre-flight. Every whole-table read keeps rows of the current label only (keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910) Developer decisions of November 11, 2026: a read that leaves out rows of another hash label logs one warning with the total; a side-effect node whose invocation log holds deliveries logged under another Arrow label raises SideEffectLogHashLabelError before delivering anything. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reported, kept) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…eck (ITL-944) Add the manifest v1.json (key std:v1:default) and make it the default data context. released_manifests.json pins the sha256 of every released manifest; v1 stays open (null) until the release-preparation commit sets its digest. The manifest schema accepts v<int> as well as v<int>.<int>. The content hash of a stream without a producer covers its _context_key column, so three golden entries (schema:source_stream_builder.*.content_hash) and the stream hashes pinned in test_struct_layout_identity move with the key. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d-layout report (ITL-944) Work in progress: existing tests that name the old paths, columns and ResultCache signature are not updated yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ITL-944) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ld layouts reported (ITL-944) Every stored row names the data context that wrote it (__data_context): function pipeline and result tables, operator-cache tables, cached-source tables and side-effect logs. It replaces __input_data_context_key and the result table's _context_key. Layouts move to pdb_v2, rdb_v2, tdb_v2, and operator-cache and cached-source tables gain odb_v1 and sdb_v1. Reads and lookups keep rows of the current data context only. A table at an older layout is counted and reported, never read as current, never written; a side-effect log at an older layout, or holding deliveries of another data context, raises SideEffectLogDataContextError before any delivery. The result table also stores the full function version (__pf_var_function_version). The label-based row filtering of PR #309 (hashing/stored_hash_labels.py) is removed; orcapod.contexts.stored_data_contexts lists the data contexts a table holds. Golden: pipeline:job.table_paths and pipeline:polars_filter_expr_job.table_paths change (layout component only), and the four table[...] entry names under each job follow the new paths with unchanged values. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…es move to an unfrozen registration (ITL-944) The v1 manifest gains released_in, previous, definitions, libraries, golden and changes (ITL-938 design document, section 7.4). The registered logical types and the value handlers of their Python values move from the manifest to contexts/logical_types.json, which is not frozen; a manifest whose type converter refers to logical_type_registry takes both from that file. What the default context registers is unchanged (pinned by a test). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…n older store (ITL-944) A function pod with ctx_arg performs an external action, and its completions are its function tables. On a store written by an earlier version, or holding completions of another data context, it would deliver every effect again without an error. It now raises SideEffectLogDataContextError, naming the tables, before anything is delivered, like a side-effect pod does for its log. Whether a node reports and skips stored rows of other generations or fails is decided in one place, contexts/stored.account_for_stale_tables. The error carries record_paths (every table found). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… machinery (ITL-944) std:v0.1:default no longer resolves: the error names it as the data context of Orcapod before v0.3, whose manifest is to live in orcapod-sanctuary. A node saved under it loads under the current data context with a warning. Removed: SchemaVersionError, NodeConfig(ignore_schema=...), orcapod.migrations and the 'orcapod migrate' command. A table at a path without a layout component (Orcapod v0.1) is counted like any other old layout. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A node, cache or cached source that would store a row for an input whose data context differs from its own raises InputValidationError naming both keys, before anything is computed, delivered or written. For a function node the whole input is checked outside the row error policy. The eventual behaviour (the node's data context wins) is ITL-951. Also records the developer's answers and the outcome in the plan, and moves the pre-v1 data context constants to system_constants to avoid an import cycle. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- attach_databases resets the stale-table check of function and operator nodes, so a newly attached store is checked again. - ResultCache.store refuses an output of another data context, and a caller's additional_constraints cannot replace the data context filter of ResultCache.lookup. - Streams rebuilt from stored rows (operator cache, cached source) get the node's data context instead of the default one. - resolve_context raises for a full key that is not its manifest's key. - The open-manifest gate also skips pre-releases. - The manifest schema's example is a v1 context; CHANGELOG and docs say the other-context refusal is not reached inside a PipelineJob. - test-objective: the two tests of the removed v0.1 context use v1. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…x (ITL-954) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… with moved hashes (ITL-954) Failing before the fix: with moved hashes the side-effecting nodes deliver again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tever their hashes (ITL-954) Side-effecting nodes of both kinds raise before delivering; other function tables are reported; a job run reports one total of the remaining tables. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…-954) Hashing reference, function pods page, CHANGELOG, agent instructions, DESIGN_ISSUES DC2. The fixture is regenerated from the final pods module. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… (ITL-954) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
ITL-954 Find tables written before v0.3 by scanning the store under a node's stable path prefix, so side-effecting nodes fail and old tables are reported
OverviewThe developer decided (ITL-938, November 11, 2026) that v0.3 counts and reports tables written by an earlier version, and that a side-effecting node (a ITL-944 implements this by looking at the node's own path with the older layout suffix. That stops working once the v0.3 hash changes are combined: the label rule (ITL-945) and the other changes move every pipeline hash and function URI, so a v0.2 table is at a path v0.3 never computes. The report would then find nothing, and every side effect would be delivered again on the first v0.3 run against a v0.2 store. Found by the ITL-945 worker; confirmed by the ITL-944 worker's analysis of the real v0.2.0 store fixture. What is stable across versions (measured on the v0.2.0 fixture): the path prefix Goals & Success Criteria
Dependencies & Risks
Resources & References
|
|
Warning Review limit reachedThe included review limit has been reached and this organization has disabled usage-based review continuation. Wait for reviews to reset or ask a billing admin to change After included review limits. Next included review available in 16 minutes. View limit details
|
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
…ce ids, native containers, pre-v0.3 table scan, GroupBy tags Bring the design document in line with the developer's decisions of November 11, 2026 made after the design session, and with PRs #313 to #318: - finding E (MergeJoin) withdrawn; ITL-942 described as in PR #315 (7 golden entries changed, 32 added, none removed); L2 is ITL-953 - a default source id built from a hash is <source kind>:<algorithm label>:<hex>, the stated exception to bare hashes inside names (rule 5a) - native list/set container logical types carry no version of their own - tables written before v0.3 are found by a prefix scan (list_table_paths, ITL-954); a store that cannot list warns once per node and proceeds - GroupBy keeps non-key tags as tags (ITL-955); tags as data via a derived source (ITL-939) - issue and PR numbers in the cut list and Part 8 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
…961) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
|
@coderabbitai review Posted by a Claude Code agent on behalf of @eywalker. |
Action not completed
|
…tten-before-v03-by-scanning' (#318, ITL-954) into ITL-967 Conflicts, all additions on both sides against main's ITL-919 changes, both kept: cached_function_pod.py (set_auto_flush and stale_result_tables_accounted_for), delta_lake_databases.py imports, job.py imports. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Closing as superseded: this branch's head is an ancestor of #324's merged head (6dc2a51), and #324 is merged to Posted by a Claude Code agent on behalf of @eywalker. |
Stacked on #313 (which is stacked on #309)
Fixes ITL-954
Summary
Only the commits after
8b050710belong to this PR; the rest is #313 and #309.#313 looks for a table of an earlier version at the node's own path with an older layout component. The other v0.3 changes (ITL-945, ITL-935, ITL-942) move every pipeline hash and function uri, so a real v0.2 table is at a path v0.3 never computes: nothing would be reported and side-effecting nodes would deliver again. This PR finds those tables by name.
list_table_paths(prefix=())onArrowDatabaseProtocol(public protocol addition, confirmed by the developer): the sorted paths of the tables under a path prefix, nested tables included, without opening a table. A store that cannot enumerate raises the neworcapod.errors.TableListingNotSupportedError.InMemoryArrowDatabaseDeltaTableDatabase, localpdb_v1under them;odb_v1/sdb_v1sit under a v0.2 table) and never enters_delta_logDeltaTableDatabase,s3:///s3a://s3fs(already a dependency), one LIST request per directory; credentials, endpoint and region are translated from the deltalake storage options, nothing elseDeltaTableDatabase,gs:///az:///abfs://fsspecpackage is installed and reaches the store with its default credentials; otherwise it cannot listConnectorArrowDatabase(SQLite, PostgreSQL, SpiralDB):and.as_), a component holding__comes back as two, and a different path with the same sanitized beginning is matched too. The last component (the layout) is exactExtensionAwareDatabaseNoOpArrowDatabaseOld versus current is decided by layout, not by hash (
contexts/stored.CURRENT_LAYOUTS): under<function>/,side_effect_function/<function>/orside_effects/<pod>/(and the same below_result/), a table whose last component is notpdb_v2,rdb_v2,tdb_v2,odb_v1orsdb_v1was written before v0.3.Side-effecting nodes of both kinds (
SideEffectPod, function pods withctx_arg) raiseSideEffectLogDataContextErrornaming every table, before anything is delivered. The check for rows of another data context in the current table and the sibling-path check stay.Ordinary function nodes report the tables in one warning per node.
PipelineJob.run()reports one total per run of the old tables under the job that no node claims by name: operator caches, cached sources stored under the job, nodes that left the job. ACachedSourcewithout an explicitcache_pathalso looks under<cache_path_prefix>/source/.A store that cannot list its tables (developer's decision, November 11): a side-effecting node logs one warning that it could not check, and proceeds. No setting.
Accepted false positives
They err towards failing, and the error names the tables so the user can tell: an old table of another function or pod with the same name in the same job, and of another node of the same function. Pinned in
TestTablesCountedByNameAlone.Cost
test_s3_lists_tables_and_nested_tables) did not run locally; the cloud walk itself is tested against a local directory standing in for the bucket.Tests on real v0.2 bytes
tests/test_arrays/data/v02_full_store.tar.gz(34 KB) was written by Orcapod v0.2.0 installed from the tag with its own pins; generatortests/test_arrays/data/make_v02_full_store.py, pods intests/test_arrays/v02_full_store_pods.py. Nine tables: 3pdb_v1, 3rdb_v1(two function pods and onectx_argpod),tdb_v1(a sink), a bare operator cache (cache_mode=LOG), a bare cached source. The existing fixture is untouched.tests/test_pipeline/test_pre_v03_tables.pyruns the same job under this version on that store twice: as written, and with every hash in its paths replaced on disk, which is what the combined stack sees. Before the fix, with replaced hashes the sink and thectx_argpod delivered again (24 tests failed); now they raise, nothing is delivered, and the v0.2 files stay byte-identical. Sync and async.Re-verification after the hash-moving branches are combined (the
as_writtencases then exercise real moved hashes):Differences from the approved plan
StoreReadError:s3fsmay not get the credentials deltalake has. A failed local directory listing raisesStoreReadError.Known limits
Listed in
DESIGN_ISSUES.md(DC2). In short: on a connector store a table found both at the sibling path and by listing is named twice; the job total needs a positive mark (pdb_v1/rdb_v1/tdb_v1, or a last component starting withschema:,instance:,node:), so a v0.2 cached source with an explicitcache_pathis found by the sibling check only.Finding outside this issue (not fixed)
PostgreSQL truncates identifiers to 63 bytes, and Orcapod's table names for function tables are much longer. Whether such tables work at all, and what truncation does to the layout suffix, was not verified (no server here).
Test plan
uv run pytest tests/on Python 3.12: 7866 passed, 97 skipped, 18 xfaileduv run pytest test-objective/passesuv run --with ruff ruff check .,ruff format --check .uv run --with basedpyright basedpyright: exit 0🤖 Generated with Claude Code