Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
0886ffd
docs(plan): ITL-910 adopt starfix 0.5 investigation and plan (WIP)
Nov 11, 2026
1b57f9b
docs(plan): ITL-910 suite findings under starfix 0.5 (WIP)
Nov 11, 2026
b9f4465
test(hashing): failing tests for starfix hash format 2 under arrow_v0…
Nov 11, 2026
ca5d8cd
feat(hashing): adopt starfix 0.5 (hash format 2) under label arrow_v0…
Nov 11, 2026
bd252f2
test(golden): regenerate golden hashes for starfix hash format 2 / ar…
Nov 11, 2026
a5ceb0a
test: re-pin format-1 digests and record-id order in starfix format 2…
Nov 11, 2026
f5433ff
docs: starfix hash format 2 / arrow_v0.2 in CHANGELOG, hashing refere…
Nov 11, 2026
bf8784f
style: ruff format and type the format-2 tests (ITL-910)
Nov 11, 2026
6dfb456
docs(plan): ITL-910 D1 design (explicit failure on foreign hash labels)
Nov 11, 2026
722a098
test(pipeline): failing tests: stores with rows under another Arrow h…
Nov 11, 2026
0c05cbd
feat(store): refuse stored rows hashed under another Arrow hash label…
Nov 12, 2026
9bef94e
fix(store): hash-label check ignores non-digest record ids, works on …
Nov 12, 2026
6df680c
docs: v0.3 refuses stores written by v0.2 (StoreHashLabelError); corr…
Nov 12, 2026
a49ba10
style: type the stored-hash-label helpers (ITL-910)
Nov 12, 2026
d5d564e
refactor: break the stored-hash-label import cycle; type the new test…
Nov 12, 2026
5363bfb
docs(plan): ITL-910 worker report
Nov 12, 2026
34960c8
feat(store): ignore rows hashed under another Arrow label instead of …
Nov 12, 2026
66517c5
docs: v0.3 ignores rows hashed under arrow_v0.1, recomputes and keeps…
Nov 12, 2026
92ffb2e
style: type the stored-hash-label filter and its tests (ITL-910)
Nov 12, 2026
bcd905b
feat(store): warn with the number of skipped rows; side-effect logs u…
Nov 12, 2026
b88f7eb
docs(plan): ITL-910 follow-up status (rows of another label ignored, …
Nov 12, 2026
7bfcb54
docs(plan): ITL-944 step 1 plan for data context v1
Nov 12, 2026
26dcbc5
feat(contexts): data context std:v1:default with a frozen-manifest ch…
Nov 12, 2026
87df847
wip(store): __data_context column, new table layouts, read filter, ol…
Nov 12, 2026
02db3f0
wip(store): function version column, tests moved to the new layouts (…
Nov 12, 2026
9b1c2be
feat(store): __data_context on every stored row, new table layouts, o…
Nov 12, 2026
2e619aa
feat(contexts): manifest fields of the versioning design; logical typ…
Nov 12, 2026
a7691dc
feat(store): function pods that take the invocation context fail on a…
Nov 12, 2026
07ccf30
refactor(store)!: remove the v0.1 manifest and the v0 to v1 migration…
Nov 12, 2026
18a110b
feat(store): refuse an input of another data context (ITL-944)
Nov 12, 2026
8b05071
fix(store): address CodeRabbit review of PR #313 (ITL-944)
Nov 12, 2026
7b6626b
docs(plan): ITL-954 plan and the store fixture written by Orcapod v0.2.0
Nov 12, 2026
0111b39
feat(databases): list_table_paths lists the tables under a path prefi…
Nov 12, 2026
03b4be2
test(store): pre-v0.3 tables on the real v0.2.0 store, as written and…
Nov 12, 2026
930e94b
feat(store): find tables written before v0.3 under a node's name, wha…
Nov 12, 2026
5bf7935
docs(store): pre-v0.3 tables are found by name; list_table_paths (ITL…
Nov 12, 2026
6addbf0
fix(job): a node without a name path claims no table in the job total…
Nov 12, 2026
8b291c9
docs(changelog): move the ITL-954 entry to a changelog fragment (ITL-…
Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .zed/rules
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,57 @@ hashes, forming a Merkle chain.

Prefixes are computed from SystemConstant in system_constants.py.

Stored tables: data context and layout versions (developer, November 11, 2026, ITL-944)

Every row Orcapod stores carries __data_context (constants.DATA_CONTEXT_COL, large_string):
the key of the data context that wrote it (std:v1:default since v0.3). The key names one
generation of hashing; rows of the current data context are the current entries.

Function pipeline — pdb_v2 (PIPELINE_DB_SCHEMA_VERSION)
Function result — rdb_v2 (RESULT_DB_SCHEMA_VERSION)
Side-effect log — tdb_v2 (TRACKING_DB_SCHEMA_VERSION)
Operator cache — odb_v1 (OPERATOR_DB_SCHEMA_VERSION)
Cached source — sdb_v1 (SOURCE_DB_SCHEMA_VERSION)

- The layout version is the last component of a table path. A change to the bookkeeping
columns of a table kind moves its layout version, so the table gets a new path. Never add,
rename or retype a stored column without it.
- Every new stored table kind writes __data_context (contexts/stored.with_data_context), and
every read or lookup keeps rows of the current data context only
(keep_rows_of_data_context, or a lookup constraint on the column).
- A table at an older layout is never read as current and never modified. What a node does
about such tables, and about rows of another data context, is decided in one place,
contexts/stored.account_for_stale_tables: count and report, or, for a node that delivers
side effects (a SideEffectPod node, or a function node whose pod takes the invocation
context, ctx_arg), raise SideEffectLogDataContextError before any delivery. A new kind of
node goes through that function; never decide it locally.
- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between versions. find_stale_tables(..., name_path=...) lists the
store under the hash-free beginning of the node's path (ArrowDatabaseProtocol.list_table_paths;
<function>, side_effect_function/<function>, side_effects/<pod>, relative to the node's
database) and every table there whose last component is not in
contexts/stored.CURRENT_LAYOUTS is pre-v0.3. A new layout constant must be added to
CURRENT_LAYOUTS, and a superseded one to the OLD_*_SCHEMA_VERSIONS tuples. A store that
cannot list raises TableListingNotSupportedError: side-effecting nodes then warn and
proceed (developer, November 11, 2026). Operator caches and cached sources have no stable
name; PipelineJob.run() reports one total for them. Every database class implements
list_table_paths.
- A released data context manifest (contexts/data/<version>.json) is frozen: its sha256 is
pinned in contexts/released_manifests.json. A change that moves an existing golden hash
value for a reason other than a logical type's own change needs a new data context version
with its own manifest file, never an edit of a released one.
- In memory the data context column is still _context_key (ColumnConfig context).
- A writer refuses an input of another data context than its own
(contexts/stored.require_same_data_context, InputValidationError naming both keys) before
anything is written; a whole-input condition is checked outside the row error policy. The
eventual behaviour is ITL-951.
- std:v0.1:default is not shipped; saved files that name it load through
contexts.resolve_saved_context. There is no SchemaVersionError, ignore_schema or
orcapod.migrations: do not reintroduce a per-layout migration path without a decision.
- Logical types and the value handlers that hash their Python values are registered in
contexts/logical_types.json, never in a data context manifest: a logical type is
versioned by its own identifier, and that file is not frozen.

### System tag evolution rules

1. Name-preserving — single-stream ops. Column name/value pass through unchanged.
Expand Down
180 changes: 174 additions & 6 deletions CHANGELOG.md

Large diffs are not rendered by default.

53 changes: 53 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -403,6 +403,59 @@ The pipeline hash uses a **resolver pattern** — `PipelineElementProtocol` obje
Prefixes are computed from `SystemConstant` in `system_constants.py`. The `constants` singleton
(with no global prefix) is used throughout.

### Stored tables: data context and layout versions

Every row Orcapod stores carries `__data_context` (`constants.DATA_CONTEXT_COL`,
`large_string`): the key of the data context that wrote it (`std:v1:default` since v0.3;
developer, November 11, 2026, ITL-944). The key names one generation of hashing; rows of the
current data context are the current entries.

| Table | Layout (last path component) | Constant |
|---|---|---|
| Function pipeline | `pdb_v2` | `PIPELINE_DB_SCHEMA_VERSION` |
| Function result | `rdb_v2` | `RESULT_DB_SCHEMA_VERSION` |
| Side-effect log | `tdb_v2` | `TRACKING_DB_SCHEMA_VERSION` |
| Operator cache | `odb_v1` | `OPERATOR_DB_SCHEMA_VERSION` |
| Cached source | `sdb_v1` | `SOURCE_DB_SCHEMA_VERSION` |

- A change to the bookkeeping columns of a table kind moves its layout version, so the table
gets a new path. Never add, rename or retype a stored column without it.
- Every new stored table kind writes `__data_context` (`contexts/stored.with_data_context`),
and every read or lookup keeps rows of the current data context only
(`keep_rows_of_data_context`, or a lookup constraint on the column).
- A table at an older layout is never read as current and never modified. What a node does
about such tables, and about rows of another data context, is decided in one place,
`contexts/stored.account_for_stale_tables`: count and report, or, for a node that delivers
side effects (a `SideEffectPod` node, or a function node whose pod takes the invocation
context, `ctx_arg`), raise `SideEffectLogDataContextError` before any delivery. A new kind
of node goes through that function; never decide it locally.
- A table written before v0.3 is found by NAME, never by a computed hash (ITL-954): the
hashes in a path differ between versions. `find_stale_tables(..., name_path=...)` lists the
store under the hash-free beginning of the node's path (`ArrowDatabaseProtocol.list_table_paths`;
`<function>`, `side_effect_function/<function>`, `side_effects/<pod>`, relative to the
node's database) and every table there whose last component is not in
`contexts/stored.CURRENT_LAYOUTS` is pre-v0.3. A new layout constant must be added to
`CURRENT_LAYOUTS`, and a superseded one to the `OLD_*_SCHEMA_VERSIONS` tuples. A store that
cannot list raises `TableListingNotSupportedError`: side-effecting nodes then warn and
proceed (developer, November 11, 2026). Operator caches and cached sources have no stable
name; `PipelineJob.run()` reports one total for them. Every database class implements
`list_table_paths`.
- A released data context manifest (`contexts/data/<version>.json`) is frozen: its sha256 is
pinned in `contexts/released_manifests.json`. A change that moves an existing golden hash
value for a reason other than a logical type's own change needs a new data context version
with its own manifest file, never an edit of a released one.
- In memory the data context column is still `_context_key` (`ColumnConfig(context=True)`).
- A writer refuses an input of another data context than its own
(`contexts/stored.require_same_data_context`, `InputValidationError` naming both keys)
before anything is written; a whole-input condition is checked outside the row error
policy. The eventual behaviour is ITL-951.
- `std:v0.1:default` is not shipped; saved files that name it load through
`contexts.resolve_saved_context`. There is no `SchemaVersionError`, `ignore_schema` or
`orcapod.migrations`: do not reintroduce a per-layout migration path without a decision.
- Logical types and the value handlers that hash their Python values are registered in
`contexts/logical_types.json`, never in a data context manifest: a logical type is
versioned by its own identifier, and that file is not frozen.

### System tag evolution rules

1. **Name-preserving** — single-stream ops (filter, select, map). Column name and value pass
Expand Down
Loading
Loading