From 3b1b1df8835b28a71d6917bd000888638170badc Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 21:44:55 -0800 Subject: [PATCH 01/37] docs(plans): ITL-1034 row-alignment suite plan Co-Authored-By: Claude Opus 5.5 --- ...2026-11-12-itl-1034-row-alignment-suite.md | 142 ++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md diff --git a/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md b/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md new file mode 100644 index 000000000..c858f4e4c --- /dev/null +++ b/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md @@ -0,0 +1,142 @@ +# ITL-1034 Row-alignment test suite Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use sensei:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A test suite that proves, by recomputation, that every per-row derived column of every +table Orcapod produces belongs to its own row, on sync and async paths, under forced completion +orders; plus a code sweep for the "built in one order, attached by position in another" pattern. + +**Spec:** Linear ITL-1034 plus the coordinator's brief (task_a7a8603f2f52). The brief is the design; +no separate spec. ITL-1029 (the `_shape_table` site) is fixed on another branch: this suite must +FAIL on main because of it and pass once that fix is merged. No xfail, no fix of that site here. + +**Architecture:** `tests/test_core/alignment/` holds three helper modules and the test modules. +`gate.py` forces completion order with asyncio events (no sleeps). `oracle.py` derives, from the +Python rows the test itself wrote, every origin row and its record id by the documented recipe +(`docs/reference/hashing.md` sites 5 and 6b). `checker.py` walks every row of a produced table and +recomputes each derived column from that row alone (content hash, source-info tokens, system tags, +record ids, context key, function outputs), falling back to an identity-keyed comparison with the +stored records only where a column embeds a per-run uuid. + +**Tech stack:** pytest, pyarrow, asyncio; public Orcapod API (`ArrowTableSource`, `function_pod`, +operators, `PipelineJob`, `node.as_table / head / get_all_records / as_source`). + +--- + +## Measured facts (prototype in the scratchpad, Nov 12 2026) + +* ITL-1029 reproduces deterministically: async job, completion order reversed, 5 rows, + `node.as_table(all_info=True)` shows each row with the hash of the mirrored row. +* The sync path is affected too whenever `sort_by_tags` permutes the rows (input not in tag + order): `_shape_table` sorts, then appends hashes in `iter_data` order. Same site, same fix. +* Node `head()` goes through `_shape_table` too (`_head_of_pairs`): same site. +* `_content_hash` recomputes as `data_context.arrow_hasher.hash_table(row.select(data_keys)).to_string()` + on a one-row slice of the produced table (verified equal on source and node tables). +* Source tokens: `"::::"`, token `"="` with + `record_id_column`, else `"row_"`. Record id: `uuid5(_SOURCE_RECORD_ID_NAMESPACE, + f"{source_id}::{token}")`, namespace `uuid5(NAMESPACE_URL, "https://orcapod.org/namespaces/source-record-id")`. + (docs/reference/hashing.md site 5 states a different namespace string: a docs defect, report it.) +* Function output token: `"::::"`; the stored record's + `__data_id` is that uuid. `__output_data_hash` / `__input_data_hash` are prefixed digests. +* Async concurrency is unlimited by default (`resolve_concurrency` -> None), so gating every row + on an event cannot deadlock for small tables; each wait still has a 30 s safety timeout that + raises a named error (a deadlock guard, never the mechanism of an assertion). + +## File structure + +* `tests/test_core/alignment/__init__.py` — empty. +* `tests/test_core/alignment/gate.py` — `CompletionGate`, `ORDERS` (natural, reversed, + interleaved, one_slow, random seeds), the gated function pods (`async def` functions keyed on a + unique data value). +* `tests/test_core/alignment/oracle.py` — `Origin`, `Oracle` (built from the test's own rows and + source ids; record id -> origin, token -> origin), recipe functions `source_record_id`, + `fold_record_ids`, `fold_values`. +* `tests/test_core/alignment/checker.py` — `assert_rows_aligned(table, *, data_keys, tag_keys, + oracle, data_context_key, ...)` and the per-column rules; `assert_same_rows_by_identity(a, b)` + for sync == async keyed on record ids. +* `tests/test_core/alignment/fixtures.py` — sources for the three tie scenarios (unique tags in + shuffled order, duplicate tags, null tags), `COLUMN_CONFIGS`. +* `tests/test_core/alignment/test_checker.py` — the checker is not vacuous: each rule rejects a + table with that one column permuted. +* `tests/test_core/alignment/test_gate.py` — the gate realises each order (observed via the + node's `iter_data` order). +* `tests/test_core/alignment/test_streams.py` — `ArrowTableStream`, sources, `FunctionPodStream` + (sync and the concurrent async-function path), `head()`. +* `tests/test_core/alignment/test_operators.py` — every operator x column config x tie scenario + (pairwise reduction over operators x configs, full over scenarios). +* `tests/test_core/alignment/test_nodes.py` — function and operator job nodes: sync/async x + orders x stores (in-memory, Delta) x fresh/readback (rerun, `PipelineJob.load`) x column + configs x scenarios (full product), `get_all_records`, `head()`, `as_source()` (DerivedSource). +* `tests/test_core/alignment/test_job.py` — `run`, `run_async`, run inside a running loop; run + summary under forced orders with failing rows; sync == async by identity. + +## Checker rules (per produced row) + +1. `_content_hash` (or a custom name): recompute from the row's data columns. +2. `_context_key`: equals the stream's `data_context_key`. +3. `_source_`: scalar token -> its origin (oracle by `(source_id, token)`) -> row[c] must equal + the origin's value of the column `c` came from (rename map), or the declared derivation + (function output: `y == f(origin)`). A list token (Batch, GroupBy, MergeJoin) is checked element + by element against the list value. A token whose prefix is a function uri is checked by + identity against the stored record (`__data_id` hex) when records are given. +4. System tags, grouped by column-name suffix: `_tag_source_id::S` / `_tag_record_id::S`. A scalar + record id must be an oracle record id whose source id equals the row's source-id value, and + every source token of a column of that source must give the same record id. Folded (aggregated) + ones must equal the documented fold of the member record ids recomputed from the row's + list-valued tokens. +5. User tags and data without provenance columns: (tag, data) must match the row's origin (via + record id when system tags are present, else by the set of origins with equal tags; with + duplicate tags the value must be one of the candidates' values). +6. Stored records (`get_all_records(all_info=True)`): `__output_data_hash` recomputed from the + data, `__input_data_hash` = content hash of the origin's input data (recomputed from the + origin's own source row through `Data.content_hash` of the source stream), `__data_context` + = context key, `_source_y` embeds `__data_id`. + +## Coverage matrix (as designed) + +| Path | sync/async | orders | ColumnConfig | scenarios | stores | +|---|---|---|---|---|---| +| ArrowTableStream / source as_table, head | sync | n/a | all 8 | 3 | n/a | +| FunctionPodStream (`pod(src)`), head | sync + async gather | 5 | all 8 | 3 | n/a | +| FunctionJobNode as_table/head/get_all_records/as_source | sync + async (run, run_async, in-loop) | 5 (+3 seeds) | all 8 | 3 | mem, delta; fresh + rerun + load | +| OperatorJobNode (Join, GroupBy downstream of the gated fn node) | sync + async | 5 | all 8 | 3 | mem, delta; fresh + rerun | +| Each operator on a source stream | sync (+ async via job for 3 ops) | n/a | pairwise | 3 | n/a | +| Run summary (failing rows) | sync + async | 5 | n/a | 1 | mem | + +ColumnConfig set: `{}` (default), `sort_by_tags=True`, `content_hash=True`, +`content_hash="h"`, `source=True`, `system_tags=True`, `context=True`, `meta=True`, +`all_info=True`, and `all_info` with `sort_by_tags=False`. + +## Tasks + +### Task 1: gate + oracle + checker + checker self-test +- [ ] Write `gate.py`, `oracle.py`, `checker.py`, `fixtures.py`. +- [ ] `test_checker.py`: build a correct source table; assert the checker passes; for each rule, + permute one column (`table.set_column(i, name, column.take(perm))`) and assert the checker + raises `AssertionError` naming that column. Run, expect PASS. +- [ ] `test_gate.py`: run the gated node for each order; assert `iter_data` order == the order. +- [ ] Commit `test(alignment): row-alignment checker and completion-order gate (ITL-1034)`. + +### Task 2: node paths (reproduces ITL-1029) +- [ ] `test_nodes.py` with the full product above. Run; expect FAILURES only on rows with + `_content_hash` under `sort_by_tags` / async orders (the ITL-1029 site). Record the failing + ids in this plan. +- [ ] Commit, push, open DRAFT PR (Fixes ITL-1034), status message to coordinator. + +### Task 3: streams, operators, job, run summary +- [ ] `test_streams.py`, `test_operators.py`, `test_job.py`. Any failure outside the ITL-1029 site + is a new defect: send it to the coordinator at once with a minimal repro, then fix here if a + few lines (failing-first test) or wait for the coordinator. +- [ ] Commit. + +### Task 4: sweep +- [ ] Verify every suspect line of the sweep report; put the table in the PR body. + +### Task 5: finish +- [ ] changelog.d/ITL-1034.md (`### Added`: the suite; no library change). +- [ ] ruff format/check, basedpyright, full `uv run pytest tests/ -n 4` (or fewer processes). +- [ ] Merge origin/main when the coordinator says the ITL-1029 fix is in; rerun; PR ready; + CodeRabbit rounds; worker_done. + +## Progress notes +(Successors: append here.) From ed3334122469bb35177aeefdd8d09d301b5c71b2 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 22:01:20 -0800 Subject: [PATCH 02/37] test(alignment): row-alignment checker, forced completion orders and node matrix (ITL-1034) WIP: reproduces ITL-1029 (fails on main until that fix merges). Co-Authored-By: Claude Opus 5.5 --- ...2026-11-12-itl-1034-row-alignment-suite.md | 7 +- tests/test_core/alignment/__init__.py | 0 tests/test_core/alignment/checker.py | 438 ++++++++++++++++++ tests/test_core/alignment/fixtures.py | 195 ++++++++ tests/test_core/alignment/gate.py | 198 ++++++++ tests/test_core/alignment/jobs.py | 212 +++++++++ tests/test_core/alignment/oracle.py | 173 +++++++ tests/test_core/alignment/records.py | 106 +++++ tests/test_core/alignment/test_checker.py | 127 +++++ tests/test_core/alignment/test_nodes.py | 240 ++++++++++ 10 files changed, 1695 insertions(+), 1 deletion(-) create mode 100644 tests/test_core/alignment/__init__.py create mode 100644 tests/test_core/alignment/checker.py create mode 100644 tests/test_core/alignment/fixtures.py create mode 100644 tests/test_core/alignment/gate.py create mode 100644 tests/test_core/alignment/jobs.py create mode 100644 tests/test_core/alignment/oracle.py create mode 100644 tests/test_core/alignment/records.py create mode 100644 tests/test_core/alignment/test_checker.py create mode 100644 tests/test_core/alignment/test_nodes.py diff --git a/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md b/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md index c858f4e4c..62dda677d 100644 --- a/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md +++ b/superpowers/plans/2026-11-12-itl-1034-row-alignment-suite.md @@ -29,7 +29,8 @@ operators, `PipelineJob`, `node.as_table / head / get_all_records / as_source`). `node.as_table(all_info=True)` shows each row with the hash of the mirrored row. * The sync path is affected too whenever `sort_by_tags` permutes the rows (input not in tag order): `_shape_table` sorts, then appends hashes in `iter_data` order. Same site, same fix. -* Node `head()` goes through `_shape_table` too (`_head_of_pairs`): same site. +* Node `head()` goes through `_shape_table` too, but `head_column_config` forces + `sort_by_tags=False`, so it is aligned (sweep). * `_content_hash` recomputes as `data_context.arrow_hasher.hash_table(row.select(data_keys)).to_string()` on a one-row slice of the produced table (verified equal on source and node tables). * Source tokens: `"::::"`, token `"="` with @@ -140,3 +141,7 @@ ColumnConfig set: `{}` (default), `sort_by_tags=True`, `content_hash=True`, ## Progress notes (Successors: append here.) + +* Helpers + checker self-test done (16 pass). Sweep done (subagent). Second defect found and + sent to coordinator (msg_c718d4fe9800): FunctionPodStream resumed iterator re-enumerates + from 0 (function_pod.py:1043/1069) and drops rows; awaiting decision to fix here. diff --git a/tests/test_core/alignment/__init__.py b/tests/test_core/alignment/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/test_core/alignment/checker.py b/tests/test_core/alignment/checker.py new file mode 100644 index 000000000..f9e0b1aca --- /dev/null +++ b/tests/test_core/alignment/checker.py @@ -0,0 +1,438 @@ +"""Check that every per-row column of a produced table belongs to its own row (ITL-1034). + +``assert_rows_aligned`` walks every row and recomputes each derived column from that row +alone, never by comparing two runs (two wrong runs can agree): + +* data values: each value must be the value its origin row has (``Oracle``), and every + column of one row must come from the same origin row of each source; +* ``_source_``: the token names the origin row whose value the row holds (element by + element for list columns); +* system tags: a record id is the documented UUID v5 of the origin row the data came from, + and the source id its source's id; a folded (aggregated) pair is the documented fold of + the row's own members; +* user tags: the origin row's tag values (element by element for list-valued tags); +* ``_content_hash``: the data context's Arrow hash of the row's data columns; +* ``_context_key``: the stream's data context key. + +A function's output token embeds the output datagram's uuid, made at run time: it cannot be +recomputed from the row, so it is compared, keyed on the row's record id, with the stored +record's (``function_tokens``). A column no rule covers fails the check, so a new kind of +per-row column cannot slip by unchecked. +""" + +from __future__ import annotations + +from collections.abc import Collection, Mapping +from dataclasses import dataclass, field +from typing import Any + +import pyarrow as pa + +from orcapod.system_constants import constants + +from .oracle import Oracle, Origin, fold_record_ids, fold_values + +MAX_REPORTED = 12 +DEFAULT_CONTENT_HASH_COLUMN = "_content_hash" + + +@dataclass(frozen=True) +class Layout: + """How the columns of a produced table relate to its origin rows. + + Attributes: + tag_keys: The user tag columns. + data_keys: The data columns. + members: Columns holding one element per member row (``Batch``, ``GroupBy``): the + row's members are the elements, the same members in every such column. + merged: Columns holding one element per input (``MergeJoin``), each element from + another input of the same row. + content_hash_column: The content hash column's name when it is not the default. + """ + + tag_keys: tuple[str, ...] + data_keys: tuple[str, ...] + members: frozenset[str] = frozenset() + merged: frozenset[str] = frozenset() + content_hash_column: str = DEFAULT_CONTENT_HASH_COLUMN + + +@dataclass +class _Row: + """The origin rows one produced row was found to come from, per member.""" + + members: list[dict[str, Origin]] = field(default_factory=lambda: [{}]) + + +def split_token(token: str) -> tuple[str, str, str]: + """Split a source-info token into source id (or function uri), row token and column.""" + prefix, column = token.rsplit(constants.BLOCK_SEPARATOR, 1) + source, row_token = prefix.rsplit(constants.BLOCK_SEPARATOR, 1) + return source, row_token, column + + +class _RowChecker: + """Check one row; collect every failure.""" + + def __init__( + self, + index: int, + row: dict[str, Any], + layout: Layout, + oracle: Oracle, + function_tokens: Mapping[tuple[str, bytes], str] | None, + ) -> None: + self.index = index + self.row = row + self.layout = layout + self.oracle = oracle + self.function_tokens = function_tokens + self.failures: list[str] = [] + self.state = _Row() + self.checked: set[str] = set() + + def fail(self, column: str, message: str) -> None: + """Record a failure of ``column`` in this row.""" + self.failures.append(f"row {self.index}, column {column!r}: {message}") + + # -- origins ----------------------------------------------------------- + + def _elements(self, column: str) -> list[Any]: + """The elements of ``column``: the list for a member or merged column.""" + value = self.row[column] + if column in self.layout.members or column in self.layout.merged: + return list(value) if value is not None else [] + return [value] + + def _member_count(self) -> int: + """How many members the row has: the length of its member columns.""" + lengths = {len(self._elements(c)) for c in self.layout.members} + if len(lengths) > 1: + self.fail("members", f"member columns have different lengths {lengths}") + return lengths.pop() if lengths else 1 + + def _assign(self, member: int, column: str, origin: Origin) -> None: + """Record that ``column`` of ``member`` comes from ``origin``; it must agree.""" + known = self.state.members[member].get(origin.source_id) + if known is None: + self.state.members[member][origin.source_id] = origin + elif known is not origin: + self.fail( + column, + f"comes from row {origin.token!r} of {origin.source_id!r}, but another " + f"column of the same row comes from row {known.token!r}", + ) + + def _resolve(self, column: str, value: Any, token: str | None) -> Origin | None: + """The origin of one element of a data column, from its token and its value.""" + from_token: Origin | None = None + token_name = f"{constants.SOURCE_PREFIX}{column}" + if token is not None: + self.checked.add(token_name) + source, row_token, token_column = split_token(token) + if column in self.oracle.derivations: + if token_column != column: + self.fail(token_name, f"token {token!r} names column {token_column!r}") + else: + if token_column != self.oracle.base_name(column): + self.fail(token_name, f"token {token!r} names column {token_column!r}") + from_token = self.oracle.by_token.get((source, row_token)) + if from_token is None: + self.fail(token_name, f"token {token!r} names no source row") + if from_token is not None: + expected = self.oracle.expected_value(column, from_token) + if expected != value: + self.fail( + token_name, + f"value {value!r} but its token {token!r} names a row whose value " + f"is {expected!r}", + ) + return from_token + if value is None: + return None + candidates = self.oracle.origins_for_value(column, value) + if not candidates: + self.fail(column, f"value {value!r} is the value of no origin row") + return None + return candidates[0] if len(candidates) == 1 else None + + def _tokens(self, column: str, count: int) -> list[str | None]: + """The source-info tokens of ``column``, one per element, or ``None`` each.""" + name = f"{constants.SOURCE_PREFIX}{column}" + if name not in self.row: + return [None] * count + value = self.row[name] + if column in self.layout.members or column in self.layout.merged: + tokens = list(value) if value is not None else [] + if len(tokens) != count: + self.fail(name, f"{len(tokens)} tokens for {count} values") + return [None] * count + return tokens + return [value] + + def resolve_data(self) -> None: + """Find each data element's origin; all columns of a member must agree.""" + count = self._member_count() + self.state.members = [{} for _ in range(count)] + for column in self.layout.data_keys: + elements = self._elements(column) + tokens = self._tokens(column, len(elements)) + for position, (value, token) in enumerate( + zip(elements, tokens, strict=True) + ): + origin = self._resolve(column, value, token) + if origin is None: + continue + member = position if column in self.layout.members else 0 + if column in self.layout.merged: + self._assign_merged(column, origin) + else: + self._assign(member, column, origin) + self.checked.add(column) + if token is not None and column in self.oracle.derivations: + self._check_function_token(column, origin, token) + + def _assign_merged(self, column: str, origin: Origin) -> None: + """Record the origin of one element of a merged column (one element per input).""" + self._assign(0, column, origin) + self.checked.add(column) + + def _check_function_token(self, column: str, origin: Origin, token: str) -> None: + """A function output's token must be the one stored for this row's record.""" + name = f"{constants.SOURCE_PREFIX}{column}" + if self.function_tokens is None: + _, row_token, _ = split_token(token) + if len(row_token) != 32: + self.fail(name, f"token {token!r} does not name an output datagram") + return + expected = self.function_tokens.get((column, origin.record_id)) + if expected != token: + self.fail( + name, + f"token {token!r}, but the record of row {origin.token!r} stores " + f"{expected!r}", + ) + + # -- system tags --------------------------------------------------------- + + def check_system_tags(self) -> None: + """Every system tag pair belongs to this row's origin rows (or folds its members).""" + source_prefix = constants.SYSTEM_TAG_SOURCE_ID_PREFIX + record_prefix = constants.SYSTEM_TAG_RECORD_ID_PREFIX + folded = bool(self.layout.members) + for name in self.row: + if not name.startswith(source_prefix): + continue + suffix = name[len(source_prefix) :] + record_name = f"{record_prefix}{suffix}" + if record_name not in self.row: + self.fail(name, f"no matching record id column {record_name!r}") + continue + self.checked.update((name, record_name)) + source_value = self.row[name] + record_value = self.row[record_name] + if folded: + self._check_folded(name, source_value, record_value) + else: + self._check_scalar(record_name, source_value, record_value) + + def _check_scalar(self, name: str, source_value: Any, record_value: Any) -> None: + """A scalar record id is the record id of this row's origin row of its source.""" + origin = self.oracle.by_record_id.get(record_value) + if origin is None: + self.fail(name, f"record id {record_value!r} is no source row's") + return + if origin.source_id != source_value: + self.fail( + name, + f"record id of a row of {origin.source_id!r} beside source id " + f"{source_value!r}", + ) + return + self._assign(0, name, origin) + + def _check_folded(self, name: str, source_value: Any, record_value: Any) -> None: + """A folded pair is the documented fold of this row's members of one source.""" + sources = {s for member in self.state.members for s in member} + for source in sorted(sources): + origins = [member.get(source) for member in self.state.members] + if fold_values([source] * len(origins)) != source_value: + continue + if any(o is None for o in origins): + self.fail(name, f"members of {source!r} not all known; cannot fold") + return + expected = fold_record_ids([o.record_id for o in origins if o is not None]) + if expected != record_value: + self.fail( + name, + "folded record id is not the fold of the row's own members " + f"{[o.token for o in origins if o is not None]!r}", + ) + return + self.fail(name, f"folded source id {source_value!r} folds no member list") + + # -- values -------------------------------------------------------------- + + def check_values(self) -> None: + """Every tag and data value is its origin row's value.""" + for column in (*self.layout.tag_keys, *self.layout.data_keys): + if column in self.layout.merged: + self._check_merged_values(column) + continue + elements = self._elements(column) + if column in self.layout.members: + if len(elements) != len(self.state.members): + self.fail(column, "length differs from the row's member count") + continue + pairs = list(zip(elements, self.state.members, strict=True)) + else: + pairs = [(elements[0], m) for m in self.state.members] + for value, member in pairs: + for origin in member.values(): + if not self.oracle.holds(column, origin): + continue + self.checked.add(column) + expected = self.oracle.expected_value(column, origin) + if expected != value: + self.fail( + column, + f"value {value!r}, but its row {origin.token!r} of " + f"{origin.source_id!r} has {expected!r}", + ) + + def _check_merged_values(self, column: str) -> None: + """A merged column holds exactly the values of the row's inputs that have it.""" + expected = [ + self.oracle.expected_value(column, o) + for o in self.state.members[0].values() + if self.oracle.holds(column, o) + ] + actual = self._elements(column) + if sorted(map(repr, expected)) != sorted(map(repr, actual)): + self.fail(column, f"holds {actual!r}, its inputs have {expected!r}") + self.checked.add(column) + + +def assert_rows_aligned( + table: pa.Table, + layout: Layout, + oracle: Oracle, + data_context: Any, + *, + function_tokens: Mapping[tuple[str, bytes], str] | None = None, + require_rows: bool = True, +) -> set[str]: + """Assert that every per-row column of every row of ``table`` belongs to that row. + + Args: + table: The produced table. + layout: How its columns relate to the origin rows. + oracle: The origin rows, from the tables the test wrote. + data_context: The stream's data context: its key and Arrow hasher. + function_tokens: The stored source-info token of each function output, keyed by + ``(column, origin record id)``; without it a function token's form is checked. + require_rows: Fail on a table with no rows (a check of nothing). + + Returns: + The names of the columns at least one rule compared. + + Raises: + AssertionError: Naming every misaligned row and column (the first few). + """ + if require_rows: + assert table.num_rows > 0, "the table has no rows: the check would check nothing" + hasher = data_context.arrow_hasher + context_key = data_context.context_key + hash_column = layout.content_hash_column + known = set(layout.tag_keys) | set(layout.data_keys) + unknown = [ + c + for c in table.column_names + if c not in known + and c != hash_column + and c != constants.CONTEXT_KEY + and not c.startswith(constants.SYSTEM_TAG_PREFIX) + and not ( + c.startswith(constants.SOURCE_PREFIX) + and c[len(constants.SOURCE_PREFIX) :] in layout.data_keys + ) + ] + assert not unknown, f"no alignment rule covers the columns {unknown!r}" + + failures: list[str] = [] + checked: set[str] = set() + rows = table.to_pylist() + for index, row in enumerate(rows): + checker = _RowChecker(index, row, layout, oracle, function_tokens) + checker.resolve_data() + checker.check_system_tags() + checker.check_values() + if hash_column in row: + data_row = table.slice(index, 1).select(list(layout.data_keys)) + expected = hasher.hash_table(data_row).to_string() + if row[hash_column] != expected: + checker.fail( + hash_column, + f"{row[hash_column]!r} is not the hash of the row's data {expected!r}", + ) + checker.checked.add(hash_column) + if constants.CONTEXT_KEY in row: + if row[constants.CONTEXT_KEY] != context_key: + checker.fail(constants.CONTEXT_KEY, f"{row[constants.CONTEXT_KEY]!r}") + checker.checked.add(constants.CONTEXT_KEY) + failures.extend(checker.failures) + checked |= checker.checked + if failures: + shown = "\n ".join(failures[:MAX_REPORTED]) + raise AssertionError( + f"{len(failures)} misaligned cells in {table.num_rows} rows:\n {shown}" + ) + return checked + + +def assert_all_columns_checked(table: pa.Table, checked: Collection[str]) -> None: + """Assert that the checker compared every column of ``table`` at least once.""" + missed = [c for c in table.column_names if c not in checked] + assert not missed, f"the check compared nothing in the columns {missed!r}" + + +def rows_by_identity( + table: pa.Table, ignore: Collection[str] = () +) -> dict[tuple[Any, ...], dict[str, Any]]: + """The rows of ``table`` keyed by their record ids (every system tag record id column). + + Raises: + AssertionError: If the table has no record id column or two rows share a key. + """ + keys = sorted( + c + for c in table.column_names + if c.startswith(constants.SYSTEM_TAG_RECORD_ID_PREFIX) + ) + assert keys, "the table has no record id column to key rows by" + keyed: dict[tuple[Any, ...], dict[str, Any]] = {} + for row in table.to_pylist(): + key = tuple(row[k] for k in keys) + assert key not in keyed, f"two rows share the record ids {key!r}" + keyed[key] = {c: v for c, v in row.items() if c not in ignore} + return keyed + + +def assert_same_rows_by_identity( + left: pa.Table, right: pa.Table, ignore: Collection[str] = () +) -> None: + """Assert two tables hold the same rows, matched by record ids, never by position. + + Args: + left: One table. + right: The other; same columns in the same order. + ignore: Columns that differ between runs by design (a per-run uuid, a time). + """ + assert left.column_names == right.column_names, ( + f"columns differ: {left.column_names} vs {right.column_names}" + ) + left_rows = rows_by_identity(left, ignore) + right_rows = rows_by_identity(right, ignore) + assert left_rows.keys() == right_rows.keys(), "the tables hold different records" + for key, row in left_rows.items(): + assert row == right_rows[key], f"record {key!r}: {row} != {right_rows[key]}" diff --git a/tests/test_core/alignment/fixtures.py b/tests/test_core/alignment/fixtures.py new file mode 100644 index 000000000..df84f325f --- /dev/null +++ b/tests/test_core/alignment/fixtures.py @@ -0,0 +1,195 @@ +"""Sources and column configurations shared by the row-alignment tests (ITL-1034). + +Three tie scenarios of six rows, each given in three input orders (``INPUT_ORDERS``): +shuffled (neither tag order nor data order, the default), tag-sorted, and reverse +tag-sorted. The tie scenarios: + +* ``unique``: every tag tuple appears once; +* ``duplicate``: tag tuples repeat (ties under ``sort_by_tags``); +* ``null``: tags are null in some rows, and repeat. + +Every data value is unique within its column and across sources sharing the column, so a +value names its origin row. Source ``A`` identifies rows by ``record_id_column="x"``; source +``C`` by position (``row_`` tokens), so both provenance token forms are covered. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any + +import pyarrow as pa + +from orcapod.sources import ArrowTableSource + +from .oracle import Oracle + +TAG_COLUMNS = ("g", "n") + +# Each scenario's rows of A as (g, n, x), in a "shuffled" order that is neither tag order +# nor data order. +SCENARIOS: dict[str, list[tuple[str | None, int | None, int]]] = { + "unique": [("d", 4, 5), ("b", 2, 2), ("f", 6, 0), ("a", 1, 4), ("e", 5, 1), ("c", 3, 3)], + "duplicate": [ + ("b", 1, 5), + ("a", 1, 2), + ("b", 1, 0), + ("a", 2, 4), + ("c", 1, 1), + ("b", 1, 3), + ], + "null": [ + (None, 1, 5), + ("a", None, 2), + (None, 1, 0), + ("b", 2, 4), + (None, None, 1), + ("a", 1, 3), + ], +} + +# The order the input rows are given in. Existing tests almost always feed rows already in +# tag order, which hides any order assumption; here "shuffled" is the default. +INPUT_ORDERS = ("shuffled", "sorted", "reversed") + + +def _tag_sort_key(row: tuple[str | None, int | None, int]) -> tuple[Any, ...]: + """Tag order, nulls first, then the data key (a stable order for tied tags).""" + g, n, x = row + return (g is not None, g or "", n is not None, n or 0, x) + + +def rows_a(scenario: str, input_order: str) -> list[tuple[str | None, int | None, int]]: + """The rows of A of ``scenario`` in ``input_order``.""" + rows = SCENARIOS[scenario] + if input_order == "shuffled": + return list(rows) + ordered = sorted(rows, key=_tag_sort_key) + if input_order == "sorted": + return ordered + if input_order == "reversed": + return list(reversed(ordered)) + raise ValueError(f"unknown input order {input_order!r}") + + +def keys_a(scenario: str, input_order: str) -> list[int]: + """The data key ``x`` of A's rows in input order: the keys of the completion orders.""" + return [x for _, _, x in rows_a(scenario, input_order)] + + +TAG_FIELDS = [pa.field("g", pa.large_string()), pa.field("n", pa.int64())] + + +def table_a(scenario: str, input_order: str = "shuffled") -> pa.Table: + """Source A: tags ``g``, ``n``; data ``x`` (unique), ``s`` (unique), ``m`` (nullable).""" + rows = rows_a(scenario, input_order) + xs = [x for _, _, x in rows] + return pa.table( + { + "g": [g for g, _, _ in rows], + "n": [n for _, n, _ in rows], + "x": xs, + "s": [f"s{x}" for x in xs], + "m": [None if x % 2 else 1000 + x for x in xs], + }, + schema=pa.schema( + [ + *TAG_FIELDS, + pa.field("x", pa.int64(), nullable=False), + pa.field("s", pa.large_string(), nullable=False), + pa.field("m", pa.int64()), + ] + ), + ) + + +def table_b(scenario: str) -> pa.Table: + """Source B: the tags of A in another order; data ``z`` (unique).""" + rows = SCENARIOS[scenario] + order = [3, 0, 5, 1, 4, 2] + return pa.table( + { + "g": [rows[i][0] for i in order], + "n": [rows[i][1] for i in order], + "z": [100 + i for i in order], + }, + schema=pa.schema([*TAG_FIELDS, pa.field("z", pa.int64(), nullable=False)]), + ) + + +def table_c(scenario: str) -> pa.Table: + """Source C: the tags of A reversed; data ``x`` from another range, positional ids.""" + rows = SCENARIOS[scenario] + order = [5, 4, 3, 2, 1, 0] + return pa.table( + { + "g": [rows[i][0] for i in order], + "n": [rows[i][1] for i in order], + "x": [200 + i for i in order], + }, + schema=pa.schema([*TAG_FIELDS, pa.field("x", pa.int64(), nullable=False)]), + ) + + +@dataclass +class Sources: + """The three sources of a scenario and the oracle that knows their rows.""" + + a: ArrowTableSource + b: ArrowTableSource + c: ArrowTableSource + oracle: Oracle + + +def make_sources(scenario: str, input_order: str = "shuffled") -> Sources: + """Build sources A, B and C of ``scenario`` and register their rows in an oracle. + + ``input_order`` orders the rows of A; B and C keep their own orders. + """ + oracle = Oracle() + built: dict[str, ArrowTableSource] = {} + for name, table, record_id_column in ( + ("A", table_a(scenario, input_order), "x"), + ("B", table_b(scenario), "z"), + ("C", table_c(scenario), None), + ): + built[name] = ArrowTableSource( + table, + tag_columns=list(TAG_COLUMNS), + source_id=name, + record_id_column=record_id_column, + ) + oracle.add_source( + name, table, tag_columns=TAG_COLUMNS, record_id_column=record_id_column + ) + return Sources(a=built["A"], b=built["B"], c=built["C"], oracle=oracle) + + +ALL_GROUPS = { + "meta": True, + "context": True, + "source": True, + "system_tags": True, + "content_hash": True, +} + +# Every ColumnConfig that changes which columns a table has, their order, or its row order. +COLUMN_CONFIGS: dict[str, dict[str, Any]] = { + "default": {}, + "sort_by_tags": {"columns": {"sort_by_tags": True}}, + "content_hash": {"columns": {"content_hash": True}}, + "content_hash_named": {"columns": {"content_hash": "h"}}, + "content_hash_sorted": {"columns": {"content_hash": True, "sort_by_tags": True}}, + "source": {"columns": {"source": True}}, + "system_tags": {"columns": {"system_tags": True}}, + "context": {"columns": {"context": True}}, + "meta": {"columns": {"meta": True}}, + "all_groups_unsorted": {"columns": {**ALL_GROUPS, "sort_by_tags": False}}, + "all_info": {"all_info": True}, +} + + +def content_hash_column(config: dict[str, Any]) -> str: + """The name of the content hash column ``config`` asks for (default when none).""" + value = config.get("columns", {}).get("content_hash") + return value if isinstance(value, str) else "_content_hash" diff --git a/tests/test_core/alignment/gate.py b/tests/test_core/alignment/gate.py new file mode 100644 index 000000000..5867c41ff --- /dev/null +++ b/tests/test_core/alignment/gate.py @@ -0,0 +1,198 @@ +"""Force the order in which async rows complete, without wall-clock time (ITL-1034). + +A gated function awaits its turn before it returns: the row keyed ``k`` returns only after +the row before it in the prescribed order has returned. The order is then the order the +rows complete in, whatever the scheduler would otherwise do. Nothing sleeps: rows wait on +``asyncio.Event`` objects, and the timeout on each wait is a deadlock guard that raises a +named error, never the mechanism of an assertion. + +The gate is armed only for async runs. A sync run calls each function in input order on a +fresh event loop, so a gate armed there would wait forever for a row that has not started. +""" + +from __future__ import annotations + +import asyncio +import random +from collections.abc import Callable, Hashable, Sequence + +from orcapod import function_pod + +# Seconds a gated row waits for its predecessor before the gate reports a deadlock. A +# deadlock guard only: in a passing run every wait ends as soon as its predecessor returns. +DEADLOCK_GUARD_SECONDS = 30.0 + + +class CompletionOrderDeadlock(RuntimeError): + """A gated row waited too long for its predecessor: the order was not realisable.""" + + +class CompletionGate: + """Make gated rows complete in a prescribed order. + + Attributes: + completed: The keys of the rows that returned, in the order they returned. Every + gated call appends to it, armed or not, so a test can assert the order a run + actually realised. + """ + + def __init__(self) -> None: + """Build a disarmed gate.""" + self._order: list[Hashable] | None = None + self._events: dict[Hashable, asyncio.Event] = {} + self.completed: list[Hashable] = [] + + def arm(self, order: Sequence[Hashable]) -> None: + """Make the next rows complete in ``order``; forget earlier completions. + + Args: + order: Every key the run will compute, each once, in the order the rows must + complete in. + + Raises: + ValueError: If a key appears twice. + """ + if len(set(order)) != len(order): + raise ValueError(f"a completion order names a key twice: {list(order)!r}") + self._order = list(order) + self._events = {} + self.completed = [] + + def disarm(self) -> None: + """Let rows complete as the scheduler runs them; forget earlier completions.""" + self._order = None + self._events = {} + self.completed = [] + + def _event(self, key: Hashable) -> asyncio.Event: + """Return the event set when the row ``key`` returns, created on first use.""" + if key not in self._events: + self._events[key] = asyncio.Event() + return self._events[key] + + async def wait_turn(self, key: Hashable) -> None: + """Return once every row before ``key`` in the armed order has returned. + + Raises: + CompletionOrderDeadlock: If the predecessor does not return within the guard. + ValueError: If the gate is armed and ``key`` is not in its order. + """ + if self._order is None: + return + position = self._order.index(key) + if position == 0: + return + before = self._order[position - 1] + try: + await asyncio.wait_for( + self._event(before).wait(), timeout=DEADLOCK_GUARD_SECONDS + ) + except TimeoutError as exc: + raise CompletionOrderDeadlock( + f"row {key!r} waited for row {before!r}, which never returned; " + f"order {self._order!r}" + ) from exc + + def finish(self, key: Hashable) -> None: + """Record that the row ``key`` returned and release the next row.""" + self.completed.append(key) + if self._order is not None: + self._event(key).set() + + +# --------------------------------------------------------------------------- +# Orders +# --------------------------------------------------------------------------- + + +def natural(keys: Sequence[Hashable]) -> list[Hashable]: + """The input order.""" + return list(keys) + + +def reversed_order(keys: Sequence[Hashable]) -> list[Hashable]: + """The input order reversed.""" + return list(reversed(keys)) + + +def interleaved(keys: Sequence[Hashable]) -> list[Hashable]: + """First, last, second, second to last, and so on.""" + out: list[Hashable] = [] + low, high = 0, len(keys) - 1 + while low <= high: + out.append(keys[low]) + if low != high: + out.append(keys[high]) + low += 1 + high -= 1 + return out + + +def one_slow(keys: Sequence[Hashable]) -> list[Hashable]: + """The input order with the first row completing last.""" + return [*keys[1:], keys[0]] + + +def seeded(seed: int) -> Callable[[Sequence[Hashable]], list[Hashable]]: + """A random order, the same for the same ``seed`` and keys.""" + + def order(keys: Sequence[Hashable]) -> list[Hashable]: + """The keys shuffled with ``random.Random(seed)``.""" + shuffled = list(keys) + random.Random(seed).shuffle(shuffled) + return shuffled + + return order + + +ORDERS: dict[str, Callable[[Sequence[Hashable]], list[Hashable]]] = { + "natural": natural, + "reversed": reversed_order, + "interleaved": interleaved, + "one_slow": one_slow, + "random_0": seeded(0), + "random_1": seeded(1), + "random_2": seeded(2), +} + + +# --------------------------------------------------------------------------- +# Gated function pods +# --------------------------------------------------------------------------- + +GATE = CompletionGate() + + +def scale(x: int) -> int: + """What ``gated_scale`` returns for ``x``: the oracle's derivation of ``y``.""" + return 10 * x + + +FAILING_REMAINDER = 1 + + +def fails(x: int) -> bool: + """Whether ``gated_fragile`` raises for ``x``.""" + return x % 3 == FAILING_REMAINDER + + +@function_pod(output_keys="y") +async def gated_scale(x: int) -> int: + """Return ``10 * x`` once ``GATE`` lets the row keyed ``x`` complete.""" + await GATE.wait_turn(x) + try: + return scale(x) + finally: + GATE.finish(x) + + +@function_pod(output_keys="y") +async def gated_fragile(x: int) -> int: + """As ``gated_scale``, but raise for every ``x`` where ``fails(x)``.""" + await GATE.wait_turn(x) + try: + if fails(x): + raise ValueError(f"gated_fragile fails for x={x}") + return scale(x) + finally: + GATE.finish(x) diff --git a/tests/test_core/alignment/jobs.py b/tests/test_core/alignment/jobs.py new file mode 100644 index 000000000..460e29a81 --- /dev/null +++ b/tests/test_core/alignment/jobs.py @@ -0,0 +1,212 @@ +"""Jobs the row-alignment tests run, and the tables they read from them (ITL-1034). + +The job: source ``A`` feeds the gated function node ``scaled`` (``y = 10 * x``); ``scaled`` +feeds an operator node ``joined`` (``Join`` with source ``B``, cached in the store so it can +be read back) and an operator node ``grouped`` (``GroupBy`` on ``g``). A run is made with one +of four entry points and, on the async ones, with the gate forcing a completion order. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pyarrow as pa + +from orcapod import PipelineJob +from orcapod.core.operators import GroupBy, Join +from orcapod.databases import DeltaTableDatabase, InMemoryArrowDatabase +from orcapod.pipeline import AsyncPipelineOrchestrator, SyncPipelineOrchestrator +from orcapod.pipeline.run_summary import RunSummary +from orcapod.system_constants import constants +from orcapod.types import CacheMode + +from .checker import Layout +from .fixtures import Sources, content_hash_column, keys_a, make_sources +from .gate import GATE, gated_fragile, gated_scale, scale +from .oracle import Origin + +JOB_NAME = "alignment" + +# How a job is run. "sync" runs rows in input order; the others run them concurrently. +MODES = ("sync", "async_run", "async_run_async", "async_in_loop") + +STORES: dict[str, Callable[[Path], Any]] = { + "in_memory": lambda _: InMemoryArrowDatabase(), + "delta": lambda path: DeltaTableDatabase(path), +} + +# Columns that differ between two runs by design: a function output's token carries the +# output datagram's uuid, made when the row is computed. +PER_RUN_COLUMNS = ("_source_y",) + + +def run_job(job: PipelineJob, mode: str) -> RunSummary: + """Run ``job`` with the entry point ``mode`` names and return its run summary.""" + if mode == "sync": + return job.run(orchestrator=SyncPipelineOrchestrator()) + if mode == "async_run": + return job.run(orchestrator=AsyncPipelineOrchestrator()) + if mode == "async_run_async": + return asyncio.run(job.run_async()) + + async def notebook_cell() -> RunSummary: + """A notebook cell: the job run inside the kernel's running loop.""" + return job.run(orchestrator=AsyncPipelineOrchestrator()) + + if mode == "async_in_loop": + return asyncio.run(notebook_cell()) + raise ValueError(f"unknown mode {mode!r}") + + +def derive_y(sources: Sources) -> None: + """Tell the oracle that ``y`` is ``gated_scale`` of an ``A`` row.""" + + def y_of(origin: Origin) -> Any: + """``gated_scale``'s output for ``origin``.""" + return scale(origin.values["x"]) + + sources.oracle.derive("y", "A", y_of) + + +def build_job( + scenario: str, + input_order: str, + store: Any, + *, + fragile: bool = False, + cache: CacheMode = CacheMode.LOG, +) -> tuple[PipelineJob, Sources]: + """Record the alignment job of ``scenario`` on ``store``; the oracle knows ``y``.""" + sources = make_sources(scenario, input_order) + derive_y(sources) + job = PipelineJob(name=JOB_NAME, store=store) + pod = gated_fragile if fragile else gated_scale + with job: + scaled = pod.pod(sources.a, label="scaled") + Join()(scaled, sources.b, label="joined") + GroupBy(by=["g"])(scaled, label="grouped") + # OperatorNode.attach_databases is the public way to store an operator's rows. + job.nodes["joined"].attach_databases( + pipeline_database=store.at(JOB_NAME), cache_mode=cache + ) + return job, sources + + +@dataclass +class Run: + """A finished run: the job, its sources and what the gate saw.""" + + job: PipelineJob + sources: Sources + store: Any + summary: RunSummary + completed: list[Any] + + +def make_run( + scenario: str, + input_order: str, + mode: str, + order: list[Any] | None, + store_kind: str, + directory: Path, + *, + fragile: bool = False, +) -> Run: + """Run the alignment job; on an async mode the gate forces ``order``.""" + store = STORES[store_kind](directory / "store") + job, sources = build_job(scenario, input_order, store, fragile=fragile) + if order is None: + GATE.disarm() + else: + GATE.arm(order) + try: + summary = run_job(job, mode) + finally: + completed = list(GATE.completed) + GATE.disarm() + return Run(job, sources, store, summary, completed) + + +def function_tokens(job: PipelineJob) -> dict[tuple[str, bytes], str]: + """The stored ``_source_y`` of every ``scaled`` record, keyed by its input record id.""" + records = job.nodes["scaled"].get_all_records(all_info=True) + assert records is not None, "the function node stored no records" + (record_id,) = [ + c + for c in records.column_names + if c.startswith(constants.SYSTEM_TAG_RECORD_ID_PREFIX) + ] + return { + ("y", row[record_id]): row["_source_y"] + for row in records.select([record_id, "_source_y"]).to_pylist() + } + + +# --------------------------------------------------------------------------- +# Views: the tables one node gives, and their layouts +# --------------------------------------------------------------------------- + + +@dataclass(frozen=True) +class View: + """One table-producing call on one node of the job. + + Attributes: + node: The node's label. + call: ``as_table``, ``head`` or ``as_source``. + """ + + node: str + call: str + + @property + def name(self) -> str: + """``.``, the test id.""" + return f"{self.node}.{self.call}" + + +VIEWS = ( + View("scaled", "as_table"), + View("scaled", "head"), + View("scaled", "as_source"), + View("joined", "as_table"), + View("joined", "head"), + View("grouped", "as_table"), + View("grouped", "head"), +) + + +def produce(job: PipelineJob, view: View, config: dict[str, Any]) -> pa.Table: + """The table ``view`` gives with the column configuration ``config``.""" + node = job.nodes[view.node] + if view.call == "as_table": + return node.as_table(**config) + if view.call == "head": + return node.head(100, **config).as_table() + if view.call == "as_source": + return node.as_source().as_table(**config) + raise ValueError(f"unknown call {view.call!r}") + + +def layout(job: PipelineJob, view: View, config: dict[str, Any]) -> Layout: + """How the columns of ``view``'s table relate to the origin rows.""" + tag_keys, data_keys = job.nodes[view.node].keys() + members: frozenset[str] = frozenset() + if view.node == "grouped": + members = frozenset(c for c in (*tag_keys, *data_keys) if c != "g") + return Layout( + tag_keys=tuple(tag_keys), + data_keys=tuple(data_keys), + members=members, + content_hash_column=content_hash_column(config), + ) + + +def natural_keys(scenario: str, input_order: str) -> list[int]: + """The keys of A's rows in input order: the order a sync run computes them in.""" + return keys_a(scenario, input_order) diff --git a/tests/test_core/alignment/oracle.py b/tests/test_core/alignment/oracle.py new file mode 100644 index 000000000..07657892f --- /dev/null +++ b/tests/test_core/alignment/oracle.py @@ -0,0 +1,173 @@ +"""What every row of a test's sources is, derived from the rows the test wrote (ITL-1034). + +The oracle never reads a table Orcapod produced. It takes the Arrow table a test hands to a +source, and recomputes each row's provenance token and record id by the recipe the library +documents (``docs/reference/hashing.md``, site 5): the token is ``"="`` with a +``record_id_column``, else ``"row_"``; the record id is the UUID v5 of +``"::"``. Aggregated system tags fold by site 6b. +""" + +from __future__ import annotations + +import hashlib +import json +import uuid +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass, field +from typing import Any + +import pyarrow as pa + +# The namespace of a source row's record id. The code (``core/sources/stream_builder.py``, +# ``_SOURCE_RECORD_ID_NAMESPACE``) is the authority: docs/reference/hashing.md site 5 names +# another string, a documentation error reported with ITL-1034. +SOURCE_RECORD_ID_NAMESPACE = uuid.uuid5( + uuid.NAMESPACE_URL, "https://orcapod.org/namespaces/source-record-id" +) +# The namespace of a folded (aggregated) record id: docs/reference/hashing.md site 6b. +AGGREGATED_RECORD_ID_NAMESPACE = uuid.uuid5( + uuid.NAMESPACE_URL, "https://orcapod.org/namespaces/aggregated-record-id" +) +BLOCK_SEPARATOR = "::" + + +def source_record_id(source_id: str, token: str) -> bytes: + """The record id of the source row ``token`` of ``source_id`` (site 5).""" + return uuid.uuid5( + SOURCE_RECORD_ID_NAMESPACE, f"{source_id}{BLOCK_SEPARATOR}{token}" + ).bytes + + +def fold_record_ids(record_ids: Sequence[bytes | None]) -> bytes: + """The folded record id of an ordered member list (site 6b).""" + name = BLOCK_SEPARATOR.join("null" if v is None else v.hex() for v in record_ids) + return uuid.uuid5(AGGREGATED_RECORD_ID_NAMESPACE, name).bytes + + +def fold_values(values: Sequence[Any]) -> str: + """The folded value of any other system tag of an ordered member list (site 6b).""" + members = json.dumps( + [None if v is None else str(v) for v in values], + separators=(",", ":"), + ensure_ascii=False, + ) + return hashlib.sha256(members.encode("utf-8")).hexdigest() + + +@dataclass(frozen=True) +class Origin: + """One row of one source, as the test wrote it. + + Attributes: + source_id: The source's id. + token: The row's provenance token within the source. + record_id: The row's record id, recomputed from ``source_id`` and ``token``. + values: Every column of the row (tags and data), as Python values. + data_table: The row's data columns as a one-row Arrow table with the source's + types: what the source's ``Data`` hashes. + """ + + source_id: str + token: str + record_id: bytes + values: Mapping[str, Any] + data_table: pa.Table = field(compare=False, repr=False) + + +@dataclass(frozen=True) +class Derivation: + """A column a function computes from one origin row. + + Attributes: + source_id: The source whose row is the function's input. + compute: The value the function returns for that row. + """ + + source_id: str + compute: Callable[[Origin], Any] + + +class Oracle: + """Every origin row of a test, and how each output column relates to them.""" + + def __init__(self) -> None: + """Build an oracle with no sources.""" + self.by_record_id: dict[bytes, Origin] = {} + self.by_token: dict[tuple[str, str], Origin] = {} + self.origins: list[Origin] = [] + self.derivations: dict[str, Derivation] = {} + self.renames: dict[str, str] = {} + + def add_source( + self, + source_id: str, + table: pa.Table, + *, + tag_columns: Sequence[str], + record_id_column: str | None = None, + ) -> None: + """Register the rows of a source from the table the test gave it. + + Args: + source_id: The id the test gave the source. + table: The table the test gave the source, in its row order. + tag_columns: The source's tag columns. + record_id_column: The source's record id column, if any. + """ + data_columns = [c for c in table.column_names if c not in tag_columns] + for index, values in enumerate(table.to_pylist()): + if record_id_column is not None: + token = f"{record_id_column}={values[record_id_column]}" + else: + token = f"row_{index}" + origin = Origin( + source_id=source_id, + token=token, + record_id=source_record_id(source_id, token), + values=values, + data_table=table.slice(index, 1).select(data_columns), + ) + self.origins.append(origin) + self.by_record_id[origin.record_id] = origin + self.by_token[(source_id, token)] = origin + + def derive( + self, column: str, source_id: str, compute: Callable[[Origin], Any] + ) -> None: + """Declare that the output column ``column`` is computed from a ``source_id`` row.""" + self.derivations[column] = Derivation(source_id, compute) + + def rename(self, new: str, old: str) -> None: + """Declare that the output column ``new`` is the input column ``old`` renamed.""" + self.renames[new] = old + + def base_name(self, column: str) -> str: + """The input column an output column holds, following renames.""" + return self.renames.get(column, column) + + def origins_for_value(self, column: str, value: Any) -> list[Origin]: + """Every origin whose value of the output column ``column`` is ``value``.""" + derivation = self.derivations.get(column) + if derivation is not None: + return [ + o + for o in self.origins + if o.source_id == derivation.source_id + and derivation.compute(o) == value + ] + name = self.base_name(column) + return [o for o in self.origins if name in o.values and o.values[name] == value] + + def expected_value(self, column: str, origin: Origin) -> Any: + """The value the output column ``column`` has for a row whose origin is ``origin``.""" + derivation = self.derivations.get(column) + if derivation is not None: + return derivation.compute(origin) + return origin.values[self.base_name(column)] + + def holds(self, column: str, origin: Origin) -> bool: + """Whether a row of ``origin``'s source gives the output column ``column``.""" + derivation = self.derivations.get(column) + if derivation is not None: + return derivation.source_id == origin.source_id + return self.base_name(column) in origin.values diff --git a/tests/test_core/alignment/records.py b/tests/test_core/alignment/records.py new file mode 100644 index 000000000..ac26c3f37 --- /dev/null +++ b/tests/test_core/alignment/records.py @@ -0,0 +1,106 @@ +"""Check the function node's stored records row by row (ITL-1034). + +A record of ``scaled`` holds the input row's tags and system tags, the input's source-info +token, the output, and bookkeeping. Each per-row column is recomputed from the record and +its origin row: + +* the record id is the origin row's, the input token ``_source_x`` names that row; +* ``y`` is ``gated_scale`` of the origin's ``x``; +* ``__output_data_hash`` is the Arrow hash of the record's own ``y``; +* ``__input_data_hash`` (and its ``_right`` copy) is the Arrow hash of the origin's ``x``, + built from the table the test wrote; +* ``_source_y`` names the output datagram ``__data_id``; +* ``__data_context`` and ``_context_key`` are the data context's key. + +The remaining columns are not derived from the row: ``__pod_ts`` is the time the row was +computed; ``__computed``, ``__is_ephemeral``, ``__pf_var_*`` and ``__pf_exec_*`` describe the +function and the run, the same in every row of one run. They are listed by name, so a new +per-row column fails the check until a rule covers it. +""" + +from __future__ import annotations + +from orcapod import PipelineJob +from orcapod.system_constants import constants + +from .checker import split_token +from .fixtures import Sources + +NOT_ROW_DERIVED_PREFIXES = ("__pf_var_", "__pf_exec_") +NOT_ROW_DERIVED = {"__pod_ts", "__computed", "__is_ephemeral"} + + +def assert_records_aligned(job: PipelineJob, sources: Sources) -> None: + """Assert every stored record of ``scaled`` belongs, column by column, to its row.""" + node = job.nodes["scaled"] + records = node.get_all_records(all_info=True) + assert records is not None and records.num_rows > 0, "no records stored" + context = node.data_context + hasher = context.arrow_hasher + oracle = sources.oracle + tag_keys = node.keys()[0] + (source_id_column,) = [ + c + for c in records.column_names + if c.startswith(constants.SYSTEM_TAG_SOURCE_ID_PREFIX) + ] + (record_id_column,) = [ + c + for c in records.column_names + if c.startswith(constants.SYSTEM_TAG_RECORD_ID_PREFIX) + ] + covered = { + *tag_keys, + source_id_column, + record_id_column, + "_source_x", + "y", + "_source_y", + "__data_id", + "__output_data_hash", + "__input_data_hash", + "__input_data_hash_right", + "__data_context", + constants.CONTEXT_KEY, + *NOT_ROW_DERIVED, + } + unknown = [ + c + for c in records.column_names + if c not in covered and not c.startswith(NOT_ROW_DERIVED_PREFIXES) + ] + assert not unknown, f"no alignment rule covers the record columns {unknown!r}" + + failures: list[str] = [] + for index, row in enumerate(records.to_pylist()): + + def fail(column: str, message: str, index: int = index) -> None: + """Record a failure of ``column`` in this record.""" + failures.append(f"record {index}, column {column!r}: {message}") + + origin = oracle.by_record_id.get(row[record_id_column]) + if origin is None or origin.source_id != row[source_id_column]: + fail(record_id_column, "record id is not a row of its source") + continue + for tag in tag_keys: + if row[tag] != origin.values[tag]: + fail(tag, f"{row[tag]!r}, its row has {origin.values[tag]!r}") + if row["_source_x"] != f"A::{origin.token}::x": + fail("_source_x", f"{row['_source_x']!r} does not name row {origin.token!r}") + if row["y"] != oracle.expected_value("y", origin): + fail("y", f"{row['y']!r} is not the output for row {origin.token!r}") + _, datagram, column = split_token(row["_source_y"]) + if datagram != row["__data_id"].hex() or column != "y": + fail("_source_y", f"{row['_source_y']!r} does not name __data_id") + one_row = records.slice(index, 1) + output_hash = hasher.hash_table(one_row.select(["y"])).to_prefixed_digest() + if row["__output_data_hash"] != output_hash: + fail("__output_data_hash", "is not the hash of the record's own y") + input_hash = hasher.hash_table(origin.data_table.select(["x"])) + for column in ("__input_data_hash", "__input_data_hash_right"): + if row[column] != input_hash.to_prefixed_digest(): + fail(column, f"is not the hash of row {origin.token!r}'s input") + for column in ("__data_context", constants.CONTEXT_KEY): + if row[column] != context.context_key: + fail(column, f"{row[column]!r}") + assert not failures, "\n".join(failures[:12]) diff --git a/tests/test_core/alignment/test_checker.py b/tests/test_core/alignment/test_checker.py new file mode 100644 index 000000000..71a548cd6 --- /dev/null +++ b/tests/test_core/alignment/test_checker.py @@ -0,0 +1,127 @@ +"""The row-alignment checker accepts aligned tables and rejects every kind of misalignment. + +A checker that passes everything proves nothing. Each test here takes a table the checker +accepts and moves the cells of one column to other rows (a rotation, the shape of ITL-1029), +and the checker must name that column. +""" + +from __future__ import annotations + +import pyarrow as pa +import pytest + +from orcapod.core.operators import Batch, Join +from orcapod.system_constants import constants + +from .checker import ( + Layout, + assert_all_columns_checked, + assert_rows_aligned, + assert_same_rows_by_identity, +) +from .fixtures import TAG_COLUMNS, make_sources + + +def rotated(table: pa.Table, column: str) -> pa.Table: + """``table`` with the cells of ``column`` each moved one row down (the last to the top).""" + index = table.column_names.index(column) + values = table.column(column) + order = [table.num_rows - 1, *range(table.num_rows - 1)] + return table.set_column(index, table.field(column), values.take(order)) + + +def source_table() -> tuple[pa.Table, Layout, object, object]: + """Source A of the unique scenario with every column group, its layout and oracle.""" + sources = make_sources("unique") + table = sources.a.as_table(all_info=True) + layout = Layout(tag_keys=TAG_COLUMNS, data_keys=sources.a.keys()[1]) + return table, layout, sources.oracle, sources.a.data_context + + +def test_an_aligned_source_table_passes_and_every_column_is_compared() -> None: + table, layout, oracle, context = source_table() + checked = assert_rows_aligned(table, layout, oracle, context) # type: ignore[arg-type] + assert_all_columns_checked(table, checked) + + +def _misalignable_columns() -> list[str]: + """Every column of source A's full table whose rotation changes the table.""" + table, *_ = source_table() + # A column with one value in every row (the context key, a single source's id) is + # unchanged by a rotation; it is checked by test_a_wrong_context_key_is_named and the + # join test instead. + return [c for c in table.column_names if len(set(table.column(c).to_pylist())) > 1] + + +@pytest.mark.parametrize("column", _misalignable_columns()) +def test_a_rotated_column_is_named(column: str) -> None: + table, layout, oracle, context = source_table() + with pytest.raises(AssertionError, match=rf"column '{column}'"): + assert_rows_aligned(rotated(table, column), layout, oracle, context) # type: ignore[arg-type] + + +def test_a_wrong_context_key_is_named() -> None: + table, layout, oracle, context = source_table() + index = table.column_names.index(constants.CONTEXT_KEY) + wrong = table.set_column( + index, + table.field(constants.CONTEXT_KEY), + pa.array(["other"] * table.num_rows, type=pa.large_string()), + ) + with pytest.raises(AssertionError, match=constants.CONTEXT_KEY): + assert_rows_aligned(wrong, layout, oracle, context) # type: ignore[arg-type] + + +def test_a_column_no_rule_covers_fails() -> None: + table, layout, oracle, context = source_table() + extra = table.append_column("__unexplained", pa.array(range(table.num_rows))) + with pytest.raises(AssertionError, match="__unexplained"): + assert_rows_aligned(extra, layout, oracle, context) # type: ignore[arg-type] + + +def test_rotated_join_record_ids_are_named() -> None: + sources = make_sources("unique") + joined = Join()(sources.a, sources.b) + table = joined.as_table(all_info=True) + layout = Layout(tag_keys=TAG_COLUMNS, data_keys=joined.keys()[1]) + assert_rows_aligned(table, layout, sources.oracle, joined.data_context) + record_ids = [ + c + for c in table.column_names + if c.startswith(constants.SYSTEM_TAG_RECORD_ID_PREFIX) + ] + assert len(record_ids) == 2 + for column in record_ids: + with pytest.raises(AssertionError, match=f"column '{column}'"): + assert_rows_aligned( + rotated(table, column), layout, sources.oracle, joined.data_context + ) + + +def test_a_rotated_folded_record_id_is_named() -> None: + sources = make_sources("unique") + batched = Batch(2)(sources.a) + table = batched.as_table(all_info=True) + tag_keys, data_keys = batched.keys() + layout = Layout( + tag_keys=tag_keys, data_keys=data_keys, members=frozenset(tag_keys + data_keys) + ) + checked = assert_rows_aligned(table, layout, sources.oracle, batched.data_context) + assert_all_columns_checked(table, checked) + (record_id,) = [ + c + for c in table.column_names + if c.startswith(constants.SYSTEM_TAG_RECORD_ID_PREFIX) + ] + with pytest.raises(AssertionError, match="folded record id"): + assert_rows_aligned( + rotated(table, record_id), layout, sources.oracle, batched.data_context + ) + + +def test_identity_comparison_ignores_row_order_but_not_cell_moves() -> None: + table, *_ = source_table() + reversed_rows = table.take(list(reversed(range(table.num_rows)))) + assert_same_rows_by_identity(table, reversed_rows) + with pytest.raises(AssertionError): + assert_same_rows_by_identity(table, rotated(table, "x")) diff --git a/tests/test_core/alignment/test_nodes.py b/tests/test_core/alignment/test_nodes.py new file mode 100644 index 000000000..32800c13d --- /dev/null +++ b/tests/test_core/alignment/test_nodes.py @@ -0,0 +1,240 @@ +"""Every row of every job node's table carries its own derived columns (ITL-1034, ITL-1029). + +The product: input row order (shuffled, tag-sorted, reverse tag-sorted) x tie scenario +(unique, duplicate, null tags) x store (in memory, Delta) x entry point and completion +order x node view (function node ``as_table`` / ``head`` / ``as_source``, operator nodes +``joined`` and ``grouped``) x every ``ColumnConfig`` that changes columns or row order. +Entry points and orders: sync ``run`` (input order); async ``run`` in every forced order +(``ORDERS``); ``run_async`` and ``run`` inside a running loop in the orders +``ENTRY_POINT_ORDERS`` (they share the async execution path with async ``run`` and differ +only in how the run is entered). Each run is made once and shared by the tests that read +it; each test checks every column configuration of one view and names every configuration +that fails. ``head`` refuses ``sort_by_tags``, so it is checked without it. + +Readback: the stored rows read by a second run (rows from the store, not computed) and by +a job loaded with ``PipelineJob.load``, after writes in input order and in forced orders. +""" + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pytest + +from orcapod import PipelineJob + +from .checker import ( + Layout, + assert_rows_aligned, + assert_same_rows_by_identity, +) +from .fixtures import COLUMN_CONFIGS, INPUT_ORDERS, SCENARIOS +from .gate import ORDERS +from .jobs import ( + MODES, + PER_RUN_COLUMNS, + STORES, + VIEWS, + Run, + View, + build_job, + function_tokens, + layout, + make_run, + natural_keys, + produce, + run_job, +) + +RunKey = tuple[str, str, str, str, str] + +ENTRY_POINT_ORDERS = ("reversed", "one_slow", "random_0") + + +@pytest.fixture(scope="module") +def runs(tmp_path_factory: pytest.TempPathFactory) -> Callable[[RunKey], Run]: + """Make each run once per module: ``(scenario, input order, mode, order, store)``.""" + made: dict[RunKey, Run] = {} + + def get(key: RunKey) -> Run: + """The run ``key`` names, made on first use.""" + if key not in made: + scenario, input_order, mode, order_name, store_kind = key + keys = natural_keys(scenario, input_order) + order = None if mode == "sync" else ORDERS[order_name](keys) + made[key] = make_run( + scenario, + input_order, + mode, + order, + store_kind, + tmp_path_factory.mktemp("run"), + ) + return made[key] + + return get + + +def run_keys() -> list[pytest.ParameterSet]: + """Every run of the product (see the module docstring).""" + orders_of = { + "sync": ["natural"], + "async_run": list(ORDERS), + "async_run_async": list(ENTRY_POINT_ORDERS), + "async_in_loop": list(ENTRY_POINT_ORDERS), + } + keys = [] + for store_kind in STORES: + for scenario in SCENARIOS: + for input_order in INPUT_ORDERS: + for mode in MODES: + for order in orders_of[mode]: + key = (scenario, input_order, mode, order, store_kind) + keys.append(pytest.param(key, id="-".join(key))) + return keys + + +def check_every_config( + job: PipelineJob, + view: View, + oracle: Any, + tokens: dict[tuple[str, bytes], str] | None, +) -> None: + """Check ``view`` under every column configuration; name every one that fails.""" + failures: list[str] = [] + for name, config in configs_for(view): + table = produce(job, view, config) + view_layout: Layout = layout(job, view, config) + context = job.nodes[view.node].data_context + try: + assert_rows_aligned( + table, view_layout, oracle, context, function_tokens=tokens + ) + except AssertionError as exc: + failures.append(f"[{name}] {exc}") + assert not failures, f"{view.name}: " + "\n".join(failures) + + +def configs_for(view: View) -> list[tuple[str, dict[str, Any]]]: + """The column configurations ``view`` accepts: ``head`` refuses ``sort_by_tags``.""" + return [ + (name, config) + for name, config in COLUMN_CONFIGS.items() + if not (view.call == "head" and config.get("columns", {}).get("sort_by_tags")) + ] + + +@pytest.mark.parametrize("view", VIEWS, ids=[v.name for v in VIEWS]) +@pytest.mark.parametrize("key", run_keys()) +def test_every_row_of_a_node_view_is_aligned( + key: RunKey, view: View, runs: Callable[[RunKey], Run] +) -> None: + run = runs(key) + check_every_config(run.job, view, run.sources.oracle, function_tokens(run.job)) + + +@pytest.mark.parametrize("key", run_keys()) +def test_the_gate_realised_the_order( + key: RunKey, runs: Callable[[RunKey], Run] +) -> None: + """The run completed its rows in the order the test meant to force.""" + scenario, input_order, mode, order_name, _ = key + keys = natural_keys(scenario, input_order) + expected = keys if mode == "sync" else ORDERS[order_name](keys) + assert runs(key).completed == expected + + +@pytest.mark.parametrize("key", run_keys()) +def test_stored_records_are_aligned( + key: RunKey, runs: Callable[[RunKey], Run] +) -> None: + """The function node's stored records: hashes, tokens and ids belong to their rows.""" + from .records import assert_records_aligned + + run = runs(key) + assert_records_aligned(run.job, run.sources) + + +@pytest.mark.parametrize("key", [k for k in run_keys() if k.values[0][2] != "sync"]) +def test_async_equals_sync_row_by_row( + key: RunKey, runs: Callable[[RunKey], Run] +) -> None: + """Keyed on record ids, every async table equals the sync one, cell for cell.""" + scenario, input_order, _, _, store_kind = key + sync = runs((scenario, input_order, "sync", "natural", store_kind)) + run = runs(key) + failures = [] + for view in VIEWS: + for name, config in configs_for(view): + expected = produce(sync.job, view, config) + if not _has_record_ids(expected): + continue + try: + assert_same_rows_by_identity( + expected, produce(run.job, view, config), ignore=PER_RUN_COLUMNS + ) + except AssertionError as exc: + failures.append(f"{view.name} [{name}]: {exc}") + assert not failures, "\n".join(failures) + + +def _has_record_ids(table: pa.Table) -> bool: + """Whether ``table`` has the system tags rows are keyed by.""" + return any(c.startswith("_tag_record_id") for c in table.column_names) + + +# --------------------------------------------------------------------------- +# Readback +# --------------------------------------------------------------------------- + +WRITES = ("sync-natural", "async_run-reversed", "async_run-random_0") +READS = ("rerun_sync", "rerun_async", "load") + + +def read_back( + run: Run, scenario: str, input_order: str, how: str, directory: Path +) -> PipelineJob: + """A job that reads ``run``'s stored rows: a second run, or ``PipelineJob.load``.""" + if how == "load": + saved = directory / "job.json" + run.job.save(saved) + return PipelineJob.load(str(saved), store=run.store) + job, _ = build_job(scenario, input_order, run.store) + summary = run_job(job, "sync" if how == "rerun_sync" else "async_run") + assert summary.nodes["scaled"].computed == 0, "the rows were not read back" + return job + + +@pytest.mark.parametrize("how", READS) +@pytest.mark.parametrize("write", WRITES) +@pytest.mark.parametrize("store_kind", list(STORES)) +@pytest.mark.parametrize("input_order", INPUT_ORDERS) +@pytest.mark.parametrize("scenario", list(SCENARIOS)) +def test_rows_read_back_from_the_store_are_aligned( + scenario: str, + input_order: str, + store_kind: str, + write: str, + how: str, + runs: Callable[[RunKey], Run], + tmp_path: Path, +) -> None: + mode, order = write.split("-") + run = runs((scenario, input_order, mode, order, store_kind)) + job = read_back(run, scenario, input_order, how, tmp_path) + tokens = function_tokens(run.job) + # A loaded job reads its function node from the store. Its operator nodes are left + # out: ``grouped`` stores nothing, and a loaded ``joined`` cannot read its stored + # cache (reported with ITL-1034: re-attached in REPLAY mode it reads no rows, or + # looks for another path when a tag column is nullable). + views = [v for v in VIEWS if not (how == "load" and v.node != "scaled")] + failures = [] + for view in views: + try: + check_every_config(job, view, run.sources.oracle, tokens) + except AssertionError as exc: + failures.append(str(exc)) + assert not failures, "\n".join(failures) From 621cddf994c4c9125e6fa93370a1be914e185374 Mon Sep 17 00:00:00 2001 From: "Edgar Y. Walker" <> Date: Thu, 12 Nov 2026 23:04:48 -0800 Subject: [PATCH 03/37] docs(specs): repro scripts from the ITL-1034 code sweep (ITL-1043, ITL-1045, ITL-1046, ITL-1047) Co-Authored-By: Claude Opus 5.5 --- superpowers/specs/itl-1034-sweep/README.md | 12 ++ .../itl1043_loaded_operator_cache.py | 122 ++++++++++++++++++ .../itl1045_two_pass_as_table.py | 47 +++++++ .../itl-1034-sweep/itl1046_index_token.py | 38 ++++++ .../itl1047_same_stream_twice.py | 43 ++++++ 5 files changed, 262 insertions(+) create mode 100644 superpowers/specs/itl-1034-sweep/README.md create mode 100644 superpowers/specs/itl-1034-sweep/itl1043_loaded_operator_cache.py create mode 100644 superpowers/specs/itl-1034-sweep/itl1045_two_pass_as_table.py create mode 100644 superpowers/specs/itl-1034-sweep/itl1046_index_token.py create mode 100644 superpowers/specs/itl-1034-sweep/itl1047_same_stream_twice.py diff --git a/superpowers/specs/itl-1034-sweep/README.md b/superpowers/specs/itl-1034-sweep/README.md new file mode 100644 index 000000000..93d5366be --- /dev/null +++ b/superpowers/specs/itl-1034-sweep/README.md @@ -0,0 +1,12 @@ +# Repro scripts from the ITL-1034 code sweep + +Evidence for issues found by the row-alignment suite and its code sweep. Run each script from +the repository root with `uv run python superpowers/specs/itl-1034-sweep/