Skip to content

feat(hashing): adopt starfix 0.5 (hash format 2) under arrow_v0.2; ignore and keep rows hashed under arrow_v0.1 (ITL-910) - #309

Closed
eywalker wants to merge 21 commits into
mainfrom
eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod
Closed

eywalker wants to merge 21 commits into
mainfrom
eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod

Conversation

@eywalker

@eywalker eywalker commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Draft. Kept a draft on purpose: ITL-942 (wrong-hash fixes) is stacked on this branch and is meant to land with this golden regeneration, and the versioning design discussion (ITL-938) is still running. The handling of stores written by v0.2 below follows the developer's decisions of November 11, 2026.

Summary

Orcapod now hashes Arrow data with starfix 0.5.0 (hash format 2, prefix 00 00 02) under the new label arrow_v0.2 (was arrow_v0.1, starfix 0.4, format 1). Format 1 lost information, so different data reached the same row hash (ITL-848): nested-list lengths ([[[1]], [[2], [3]]] vs [[[1], [2]], [[3]]]), a null struct vs a struct of nulls, and non-nullable Booleans ([True] vs [True, False, False]); Boolean hashes also depended on chunking. Format 2 fixes these and hashes fixed_size_list data (ITL-927). Following developer decisions 1–6 of November 11, 2026: the label is bumped, the main path uses format 2 only (no legacy 0.4 mode), the order options stay off (every hasher call keeps today's order-independent behaviour), and the golden files were regenerated explicitly for this change.

Fixes ITL-910. Related: ITL-848 (the starfix-side gaps), ITL-927 (fixed-size lists), ITL-846 (hash_schema on extension types), ITL-913 / ITL-914 / ITL-938 (migration and versioning).

Dependency

  • pyproject.toml: starfix~=0.4.0 → starfix>=0.5.0,<0.6. starfix 0.5 needs pyarrow>=19; Orcapod's floor is already pyarrow>=20 (locked 23.0.1), so the pyarrow floor is unchanged.
  • uv.lock: only the starfix entry moves (0.4.0 → 0.5.0, sdist and wheel hashes) and the specifier line. (uv lock with uv 0.13.7 also rewrote unrelated markers and the lock revision; those changes were not kept.)

Label and context

  • hashing/versioned_hashers.py _CURRENT_ARROW_HASHER_ID, contexts/data/v0.1.json and the example in contexts/data/schemas/context_schema.json: arrow_v0.2. The context key and version stay std:v0.1:default / v0.1 (decision D4, precedent ITL-888); the context changelog records "arrow_v0.2 = starfix hash format 2".

Hasher changes besides the label

  • pyarrow's canonical extension types (arrow.uuid, arrow.fixed_shape_tensor) derive from pa.BaseExtensionType, not pa.ExtensionType, so Orcapod's normalization skipped them, and on starfix 0.5 they would have hashed as their storage type (an arrow.uuid table == a fixed_size_binary(16) table). They are now hashed with their extension name and metadata (include_metadata whenever a table has an extension type) (D2).
  • StarfixArrowHasher.hash_schema normalizes extension columns as hash_table does, so a Path column's schema no longer hashes like large_string (ITL-846; the strict xfail became a real test) (D3).
  • Neither moves an existing value: both inputs were unhashable before (verified: the regenerated golden values equal a simulation without these two changes).

Behaviour changes visible to users (besides hash values)

Input starfix 0.4 (arrow_v0.1) now
fixed_size_list column at any depth (source, operator) NotImplementedError at source construction hashes; type kept; fixed_size_list<T>[n] ≠ large_list<T>
arrow.uuid, arrow.fixed_shape_tensor columns NotImplementedError hash with extension identity
hash_schema with extension / null / view / interval types raised hashes
non-nullable field holding a null hashed silently ValueError: Non-nullable field 'x' contains nulls
zero-row table with a map column hashed ValueError
duplicate column names (e.g. a pa.Table value) hashed ValueError: Ambiguous node path 'x'
column "a/b" next to struct a{b} hashed ValueError: Ambiguous node path 'a/b'
null / map / interval / view data, union, run-end-encoded NotImplementedError ValueError (same rejection; starfix names column and type)
a store written by v0.2 (n/a) its rows are ignored, recomputed and kept; a side-effect log fails with SideEffectLogHashLabelError; see below

Each row has a test in tests/test_hashing/test_starfix_format2.py (and the extension-schema case in test_async_sync_equivalence.py). starfix's ValueError propagates unwrapped (D5). CachedSource._build_merged_stream keeps its except NotImplementedError (meant for an unbound SourceProxy); hashing rejections are now ValueError and propagate instead of being taken for "source cannot provide data" — in practice they already fail at source construction. Function pods still cannot declare tuple[float, float] (pre-existing, "Unsupported annotation"), so a fixed_size_list column reaches pods only through operators.

For PR #305 (not yet on main): its _unhashable_columns wrapper in stream_builder.py catches NotImplementedError from hash_table. Once both are on main, starfix raises ValueError and names the column path and type itself, and fixed_size_list hashes, so the wrapper should be removed (or catch ValueError), and #305's CHANGELOG / sources.md / DESIGN_ISSUES text saying fixed_size_list cannot be hashed should go; map stays unsupported (now ValueError).

Stores written by v0.2: rows ignored, recomputed, kept (developer decisions of November 11, 2026, ITL-938 session)

The earlier decision (refuse the store with StoreHashLabelError) was reversed by the developer: "As long as [we] just store and make use of arrow_v0.2, then it's ok that arrow_v0.1-based values are present next [to] arrow_v0.2", "NOT fail but still be possible/completely clear to distinguish old vs new entries", and a later migration must stay possible. Only a change of a table's schema goes into its path; this value-only change stays in the same tables. So:

  • Removed: orcapod.errors.StoreHashLabelError (never released), every check_stored_hash_labels() and the PipelineJob.run pre-flight.
  • v0.3 uses rows of the current label only. Key lookups cannot match an old row (the labelled bytes differ). Every read of a whole table goes through one filter, hashing/stored_hash_labels.keep_rows_hashed_under(table, column, current). Without it every reader below returned both generations (4 rows for 2 inputs; the 2 v0.2 rows before the first v0.3 run).
  • Old rows are untouched and listable: stored_hash_labels(database, record_path) returns e.g. {"arrow_v0.1": 3, "arrow_v0.2": 3} (documented in docs/reference/hashing.md, section 9, with an executed sample).
  • Skipped rows are reported (developer: "just total counts I think would be sufficient"): a logging WARNING on the logger orcapod.hashing.stored_hash_labels, through one function, report_skipped_rows(record_path, skipped). Proposed wording, for review:
    Skipped 3 rows of stored table 'my_job/double/.../pdb_v1': written under another hash label than the current one. The rows are left in place unchanged and can be migrated later.
    It is logged once per reader object (node, result cache, cached source) and table, not once per process: stored paths are relative to the store, so a process-wide record would silence a second store with the same layout. No per-row detail.
  • Side-effect logs fail instead (developer: "let's do fail for side-effect log only although I want to separately step into this later and discuss more of what's present"): new public orcapod.errors.SideEffectLogHashLabelError (a StoreError: fails the run whatever the error_policy; record_path, found_label, current_label, node_label). A side-effect node whose invocation log holds deliveries logged under another Arrow label raises it at the start of execute / async_execute, before delivering or writing anything. Function, operator and source nodes of the same job are not refused. The developer wants a separate, later discussion of side-effect logs.
Table (owner) Whole-table readers Filter Test on a table holding both labels (tests/test_pipeline/test_stored_hash_labels.py, in-memory + Delta)
Function pipeline table (FunctionJobNode) _fetch_joined_records: phase 1 of a run, get_cached_results, as_table, iter_data, get_all_records, as_source (DerivedSource), loaded jobs (also a job file saved by v0.2) label in __pipeline_base_entry_id TestFunctionNodeReads (5 tests, sync + async run), TestRun
Function result table (ResultCache) joined through the pipeline rows kept above; ResultCache.get_all_records (get_all_cached_outputs of CachedFunctionPod / CachedDataFunction), which now takes the current label label in __input_data_hash TestResultCacheReads
Operator-node table (OperatorJobNode, LOG / REPLAY) _read_stored_records: get_all_records, REPLAY as_table / iter_data / run, as_source, loaded jobs starfix format at the start of the _record_hash record id (000001 / 000002), then the node-content-hash filter TestOperatorNodeReads (4 tests)
Cached-source table (CachedSource) _read_cached_records: as_table, iter_data, get_all_records starfix format of _record_hash TestCachedSource
Side-effect invocation log (SideEffectJobNode) none: read by key only (get_record_by_id) not filtered: fails, see above TestSideEffects (sync + async, in-memory + Delta: nothing delivered, log and Delta files untouched)
StatusObserver / LoggingObserver tables read in full not filtered: run history, no Arrow-keyed rows n/a

Rows without an Orcapod label (operator-node and cached-source tables). Their record hashes are hex digests; only starfix's format number at the start names the generation. Measured: the node-content-hash filter does not separate generations (a source with an explicit source_id has the same node content hash under both labels), so without a filter those tables returned both generations. The format filter is exact for arrow_v0.1 (format 1) vs arrow_v0.2 (format 2), and nothing of the old generation is returned now. Limit: a later Arrow label that keeps starfix format 2 could not be told apart in these two kinds of table without storing a label there (a stored-schema change, not done here; a global hash-version identifier per row is being designed, not decided). Values that are not a 70-hex starfix digest, and labelled keys that carry no arrow_v... label (hand-made ids), are kept.

Consequences (accepted by the developer: "old values looking absent without explicit migration is something we have to accept")

  • The first v0.3 run on a v0.2 store computes every input again. The ITL-888 claim "an array-free pod reuses its v0.2 results" is false and the CHANGELOG now says so.
  • A node read on a v0.2 store before any v0.3 run returns no rows (with the warning above); a job file saved by v0.2 and loaded by v0.3 shows no rows until v0.3 has run.

Tests: tests/test_pipeline/test_stored_hash_labels.py (v0.2 rows ignored and inputs recomputed; old rows still present row for row, Delta files byte-identical; a second v0.3 run reuses v0.3 rows; every reader above; stored_hash_labels on pipeline, result, operator and log tables; the warning, once, with the total; the side-effect failure), tests/test_hashing/test_stored_hash_labels_helpers.py, and the real Orcapod v0.2.0 fixture in tests/test_arrays/test_array_v02_store.py (rewritten again: every input recomputed, 3 rows per node for 3 inputs, every v0.2 file byte-identical, every v0.2 table still lists 3 arrow_v0.1 rows). tests/test_pipeline/test_store_read_failure.py is back to main's version. Before the filters, 39 of these tests failed (both generations returned).

Found, not fixed (not specific to hash labels): a DeltaTableDatabase object keeps the table version it read first. A job object that read a table before a run does not see the rows the run added: on a v0.3-only Delta store holding 2 rows, a run with 7 inputs followed by job.nodes["dbl"].get_all_records() on the same job returns 2 rows (a newly built job returns 7). Any incremental run on an existing Delta store can show it; a run on a v0.2 store is one such case, so the fixture test reads through a newly built job.

Golden hashes (explicitly approved regeneration, decision 3)

Regenerated once with generate_golden_hashes.py. 184 values changed, 21 added, none removed; every changed value is Arrow-derived: an arrow_v0.x-labelled hash (table/schema hashes, per-row data hashes, input-data hashes, stored record ids), a default source_id (an Arrow table hash) and the record ids / system-tag values built from it, or a source content hash that embeds the table hash. No pipeline hash, schema hash, system-tag column name or table path moved; all semantic_v0.1-only entries are unchanged except value.dataframe.basic (a pandas DataFrame, hashed through StarfixArrowHasher).

Representative values (main → branch)

File Key main branch
arrow table.bool arrow_v0.1:0000012fafed9df9923… arrow_v0.2:000002dee7bb2ff6dce…
arrow table.nested.list_list arrow_v0.1:000001639751855e661… arrow_v0.2:000002d0d198475a44d…
arrow schema.width.int64 arrow_v0.1:000001f31bd23e3b248… arrow_v0.2:000002756a354ce6028…
arrow extension_table.extension.path arrow_v0.1:000001e32d9b28a94c4… arrow_v0.2:000002f558ddf91c1c5…
semantic value.dataframe.basic arrow_v0.1:000001b0e17dcec7cd7… arrow_v0.2:000002bac72e84776fc…
schema source_stream_builder.default_source_id.source_id 000001630ee60cef9dfe3cce7a6782… 000002e12eafa1dc72203ca9da1ca5…
schema source_stream_builder.default_source_id.content_hash semantic_v0.1:7eafdaffcb2ac49d… semantic_v0.1:167c1d63d67f2fa1…
datagram arrow_source.int32.row[{"k":"r1"}].data_hash arrow_v0.1:000001a01cd0f32e44a… arrow_v0.2:000002d92f4d29531f4…
pipeline batch.row[{"subject":["s1","s2"]}].data_hash arrow_v0.1:0000010819e767e277e… arrow_v0.2:0000020b638d3ccc763…
pipeline job.table[golden_job/golden_adjust/…] (stored record ids) ['arrow_v0.1:00000137cf61ccacb… ['arrow_v0.2:000002118147ab4ef…

The 21 added entries pin the newly hashable inputs: fixed-size lists, the ITL-848 collision pairs (now different), a chunked Boolean column (equal to the unchunked one), arrow.uuid / arrow.fixed_shape_tensor, and hash_schema on extension columns.

Every moved value

File Entries (main → branch) Changed Unchanged Added Removed
arrow.json 53 → 74 53 0 21 0
datagram.json 171 → 171 40 131 0 0
pipeline.json 305 → 305 68 237 0 0
schema.json 92 → 92 22 70 0 0
semantic.json 64 → 64 1 63 0 0
arrow.json: 53 changed, 21 added
Key main branch
extension_table.extension.path arrow_v0.1:000001e32d9b28a94c4f47e… arrow_v0.2:000002f558ddf91c1c5f706…
extension_table.extension.unregistered_passthrough arrow_v0.1:00000197e4832ff6efd7436… arrow_v0.2:000002b9c5f64eb011d128c…
extension_table.extension.uuid arrow_v0.1:0000016efd6e3b9a4c03e2c… arrow_v0.2:000002062ec7b363fa63cfb…
schema.bool arrow_v0.1:000001e1a1ab1078fd92ded… arrow_v0.2:00000292f4b85e8651fe4c7…
schema.empty.zero_rows arrow_v0.1:000001f31bd23e3b2489f68… arrow_v0.2:000002756a354ce60282443…
schema.float.float32 arrow_v0.1:000001fcd0edfcd85a395e7… arrow_v0.2:0000026a27b2c2276546fc4…
schema.float.float64 arrow_v0.1:000001f68c38ed991e7d525… arrow_v0.2:000002a8bb0c99c2d60fd20…
schema.layout.binary arrow_v0.1:0000011b8b0793ca3db4779… arrow_v0.2:000002346ed5761cafbd7a6…
schema.layout.large_binary arrow_v0.1:0000011b8b0793ca3db4779… arrow_v0.2:000002346ed5761cafbd7a6…
schema.layout.large_list arrow_v0.1:000001cec41debef42a67a5… arrow_v0.2:000002409130499175385ea…
schema.layout.large_string arrow_v0.1:00000191f7f5c26abcceb89… arrow_v0.2:000002d9023abe34e3bde2c…
schema.layout.list arrow_v0.1:000001cec41debef42a67a5… arrow_v0.2:000002409130499175385ea…
schema.layout.string arrow_v0.1:00000191f7f5c26abcceb89… arrow_v0.2:000002d9023abe34e3bde2c…
schema.nested.list_list arrow_v0.1:0000014387f6de75a16b23c… arrow_v0.2:0000029753856b34b9c54c1…
schema.nested.list_struct arrow_v0.1:000001a257fe950b239e9d3… arrow_v0.2:000002cd3954fa9d6958fc1…
schema.nested.struct arrow_v0.1:000001402dbf363e9e7ebaf… arrow_v0.2:0000021c6ab7531323b6ed5…
schema.nullability.non_nullable arrow_v0.1:0000013487bfb8486c6973d… arrow_v0.2:000002389fdc8277c962a83…
schema.nullability.nullable arrow_v0.1:000001f31bd23e3b2489f68… arrow_v0.2:000002756a354ce60282443…
schema.nulls.int64 arrow_v0.1:000001f31bd23e3b2489f68… arrow_v0.2:000002756a354ce60282443…
schema.nulls.string arrow_v0.1:00000191f7f5c26abcceb89… arrow_v0.2:000002d9023abe34e3bde2c…
schema.order.ab arrow_v0.1:00000100ad4d558238e9562… arrow_v0.2:000002b8f677fa5b62affbb…
schema.order.ba arrow_v0.1:00000100ad4d558238e9562… arrow_v0.2:000002b8f677fa5b62affbb…
schema.order.rows_reversed arrow_v0.1:00000100ad4d558238e9562… arrow_v0.2:000002b8f677fa5b62affbb…
schema.width.int16 arrow_v0.1:00000178c6959f782c8a48a… arrow_v0.2:000002c9ea066d6e1b441ad…
schema.width.int32 arrow_v0.1:0000011844a51e73c25585a… arrow_v0.2:000002447a113da124fde87…
schema.width.int64 arrow_v0.1:000001f31bd23e3b2489f68… arrow_v0.2:000002756a354ce60282443…
schema.width.int8 arrow_v0.1:000001d22fd4d64d8554ce4… arrow_v0.2:00000233ec0dccf2b00dbbd…
schema.width.uint8 arrow_v0.1:000001c63af839f862f0328… arrow_v0.2:00000252e16232ff906ecf1…
table.bool arrow_v0.1:0000012fafed9df99235be9… arrow_v0.2:000002dee7bb2ff6dce8ddc…
table.empty.zero_rows arrow_v0.1:000001d93c4003276f674c8… arrow_v0.2:000002d655beec05f54e13a…
table.float.float32 arrow_v0.1:00000177212619946903d4b… arrow_v0.2:00000287ee09555fc05dfba…
table.float.float64 arrow_v0.1:000001c1d762b8d385549ea… arrow_v0.2:000002d83df46319a27052e…
table.layout.binary arrow_v0.1:000001b19bcc9f2f94795bf… arrow_v0.2:0000021977fb0b4aac38946…
table.layout.large_binary arrow_v0.1:000001b19bcc9f2f94795bf… arrow_v0.2:0000021977fb0b4aac38946…
table.layout.large_list arrow_v0.1:00000189f2c4331bbc82425… arrow_v0.2:0000027db606bdc3327cd8b…
table.layout.large_string arrow_v0.1:000001590b6b8a65a493487… arrow_v0.2:0000021aedf65c94c24afaa…
table.layout.list arrow_v0.1:00000189f2c4331bbc82425… arrow_v0.2:0000027db606bdc3327cd8b…
table.layout.string arrow_v0.1:000001590b6b8a65a493487… arrow_v0.2:0000021aedf65c94c24afaa…
table.nested.list_list arrow_v0.1:000001639751855e6618fdf… arrow_v0.2:000002d0d198475a44dc9ab…
table.nested.list_struct arrow_v0.1:0000013c0c348a94fa935da… arrow_v0.2:0000029919cc15883d26c03…
table.nested.struct arrow_v0.1:0000012af9c2ad25c501bbf… arrow_v0.2:0000022efc170a4f2578623…
table.nullability.non_nullable arrow_v0.1:000001a55bac289fa4cb83a… arrow_v0.2:000002b37da33f664f9dc4a…
table.nullability.nullable arrow_v0.1:000001d469c2c16d7e2c80b… arrow_v0.2:00000236be7e40075274bab…
table.nulls.int64 arrow_v0.1:000001604458bd26489ed61… arrow_v0.2:0000023ba22c1f5257d8cf5…
table.nulls.string arrow_v0.1:00000189124cc15f2fbfc1d… arrow_v0.2:0000026885780c872492587…
table.order.ab arrow_v0.1:000001fb52c5ca8baed03ac… arrow_v0.2:000002d29b71de1ff0980cd…
table.order.ba arrow_v0.1:000001fb52c5ca8baed03ac… arrow_v0.2:000002d29b71de1ff0980cd…
table.order.rows_reversed arrow_v0.1:00000151aa84ec1e72539ee… arrow_v0.2:000002edcc170ff123dccde…
table.width.int16 arrow_v0.1:0000013e000f3e0b4beeba1… arrow_v0.2:000002c53e3f002b007a80a…
table.width.int32 arrow_v0.1:000001d831e96457f732f78… arrow_v0.2:000002ee36b64c06c9157f5…
table.width.int64 arrow_v0.1:000001d469c2c16d7e2c80b… arrow_v0.2:00000236be7e40075274bab…
table.width.int8 arrow_v0.1:00000160707f6988db37faf… arrow_v0.2:000002f195acdfc1ca1214d…
table.width.uint8 arrow_v0.1:000001dded35c6cee950b7a… arrow_v0.2:00000254a73782eb935f014…
extension_schema.extension.canonical_fixed_shape_tensor (added) arrow_v0.2:0000022df2fb43d6457645d…
extension_schema.extension.canonical_uuid (added) arrow_v0.2:00000236be584e0cd3a2668…
extension_schema.extension.path (added) arrow_v0.2:0000024887bbb622b764650…
extension_schema.extension.unregistered_passthrough (added) arrow_v0.2:000002853ab951d35ce3198…
extension_schema.extension.uuid (added) arrow_v0.2:000002bfbbdea0a2cf9f094…
extension_table.extension.canonical_fixed_shape_tensor (added) arrow_v0.2:0000029cdbbb5f9c731c669…
extension_table.extension.canonical_uuid (added) arrow_v0.2:0000024cf6272f99fa3465a…
schema.bool.chunked (added) arrow_v0.2:00000292f4b85e8651fe4c7…
schema.collision.list_lengths_a (added) arrow_v0.2:00000217b4f541f59698108…
schema.collision.list_lengths_b (added) arrow_v0.2:00000217b4f541f59698108…
schema.collision.null_struct (added) arrow_v0.2:000002377ba0fc7c623d903…
schema.collision.struct_of_nulls (added) arrow_v0.2:000002377ba0fc7c623d903…
schema.fixed_size_list.float32_2 (added) arrow_v0.2:00000256387ebbc501cb6fc…
schema.fixed_size_list.null_slot (added) arrow_v0.2:000002a91c8dfd080d7012a…
table.bool.chunked (added) arrow_v0.2:000002dee7bb2ff6dce8ddc…
table.collision.list_lengths_a (added) arrow_v0.2:000002295a4cad9a00a66f4…
table.collision.list_lengths_b (added) arrow_v0.2:0000026806409bd5df9d075…
table.collision.null_struct (added) arrow_v0.2:00000238e6e122820c9902b…
table.collision.struct_of_nulls (added) arrow_v0.2:0000024ef1bfba9ecf0d650…
table.fixed_size_list.float32_2 (added) arrow_v0.2:000002a963ce04c642d126d…
table.fixed_size_list.null_slot (added) arrow_v0.2:00000223564917b29cabbfd…
datagram.json: 40 changed, 0 added
Key main branch
arrow_source.int32.row[{"k":"r1"}].data_hash arrow_v0.1:000001a01cd0f32e44a4839… arrow_v0.2:000002d92f4d29531f41258…
arrow_source.int32.row[{"k":"r2"}].data_hash arrow_v0.1:000001cbdc2fbc671d4157f… arrow_v0.2:00000243c87e6d05d3588ed…
arrow_source.int32.row[{"k":"r3"}].data_hash arrow_v0.1:0000012f75fdca845f528ca… arrow_v0.2:000002825e1c06b5514e562…
arrow_source.int64.row[{"k":"r1"}].data_hash arrow_v0.1:000001c028af694082af2f2… arrow_v0.2:000002784ccdd2b780948d2…
arrow_source.int64.row[{"k":"r2"}].data_hash arrow_v0.1:0000017e332b258dbc5fe1f… arrow_v0.2:0000021d3e14bd65d5ca263…
arrow_source.int64.row[{"k":"r3"}].data_hash arrow_v0.1:0000015c299c665cda513e3… arrow_v0.2:0000023846c7aef08fc1400…
arrow_source.int64_nullable.row[{"k":"r1"}].data_hash arrow_v0.1:0000012081a40ec7eba7115… arrow_v0.2:000002521ff3d4701e70a5a…
arrow_source.int64_nullable.row[{"k":"r2"}].data_hash arrow_v0.1:000001ebc7fce25941def04… arrow_v0.2:000002601fc387de4781312…
arrow_source.int64_nullable.row[{"k":"r3"}].data_hash arrow_v0.1:000001951dc05c8c79ecd4a… arrow_v0.2:000002565243c66b8ae12de…
arrow_source.int8.row[{"k":"r1"}].data_hash arrow_v0.1:000001760d63148f6020615… arrow_v0.2:000002b17bad7455bf81d47…
arrow_source.int8.row[{"k":"r2"}].data_hash arrow_v0.1:0000012622e7bf473c8c58d… arrow_v0.2:000002ff7e5abaafd6df855…
arrow_source.int8.row[{"k":"r3"}].data_hash arrow_v0.1:000001c502c37a181f7c3e1… arrow_v0.2:000002fa70cfa37eaab0c4e…
arrow_source.large_string.row[{"k":"r1"}].data_hash arrow_v0.1:000001b4407c40d91327434… arrow_v0.2:0000027dc69926b790513bb…
arrow_source.large_string.row[{"k":"r2"}].data_hash arrow_v0.1:000001cd356426fd64cb498… arrow_v0.2:000002e2668873da695349e…
arrow_source.large_string.row[{"k":"r3"}].data_hash arrow_v0.1:000001e0c4e88cf1c63c94e… arrow_v0.2:000002b08595f56b1cd4a87…
arrow_source.string.row[{"k":"r1"}].data_hash arrow_v0.1:000001b4407c40d91327434… arrow_v0.2:0000027dc69926b790513bb…
arrow_source.string.row[{"k":"r2"}].data_hash arrow_v0.1:000001cd356426fd64cb498… arrow_v0.2:000002e2668873da695349e…
arrow_source.string.row[{"k":"r3"}].data_hash arrow_v0.1:000001e0c4e88cf1c63c94e… arrow_v0.2:000002b08595f56b1cd4a87…
dataclass.row[{"k":"r1"}].data_hash arrow_v0.1:0000015e306dd402af59e45… arrow_v0.2:0000027b15d8db313a05d7a…
dataclass.row[{"k":"r2"}].data_hash arrow_v0.1:000001b088edbc3232e6d83… arrow_v0.2:000002abd8873e34ca7ee3f…
dataframe.row[{"k":"r1"}].data_hash arrow_v0.1:00000108d551603b5a5df12… arrow_v0.2:00000229c5394be73a9c867…
dataframe.row[{"k":"r2"}].data_hash arrow_v0.1:00000134950bb185ada91e9… arrow_v0.2:0000024dc4bbb7a657d7ba9…
list_path.row[{"k":"r1"}].data_hash arrow_v0.1:00000135fc611ef9ad245d4… arrow_v0.2:000002cc2ece0ae8ac64848…
list_path.row[{"k":"r2"}].data_hash arrow_v0.1:0000018efe66b052bc46996… arrow_v0.2:000002cdaddd554ca855740…
ndarray.row[{"k":"r1"}].data_hash arrow_v0.1:000001c80cacf9821f15fd1… arrow_v0.2:000002e79fea6f79a5a7e61…
ndarray.row[{"k":"r2"}].data_hash arrow_v0.1:000001ba0bc856108ff6198… arrow_v0.2:000002922f3a9172b4d7363…
nested.row[{"k":"r1"}].data_hash arrow_v0.1:000001c31188a44c589ce3d… arrow_v0.2:00000261f569412bb12da24…
nested.row[{"k":"r2"}].data_hash arrow_v0.1:000001a7658c283bcd7a1c6… arrow_v0.2:000002f4312647b9ca61759…
optional.row[{"k":"r1"}].data_hash arrow_v0.1:000001f910e5b3279111cf4… arrow_v0.2:000002f774b1bc717e2ddf7…
optional.row[{"k":"r2"}].data_hash arrow_v0.1:0000010d55a89a30404439b… arrow_v0.2:0000027714fb655c044c9c1…
path.row[{"k":"r1"}].data_hash arrow_v0.1:00000135bf11f7d6830036c… arrow_v0.2:000002a230080e1f03f2254…
path.row[{"k":"r2"}].data_hash arrow_v0.1:0000015b4f8156dc62851f9… arrow_v0.2:00000234b6166fa805040e4…
pydantic.row[{"k":"r1"}].data_hash arrow_v0.1:0000012280b233c56771fa4… arrow_v0.2:000002459c1d7ed12f91d7f…
pydantic.row[{"k":"r2"}].data_hash arrow_v0.1:00000138d77ff36d7b2f3c3… arrow_v0.2:000002ce57007b0730b9021…
scalars.row[{"k":"r1"}].data_hash arrow_v0.1:00000162a09a0c2dd7d8e7c… arrow_v0.2:000002c8161def16592dc28…
scalars.row[{"k":"r2"}].data_hash arrow_v0.1:000001d49094978c1867f74… arrow_v0.2:0000020528b8069a7f49fd5…
set_uuid.row[{"k":"r1"}].data_hash arrow_v0.1:0000011eb69965d474d2209… arrow_v0.2:000002648eddf1e15a4ed2e…
set_uuid.row[{"k":"r2"}].data_hash arrow_v0.1:000001fc38bc3b27de6a066… arrow_v0.2:000002ec023f0196ddaaa81…
uuid.row[{"k":"r1"}].data_hash arrow_v0.1:000001e05de561e03537d1d… arrow_v0.2:00000228d36dcdde1b6e7ff…
uuid.row[{"k":"r2"}].data_hash arrow_v0.1:0000014c189ef879ddf91cf… arrow_v0.2:00000236a95b5f9e34143f9…
pipeline.json: 68 changed, 0 added
Key main branch
batch.row[{"subject":["s1","s2"]}].data_hash arrow_v0.1:0000010819e767e277e136c… arrow_v0.2:0000020b638d3ccc763f2a8…
batch.row[{"subject":["s3"]}].data_hash arrow_v0.1:0000014957b9ee25e0ea51c… arrow_v0.2:000002928bbf5d31fadf74e…
column_selection.drop_data.row[{"subject":"s1"}].data_hash arrow_v0.1:000001ee27bd12d66af74db… arrow_v0.2:000002c734791b7452bb2af…
column_selection.drop_data.row[{"subject":"s2"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
column_selection.drop_data.row[{"subject":"s3"}].data_hash arrow_v0.1:000001966c5e58c797ab332… arrow_v0.2:0000025f33a2f5bf3e956ed…
column_selection.drop_tag.row[{"cohort":"a"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
column_selection.drop_tag.row[{"cohort":"b"}].data_hash arrow_v0.1:000001966c5e58c797ab332… arrow_v0.2:0000025f33a2f5bf3e956ed…
column_selection.select_data.row[{"subject":"s1"}].data_hash arrow_v0.1:0000017cb817a7c2cfd6c71… arrow_v0.2:00000281f4d4a4e7742a2ca…
column_selection.select_data.row[{"subject":"s2"}].data_hash arrow_v0.1:00000124d0ecb4bf9d093b6… arrow_v0.2:0000022420dc43cfa6c4d36…
column_selection.select_data.row[{"subject":"s3"}].data_hash arrow_v0.1:000001810e6ae31eebcd4f2… arrow_v0.2:000002805d579510905b591…
column_selection.select_tag.row[{"cohort":"a"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
column_selection.select_tag.row[{"cohort":"b"}].data_hash arrow_v0.1:000001966c5e58c797ab332… arrow_v0.2:0000025f33a2f5bf3e956ed…
group_by.row[{"cohort":"a"}].data_hash arrow_v0.1:0000018910e185c56ec7890… arrow_v0.2:0000022b0f148a444e75dad…
group_by.row[{"cohort":"b"}].data_hash arrow_v0.1:000001ba6adaea3c56bf0be… arrow_v0.2:0000028b04ecd69f72beab1…
index_pick.index.row[{"subject":"s1"}].data_hash arrow_v0.1:00000189f1942fbc40737ff… arrow_v0.2:0000025f5739a1fc427e583…
index_pick.index.row[{"subject":"s2"}].data_hash arrow_v0.1:00000164b528c22b7183231… arrow_v0.2:000002d9f79b11b0d692847…
index_pick.pick.row[{"subject":"s1"}].data_hash arrow_v0.1:000001380f07142b3132f93… arrow_v0.2:000002d3aa3df31c715ed95…
index_pick.pick.row[{"subject":"s2"}].data_hash arrow_v0.1:00000157b70714f9f88f916… arrow_v0.2:00000218fa0f5bdf561a92c…
job.node[adjust].row[{"subject":"s1"}].data_hash arrow_v0.1:0000014a3871996b0dc2563… arrow_v0.2:000002df3a95f3b9eddaf07…
job.node[adjust].row[{"subject":"s2"}].data_hash arrow_v0.1:0000014c738e65c53a408df… arrow_v0.2:0000023565098d1a88ecd75…
job.node[joined].row[{"subject":"s1"}].data_hash arrow_v0.1:00000130ab270ba209296b0… arrow_v0.2:000002c945ad08138f51298…
job.node[joined].row[{"subject":"s2"}].data_hash arrow_v0.1:00000186afa42ffc71c4517… arrow_v0.2:0000026b8369e60be10e43d…
job.node[scans].row[{"subject":"s1"}].data_hash arrow_v0.1:000001ac375e92d49c253f9… arrow_v0.2:000002ef2b37a1979bd7348…
job.node[scans].row[{"subject":"s2"}].data_hash arrow_v0.1:000001f7fa6c7bb04912cca… arrow_v0.2:00000210b14e22145ce1d5f…
job.node[subjects].row[{"subject":"s1"}].data_hash arrow_v0.1:00000137de028aa69769a5d… arrow_v0.2:000002d1c5b49de38ef6efc…
job.node[subjects].row[{"subject":"s2"}].data_hash arrow_v0.1:000001584cc7c0907761513… arrow_v0.2:000002338ce8710f5ecb156…
job.node[subjects].row[{"subject":"s3"}].data_hash arrow_v0.1:000001562e4c70b60df10ed… arrow_v0.2:0000023401f1ce40c8cfdd6…
job.table[golden_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes ['arrow_v0.1:00000130ab270ba209296… ['arrow_v0.2:0000026b8369e60be10e4…
job.table[golden_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:b61a42486c29af296f5c2f1d8129f20b61e023d492747e397a50756cb153aff8/pdb_v1].record_ids ['arrow_v0.1:00000137cf61ccacb0c63… ['arrow_v0.2:000002118147ab4ef2d7e…
join_pod.join.row[{"subject":"s1"}].data_hash arrow_v0.1:00000130ab270ba209296b0… arrow_v0.2:000002c945ad08138f51298…
join_pod.join.row[{"subject":"s2"}].data_hash arrow_v0.1:00000186afa42ffc71c4517… arrow_v0.2:0000026b8369e60be10e43d…
join_pod.pod.row[{"subject":"s1"}].data_hash arrow_v0.1:0000014a3871996b0dc2563… arrow_v0.2:000002df3a95f3b9eddaf07…
join_pod.pod.row[{"subject":"s2"}].data_hash arrow_v0.1:0000014c738e65c53a408df… arrow_v0.2:0000023565098d1a88ecd75…
map_data.row[{"subject":"s1"}].data_hash arrow_v0.1:0000012b54015317016ea37… arrow_v0.2:0000022ad1363967c26697b…
map_data.row[{"subject":"s2"}].data_hash arrow_v0.1:000001bf869237358aa3c91… arrow_v0.2:000002a61fcaac02c3fd261…
map_data.row[{"subject":"s3"}].data_hash arrow_v0.1:000001e87c392dee12edd21… arrow_v0.2:000002ef21c37efbb54c93f…
merge_join.row[{"subject":"s1"}].data_hash arrow_v0.1:000001ec09b4a15ec5e06d5… arrow_v0.2:000002654a49c9ec7661bbb…
merge_join.row[{"subject":"s3"}].data_hash arrow_v0.1:0000013aeb0e14ea94b020f… arrow_v0.2:000002b78038f0da00d8b6e…
pod.row[{"subject":"s1"}].data_hash arrow_v0.1:00000155d85a85806e3601d… arrow_v0.2:0000024da91c65d3ebacee0…
pod.row[{"subject":"s2"}].data_hash arrow_v0.1:0000017db75343362e61ae6… arrow_v0.2:0000021afa3e9774d8a2852…
pod.row[{"subject":"s3"}].data_hash arrow_v0.1:000001a3b408003eb5481ee… arrow_v0.2:000002c2a1324b1334c4c49…
polars_filter.row[{"cohort":"a","subject":"s1"}].data_hash arrow_v0.1:000001ee27bd12d66af74db… arrow_v0.2:000002c734791b7452bb2af…
polars_filter.row[{"cohort":"a","subject":"s2"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
polars_filter_expr.expr.row[{"cohort":"a","subject":"s1"}].data_hash arrow_v0.1:000001ee27bd12d66af74db… arrow_v0.2:000002c734791b7452bb2af…
polars_filter_expr.expr.row[{"cohort":"a","subject":"s2"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
polars_filter_expr.mixed.row[{"cohort":"a","subject":"s1"}].data_hash arrow_v0.1:000001ee27bd12d66af74db… arrow_v0.2:000002c734791b7452bb2af…
polars_filter_expr.mixed.row[{"cohort":"a","subject":"s2"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
polars_filter_expr_job.node[adjust].row[{"subject":"s2"}].data_hash arrow_v0.1:0000014c738e65c53a408df… arrow_v0.2:0000023565098d1a88ecd75…
polars_filter_expr_job.node[joined].row[{"subject":"s2"}].data_hash arrow_v0.1:00000186afa42ffc71c4517… arrow_v0.2:0000026b8369e60be10e43d…
polars_filter_expr_job.node[older].row[{"subject":"s2"}].data_hash arrow_v0.1:000001584cc7c0907761513… arrow_v0.2:000002338ce8710f5ecb156…
polars_filter_expr_job.node[older].row[{"subject":"s3"}].data_hash arrow_v0.1:000001562e4c70b60df10ed… arrow_v0.2:0000023401f1ce40c8cfdd6…
polars_filter_expr_job.node[scans].row[{"subject":"s1"}].data_hash arrow_v0.1:000001ac375e92d49c253f9… arrow_v0.2:000002ef2b37a1979bd7348…
polars_filter_expr_job.node[scans].row[{"subject":"s2"}].data_hash arrow_v0.1:000001f7fa6c7bb04912cca… arrow_v0.2:00000210b14e22145ce1d5f…
polars_filter_expr_job.node[subjects].row[{"subject":"s1"}].data_hash arrow_v0.1:00000137de028aa69769a5d… arrow_v0.2:000002d1c5b49de38ef6efc…
polars_filter_expr_job.node[subjects].row[{"subject":"s2"}].data_hash arrow_v0.1:000001584cc7c0907761513… arrow_v0.2:000002338ce8710f5ecb156…
polars_filter_expr_job.node[subjects].row[{"subject":"s3"}].data_hash arrow_v0.1:000001562e4c70b60df10ed… arrow_v0.2:0000023401f1ce40c8cfdd6…
polars_filter_expr_job.table[golden_filter_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes ['arrow_v0.1:00000186afa42ffc71c45… ['arrow_v0.2:0000026b8369e60be10e4…
polars_filter_expr_job.table[golden_filter_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:1bf083d50625083700e69f699b90f9f01c9c1e93d56e1a56feb4ad1506005e93/pdb_v1].record_ids ['arrow_v0.1:000001ea2218fa4e75231… ['arrow_v0.2:0000022ad2e7cf689d570…
semi_join.row[{"subject":"s1"}].data_hash arrow_v0.1:00000137de028aa69769a5d… arrow_v0.2:000002d1c5b49de38ef6efc…
semi_join.row[{"subject":"s2"}].data_hash arrow_v0.1:000001584cc7c0907761513… arrow_v0.2:000002338ce8710f5ecb156…
sources.cohorts.row[{"cohort":"a","subject":"s1"}].data_hash arrow_v0.1:000001ee27bd12d66af74db… arrow_v0.2:000002c734791b7452bb2af…
sources.cohorts.row[{"cohort":"a","subject":"s2"}].data_hash arrow_v0.1:00000133552dc58a31ff558… arrow_v0.2:00000252e4f32a133cd6a88…
sources.cohorts.row[{"cohort":"b","subject":"s3"}].data_hash arrow_v0.1:000001966c5e58c797ab332… arrow_v0.2:0000025f33a2f5bf3e956ed…
sources.scans.row[{"subject":"s1"}].data_hash arrow_v0.1:000001ac375e92d49c253f9… arrow_v0.2:000002ef2b37a1979bd7348…
sources.scans.row[{"subject":"s2"}].data_hash arrow_v0.1:000001f7fa6c7bb04912cca… arrow_v0.2:00000210b14e22145ce1d5f…
sources.subjects.row[{"subject":"s1"}].data_hash arrow_v0.1:00000137de028aa69769a5d… arrow_v0.2:000002d1c5b49de38ef6efc…
sources.subjects.row[{"subject":"s2"}].data_hash arrow_v0.1:000001584cc7c0907761513… arrow_v0.2:000002338ce8710f5ecb156…
sources.subjects.row[{"subject":"s3"}].data_hash arrow_v0.1:000001562e4c70b60df10ed… arrow_v0.2:0000023401f1ce40c8cfdd6…
schema.json: 22 changed, 0 added
Key main branch
source_stream_builder.default_source_id.content_hash semantic_v0.1:7eafdaffcb2ac49d7607… semantic_v0.1:167c1d63d67f2fa19980…
source_stream_builder.default_source_id.row[{"subject":"s1"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 71c09013eb5f5f5582ad41da30acbcf7 2d25e22e5ed45f2ebcd9ba0e574ce4ab
source_stream_builder.default_source_id.row[{"subject":"s1"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 000001630ee60cef9dfe3cce7a6782fa30… 000002e12eafa1dc72203ca9da1ca5d064…
source_stream_builder.default_source_id.row[{"subject":"s1"}].data_hash arrow_v0.1:00000128b75b53ced7f9752… arrow_v0.2:00000289f61a7619845a37a…
source_stream_builder.default_source_id.row[{"subject":"s2"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 e2960b661cf05e14874582345bdde8e8 7a85d0e335bf539d990068a86fb4409e
source_stream_builder.default_source_id.row[{"subject":"s2"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 000001630ee60cef9dfe3cce7a6782fa30… 000002e12eafa1dc72203ca9da1ca5d064…
source_stream_builder.default_source_id.row[{"subject":"s2"}].data_hash arrow_v0.1:000001a39a4d7b517d56c4c… arrow_v0.2:0000029f0b00a74de3d3f45…
source_stream_builder.default_source_id.row[{"subject":"s3"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 5d4deb7fe5675941942e2b590adaf240 174dcf5eb0c05b08967518af5d71cd7b
source_stream_builder.default_source_id.row[{"subject":"s3"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367 000001630ee60cef9dfe3cce7a6782fa30… 000002e12eafa1dc72203ca9da1ca5d064…
source_stream_builder.default_source_id.row[{"subject":"s3"}].data_hash arrow_v0.1:0000010364967ccec6827a7… arrow_v0.2:0000026b8c4f7e93665f79d…
source_stream_builder.default_source_id.source_id 000001630ee60cef9dfe3cce7a6782fa30… 000002e12eafa1dc72203ca9da1ca5d064…
source_stream_builder.default_source_id.table_hash arrow_v0.1:000001630ee60cef9dfe3cc… arrow_v0.2:000002e12eafa1dc72203ca…
source_stream_builder.named_source_id.content_hash semantic_v0.1:f424ce4643b9e4dd3757… semantic_v0.1:69b4b3098babd295b560…
source_stream_builder.named_source_id.row[{"subject":"s1"}].data_hash arrow_v0.1:00000128b75b53ced7f9752… arrow_v0.2:00000289f61a7619845a37a…
source_stream_builder.named_source_id.row[{"subject":"s2"}].data_hash arrow_v0.1:000001a39a4d7b517d56c4c… arrow_v0.2:0000029f0b00a74de3d3f45…
source_stream_builder.named_source_id.row[{"subject":"s3"}].data_hash arrow_v0.1:0000010364967ccec6827a7… arrow_v0.2:0000026b8c4f7e93665f79d…
source_stream_builder.named_source_id.table_hash arrow_v0.1:000001630ee60cef9dfe3cc… arrow_v0.2:000002e12eafa1dc72203ca…
source_stream_builder.record_id_column.content_hash semantic_v0.1:6983755521b8c4ca1771… semantic_v0.1:0bbf3af4ed09b723aa34…
source_stream_builder.record_id_column.row[{"subject":"s1"}].data_hash arrow_v0.1:00000128b75b53ced7f9752… arrow_v0.2:00000289f61a7619845a37a…
source_stream_builder.record_id_column.row[{"subject":"s2"}].data_hash arrow_v0.1:000001a39a4d7b517d56c4c… arrow_v0.2:0000029f0b00a74de3d3f45…
source_stream_builder.record_id_column.row[{"subject":"s3"}].data_hash arrow_v0.1:0000010364967ccec6827a7… arrow_v0.2:0000026b8c4f7e93665f79d…
source_stream_builder.record_id_column.table_hash arrow_v0.1:000001630ee60cef9dfe3cc… arrow_v0.2:000002e12eafa1dc72203ca…
semantic.json: 1 changed, 0 added
Key main branch
value.dataframe.basic arrow_v0.1:000001b0e17dcec7cd73f09… arrow_v0.2:000002bac72e84776fc6d62…

Other pinned values re-recorded in format 2 (outside the golden files): test_starfix_arrow_hasher.py (schema, nullability and table digests), test_materialize_source_info.py (operator record ids and node content hash; the pre-fix ids are re-derived with _broadcast_row_zero), test_async_sync_equivalence.py (ITL-837 row hashes; stream hashes unchanged), test_struct_layout_identity.py (stream content and row hashes; small and large layouts still equal), test_group_by.py (member order by record_id, which derives from the Arrow-hashed default source_id: ["a","b"] → ["b","a"]), and the worked example in docs/reference/hashing.md.

Golden files touched (conflicts with every open PR adding golden entries, e.g. #301, #300, #305, #306): tests/test_hashing/golden/data/{arrow,datagram,pipeline,schema,semantic}.json, tests/test_hashing/golden/cases.py, tests/test_hashing/golden/generate_golden_hashes.py. Rebasing such a PR onto this one means regenerating only its added entries.

Performance (starfix-python format 2 is pure Python)

Orcapod's StarfixArrowHasher.hash_table on 100,000-row tables (best of 3, Apple Silicon):

Table starfix 0.4 starfix 0.5
int64 + float64 + bool 50.7 ms 0.7 ms
large_string 44.2 ms 31.1 ms
large_list 4436.5 ms 500.3 ms
struct<int64, large_string> 540.1 ms 533.9 ms
1-row table (per call, ×2000) 36 µs 43 µs

Whole-table hashing is as fast or faster; single-row hashes (record ids, per-row data hashes) are about 20% slower. tests/test_hashing + tests/test_core/operators: 27.6 s on main, 27.8 s on this branch.

Test plan

  • New tests failed first: on starfix 0.4 (21 in test_starfix_format2.py + the ITL-846 test), and for the v0.2-store handling 39 tests with both generations returned before the filters were added; pass now.
  • Full suite on Python 3.11, 3.12, 3.13, one at a time: 7755 passed, 96 skipped, 18 xfailed, 0 failed on each. Skips are infrastructure only: SpiralDB integration (SPIRAL_INTEGRATION_TESTS), S3/MinIO (no container), Graphviz dot not on PATH, PostgreSQL opt-in.
  • The follow-up (v0.2-store handling) moved no golden value: tests/test_hashing/golden is identical to the previous head of this branch. origin/main had no commits the branch lacks, so no merge and no further golden regeneration were needed.
  • Golden tests pass on all three versions (values identical across interpreters and hash seeds).
  • ruff check, ruff format --check: clean. basedpyright: exit 0, 0 errors.
  • Docs samples and mkdocs build --strict (tests/test_docs) pass.
  • Not done: the tutorial notebook notebooks/tutorials/01_introduction_to_orcapod.ipynb still shows arrow_v0.1 values in two stored outputs (not executed by tests; re-running it would rewrite unrelated outputs).

Linear

Fixes ITL-910
ITL-848
ITL-927
ITL-846
ITL-938

🤖 Generated with Claude Code

Edgar Y. Walker and others added 15 commits November 11, 2026 15:31
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910)

Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash
pyarrow's canonical extension types and extension-typed schemas with their
extension identity instead of as their storage type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910)

Explicitly approved regeneration (developer decision 3, November 11, 2026):
184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline
68/305, schema 22/92, semantic 1/64); no key added or removed among them.
21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a
chunked Boolean column, canonical extension types and extension schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910)

Adds orcapod.errors.StoreHashLabelError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910)

A v0.3 node whose pipeline, result, operator, cached-source or side-effect
table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now
raises StoreHashLabelError on read and before its first write, and
PipelineJob.run checks every node before the first one runs, so a refused
store is left exactly as it was. No opt-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
ITL-910 Adopt starfix 0.5 (hash format 2) in Orcapod: dependency bump, label arrow_v0.2, golden regeneration

Decisions — November 11, 2026 (developer)

  • starfix hash format 2 is approved (ITL-898): a corrected, lossless layout with a new format number, so every starfix hash changes. Release 0.5.0 is planned for November 12, 2026.
  • A hash bug overrides hash stability: erroneous values move, and the change is marked explicitly as a fix. Moving every hash before the public release is acceptable.
  • Orcapod's label rule (accepted): bump Orcapod's Arrow hash label whenever the hash of any table can change, whatever the cause. For v0.3 the label moves from arrow_v0.1 to arrow_v0.2. The label's number is not kept equal to the starfix format number; the context file records which starfix format each label uses.

Overview

Orcapod hashes Arrow tables through StarfixArrowHasher, which preprocesses the table and calls the starfix package (0.4.x today, the pure-Python implementation). starfix 0.4 has collisions that reach Orcapod's row hashes (nested list lengths, nullable struct validity, Booleans; see ITL-848), and its hashes of Boolean columns depend on chunking. starfix 0.5 fixes these in a new format.

This issue moves Orcapod onto starfix 0.5.

Goals & Success Criteria

  • pyproject.toml and uv.lock require starfix 0.5.x.
  • The Arrow hasher label is arrow_v0.2 everywhere it is configured (hashing/versioned_hashers.py, the context data file and its schema), with a new context version or changelog entry that records "starfix format 2".
  • Every golden file is regenerated once, explicitly, as a format change. The PR states the counts (entries changed, entries unchanged such as semantic_v0.1 values that do not involve Arrow hashing) and confirms nothing changed for any other reason.
  • Tests that pinned the old collisions as known behaviour are replaced by tests that the collisions are gone (the nested-list row-hash case and the Boolean chunking case from ITL-848).
  • Orcapod inputs that starfix 0.5 now rejects in every mode are identified and handled with clear errors: duplicate or aliased column paths, a non-nullable field containing nulls, and data of null-typed, fixed-size-list, map, interval and view columns.
  • Migration notes for existing v0.2.0 stores: every cached result recomputes once; old rows remain; nothing is deleted. Stated in the changelog and the docs.
  • Hashing throughput before and after on a representative table.

Scope & Boundaries

In scope:

  • The dependency bump, the label, the golden regeneration, and the adjustments the new rejections require.

Out of scope:

  • Order as identity for Polars, pandas and pa.Table values (ITL-578, ITL-897), which build on this.
  • Replacing the pure-Python starfix with a Rust-backed wrapper (ITL-909).

Dependencies & Risks

  • Needs starfix 0.5.0 on PyPI.
  • Every open PR that touches golden files conflicts with the regeneration; merge order has to be planned.

Resources & References

ITL-848 Starfix Arrow hasher gaps: hash_schema crashes on extension types; null, uuid, map and fixed-size-list columns cannot be hashed

Overview

Several workers hit limits of StarfixArrowHasher while writing hash tests for v0.2. None blocks v0.2, but each leaves a type that Orcapod cannot hash or cannot test.

Reported on November 10, 2026 (PR metamorphic-labs/orcapod-python#278 and ITL-837 worker reports); each needs a minimal reproduction confirmed on main:

  • hash_schema crashes when the schema contains a live extension type. It is not called in production today, only in tests.
  • A null-typed column cannot be hashed.
  • Canonical PyArrow extension types such as arrow.uuid cannot be hashed. A MergeJoin over such a column fails while computing the pipeline hash, before the operator runs.
  • Map values and fixed-size-list values cannot be hashed, so tests compare those types by schema hash only.
  • ArrowHasherProtocol does not declare hash_schema, so tests have to narrow to the concrete StarfixArrowHasher.

Goals & Success Criteria

  • Each item above is confirmed with a minimal reproduction, then fixed or explicitly declared unsupported with a clear error that names the column and type.
  • hash_schema is on ArrowHasherProtocol if tests and operators are meant to use it.
  • Golden tests (ITL-626) cover every type the hasher supports.

Scope & Boundaries

In scope:

  • src/orcapod/hashing/arrow_hashers.py and the hashing protocol.

Out of scope:

  • Changes to the starfix library itself; if a gap is upstream, file it there and reference it here.

Dependencies & Risks

  • Any change to how a type is hashed changes stored hashes for data of that type. Each fix must state which hashes move.

Resources & References

ITL-846 Audit and test Arrow schema and data preservation across every operation, sync and async

Overview

Orcapod is type strict. Principle stated by the developer on November 10, 2026:

Except for very specific exceptions (the large variants of a type), Orcapod is type strict and does not perform implicit type expansion or conversion. When data types disagree, raise an error rather than silently widening.

During the v0.2 work, each operator that was looked at closely turned out to change something it should only pass through: Index, Pick and MergeJoin widened integers and floats; Join's async path re-inferred every type; PolarsFilter and MergeJoin changed the record-ID column from fixed_size_binary[16] to large_binary; normalize_to_large_types dropped list-item nullability; materializing rows stamped row 0's source info on every row; sync Join silently took one side's type for a key with disagreeing types. Each was found by hand, one at a time.

This issue is one thorough, test-backed check that every operation preserves the Arrow schema and the data it does not own, so the remaining cases are found systematically instead of by accident.

Goals & Success Criteria

  • A reusable preservation test harness: given an operation and an input stream covering the full type matrix, it asserts for every pass-through column and row:
    • Arrow type equals the large-normalized input type (width, signedness, precision, timezone, struct fields and logical/extension types unchanged),
    • nullability unchanged, top level and nested,
    • values unchanged,
    • source-info values and system-tag names and values follow the documented rules, per row,
    • column order follows the documented rule,
    • the hasher gives the same schema hash and value hashes as the input for pass-through columns.
  • The harness is applied to every operator (Join, MergeJoin, SemiJoin, Batch, GroupBy, MapTags, MapData, the four column-selection operators, PolarsFilter, Index, Pick), function pods, side-effect pods, sources, and function and operator nodes.
  • Every execution path is covered: static_process, per-row iter_data, as_table(), the async path, results read back from each database backend, and cached results on a re-run.
  • Sync and async paths give identical results for the same inputs.
  • Predicted output_schema() equals the materialized schema in every case.
  • Inputs whose types disagree (other than large variants) raise an error naming the column and both types, in every multi-input operator, sync and async. No operator promotes or widens.
  • Every deviation is fixed or filed as its own issue with a failing test.
  • The type-strict principle is written into orcapod-design.md, CLAUDE.md and .zed/rules.

Scope & Boundaries

In scope:

  • The type matrix: all integer and float widths, bool, string and binary in both layouts, fixed-size binary, date, time, timestamp with and without timezone, duration, decimal, list, large list, fixed-size list, struct, map, null, and every registered logical type.
  • Sub-issue ITL-842 (nullability audit) is the nullability part of this work.

Out of scope:

  • Designing where large-normalization happens (ITL-841).
  • Arrow types in Schema and how disagreement is detected at graph-build time (ITL-835).
  • Golden hash regression files (ITL-626), though the harness should feed them.

Dependencies & Risks

  • Each fix can change row content hashes for data with non-default types. Every fix must state which hashes move.

Resources & References

ITL-927 A fixed_size_list column crashes source construction: starfix cannot hash FixedSizeList data

Overview

Any source whose table has a fixed_size_list column fails at construction: starfix 0.4.0 raises NotImplementedError in hash_table. Found by the ITL-926 worker on November 11, 2026. It affects every source type, not only Spiral.

The workshop's sample project (edi-sample-v1-855632) has 13 fixed_size_list columns, for example monitor_gamma_table: fixed_size_list<float>[256], monitor_resolution_px: fixed_size_list<uint16>[2] and fix_spot_color: fixed_size_list<uint8>[3], in experiment_display_params, experiment_freeviewing_params, experiment_screen_capture_event, lfp_timeseries and others.

starfix 0.5.0 (hash format 2, about to be released) also rejects FixedSizeList data, so adopting 0.5.0 (ITL-910) does not fix this.

Goals & Success Criteria

  • Decide where FixedSizeList is handled: support in starfix (a format-2 addition: how the type and its values are hashed, and whether fixed_size_list<T>[n] is the same as or distinct from list<T>), or a rule in Orcapod.
  • Adding support for a type that is rejected today changes no existing hash; confirm that in the starfix change.
  • Golden values generated on the Rust side and matched by starfix-python.
  • A source with fixed_size_list columns constructs, hashes and runs through a pipeline, with the type preserved (no conversion to large_list).

Dependencies & Risks

  • Needed before the workshop on November 17, 2026 if attendees use the affected tables.

Resources & References

ITL-938 Design: one coherent versioning system for hashes, hash labels and stored paths

Overview

Developer, November 11, 2026: "I want a thorough discussion & analyses just about the path for the hash. I also want to have a thorough consideration of everywhere we have some hash versioning (e.g. we have name_v0.2 eg for some of the hash values). Rather than keeping all of these things scattered, I want one very coherent and robust system."

Orcapod records versions in many unrelated places, each added for its own reason:

  • Hash labels in front of digests: semantic_v0.1, arrow_v0.1 (becoming arrow_v0.2 with starfix 0.5.0, ITL-910), and per-type value labels such as ndarray_v1, polars_expr_v1, torch_tensor_v1, pandas_dataframe_v1, arrow_table_v1.
  • The starfix format prefix inside the digest (000002).
  • The data context key and version (std:v0.1:default, v0.1) and its changelog.
  • Stored table path components and suffixes (pdb_v1, rdb_v1, the function's major version, table_scope), and the stored-schema version checked by SchemaVersionError.
  • Representation versions inside stored data: the array format's {"version":1} extension metadata and the Spiral nullability record's version (ITL-936).

Stored table paths come from the pipeline hash, which does not change when a hash label changes, so results written under different hash generations land in the same tables. The developer decided on November 11, 2026 that v0.3 fails explicitly when it meets v0.2 rows, and then asked whether an explicit name should make every path change for v0.3.

This issue is a design discussion, held by the developer directly with a dedicated worker. It produces a design, not code.

Goals & Success Criteria

  • An inventory of every place a version, label or format marker appears in a hash, a stored value, a stored schema or a stored path, with what bumps it today and what depends on it.
  • One system that answers, for any change to hashing or representation: what gets a new name, where that name appears (digest label, path, stored metadata), what happens to data written under the old name (found, refused, migrated), and how an old generation is read.
  • A decision on whether stored paths carry a generation name, what the name denotes, where it sits, and what v0.3 does on meeting an older generation.
  • Fit with the related work: the logical type version design (ITL-935), migratability and orcapod-sanctuary (ITL-914, ITL-913), the starfix adoption (ITL-910), and the nullability record (ITL-936).
  • A list of what must land before the v0.3 tag (target November 16, 2026) and what can follow, with the hashes and paths that move.

Scope & Boundaries

Out of scope:

  • Implementation. Follow-up issues are filed from the design's conclusions.

Dependencies & Risks

  • ITL-910 is in progress and regenerates golden values; its handling of v0.2 stores depends on this design.
  • v0.3 is the public baseline: whatever scheme v0.3 writes becomes the first generation every later reader must understand.

Resources & References

Review in Linear

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Edgar Y. Walker and others added 5 commits November 11, 2026 16:28
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910)

Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses
arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove
StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run
pre-flight. Every whole-table read keeps rows of the current label only
(keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910)

Developer decisions of November 11, 2026: a read that leaves out rows of another
hash label logs one warning with the total; a side-effect node whose invocation
log holds deliveries logged under another Arrow label raises
SideEffectLogHashLabelError before delivering anything.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@eywalker eywalker changed the title feat(hashing): adopt starfix 0.5 (hash format 2) under arrow_v0.2; refuse stores written by v0.2 (ITL-910) feat(hashing): adopt starfix 0.5 (hash format 2) under arrow_v0.2; ignore and keep rows hashed under arrow_v0.1 (ITL-910) Nov 12, 2026
…reported, kept)

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eywalker pushed a commit that referenced this pull request Nov 12, 2026
…ld layouts reported (ITL-944)

Every stored row names the data context that wrote it (__data_context):
function pipeline and result tables, operator-cache tables, cached-source
tables and side-effect logs. It replaces __input_data_context_key and the
result table's _context_key. Layouts move to pdb_v2, rdb_v2, tdb_v2, and
operator-cache and cached-source tables gain odb_v1 and sdb_v1.

Reads and lookups keep rows of the current data context only. A table at an
older layout is counted and reported, never read as current, never written;
a side-effect log at an older layout, or holding deliveries of another data
context, raises SideEffectLogDataContextError before any delivery.

The result table also stores the full function version
(__pf_var_function_version). The label-based row filtering of PR #309
(hashing/stored_hash_labels.py) is removed; orcapod.contexts.stored_data_contexts
lists the data contexts a table holds.

Golden: pipeline:job.table_paths and pipeline:polars_filter_expr_job.table_paths
change (layout component only), and the four table[...] entry names under each
job follow the new paths with unchanged values.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
eywalker pushed a commit that referenced this pull request Nov 12, 2026
Golden files regenerated on this branch to keep its CI meaningful; the final regeneration
is shared with ITL-910, ITL-944, ITL-935 and ITL-942. Compared with the base (PR #309):
225 values change in label text only, 195 change digest or name, 250 entries are renamed
(their key holds a system-tag column name), 36 are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@eywalker

Copy link
Copy Markdown
Contributor Author

Closing as superseded: this branch's head is an ancestor of #324's merged head (6dc2a51), and #324 is merged to main as e140862. The findings CodeRabbit raised on this code were handled in #324's review rounds; the few that remain are tracked in ITL-975, ITL-951 and ITL-904.

Posted by a Claude Code agent on behalf of @eywalker.

@eywalker eywalker closed this Nov 12, 2026
@eywalker
eywalker deleted the eywalker/itl-910-adopt-starfix-05-hash-format-2-in-orcapod branch November 17, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant