Repository navigation
feat(hashing): adopt starfix 0.5 (hash format 2) under arrow_v0.2; ignore and keep rows hashed under arrow_v0.1 (ITL-910) - #309
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….2 (ITL-910) Bump starfix to >=0.5.0,<0.6 and the Arrow hasher label to arrow_v0.2. Hash pyarrow's canonical extension types and extension-typed schemas with their extension identity instead of as their storage type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…row_v0.2 (ITL-910) Explicitly approved regeneration (developer decision 3, November 11, 2026): 184 Arrow-derived values change (arrow 53/53, datagram 40/171, pipeline 68/305, schema 22/92, semantic 1/64); no key added or removed among them. 21 arrow entries added: fixed-size lists, the ITL-848 collision pairs, a chunked Boolean column, canonical extension types and extension schemas. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nce and DESIGN_ISSUES (ITL-910, WIP) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ash label fail explicitly (ITL-910) Adds orcapod.errors.StoreHashLabelError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (ITL-910) A v0.3 node whose pipeline, result, operator, cached-source or side-effect table holds rows keyed under arrow_v0.1 (Orcapod v0.2, starfix format 1) now raises StoreHashLabelError on read and before its first write, and PipelineJob.run checks every node before the first one runs, so a refused store is left exactly as it was. No opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loaded jobs; tests (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ect the ITL-888 reuse claim (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (ITL-910) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ITL-910 Adopt starfix 0.5 (hash format 2) in Orcapod: dependency bump, label arrow_v0.2, golden regeneration
Decisions — November 11, 2026 (developer)
OverviewOrcapod hashes Arrow tables through This issue moves Orcapod onto starfix 0.5. Goals & Success Criteria
Scope & BoundariesIn scope:
Out of scope:
Dependencies & Risks
Resources & ReferencesITL-848 Starfix Arrow hasher gaps: hash_schema crashes on extension types; null, uuid, map and fixed-size-list columns cannot be hashed
OverviewSeveral workers hit limits of Reported on November 10, 2026 (PR metamorphic-labs/orcapod-python#278 and ITL-837 worker reports); each needs a minimal reproduction confirmed on
Goals & Success Criteria
Scope & BoundariesIn scope:
Out of scope:
Dependencies & Risks
Resources & References
ITL-846 Audit and test Arrow schema and data preservation across every operation, sync and async
OverviewOrcapod is type strict. Principle stated by the developer on November 10, 2026:
During the v0.2 work, each operator that was looked at closely turned out to change something it should only pass through: This issue is one thorough, test-backed check that every operation preserves the Arrow schema and the data it does not own, so the remaining cases are found systematically instead of by accident. Goals & Success Criteria
Scope & BoundariesIn scope:
Out of scope:
Dependencies & Risks
Resources & References
ITL-927 A fixed_size_list column crashes source construction: starfix cannot hash FixedSizeList data
OverviewAny source whose table has a The workshop's sample project ( starfix 0.5.0 (hash format 2, about to be released) also rejects FixedSizeList data, so adopting 0.5.0 (ITL-910) does not fix this. Goals & Success Criteria
Dependencies & Risks
Resources & ReferencesITL-938 Design: one coherent versioning system for hashes, hash labels and stored paths
OverviewDeveloper, November 11, 2026: "I want a thorough discussion & analyses just about the path for the hash. I also want to have a thorough consideration of everywhere we have some hash versioning (e.g. we have name_v0.2 eg for some of the hash values). Rather than keeping all of these things scattered, I want one very coherent and robust system." Orcapod records versions in many unrelated places, each added for its own reason:
Stored table paths come from the pipeline hash, which does not change when a hash label changes, so results written under different hash generations land in the same tables. The developer decided on November 11, 2026 that v0.3 fails explicitly when it meets v0.2 rows, and then asked whether an explicit name should make every path change for v0.3. This issue is a design discussion, held by the developer directly with a dedicated worker. It produces a design, not code. Goals & Success Criteria
Scope & BoundariesOut of scope:
Dependencies & Risks
Resources & References |
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: truePre-merge checks:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refusing the store (ITL-910) Developer decision of November 11, 2026 (ITL-938 session): v0.3 writes and uses arrow_v0.2 rows in the same tables next to arrow_v0.1 rows. Remove StoreHashLabelError, every check_stored_hash_labels() and the PipelineJob.run pre-flight. Every whole-table read keeps rows of the current label only (keep_rows_hashed_under); stored_hash_labels() lists the labels a table holds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… them (ITL-910) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…nder another label fail (ITL-910) Developer decisions of November 11, 2026: a read that leaves out rows of another hash label logs one warning with the total; a side-effect node whose invocation log holds deliveries logged under another Arrow label raises SideEffectLogHashLabelError before delivering anything. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…reported, kept) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ld layouts reported (ITL-944) Every stored row names the data context that wrote it (__data_context): function pipeline and result tables, operator-cache tables, cached-source tables and side-effect logs. It replaces __input_data_context_key and the result table's _context_key. Layouts move to pdb_v2, rdb_v2, tdb_v2, and operator-cache and cached-source tables gain odb_v1 and sdb_v1. Reads and lookups keep rows of the current data context only. A table at an older layout is counted and reported, never read as current, never written; a side-effect log at an older layout, or holding deliveries of another data context, raises SideEffectLogDataContextError before any delivery. The result table also stores the full function version (__pf_var_function_version). The label-based row filtering of PR #309 (hashing/stored_hash_labels.py) is removed; orcapod.contexts.stored_data_contexts lists the data contexts a table holds. Golden: pipeline:job.table_paths and pipeline:polars_filter_expr_job.table_paths change (layout component only), and the four table[...] entry names under each job follow the new paths with unchanged values. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Golden files regenerated on this branch to keep its CI meaningful; the final regeneration is shared with ITL-910, ITL-944, ITL-935 and ITL-942. Compared with the base (PR #309): 225 values change in label text only, 195 change digest or name, 250 entries are renamed (their key holds a system-tag column name), 36 are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Closing as superseded: this branch's head is an ancestor of #324's merged head (6dc2a51), and #324 is merged to Posted by a Claude Code agent on behalf of @eywalker. |
Summary
Orcapod now hashes Arrow data with starfix 0.5.0 (hash format 2, prefix
00 00 02) under the new labelarrow_v0.2(wasarrow_v0.1, starfix 0.4, format 1). Format 1 lost information, so different data reached the same row hash (ITL-848): nested-list lengths ([[[1]], [[2], [3]]]vs[[[1], [2]], [[3]]]), a null struct vs a struct of nulls, and non-nullable Booleans ([True]vs[True, False, False]); Boolean hashes also depended on chunking. Format 2 fixes these and hashesfixed_size_listdata (ITL-927). Following developer decisions 1–6 of November 11, 2026: the label is bumped, the main path uses format 2 only (no legacy 0.4 mode), the order options stay off (every hasher call keeps today's order-independent behaviour), and the golden files were regenerated explicitly for this change.Fixes ITL-910. Related: ITL-848 (the starfix-side gaps), ITL-927 (fixed-size lists), ITL-846 (
hash_schemaon extension types), ITL-913 / ITL-914 / ITL-938 (migration and versioning).Dependency
pyproject.toml:starfix~=0.4.0→starfix>=0.5.0,<0.6. starfix 0.5 needspyarrow>=19; Orcapod's floor is alreadypyarrow>=20(locked 23.0.1), so the pyarrow floor is unchanged.uv.lock: only thestarfixentry moves (0.4.0 → 0.5.0, sdist and wheel hashes) and the specifier line. (uv lockwith uv 0.13.7 also rewrote unrelated markers and the lock revision; those changes were not kept.)Label and context
hashing/versioned_hashers.py_CURRENT_ARROW_HASHER_ID,contexts/data/v0.1.jsonand the example incontexts/data/schemas/context_schema.json:arrow_v0.2. The context key and version staystd:v0.1:default/v0.1(decision D4, precedent ITL-888); the context changelog records "arrow_v0.2 = starfix hash format 2".Hasher changes besides the label
arrow.uuid,arrow.fixed_shape_tensor) derive frompa.BaseExtensionType, notpa.ExtensionType, so Orcapod's normalization skipped them, and on starfix 0.5 they would have hashed as their storage type (anarrow.uuidtable == afixed_size_binary(16)table). They are now hashed with their extension name and metadata (include_metadatawhenever a table has an extension type) (D2).StarfixArrowHasher.hash_schemanormalizes extension columns ashash_tabledoes, so aPathcolumn's schema no longer hashes likelarge_string(ITL-846; the strict xfail became a real test) (D3).Behaviour changes visible to users (besides hash values)
arrow_v0.1)fixed_size_listcolumn at any depth (source, operator)NotImplementedErrorat source constructionfixed_size_list<T>[n]≠large_list<T>arrow.uuid,arrow.fixed_shape_tensorcolumnsNotImplementedErrorhash_schemawith extension / null / view / interval typesValueError: Non-nullable field 'x' contains nullsmapcolumnValueErrorpa.Tablevalue)ValueError: Ambiguous node path 'x'"a/b"next to structa{b}ValueError: Ambiguous node path 'a/b'null/map/ interval / view data, union, run-end-encodedNotImplementedErrorValueError(same rejection; starfix names column and type)SideEffectLogHashLabelError; see belowEach row has a test in
tests/test_hashing/test_starfix_format2.py(and the extension-schema case intest_async_sync_equivalence.py). starfix'sValueErrorpropagates unwrapped (D5).CachedSource._build_merged_streamkeeps itsexcept NotImplementedError(meant for an unboundSourceProxy); hashing rejections are nowValueErrorand propagate instead of being taken for "source cannot provide data" — in practice they already fail at source construction. Function pods still cannot declaretuple[float, float](pre-existing, "Unsupported annotation"), so afixed_size_listcolumn reaches pods only through operators.For PR #305 (not yet on main): its
_unhashable_columnswrapper instream_builder.pycatchesNotImplementedErrorfromhash_table. Once both are on main, starfix raisesValueErrorand names the column path and type itself, andfixed_size_listhashes, so the wrapper should be removed (or catchValueError), and #305's CHANGELOG / sources.md / DESIGN_ISSUES text sayingfixed_size_listcannot be hashed should go;mapstays unsupported (nowValueError).Stores written by v0.2: rows ignored, recomputed, kept (developer decisions of November 11, 2026, ITL-938 session)
The earlier decision (refuse the store with
StoreHashLabelError) was reversed by the developer: "As long as [we] just store and make use of arrow_v0.2, then it's ok that arrow_v0.1-based values are present next [to] arrow_v0.2", "NOT fail but still be possible/completely clear to distinguish old vs new entries", and a later migration must stay possible. Only a change of a table's schema goes into its path; this value-only change stays in the same tables. So:orcapod.errors.StoreHashLabelError(never released), everycheck_stored_hash_labels()and thePipelineJob.runpre-flight.hashing/stored_hash_labels.keep_rows_hashed_under(table, column, current). Without it every reader below returned both generations (4 rows for 2 inputs; the 2 v0.2 rows before the first v0.3 run).stored_hash_labels(database, record_path)returns e.g.{"arrow_v0.1": 3, "arrow_v0.2": 3}(documented indocs/reference/hashing.md, section 9, with an executed sample).loggingWARNING on the loggerorcapod.hashing.stored_hash_labels, through one function,report_skipped_rows(record_path, skipped). Proposed wording, for review:Skipped 3 rows of stored table 'my_job/double/.../pdb_v1': written under another hash label than the current one. The rows are left in place unchanged and can be migrated later.It is logged once per reader object (node, result cache, cached source) and table, not once per process: stored paths are relative to the store, so a process-wide record would silence a second store with the same layout. No per-row detail.
orcapod.errors.SideEffectLogHashLabelError(aStoreError: fails the run whatever theerror_policy;record_path,found_label,current_label,node_label). A side-effect node whose invocation log holds deliveries logged under another Arrow label raises it at the start ofexecute/async_execute, before delivering or writing anything. Function, operator and source nodes of the same job are not refused. The developer wants a separate, later discussion of side-effect logs.tests/test_pipeline/test_stored_hash_labels.py, in-memory + Delta)FunctionJobNode)_fetch_joined_records: phase 1 of a run,get_cached_results,as_table,iter_data,get_all_records,as_source(DerivedSource), loaded jobs (also a job file saved by v0.2)__pipeline_base_entry_idTestFunctionNodeReads(5 tests, sync + async run),TestRunResultCache)ResultCache.get_all_records(get_all_cached_outputsofCachedFunctionPod/CachedDataFunction), which now takes the current label__input_data_hashTestResultCacheReadsOperatorJobNode, LOG / REPLAY)_read_stored_records:get_all_records, REPLAYas_table/iter_data/ run,as_source, loaded jobs_record_hashrecord id (000001/000002), then the node-content-hash filterTestOperatorNodeReads(4 tests)CachedSource)_read_cached_records:as_table,iter_data,get_all_records_record_hashTestCachedSourceSideEffectJobNode)get_record_by_id)TestSideEffects(sync + async, in-memory + Delta: nothing delivered, log and Delta files untouched)StatusObserver/LoggingObservertablesRows without an Orcapod label (operator-node and cached-source tables). Their record hashes are hex digests; only starfix's format number at the start names the generation. Measured: the node-content-hash filter does not separate generations (a source with an explicit
source_idhas the same node content hash under both labels), so without a filter those tables returned both generations. The format filter is exact forarrow_v0.1(format 1) vsarrow_v0.2(format 2), and nothing of the old generation is returned now. Limit: a later Arrow label that keeps starfix format 2 could not be told apart in these two kinds of table without storing a label there (a stored-schema change, not done here; a global hash-version identifier per row is being designed, not decided). Values that are not a 70-hex starfix digest, and labelled keys that carry noarrow_v...label (hand-made ids), are kept.Consequences (accepted by the developer: "old values looking absent without explicit migration is something we have to accept")
Tests:
tests/test_pipeline/test_stored_hash_labels.py(v0.2 rows ignored and inputs recomputed; old rows still present row for row, Delta files byte-identical; a second v0.3 run reuses v0.3 rows; every reader above;stored_hash_labelson pipeline, result, operator and log tables; the warning, once, with the total; the side-effect failure),tests/test_hashing/test_stored_hash_labels_helpers.py, and the real Orcapod v0.2.0 fixture intests/test_arrays/test_array_v02_store.py(rewritten again: every input recomputed, 3 rows per node for 3 inputs, every v0.2 file byte-identical, every v0.2 table still lists 3arrow_v0.1rows).tests/test_pipeline/test_store_read_failure.pyis back to main's version. Before the filters, 39 of these tests failed (both generations returned).Found, not fixed (not specific to hash labels): a
DeltaTableDatabaseobject keeps the table version it read first. A job object that read a table before a run does not see the rows the run added: on a v0.3-only Delta store holding 2 rows, a run with 7 inputs followed byjob.nodes["dbl"].get_all_records()on the same job returns 2 rows (a newly built job returns 7). Any incremental run on an existing Delta store can show it; a run on a v0.2 store is one such case, so the fixture test reads through a newly built job.Golden hashes (explicitly approved regeneration, decision 3)
Regenerated once with
generate_golden_hashes.py. 184 values changed, 21 added, none removed; every changed value is Arrow-derived: anarrow_v0.x-labelled hash (table/schema hashes, per-row data hashes, input-data hashes, stored record ids), a defaultsource_id(an Arrow table hash) and the record ids / system-tag values built from it, or a source content hash that embeds the table hash. No pipeline hash, schema hash, system-tag column name or table path moved; allsemantic_v0.1-only entries are unchanged exceptvalue.dataframe.basic(a pandas DataFrame, hashed throughStarfixArrowHasher).Representative values (main → branch)
arrowtable.boolarrow_v0.1:0000012fafed9df9923…arrow_v0.2:000002dee7bb2ff6dce…arrowtable.nested.list_listarrow_v0.1:000001639751855e661…arrow_v0.2:000002d0d198475a44d…arrowschema.width.int64arrow_v0.1:000001f31bd23e3b248…arrow_v0.2:000002756a354ce6028…arrowextension_table.extension.patharrow_v0.1:000001e32d9b28a94c4…arrow_v0.2:000002f558ddf91c1c5…semanticvalue.dataframe.basicarrow_v0.1:000001b0e17dcec7cd7…arrow_v0.2:000002bac72e84776fc…schemasource_stream_builder.default_source_id.source_id000001630ee60cef9dfe3cce7a6782…000002e12eafa1dc72203ca9da1ca5…schemasource_stream_builder.default_source_id.content_hashsemantic_v0.1:7eafdaffcb2ac49d…semantic_v0.1:167c1d63d67f2fa1…datagramarrow_source.int32.row[{"k":"r1"}].data_hasharrow_v0.1:000001a01cd0f32e44a…arrow_v0.2:000002d92f4d29531f4…pipelinebatch.row[{"subject":["s1","s2"]}].data_hasharrow_v0.1:0000010819e767e277e…arrow_v0.2:0000020b638d3ccc763…pipelinejob.table[golden_job/golden_adjust/…](stored record ids)['arrow_v0.1:00000137cf61ccacb…['arrow_v0.2:000002118147ab4ef…The 21 added entries pin the newly hashable inputs: fixed-size lists, the ITL-848 collision pairs (now different), a chunked Boolean column (equal to the unchunked one),
arrow.uuid/arrow.fixed_shape_tensor, andhash_schemaon extension columns.Every moved value
arrow.jsondatagram.jsonpipeline.jsonschema.jsonsemantic.jsonarrow.json: 53 changed, 21 addedextension_table.extension.patharrow_v0.1:000001e32d9b28a94c4f47e…arrow_v0.2:000002f558ddf91c1c5f706…extension_table.extension.unregistered_passthrougharrow_v0.1:00000197e4832ff6efd7436…arrow_v0.2:000002b9c5f64eb011d128c…extension_table.extension.uuidarrow_v0.1:0000016efd6e3b9a4c03e2c…arrow_v0.2:000002062ec7b363fa63cfb…schema.boolarrow_v0.1:000001e1a1ab1078fd92ded…arrow_v0.2:00000292f4b85e8651fe4c7…schema.empty.zero_rowsarrow_v0.1:000001f31bd23e3b2489f68…arrow_v0.2:000002756a354ce60282443…schema.float.float32arrow_v0.1:000001fcd0edfcd85a395e7…arrow_v0.2:0000026a27b2c2276546fc4…schema.float.float64arrow_v0.1:000001f68c38ed991e7d525…arrow_v0.2:000002a8bb0c99c2d60fd20…schema.layout.binaryarrow_v0.1:0000011b8b0793ca3db4779…arrow_v0.2:000002346ed5761cafbd7a6…schema.layout.large_binaryarrow_v0.1:0000011b8b0793ca3db4779…arrow_v0.2:000002346ed5761cafbd7a6…schema.layout.large_listarrow_v0.1:000001cec41debef42a67a5…arrow_v0.2:000002409130499175385ea…schema.layout.large_stringarrow_v0.1:00000191f7f5c26abcceb89…arrow_v0.2:000002d9023abe34e3bde2c…schema.layout.listarrow_v0.1:000001cec41debef42a67a5…arrow_v0.2:000002409130499175385ea…schema.layout.stringarrow_v0.1:00000191f7f5c26abcceb89…arrow_v0.2:000002d9023abe34e3bde2c…schema.nested.list_listarrow_v0.1:0000014387f6de75a16b23c…arrow_v0.2:0000029753856b34b9c54c1…schema.nested.list_structarrow_v0.1:000001a257fe950b239e9d3…arrow_v0.2:000002cd3954fa9d6958fc1…schema.nested.structarrow_v0.1:000001402dbf363e9e7ebaf…arrow_v0.2:0000021c6ab7531323b6ed5…schema.nullability.non_nullablearrow_v0.1:0000013487bfb8486c6973d…arrow_v0.2:000002389fdc8277c962a83…schema.nullability.nullablearrow_v0.1:000001f31bd23e3b2489f68…arrow_v0.2:000002756a354ce60282443…schema.nulls.int64arrow_v0.1:000001f31bd23e3b2489f68…arrow_v0.2:000002756a354ce60282443…schema.nulls.stringarrow_v0.1:00000191f7f5c26abcceb89…arrow_v0.2:000002d9023abe34e3bde2c…schema.order.abarrow_v0.1:00000100ad4d558238e9562…arrow_v0.2:000002b8f677fa5b62affbb…schema.order.baarrow_v0.1:00000100ad4d558238e9562…arrow_v0.2:000002b8f677fa5b62affbb…schema.order.rows_reversedarrow_v0.1:00000100ad4d558238e9562…arrow_v0.2:000002b8f677fa5b62affbb…schema.width.int16arrow_v0.1:00000178c6959f782c8a48a…arrow_v0.2:000002c9ea066d6e1b441ad…schema.width.int32arrow_v0.1:0000011844a51e73c25585a…arrow_v0.2:000002447a113da124fde87…schema.width.int64arrow_v0.1:000001f31bd23e3b2489f68…arrow_v0.2:000002756a354ce60282443…schema.width.int8arrow_v0.1:000001d22fd4d64d8554ce4…arrow_v0.2:00000233ec0dccf2b00dbbd…schema.width.uint8arrow_v0.1:000001c63af839f862f0328…arrow_v0.2:00000252e16232ff906ecf1…table.boolarrow_v0.1:0000012fafed9df99235be9…arrow_v0.2:000002dee7bb2ff6dce8ddc…table.empty.zero_rowsarrow_v0.1:000001d93c4003276f674c8…arrow_v0.2:000002d655beec05f54e13a…table.float.float32arrow_v0.1:00000177212619946903d4b…arrow_v0.2:00000287ee09555fc05dfba…table.float.float64arrow_v0.1:000001c1d762b8d385549ea…arrow_v0.2:000002d83df46319a27052e…table.layout.binaryarrow_v0.1:000001b19bcc9f2f94795bf…arrow_v0.2:0000021977fb0b4aac38946…table.layout.large_binaryarrow_v0.1:000001b19bcc9f2f94795bf…arrow_v0.2:0000021977fb0b4aac38946…table.layout.large_listarrow_v0.1:00000189f2c4331bbc82425…arrow_v0.2:0000027db606bdc3327cd8b…table.layout.large_stringarrow_v0.1:000001590b6b8a65a493487…arrow_v0.2:0000021aedf65c94c24afaa…table.layout.listarrow_v0.1:00000189f2c4331bbc82425…arrow_v0.2:0000027db606bdc3327cd8b…table.layout.stringarrow_v0.1:000001590b6b8a65a493487…arrow_v0.2:0000021aedf65c94c24afaa…table.nested.list_listarrow_v0.1:000001639751855e6618fdf…arrow_v0.2:000002d0d198475a44dc9ab…table.nested.list_structarrow_v0.1:0000013c0c348a94fa935da…arrow_v0.2:0000029919cc15883d26c03…table.nested.structarrow_v0.1:0000012af9c2ad25c501bbf…arrow_v0.2:0000022efc170a4f2578623…table.nullability.non_nullablearrow_v0.1:000001a55bac289fa4cb83a…arrow_v0.2:000002b37da33f664f9dc4a…table.nullability.nullablearrow_v0.1:000001d469c2c16d7e2c80b…arrow_v0.2:00000236be7e40075274bab…table.nulls.int64arrow_v0.1:000001604458bd26489ed61…arrow_v0.2:0000023ba22c1f5257d8cf5…table.nulls.stringarrow_v0.1:00000189124cc15f2fbfc1d…arrow_v0.2:0000026885780c872492587…table.order.abarrow_v0.1:000001fb52c5ca8baed03ac…arrow_v0.2:000002d29b71de1ff0980cd…table.order.baarrow_v0.1:000001fb52c5ca8baed03ac…arrow_v0.2:000002d29b71de1ff0980cd…table.order.rows_reversedarrow_v0.1:00000151aa84ec1e72539ee…arrow_v0.2:000002edcc170ff123dccde…table.width.int16arrow_v0.1:0000013e000f3e0b4beeba1…arrow_v0.2:000002c53e3f002b007a80a…table.width.int32arrow_v0.1:000001d831e96457f732f78…arrow_v0.2:000002ee36b64c06c9157f5…table.width.int64arrow_v0.1:000001d469c2c16d7e2c80b…arrow_v0.2:00000236be7e40075274bab…table.width.int8arrow_v0.1:00000160707f6988db37faf…arrow_v0.2:000002f195acdfc1ca1214d…table.width.uint8arrow_v0.1:000001dded35c6cee950b7a…arrow_v0.2:00000254a73782eb935f014…extension_schema.extension.canonical_fixed_shape_tensorarrow_v0.2:0000022df2fb43d6457645d…extension_schema.extension.canonical_uuidarrow_v0.2:00000236be584e0cd3a2668…extension_schema.extension.patharrow_v0.2:0000024887bbb622b764650…extension_schema.extension.unregistered_passthrougharrow_v0.2:000002853ab951d35ce3198…extension_schema.extension.uuidarrow_v0.2:000002bfbbdea0a2cf9f094…extension_table.extension.canonical_fixed_shape_tensorarrow_v0.2:0000029cdbbb5f9c731c669…extension_table.extension.canonical_uuidarrow_v0.2:0000024cf6272f99fa3465a…schema.bool.chunkedarrow_v0.2:00000292f4b85e8651fe4c7…schema.collision.list_lengths_aarrow_v0.2:00000217b4f541f59698108…schema.collision.list_lengths_barrow_v0.2:00000217b4f541f59698108…schema.collision.null_structarrow_v0.2:000002377ba0fc7c623d903…schema.collision.struct_of_nullsarrow_v0.2:000002377ba0fc7c623d903…schema.fixed_size_list.float32_2arrow_v0.2:00000256387ebbc501cb6fc…schema.fixed_size_list.null_slotarrow_v0.2:000002a91c8dfd080d7012a…table.bool.chunkedarrow_v0.2:000002dee7bb2ff6dce8ddc…table.collision.list_lengths_aarrow_v0.2:000002295a4cad9a00a66f4…table.collision.list_lengths_barrow_v0.2:0000026806409bd5df9d075…table.collision.null_structarrow_v0.2:00000238e6e122820c9902b…table.collision.struct_of_nullsarrow_v0.2:0000024ef1bfba9ecf0d650…table.fixed_size_list.float32_2arrow_v0.2:000002a963ce04c642d126d…table.fixed_size_list.null_slotarrow_v0.2:00000223564917b29cabbfd…datagram.json: 40 changed, 0 addedarrow_source.int32.row[{"k":"r1"}].data_hasharrow_v0.1:000001a01cd0f32e44a4839…arrow_v0.2:000002d92f4d29531f41258…arrow_source.int32.row[{"k":"r2"}].data_hasharrow_v0.1:000001cbdc2fbc671d4157f…arrow_v0.2:00000243c87e6d05d3588ed…arrow_source.int32.row[{"k":"r3"}].data_hasharrow_v0.1:0000012f75fdca845f528ca…arrow_v0.2:000002825e1c06b5514e562…arrow_source.int64.row[{"k":"r1"}].data_hasharrow_v0.1:000001c028af694082af2f2…arrow_v0.2:000002784ccdd2b780948d2…arrow_source.int64.row[{"k":"r2"}].data_hasharrow_v0.1:0000017e332b258dbc5fe1f…arrow_v0.2:0000021d3e14bd65d5ca263…arrow_source.int64.row[{"k":"r3"}].data_hasharrow_v0.1:0000015c299c665cda513e3…arrow_v0.2:0000023846c7aef08fc1400…arrow_source.int64_nullable.row[{"k":"r1"}].data_hasharrow_v0.1:0000012081a40ec7eba7115…arrow_v0.2:000002521ff3d4701e70a5a…arrow_source.int64_nullable.row[{"k":"r2"}].data_hasharrow_v0.1:000001ebc7fce25941def04…arrow_v0.2:000002601fc387de4781312…arrow_source.int64_nullable.row[{"k":"r3"}].data_hasharrow_v0.1:000001951dc05c8c79ecd4a…arrow_v0.2:000002565243c66b8ae12de…arrow_source.int8.row[{"k":"r1"}].data_hasharrow_v0.1:000001760d63148f6020615…arrow_v0.2:000002b17bad7455bf81d47…arrow_source.int8.row[{"k":"r2"}].data_hasharrow_v0.1:0000012622e7bf473c8c58d…arrow_v0.2:000002ff7e5abaafd6df855…arrow_source.int8.row[{"k":"r3"}].data_hasharrow_v0.1:000001c502c37a181f7c3e1…arrow_v0.2:000002fa70cfa37eaab0c4e…arrow_source.large_string.row[{"k":"r1"}].data_hasharrow_v0.1:000001b4407c40d91327434…arrow_v0.2:0000027dc69926b790513bb…arrow_source.large_string.row[{"k":"r2"}].data_hasharrow_v0.1:000001cd356426fd64cb498…arrow_v0.2:000002e2668873da695349e…arrow_source.large_string.row[{"k":"r3"}].data_hasharrow_v0.1:000001e0c4e88cf1c63c94e…arrow_v0.2:000002b08595f56b1cd4a87…arrow_source.string.row[{"k":"r1"}].data_hasharrow_v0.1:000001b4407c40d91327434…arrow_v0.2:0000027dc69926b790513bb…arrow_source.string.row[{"k":"r2"}].data_hasharrow_v0.1:000001cd356426fd64cb498…arrow_v0.2:000002e2668873da695349e…arrow_source.string.row[{"k":"r3"}].data_hasharrow_v0.1:000001e0c4e88cf1c63c94e…arrow_v0.2:000002b08595f56b1cd4a87…dataclass.row[{"k":"r1"}].data_hasharrow_v0.1:0000015e306dd402af59e45…arrow_v0.2:0000027b15d8db313a05d7a…dataclass.row[{"k":"r2"}].data_hasharrow_v0.1:000001b088edbc3232e6d83…arrow_v0.2:000002abd8873e34ca7ee3f…dataframe.row[{"k":"r1"}].data_hasharrow_v0.1:00000108d551603b5a5df12…arrow_v0.2:00000229c5394be73a9c867…dataframe.row[{"k":"r2"}].data_hasharrow_v0.1:00000134950bb185ada91e9…arrow_v0.2:0000024dc4bbb7a657d7ba9…list_path.row[{"k":"r1"}].data_hasharrow_v0.1:00000135fc611ef9ad245d4…arrow_v0.2:000002cc2ece0ae8ac64848…list_path.row[{"k":"r2"}].data_hasharrow_v0.1:0000018efe66b052bc46996…arrow_v0.2:000002cdaddd554ca855740…ndarray.row[{"k":"r1"}].data_hasharrow_v0.1:000001c80cacf9821f15fd1…arrow_v0.2:000002e79fea6f79a5a7e61…ndarray.row[{"k":"r2"}].data_hasharrow_v0.1:000001ba0bc856108ff6198…arrow_v0.2:000002922f3a9172b4d7363…nested.row[{"k":"r1"}].data_hasharrow_v0.1:000001c31188a44c589ce3d…arrow_v0.2:00000261f569412bb12da24…nested.row[{"k":"r2"}].data_hasharrow_v0.1:000001a7658c283bcd7a1c6…arrow_v0.2:000002f4312647b9ca61759…optional.row[{"k":"r1"}].data_hasharrow_v0.1:000001f910e5b3279111cf4…arrow_v0.2:000002f774b1bc717e2ddf7…optional.row[{"k":"r2"}].data_hasharrow_v0.1:0000010d55a89a30404439b…arrow_v0.2:0000027714fb655c044c9c1…path.row[{"k":"r1"}].data_hasharrow_v0.1:00000135bf11f7d6830036c…arrow_v0.2:000002a230080e1f03f2254…path.row[{"k":"r2"}].data_hasharrow_v0.1:0000015b4f8156dc62851f9…arrow_v0.2:00000234b6166fa805040e4…pydantic.row[{"k":"r1"}].data_hasharrow_v0.1:0000012280b233c56771fa4…arrow_v0.2:000002459c1d7ed12f91d7f…pydantic.row[{"k":"r2"}].data_hasharrow_v0.1:00000138d77ff36d7b2f3c3…arrow_v0.2:000002ce57007b0730b9021…scalars.row[{"k":"r1"}].data_hasharrow_v0.1:00000162a09a0c2dd7d8e7c…arrow_v0.2:000002c8161def16592dc28…scalars.row[{"k":"r2"}].data_hasharrow_v0.1:000001d49094978c1867f74…arrow_v0.2:0000020528b8069a7f49fd5…set_uuid.row[{"k":"r1"}].data_hasharrow_v0.1:0000011eb69965d474d2209…arrow_v0.2:000002648eddf1e15a4ed2e…set_uuid.row[{"k":"r2"}].data_hasharrow_v0.1:000001fc38bc3b27de6a066…arrow_v0.2:000002ec023f0196ddaaa81…uuid.row[{"k":"r1"}].data_hasharrow_v0.1:000001e05de561e03537d1d…arrow_v0.2:00000228d36dcdde1b6e7ff…uuid.row[{"k":"r2"}].data_hasharrow_v0.1:0000014c189ef879ddf91cf…arrow_v0.2:00000236a95b5f9e34143f9…pipeline.json: 68 changed, 0 addedbatch.row[{"subject":["s1","s2"]}].data_hasharrow_v0.1:0000010819e767e277e136c…arrow_v0.2:0000020b638d3ccc763f2a8…batch.row[{"subject":["s3"]}].data_hasharrow_v0.1:0000014957b9ee25e0ea51c…arrow_v0.2:000002928bbf5d31fadf74e…column_selection.drop_data.row[{"subject":"s1"}].data_hasharrow_v0.1:000001ee27bd12d66af74db…arrow_v0.2:000002c734791b7452bb2af…column_selection.drop_data.row[{"subject":"s2"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…column_selection.drop_data.row[{"subject":"s3"}].data_hasharrow_v0.1:000001966c5e58c797ab332…arrow_v0.2:0000025f33a2f5bf3e956ed…column_selection.drop_tag.row[{"cohort":"a"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…column_selection.drop_tag.row[{"cohort":"b"}].data_hasharrow_v0.1:000001966c5e58c797ab332…arrow_v0.2:0000025f33a2f5bf3e956ed…column_selection.select_data.row[{"subject":"s1"}].data_hasharrow_v0.1:0000017cb817a7c2cfd6c71…arrow_v0.2:00000281f4d4a4e7742a2ca…column_selection.select_data.row[{"subject":"s2"}].data_hasharrow_v0.1:00000124d0ecb4bf9d093b6…arrow_v0.2:0000022420dc43cfa6c4d36…column_selection.select_data.row[{"subject":"s3"}].data_hasharrow_v0.1:000001810e6ae31eebcd4f2…arrow_v0.2:000002805d579510905b591…column_selection.select_tag.row[{"cohort":"a"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…column_selection.select_tag.row[{"cohort":"b"}].data_hasharrow_v0.1:000001966c5e58c797ab332…arrow_v0.2:0000025f33a2f5bf3e956ed…group_by.row[{"cohort":"a"}].data_hasharrow_v0.1:0000018910e185c56ec7890…arrow_v0.2:0000022b0f148a444e75dad…group_by.row[{"cohort":"b"}].data_hasharrow_v0.1:000001ba6adaea3c56bf0be…arrow_v0.2:0000028b04ecd69f72beab1…index_pick.index.row[{"subject":"s1"}].data_hasharrow_v0.1:00000189f1942fbc40737ff…arrow_v0.2:0000025f5739a1fc427e583…index_pick.index.row[{"subject":"s2"}].data_hasharrow_v0.1:00000164b528c22b7183231…arrow_v0.2:000002d9f79b11b0d692847…index_pick.pick.row[{"subject":"s1"}].data_hasharrow_v0.1:000001380f07142b3132f93…arrow_v0.2:000002d3aa3df31c715ed95…index_pick.pick.row[{"subject":"s2"}].data_hasharrow_v0.1:00000157b70714f9f88f916…arrow_v0.2:00000218fa0f5bdf561a92c…job.node[adjust].row[{"subject":"s1"}].data_hasharrow_v0.1:0000014a3871996b0dc2563…arrow_v0.2:000002df3a95f3b9eddaf07…job.node[adjust].row[{"subject":"s2"}].data_hasharrow_v0.1:0000014c738e65c53a408df…arrow_v0.2:0000023565098d1a88ecd75…job.node[joined].row[{"subject":"s1"}].data_hasharrow_v0.1:00000130ab270ba209296b0…arrow_v0.2:000002c945ad08138f51298…job.node[joined].row[{"subject":"s2"}].data_hasharrow_v0.1:00000186afa42ffc71c4517…arrow_v0.2:0000026b8369e60be10e43d…job.node[scans].row[{"subject":"s1"}].data_hasharrow_v0.1:000001ac375e92d49c253f9…arrow_v0.2:000002ef2b37a1979bd7348…job.node[scans].row[{"subject":"s2"}].data_hasharrow_v0.1:000001f7fa6c7bb04912cca…arrow_v0.2:00000210b14e22145ce1d5f…job.node[subjects].row[{"subject":"s1"}].data_hasharrow_v0.1:00000137de028aa69769a5d…arrow_v0.2:000002d1c5b49de38ef6efc…job.node[subjects].row[{"subject":"s2"}].data_hasharrow_v0.1:000001584cc7c0907761513…arrow_v0.2:000002338ce8710f5ecb156…job.node[subjects].row[{"subject":"s3"}].data_hasharrow_v0.1:000001562e4c70b60df10ed…arrow_v0.2:0000023401f1ce40c8cfdd6…job.table[golden_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes['arrow_v0.1:00000130ab270ba209296…['arrow_v0.2:0000026b8369e60be10e4…job.table[golden_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:b61a42486c29af296f5c2f1d8129f20b61e023d492747e397a50756cb153aff8/pdb_v1].record_ids['arrow_v0.1:00000137cf61ccacb0c63…['arrow_v0.2:000002118147ab4ef2d7e…join_pod.join.row[{"subject":"s1"}].data_hasharrow_v0.1:00000130ab270ba209296b0…arrow_v0.2:000002c945ad08138f51298…join_pod.join.row[{"subject":"s2"}].data_hasharrow_v0.1:00000186afa42ffc71c4517…arrow_v0.2:0000026b8369e60be10e43d…join_pod.pod.row[{"subject":"s1"}].data_hasharrow_v0.1:0000014a3871996b0dc2563…arrow_v0.2:000002df3a95f3b9eddaf07…join_pod.pod.row[{"subject":"s2"}].data_hasharrow_v0.1:0000014c738e65c53a408df…arrow_v0.2:0000023565098d1a88ecd75…map_data.row[{"subject":"s1"}].data_hasharrow_v0.1:0000012b54015317016ea37…arrow_v0.2:0000022ad1363967c26697b…map_data.row[{"subject":"s2"}].data_hasharrow_v0.1:000001bf869237358aa3c91…arrow_v0.2:000002a61fcaac02c3fd261…map_data.row[{"subject":"s3"}].data_hasharrow_v0.1:000001e87c392dee12edd21…arrow_v0.2:000002ef21c37efbb54c93f…merge_join.row[{"subject":"s1"}].data_hasharrow_v0.1:000001ec09b4a15ec5e06d5…arrow_v0.2:000002654a49c9ec7661bbb…merge_join.row[{"subject":"s3"}].data_hasharrow_v0.1:0000013aeb0e14ea94b020f…arrow_v0.2:000002b78038f0da00d8b6e…pod.row[{"subject":"s1"}].data_hasharrow_v0.1:00000155d85a85806e3601d…arrow_v0.2:0000024da91c65d3ebacee0…pod.row[{"subject":"s2"}].data_hasharrow_v0.1:0000017db75343362e61ae6…arrow_v0.2:0000021afa3e9774d8a2852…pod.row[{"subject":"s3"}].data_hasharrow_v0.1:000001a3b408003eb5481ee…arrow_v0.2:000002c2a1324b1334c4c49…polars_filter.row[{"cohort":"a","subject":"s1"}].data_hasharrow_v0.1:000001ee27bd12d66af74db…arrow_v0.2:000002c734791b7452bb2af…polars_filter.row[{"cohort":"a","subject":"s2"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…polars_filter_expr.expr.row[{"cohort":"a","subject":"s1"}].data_hasharrow_v0.1:000001ee27bd12d66af74db…arrow_v0.2:000002c734791b7452bb2af…polars_filter_expr.expr.row[{"cohort":"a","subject":"s2"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…polars_filter_expr.mixed.row[{"cohort":"a","subject":"s1"}].data_hasharrow_v0.1:000001ee27bd12d66af74db…arrow_v0.2:000002c734791b7452bb2af…polars_filter_expr.mixed.row[{"cohort":"a","subject":"s2"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…polars_filter_expr_job.node[adjust].row[{"subject":"s2"}].data_hasharrow_v0.1:0000014c738e65c53a408df…arrow_v0.2:0000023565098d1a88ecd75…polars_filter_expr_job.node[joined].row[{"subject":"s2"}].data_hasharrow_v0.1:00000186afa42ffc71c4517…arrow_v0.2:0000026b8369e60be10e43d…polars_filter_expr_job.node[older].row[{"subject":"s2"}].data_hasharrow_v0.1:000001584cc7c0907761513…arrow_v0.2:000002338ce8710f5ecb156…polars_filter_expr_job.node[older].row[{"subject":"s3"}].data_hasharrow_v0.1:000001562e4c70b60df10ed…arrow_v0.2:0000023401f1ce40c8cfdd6…polars_filter_expr_job.node[scans].row[{"subject":"s1"}].data_hasharrow_v0.1:000001ac375e92d49c253f9…arrow_v0.2:000002ef2b37a1979bd7348…polars_filter_expr_job.node[scans].row[{"subject":"s2"}].data_hasharrow_v0.1:000001f7fa6c7bb04912cca…arrow_v0.2:00000210b14e22145ce1d5f…polars_filter_expr_job.node[subjects].row[{"subject":"s1"}].data_hasharrow_v0.1:00000137de028aa69769a5d…arrow_v0.2:000002d1c5b49de38ef6efc…polars_filter_expr_job.node[subjects].row[{"subject":"s2"}].data_hasharrow_v0.1:000001584cc7c0907761513…arrow_v0.2:000002338ce8710f5ecb156…polars_filter_expr_job.node[subjects].row[{"subject":"s3"}].data_hasharrow_v0.1:000001562e4c70b60df10ed…arrow_v0.2:0000023401f1ce40c8cfdd6…polars_filter_expr_job.table[golden_filter_job/_result/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/rdb_v1].input_data_hashes['arrow_v0.1:00000186afa42ffc71c45…['arrow_v0.2:0000026b8369e60be10e4…polars_filter_expr_job.table[golden_filter_job/golden_adjust/semantic_v0.1:5cd978b399b203b46d0d8a7d395dcbb592d63aaeeefbd3da97f59149202d48f3/v0/python.function.v0/schema:semantic_v0.1:1bf083d50625083700e69f699b90f9f01c9c1e93d56e1a56feb4ad1506005e93/pdb_v1].record_ids['arrow_v0.1:000001ea2218fa4e75231…['arrow_v0.2:0000022ad2e7cf689d570…semi_join.row[{"subject":"s1"}].data_hasharrow_v0.1:00000137de028aa69769a5d…arrow_v0.2:000002d1c5b49de38ef6efc…semi_join.row[{"subject":"s2"}].data_hasharrow_v0.1:000001584cc7c0907761513…arrow_v0.2:000002338ce8710f5ecb156…sources.cohorts.row[{"cohort":"a","subject":"s1"}].data_hasharrow_v0.1:000001ee27bd12d66af74db…arrow_v0.2:000002c734791b7452bb2af…sources.cohorts.row[{"cohort":"a","subject":"s2"}].data_hasharrow_v0.1:00000133552dc58a31ff558…arrow_v0.2:00000252e4f32a133cd6a88…sources.cohorts.row[{"cohort":"b","subject":"s3"}].data_hasharrow_v0.1:000001966c5e58c797ab332…arrow_v0.2:0000025f33a2f5bf3e956ed…sources.scans.row[{"subject":"s1"}].data_hasharrow_v0.1:000001ac375e92d49c253f9…arrow_v0.2:000002ef2b37a1979bd7348…sources.scans.row[{"subject":"s2"}].data_hasharrow_v0.1:000001f7fa6c7bb04912cca…arrow_v0.2:00000210b14e22145ce1d5f…sources.subjects.row[{"subject":"s1"}].data_hasharrow_v0.1:00000137de028aa69769a5d…arrow_v0.2:000002d1c5b49de38ef6efc…sources.subjects.row[{"subject":"s2"}].data_hasharrow_v0.1:000001584cc7c0907761513…arrow_v0.2:000002338ce8710f5ecb156…sources.subjects.row[{"subject":"s3"}].data_hasharrow_v0.1:000001562e4c70b60df10ed…arrow_v0.2:0000023401f1ce40c8cfdd6…schema.json: 22 changed, 0 addedsource_stream_builder.default_source_id.content_hashsemantic_v0.1:7eafdaffcb2ac49d7607…semantic_v0.1:167c1d63d67f2fa19980…source_stream_builder.default_source_id.row[{"subject":"s1"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b417736771c09013eb5f5f5582ad41da30acbcf72d25e22e5ed45f2ebcd9ba0e574ce4absource_stream_builder.default_source_id.row[{"subject":"s1"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367000001630ee60cef9dfe3cce7a6782fa30…000002e12eafa1dc72203ca9da1ca5d064…source_stream_builder.default_source_id.row[{"subject":"s1"}].data_hasharrow_v0.1:00000128b75b53ced7f9752…arrow_v0.2:00000289f61a7619845a37a…source_stream_builder.default_source_id.row[{"subject":"s2"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367e2960b661cf05e14874582345bdde8e87a85d0e335bf539d990068a86fb4409esource_stream_builder.default_source_id.row[{"subject":"s2"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367000001630ee60cef9dfe3cce7a6782fa30…000002e12eafa1dc72203ca9da1ca5d064…source_stream_builder.default_source_id.row[{"subject":"s2"}].data_hasharrow_v0.1:000001a39a4d7b517d56c4c…arrow_v0.2:0000029f0b00a74de3d3f45…source_stream_builder.default_source_id.row[{"subject":"s3"}]._tag_record_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b41773675d4deb7fe5675941942e2b590adaf240174dcf5eb0c05b08967518af5d71cd7bsource_stream_builder.default_source_id.row[{"subject":"s3"}]._tag_source_id::f639932798b21a97cbf494c5b56b76142293dd248b775ecdcfdce639b4177367000001630ee60cef9dfe3cce7a6782fa30…000002e12eafa1dc72203ca9da1ca5d064…source_stream_builder.default_source_id.row[{"subject":"s3"}].data_hasharrow_v0.1:0000010364967ccec6827a7…arrow_v0.2:0000026b8c4f7e93665f79d…source_stream_builder.default_source_id.source_id000001630ee60cef9dfe3cce7a6782fa30…000002e12eafa1dc72203ca9da1ca5d064…source_stream_builder.default_source_id.table_hasharrow_v0.1:000001630ee60cef9dfe3cc…arrow_v0.2:000002e12eafa1dc72203ca…source_stream_builder.named_source_id.content_hashsemantic_v0.1:f424ce4643b9e4dd3757…semantic_v0.1:69b4b3098babd295b560…source_stream_builder.named_source_id.row[{"subject":"s1"}].data_hasharrow_v0.1:00000128b75b53ced7f9752…arrow_v0.2:00000289f61a7619845a37a…source_stream_builder.named_source_id.row[{"subject":"s2"}].data_hasharrow_v0.1:000001a39a4d7b517d56c4c…arrow_v0.2:0000029f0b00a74de3d3f45…source_stream_builder.named_source_id.row[{"subject":"s3"}].data_hasharrow_v0.1:0000010364967ccec6827a7…arrow_v0.2:0000026b8c4f7e93665f79d…source_stream_builder.named_source_id.table_hasharrow_v0.1:000001630ee60cef9dfe3cc…arrow_v0.2:000002e12eafa1dc72203ca…source_stream_builder.record_id_column.content_hashsemantic_v0.1:6983755521b8c4ca1771…semantic_v0.1:0bbf3af4ed09b723aa34…source_stream_builder.record_id_column.row[{"subject":"s1"}].data_hasharrow_v0.1:00000128b75b53ced7f9752…arrow_v0.2:00000289f61a7619845a37a…source_stream_builder.record_id_column.row[{"subject":"s2"}].data_hasharrow_v0.1:000001a39a4d7b517d56c4c…arrow_v0.2:0000029f0b00a74de3d3f45…source_stream_builder.record_id_column.row[{"subject":"s3"}].data_hasharrow_v0.1:0000010364967ccec6827a7…arrow_v0.2:0000026b8c4f7e93665f79d…source_stream_builder.record_id_column.table_hasharrow_v0.1:000001630ee60cef9dfe3cc…arrow_v0.2:000002e12eafa1dc72203ca…semantic.json: 1 changed, 0 addedvalue.dataframe.basicarrow_v0.1:000001b0e17dcec7cd73f09…arrow_v0.2:000002bac72e84776fc6d62…Other pinned values re-recorded in format 2 (outside the golden files):
test_starfix_arrow_hasher.py(schema, nullability and table digests),test_materialize_source_info.py(operator record ids and node content hash; the pre-fix ids are re-derived with_broadcast_row_zero),test_async_sync_equivalence.py(ITL-837 row hashes; stream hashes unchanged),test_struct_layout_identity.py(stream content and row hashes; small and large layouts still equal),test_group_by.py(member order byrecord_id, which derives from the Arrow-hashed defaultsource_id:["a","b"]→["b","a"]), and the worked example indocs/reference/hashing.md.Golden files touched (conflicts with every open PR adding golden entries, e.g. #301, #300, #305, #306):
tests/test_hashing/golden/data/{arrow,datagram,pipeline,schema,semantic}.json,tests/test_hashing/golden/cases.py,tests/test_hashing/golden/generate_golden_hashes.py. Rebasing such a PR onto this one means regenerating only its added entries.Performance (starfix-python format 2 is pure Python)
Orcapod's
StarfixArrowHasher.hash_tableon 100,000-row tables (best of 3, Apple Silicon):Whole-table hashing is as fast or faster; single-row hashes (record ids, per-row data hashes) are about 20% slower.
tests/test_hashing+tests/test_core/operators: 27.6 s on main, 27.8 s on this branch.Test plan
test_starfix_format2.py+ the ITL-846 test), and for the v0.2-store handling 39 tests with both generations returned before the filters were added; pass now.SPIRAL_INTEGRATION_TESTS), S3/MinIO (no container), Graphvizdotnot on PATH, PostgreSQL opt-in.tests/test_hashing/goldenis identical to the previous head of this branch.origin/mainhad no commits the branch lacks, so no merge and no further golden regeneration were needed.ruff check,ruff format --check: clean.basedpyright: exit 0, 0 errors.mkdocs build --strict(tests/test_docs) pass.notebooks/tutorials/01_introduction_to_orcapod.ipynbstill showsarrow_v0.1values in two stored outputs (not executed by tests; re-running it would rewrite unrelated outputs).Linear
Fixes ITL-910
ITL-848
ITL-927
ITL-846
ITL-938
🤖 Generated with Claude Code