Repository navigation
Expose Arrow graph batches for foundation-model consumers - #365
Conversation
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Change mapRequested behavior: expose graph columns that a foundation-model user can consume directly. Previously, native Arrow buffers became inspectable tensor/mask dictionaries that a columnar payload consumer must convert back into Arrow. The change adds one explicitly bounded public output format and preserves existing outputs. Consumers can normalize pairs during payload assembly, or request stable per-relation edge deduplication with
|
|
The Arrow output makes sense when the downstream RFM repo owns model-batch construction. Could you clarify the separate edge-deduplication tradeoff? Existing graph output deduplicates logical edge pairs, while If RFM needs unique edges, please identify where that normalization occurs and confirm the reported full-payload timings include it. This would clarify the consumer contract and whether each downstream model otherwise needs to implement the same topology normalization. |
04bdf1c to
0fffd73
Compare
A foundation-model consumer already normalizes these pairs while building its final payload (it maps nodes to their parents so duplicates get assigned to the same map slot repeatedly), so deduplicating upstream adds redundant work. That downstream normalization is included in the benchmark timings. I’ve added |
|
/ok to test 0f550da |
Victsm
left a comment
There was a problem hiding this comment.
Reviewed the native Arrow output path and the updated edge-deduplication contract. Native AnchorTable iteration, example isolation, edge orientation and existing output defaults are preserved; the new format is explicitly guarded to its supported scope. Earlier deduplication question is addressed, regression coverage and CI are green, and I found no remaining correctness blocker.
A foundation-model user may need Arrow tables to assemble model-specific payloads. The inspectable Graph output first encodes sampled nodes into tensor/mask dictionaries, requiring a columnar consumer to reconstruct Arrow columns. Add
Task.batches(output_format="columnar")to return publicColumnarGraph,ColumnarNodes, andColumnarEdgesdirectly from native Arrow buffers.deduplicate_edges=Truefor unique pairs per relation in first-occurrence order, preserving example isolation. Document the additional allocation/work and consumer normalization contract. Existing inspectable/model outputs remain unchanged.Stacked on #340. No native/Rust changes.
Benchmark
A foundation-model payload consumer was evaluated using the same indexes, native binary and frozen 10k-seed batches for both output formats. Each format received one warmup and five alternating measured draws. Columnar measurements use the default
deduplicate_edges=False; the opt-in deduplication path has not been benchmarked. Medians include the consumer's complete payload assembly and JSON encoding; preparation and validation are excluded.AMD EPYC 9654P, 16 sampler threads. All measured payloads are byte-identical across formats. Full 10k native/Python encoder parity and independent DuckDB source validation on 128 distributed anchors per workload pass. These measurements retain the native binary used for the #340 baseline and isolate the output-format change.
Testing
237 focused engine/API tests pass, including 56 columnar tests, existing graph direction/uniqueness/temporal/native-anchor suites, and generated API-reference checks. Tests cover both deduplication settings, stable uniqueness within/across chunks, empty/singleton edges, uint32/uint64 indexes, repeated entities in separate examples, reverse/self-relations, and rejection of changed deduplication settings on resume. Buffer-address and closed-task lifetime tests cover Arrow sharing; sharding, padding, shuffle and rejection tests cover the public contract. Consumer integration checks additionally cover asymmetric sampling radii, shuffled-anchor mapping and payload equality. Ruff and diff checks pass.
Change map
#365 (comment)