feat(blob): support MAP<K, BLOB> reads in Go - #784
Conversation
6fa5e23 to
9e2e62a
Compare
9e2e62a to
85a3482
Compare
| }) | ||
| .collect::<crate::Result<Vec<_>>>()? | ||
| } else { | ||
| let payload = read_blob_entry(reader, blob_entry_range(&payload_range)).await?; |
There was a problem hiding this comment.
Please reject oversized inline payloads before this read. total_value_length is a u64, but these values are later collected into an Arrow BinaryArray, whose 32-bit offsets panic once the total byte length exceeds i32::MAX. ARRAY already preflights this case; MAP should do the same and also validate the flattened batch total.
There was a problem hiding this comment.
Fixed in cd19225. Inline MAP values are now rejected before the complete entry read, and flattened key/value batches enforce Arrow i32 offset limits. Added sparse >2 GiB regression coverage.
| } | ||
| } | ||
|
|
||
| func TestStringBlobMapDescriptors(t *testing.T) { |
There was a problem hiding this comment.
Could this test cover the actual read path instead of starting from a hand-built Arrow map? As written, it only tests descriptor extraction and BlobReader. A writer-generated fixture read through NewReadBuilderWithOptions would also cover the Rust decoder, C Data conversion, and Go API.
There was a problem hiding this comment.
Fixed in cd19225. The Go test now reads a PyPaimon-generated table through NewReadBuilderWithOptions, Rust decoding, C Data import, Go map extraction, and ReadBlobs.
| "dlf.oss-endpoint": os.Getenv("DLF_OSS_ENDPOINT"), | ||
| "dlf.token-loader": "ecs", | ||
| "dlf.token-ecs-role-name": os.Getenv("DLF_ECS_ROLE"), | ||
| "dlf.access-key-id": os.Getenv("DLF_ACCESS_KEY_ID"), |
There was a problem hiding this comment.
This changes the DLF example from the ECS RAM-role loader to static credentials, which is unrelated to MAP<K, BLOB> support and drops automatic credential refresh. Please revert it here or move it to a separate PR with its own rationale.
There was a problem hiding this comment.
Fixed in cd19225. Restored the ECS RAM-role loader example with dlf.token-loader and dlf.token-ecs-role-name.
cd19225 to
5ca3aad
Compare
5ca3aad to
1b3e7b7
Compare
Summary
MAP<K, BLOB>layout as values or descriptors in RustMAP<STRING, BLOB>descriptors as a native Go mapBlobReader:OpenBlobstreams and seeks;ReadBlobsbatches with range mergingNewReadBuilderWithOptionsexposes the existing C API and is equivalent to Java/PyPaimon'stable.copy(options).newReadBuilder().This is read support only. Blob storage IO, descriptor parsing, merge limits, concurrency, and DLF data-token refresh remain in Rust core. Null map keys cannot be represented by Arrow maps and return an error, matching PyPaimon's Arrow reader.
Go usage
Use
OpenBlobfor large values. Pass descriptors for small values to oneReadBlobscall to use range merging.Verification
cargo test -p paimon --lib blob: 128 passedcargo clippy -p paimon --all-targets -- -D warningscargo build -p paimon-c --releaseNewReadBuilderWithOptions→ Rust decoder → C Data → native Go map →ReadBlobsgo vet ./...morax_test: PyPaimon wrote aMAP<STRING, BLOB>table; Go traversed descriptors after releasing Arrow records, streamed and checksummed a 10,485,883-byte BLOB, sought a 4 KiB range, and batch-read multi-file, empty, and duplicate descriptors