Skip to content

feat(blob): support MAP<K, BLOB> reads in Go - #784

Merged
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/go-map-blob-reader
Sep 3, 2026
Merged

feat(blob): support MAP<K, BLOB> reads in Go#784
JingsongLi merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:codex/go-map-blob-reader

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • read Java/PyPaimon MAP<K, BLOB> layout as values or descriptors in Rust
  • expose per-read options and MAP<STRING, BLOB> descriptors as a native Go map
  • keep copied keys and descriptors valid after the Arrow record is released
  • reuse the existing BlobReader: OpenBlob streams and seeks; ReadBlobs batches with range merging

NewReadBuilderWithOptions exposes the existing C API and is equivalent to Java/PyPaimon's table.copy(options).newReadBuilder().

This is read support only. Blob storage IO, descriptor parsing, merge limits, concurrency, and DLF data-token refresh remain in Rust core. Null map keys cannot be represented by Arrow maps and return an error, matching PyPaimon's Arrow reader.

Go usage

builder, err := table.NewReadBuilderWithOptions(map[string]string{
    "blob-as-descriptor": "true",
})
// scan/read an Arrow record, then:
descriptors, err := paimon.StringBlobMapDescriptors(record.Column(mapColumn), row)
record.Release()
for key, descriptor := range descriptors {
    if descriptor == nil {
        continue
    }
    stream, err := blobReader.OpenBlob(descriptor)
    // io.Copy(destinationFor(key), stream)
}

Use OpenBlob for large values. Pass descriptors for small values to one ReadBlobs call to use range merging.

Verification

  • cargo test -p paimon --lib blob: 128 passed
  • cargo clippy -p paimon --all-targets -- -D warnings
  • cargo build -p paimon-c --release
  • Go writer-generated table fixture: NewReadBuilderWithOptions → Rust decoder → C Data → native Go map → ReadBlobs
  • focused Go BLOB tests and go vet ./...
  • real DLF E2E in regression morax_test: PyPaimon wrote a MAP<STRING, BLOB> table; Go traversed descriptors after releasing Arrow records, streamed and checksummed a 10,485,883-byte BLOB, sought a 4 KiB range, and batch-read multi-file, empty, and duplicate descriptors

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-map-blob-reader branch from 6fa5e23 to 9e2e62a Compare September 2, 2026 11:19
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title feat(go): add map blob readers feat(blob): support MAP<K, BLOB> reads in Go Sep 2, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-map-blob-reader branch from 9e2e62a to 85a3482 Compare September 2, 2026 11:25
})
.collect::<crate::Result<Vec<_>>>()?
} else {
let payload = read_blob_entry(reader, blob_entry_range(&payload_range)).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reject oversized inline payloads before this read. total_value_length is a u64, but these values are later collected into an Arrow BinaryArray, whose 32-bit offsets panic once the total byte length exceeds i32::MAX. ARRAY already preflights this case; MAP should do the same and also validate the flattened batch total.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cd19225. Inline MAP values are now rejected before the complete entry read, and flattened key/value batches enforce Arrow i32 offset limits. Added sparse >2 GiB regression coverage.

}
}

func TestStringBlobMapDescriptors(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this test cover the actual read path instead of starting from a hand-built Arrow map? As written, it only tests descriptor extraction and BlobReader. A writer-generated fixture read through NewReadBuilderWithOptions would also cover the Rust decoder, C Data conversion, and Go API.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cd19225. The Go test now reads a PyPaimon-generated table through NewReadBuilderWithOptions, Rust decoding, C Data import, Go map extraction, and ReadBlobs.

Comment thread docs/src/go-binding.md Outdated
"dlf.oss-endpoint": os.Getenv("DLF_OSS_ENDPOINT"),
"dlf.token-loader": "ecs",
"dlf.token-ecs-role-name": os.Getenv("DLF_ECS_ROLE"),
"dlf.access-key-id": os.Getenv("DLF_ACCESS_KEY_ID"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes the DLF example from the ECS RAM-role loader to static credentials, which is unrelated to MAP<K, BLOB> support and drops automatic credential refresh. Please revert it here or move it to a separate PR with its own rationale.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cd19225. Restored the ECS RAM-role loader example with dlf.token-loader and dlf.token-ecs-role-name.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-map-blob-reader branch 2 times, most recently from cd19225 to 5ca3aad Compare September 2, 2026 16:08
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/go-map-blob-reader branch from 5ca3aad to 1b3e7b7 Compare September 2, 2026 16:30

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit 6d1f8ea into apache:main Sep 3, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants