Native RolloutRecord schema and RolloutStore for RL rollout storage#124
Merged
Conversation
added 2 commits
July 8, 2026 16:56
Add an independent first-class RolloutRecord schema and RolloutStore for RL rollout storage, per specs/rollout-schema-design.md. RolloutStore is a separate Lance dataset with its own Arrow schema following MAI Snowflake rolloutdb conventions; the existing ContextRecord path is untouched. Artifacts are stored as their own rows (role="artifact") with the binary_payload column physically offloaded via blob v2, so column scans skip the bytes. RolloutStore::get_blob materializes payloads on demand via BlobHandling::AllBinary. Native versioning, blob offload, and the relationship graph are reused from the schema-agnostic layer (store.rs helpers promoted to pub(crate)). Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>
Co-Authored-By: Beinan Wang <beinanwang@microsoft.com>
4 tasks
beinan
added a commit
that referenced
this pull request
Jul 9, 2026
## Summary - Extends the native `RolloutRecord` / `RolloutStore` (landed in #124) across the remote store layer, mirroring the existing `ContextRecord` remote path. Purely additive — the `ContextRecord` path is untouched. - Single `POST /api/v1/rollouts/{name}/records` endpoint accepts either encoding, chosen by `Content-Type`: `application/json` inlines `binary_payload` as base64; `multipart/form-data` sends a `metadata` part (records array, payloads null) followed by one raw binary part per record-with-a-blob, named by zero-based index. Avoids ~33% base64 inflation and full-blob buffering for large artifacts. - Unified `RolloutStore` enum (Local / Remote) behind the `remote` feature, mirroring `ContextStore`. ## Multipart strict-reject A rollout append feeds training and is atomic, so the whole request is rejected with `400` before anything is written when: the first part is not `metadata`; a part name is not a valid record index (unparseable / out of bounds); an index is duplicated; a byte length disagrees with the record's `payload_size`; or a record declares `payload_size` but no binary part is supplied. ## Notes - `payload_checksum` is a caller-supplied opaque string, stored as-is (no server-side hashing — no hashing dep in the workspace). - Rollout datasets use a `.rollout.lance` suffix so a rollout store and a context store can share a name on disk. ## Test plan - [x] `cargo test --all-features` — api 6, core 102 (+1 ignored), server 26 - [x] `cargo clippy --all-features --all-targets` — clean - [x] Multipart attach-by-index round-trip + all strict-reject cases - [x] JSON inline base64 payload offloaded via blob v2, materialized via blob endpoint Co-Authored-By: Beinan Wang <beinanwang@microsoft.com> --------- Co-authored-by: Beinan Wang <beinanwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RolloutRecordschema andRolloutStorefor RL rollout storage, perspecs/rollout-schema-design.md(Design doc: native RolloutDB schema for lance-context #123).RolloutStoreis a separate Lance dataset with its own Arrow schema (MAI Snowflake rolloutdb column conventions). The existingContextRecordpath is untouched — purely additive.role="artifact") withbinary_payloadoffloaded via blob v2;get_blobmaterializes bytes on demand viaBlobHandling::AllBinary. Native versioning, blob offload, and the relationship graph are shared from the schema-agnostic layer (store.rs helpers promoted topub(crate)).Test plan
cargo test -p lance-context-core— 102 passed, 0 failed, 1 ignoredcargo clippy --all-targets— clean