Skip to content

Add bounded host-backed storage for broker files - #1770

Open
Weiteng Chen (CvvT) wants to merge 12 commits into
uliteboxfrom
weiteng/broker-file-storage
Open

Weiteng Chen (CvvT) wants to merge 12 commits into
uliteboxfrom
weiteng/broker-file-storage

Conversation

@CvvT

@CvvT Weiteng Chen (CvvT) commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds storage that lets a file's contents live in host shared memory, so that later PRs can map broker files into runners with MAP_SHARED. In-mem fs supports this by moving a file's contents into a SharedFileMemory the first time it's called and returns the host object as a SharedFile. From then on, reads, writes and truncations all use that storage, and writes behave like pwrite.

The userland broker provides the storage on each host. Linux uses a memfd while Windows uses a delete-on-close temporary file. Windows can't shrink a file while it's mapped, so the broker tracks the file's size itself, zeroes the bytes a refused shrink cuts off, and shrinks the host file once nothing maps it.

One limitation is that we need one host descriptor or handle per shared file and the total number of host fd is limited. Additionally, because we don't know when the shared file is no longer needed, we never close it.

@CvvT
Weiteng Chen (CvvT) marked this pull request as draft November 1, 2026 09:07
Base automatically changed from weiteng/broker-fork-socket to ulitebox November 1, 2026 20:49
@CvvT
Weiteng Chen (CvvT) force-pushed the weiteng/broker-file-storage branch from 79df568 to 1a42125 Compare November 2, 2026 18:05
Weiteng Chen (CvvT) and others added 12 commits November 3, 2026 11:05
Prepare lazy shared-file storage and Linux/Windows host providers without exposing a file-mapping protocol endpoint or changing guest mmap behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Grow shared file memory on writes like pwrite, reporting short writes, and report exhausted storage as NoSpace rather than an I/O error. Document that each storage call is atomic so backends access it without holding their own locks, centralize storage error conversions, and return the descriptor's writability with the shared file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only sections derived from the handle reach runners, and their protection and the
rights of the delivered section handle already enforce read-only access, so the
read-only and writable duplicates cost two handles per file for nothing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Linux, make a reader's read-only reopen of a memfd for each export and close
it once delivered, so each file in host shared memory holds one descriptor.
HostHandle may now own the handle it exports.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add BrokerCoreLimits::max_shared_files, which the userland broker applies to its
in-memory file system, so the limit is set with the broker's other limits. Zero
keeps every file in broker memory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Windows refuses to shrink a file while any section of it exists, and runners
hold sections for as long as they map the file, so truncating a mapped file
failed with EBUSY. Track the file's size in the broker instead: a shrink the
host refuses zeroes the bytes past the new end and lowers the size only, and a
later resize shrinks the host file once nothing maps it. Bytes past the end
that runners write through their mappings are zeroed before the file grows
over them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The code says what it does: map the layer that holds the file, as reads do.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A handle that shared file memory exports is now a boxed trait object that knows
its own raw value, whether the memory lends one of its own handles or makes one
for the export, as BorrowedFd and OwnedFd do. This drops the separate raw value,
the optional owner, and the two unsafe constructors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the host-file storage and the Linux and Windows modules of shared_objects.rs
into shared_objects/, unchanged, as the runner module is laid out.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
HostFileMemory keeps the shared read and write logic in host_file.rs and calls
methods that linux.rs and windows.rs each implement on it: the size, resizing,
bookkeeping around writes, and exporting handles. Windows's logical size and the
error codes of each host move to its module.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove tests that restate error-mapping match tables, drop mapped-file tests covered elsewhere, and fold the append/truncate-on-open checks into the coherence test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Match once on in-memory file contents instead of checking for shared
  memory and then reaching inline bytes that cannot be shared.
- Loop over direct reads and writes, staging a chunk only when the rest of
  a buffer cannot take a direct transfer; direct read_at and write_at take
  a buffer offset for that.
- Stop zeroing bytes past the end of mapped Windows files on growth; a
  refused shrink still zeroes the tail it cuts off.
- Drop the memfd mode change and the F_SEAL_SEAL seal: runners' seccomp
  filter allows neither open, fchmod, nor F_ADD_SEALS.
- Restore the empty-write check for unmapped files, and drop the static
  backing data test, which the contents enum already guarantees.
- Trim shared file storage docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@CvvT
Weiteng Chen (CvvT) force-pushed the weiteng/broker-file-storage branch from 43bb28b to 8222a0d Compare November 3, 2026 22:17
@CvvT
Weiteng Chen (CvvT) marked this pull request as ready for review November 3, 2026 22:54
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

🤖 SemverChecks 🤖 ⚠️ Potential breaking API changes detected ⚠️

Click for details
--- failure enum_no_repr_variant_discriminant_changed: enum variant had its discriminant change value ---

Description:
The enum's variant had its discriminant value change. This breaks downstream code that used its value via a numeric cast like `as isize`.
        ref: https://doc.rust-lang.org/reference/items/enumerations.html#assigning-discriminant-values
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_no_repr_variant_discriminant_changed.ron

Failed in:
  variant TruncateError::Io 5 -> 7 in /home/runner/work/litebox/litebox/litebox/src/fs/errors.rs:120

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_added.ron

Failed in:
  variant TruncateError:Busy in /home/runner/work/litebox/litebox/litebox/src/fs/errors.rs:116
  variant TruncateError:NoSpace in /home/runner/work/litebox/litebox/litebox/src/fs/errors.rs:118

--- failure enum_no_repr_variant_discriminant_changed: enum variant had its discriminant change value ---

Description:
The enum's variant had its discriminant value change. This breaks downstream code that used its value via a numeric cast like `as isize`.
        ref: https://doc.rust-lang.org/reference/items/enumerations.html#assigning-discriminant-values
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_no_repr_variant_discriminant_changed.ron

Failed in:
  variant TruncateError::Io 5 -> 7 in /home/runner/work/litebox/litebox/litebox_broker_core/src/fs/errors.rs:125

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_added.ron

Failed in:
  variant TruncateError:Busy in /home/runner/work/litebox/litebox/litebox_broker_core/src/fs/errors.rs:121
  variant TruncateError:NoSpace in /home/runner/work/litebox/litebox/litebox_broker_core/src/fs/errors.rs:123
  variant WriteError:NoSpace in /home/runner/work/litebox/litebox/litebox_broker_core/src/fs/errors.rs:77

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant