osdc/hf-cache: cut rclone RSS with --buffer-size 4M --use-mmap - #907
Merged
Conversation
The 1-GPU mount tier (512Mi) OOM-kills rclone during concurrent large-model (safetensors) reads; the dead node-wide FUSE mount then faults the job's mmap'd read with SIGBUS, surfacing as a spurious LocalEntryNotFoundError. rclone RSS under --vfs-cache-mode full is dominated by per-open-file read-ahead buffers, not GPU count. --buffer-size 0 drops that read-ahead (redundant — reads already come from the on-disk VFS cache), and --use-mmap returns freed buffers to the OS instead of Go holding them as RSS. This keeps rclone inside the reserve without raising the limit, and complements the tier bump in pytorch#890.
A 4x cut from rclone's 16Mi-per-open-file default caps the RSS driver while leaving a small prefetch so cold model reads don't lose throughput.
huydhn
changed the title
jeanschmidt
approved these changes
Jul 16, 2026
Contributor
Author
|
Deploy on ue1 staging, smoke and integration tests are passing |
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 17, 2026
Remove --use-mmap from the rclone mount, keeping the --buffer-size 4M read-ahead cut. Both were added together in pytorch#907; we're rolling out one lever at a time, starting with the buffer-size reduction.
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 17, 2026
pytorch#907)" This reverts commit 76fd948.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the 1-GPU mount tier (512Mi), concurrent large-model (safetensors) reads OOM-kill rclone → the node-wide
/mnt/hf_cacheFUSE mount dies → the job's mmap'd read faults with SIGBUS, surfacing as a misleadingLocalEntryNotFoundError("missing weights").Example — failing
inductor_huggingfacea10g job:Stats — live rclone
OOMKilledmounts, gpu1 tier only (cpu/gpu4/gpu8 = 0): ue2 17/59, ue1 9/80.Fix
rclone RSS under
--vfs-cache-mode fullis dominated by per-open-file read-ahead buffers, not GPU count:--buffer-size 4M— 4x cut from rclone's 16Mi-per-open-file default read-ahead (the RSS driver when a model opens many shards); kept non-zero so cold reads retain some prefetch, with the on-disk cache serving the rest.--use-mmap— return freed buffers to the OS instead of Go retaining them as RSS.Keeps rclone inside the reserve without raising the limit. Complements #890 (512Mi→1Gi headroom).