osdc/hf-cache: raise 1-GPU rclone reserve to 1Gi - #890
Closed
huydhn wants to merge 1 commit into
Closed
Conversation
The 1-GPU tier reserved only 512Mi for rclone (pytorch#883). rclone RSS under `--vfs-cache-mode full` is driven by the model files a job reads, not by GPU count, so a single-GPU inductor job OOM-kills rclone at 512Mi, dropping the node-wide `/mnt/hf_cache` FUSE mount → job pods fail with `CreateContainerError`. Observed on a g5.8xlarge (1x a10g): https://github.com/pytorch/pytorch/actions/runs/29042231396/job/86206746830 Raise the `hf-cache-mount-gpu1` tier 512Mi → 1Gi (the previously-working pre-pytorch#883 floor). The reserve is `request == limit`, so trim the untrimmed 1-GPU runner defs to keep them schedulable (`analyze-utilization` confirms 1 pod/node): a10g/l4 (g5/g6.8xlarge) 113→112Gi, t4 (g4dn.8xlarge) 115→114Gi. Labels keep their `-NNN-`. Packing overhead (daemonset_overhead) tier updated. Deploy: redeploy hf-cache + arc-runners (+arc-runners-opt) to the clusters running these 1-GPU classes.
Contributor
|
lets pause on this, as I believe this can be a problem to CPU on -opt fleet |
jeanschmidt
approved these changes
Jul 9, 2026
jeanschmidt
requested changes
Jul 9, 2026
jeanschmidt
left a comment
jeanschmidt
left a comment
Contributor
There was a problem hiding this comment.
we should reduce the memory usage, not raise memory for the pod
Contributor
|
we can SUBSTANTIALLY reduce memory usage, without significant performance drop, potentially to 128Mib. rclone is a go binary, and we can easily control go GC behavior and memory footprint without requiring infinite memory. The problem here is GC, NOT how much memory the process requires. few knobs:
|
Contributor
|
with this, we can probably get rid of the stagger multi-size memory we set up for GPUs |
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 10, 2026
rclone is a Go binary. Its OOMs on high-concurrency GPU nodes are a GC problem, not a genuine memory requirement: under bursty `--vfs-cache-mode full` buffering the Go heap grows past the container's cgroup limit before the GC runs, and the kernel OOM-kills the process — dropping the `/mnt/hf_cache` FUSE mount node-wide (dead mount → job-pod CreateContainerError). Give the runtime a soft memory ceiling at 90% of the container limit so the GC reclaims buffering before the kernel intervenes. The limit is injected in bytes via the downward API (`RCLONE_MEMORY_LIMIT_BYTES` ← `limits.memory`) and the launch script exports `GOMEMLIMIT` before starting rclone (Go reads it once at startup), so it tracks whatever per-tier reservation deploy.sh renders — no new placeholder, no hard-coded value. This targets the OOM at the *current* per-tier limits (256Mi–4Gi) rather than raising them (cf. pytorch#890). Follow-up levers Jean noted, to validate against the deployed rclone version before adding: `--buffer-size` (per-file buffer) and `--max-buffer-memory` (global buffer cap). Deploy: redeploy hf-cache.
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 10, 2026
rclone is a Go binary. Its OOMs on high-concurrency GPU nodes are a GC problem, not a genuine memory requirement: under bursty `--vfs-cache-mode full` buffering the Go heap grows past the container's cgroup limit before the GC runs, and the kernel OOM-kills the process — dropping the `/mnt/hf_cache` FUSE mount node-wide (dead mount → job-pod CreateContainerError). Give the runtime a soft memory ceiling at 90% of the container limit so the GC reclaims buffering before the kernel intervenes. The limit is injected in bytes via the downward API (`RCLONE_MEMORY_LIMIT_BYTES` ← `limits.memory`) and the launch script exports `GOMEMLIMIT` before starting rclone (Go reads it once at startup), so it tracks whatever per-tier reservation deploy.sh renders — no new placeholder, no hard-coded value. This targets the OOM at the *current* per-tier limits (256Mi–4Gi) rather than raising them (cf. pytorch#890). Follow-up levers Jean noted, to validate against the deployed rclone version before adding: `--buffer-size` (per-file buffer) and `--max-buffer-memory` (global buffer cap). Deploy: redeploy hf-cache.
Contributor
Author
|
Closing this as I haven't seen OOM for a couple of days, so I would assume that the GOMEMLIMIT fix does the trick |
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 16, 2026
…ch#907) ## Problem On the 1-GPU mount tier (512Mi), concurrent large-model (safetensors) reads OOM-kill rclone → the node-wide `/mnt/hf_cache` FUSE mount dies → the job's mmap'd read faults with **SIGBUS**, surfacing as a misleading `LocalEntryNotFoundError` ("missing weights"). Example — failing `inductor_huggingface` a10g job: ``` Loading weights: 25%|██▍| 36/146 … Run failed with return code: -7 # SIGBUS huggingface_hub.errors.LocalEntryNotFoundError: Cannot find the requested files in the disk cache ... ``` Stats — live rclone `OOMKilled` mounts, **gpu1 tier only** (cpu/gpu4/gpu8 = 0): ue2 **17/59**, ue1 **9/80**. ## Fix rclone RSS under `--vfs-cache-mode full` is dominated by per-open-file read-ahead buffers, not GPU count: - `--buffer-size 4M` — 4x cut from rclone's 16Mi-per-open-file default read-ahead (the RSS driver when a model opens many shards); kept non-zero so cold reads retain some prefetch, with the on-disk cache serving the rest. - `--use-mmap` — return freed buffers to the OS instead of Go retaining them as RSS. Keeps rclone inside the reserve without raising the limit. Complements pytorch#890 (512Mi→1Gi headroom).
Contributor
Author
|
Not need this anymore |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Raise the
hf-cache-mount-gpu1rclone memory tier 512Mi → 1Gi, and trim the three single-GPU runner defs (arc-runners+arc-runners-opt) to fit under the largerrequest == limitreserve.Why
#883 tiered rclone memory by GPU count and gave the 1-GPU tier 512Mi. But under
--vfs-cache-mode full, rclone RSS tracks the model files a job reads, not GPU count — and 1-GPU a10g nodes run theinductor_huggingfacesuite, which loads the largest models. rclone crosses 512Mi → OOMKilled → the node-wide/mnt/hf_cacheFUSE mount dies → the job's mmap'd safetensors read faults with SIGBUS (Run failed with return code: -7), which surfaces as a misleadingLocalEntryNotFoundError("missing weights"). 1Gi restores the floor these nodes ran at before #883.Evidence
Live rclone OOMs, both prod clusters, gpu1 tier only (cpu/gpu4/gpu8 = 0):
OOMKilledOOMKilledFlaky because the OOM only trips when rclone crosses 512Mi mid-load — same job, same ue1 bucket:
Loading weights: 25%|…| 36/146 … return code: -7)The file isn't missing (the load reached 25% of 146 shards); the
LocalEntryNotFoundErroris a retry cascade after the SIGBUS.Runner trims (label keeps
-NNN-)g5.8xlarge/ l4g6.8xlarge: 113 → 112Gig4dn.8xlarge: 115 → 114GiDeploy
Redeploy
hf-cache+arc-runners(+arc-runners-opt) to the clusters running these 1-GPU classes.