Revert #907 + #913: restore pre-#907 hf-cache rclone config - #915
Closed
huydhn wants to merge 2 commits into
Closed
Conversation
…ytorch#913)" This reverts commit 0772508.
pytorch#907)" This reverts commit 76fd948.
Contributor
|
I believe we should keep the |
Contributor
Author
|
Superseding this in favor of #917, which keeps a per-tier |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #913 and #907, returning the hf-cache rclone mount to its pre-#907 config (no
--use-mmap, no--buffer-size 4M— rclone defaults + the per-tierGOMEMLIMIT). This is the same configmeta-prod-aws-ue1runs today with zero small-tier OOMs.#907 added
--buffer-size 4M --use-mmapand #913 later dropped only--use-mmap; post-deploy verification on ue2 showed small-tier (256Mi CPU / 512Mi gpu1) OOMs continued regardless, so neither flag was the driver. The real cause is that rclone mount memory is tiered by GPU count while per-node RSS scales with pod density — ue2 packs runners onto 48xl/metal nodes, overwhelming the single 256Mi per-node mount. Reverting to the known-good baseline while the tiering fix is developed separately.