Skip to content

Revert #907 + #913: restore pre-#907 hf-cache rclone config - #915

Closed
huydhn wants to merge 2 commits into
pytorch:mainfrom
huydhn:revert-913-use-mmap
Closed

Revert #907 + #913: restore pre-#907 hf-cache rclone config#915
huydhn wants to merge 2 commits into
pytorch:mainfrom
huydhn:revert-913-use-mmap

Conversation

@huydhn

@huydhn huydhn commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Reverts #913 and #907, returning the hf-cache rclone mount to its pre-#907 config (no --use-mmap, no --buffer-size 4M — rclone defaults + the per-tier GOMEMLIMIT). This is the same config meta-prod-aws-ue1 runs today with zero small-tier OOMs.

#907 added --buffer-size 4M --use-mmap and #913 later dropped only --use-mmap; post-deploy verification on ue2 showed small-tier (256Mi CPU / 512Mi gpu1) OOMs continued regardless, so neither flag was the driver. The real cause is that rclone mount memory is tiered by GPU count while per-node RSS scales with pod density — ue2 packs runners onto 48xl/metal nodes, overwhelming the single 256Mi per-node mount. Reverting to the known-good baseline while the tiering fix is developed separately.

@huydhn
huydhn requested a review from jeanschmidt as a code owner July 17, 2026 16:43
@huydhn huydhn changed the title Revert #913: re-add --use-mmap to hf-cache rclone mount Revert #907 + #913: restore pre-#907 hf-cache rclone config Jul 17, 2026
@jeanschmidt

Copy link
Copy Markdown
Contributor

I believe we should keep the -buffer-size 4M anyways

@huydhn

huydhn commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Superseding this in favor of #917, which keeps a per-tier --buffer-size (CPU 0 / GPU 4M) rather than reverting to rclone defaults. Post-deploy evidence showed the OOM driver is node pod-density on the small tiers, not the mmap/buffer flags per se, so a full revert doesn't address it.

@huydhn huydhn closed this Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants