osdc/hf-cache: cap rclone Go heap with GOMEMLIMIT - #892
Merged
Conversation
huydhn
force-pushed
the
hf-cache-rclone-gomemlimit
branch
from
July 10, 2026 00:36
f6e5d39 to
3f2c51d
Compare
huydhn
marked this pull request as ready for review
July 10, 2026 00:41
jeanschmidt
approved these changes
Jul 10, 2026
huydhn
force-pushed
the
hf-cache-rclone-gomemlimit
branch
from
July 10, 2026 00:48
3f2c51d to
6c176b2
Compare
rclone is a Go binary; its OOMs on high-concurrency nodes are a GC problem, not a genuine memory requirement. Under bursty --vfs-cache-mode full buffering the Go heap grows past the container's cgroup limit before the GC runs, so the kernel OOM-kills the process and the /mnt/hf_cache FUSE mount dies node-wide (dead mount → job-pod CreateContainerError). Give the runtime a soft heap ceiling (GOMEMLIMIT) at ~90% of the reserved limit so the GC reclaims buffering before the kernel does — no limit changes. deploy.sh derives GOMEMLIMIT from each tier's memory (the value it already renders for __RCLONE_MEMORY_LIMIT__) and injects it as a literal env, so: - no runtime shell arithmetic (a bytes*90 intermediate overflows 32-bit busybox); - no downward-API indirection to resolve at runtime; - the value is a literal you can read straight out of the rendered manifest. Go needs MiB/GiB suffixes (it rejects k8s Mi/Gi), so the value is converted, not reused verbatim. Deploy: redeploy hf-cache.
huydhn
force-pushed
the
hf-cache-rclone-gomemlimit
branch
from
July 10, 2026 00:59
d35f0ae to
a5d72c9
Compare
The two hf-cache jobs (test-hf-cache-large-read, test-hf-cache-concurrent-sync) installed unpinned `torch`, so a new stable release could shift the test under us; the three pypi-cache jobs were pinned to 2.11.0. Pin all eight install sites (pip + uv) to torch==2.12.1 for a consistent, reproducible version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rclone is a Go binary, and its OOMs are a GC problem, not a genuine memory requirement: under bursty
--vfs-cache-mode fullbuffering the Go heap grows past the cgroup limit before the GC runs, so the kernel OOM-kills rclone and the/mnt/hf_cacheFUSE mount dies node-wide (→ job-podCreateContainerError).What
Give the Go runtime a soft heap ceiling (
GOMEMLIMIT) at ~90% of the reserved limit so the GC reclaims buffering before the kernel does — no limit changes.deploy.shderivesGOMEMLIMITfrom each tier's memory (the value it already renders for__RCLONE_MEMORY_LIMIT__) and injects it as a literal env value:So there's:
bytes * 90intermediate overflows 32-bit busybox);Go needs
MiB/GiBsuffixes (it rejects k8sMi/Gi), so the value is converted indeploy.sh, not reused verbatim.Per tier: 256Mi→230MiB, 512Mi→460MiB, 1Gi→921MiB, 2Gi→1843MiB, 4Gi→3686MiB.
Verification
just lint13/13,just testpass.deploy.sh's logic and validated withkubeconform -strict; confirmed the manifest carries a literalGOMEMLIMIT(noresourceFieldRef) that sits below the container limit.