osdc/hf-cache: per-tier rclone --buffer-size (CPU 0, gpu1 2M)

[huydhn]

Makes rclone --buffer-size a per-tier value in deploy.sh instead of one global flag. Read-ahead RAM scales as (concurrent open shards) × buffer-size, and peak concurrency scales with a node's pod density, so the tight small tiers need less read-ahead:

• CPU catch-all (256Mi) — densely packed (many runner pods per node share one mount) → --buffer-size 0 (read-ahead off; served from the --vfs-cache-mode full on-disk cache).
• 1-GPU tier (512Mi) → --buffer-size 2M (a single HF model opening ~100+ shards at 4M overruns the 512Mi reserve; observed OOMs on g5.16xlarge gpu1 nodes).
• multi-GPU tiers (1–4Gi) → --buffer-size 4M (roomier reserve, keep more prefetch).

Addresses the OOM driver on both small tiers via buffer sizing. Note this is buffer-size only; the deeper fix (mount reserve sized to pod density) can follow, and #876 adds self-heal so any residual OOM recovers instead of wedging.

[jeanschmidt]

do we know that --buffer-size is the culprit here? If it was, it should have been fixed when we lowered to 4MiB, correct?

[huydhn]

do we know that --buffer-size is the culprit here? If it was, it should have been fixed when we lowered to 4MiB, correct?

There is no single culprit, as both memory limit and the size of of nodes are involved. Lower buffer size does help reduce the memory footprint, but in a bigger node with multiple pods, it's not enough. 4MB is not low enough for CPU nodes that only has 256MB granted to rclone. That's why you see OOM crash concentrated on CPU nodes, not GPU ones which allocate more memory to rclone

[jeanschmidt]

yaeah, but the question is that this knob is substantial or marginal. But maybe worth releasing any way.