Repository navigation
hardware: size grace gpus by their hbm and offload moe experts - #71
Open
DevelopmentCats wants to merge 1 commit into
Open
DevelopmentCats wants to merge 1 commit into
DevelopmentCats wants to merge 1 commit into
Conversation
GH200, GB200 and GB300 were sized as one pool with system memory, which promised a GH200 about 480 GB for a 96 GB GPU and capped --gpu-memory-utilization as on a DGX Spark. In the driver's default memory mode CUDA allocates only from the HBM, so the budget is now the HBM. The Grace CPU's memory becomes a second tier for mixture-of-experts models that do not fit the HBM. inspect and the panel still recommend the largest copy that fits on the GPU, and only when none does, the largest that fits with offload. new adds the engine's offload settings from its catalogue definition: vLLM gets --cpu-offload-gb and --cpu-offload-params experts with pinning off, as NVIDIA's DGX Station recipe sets; llama.cpp places experts itself and gets a note. A DGX Station also has a display GPU, and DGX OS names the GB300 only in /etc/environment, which services do not read. On a machine whose GPUs are different models, setup records the GPUs with the most memory in config.env by UUID and doctor warns until it is set. The DGX Station report is synthetic; none of this has run on a GB300 yet.
There was a problem hiding this comment.
🟡 Changes recommended
Offload fit can be reported even when no eligible model variant can load, and CLI/web output can hide the selected variant’s CPU-memory requirement.
4 open issues
What changed in this PR
This PR sizes Grace Hopper and Grace Blackwell GPUs against their own HBM, adding CPU-memory offload recommendations for oversized mixture-of-experts models within llm-helper’s hardware and profile planning.
Changes:
- Separates GPU weight budgets from coherent CPU-memory offload pools.
- Adds engine-defined offload settings, profile planning, and CLI/panel explanations.
- Pins the largest GPU model on mixed-GPU machines and adds tests, catalogue data, and documentation.
| File | Description |
|---|---|
| internal/web/static/models.js | Displays offload verdicts and variant labels. |
| internal/web/static/machine.js | Shows CPU offload capacity in hardware details. |
| internal/resolve/resolve.go | Ranks offload candidates and applies profile settings. |
| internal/resolve/offload.go | Plans expert offload and checks flag support. |
| internal/resolve/offload_internal_test.go | Tests ranking and engine-specific offload planning. |
| internal/hardware/hardware.go | Separates coherent memory from unified memory. |
| internal/hardware/hardware_test.go | Updates coherent hardware classification expectations. |
| internal/engine/registry.go | Adds offload settings to the provider interface. |
| internal/engine/engine.go | Parses, validates, and expands offload definitions. |
| internal/engine/engine_test.go | Tests offload definitions and expansion. |
| internal/detect/hardware.go | Exposes offload capacity and GPU pinning details. |
| internal/detect/hardware_internal_test.go | Tests Grace GPU memory classification. |
| internal/detect/coherent_internal_test.go | Tests coherent budgets, offload sizing, and GPU selection. |
| internal/detect/classify.go | Identifies UUIDs for the largest GPU model. |
| internal/detect/budget.go | Calculates separate CPU offload pools and amounts. |
| internal/cli/setup.go | Reports offload capacity and saves GPU pins. |
| internal/cli/search.go | Explains offload fits in model inspection. |
| internal/cli/gpupin_internal_test.go | Tests setup pinning and CLI explanations. |
| internal/cli/detectcmds.go | Adds mixed-GPU warnings and memory-pool descriptions. |
| docs/providers.md | Documents engine offload definitions. |
| docs/memory.md | Explains Grace memory budgets and offload behavior. |
| docs/hardware.md | Updates coherent-memory sizing documentation. |
| CHANGELOG.md | Records offload, sizing, and GPU-pinning changes. |
| catalogue/hardware/reports/dgx-station-gb300.json | Adds an explicitly synthetic GB300 report. |
| catalogue/hardware/machines/nvidia_grace_hopper.json | Updates the memory-sizing rationale. |
| catalogue/hardware/machines/nvidia_grace_blackwell.json | Updates the memory-sizing rationale. |
| catalogue/engines/vllm.json | Declares expert-offload flags and environment settings. |
| catalogue/engines/llamacpp.json | Declares automatic offload behavior. |
Review effort: Balanced
Comment on lines
+360
to
+363
| if best, ok := w.BestFit(d.BudgetGB+d.OffloadGB, ggufOK); ok && many { | ||
| d.Best = &best | ||
| size = best.SizeGB | ||
| } |
| d.Best.SizeGB, display(*d.Best), d.Variants, d.BudgetGB, d.SizeGB) | ||
| case *d.FitsBudget: | ||
| fmt.Printf(" size: %.1f GB (fits under the ~%.0fGB budget%s)\n", d.SizeGB, d.BudgetGB, of) | ||
| case d.FitsWithOffload != nil && *d.FitsWithOffload: |
| fits := w.SizeGB <= d.BudgetGB | ||
| d.FitsBudget = &fits | ||
| } | ||
| d.MoE = arch.MoE() |
| `Under this machine's ~${Math.round(d.budget_gb)} GB budget${of}.`, | ||
| ); | ||
| } | ||
| } else if (d.fits_with_offload) { |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Prepares llm-helper for the DGX Station GB300 (Dell Pro Max with GB300): 252 GB HBM3e on the GPU, 496 GB LPDDR5X on the Grace CPU, joined by NVLink-C2C, plus a small RTX PRO 2000 display GPU.
Why
On Grace Hopper and Grace Blackwell the GPU's memory and the CPU's memory are coherent but separate. With the default CDMM driver mode,
cudaMalloconly allocates in HBM, so sizing a model against HBM + LPDDR5X (asunifieddid) tells someone a 400 GB model fits when vLLM will OOM. Huge MoE models (DeepSeek R1 class) can still run by keeping their experts in CPU memory, which vLLM supports with--cpu-offload-gb N --cpu-offload-params experts(NVIDIA's Station recipe uses exactly this).What changes
coherentmachines are no longer treated as unified. The budget is the GPU's own memory; the CPU memory becomes a separate offload pool (RAM minus baseline minus an 8 GiB host reserve, with a 4 GiB margin), only when the GPU is coherent and CUDA is present.catalogue/engines/vllm.jsondeclares anoffloadblock (args with{offload_gib},VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1, and awhy).llamacpp.jsondeclaresautomatic: true(its--fitalready moves experts to CPU). No engine names in Go.inspect,doctor,setupand the panel show "fits with ~N GB of its experts in CPU memory".new/ panel profile creation adds the offload args and env when needed, checks the installed vLLM accepts the flags (otherwise prints a note only), and reduces the in-HBM weights by the offloaded amount before planning KV cache.setupwritesCUDA_VISIBLE_DEVICESintoconfig.env(and fills it into a kept config only if missing, never replacing a value). DGX OS only sets it in/etc/environment, which systemd user services and containers don't inherit.doctorwarns when the shell and server disagree.--tensor-parallel-sizeis only suggested for identical GPUs.whyfor the Grace Blackwell / Grace Hopper architectures, and adgx-station-gb300report.docs/memory.md, linked from hardware and providers pages. CHANGELOG entries under Unreleased.Behavior change for GH200 / GB200
These were previously sized as unified. They now get the HBM budget plus the offload tier. The docs note that machines on older drivers in NUMA mode behave differently.
Verification
gofmt,go vet ./...,go test -race ./..., gendocs (no diff), builds for linux/amd64, linux/arm64, darwin/arm64,tools/recipediffvet+test, prettier 3.9.6: all pass.hardware report,doctor, andinspectfor Qwen3-30B-A3B, DeepSeek-R1 and unsloth/DeepSeek-R1-GGUF produce identical text; the only JSON difference is a new"moe": truefield. GB10 is unified and unaffected. No launches; the serving model stayed healthy throughout.Not verified
dgx-station-gb300report is synthetic (built from published specs and nvidia-smi values), and must be replaced with a measured one.GB300 test plan
Read-only first:
llm-helper hardware reportandllm-helper doctorecho $CUDA_VISIBLE_DEVICES,systemctl --user show-environmentnvidia-smi --query-gpu=name,memory.total,compute_cap,uuid --format=csvgrep CoherentGPUMemoryMode /proc/driver/nvidia/paramstorch.cuda.get_device_name(0),mem_get_info(), capabilityThen, with OK to load models:
6. One in-HBM model (DeepSeek-V4-Flash) — confirm no offload is added.
7. One offload model (DeepSeek-R1 NVFP4 or GLM NVFP4) — check the log for
Total CPU offloaded parameters, record tok/s.8. Replace the synthetic report with the measured one.
Out of scope
GB300 recipe, MIG, NVL72, verifying offload after launch,
--moe-cache-mibtuning.