Skip to content

hardware: size grace gpus by their hbm and offload moe experts - #71

Open
DevelopmentCats wants to merge 1 commit into
mainfrom
devcats/gb300-prep
Open

DevelopmentCats wants to merge 1 commit into
mainfrom
devcats/gb300-prep

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Prepares llm-helper for the DGX Station GB300 (Dell Pro Max with GB300): 252 GB HBM3e on the GPU, 496 GB LPDDR5X on the Grace CPU, joined by NVLink-C2C, plus a small RTX PRO 2000 display GPU.

Why

On Grace Hopper and Grace Blackwell the GPU's memory and the CPU's memory are coherent but separate. With the default CDMM driver mode, cudaMalloc only allocates in HBM, so sizing a model against HBM + LPDDR5X (as unified did) tells someone a 400 GB model fits when vLLM will OOM. Huge MoE models (DeepSeek R1 class) can still run by keeping their experts in CPU memory, which vLLM supports with --cpu-offload-gb N --cpu-offload-params experts (NVIDIA's Station recipe uses exactly this).

What changes

  • Memory budget: coherent machines are no longer treated as unified. The budget is the GPU's own memory; the CPU memory becomes a separate offload pool (RAM minus baseline minus an 8 GiB host reserve, with a 4 GiB margin), only when the GPU is coherent and CUDA is present.
  • Offload is engine data: catalogue/engines/vllm.json declares an offload block (args with {offload_gib}, VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1, and a why). llamacpp.json declares automatic: true (its --fit already moves experts to CPU). No engine names in Go.
  • Recommendation rule: the largest variant that fits in HBM wins. Only when none does, a MoE model with an offload pool gets the largest variant that fits with offload. Dense models never get offload. inspect, doctor, setup and the panel show "fits with ~N GB of its experts in CPU memory".
  • Profiles: new / panel profile creation adds the offload args and env when needed, checks the installed vLLM accepts the flags (otherwise prints a note only), and reduces the in-HBM weights by the offloaded amount before planning KV cache.
  • Mixed GPUs: when GPUs differ (GB300 + RTX PRO 2000), detection records the UUIDs of the largest GPU model. setup writes CUDA_VISIBLE_DEVICES into config.env (and fills it into a kept config only if missing, never replacing a value). DGX OS only sets it in /etc/environment, which systemd user services and containers don't inherit. doctor warns when the shell and server disagree. --tensor-parallel-size is only suggested for identical GPUs.
  • Catalogue: rewritten why for the Grace Blackwell / Grace Hopper architectures, and a dgx-station-gb300 report.
  • Docs: new "Grace Hopper and Grace Blackwell" section in docs/memory.md, linked from hardware and providers pages. CHANGELOG entries under Unreleased.

Behavior change for GH200 / GB200

These were previously sized as unified. They now get the HBM budget plus the offload tier. The docs note that machines on older drivers in NUMA mode behave differently.

Verification

  • gofmt, go vet ./..., go test -race ./..., gendocs (no diff), builds for linux/amd64, linux/arm64, darwin/arm64, tools/recipediff vet+test, prettier 3.9.6: all pass.
  • DGX Spark (GB10) regression, read-only: built main and this branch for linux/arm64 and ran both with scratch dirs on a Spark. hardware report, doctor, and inspect for Qwen3-30B-A3B, DeepSeek-R1 and unsloth/DeepSeek-R1-GGUF produce identical text; the only JSON difference is a new "moe": true field. GB10 is unified and unaffected. No launches; the serving model stayed healthy throughout.

Not verified

  • Nothing has run on a GB300. The dgx-station-gb300 report is synthetic (built from published specs and nvidia-smi values), and must be replaced with a measured one.
  • The offload path is untested end to end: the vLLM flags were not checked against a real vLLM's capabilities on the box, and no tok/s has been measured (no public GB300 numbers exist; GH200 expert offload ran 13–26 tok/s).
  • Panel JS changes were not browser-tested.

GB300 test plan

Read-only first:

  1. llm-helper hardware report and llm-helper doctor
  2. echo $CUDA_VISIBLE_DEVICES, systemctl --user show-environment
  3. nvidia-smi --query-gpu=name,memory.total,compute_cap,uuid --format=csv
  4. grep CoherentGPUMemoryMode /proc/driver/nvidia/params
  5. torch: torch.cuda.get_device_name(0), mem_get_info(), capability

Then, with OK to load models:
6. One in-HBM model (DeepSeek-V4-Flash) — confirm no offload is added.
7. One offload model (DeepSeek-R1 NVFP4 or GLM NVFP4) — check the log for Total CPU offloaded parameters, record tok/s.
8. Replace the synthetic report with the measured one.

Out of scope

GB300 recipe, MIG, NVL72, verifying offload after launch, --moe-cache-mib tuning.

GH200, GB200 and GB300 were sized as one pool with system memory, which
promised a GH200 about 480 GB for a 96 GB GPU and capped
--gpu-memory-utilization as on a DGX Spark. In the driver's default memory
mode CUDA allocates only from the HBM, so the budget is now the HBM.

The Grace CPU's memory becomes a second tier for mixture-of-experts models
that do not fit the HBM. inspect and the panel still recommend the largest
copy that fits on the GPU, and only when none does, the largest that fits
with offload. new adds the engine's offload settings from its catalogue
definition: vLLM gets --cpu-offload-gb and --cpu-offload-params experts with
pinning off, as NVIDIA's DGX Station recipe sets; llama.cpp places experts
itself and gets a note.

A DGX Station also has a display GPU, and DGX OS names the GB300 only in
/etc/environment, which services do not read. On a machine whose GPUs are
different models, setup records the GPUs with the most memory in config.env
by UUID and doctor warns until it is set.

The DGX Station report is synthetic; none of this has run on a GB300 yet.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Offload fit can be reported even when no eligible model variant can load, and CLI/web output can hide the selected variant’s CPU-memory requirement.

4 open issues
What changed in this PR

This PR sizes Grace Hopper and Grace Blackwell GPUs against their own HBM, adding CPU-memory offload recommendations for oversized mixture-of-experts models within llm-helper’s hardware and profile planning.

Changes:

  • Separates GPU weight budgets from coherent CPU-memory offload pools.
  • Adds engine-defined offload settings, profile planning, and CLI/panel explanations.
  • Pins the largest GPU model on mixed-GPU machines and adds tests, catalogue data, and documentation.
File Description
internal/​web/​static/​models.js Displays offload verdicts and variant labels.
internal/​web/​static/​machine.js Shows CPU offload capacity in hardware details.
internal/​resolve/​resolve.go Ranks offload candidates and applies profile settings.
internal/​resolve/​offload.go Plans expert offload and checks flag support.
internal/​resolve/​offload_internal_test.go Tests ranking and engine-specific offload planning.
internal/​hardware/​hardware.go Separates coherent memory from unified memory.
internal/​hardware/​hardware_test.go Updates coherent hardware classification expectations.
internal/​engine/​registry.go Adds offload settings to the provider interface.
internal/​engine/​engine.go Parses, validates, and expands offload definitions.
internal/​engine/​engine_test.go Tests offload definitions and expansion.
internal/​detect/​hardware.go Exposes offload capacity and GPU pinning details.
internal/​detect/​hardware_internal_test.go Tests Grace GPU memory classification.
internal/​detect/​coherent_internal_test.go Tests coherent budgets, offload sizing, and GPU selection.
internal/​detect/​classify.go Identifies UUIDs for the largest GPU model.
internal/​detect/​budget.go Calculates separate CPU offload pools and amounts.
internal/​cli/​setup.go Reports offload capacity and saves GPU pins.
internal/​cli/​search.go Explains offload fits in model inspection.
internal/​cli/​gpupin_internal_test.go Tests setup pinning and CLI explanations.
internal/​cli/​detectcmds.go Adds mixed-GPU warnings and memory-pool descriptions.
docs/​providers.md Documents engine offload definitions.
docs/​memory.md Explains Grace memory budgets and offload behavior.
docs/​hardware.md Updates coherent-memory sizing documentation.
CHANGELOG.md Records offload, sizing, and GPU-pinning changes.
catalogue/​hardware/​reports/​dgx-station-gb300.json Adds an explicitly synthetic GB300 report.
catalogue/​hardware/​machines/​nvidia_grace_hopper.json Updates the memory-sizing rationale.
catalogue/​hardware/​machines/​nvidia_grace_blackwell.json Updates the memory-sizing rationale.
catalogue/​engines/​vllm.json Declares expert-offload flags and environment settings.
catalogue/​engines/​llamacpp.json Declares automatic offload behavior.

Review effort: Balanced

Comment on lines +360 to +363
if best, ok := w.BestFit(d.BudgetGB+d.OffloadGB, ggufOK); ok && many {
d.Best = &best
size = best.SizeGB
}
Comment thread internal/cli/search.go
d.Best.SizeGB, display(*d.Best), d.Variants, d.BudgetGB, d.SizeGB)
case *d.FitsBudget:
fmt.Printf(" size: %.1f GB (fits under the ~%.0fGB budget%s)\n", d.SizeGB, d.BudgetGB, of)
case d.FitsWithOffload != nil && *d.FitsWithOffload:
fits := w.SizeGB <= d.BudgetGB
d.FitsBudget = &fits
}
d.MoE = arch.MoE()
`Under this machine's ~${Math.round(d.budget_gb)} GB budget${of}.`,
);
}
} else if (d.fits_with_offload) {

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants