Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,18 @@ the fact.

### Added

- Grace Hopper and Grace Blackwell (GH200, GB200, GB300, including the DGX
Station GB300): a mixture-of-experts model too big for the GPU's own memory
can keep part of its experts in the Grace CPU's memory. `inspect` and the
panel recommend the largest copy that fits on the GPU, and only when none
does, the largest that fits with offload. `new` gives such a profile
vLLM's `--cpu-offload-gb` and `--cpu-offload-params experts`; llama.cpp
places experts itself. `doctor` and `setup` print the CPU memory available
for it. Engines declare this in a new `offload` field.
- On a machine whose GPUs are different models, such as a DGX Station's GB300
and its display GPU, `setup` records the GPUs with the most memory in
`config.env` as `CUDA_VISIBLE_DEVICES`, by UUID, and `doctor` warns until
it is set.
- Warm-up: a profile's `WARMUP_FILE` names a chat request that llm-helper
sends when `up`, `restart` or `switch` sees the server healthy, and that the router sends when a
server comes back, so the server has the client's fixed prompt cached
Expand Down Expand Up @@ -103,6 +115,10 @@ the fact.

### Changed

- Grace Hopper and Grace Blackwell GPUs are sized by their own HBM rather
than as one pool with system memory, which promised a GH200 about 480 GB
for a 96 GB GPU, and `--gpu-memory-utilization` is no longer capped on
them as on a DGX Spark.
- gpt-oss profiles now carry `TIKTOKEN_ENCODINGS_BASE` themselves, and
`setup` no longer writes it into `config.env`. Existing configs keep the
line, and existing profiles serve as before.
Expand Down
4 changes: 4 additions & 0 deletions catalogue/engines/llamacpp.json
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@
"capabilities": {
"gguf_quant": true
},
"offload": {
"automatic": true,
"why": "llama-server's --fit, on by default, moves a mixture-of-experts model's expert tensors to CPU memory first when the weights outgrow the GPU, then fills the GPU back up layer by layer, so a profile needs no flags."
},
"notes": {
"hybrid": "llama.cpp can reuse a cached prompt on this model only up to the start of a user message, so keep the system prompt and tools identical between chats."
}
Expand Down
12 changes: 12 additions & 0 deletions catalogue/engines/vllm.json
Original file line number Diff line number Diff line change
Expand Up @@ -113,5 +113,17 @@
"vllm_args": true,
"gguf_quant": true,
"chat_template_kwargs_flag": "--default-chat-template-kwargs"
},
"offload": {
"args": [
"--cpu-offload-gb",
"{offload_gib}",
"--cpu-offload-params",
"experts"
],
"env": {
"VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY": "1"
},
"why": "--cpu-offload-gb keeps that many GiB of weights in CPU memory, which the GPU reads in place on every forward pass, and --cpu-offload-params experts limits that to expert weights, so attention and shared layers stay on the GPU. Pinning is off because on a coherent GPU pinned memory takes from GPU memory, as vLLM's own offloader notes; NVIDIA's DGX Station recipe for Nemotron 3 Ultra sets the same three."
}
}
2 changes: 1 addition & 1 deletion catalogue/hardware/machines/nvidia_grace_blackwell.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,5 @@
"gpu_name": "\\b(GB200|GB300|Grace Blackwell)\\b"
},
"memory": "coherent",
"why": "The GPU has HBM of its own, with the Grace CPU's memory coherently attached over NVLink-C2C. Sized as one shared pool, as before, until that is measured on the hardware. The name matches with or without the GB200 or GB300 code, so a name that spells out Grace Blackwell lands here."
"why": "The GPU has HBM of its own, with the Grace CPU's memory coherently attached over NVLink-C2C. Weights are sized against the HBM, and a mixture-of-experts model too big for it can keep experts in the CPU's memory. The name matches with or without the GB200 or GB300 code, so a name that spells out Grace Blackwell lands here."
}
2 changes: 1 addition & 1 deletion catalogue/hardware/machines/nvidia_grace_hopper.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@
},
"memory": "coherent",
"arch": "sm90",
"why": "The GPU has HBM of its own, with the Grace CPU's memory coherently attached over NVLink-C2C. Sized as one shared pool, as before, until that is measured on the hardware. The name matches with or without the GH200 code, so a name that spells out Grace Hopper lands here and gets sm90."
"why": "The GPU has HBM of its own, with the Grace CPU's memory coherently attached over NVLink-C2C. Weights are sized against the HBM, and a mixture-of-experts model too big for it can keep experts in the CPU's memory. The name matches with or without the GH200 code, so a name that spells out Grace Hopper lands here and gets sm90."
}
23 changes: 23 additions & 0 deletions catalogue/hardware/reports/dgx-station-gb300.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"provenance": "synthetic: not measured; a DGX Station GB300 (Dell Pro Max with GB300) from NVIDIA's published specifications: a GB300 with 252 GB of HBM3e, 496 GB of LPDDR5X, and an RTX PRO 2000 Blackwell display GPU, listed after the GB300",
"readings": {
"schema_version": 1,
"os": "linux",
"arch": "arm64",
"mem_total_bytes": 496000000000,
"nvidia_device": true,
"nvidia_smi": true,
"gpus": [
{ "index": 0, "name": "NVIDIA GB300", "mem_bytes": 269172604928, "compute_cap": "10.3", "addressing_mode": "ATS" },
{ "index": 1, "name": "NVIDIA RTX PRO 2000 Blackwell", "mem_bytes": 17103323136, "compute_cap": "12.0", "addressing_mode": "HMM" }
]
},
"detected": {
"hardware_id": "nvidia_grace_blackwell",
"gpu_name": "NVIDIA GB300",
"memory": "coherent",
"memory_source": "definition",
"arch": "sm103",
"cuda": true
}
}
3 changes: 2 additions & 1 deletion docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ A definition's memory is one of three kinds:

- **unified** — one pool shared with the CPU, such as a DGX Spark, a Jetson or a Mac.
- **discrete** — memory of the GPU's own.
- **coherent** — memory of the GPU's own with the CPU's memory coherently attached, as on Grace Hopper and Grace Blackwell. It is sized as unified until that has been measured on the hardware.
- **coherent** — memory of the GPU's own with the CPU's memory coherently attached, as on Grace Hopper and Grace Blackwell.
Weights are sized against the GPU's own memory, and a mixture-of-experts model too big for it can keep part of its experts in the CPU's; see "Grace Hopper and Grace Blackwell" in [Memory budget](./memory.md).

## Flags that crash an architecture

Expand Down
30 changes: 30 additions & 0 deletions docs/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,36 @@ A model larger than one GPU needs `--tensor-parallel-size` in the profile's `EXT

How llm-helper decides whether a GPU has memory of its own is described in [Hardware detection](./hardware.md).

## Grace Hopper and Grace Blackwell

A GH200, GB200 or GB300 has HBM of its own, and the Grace CPU's LPDDR5X attached over NVLink-C2C, which the GPU reads coherently.
The budget is the HBM alone, because in the driver's default memory mode CUDA allocates only from it, so that is where the weights and the KV cache go.
The CPU's memory is a second pool, and `doctor` and `setup` print how much of it a model can use on their `offload:` line: system memory, less what the OS is using and 8 GiB for the server's own process.

The second pool is offered only to mixture-of-experts models.
Each token reads only a few of their experts, where a dense model would read every offloaded weight for every token.
`inspect` and the panel recommend the largest copy of a model that fits on the GPU.
Only when none does do they recommend the largest copy that fits with part of its experts in CPU memory, and say so.
Every copy that fits either way stays selectable.

A profile `new` creates for such a copy keeps the shortfall, plus 4 GiB more for the KV cache, in CPU memory.
For vLLM that is `--cpu-offload-gb <GiB> --cpu-offload-params experts` in `EXTRA_ARGS`, and `VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1` in the profile, which are the settings NVIDIA's DGX Station recipes use.
Pinning is off because on these machines pinned memory is taken from the GPU's.
llama.cpp's `llama-server` moves experts to CPU memory by itself when the weights outgrow the GPU, so its profile gets nothing added.
Each engine says which it does in the `offload` field of its definition; see [Providers](./providers.md).

Offloaded experts are read at the CPU memory's bandwidth, 396 GB/s on a DGX Station GB300 against 7.1 TB/s for its HBM, so decode is slower than for a copy that fits on the GPU.
How much slower depends on the model and has not yet been measured on a GB300.

A DGX Station also has a small display GPU, an RTX PRO 2000.
DGX OS names the GB300 in `CUDA_VISIBLE_DEVICES` in `/etc/environment`, which login shells read and services started by systemd do not.
On a machine whose GPUs are different models, `setup` writes `CUDA_VISIBLE_DEVICES` to `config.env`, naming the GPUs with the most memory by UUID, unless `config.env` already sets it.
`doctor` warns, with the line to add, until it is set.

Older drivers default to a NUMA memory mode instead, in which the HBM is also system memory.
There the OS can fill the HBM with page cache and vLLM's free-memory check can fail at startup, so update the driver before serving.
Leave MIG off: llm-helper does not size MIG instances.

## AMD Ryzen AI Max (Strix Halo)

A Ryzen AI Max GPU (Radeon 8060S, 8050S or 8040S) has no memory of its own beyond a carve-out the firmware sets aside, 512 MB by default.
Expand Down
4 changes: 4 additions & 0 deletions docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,10 @@ A definition holds `schema_version` and the engine's fields, and says:
`chat_template_kwargs_flag` names the flag that sets default chat template kwargs, which is how a profile turns a model's thinking on or off; see [Model types](./model-types.md).
- `notes`: optional sentences `inspect` prints for some kinds of model.
`hybrid` is shown for a model whose layers are not all attention, such as one with linear-attention or Mamba layers.
- `offload`: optional, for a process engine that can keep a mixture-of-experts model's expert weights in CPU memory when they do not fit on a GPU that reads CPU memory coherently, such as a Grace Blackwell's; see "Grace Hopper and Grace Blackwell" in [Memory budget](./memory.md).
`args` are added to the profile's `EXTRA_ARGS` and must use `{offload_gib}`, the GiB to keep in CPU memory; `env` is added to the profile.
`automatic` instead says the server places the weights itself, so nothing is added.
Like everything else, it needs a `why`.

The definition is checked when it loads.
An unknown field, an unknown placeholder, a key that would land on the command line, or a container with no role that serves HTTP is refused with the reason.
Expand Down
56 changes: 51 additions & 5 deletions internal/cli/detectcmds.go
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,12 @@ func gatherDoctor() doctorReport {
}
r.Warnings = append(r.Warnings, hardwareWarnings(r.Hardware, running)...)
r.Warnings = append(r.Warnings, gpuDeviceWarnings(r.Hardware.Accelerators)...)
configPinned := false
if cfg != nil {
configPinned = cfg.File.Has("CUDA_VISIBLE_DEVICES") || (cfg.Secrets != nil && cfg.Secrets.Has("CUDA_VISIBLE_DEVICES"))
}
shell, shellSet := os.LookupEnv("CUDA_VISIBLE_DEVICES")
r.Warnings = append(r.Warnings, gpuPinWarnings(r.Hardware, configPinned, shell, shellSet)...)
if cfg != nil {
r.Warnings = append(r.Warnings, serviceScopeWarnings(cfg)...)
}
Expand Down Expand Up @@ -355,6 +361,27 @@ func hardwareWarnings(hw detect.Hardware, running int) []string {
return w
}

// gpuPinWarnings says when a server could load onto the wrong GPU: the
// machine's GPUs are different models and config.env either names none, so
// each service sees them all, or names a mix of them.
func gpuPinWarnings(hw detect.Hardware, configPinned bool, shell string, shellSet bool) []string {
switch {
case hw.GPUPin == "":
return nil
case configPinned && !hw.SameGPUs():
return []string{fmt.Sprintf(
"CUDA_VISIBLE_DEVICES gives the server different GPUs (%s), and it loads onto the first. Set CUDA_VISIBLE_DEVICES=%s in config.env to use the one with the most memory.",
hw.GPUSummary(), hw.GPUPin)}
case configPinned:
return nil
}
msg := "this machine has GPUs of different models and config.env does not set CUDA_VISIBLE_DEVICES, so each model's service sees all of them and loads onto the first in PCI order"
if shellSet {
msg += fmt.Sprintf(". This shell's CUDA_VISIBLE_DEVICES=%s reaches a service only if systemd sets it too", shell)
}
return []string{fmt.Sprintf("%s. Run '%s setup' to record CUDA_VISIBLE_DEVICES=%s in config.env, or add it by hand.", msg, cmdName, hw.GPUPin)}
}

// overlapWarnings names the definitions that claim one of this machine's
// GPUs when more than one does. The shipped ones are tested never to
// overlap, so the others come from hardware.d, and only the first is used.
Expand Down Expand Up @@ -398,6 +425,26 @@ func overlapWarnings(reg *hardware.Registry, hw detect.Hardware) []string {
return w
}

// memoryKind names how the GPU's memory relates to the CPU's.
func memoryKind(hw detect.Hardware) string {
switch {
case hw.Unified:
return "unified memory"
case hw.Memory == string(hardware.MemoryCoherent):
return "its own memory, and the CPU's over NVLink-C2C"
}
return "discrete VRAM"
}

// offloadLine says how much CPU memory a coherent GPU can keep a
// mixture-of-experts model's experts in, or "" when it cannot.
func offloadLine(hw detect.Hardware) string {
if hw.OffloadGB <= 0 {
return ""
}
return fmt.Sprintf("~%d GB more of a mixture-of-experts model's experts in CPU memory, when it does not fit the GPU (slower)", hw.OffloadGB)
}

// hardwareLine says which hardware definition recognised the GPU, its
// architecture, and what decided its memory kind when no definition did.
func hardwareLine(hw detect.Hardware) string {
Expand All @@ -424,11 +471,7 @@ func renderDoctor(r doctorReport, yes bool) {
fmt.Println("HARDWARE")
fmt.Printf(" memory: %d GB total\n", hw.TotalMemGB)
if hw.GPUName != "" {
kind := "discrete VRAM"
if hw.Unified {
kind = "unified memory"
}
fmt.Printf(" gpu: %s (%s)\n", hw.GPUSummary(), kind)
fmt.Printf(" gpu: %s (%s)\n", hw.GPUSummary(), memoryKind(hw))
if line := hardwareLine(hw); line != "" {
fmt.Printf(" hardware: %s\n", line)
}
Expand Down Expand Up @@ -465,6 +508,9 @@ func renderDoctor(r doctorReport, yes bool) {
fmt.Printf(" in use: %.1f GB by the OS and everything else running\n", hw.BaselineGB)
}
fmt.Printf(" budget: ~%d GB of model weights (%s)\n", hw.BudgetGB, hw.BudgetReason)
if line := offloadLine(hw); line != "" {
fmt.Printf(" offload: %s\n", line)
}
// An override silently replaces the figure above everywhere else, so
// showing only the measured one would make doctor disagree with search.
if r.Config != nil && r.Config.BudgetOverride != "" {
Expand Down
113 changes: 113 additions & 0 deletions internal/cli/gpupin_internal_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
package cli

import (
"os"
"path/filepath"
"strings"
"testing"

"github.com/coder/llm-helper/internal/config"
"github.com/coder/llm-helper/internal/detect"
"github.com/coder/llm-helper/internal/engine"
"github.com/coder/llm-helper/internal/hardware"
"github.com/coder/llm-helper/internal/hf"
"github.com/coder/llm-helper/internal/resolve"
)

// station is a DGX Station GB300 whose display GPU the server can also see.
var station = detect.Hardware{
GPUName: "NVIDIA GB300", Memory: string(hardware.MemoryCoherent), OffloadGB: 478,
GPUs: []detect.GPU{{Name: "NVIDIA GB300", UUID: "GPU-aaaa"}, {Name: "NVIDIA RTX PRO 2000 Blackwell", UUID: "GPU-bbbb"}},
GPUPin: "GPU-aaaa",
}

// A new config on a machine with different GPUs names the biggest, and a
// kept one gets it only when it names no GPUs itself.
func TestSetupPinsTheBiggestGPUWhenTheyDiffer(t *testing.T) {
prov, err := engine.Lookup(engine.VLLM)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "config.env")
if err := os.WriteFile(path, []byte(newConfigEnv(prov, "/opt/vllm/bin/vllm", "user", station)), 0o600); err != nil {
t.Fatal(err)
}
env, err := config.ParseEnvFile(path)
if err != nil {
t.Fatal(err)
}
if got := env.Get("CUDA_VISIBLE_DEVICES"); got != "GPU-aaaa" {
t.Errorf("new config CUDA_VISIBLE_DEVICES = %q, want the GB300's UUID", got)
}
if strings.Contains(newConfigEnv(prov, "/opt/vllm/bin/vllm", "user", detect.Hardware{}), "CUDA_VISIBLE_DEVICES") {
t.Error("a machine with one kind of GPU got a CUDA_VISIBLE_DEVICES line")
}

kept := filepath.Join(t.TempDir(), "config.env")
if err := os.WriteFile(kept, []byte("PORT=8001\n"), 0o600); err != nil {
t.Fatal(err)
}
if filled, err := fillMissingGPUPin(kept, "GPU-aaaa"); err != nil || !filled {
t.Fatalf("fillMissingGPUPin = %v, %v; want filled", filled, err)
}
if filled, err := fillMissingGPUPin(kept, "GPU-cccc"); err != nil || filled {
t.Errorf("a second fill replaced the first: %v, %v", filled, err)
}
env, err = config.ParseEnvFile(kept)
if err != nil {
t.Fatal(err)
}
if got := env.Get("CUDA_VISIBLE_DEVICES"); got != "GPU-aaaa" || env.Get("PORT") != "8001" {
t.Errorf("kept config: CUDA_VISIBLE_DEVICES=%q PORT=%q", got, env.Get("PORT"))
}
}

func TestDoctorWarnsWhenTheServerCouldPickTheWrongGPU(t *testing.T) {
t.Parallel()
if w := gpuPinWarnings(station, false, "GPU-aaaa", true); len(w) != 1 ||
!strings.Contains(w[0], "CUDA_VISIBLE_DEVICES=GPU-aaaa in config.env") || !strings.Contains(w[0], "only if systemd sets it") {
t.Errorf("unpinned, with the shell's setting: %q", w)
}
if w := gpuPinWarnings(station, true, "", false); len(w) != 1 || !strings.Contains(w[0], "different GPUs") {
t.Errorf("config.env names both GPUs: %q", w)
}
pinned := station
pinned.GPUs = pinned.GPUs[:1]
if w := gpuPinWarnings(pinned, true, "", false); len(w) != 0 {
t.Errorf("config.env names the GB300 alone: %q", w)
}
if w := gpuPinWarnings(detect.Hardware{GPUName: "NVIDIA GB10"}, false, "", false); len(w) != 0 {
t.Errorf("one GPU: %q", w)
}
}

func TestACoherentGPUIsDescribedWithBothPools(t *testing.T) {
t.Parallel()
if got := memoryKind(station); !strings.Contains(got, "NVLink-C2C") {
t.Errorf("memoryKind = %q", got)
}
if got := offloadLine(station); !strings.Contains(got, "478 GB") {
t.Errorf("offloadLine = %q", got)
}
if got := offloadLine(detect.Hardware{Unified: true}); got != "" {
t.Errorf("unified offloadLine = %q", got)
}
}

// A mixture-of-experts model too big for a Grace GPU is shown as fitting with
// offload, and its copies say which fit how.
func TestInspectShowsAModelThatFitsWithOffload(t *testing.T) {
d := resolve.Detail{
SizeGB: 395, BudgetGB: 244, BudgetKnown: true, FitsBudget: new(false),
MoE: true, OffloadGB: 478, FitsWithOffload: new(true), OffloadNeededGB: 151,
All: []hf.Variant{{Name: "big", Label: "BIG", SizeGB: 900}, {Name: "nvfp4", Label: "NVFP4", SizeGB: 395}},
}
d.Best = &d.All[1]
d.Variants = 2
out, _ := captureStdout(t, func() error { printModelDetail(d, nil); return nil })
for _, want := range []string{"fits with ~151 GB of its experts in CPU memory", "fits with experts in CPU memory, slower", "too big for this machine"} {
if !strings.Contains(out, want) {
t.Errorf("output lacks %q:\n%s", want, out)
}
}
}
Loading
Loading