Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,27 @@ the fact.
- `recipes install-service` follows `SERVICE_SCOPE`. With `user` it installs
the recipe unit and its watchdog in `~/.config/systemd/user`, enables
lingering so they start at boot, and needs no sudo.
- AMD Ryzen AI Max (Strix Halo): the machine is recognised by its PCI id and
sized against the GPU's GTT limit, the system memory amdgpu lets it map,
rather than as a CPU-only box. A hardware definition can now match by
`pci_devices`, and `hardware report` records `gtt_bytes`.
- `setup` records the server named by the provider's key in the environment,
such as `LLAMA_SERVER_BIN`, instead of the one on PATH, and refuses to run
when that key names something that is not an executable. With a
`config.env` already in place, it keeps the file and says when the key in
the environment differs from it.
- `setup --service-scope user` writes a config whose model services run in
the user's service manager, which needs no sudo. On Linux, `doctor` warns
when lingering is off for that user, since the models then do not start at
boot.
- Docs: llama.cpp on an AMD GPU under Linux (the release's Vulkan and ROCm
builds, and the `render` group), and DeepSeek V4 Flash Vision measured on a
Ryzen AI Max with both.
- `doctor` shows the GTT limit on a Strix Halo and, when it is well below
90% of RAM, the ttm `pages_limit` that raises it; see docs/memory.md.
- `doctor` warns when an amdgpu GPU is present but the user cannot open
`/dev/kfd` or `/dev/dri/renderD*`, which makes llama.cpp's ROCm build fail
with "no ROCm-capable device is detected", and names the `usermod` fix.

### Changed

Expand Down Expand Up @@ -124,6 +145,9 @@ the fact.

### Fixed

- `inspect <repo>:<quant>`, the form a llama.cpp profile's `MODEL` takes,
failed with "hub lookup: HTTP 401" because the quant was sent as part of
the repo id. It now looks up the repo and reports that quant's size.
- `inspect`, `new` and the panel sized the KV cache of GLM-5.3-Flash about
50 times too large: its 34 linear-attention layers counted as full
attention, and its 11 latent-attention layers as 64 full heads. A
Expand Down
2 changes: 1 addition & 1 deletion catalogue/engines/llamacpp.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"schema_version": 1,
"id": "llamacpp",
"display_name": "llama.cpp",
"why": "llama.cpp's OpenAI-compatible server, llama-server. It serves GGUF on Metal, CUDA or the CPU, and sizes its own context and GPU layers to fit the machine, so a profile needs only the model. MODEL is passed as --alias too, so /v1/models reports exactly MODEL whether the model comes from the Hub or a file. It checks an API key, which it reads from LLAMA_API_KEY so the key is on no command line; with one set it binds every interface, and without one it stays on 127.0.0.1. It downloads a repo's GGUF itself, and its /health answers 200 only once the model is loaded.",
"why": "llama.cpp's OpenAI-compatible server, llama-server. It serves GGUF on Metal, CUDA, Vulkan, ROCm or the CPU, and sizes its own context and GPU layers to fit the machine, so a profile needs only the model. MODEL is passed as --alias too, so /v1/models reports exactly MODEL whether the model comes from the Hub or a file. It checks an API key, which it reads from LLAMA_API_KEY so the key is on no command line; with one set it binds every interface, and without one it stays on 127.0.0.1. It downloads a repo's GGUF itself, and its /health answers 200 only once the model is loaded.",
"runtime": ["process"],
"install": {
"any": "brew install llama.cpp"
Expand Down
3 changes: 2 additions & 1 deletion catalogue/hardware/machines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@

Each file here is one kind of machine, named after its id, such as `dgx_spark_gb10.json`.
A machine matches a GPU by vendor and by a regular expression over the name the GPU reports, case-insensitively.
A GPU whose driver reports no name, such as an AMD GPU under `amdgpu`, matches by vendor and by a list of PCI device ids in `pci_devices`, such as `0x1586`.
It says how the GPU's memory relates to the CPU's: `unified`, `discrete` or `coherent`.
Recipes record a machine's id as the hardware they were measured on, so an id never changes once shipped.

Add one when llm-helper gets a GPU wrong without it, for example a GPU that shares system memory but reports none of its own.
Keep `gpu_name` narrow enough to match only your GPU, because no two machines may match the same one.
Keep `gpu_name` or `pci_devices` narrow enough to match only your GPU, because no two machines may match the same one.
Describe one chip family, not a catch-all: a GPU no machine names already gets its architecture from its compute capability and its memory from what it reports.
Give every claim a `why`, and add a report to [`reports/`](../reports) that detects the machine.

Expand Down
11 changes: 11 additions & 0 deletions catalogue/hardware/machines/amd_strix_halo.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"schema_version": 1,
"id": "amd_strix_halo",
"display_name": "AMD Ryzen AI Max (Strix Halo)",
"match": {
"vendor": "amd",
"pci_devices": ["0x1586"]
},
"memory": "unified",
"why": "The Radeon 8060S, 8050S and 8040S in a Ryzen AI Max are one PCI device, 0x1586, and amdgpu reports no name for it, so the device id is what identifies it. The GPU has only the carve-out the firmware sets aside (mem_info_vram_total, 512 MB by default) and maps the rest from system memory through GTT, up to mem_info_gtt_total, which the ttm module's pages_limit sets. The budget comes from that limit, not from VRAM or all of RAM."
}
20 changes: 20 additions & 0 deletions catalogue/hardware/reports/amd_strix_halo.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"provenance": "measured: llm-helper built from 9adb7af on an AMD Ryzen AI Max (Strix Halo, Radeon 8060S) with 128 GB, ttm pages_limit raised to 29360128",
"readings": {
"schema_version": 1,
"os": "linux",
"arch": "amd64",
"system": "Advanced Micro Devices, Inc. (AMD) RAH-001",
"mem_total_bytes": 134314070016,
"nvidia_smi": false,
"pci": [{ "vendor": "0x1002", "device": "0x1586", "class": "0x038000", "driver": "amdgpu", "vram_bytes": 536870912, "gtt_bytes": 120259084288 }]
},
"detected": {
"hardware_id": "amd_strix_halo",
"gpu_name": "AMD Ryzen AI Max (Strix Halo)",
"memory": "unified",
"memory_source": "definition",
"cuda": false,
"accelerators": ["AMD gpu (0x1002:0x1586, driver amdgpu, 536 MB)"]
}
}
6 changes: 4 additions & 2 deletions docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ An NVIDIA GPU with no definition still works: its memory comes from `nvidia-smi`
`nvidia-smi` gets 10 seconds; a wedged driver is treated as `nvidia-smi` failing, rather than hanging the command.
- **`CUDA_VISIBLE_DEVICES`** — from the environment and from `config.env`, `secrets.env` and the profile being served, so the GPUs sized are the ones the server will get.
- **Jetson markers** — `/etc/nv_tegra_release`, `/sys/class/tegra-firmware` and the device-tree model.
- **The PCI bus** — every display controller from a known vendor, and every processing accelerator, so an AMD GPU or an NPU is named even though llm-helper does not size for it yet.
- **The PCI bus** — every display controller from a known vendor, and every processing accelerator, so an AMD GPU or an NPU is named even when llm-helper does not size for it.
A GPU that a hardware definition recognises by its PCI ids, such as the AMD Ryzen AI Max (Strix Halo), is sized from what amdgpu lets it map: its GTT limit (`mem_info_gtt_total`) plus its firmware carve-out (`mem_info_vram_total`). See "AMD Ryzen AI Max (Strix Halo)" in [Memory budget](./memory.md).
- **The cgroup memory limit** — in a container or a limited systemd unit, the budget is held to the limit rather than the host's RAM.
- **Apple Silicon** — the SoC's brand string, which names its GPU.

Expand Down Expand Up @@ -55,7 +56,8 @@ Each file is named after the id it defines, starts with the format version this
`llm-helper hardware list` shows the machines in effect, including any added locally.

A machine matches a vendor and a regular expression over the GPU's name, case-insensitively.
No two machines may match the same GPU, so keep the pattern narrow enough to match only yours; the tests run every report against every machine and fail on an overlap.
A GPU that its driver gives no name, such as an AMD GPU under `amdgpu`, is matched by its vendor and its PCI device ids instead, with `"pci_devices": ["0x1586"]`; `llm-helper hardware report` prints them.
No two machines may match the same GPU, so keep the pattern or the list narrow enough to match only yours; the tests run every report against every machine and fail on an overlap.
Every definition needs a `why`, because an entry nobody can explain cannot be reviewed, and cannot be removed when the reason stops holding.

## Local definitions
Expand Down
9 changes: 7 additions & 2 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ On Linux that means an NVIDIA GPU with a vLLM usually living in a venv, or llama

3. Run `llm-helper setup`, or `llm-helper setup --provider mlx` or `--provider llamacpp` to serve with mlx-lm or llama.cpp.

If `vllm` isn't on your PATH, set `VLLM_BIN` in `~/.config/vllm-helper/config.env` to its full path.
If `vllm` isn't on your PATH, run setup with `VLLM_BIN` set to its full path in the environment, or set `VLLM_BIN` in `~/.config/vllm-helper/config.env` afterwards.
The other providers' keys, such as `LLAMA_SERVER_BIN`, work the same way, and one set in the environment wins over a server on PATH.
When `config.env` already exists, setup keeps it, and says when a key in the environment differs from the one in the file so you can change the file by hand.

## What setup writes

Expand All @@ -42,7 +44,7 @@ See [Hardware detection](./hardware.md).
| `MLX_LM_BIN` | config.env | Full path to the `mlx_lm.server` executable. Required for MLX profiles. |
| `LLAMA_SERVER_BIN` | config.env | Full path to the `llama-server` executable. Required for llama.cpp profiles. |
| `SERVICE_NAME` | config.env | The single service unit earlier releases ran: `vllm.service` on Linux, `com.coder.vllm` on macOS. Read only by `migrate`. |
| `SERVICE_SCOPE` | config.env | `system` on Linux (needs sudo to control) or `user`. macOS defaults to `user`; use `system` on a headless Mac. |
| `SERVICE_SCOPE` | config.env | `system` on Linux (needs sudo) or `user`, as `setup --service-scope` sets it. macOS defaults to `user`; use `system` on a headless Mac. |
| `DEFAULT_PROFILE` | config.env | The model Coder selects first. Written by `llm-helper default`. See [Several models at once](./multi-instance.md). |
| `ROUTER_PORT` | config.env | Port the model router listens on. Written by `llm-helper router install-service`. |
| `MODEL_BUDGET_GB` | config.env | Overrides the measured model budget. See [Memory budget](./memory.md). |
Expand All @@ -58,6 +60,9 @@ See [Hardware detection](./hardware.md).
| `HF_TOKEN` | either | HuggingFace token, needed only for gated or private models. A value in `secrets.env` wins. |
| `GITHUB_TOKEN` | either | Token for reading releases while the repository is private. A value in `secrets.env` wins; `upgrade` also accepts it in the environment. |

With `SERVICE_SCOPE=user` on Linux, each model's unit runs in the user's systemd manager, which starts at boot only when lingering is on for the user.
`up` tries to turn it on, and `doctor` warns while it is off; turn it on yourself with `sudo loginctl enable-linger $USER`.

## Upgrade

```sh
Expand Down
23 changes: 23 additions & 0 deletions docs/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,29 @@ A model larger than one GPU needs `--tensor-parallel-size` in the profile's `EXT

How llm-helper decides whether a GPU has memory of its own is described in [Hardware detection](./hardware.md).

## AMD Ryzen AI Max (Strix Halo)

A Ryzen AI Max GPU (Radeon 8060S, 8050S or 8040S) has no memory of its own beyond a carve-out the firmware sets aside, 512 MB by default.
It maps the rest from system memory through GTT, up to a limit the `ttm` kernel module sets, which `amdgpu` reports in `/sys/class/drm/card0/device/mem_info_gtt_total`.
The budget is measured against that limit plus the carve-out, not against all of RAM, and `doctor` prints the limit on its `gtt:` line.
llama.cpp sees the same memory: its ROCm build reports the GTT limit as the device's memory, and its Vulkan build reports the limit plus the carve-out.

The default limit leaves memory idle on a machine that only serves models.
On a 128 GB machine it is 101 GB, and a 103 GB model does not fit.
To raise it, set `pages_limit` in 4 KiB pages, rebuild the initramfs so the module reads it at boot, and reboot:

```sh
echo 'options ttm pages_limit=29360128' | sudo tee /etc/modprobe.d/ttm.conf
sudo update-initramfs -u
sudo reboot
```

29360128 pages is 112 GiB.
On a 128 GB machine that raised the limit to 120 GB, and with a 103 GB model loaded about 15 GB of RAM was still free.
`doctor` suggests a value for its machine, 90% of RAM in whole GiB, until the limit is within 4 GiB of it.
`update-initramfs` is the Debian and Ubuntu command; on Fedora run `sudo dracut -f` instead.
The OS shares the same memory, so the budget never counts more than RAM less what the OS is using, however high the limit is set.

## Next steps

- [Profiles](./profiles.md) — where `KV_CACHE_MEMORY_BYTES` comes from.
Expand Down
67 changes: 65 additions & 2 deletions docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,11 @@ llm-helper set <profile> -- EXTRA_ARGS='--host 0.0.0.0 --max-tokens 32768'

## llama.cpp

[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` serves GGUF models on Metal, CUDA or the CPU.
[llama.cpp](https://github.com/ggml-org/llama.cpp)'s `llama-server` serves GGUF models on Metal, CUDA, Vulkan, ROCm or the CPU.
It sizes its context and GPU layers to fit the machine, so a llama.cpp profile needs only the model.
Use a repo of GGUF files, such as one from [bartowski](https://huggingface.co/bartowski) or [unsloth](https://huggingface.co/unsloth).

1. Install llama.cpp. Homebrew has it on macOS and Linux, and the [releases](https://github.com/ggml-org/llama.cpp/releases) have prebuilt binaries:
1. Install llama.cpp. Homebrew has it on macOS and Linux, and the [releases](https://github.com/ggml-org/llama.cpp/releases) have prebuilt binaries; for an AMD GPU on Linux, see [AMD GPUs on Linux](#amd-gpus-on-linux):

```sh
brew install llama.cpp
Expand Down Expand Up @@ -129,11 +129,74 @@ Things to know about `llama-server`:
- **It downloads the model when it starts.** The first start of a profile takes as long as the download, and the GGUF goes to the same Hugging Face cache vLLM uses.
- **Its health check waits for the model.** `status` and `up --wait` report the endpoint up only once the model is loaded and it can serve.
- **Context comes from the model,** then shrinks to fit memory. Set it with `-c <tokens>` in `EXTRA_ARGS`.
- **Its token counters stand still while it reads a prompt.** `llamacpp:prompt_tokens_total` and `llamacpp:tokens_predicted_total` in `/metrics` stayed flat for 181 seconds while b12364 read a 45,000-token prompt, so a flat counter during a long prompt is not a stuck server.

```sh
llm-helper set <profile> -- EXTRA_ARGS='-c 65536'
```

### AMD GPUs on Linux

llama.cpp serves an AMD GPU through Vulkan or ROCm, and every [release](https://github.com/ggml-org/llama.cpp/releases) has a prebuilt Ubuntu x64 build of each.
A distribution's own `llama-server` can be far older: Debian's `/usr/bin/llama-server` is a 0.2.0 build, and `setup` finds it on PATH unless told otherwise.

1. Download a build and unpack it.
The Vulkan one, for example:

```sh
curl -LO https://github.com/ggml-org/llama.cpp/releases/download/b12364/llama-b12364-bin-ubuntu-vulkan-x64.tar.gz
mkdir -p ~/llama.cpp
tar xzf llama-b12364-bin-ubuntu-vulkan-x64.tar.gz -C ~/llama.cpp --strip-components=1
```

2. Let your user open the GPU.
`/dev/kfd` and `/dev/dri/renderD*` belong to the `render` group, and without it the ROCm build fails with "no ROCm-capable device is detected".
`doctor` warns when you cannot open them.

```sh
sudo usermod -aG render,video $USER
```

Log out and back in for the groups to apply.
A user's systemd manager that lingers keeps the groups it started with, so reboot if your models run with `SERVICE_SCOPE=user`.

3. Run setup with the server named in `LLAMA_SERVER_BIN`.
The user scope needs no sudo, and lingering starts the models at boot:

```sh
LLAMA_SERVER_BIN=$HOME/llama.cpp/llama-server llm-helper setup --provider llamacpp --service-scope user
sudo loginctl enable-linger $USER
```

On a Ryzen AI Max (Strix Halo), the GPU loads the model into GTT memory, which is capped well below RAM by default; raise the cap first, as "AMD Ryzen AI Max (Strix Halo)" in [Memory budget](./memory.md) describes.

### Example: DeepSeek V4 Flash Vision on a Ryzen AI Max

This ran on a Ryzen AI Max with the Radeon 8060S and 128 GB, with the GTT limit raised to 120 GB, and llama.cpp b12364's Vulkan and ROCm 10 builds from the releases.
The model is Unsloth's `UD-IQ3_XXS` GGUF (103 GB), served as a shared endpoint with 4 slots.
llama-server finds the vision file (`mmproj`) in the same repo by itself, with `--offline` too.

```sh
llm-helper new unsloth/DeepSeek-V4-Flash-Vision-Exp-GGUF --quant UD-IQ3_XXS
llm-helper set deepseek-v4-flash-vision-exp-gguf-ud-iq3_xxs -- \
EXTRA_ARGS="-c 1048576 -np 4 -ngl all --fit off -fa on -b 4096 -ub 2048 --temp 1.0 --top-p 0.95 --min-p 0.0 --metrics --offline"
```

| | ROCm 10 | Vulkan (RADV) |
| -------------------------- | ------------------------ | ------------------------ |
| Load to healthy | 22 s | 21 s |
| One request, short prompt | 15.2 tokens/sec | 16.9 tokens/sec |
| Prompt read, 10,449 tokens | 154 tokens/sec | 201 tokens/sec |
| Prompt read, 38,968 tokens | 118 tokens/sec | 128 tokens/sec |
| Two requests at once | 12.0 each | 12.0 each |
| Four requests at once | about 8.3 each, 33 total | about 8.2 each, 33 total |
| GTT memory in use | 113.7 GB | 114.5 GB |

- Vulkan writes and reads prompts faster than ROCm on one request, and the two are level with several at once.
- With the model loaded, about 15 GB of the 128 GB was still free.
At the stock GTT limit of 101 GB, the model does not fit.
- A tool call came back as correct `tool_calls`, a needle in a long context was found, and a red image was answered "Red."

### Prompt cache

`llama-server` keeps the prompts of finished requests in host memory, up to `--cache-ram` MiB (8192 by default), and reuses the longest matching start of a new prompt instead of reading it again.
Expand Down
16 changes: 16 additions & 0 deletions docs/reference/cli/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ description: "First-time setup: detect hardware and the inference server, write

<!-- DO NOT EDIT | GENERATED CONTENT -->

First-time setup: detect hardware and the inference server, write config.env and
secrets.env, and offer to adopt a running server. It never overwrites a config.

The server's path comes from the provider's key in the environment when it is set,
such as LLAMA_SERVER_BIN or VLLM_BIN, and otherwise from PATH and where installers put it.

--service-scope user runs each model's service under this user, which needs no sudo.

## Usage

```console
Expand All @@ -22,3 +30,11 @@ llm-helper setup [flags]
| Default | `vllm` |

inference server new profiles use: llamacpp, mlx, vllm (only written to a new config)

### --service-scope

| | |
| ---- | -------- |
| Type | `string` |

where each model's service runs: system (needs sudo) or user; default system on Linux, user on macOS (only written to a new config)
Loading
Loading