Skip to content

hardware: size and run AMD Strix Halo under Linux - #70

Merged
DevelopmentCats merged 15 commits into
mainfrom
bpmct/amd-strix-halo
Nov 15, 2026
Merged

DevelopmentCats merged 15 commits into
mainfrom
bpmct/amd-strix-halo

Conversation

@bpmct

@bpmct bpmct commented Nov 13, 2026

Copy link
Copy Markdown
Member

Gets llm-helper working on an AMD Ryzen AI Max (Strix Halo) box under Linux. I moved a 128 GB Strix Halo from a hand-written systemd unit to llm-helper. It serves DeepSeek-V4-Flash-Vision UD-IQ3_XXS on llama.cpp's Vulkan build. setup, new, set and restart worked, but only after a few manual steps. llm-helper also sized the box as a CPU-only machine with a 0 GB GPU. This PR fixes those gaps.

Changes

  • hardware: a machine file can match by PCI device id (match.pci_devices, with a vendor). New amd_strix_halo machine (unified memory) and a measured report.
  • detect: read amdgpu's GTT limit (mem_info_gtt_total) and size the weights budget from it, not from vLLM's CPU backend. A 536 MB carve-out now reads as 536 MB, not 0 GB.
  • doctor:
    • Shows the GTT limit, with the ttm pages_limit steps to raise it when it's well under RAM.
    • Warns when the user can't open /dev/kfd or /dev/dri/renderD* (the render group).
    • Warns when SERVICE_SCOPE=user has no linger.
  • setup:
    • Honors the engine's bin key from the environment, such as LLAMA_SERVER_BIN. Before, it picked Debian's ancient /usr/bin/llama-server, which can't run this model.
    • Adds --service-scope system|user, because system scope needs sudo, and sudo on this box asks for a password.
    • Says when an existing config.env ignores the bin key.
  • inspect: accepts <repo>:<quant>. Before, it got a 401 from the Hub.
  • docs: llama.cpp on AMD Linux (release tarballs, render group, user scope), the Strix Halo GTT note, and a measured example.

On the box (128 GB, GTT raised to 112 GiB), idle, llama.cpp b12364, -c 1048576 -np 4:

                   Vulkan        ROCm 10
1 request:         16.9 tok/s    15.2 tok/s
4 at once:         ~8.2 each, 33 total (both)
prompt read, 10k:  201 tok/s     154 tok/s
prompt read, 39k:  128 tok/s     118 tok/s
GTT used:          ~114 GB

Tool calls and images work. Before raising the limit, the stock GTT limit (94 GiB) was too small for this 103 GB quant. doctor now says so and prints the fix.

Tested:

  • gofmt -l . is empty.
  • go vet ./... and go test -race ./... pass.
  • darwin/arm64 and linux/arm64 build.
  • recipediff tests and Prettier pass.
  • gendocs leaves no diff.

On the real box with a build of 9adb7af:

  • hardware report shows amd_strix_halo and gtt_bytes.
  • doctor shows the GTT line and a ~105 GB budget, with no false warnings.
  • As nobody, doctor gives the render group warning.
  • setup picks up LLAMA_SERVER_BIN and --service-scope user. It rejects /nope and --service-scope users.
  • inspect repo:quant works.

The last two commits came after that run. The note for a kept config.env and the report swap are unit-tested only. The linger warning is unit-tested only, because turning linger off would stop the running model.

What I hit on the box, in order
  1. doctor: "AMD gpu ... present, but llm-helper only sizes NVIDIA and Apple GPUs". GPU 0 GB, and the budget came from vLLM's CPU backend.
  2. setup --provider llamacpp picked /usr/bin/llama-server (Debian llama.cpp 0.2.0) and ignored LLAMA_SERVER_BIN. I fixed config.env by hand.
  3. setup wrote SERVICE_SCOPE=system, and sudo needs a password, so I changed it to user and ran loginctl enable-linger.
  4. The ROCm build said "no ROCm-capable device" until the user joined the render group.
  5. inspect <repo>:<quant> got HTTP 401.
  6. Then new --quant UD-IQ3_XXS, set EXTRA_ARGS=... and restart worked as they are. llama-server found the mmproj with -hf ... --offline.

Generated by Coder Agents on behalf of @bpmct.

bpmct added 11 commits November 13, 2026 20:53
amdgpu gives a GPU no name: an AMD device is only its PCI vendor and
device ids, so a machine file had no way to recognise one, and an AMD
APU that shares system memory was sized as a CPU-only box. A machine's
match can now list pci_devices, which must hold together with its vendor
(device ids are only unique within one) and with gpu_name when both are
given. Matching stays data: no GPU is named in Go.
A Ryzen AI Max read as a CPU-only box: amdgpu reports only a 512 MB
carve-out as VRAM, nothing names the GPU, so doctor warned that only
NVIDIA and Apple GPUs are sized and the budget was the CPU backend's
share of RAM. The GPU actually loads weights into GTT, system memory up
to the limit the ttm module sets, which amdgpu reports in
mem_info_gtt_total.

The amd_strix_halo machine matches PCI device 0x1586 (one id for the
8060S, 8050S and 8040S) and is unified. The probe now reads
mem_info_gtt_total, locally and over SSH, and a unified GPU with a GTT
reading is sized against the limit plus its carve-out, held to RAM less
the OS when the limit is raised close to all of it. With the box's
numbers, the stock 101 GB limit budgets about 87 GB and the raised
120 GB one about 105 GB, which matches a 103 GB model needing the raise.

The report is the box's readings from 0.6.1-dev.63314bb with gtt_bytes
added from sysfs, since that build did not read it. A carve-out under a
gigabyte is shown in MB instead of "0 GB".
On a Strix Halo the GTT limit is what bounds the model, and the stock
one (101 GB on a 128 GB box) leaves memory idle: a 103 GB model only
fit after raising ttm's pages_limit by hand. Doctor now prints the limit
beside the GPU and, when it is more than 4 GiB below 90% of RAM, the
modprobe line, initramfs rebuild and reboot that raise it, the way
metal-limit does for the wired limit on macOS. 90% is what served the
103 GB model with about 15 GB still free.

It only prints the steps. Writing modprobe.d needs root and a bad value
is found out at the next boot, so llm-helper does not do it.
llama.cpp's ROCm build failed on a Strix Halo with "no ROCm-capable
device is detected", which says nothing about the cause: the user was
not in the render group, and /dev/kfd and /dev/dri/renderD128 are
root:render 0660. On Linux, when an amdgpu GPU is on the bus, doctor
now checks those nodes with access(2), which needs no open (opening
/dev/kfd sets up a GPU context), and names the ones this user cannot
use with the usermod fix and the re-login it needs. A node that does
not exist is not reported, since Vulkan serves without KFD. The rule
takes the device directory and the check as arguments, so it is tested
without devices.
On a Debian box setup found /usr/bin/llama-server, the distribution's
0.2.0 build, and ignored LLAMA_SERVER_BIN in the environment, so the
official release had to be put in config.env by hand. lookBin now takes
the server the environment names under the engine definition's bin.key
first, for every engine, including VLLM_BIN when a rerun fills in a
missing one. A bare name is looked up on PATH and recorded as a full
path. A key that names nothing executable stops setup before it writes
anything, rather than falling back to the server the user set it to
avoid. An environment variable rather than a flag, because the key is
already how config.env names the server.
setup always wrote SERVICE_SCOPE=system on Linux. On a box where sudo
asks for a password, every system-scope unit operation fails when run
without a terminal, so the scope had to be changed in config.env by
hand before the first up. --service-scope system|user picks it for a
new config, like --provider. Left empty it is still the platform's
default, which keeps the generated CLI reference the same on Linux and
macOS. On a kept config that says otherwise, setup names the line to
change rather than rewriting the file. The config text moves into
newConfigEnv so what it writes can be tested.
With SERVICE_SCOPE=user, each model's unit lives in the user's systemd
manager, which systemd starts at the user's first login and stops at the
last logout unless the user lingers. A box set up that way served
nothing after a reboot until 'sudo loginctl enable-linger' was run. up
already tries to enable lingering, but without sudo that can fail with
only a note. Doctor on Linux now checks /var/lib/systemd/linger for the
user, which needs no loginctl call, and names the command when the file
is missing. A machine without /var/lib/systemd is not checked.
A llama.cpp profile's MODEL is <repo>:<quant>, and pasting it into
inspect sent the whole string to the Hub as the repo id, which answered
with "hub lookup: HTTP 401". inspect now looks up the repo and, given a
quant, checks the repo holds it and prints its size against the budget,
or names the quants it does hold. --json prints the repo's detail as
before.
The llama.cpp section said Metal, CUDA or the CPU and offered only
Homebrew. On a Strix Halo box the working setup was the release's
Vulkan or ROCm tarball, the render group, LLAMA_SERVER_BIN at setup,
the user scope with lingering, and a raised GTT limit; each of those
cost a manual step, so they are now written down in order. The example
records the box's numbers for DeepSeek V4 Flash Vision UD-IQ3_XXS with
both builds, beside the Mac Studio ones, and the note that llama.cpp's
token counters do not move while it reads a long prompt.
Setup never edits an existing config.env, so a second run with
LLAMA_SERVER_BIN (or any provider's bin key) set in the environment did
nothing and printed nothing, even when the key named a path that does not
exist. It looked as if the new server had been recorded.

Setup now prints a note for each provider bin key set in the environment
that differs from config.env, or that names nothing runnable, with the
line to put in config.env by hand. It still does not change the file.
The report's gtt_bytes was read from sysfs by hand, because the build that
wrote it did not record the GTT limit. This one is the output of a build
that does, on the same machine with the ttm pages_limit raised, so every
reading comes from llm-helper itself. The stock limit stays covered by the
unit tests, which build their readings rather than load this file.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues can give discrete GPUs RAM-based model budgets instead of VRAM limits, misstate Vulkan access requirements, and break shard commands when VLLM_BIN points to a virtualenv symlink.

3 open issues
What changed in this PR

This PR adds Linux support for AMD Strix Halo to llm-helper’s hardware detection and memory budgeting, and addresses setup and diagnostic gaps encountered when serving models with llama.cpp.

Changes:

  • Recognizes Strix Halo by PCI device ID and budgets model weights using the GPU’s GTT limit and firmware carve-out.
  • Adds environment-selected server binaries, configurable service scope, GPU-access and linger warnings, and inspect <repo>:<quant> support.
  • Documents AMD Linux setup, GTT tuning, and measured Vulkan/ROCm results, with regression tests and a hardware report.
File Description
internal/​hardware/​hardware.go Adds vendor-scoped PCI device matching and validation.
internal/​hardware/​hardware_test.go Tests PCI matching rules and invalid definitions.
internal/​detect/​remote.go Collects and parses remote GTT limits.
internal/​detect/​hardware.go Adds PCI identity, GTT fields, and memory-pool selection.
internal/​detect/​hardware_linux.go Reads GTT limits from sysfs.
internal/​detect/​hardware_linux_internal_test.go Tests sysfs GTT readings.
internal/​detect/​gpus_internal_test.go Tests remote GTT parsing and backward compatibility.
internal/​detect/​fixtures_internal_test.go Checks PCI observations for overlapping machine definitions.
internal/​detect/​classify.go Classifies PCI-matched GPUs and displays sub-GB carve-outs.
internal/​detect/​budget.go Adds GTT-based budgeting with system-memory constraints.
internal/​detect/​budget_internal_test.go Tests GTT limits, carve-outs, and RAM fallback.
internal/​cli/​setup.go Honors environment binary paths and adds service-scope selection.
internal/​cli/​setup_internal_test.go Tests binary overrides, scope validation, and kept-config notices.
internal/​cli/​search.go Accepts quant-qualified repository names in inspect.
internal/​cli/​linger.go Defines warnings for user services without lingering.
internal/​cli/​linger_other.go Disables linger checks outside Linux.
internal/​cli/​linger_linux.go Checks lingering for the current Linux user.
internal/​cli/​linger_internal_test.go Tests linger warning conditions.
internal/​cli/​inspect_internal_test.go Tests repository parsing and missing-quant errors.
internal/​cli/​gttlimit.go Formats GTT diagnostics and limit-raising guidance.
internal/​cli/​gttlimit_internal_test.go Tests displayed limits and suggested page counts.
internal/​cli/​doctor_internal_test.go Tests recognized APUs and PCI overlap warnings.
internal/​cli/​devaccess.go Defines AMD device-permission warnings.
internal/​cli/​devaccess_other.go Disables device checks outside Linux.
internal/​cli/​devaccess_linux.go Checks GPU device permissions without opening devices.
internal/​cli/​devaccess_internal_test.go Tests denied, allowed, and missing device nodes.
internal/​cli/​detectcmds.go Integrates GTT, device-access, linger, and PCI diagnostics.
docs/​reference/​cli/​setup.md Updates generated setup help and service-scope documentation.
docs/​providers.md Documents AMD llama.cpp setup and measured serving results.
docs/​memory.md Explains Strix Halo budgeting and GTT tuning.
docs/​install.md Documents binary overrides, service scope, and lingering.
docs/​hardware.md Explains PCI matching and GTT detection.
CHANGELOG.md Records user-visible support and CLI changes.
catalogue/​hardware/​reports/​amd_strix_halo.json Adds measured Strix Halo detection readings.
catalogue/​hardware/​machines/​README.md Documents PCI-based machine definitions.
catalogue/​hardware/​machines/​amd_strix_halo.json Defines Strix Halo identity and unified-memory behavior.
catalogue/​engines/​llamacpp.json Updates descriptive metadata to mention Vulkan and ROCm.

Review effort: Balanced

Comment thread internal/detect/classify.go Resolved
Comment thread internal/cli/devaccess.go Resolved
Comment thread internal/cli/setup.go Resolved
A machine file that matches by PCI id and says memory is discrete set the
GPU's identity but left GPUMemBytes at 0, so Pool() fell back to system
RAM. A 24 GB AMD card in a 128 GB host got a RAM budget, and doctor no
longer warned about it. Use the VRAM amdgpu reports for the card.

Generated by Coder Agents on behalf of @bpmct.
The render-group warning said a server on ROCm or Vulkan would not find
the GPU whenever any node was denied. Vulkan only needs the render nodes,
so a user who can open those but not /dev/kfd was told Vulkan, the
backend we recommend on these boxes, would fail. Name ROCm alone then.

Generated by Coder Agents on behalf of @bpmct.
envBin recorded a VLLM_BIN symlink as given, while findVLLM resolves it.
Shard commands look for the venv's python and ray beside VLLM_BIN, so a
link into a venv broke them. Run vLLM paths through stableVLLMPath, and
keep other servers' links as named so a package manager's link survives
upgrades.

Generated by Coder Agents on behalf of @bpmct.
envBin now resolves a VLLM_BIN path, and on macOS t.TempDir() is under
/var, a link to /private/var, so the test's unresolved path no longer
matched and go (macos-latest) failed. Compare against the resolved path.

Generated by Coder Agents on behalf of @bpmct.
@DevelopmentCats
DevelopmentCats merged commit 5e85fc9 into main Nov 15, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants