Repository navigation
hardware: size and run AMD Strix Halo under Linux - #70
Merged
Merged
Conversation
amdgpu gives a GPU no name: an AMD device is only its PCI vendor and device ids, so a machine file had no way to recognise one, and an AMD APU that shares system memory was sized as a CPU-only box. A machine's match can now list pci_devices, which must hold together with its vendor (device ids are only unique within one) and with gpu_name when both are given. Matching stays data: no GPU is named in Go.
A Ryzen AI Max read as a CPU-only box: amdgpu reports only a 512 MB carve-out as VRAM, nothing names the GPU, so doctor warned that only NVIDIA and Apple GPUs are sized and the budget was the CPU backend's share of RAM. The GPU actually loads weights into GTT, system memory up to the limit the ttm module sets, which amdgpu reports in mem_info_gtt_total. The amd_strix_halo machine matches PCI device 0x1586 (one id for the 8060S, 8050S and 8040S) and is unified. The probe now reads mem_info_gtt_total, locally and over SSH, and a unified GPU with a GTT reading is sized against the limit plus its carve-out, held to RAM less the OS when the limit is raised close to all of it. With the box's numbers, the stock 101 GB limit budgets about 87 GB and the raised 120 GB one about 105 GB, which matches a 103 GB model needing the raise. The report is the box's readings from 0.6.1-dev.63314bb with gtt_bytes added from sysfs, since that build did not read it. A carve-out under a gigabyte is shown in MB instead of "0 GB".
On a Strix Halo the GTT limit is what bounds the model, and the stock one (101 GB on a 128 GB box) leaves memory idle: a 103 GB model only fit after raising ttm's pages_limit by hand. Doctor now prints the limit beside the GPU and, when it is more than 4 GiB below 90% of RAM, the modprobe line, initramfs rebuild and reboot that raise it, the way metal-limit does for the wired limit on macOS. 90% is what served the 103 GB model with about 15 GB still free. It only prints the steps. Writing modprobe.d needs root and a bad value is found out at the next boot, so llm-helper does not do it.
llama.cpp's ROCm build failed on a Strix Halo with "no ROCm-capable device is detected", which says nothing about the cause: the user was not in the render group, and /dev/kfd and /dev/dri/renderD128 are root:render 0660. On Linux, when an amdgpu GPU is on the bus, doctor now checks those nodes with access(2), which needs no open (opening /dev/kfd sets up a GPU context), and names the ones this user cannot use with the usermod fix and the re-login it needs. A node that does not exist is not reported, since Vulkan serves without KFD. The rule takes the device directory and the check as arguments, so it is tested without devices.
On a Debian box setup found /usr/bin/llama-server, the distribution's 0.2.0 build, and ignored LLAMA_SERVER_BIN in the environment, so the official release had to be put in config.env by hand. lookBin now takes the server the environment names under the engine definition's bin.key first, for every engine, including VLLM_BIN when a rerun fills in a missing one. A bare name is looked up on PATH and recorded as a full path. A key that names nothing executable stops setup before it writes anything, rather than falling back to the server the user set it to avoid. An environment variable rather than a flag, because the key is already how config.env names the server.
setup always wrote SERVICE_SCOPE=system on Linux. On a box where sudo asks for a password, every system-scope unit operation fails when run without a terminal, so the scope had to be changed in config.env by hand before the first up. --service-scope system|user picks it for a new config, like --provider. Left empty it is still the platform's default, which keeps the generated CLI reference the same on Linux and macOS. On a kept config that says otherwise, setup names the line to change rather than rewriting the file. The config text moves into newConfigEnv so what it writes can be tested.
With SERVICE_SCOPE=user, each model's unit lives in the user's systemd manager, which systemd starts at the user's first login and stops at the last logout unless the user lingers. A box set up that way served nothing after a reboot until 'sudo loginctl enable-linger' was run. up already tries to enable lingering, but without sudo that can fail with only a note. Doctor on Linux now checks /var/lib/systemd/linger for the user, which needs no loginctl call, and names the command when the file is missing. A machine without /var/lib/systemd is not checked.
A llama.cpp profile's MODEL is <repo>:<quant>, and pasting it into inspect sent the whole string to the Hub as the repo id, which answered with "hub lookup: HTTP 401". inspect now looks up the repo and, given a quant, checks the repo holds it and prints its size against the budget, or names the quants it does hold. --json prints the repo's detail as before.
The llama.cpp section said Metal, CUDA or the CPU and offered only Homebrew. On a Strix Halo box the working setup was the release's Vulkan or ROCm tarball, the render group, LLAMA_SERVER_BIN at setup, the user scope with lingering, and a raised GTT limit; each of those cost a manual step, so they are now written down in order. The example records the box's numbers for DeepSeek V4 Flash Vision UD-IQ3_XXS with both builds, beside the Mac Studio ones, and the note that llama.cpp's token counters do not move while it reads a long prompt.
Setup never edits an existing config.env, so a second run with LLAMA_SERVER_BIN (or any provider's bin key) set in the environment did nothing and printed nothing, even when the key named a path that does not exist. It looked as if the new server had been recorded. Setup now prints a note for each provider bin key set in the environment that differs from config.env, or that names nothing runnable, with the line to put in config.env by hand. It still does not change the file.
The report's gtt_bytes was read from sysfs by hand, because the build that wrote it did not record the GTT limit. This one is the output of a build that does, on the same machine with the ttm pages_limit raised, so every reading comes from llm-helper itself. The stock limit stays covered by the unit tests, which build their readings rather than load this file.
There was a problem hiding this comment.
🟡 Changes recommended
Active issues can give discrete GPUs RAM-based model budgets instead of VRAM limits, misstate Vulkan access requirements, and break shard commands when VLLM_BIN points to a virtualenv symlink.
3 open issues
What changed in this PR
This PR adds Linux support for AMD Strix Halo to llm-helper’s hardware detection and memory budgeting, and addresses setup and diagnostic gaps encountered when serving models with llama.cpp.
Changes:
- Recognizes Strix Halo by PCI device ID and budgets model weights using the GPU’s GTT limit and firmware carve-out.
- Adds environment-selected server binaries, configurable service scope, GPU-access and linger warnings, and
inspect <repo>:<quant>support. - Documents AMD Linux setup, GTT tuning, and measured Vulkan/ROCm results, with regression tests and a hardware report.
| File | Description |
|---|---|
internal/hardware/hardware.go |
Adds vendor-scoped PCI device matching and validation. |
internal/hardware/hardware_test.go |
Tests PCI matching rules and invalid definitions. |
internal/detect/remote.go |
Collects and parses remote GTT limits. |
internal/detect/hardware.go |
Adds PCI identity, GTT fields, and memory-pool selection. |
internal/detect/hardware_linux.go |
Reads GTT limits from sysfs. |
internal/detect/hardware_linux_internal_test.go |
Tests sysfs GTT readings. |
internal/detect/gpus_internal_test.go |
Tests remote GTT parsing and backward compatibility. |
internal/detect/fixtures_internal_test.go |
Checks PCI observations for overlapping machine definitions. |
internal/detect/classify.go |
Classifies PCI-matched GPUs and displays sub-GB carve-outs. |
internal/detect/budget.go |
Adds GTT-based budgeting with system-memory constraints. |
internal/detect/budget_internal_test.go |
Tests GTT limits, carve-outs, and RAM fallback. |
internal/cli/setup.go |
Honors environment binary paths and adds service-scope selection. |
internal/cli/setup_internal_test.go |
Tests binary overrides, scope validation, and kept-config notices. |
internal/cli/search.go |
Accepts quant-qualified repository names in inspect. |
internal/cli/linger.go |
Defines warnings for user services without lingering. |
internal/cli/linger_other.go |
Disables linger checks outside Linux. |
internal/cli/linger_linux.go |
Checks lingering for the current Linux user. |
internal/cli/linger_internal_test.go |
Tests linger warning conditions. |
internal/cli/inspect_internal_test.go |
Tests repository parsing and missing-quant errors. |
internal/cli/gttlimit.go |
Formats GTT diagnostics and limit-raising guidance. |
internal/cli/gttlimit_internal_test.go |
Tests displayed limits and suggested page counts. |
internal/cli/doctor_internal_test.go |
Tests recognized APUs and PCI overlap warnings. |
internal/cli/devaccess.go |
Defines AMD device-permission warnings. |
internal/cli/devaccess_other.go |
Disables device checks outside Linux. |
internal/cli/devaccess_linux.go |
Checks GPU device permissions without opening devices. |
internal/cli/devaccess_internal_test.go |
Tests denied, allowed, and missing device nodes. |
internal/cli/detectcmds.go |
Integrates GTT, device-access, linger, and PCI diagnostics. |
docs/reference/cli/setup.md |
Updates generated setup help and service-scope documentation. |
docs/providers.md |
Documents AMD llama.cpp setup and measured serving results. |
docs/memory.md |
Explains Strix Halo budgeting and GTT tuning. |
docs/install.md |
Documents binary overrides, service scope, and lingering. |
docs/hardware.md |
Explains PCI matching and GTT detection. |
CHANGELOG.md |
Records user-visible support and CLI changes. |
catalogue/hardware/reports/amd_strix_halo.json |
Adds measured Strix Halo detection readings. |
catalogue/hardware/machines/README.md |
Documents PCI-based machine definitions. |
catalogue/hardware/machines/amd_strix_halo.json |
Defines Strix Halo identity and unified-memory behavior. |
catalogue/engines/llamacpp.json |
Updates descriptive metadata to mention Vulkan and ROCm. |
Review effort: Balanced
A machine file that matches by PCI id and says memory is discrete set the GPU's identity but left GPUMemBytes at 0, so Pool() fell back to system RAM. A 24 GB AMD card in a 128 GB host got a RAM budget, and doctor no longer warned about it. Use the VRAM amdgpu reports for the card. Generated by Coder Agents on behalf of @bpmct.
The render-group warning said a server on ROCm or Vulkan would not find the GPU whenever any node was denied. Vulkan only needs the render nodes, so a user who can open those but not /dev/kfd was told Vulkan, the backend we recommend on these boxes, would fail. Name ROCm alone then. Generated by Coder Agents on behalf of @bpmct.
envBin recorded a VLLM_BIN symlink as given, while findVLLM resolves it. Shard commands look for the venv's python and ray beside VLLM_BIN, so a link into a venv broke them. Run vLLM paths through stableVLLMPath, and keep other servers' links as named so a package manager's link survives upgrades. Generated by Coder Agents on behalf of @bpmct.
envBin now resolves a VLLM_BIN path, and on macOS t.TempDir() is under /var, a link to /private/var, so the test's unresolved path no longer matched and go (macos-latest) failed. Compare against the resolved path. Generated by Coder Agents on behalf of @bpmct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Gets llm-helper working on an AMD Ryzen AI Max (Strix Halo) box under Linux. I moved a 128 GB Strix Halo from a hand-written systemd unit to llm-helper. It serves DeepSeek-V4-Flash-Vision
UD-IQ3_XXSon llama.cpp's Vulkan build.setup,new,setandrestartworked, but only after a few manual steps. llm-helper also sized the box as a CPU-only machine with a 0 GB GPU. This PR fixes those gaps.Changes
match.pci_devices, with a vendor). Newamd_strix_halomachine (unified memory) and a measured report.mem_info_gtt_total) and size the weights budget from it, not from vLLM's CPU backend. A 536 MB carve-out now reads as 536 MB, not 0 GB.ttm pages_limitsteps to raise it when it's well under RAM./dev/kfdor/dev/dri/renderD*(the render group).SERVICE_SCOPE=userhas no linger.LLAMA_SERVER_BIN. Before, it picked Debian's ancient/usr/bin/llama-server, which can't run this model.--service-scope system|user, because system scope needs sudo, and sudo on this box asks for a password.<repo>:<quant>. Before, it got a 401 from the Hub.On the box (128 GB, GTT raised to 112 GiB), idle, llama.cpp b12364,
-c 1048576 -np 4:Tool calls and images work. Before raising the limit, the stock GTT limit (94 GiB) was too small for this 103 GB quant.
doctornow says so and prints the fix.Tested:
gofmt -l .is empty.go vet ./...andgo test -race ./...pass.On the real box with a build of
9adb7af:hardware reportshowsamd_strix_haloandgtt_bytes.doctorshows the GTT line and a ~105 GB budget, with no false warnings.nobody,doctorgives the render group warning.setuppicks upLLAMA_SERVER_BINand--service-scope user. It rejects/nopeand--service-scope users.inspect repo:quantworks.The last two commits came after that run. The note for a kept config.env and the report swap are unit-tested only. The linger warning is unit-tested only, because turning linger off would stop the running model.
What I hit on the box, in order
doctor: "AMD gpu ... present, but llm-helper only sizes NVIDIA and Apple GPUs". GPU 0 GB, and the budget came from vLLM's CPU backend.setup --provider llamacpppicked/usr/bin/llama-server(Debian llama.cpp 0.2.0) and ignoredLLAMA_SERVER_BIN. I fixed config.env by hand.setupwroteSERVICE_SCOPE=system, and sudo needs a password, so I changed it touserand ranloginctl enable-linger.rendergroup.inspect <repo>:<quant>got HTTP 401.new --quant UD-IQ3_XXS,set EXTRA_ARGS=...andrestartworked as they are. llama-server found the mmproj with-hf ... --offline.Generated by Coder Agents on behalf of @bpmct.