Local Inference Stack

OpenAI-compatible LLM serving on one AMD RX 9070 XT. The repo contains the gateway, MCP server, Prometheus metrics, cost-aware routing, and a Nix deploy.

Results

RX 9070 XT, 16 GB VRAM, Vulkan. DeepSeek R1 Distill 7B Q4_K_M, 32k context, 8 slots, f16 KV. Three runs, no warm-up.

Measurement	Value
Generation	116 tok/s at 4k context; 107 tok/s on a 1.9k-token prompt
Prefill	4,055 tok/s (1,943 tokens, 479 ms); 2,400 tok/s at 981 tokens
Aggregate, 8 requests	424 tok/s (256 output tokens each; 417 with f16 KV)
First token	479 ms cold; 12 ms when the prompt is cached
VRAM	7.5 GB of 16 GB

Qwen3.8 9B at Q6_K runs at 60 tok/s in 12.1 GB. The 7B is faster and leaves more VRAM for KV; the 9B's Q6_K precision costs throughput and memory.

Per-op breakdown and bandwidth analysis: docs/perf-analysis.md.

Run

$LLAMA_SERVER -m $MODEL_DIR/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf \
  --alias deepseek-r1-7b --host 127.0.0.1 --port 8082 \
  --ctx-size 32768 --parallel 8 --n-gpu-layers 99 \
  -ctk f16 -ctv f16 --cache-reuse 256 --reasoning off

npm run gateway

The gateway speaks the OpenAI /v1 API.

curl -s http://localhost:3000/health | jq
curl -s http://localhost:3000/v1/models | jq '.data[].id'

curl -s http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-r1-7b","messages":[{"role":"user","content":"Explain a hash table in three sentences."}],"max_tokens":512}' \
  | jq '.choices[0].message.content'

stream: true gets SSE. R1 distills always emit a think block, so budget max_tokens for both reasoning_content and content.

Routing

model value	Behavior
a model name	that model, or a refusal if it is down
auto	cheapest healthy model
quality	cheapest model at or above that tier
max_cost_per_1m	ceiling; a model that cannot meet it is not used

The gateway reads prices from models.json, generated from bench/cost.json. Spend is available at /cost and in Prometheus.

Cost

Three models served from one card. Inputs are recorded in bench/cost.json: $0.15/kWh, $600 GPU, 3 years at 2,000 h/yr.

Model	tok/s	Watts	VRAM	$ / 1M tok (power)	$ / 1M tok (amortized)
Qwen3 4B Q4_K_M	144.8	252.4	6.4 GB	$0.073	$0.19
DeepSeek R1 7B Q4_K_M	105.2	289.3	8.2 GB	$0.115	$0.26
Qwen3.8 9B Q6_K	68.5	299.1	10.6 GB	$0.182	$0.41

The power figure is the marginal cost when the card is already running. The amortized figure includes hardware. A request on the 4B costs about half of the same request on the 9B.

MCP

npm run mcp starts an MCP server over stdio or HTTP/SSE with two tools: complete and health.

Tuning notes

• Parallel slots: 105 tok/s at 1 slot, 424 at 8, 459 at 16. Eight slots give 4x aggregate throughput; a 256-token answer takes 4.8 s under full load.
• KV cache: f16 is 2-6% faster than q8_0 at 4k and 8k. Use q8_0 only when the weights need the VRAM.
• Quantization: IQ4_XS is 9-11% faster single-stream but drops to 174 tok/s at 8 slots. Q4_K_M is the serving default.
• Speculative decoding: slower on this card in all four configurations tested. Details are in docs/perf-analysis.md.
• Context: llama-server splits the total window across slots. Keep ctx = np * 4096 or each request gets less context than expected.
• Prefix cache: --cache-reuse 256 cuts a repeated 981-token prompt from 413 ms to 155 ms.

Limits

• One GPU and one model family. No multi-node serving.
• Vulkan only. ROCm was not repeatable on this card.
• Cost numbers use the inputs above, not a cloud price list.
• The 7B and 4B models use f16 KV; the 9B uses q8_0 to fit.

Nix

nix run .#list
nix run .#start -- deepseek
nix run .#status
nix run .#stop -- deepseek

Model settings live in models.nix. The runner regenerates the wrapper on the next nix run.

License

MIT