Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,22 @@ the fact.

### Added

- Models from an external drive: `llm-helper export <org/model>... <drive-path>`
copies models from the Hugging Face cache onto a drive, and
`llm-helper import <drive-path>` copies them into the cache the server
reads, from a drive written by `export`, a copy of a cache, an
`hf download --local-dir` folder or a model folder. Files are verified
against their hashes as they are copied, an interrupted copy resumes,
incomplete models and a drive that is too small or FAT32 are refused, and
`import --recipe <id>` stages the models a tested recipe pins. See the new
"Offline machines and external drives" page.
- Offline machines: with `HF_HUB_OFFLINE=1` in `config.env`, or once the Hub
does not answer, `new`, `inspect` and `search` read the local cache instead
of the Hub, and a new profile tells its server to stay off the network
(`HF_HUB_OFFLINE=1` for vLLM and MLX, `--offline` for a llama.cpp that lists
it). Engines declare this in `offline` and `hub_cache_env`.
- `doctor` prints the model cache the server reads and its free space, and
warns when the shell's `HF_HOME` or `HF_HUB_CACHE` points elsewhere.
- Warm-up: a profile's `WARMUP_FILE` names a chat request that llm-helper
sends when `up`, `restart` or `switch` sees the server healthy, and that the router sends when a
server comes back, so the server has the client's fixed prompt cached
Expand Down
1 change: 1 addition & 0 deletions catalogue/engines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Each file here is one inference server llm-helper can launch, named after the en
To add an engine, copy the closest file to `<id>.json` and change it.
No Go code is needed: shared code acts on the engine's `capabilities`, never on its name.
`chat_template_kwargs_flag` names the flag that sets default chat template kwargs, which is how a profile turns a model's thinking on or off; leave it out if the engine has none.
`hub_cache_env` names the variable that sets the hub cache directly, and `offline` says how to keep the server off the network (`env`, `args`, and `probe_args` to check the installed server accepts them); leave either out if the engine has no such setting.
`notes.hybrid` is a sentence `inspect` prints for a model whose layers are not all attention; leave it out if there is nothing to say.
The file is checked when it loads, and the tests in `internal/engine` render every file here for every rank.

Expand Down
5 changes: 5 additions & 0 deletions catalogue/engines/llamacpp.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,11 @@
"with_key": "0.0.0.0",
"without_key": "127.0.0.1"
},
"hub_cache_env": "LLAMA_CACHE",
"offline": {
"args": ["--offline"],
"probe_args": ["--help"]
},
"api_key": "env:LLAMA_API_KEY",
"health": {
"path": "/health",
Expand Down
3 changes: 3 additions & 0 deletions catalogue/engines/mlx.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
"names": ["mlx_lm.server"]
},
"serve": ["{bin}", "--model", "{model}", "--port", "{port}"],
"offline": {
"env": { "HF_HUB_OFFLINE": "1" }
},
"api_key": "none",
"health": {
"path": "/health",
Expand Down
3 changes: 3 additions & 0 deletions catalogue/engines/vllm.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"--enable-prompt-tokens-details"
],
"optional_args": [["--kv-cache-memory-bytes", "{kv_cache_bytes}"]],
"offline": {
"env": { "HF_HUB_OFFLINE": "1" }
},
"api_key": "flag:--api-key",
"health": {
"path": "/health",
Expand Down
15 changes: 15 additions & 0 deletions docs/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@
"description": "Finding a model that fits this machine, and choosing a quant.",
"path": "./models.md"
},
{
"title": "Offline machines and external drives",
"description": "Move models between machines on a drive, and serve a machine that cannot reach the Hugging Face Hub.",
"path": "./offline.md"
},
{
"title": "Model types",
"description": "What a model's architecture needs to run correctly, and how to add or change it.",
Expand Down Expand Up @@ -133,6 +138,11 @@
}
]
},
{
"title": "llm-helper export",
"description": "Copy models from the Hugging Face cache onto an external drive",
"path": "./reference/cli/export.md"
},
{
"title": "llm-helper hardware",
"description": "List hardware definitions and report this machine's hardware",
Expand All @@ -150,6 +160,11 @@
}
]
},
{
"title": "llm-helper import",
"description": "Copy models from an external drive into the Hugging Face cache",
"path": "./reference/cli/import.md"
},
{
"title": "llm-helper inspect",
"description": "Show a model's license, weight size against this machine, and detected flags",
Expand Down
96 changes: 96 additions & 0 deletions docs/offline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
title: Offline machines and external drives
description: Move models between machines on a drive, and serve a machine that cannot reach the Hugging Face Hub.
---

A machine with no route to the Hugging Face Hub can still serve any model that is in its Hugging Face cache.
`llm-helper export` copies models from a machine that has them onto a drive, and `llm-helper import` copies them into the cache of the machine that does not.

## Move a model on a drive

On a machine that has the model, copy it onto the mounted drive:

```sh
llm-helper export Qwen/Qwen3-8B-FP8 /media/usb
```

On the offline machine, copy it into the cache the server reads, then create the profile:

```sh
llm-helper import /media/usb Qwen/Qwen3-8B-FP8
llm-helper new Qwen/Qwen3-8B-FP8
```

`import` lists the models it finds when you name none, and asks which to copy.
`--all` copies every complete model, and `--dry-run` shows what would be copied and stops.

Both commands:

- Refuse a model that is incomplete, because it would not load on the other machine. A missing file, a link to a blob that is not there, no weights, or a shard an index names but the folder lacks all count.
- Check every file against its recorded hash as it is copied, and fail on a damaged one without keeping it.
- Resume after an interruption, skip files already there, and never overwrite what is in the cache.
- Check the free space first, and say how much is needed.

`export` writes a plain copy of each file under `hub/` on the drive, with no links, so any filesystem can hold it.
FAT32 cannot hold a file over 4 GiB, so `export` refuses a model with one: format the drive as exFAT, or use ext4 or APFS.

## What `import` reads

`import` finds a Hugging Face cache at the path you give, or under it in `hub/`, `huggingface/hub/` or `.cache/huggingface/hub/`.
That covers a drive written by `export`, and a copy of `~/.cache/huggingface`.
A cache that shares one blob between several snapshots is read as it is.

Two other layouts record no repo id, so name it:

```sh
llm-helper import /media/usb/qwen --repo Qwen/Qwen3-8B-FP8 --revision <commit>
```

- A folder written by `hf download --local-dir` needs `--repo`. Its commit is read from the folder.
- A plain model folder needs `--repo` and a commit. A git clone supplies its own, and any other folder needs `--revision`, because llm-helper does not invent one.

## Which cache the server reads

`import` and `export` work on the cache the server reads, which is the one `llm-helper doctor` prints under `MODEL CACHE`.
The service sources `config.env` and `secrets.env`, not your login shell, so an `HF_HOME` exported in a shell is not the server's.
In order, the first of these that `config.env` sets wins:

1. `LLAMA_CACHE`, for llama.cpp only.
2. `HF_HUB_CACHE`, then `HUGGINGFACE_HUB_CACHE`.
3. `HF_HOME`, with `/hub` added.
4. `XDG_CACHE_HOME`, with `/huggingface/hub` added.
5. `~/.cache/huggingface/hub`.

`doctor` warns when your shell's `HF_HOME` or `HF_HUB_CACHE` points somewhere else.
`--cache <dir>` names a different cache for one command.

## Serve with no network

Set `HF_HUB_OFFLINE=1` in `config.env` on a machine that never reaches the Hub.
`llm-helper new`, `inspect` and `search` then read the cache instead of the Hub:

- `new` and `inspect` read the model's config, chat template and file list from the cached snapshot.
- `search` lists the cached models that match your terms.
- A model that is not in the cache is reported as not cached, instead of failing on a network error.

Without the setting, the same happens once a request cannot reach the Hub, and the Hub is tried again a minute later.

A profile made while offline tells its server to stay off the network, as the engine defines it:

| Engine | Setting in the new profile |
| --------- | ---------------------------------------------------------------------- |
| vLLM | `HF_HUB_OFFLINE=1` |
| MLX | `HF_HUB_OFFLINE=1` |
| llama.cpp | `--offline` in `EXTRA_ARGS`, only when the installed `--help` lists it |

For llama.cpp, set `LLAMA_CACHE` in `config.env` if the cache is not the default one, because `llama-server -hf` reads that first.
A build that does not list `--offline` is left without it, and `new` says so.
Existing profiles are not changed: add the setting to one with `llm-helper edit <profile>`.

## Tested recipes

A recipe launches containers that load a pinned revision from each node's cache and refuse to start when it is missing.
`llm-helper import <drive-path> --recipe <id>` copies exactly the models the recipe pins into the cache its engine reads, and fails when the drive has another revision.
Run it on each node.

See [Models and quantizations](./models.md) for choosing a model, and [Recipes](./recipes.md) for the launch configurations.
2 changes: 2 additions & 0 deletions docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,8 @@ A definition holds `schema_version` and the engine's fields, and says:
- `health`: the path that answers 200, and whether it does so only once the model is loaded (`ready`) or as soon as the process listens (`live`).
- `capabilities`: the few differences shared code acts on, such as `vllm_args` for vLLM's sizing and flag checks. Launching and health checks act on a capability, never on an engine's name.
`chat_template_kwargs_flag` names the flag that sets default chat template kwargs, which is how a profile turns a model's thinking on or off; see [Model types](./model-types.md).
- `hub_cache_env`: the variable that sets the hub cache directly, which the server reads before `HF_HUB_CACHE`. `llm-helper import` and `doctor` look there first.
- `offline`: how to keep the server off the network when a new profile is made for a machine that does not use the Hub, as `env` entries and `args`. `probe_args` is run against the installed server first, and `args` are added only when its output lists the first of them. See [Offline machines and external drives](./offline.md).
- `notes`: optional sentences `inspect` prints for some kinds of model.
`hybrid` is shown for a model whose layers are not all attention, such as one with linear-attention or Mamba layers.

Expand Down
2 changes: 2 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,9 @@ sharding, replica pools, recipes, optional Coder Agents sync).
| [`down`](cli/down.md) | Stop serving a profile (the only one running when none is named) |
| [`edit`](cli/edit.md) | Open a profile in $EDITOR and validate the result |
| [`engines`](cli/engines.md) | List the inference engines a profile or recipe can run on |
| [`export`](cli/export.md) | Copy models from the Hugging Face cache onto an external drive |
| [`hardware`](cli/hardware.md) | List hardware definitions and report this machine's hardware |
| [`import`](cli/import.md) | Copy models from an external drive into the Hugging Face cache |
| [`inspect`](cli/inspect.md) | Show a model's license, weight size against this machine, and detected flags |
| [`install-gguf`](cli/install-gguf.md) | Install the plugin that lets vLLM load GGUF models |
| [`list`](cli/list.md) | List saved model profiles |
Expand Down
39 changes: 39 additions & 0 deletions docs/reference/cli/export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
# Code generated by go run ./scripts/gendocs. DO NOT EDIT.
title: "llm-helper export"
description: "Copy models from the Hugging Face cache onto an external drive"
---

<!-- DO NOT EDIT | GENERATED CONTENT -->

Copy models from the Hugging Face cache the server reads onto an external drive, to
carry to a machine with no internet. The drive gets a plain copy of each file in a
hub/ folder, which any filesystem holds, and 'llm-helper import' reads it back.

Only complete models are copied, and each file is checked against its recorded
hash as it is copied. FAT32 cannot hold a file over 4 GiB; format the drive as
exFAT.

## Usage

```console
llm-helper export <org/model>... <drive-path> [flags]
```

## Options

### --cache

| | |
| ---- | -------- |
| Type | `string` |

hub cache directory to copy from (default: the one the server reads)

### --dry-run

| | |
| ---- | ------ |
| Type | `bool` |

show what would be copied and stop
75 changes: 75 additions & 0 deletions docs/reference/cli/import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
# Code generated by go run ./scripts/gendocs. DO NOT EDIT.
title: "llm-helper import"
description: "Copy models from an external drive into the Hugging Face cache"
---

<!-- DO NOT EDIT | GENERATED CONTENT -->

Copy models from an external drive, or any folder, into the Hugging Face cache the
server reads, so 'llm-helper new <org/model>' loads them with no download.

The drive can hold a Hugging Face cache (also one made by 'llm-helper export'), an
'hf download --local-dir' folder, or a model folder. The last two do not record
which repo they are, so name it with --repo (and --revision unless the folder is
a git clone).

Every file is checked against its recorded hash as it is copied, an interrupted
import resumes where it stopped, and nothing already in the cache is overwritten.
With --recipe the models go where that recipe's engine reads them.

## Usage

```console
llm-helper import <drive-path> [org/model...] [flags]
```

## Options

### --all

| | |
| ---- | ------ |
| Type | `bool` |

import every complete model on the drive

### --cache

| | |
| ---- | -------- |
| Type | `string` |

hub cache directory to import into (default: the one the server reads)

### --dry-run

| | |
| ---- | ------ |
| Type | `bool` |

show what would be copied and stop

### --recipe

| | |
| ---- | -------- |
| Type | `string` |

import the models a tested recipe pins, into the cache its engine reads

### --repo

| | |
| ---- | -------- |
| Type | `string` |

org/model of a folder that is not a Hugging Face cache

### --revision

| | |
| ---- | -------- |
| Type | `string` |

commit of a folder that is not a Hugging Face cache or git clone
Loading
Loading