Skip to content

cli: import and export models on an external drive, and work offline - #72

Open
DevelopmentCats wants to merge 5 commits into
mainfrom
devcats/offline-drive
Open

DevelopmentCats wants to merge 5 commits into
mainfrom
devcats/offline-drive

Conversation

@DevelopmentCats

Copy link
Copy Markdown
Collaborator

Lets someone move models to a machine on an external drive and serve them without the Hub, for offline or air-gapped machines.

Commands

  • llm-helper export <org/model>... <drive-path> copies models from the Hugging Face cache onto a drive. The drive holds real files with no symlinks, so exFAT works.
  • llm-helper import <drive-path> copies them into the cache the server reads. It accepts a drive written by export, a copy of a cache, an hf download --local-dir folder (with --repo), or a plain model folder (with --repo and a commit from --revision or git HEAD). import --recipe <id> stages the models a recipe pins.

Files are verified against their hashes as they are copied. An interrupted copy resumes. Incomplete models, a drive that is too small, and FAT32 (4 GiB file limit) are refused. The result is the standard blobs/ + snapshots/ + refs/main layout, so huggingface_hub treats it as its own cache.

Offline mode

With HF_HUB_OFFLINE=1 in config.env, or once the Hub doesn't answer and the cache holds a model, new, inspect and search read the local cache. A new profile tells its server to stay off the network. How is engine data: offline in catalogue/engines/*.json (HF_HUB_OFFLINE=1 for vLLM and MLX; --offline for llama.cpp, added only if the installed llama-server --help lists it). hub_cache_env names an engine's own cache variable (LLAMA_CACHE). No engine names in Go.

doctor prints the cache the server reads and its free space, and warns when the shell's HF_HOME/HF_HUB_CACHE differ from what the service sees.

Design notes

  • The server's cache comes from config.env/secrets.env, which is what the service sources, not the login shell.
  • refs/main is written only when absent; a different existing ref is left alone with a note.
  • trees/<commit>.json is copied when present, never synthesized.
  • Partial copies are *.llm-helper.incomplete in blobs/, so an hf partial is never touched.
  • Repo ids, revisions and blob names read from a drive are validated against strict patterns before they become paths.
  • Shared behavior lives in internal/control (HubCache, HubOffline, KeepOffline) and is used by both the CLI and the panel's new-profile path.
  • The Hub fallback lasts one minute, so a long-running panel recovers after a network drop.

Verification

  • gofmt, go vet ./..., go test -race ./..., gendocs, builds for linux/amd64, linux/arm64, darwin/arm64, prettier 3.9.6: all pass.
  • Round trip with real huggingface_hub 0.36.2 on scratch dirs with trl-internal-testing/tiny-Qwen3ForCausalLM: cache → export → drive (0 symlinks) → import into a new cache (12 symlinks); a second import skipped all files; snapshot_download(local_files_only=True) and scan_cache_dir read the imported cache; an online snapshot_download against it downloaded nothing. --local-dir import works with --repo and fails clearly without it. Offline search/inspect/doctor answered from the cache.

Not verified

  • llama.cpp --offline on a real build (only a fake --help in tests). Offline loading from the cache had upstream bugs around April 2026 (Misc. bug: --offline flag broken ggml-org/llama.cpp#21302, #21364).
  • macOS, and real exFAT/FAT32 drives. FAT32 and free-space behavior is covered by test hooks.
  • new end to end, since it starts services; offline profile creation is tested through KeepOffline and profile.Create per engine.

Follow-ups

Panel import/export, an --update-ref flag, copying to several nodes in one command, serving directly off the drive, and accounting for a profile's own HF_HOME override.

Note

This conflicts mechanically with #71 (both add fields/methods at the same place in internal/engine/engine.go, registry.go, engine_test.go, and both add CHANGELOG entries). Whichever merges second gets rebased.

Engines say how their server stays off the network and which variable sets its
hub cache, so no code has to name an engine to do it.
Reads the cache layout all engines share, and copies complete models between
caches with every file checked against its hash.
new, inspect and search answer from the cache when the Hub is not used or does
not answer, instead of failing on the network.
import and export move models on a drive. A new profile made offline tells its
server to stay off the network, and doctor shows which cache the server reads.
Explains moving models on a drive, which cache the server reads, and what a
profile made offline sets.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 16:33
Comment thread internal/hf/local.go
if !ok {
return nil, fmt.Errorf("%s: %w", repo, ErrOffline)
}
f, err := os.Open(filepath.Join(dir, filepath.FromSlash(name)))
return "", "", false
}
dir = filepath.Join(RepoDir(hub, repo), "snapshots", rev)
if st, err := os.Stat(dir); err != nil || !st.IsDir() {
// mainRevision reads refs/main, or names the one snapshot a repo has when it
// has no ref.
func mainRevision(repoDir string) (string, error) {
if b, err := os.ReadFile(filepath.Join(repoDir, "refs", "main")); err == nil {
return rev, nil
}
}
entries, err := os.ReadDir(filepath.Join(repoDir, "snapshots"))

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Predictable temporary-file paths can follow attacker-controlled symlinks and overwrite files outside the cache before hash verification.

11 open issues
What changed in this PR

This PR adds drive-based model transfer and offline cache access to llm-helper, connecting both to the existing engine definitions and shared profile-creation logic.

Changes:

  • Adds import/export commands with hash checks, resumable copies, space checks, and recipe staging.
  • Adds local-cache fallback for Hub readers and engine-specific offline settings for new profiles.
  • Adds cache diagnostics, regression tests, and user documentation.
File Description
internal/​web/​models.go Applies offline settings during panel profile creation.
internal/​resolve/​offline.go Adds engine-declared offline settings to profile plans.
internal/​resolve/​offline_internal_test.go Tests offline environment and argument handling.
internal/​hfcache/​space_other.go Provides unsupported-platform filesystem fallbacks.
internal/​hfcache/​space_linux.go Detects Linux free space and FAT filesystems.
internal/​hfcache/​space_darwin.go Detects macOS free space and FAT filesystems.
internal/​hfcache/​scan.go Scans caches, download directories, and model folders.
internal/​hfcache/​hfcache.go Defines cache paths, identifiers, and offline detection.
internal/​hfcache/​hfcache_internal_test.go Tests scanning, copying, integrity, and filesystem checks.
internal/​hfcache/​copy.go Implements model copying, hashing, and capacity checks.
internal/​hf/​local.go Adds local-cache readers and temporary Hub fallback.
internal/​hf/​local_internal_test.go Tests offline reads and Hub recovery.
internal/​hf/​hf.go Integrates cache fallback into Hub operations.
internal/​hf/​arch.go Reads model configuration through the shared fallback.
internal/​engine/​registry.go Extends the provider interface with cache and offline settings.
internal/​engine/​engine.go Defines and validates engine offline metadata.
internal/​engine/​engine_test.go Tests offline declarations and validation.
internal/​control/​hubcache.go Shares cache resolution and offline profile setup.
internal/​control/​hubcache_internal_test.go Tests cache selection and offline profile creation.
internal/​cli/​search.go Identifies offline results and empty-cache matches.
internal/​cli/​root.go Registers transfer commands and initializes cache access.
internal/​cli/​root_internal_test.go Updates the expected command set.
internal/​cli/​recipelaunch.go Adds drive-import guidance for missing recipe models.
internal/​cli/​profiles.go Applies offline settings during CLI profile creation.
internal/​cli/​hfcache.go Implements transfer commands and model selection.
internal/​cli/​hfcache_test.go Tests transfers, recipe staging, and offline search.
internal/​cli/​engines_internal_test.go Checks recipe import guidance.
internal/​cli/​doctor_internal_test.go Tests cache reporting and shell mismatch warnings.
internal/​cli/​detectcmds.go Reports cache location, space, and offline status.
docs/​reference/​cli/​import.md Adds the generated import reference.
docs/​reference/​cli/​export.md Adds the generated export reference.
docs/​reference/​cli.md Lists the new transfer commands.
docs/​providers.md Documents engine cache and offline fields.
docs/​offline.md Explains drive transfers and offline operation.
docs/​manifest.json Routes the new documentation pages.
CHANGELOG.md Records transfer, offline, and diagnostic features.
catalogue/​engines/​vllm.json Declares vLLM’s offline environment setting.
catalogue/​engines/​README.md Documents the new engine metadata fields.
catalogue/​engines/​mlx.json Declares MLX’s offline environment setting.
catalogue/​engines/​llamacpp.json Declares llama.cpp’s cache variable and probed offline flag.

Review effort: Balanced

Comment thread internal/hfcache/copy.go
if st, err := in.Stat(); err != nil || st.Size() != size {
return "", fmt.Errorf("%s changed size while being read", src)
}
out, err := os.OpenFile(tmp, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0o644)
Comment thread internal/cli/hfcache.go
Comment on lines +289 to +290
case m.Revision != p.Revision:
return nil, fmt.Errorf("the recipe pins %s at %s, but the drive has %s", p.ID, p.Revision, m.Revision)
Comment thread internal/hf/hf.go
Comment on lines +641 to +643
if hub := canFallBack(ctx, err); hub != "" && page == 0 {
return localTree(hub, repo)
}
Comment thread internal/hf/local.go
if err != nil {
return &http.Response{StatusCode: http.StatusNotFound, Body: http.NoBody}, nil
}
return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(f), ContentLength: -1}, nil
Comment thread internal/hf/local.go
if _, _, ok := hfcache.Snapshot(hub, repo); !ok {
return Model{}, fmt.Errorf("%s: %w", repo, ErrOffline)
}
return Model{ID: repo}, nil
Comment thread internal/hfcache/copy.go
Comment on lines +184 to +188
tmp := partialPath(repoDir, f.Path)
if _, err := copyFile(f.Abs, tmp, f.Etag, f.Size); err != nil {
return false, err
}
return true, os.Rename(tmp, dst)
Comment thread internal/hfcache/copy.go
Comment on lines +193 to +194
name := f.Etag
if name == "" || !sizeIs(filepath.Join(repoDir, "blobs", name), f.Size) {
Comment on lines +55 to +56
v = strings.ReplaceAll(v, "${HOME}", home)
return strings.ReplaceAll(v, "$HOME", home)
Comment thread internal/hfcache/scan.go
Comment on lines +221 to +223
if err != nil || d.IsDir() {
return nil
}
Comment thread internal/hfcache/scan.go
Comment on lines +313 to +315
for _, f := range m.Files {
if !strings.HasSuffix(f.Path, ".index.json") {
continue

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants