Local LLM training monorepo

Python FastAPI backend that runs LoRA supervised fine-tuning (TRL SFTTrainer + PEFT) on OpenAI-style messages JSONL, plus a Next.js UI to validate data, preview configuration, start a single local job at a time, stream logs (SSE), cancel runs, and download a zip of artifacts.

GGUF conversion and Ollama import are documentation-only in this repo (see appendix).

Prerequisites

• Python 3.10+ (3.11 recommended). This repo was tested with a local venv under backend/.venv.
• PyTorch installed for your platform (pytorch.org — pick CUDA, CPU, or MPS).
• Node.js LTS and npm (for the frontend; see frontend/package.json for pinned Next.js / React versions).
• Disk space for the base model (often many GB), uploaded datasets, and checkpoints.
• A local Hugging Face–style model folder (config.json + weights) for base_model_path. Download with Hugging Face tooling or copy from elsewhere; the UI does not ship multi‑GB base weights for you.

Backend Python dependencies

From the repo root:

cd backend
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

requirements.txt includes rich (required by recent trl imports). If pip install torch fails on your OS, follow PyTorch’s official install command first, then install the rest.

Frontend dependencies

cd frontend
npm install

Configuration

• Root .env.example — copy to backend/.env (or export vars) before running uvicorn from backend/ (Pydantic loads .env from the process working directory). Key fields:
  • DATA_DIR — where jobs, uploads, and artifacts live (default ./data).
  • CORS_ORIGINS — comma-separated origins; include your Next.js origin (default http://localhost:3000).
  • USE_FAKE_TRAINER — set to 1 for a dry/demo run that writes placeholder artifacts without loading TRL training (useful for wiring tests).
  • ENABLE_JOB_PLAYGROUND — set to 0 to disable POST /jobs/{id}/playground/compare and hide the job-page compare UI (RAM-heavy; loads checkpoints per request).
  • MODEL_PRESETS_SCAN_DIR — optional directory (path is resolved relative to the backend cwd) whose subfolders containing config.json appear as base model shortcuts in the UI via GET /system/model-shortcuts.
  • BASE_MODEL_SHORTCUTS — optional JSON array of explicit shortcuts, e.g. [{"label":"Llama 8B","path":"/abs/path/to/model"}]. Entries are merged with scan results; invalid JSON is surfaced in the API response’s parse_error field.
• frontend/.env.local — copy from frontend/.env.example; set NEXT_PUBLIC_API_URL to the API base (e.g. http://localhost:8000).

Run locally

Terminal 1 — API

cd backend
source .venv/bin/activate
export DATA_DIR="$(pwd)/../data"          # optional explicit path
export USE_FAKE_TRAINER=0                  # set to 1 for demo trainer
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

Terminal 2 — UI

cd frontend
npm run dev

Open http://localhost:3000. The home page loads GET /system/capabilities (CUDA / MPS / CPU, best‑effort VRAM on CUDA, warnings) and GET /system/model-shortcuts (preset base models from MODEL_PRESETS_SCAN_DIR / BASE_MODEL_SHORTCUTS). GET /health returns {"status":"ok"} for simple uptime checks.

Workflow

1. Upload — POST /datasets/upload (multipart file) stores under DATA_DIR/uploads/ and returns a relative path, or set dataset_path manually to a file under DATA_DIR (e.g. uploads/<id>.jsonl). Allowed upload suffixes are .jsonl or .json; content must still be JSON Lines (one chat record per line), same as validation expects.
2. POST /datasets/validate (or Validate sample in the UI) — parses the first N lines, checks messages roles/content, optional tokenizer length estimate when base_model_path is set.
3. Fill base_model_path (absolute path on the machine running the API) and hyperparameters. Use shortcuts from the UI when configured.
4. Preview / validate — POST /jobs/preview runs the same validators plus device/dtype warnings.
5. Approve & start — POST /jobs creates the job and starts training. Only one non‑terminal job is allowed globally (409 with active_job_id if busy).
6. Job page streams GET /jobs/{id}/events (SSE), polls GET /jobs/{id}, and loads loss curves from GET /jobs/{id}/training-metrics (backed by output/metrics.jsonl when present). Cancel via POST /jobs/{id}/cancel (cooperative stop).
7. When finished, GET /jobs/{id}/artifacts.zip contains config.resolved.json, logs, and output/ (adapter under output/adapter/, optional merged HF model under output/merged/, plus output/ollama/ — Modelfiles and README for optional GGUF/Ollama export). The job UI calls GET /jobs/{id}/ollama-recipe for that recipe as JSON. POST /jobs/{id}/playground/compare powers the job-page side‑by‑side text boxes (Transformers inference on the API host — no GGUF required). GET /jobs/{id}/playground/hints returns the first system message from the training JSONL (if any), which the UI can use when aligning the playground with dataset-style prompting.

Dataset format (JSONL)

One JSON object per line, OpenAI-style chat messages (roles: system, user, assistant, tool):

{"messages":[{"role":"user","content":"Say hello."},{"role":"assistant","content":"Hello!"}]}

Each record must include at least one assistant message with non-empty content.

Example data

Under examples/:

• examples/sample.jsonl — minimal JSONL you can upload or point dataset_path at (after copying under DATA_DIR or via upload).
• examples/nascar_company_dataset.jsonl — larger fictional retail/B2B chat set; regenerate with examples/generate_nascar_company_dataset.py.

Sample TrainingConfig (JSON body)

{
  "job_name": "my-run",
  "base_model_path": "/models/Meta-Llama-3-8B-Instruct",
  "dataset_path": "uploads/abc123.jsonl",
  "max_seq_length": 2048,
  "lora_r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "lora_target_modules": null,
  "learning_rate": 0.0002,
  "num_train_epochs": 1,
  "max_steps": null,
  "per_device_train_batch_size": 1,
  "gradient_accumulation_steps": 4,
  "warmup_ratio": 0.03,
  "logging_steps": 10,
  "save_steps": "epoch",
  "seed": 42,
  "fp16": false,
  "bf16": true,
  "device_preference": "auto",
  "merge_adapter_after_train": false
}

Paths:

• dataset_path must resolve under DATA_DIR (relative paths are rooted there).
• base_model_path can be any readable directory on the host with config.json (not restricted to DATA_DIR).

job_name must start with a letter or digit and may contain only letters, digits, ., _, and - (max 128 characters).

Troubleshooting

• CUDA OOM / MPS errors — lower max_seq_length, per_device_train_batch_size, or lora_r; increase gradient_accumulation_steps; try fp16 instead of bf16 on MPS; enable CPU only as a last resort.
• bf16 on CPU/MPS — may be unsupported or flaky; switch to fp32 (both fp16 and bf16 false) or fp16 where appropriate.
• Slow CPU training — expected; use a GPU or a smaller model for experimentation.
• 409 Active job — wait for the running job to finish, cancel it, or inspect data/jobs/active.json under DATA_DIR if the process crashed mid‑run (you may clear stale state manually in development).
• Job playground & system prompts — In POST /jobs/{id}/playground/compare, omitting system_prompt (or sending JSON null) uses the first system message found in the job’s training JSONL for both base and fine-tuned generations. An empty string uses no system message on either side. Any other string is used as the same explicit system prompt on both sides (useful for matching production prompts; a fixed system can mask what the adapter alone changed). The job UI defaults to no system (""); switch to dataset mode there to omit the field and match API null behavior.
• Job playground OOM / slow — POST /jobs/{id}/playground/compare reloads weights each request; use a smaller base model, lower max new tokens, or turn off bf16 / use fp32. Set ENABLE_JOB_PLAYGROUND=0 to hide the feature.

Appendix: Ollama / GGUF (documentation only)

This project outputs Hugging Face–compatible weights (LoRA adapter and optionally merged full weights). Ollama typically consumes GGUF. Completed runs also write output/ollama/ and expose GET /jobs/{job_id}/ollama-recipe so you can copy Modelfiles and paired ollama run … commands for side‑by‑side testing (you still produce the .gguf files locally).

High-level steps (details depend on model family and llama.cpp support):

1. Convert the base and merged HF checkpoints with llama.cpp tooling (see upstream docs and scripts such as convert_hf_to_gguf.py in the llama.cpp repository).
2. Create two Ollama models using Modelfiles — see Ollama’s Modelfile documentation for directives like FROM pointing at a .gguf file (Ollama Modelfile docs).

There is no bundled GGUF conversion binary in this repo; conversion stays external.