Virgil Lemma foundations

.codecov.yml

@@ -2,22 +2,11 @@ coverage:
   status:
     project:
       default:
-        target: 80%
-        threshold: 1%
+        target: 98%
+        threshold: 8%
     patch:
       default:
         target: 70%
 
 ignore:
-  # Hardware/native runtime paths need a separate Metal-backed integration gate.
-  - "go/*_darwin.go"
-  - "go/register_metal.go"
-  - "go/internal/metal/**"
-
-  # Adapter shells and sidecars are tested, but not part of the core library gate.
-  - "go/training.go"
-  - "go/mlxlm/**"
-  - "go/pkg/daemon/**"
-  - "go/pkg/memvid/cli/**"
-  - "go/cmd/**"
   - "go/tests/**"

.gitignore

@@ -1,18 +1,27 @@
 # Build artifacts
 build/
+bin/
 *.dylib
 *.so
 *.a
 
+# `go build ./go/cmd/mlx/` without -o lands the binary at repo root.
+# Convention is `go build -o bin/mlx` (bin/ already ignored above);
+# this catches the shortcut form too.
+/mlx
+
 # CMake
 CMakeCache.txt
 CMakeFiles/
 cmake_install.cmake
 Makefile
 
-# CMake install output (keep headers for Go module consumers)
-dist/*
-!dist/include/
+# CMake install output
+dist/
+
+# Local Go build/test shortcuts
+/go/mlx
+/*.test
 
 # IDE
 .idea/
@@ -22,6 +31,11 @@ dist/*
 # macOS
 .DS_Store
 
+# lthn/desktop frontend dist — copied at build time by
+# scripts/make-app-bundle.sh, embedded in cmd/mlx via go:embed.
+# Single source of truth lives in lthn/desktop/frontend/.
+go/cmd/mlx/frontend/dist/
+
 # Knowledge base
 KB/
 .core/

.gitmodules

@@ -22,3 +22,26 @@
 	path = external/go-io
 	url = https://github.com/dappcore/go-io.git
 	branch = dev
+[submodule "external/go-ai"]
+	path = external/go-ai
+	url = https://github.com/dappcore/go-ai.git
+	branch = dev
+[submodule "external/go-ml"]
+	path = external/go-ml
+	url = https://github.com/dappcore/go-ml.git
+	branch = dev
+[submodule "external/go-cgo"]
+	path = external/go-cgo
+	url = https://github.com/dappcore/go-cgo.git
+	branch = dev
+[submodule "external/go-i18n"]
+	path = external/go-i18n
+	url = https://github.com/dappcore/go-i18n.git
+	branch = dev
+[submodule "external/go-log"]
+	path = external/go-log
+	url = https://github.com/dappcore/go-log.git
+[submodule "external/go-store"]
+	path = external/go-store
+	url = https://github.com/dappcore/go-store.git
+	branch = dev

AGENTS.md

@@ -1,66 +1,11 @@
-# go-mlx Agent Guide
-
-This repository provides Go bindings, model loaders, and a research-grade
-training pipeline for MLX on Apple Silicon. Module: `dappco.re/go/mlx`.
-
-## Layout (post Mantis #1241)
-
-All Go code lives under `go/`:
-
-- `go/` — root package: public model, tokenizer, compute, training, eval,
-  distill, GRPO, hf-fit, merge, gguf-quantize, kv-snapshot, lora-fuse APIs
-- `go/internal/metal/` — CGO boundary to `mlx-c`; do not move CGO code out
-- `go/mlxlm/` — subprocess backend for Python `mlx-lm` (CGO-free; build tag
-  `nomlxlm` removes it)
-- `go/cmd/violet/` and `go/pkg/daemon/` — local Violet Unix-socket sidecar
-- `cpp/` — C++ side companion (CLion-side worktree)
-- `lib/mlx/` — upstream MLX submodule pinned at `v0.30.1`
-- `patches/` — local patches against `lib/mlx` (manual apply only)
-- `docs/`, `examples/` — markdown documentation and per-feature usage examples
-
-## Platform Boundaries
+Rule -2: Do not use `GOAL.md` or tracker files to record progress, savings, benchmark notes, proof, status, or "what got faster". Only remove completed task lines unless the user explicitly asks for tracker-file edits in the current turn.
 
-Files that need the native MLX runtime use `//go:build darwin && arm64`.
-Unsupported builds compile against the `*_stub.go` files and a stub
-`MetalAvailable() bool` that returns false. Do not move CGO code out of
-`go/internal/metal/`.
+Rule -1: `GOAL.md` and tracker files are compact task queues, not progress logs. Never read, add, preserve, or update tracking, proof, benchmark, savings, performance, status, changelog, or "what got faster" notes there unless the user explicitly asks for that tracker-file edit in the current turn; report evidence in chat or commits instead.
 
-## Conventions
-
-- UK English in code, comments, and docs (colour, organisation, behaviour)
-- SPDX header on every new file: `// SPDX-Licence-Identifier: EUPL-1.2`
-- Conventional commits: `type(scope): description` — scopes include `metal`,
-  `api`, `mlxlm`, `cpp`, `docs`, `repo`, `deps`
-- Co-Author trailer: `Co-Authored-By: Virgil <virgil@lethean.io>`
-- Use `dappco.re/go` core helpers for fmt, errors, JSON, filesystem, path,
-  env, byte buffers, and string ops — do not import the wrapped stdlib
-  packages directly
-
-## Test Patterns
-
-Tests are file-aware. Public functions and methods in `foo.go` have their
-Good, Bad, and Ugly triplets in `foo_test.go`, and runnable examples in
-`foo_example_test.go`. Native tests skip only when the local machine lacks
-the required Metal runtime or test model assets. Keep examples small and
-checkable so they document the public API without requiring heavyweight
-model downloads.
-
-## Sandboxing Notes
-
-Before handing off, run the repository gates from the brief with `GOWORK=off`.
-On sandboxed systems, set `GOCACHE` to a writable directory such as
-`/tmp/codex-go-mlx-cache` so Go can compile without touching the user
-cache. If the sandbox cannot resolve the bundled `mlx.metallib`, apply
-`patches/mlx-metallib-path.patch` inside `lib/mlx` to enable the
-`MLX_METALLIB_PATH` env-var override (not auto-applied).
+# go-mlx Agent Guide
 
-## What's Inside the Public Surface
+Module `dappco.re/go/mlx`; Go lives in `go/`.
 
-Beyond the inference path, the root package owns the research-grade
-pipeline: knowledge distillation (`RunKnowledgeDistillation`), GRPO
-(`RunGRPOReasoningTraining`), dataset eval (`RunModelEval`), LoRA fusion
-(`FuseLoRAIntoModelPack`), model merging (`MergeModelPacks`), native
-GGUF quantisation (`QuantizeModelPackToGGUF`), KV snapshots
-(`KVSnapshot.Save` / `LoadKVSnapshot`), and HuggingFace Hub metadata
-(`HuggingFaceModelSource`). See `docs/` and `examples/` for full
-walkthroughs.
+- Native: `go/pkg/native` stays CGO-free; CGO stays in `go/internal/metal`; `darwin && arm64`, macOS 26+.
+- Test env: repo `go.work`; `GOCACHE=/private/tmp/codex-go-mlx-cache`; `MLX_METALLIB_PATH=/Users/snider/Code/core/go-mlx/dist/lib/mlx.metallib`; `-ldflags "-extldflags=-mmacosx-version-min=26.0"`.
+- Style/tests: UK English; EUPL SPDX on new files; use `dappco.re/go` helpers; file-aware tests; native skips only for missing runtime/assets.

CLAUDE.md

@@ -12,30 +12,61 @@ Implements the `inference.Backend` and `inference.TextModel` interfaces from `da
 
 **darwin/arm64 only.** All CGO files carry `//go:build darwin && arm64`. A stub provides `MetalAvailable() bool` returning false on every other platform. Files that need the native MLX runtime use the build tag; non-Darwin builds compile against `*_stub.go` files.
 
-## Build & Test
+## How to use this repo — drive `task`, never hand-roll the build
+
+**This repo is driven by [Taskfile.yml](Taskfile.yml) (`go-task`). Run `task <target>` — do NOT reconstruct the build with bare `go build` / `go test` and manual env exports.** The Taskfile already bakes in everything the build needs:
+
+- `GOCACHE` (default `/private/tmp/codex-go-mlx-cache` — the repo's shared build cache; not a "codex env", just the default)
+- `GO_DARWIN_LDFLAGS = -extldflags=-mmacosx-version-min=26.0` (Metal 4 floor — the build fails without it)
+- `MLX_METALLIB_PATH = {{.ROOT_DIR}}/dist/lib/mlx.metallib`
+- `-tags metal_runtime` on the test path
+
+Hand-exporting those yourself is how you get a subtly-wrong build and waste a session. `task fmt; go-task --list` shows every target. The Go module lives in `go/`; the Taskfile `dir: go` handles the `cd` for you.
 
 ```bash
-# Build mlx-c C library (required on fresh checkout, ~2min on M3 Ultra)
-git submodule update --init --recursive
-go generate ./...
+# Build the binary (self-contained — embeds the gzipped metallib). Output: bin/lthn-mlx
+task build:lthn
 
-# Run all tests
-go test ./...
+# Compile the native engine's OWN fused Metal kernels (router-topk, q4 lm-head argmax, bf16)
+#   -> dist/lib/lthn_kernels.metallib, loaded beside mlx.metallib. Build after editing pkg/native/kernels/*.metal.
+task build:kernels
 
-# Run a single test
-go test -run TestRMSNorm_Good ./go/internal/metal/
+# Run the whole Go suite (metal_runtime tag + ldflags, on the GPU)
+task test
 
-# Run benchmarks
-go test -bench=. -benchtime=2s ./go/internal/metal/
+# fmt + vet + test
+task qa
 
-# Lint
-golangci-lint run ./...
+# Real coverage figure (metal_runtime + model_eval tags) -> /tmp/go-mlx-coverage.out
+task cov
+
+# C++ side (standalone lib/mlx build + the kernel-bridge tests). First run cold-builds MLX ~15 min.
+task test:cpp        # vendored MLX suite
+task test:cpp:kernels  # go-mlx's own Metal-kernel bridges
+
+# Clean
+task clean
+```
+
+**Fresh checkout:** `git submodule update --init --recursive`, then build the MLX C library + metallib (`go generate ./...` from `go/`, or the CMake path in `docs/build.md`) so `dist/lib/mlx.metallib` exists before `task build:lthn` can embed it. `dist/lib/` is gitignored (rebuilt per checkout); `dist/include/` headers are committed for module consumers.
+
+### The two engines, and trying them
+
+| Engine | Path | How to run |
+|--------|------|-----------|
+| **cgo metal** (mature: MTP, paged KV, cache modes) | `go/internal/metal/` | `./bin/lthn-mlx generate <model-dir>` / `serve` |
+| **no-cgo native** (`pkg/native` + `pkg/model` — the contract engine, current focus) | `go/pkg/native/` | add `-native`: `./bin/lthn-mlx generate -native …` |
+
+```bash
+# Try the native engine end-to-end (greedy, deterministic). Model = a gemma4 HF/MLX snapshot dir.
+./bin/lthn-mlx generate -native -temp 0 -max-tokens 64 -prompt "Explain RoPE in one sentence." \
+  ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e2b-it-4bit/snapshots/*/
 
-# Clean rebuild (if dist/ is stale)
-rm -rf build dist && go generate ./...
+# Serve it over the OpenAI/Anthropic/Ollama API, then curl /v1/chat/completions
+./bin/lthn-mlx serve -native <model-dir>          # default :11434
 ```
 
-The compiled libraries (`dist/lib/`) are gitignored and must be rebuilt on each fresh checkout. Headers in `dist/include/` are committed for Go module consumers. On sandboxed systems, set `GOCACHE` to a writable directory such as `/tmp/codex-go-mlx-cache`.
+`generate -native` does NOT yet support `-trace` or `-state` (cgo-engine only); those exit 2. A run-quick smoke of the served engine also exists as the `lethean-lem` skill (`lem.sh smoke e2b`), but for *driving the repo itself* prefer `task` + the binary directly.
 
 ## Repository Layout
 
@@ -44,17 +75,18 @@ After Mantis #1241, all Go code lives under `go/`:
 ```
 go/                          Go module root (dappco.re/go/mlx)
   *.go                       Public root API: model, tokenizer, compute, training, eval, distill, GRPO, hf-fit, merge, gguf-quantize, kv-snapshot, lora-fuse
+  cmd/mlx/                   CLI tool (built with `-o core-mlx`; consumers rename: lthn-mlx)
   cmd/violet/                Unix-socket sidecar daemon
   internal/metal/            All CGO code (mlx-c bindings)
   mlxlm/                     CGO-free Python subprocess backend
   pkg/daemon/                Daemon implementation
-  pkg/memvid/                Memvid storage CLI
+  pkg/memvid/                Deprecated State codec compatibility shim
   tests/                     Integration tests
 cpp/                         C++ side (CLion-side companion)
 docs/                        Markdown documentation
 examples/                    Per-feature usage examples (markdown)
 external/                    Vendored core libraries
-lib/mlx/                     Upstream mlx submodule (pinned at v0.30.1)
+lib/mlx/                     Upstream mlx submodule (pinned at v0.31.1)
 patches/                     Local patches to lib/mlx (not auto-applied)
 ```
 
@@ -127,7 +159,7 @@ Architecture is detected from `config.json` (`model_type`) for safetensors and f
 
 ## Submodule Patches
 
-`lib/mlx` is pinned at upstream tag `v0.30.1`. Local patches that we do not upstream live in `patches/` as standalone diff files (e.g. `patches/mlx-metallib-path.patch` for the `MLX_METALLIB_PATH` env-var override). Patches are not auto-applied — run them inside the submodule manually when their function is needed:
+`lib/mlx` is pinned at upstream tag `v0.31.1`. Local patches that we do not upstream live in `patches/` as standalone diff files (e.g. `patches/mlx-metallib-path.patch` for the `MLX_METALLIB_PATH` env-var override). Patches are not auto-applied — run them inside the submodule manually when their function is needed:
 
 ```bash
 git -C lib/mlx apply ../../patches/mlx-metallib-path.patch