MCP server, model discovery, CI/packaging lanes, and parallel vision preprocessing - #23
Open
CrimsonMartin wants to merge 6 commits into
Open
Conversation
Re-applies the fork's MCP (Model Context Protocol) server — JSON-RPC 2.0 over stdio with generate/chat/embed tools and a fox://models resource — on top of the upstream tree. Adapted to the current engine API: submit_request's SubmitError result is surfaced as a JSON-RPC error, sampling uses the centralized defaults with SamplingParams::default(), and registry setup goes through the new RegistryConfig::embedded() constructor (mirrors ServeArgs defaults) shared with tests.
Re-applies the fork's `fox discover` (scans ferrumox, HuggingFace, Ollama and LM Studio model directories plus FOX_MODEL_DIRS for GGUF files, with shard grouping and dedup) and `fox gpu-info` (CUDA/Metal backend, VRAM and driver diagnostics) on the upstream base. Both were self-contained; only CLI wiring changed. Adds the walkdir dependency.
…e lanes Re-applies the fork's CI additions on the upstream base: - secret-scanning.yml: TruffleHog verified-secrets scan on push/PR. - update-llamacpp.yml: weekly job that bumps vendor/llama.cpp to upstream master, verifies a stub cargo check, and opens a labeled PR. - docker.yml: matrix build adding a -rocm image variant (Dockerfile.rocm) next to the existing CUDA image, with per-variant build cache and a disk-space step for the large ROCm base image. - release.yml: x86_64-linux-rocm release lane (ROCm 6.2 HIP SDK from AMD's repo, hipcc auto-detected by build.rs), plus a .deb package on the CPU leg via cargo-deb — smoke-tested by extracting and running the packaged binary, mirroring the existing tarball gate. The .deb asset globs lib*.so* so it cannot repeat the libmtmd.so.0 omission.
…split prefill Ports the fork's vision-parallelism work onto the current vision implementation, redesigned around today's mtmd helper API. Before this, a multimodal prefill was one atomic mtmd_helper_eval_chunks call that ran the CLIP encode while holding the llama context lock, so every other request — text ones included — stalled behind each image. Now tokenize_multimodal CLIP-encodes each media chunk up front (mtmd_encode_chunk + mtmd_get_output_embd) and stores the embeddings on MultimodalChunks; prefill walks the chunks and decodes text as tokens and images from the stored embeddings via mtmd_helper_decode_image_chunk, which handles M-RoPE and non-causal attention internally. The atomic path remains as fallback for chunks without embeddings. - MtmdPool: mtmd contexts become a checkout pool. --vision-contexts N (FOX_VISION_CONTEXTS, default 1) sizes it, so N requests encode in parallel; N=1 preserves today's serialization, but the encode still moves off the engine thread. - CLIP cache: per-model LRU (16 images) keyed by the raw image bytes' hash — chat clients re-send the history's images every turn, and each now encodes once. - Handlers call prepare_multimodal_prompt through a spawn_blocking wrapper so the encode doesn't stall tokio workers. Validated: make ci (stub fmt/clippy/tests/build + real cargo check), real-build clippy clean, 521 real-mode lib tests pass, docs flag check. Runtime validation on GPU hardware with a vision model still pending — this changes prefill mechanics for every multimodal request.
CI runs the latest stable clippy, and 1.98's new result_large_err lint flags this function's ~128-byte Err. That Err is the finished HTTP response a handler returns — built at most once per rejected request and consumed immediately — so boxing it would add an unbox at every call site to save a move that is on no hot path. Documented allow instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
This contributes the features from the CrimsonMartin fork, rebuilt as clean commits on top of current
develop(70e4bc0):fox mcp): JSON-RPC 2.0 over stdio withgenerate/chat/embedtools and afox://modelsresource, for IDE integration. Built against the current engine API (SubmitErrorbackpressure, centralized sampling defaults, a sharedRegistryConfig::embedded()constructor).fox discover,fox gpu-info): multi-path GGUF scan (ferrumox/HuggingFace/Ollama/LM Studio dirs) with shard grouping; CUDA/Metal VRAM + driver diagnostics.-rocmDocker variant, anx86_64-linux-rocmrelease lane (ROCm 6.2), and a.debpackage on the CPU release leg with an extract-and-run smoke test.mtmd_helper_eval_chunksprefill (which held the llama context lock, stalling every other request) into tokenize time on a checkout pool of mtmd contexts (--vision-contexts N, default 1). Prefill decodes text as tokens and images from stored embeddings viamtmd_helper_decode_image_chunk(M-RoPE/non-causal via the helper), atomic path kept as fallback. Adds a per-model CLIP LRU cache keyed by image bytes and aspawn_blockingwrapper so encodes don't stall tokio workers.How was it tested?
make cipasses (fmt, clippy-D warnings, tests,check_docs_flags.py,check_prompt_tokenization.py); real llama.cpp build:cargo test --lib— 521 tests pass; 20 new MCP unit tests.Checklist
make cipasses locally (cargo fmt --check,clippy -D warnings, tests)docs/cli/serve.mdgained--vision-contexts; CHANGELOG entry left for the next release cut)developbranch🤖 Generated with Claude Code