From 4d04001e14b05ec8da300e38e969fe390c72270a Mon Sep 17 00:00:00 2001 From: midagedev Date: Thu, 15 Oct 2026 23:58:24 +0900 Subject: [PATCH] docs: README and models.md list Nemotron 3 Nano 4B, served by -m until its Hugging Face repo is named Co-Authored-By: Claude Opus 5.5 --- README.md | 9 +++++++-- docs/models.md | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8a6a60cec..16daf06ab 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --port 8080 ## Status (0.3.3) -0.3.3 adds MiniCPM-V 4.7 35B-A3B image input ([#147](https://github.com/midagedev/bloomery/pull/147)), with `--mmproj` on the `qwen3` seat; a tile is capped at 70 tokens. The Qwen3.8 limits 0.3.2 listed are fixed: the paged log names the asked draft before the yield ([#121](https://github.com/midagedev/bloomery/pull/121)), and `gate-gpu-qwen4exp-mtp` (pg) plans its card experts with their route scratch ([#255](https://github.com/midagedev/bloomery/pull/255)). The release pass for this tarball is in the [0.3.3 release notes](https://github.com/midagedev/bloomery/releases/tag/v0.3.3). +0.3.3 adds Nemotron 3 Nano 4B on its own seat (`--model nemotron`, [#259](https://github.com/midagedev/bloomery/pull/259)) and MiniCPM-V 4.7 35B-A3B image input ([#147](https://github.com/midagedev/bloomery/pull/147)), with `--mmproj` on the `qwen3` seat; a tile is capped at 70 tokens. The Qwen3.8 limits 0.3.2 listed are fixed: the paged log names the asked draft before the yield ([#121](https://github.com/midagedev/bloomery/pull/121)), and `gate-gpu-qwen4exp-mtp` (pg) plans its card experts with their route scratch ([#255](https://github.com/midagedev/bloomery/pull/255)). The release pass for this tarball is in the [0.3.3 release notes](https://github.com/midagedev/bloomery/releases/tag/v0.3.3). | Area | State | |---|---| @@ -65,6 +65,7 @@ bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --port 8080 - **On Qwen3.8 loads that page experts through the NVMe tier, `gate-gpu-qwen38-serve`'s real tier was 70/72 on 0.3.2's baseline (`f382b474`), with two preexisting paged failures.** It was not re-run for this note; the 0.3.3 release pass re-checks it. A default load keeps the draft off. +- **Nemotron 3 Nano 4B reads a prompt at its decode rate:** the body has no prompt batch, so a prompt runs one decode step per token. It serves one slot on one card; `--parallel` above 1, a placement of more than one card or with a tier card, and a draft lever are each refused by name. A request keeps what the slot holds only when its prompt extends the whole held sequence (the Mamba-2 state cannot step back); the seat runs no prompt cache. - **Unset context is the seats' ladder (PIN(2026-10-14)).** With `--ctx-size` unset, one slot asks the lesser of 262,144 and the file's limit and halves while the plan is short of the card, and of the host where the body keeps no tier past it. A set `--ctx-size` that does not fit is refused @@ -104,7 +105,7 @@ The tarballs are on the [releases page](https://github.com/midagedev/bloomery/re ## Use -One binary, six seats; the file's architecture picks the seat, and `--place` can stay unset on every model. +One binary, seven seats; the file's architecture picks the seat, and `--place` can stay unset on every model. | Seat | Serves | One command | |---|---|---| @@ -115,6 +116,10 @@ One binary, six seats; the file's architecture picks the seat, and `--place` can | `mimo2` | [MiMo-V2.6-Flash](https://huggingface.co/ggml-org/MiMo-V2.6-Flash-MOPD-GGUF) | `bloomery-serve --hf ggml-org/MiMo-V2.6-Flash-MOPD-GGUF:MXFP4` | | `decide` | [Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) and [lev](https://huggingface.co/ggml-org/lev-GGUF) (decision models) | `bloomery-serve --hf bartowski/Cloudflare_clef-flash-GGUF:Q5_K_M` | +The seventh seat, `nemotron`, serves Nemotron 3 Nano 4B (new in 0.3.3) from a file on disk, tested as +`NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf`: `bloomery-serve --model nemotron -m PATH/NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf`. +The table above lists the seats with a checked `--hf` row. + ```sh curl -s http://127.0.0.1:8080/v1/chat/completions \ -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 64}' diff --git a/docs/models.md b/docs/models.md index a8f498001..0ff7b4f1e 100644 --- a/docs/models.md +++ b/docs/models.md @@ -12,6 +12,7 @@ | Qwen3-30B-A3B-Instruct-2507 | [`Q4_K_M`](https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF) (unsloth) | whole on one 24 GB card, or split onto the CPU on 12-16 GB (automatic) | | MiniCPM-V 4.7 35B-A3B | [`Q4_K_M`](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf) (openbmb) | on the `qwen3` seat (a `qwen35moe` file); image input with `--mmproj` since 0.3.3, a tile capped at 70 tokens | | Clef-Flash | [`Q8_0` to `Q3_K_S`](https://huggingface.co/bartowski/Cloudflare_clef-flash-GGUF) (bartowski's upload of 2026-10-05; llama.cpp's `clef` layout carries the head, and the Q3_K files' Q3_K token embedding runs on the card; an older `qwen35`-layout file takes the release's [head file](BUILD.md#clef-flash)) | whole on one GPU, its head on the CPU; image input with an encoder | +| Nemotron 3 Nano 4B | `Q4_K_M` (`NVIDIA-Nemotron3-Nano-4B-Q4_K_M.gguf`, by `-m`) | whole on one GPU, one slot (`--model nemotron`); a prompt runs one step per token | | lev | [`Q4_K_M`](https://huggingface.co/ggml-org/lev-GGUF) (ggml-org) | whole on one GPU; its label readout in the file | | DeepSeek-V2-Lite-Chat | [`Q3_K_M`](https://huggingface.co/mradermacher/DeepSeek-V2-Lite-Chat-GGUF) (mradermacher) | CPU or GPU; the first model, still gated |