From 2389edf332c000b44e3a15bba90f5c928c6ff693 Mon Sep 17 00:00:00 2001 From: midagedev Date: Fri, 16 Oct 2026 01:04:14 +0900 Subject: [PATCH] docs: BUILD.md and serving.md name the nemotron seat, and the README's limit line points to serving.md Co-Authored-By: Claude Opus 5.5 --- README.md | 2 +- docs/BUILD.md | 2 +- docs/serving.md | 10 ++++++++-- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 16daf06ab..adacd0c0b 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --port 8080 - **On Qwen3.8 loads that page experts through the NVMe tier, `gate-gpu-qwen38-serve`'s real tier was 70/72 on 0.3.2's baseline (`f382b474`), with two preexisting paged failures.** It was not re-run for this note; the 0.3.3 release pass re-checks it. A default load keeps the draft off. -- **Nemotron 3 Nano 4B reads a prompt at its decode rate:** the body has no prompt batch, so a prompt runs one decode step per token. It serves one slot on one card; `--parallel` above 1, a placement of more than one card or with a tier card, and a draft lever are each refused by name. A request keeps what the slot holds only when its prompt extends the whole held sequence (the Mamba-2 state cannot step back); the seat runs no prompt cache. +- **Nemotron 3 Nano 4B reads a prompt at its decode rate and serves one slot on one card** ([the nemotron seat](docs/serving.md#the-seats) names what it refuses and when it keeps a prefix). - **Unset context is the seats' ladder (PIN(2026-10-14)).** With `--ctx-size` unset, one slot asks the lesser of 262,144 and the file's limit and halves while the plan is short of the card, and of the host where the body keeps no tier past it. A set `--ctx-size` that does not fit is refused diff --git a/docs/BUILD.md b/docs/BUILD.md index 5da7b93ad..3170dc92d 100644 --- a/docs/BUILD.md +++ b/docs/BUILD.md @@ -69,7 +69,7 @@ This builds the V4.1 CLI, the GPU kernels and the host expert tier into one bina |---|---|---| | `generate_ds41`, `bloomery-chat`, `bloomery-serve-ds41` | `cargo oxide build --arch sm_86 -- -p bloomery-gpu-gates --features deepseek41 --release --bin ` | DeepSeek-V4.1-Flash | | `bloomery-serve-qwen38` | the same, `--features deepseek41` (the feature scopes the server code; it runs no V4.1 code) | Qwen3.8-Flash-Next | -| `bloomery-serve` | `… --features glm5next,clef,vision,mimo2 --release --bin bloomery-serve` (`--model ds41\|qwen38\|glm\|qwen3\|mimo2\|decide`; the ds41 and qwen38 seats are the two binaries above; `vision` adds V4.1's `--mmproj`) | DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next, GLM-5.3-Flash, Qwen3.6-35B-A3B, Qwen3-30B-A3B, MiMo-V2.6-Flash, Clef-Flash, lev | +| `bloomery-serve` | `… --features glm5next,clef,vision,mimo2 --release --bin bloomery-serve` (`--model ds41\|qwen38\|glm\|qwen3\|mimo2\|nemotron\|decide`; the ds41 and qwen38 seats are the two binaries above; `vision` adds V4.1's `--mmproj`) | DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next, GLM-5.3-Flash, Qwen3.6-35B-A3B, Qwen3-30B-A3B, MiMo-V2.6-Flash, Nemotron 3 Nano 4B, Clef-Flash, lev | | `generate_qwen3moe` | `… --features gpu --release --bin generate_qwen3moe` | Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, Qwen3-30B-A3B | | `generate_glm5next` | `… --features glm5next --release --bin generate_glm5next` | GLM-5.3-Flash | | `r8conv` | `cargo build --release -p bloomery-model --bin r8conv` | the V4.1 sidecar | diff --git a/docs/serving.md b/docs/serving.md index 40caf8010..438d32026 100644 --- a/docs/serving.md +++ b/docs/serving.md @@ -81,6 +81,12 @@ Where it ends today: one model a server, sm_86+ GPUs, and the seats' own default ## The seats +The nemotron seat serves Nemotron 3 Nano 4B (`--model nemotron -m PATH`) whole on one card, one slot: `--parallel` +above 1, a placement of more than one card or with a tier card, and a draft lever are refused by name. The body has +no prompt batch, so a prompt runs one decode step per token, at the decode rate. A Mamba-2 state cannot step back: a +request keeps what the slot holds only when its prompt extends the whole held sequence, and any other prompt resets +the slot and is read again. The seat runs no prompt cache; a snapshot or a resume is refused by name. + The decide seat answers SystemOne requests, one at a time. Clef-Flash also reads images (Image input, below); lev reads text states only. It serves two decision models: Clef-Flash (a joint schema head) and lev (`--hf ggml-org/lev-GGUF:Q4_K_M`, a label readout; its file's `qwen35.decision.type` picks it, and its request is rendered by the file's own `systemone` template): ```sh @@ -95,7 +101,7 @@ bartowski/Cloudflare_clef-flash-GGUF:Q5_K_M` needs no head flag. The earlier `qw file: `--head joint_head.safetensors`, its `joint_head_config.json` beside it. A `--head` beside a `clef` file names the head instead of the file's own. -**Concurrent requests.** Every generative seat takes `--parallel N` (unset, one slot; a set +**Concurrent requests.** Every generative seat but nemotron (one slot, above) takes `--parallel N` (unset, one slot; a set `--ctx-size` with no `--parallel` serves one slot at the whole of it — the flag is one request's context, as llama-server reads it, and the startup line says so). A count you set is never lowered. The N slots are resident sequences inside the one model, switched by pointer exchange: nothing parks, and each round advances every busy slot by one @@ -213,7 +219,7 @@ binary's `--help`. - One model a server; one expert tier card at most (`--place bp`). - Concurrent streams run as one pass on a whole-card Qwen3-30B or Qwen3.6, on V4.1, MiMo-V2.6-Flash, GLM-5.3 and Qwen3.8; a placed Qwen3-30B or Qwen3.6 and a sampled request on a drafting load step in turn. A new request's prompt runs whole while the other - streams wait. The decide seat serves one request at a time; DSpark drafts never rejoin — `--parallel > 1` with + streams wait. The decide seat serves one request at a time, and the nemotron seat one slot; DSpark drafts never rejoin — `--parallel > 1` with `BLOOMERY_DRAFT=dspark` is refused by name. - With adaptive residency on, a request's tokens follow the placement its passes ran on, and the placement follows the history of passes (the requests before it and the streams beside it): the same history gives the same tokens