Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ bloomery-serve --hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --port 8080
- **On Qwen3.8 loads that page experts through the NVMe tier, `gate-gpu-qwen38-serve`'s real tier was 70/72 on 0.3.2's
baseline (`f382b474`), with two preexisting paged failures.** It was not re-run for this note; the 0.3.3 release pass
re-checks it. A default load keeps the draft off.
- **Nemotron 3 Nano 4B reads a prompt at its decode rate:** the body has no prompt batch, so a prompt runs one decode step per token. It serves one slot on one card; `--parallel` above 1, a placement of more than one card or with a tier card, and a draft lever are each refused by name. A request keeps what the slot holds only when its prompt extends the whole held sequence (the Mamba-2 state cannot step back); the seat runs no prompt cache.
- **Nemotron 3 Nano 4B reads a prompt at its decode rate and serves one slot on one card** ([the nemotron seat](docs/serving.md#the-seats) names what it refuses and when it keeps a prefix).
- **Unset context is the seats' ladder (PIN(2026-10-14)).** With `--ctx-size` unset, one slot asks the lesser of
262,144 and the file's limit and halves while the plan is short of the card, and of the host where the body keeps
no tier past it. A set `--ctx-size` that does not fit is refused
Expand Down
2 changes: 1 addition & 1 deletion docs/BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ This builds the V4.1 CLI, the GPU kernels and the host expert tier into one bina
|---|---|---|
| `generate_ds41`, `bloomery-chat`, `bloomery-serve-ds41` | `cargo oxide build --arch sm_86 -- -p bloomery-gpu-gates --features deepseek41 --release --bin <bin>` | DeepSeek-V4.1-Flash |
| `bloomery-serve-qwen38` | the same, `--features deepseek41` (the feature scopes the server code; it runs no V4.1 code) | Qwen3.8-Flash-Next |
| `bloomery-serve` | `… --features glm5next,clef,vision,mimo2 --release --bin bloomery-serve` (`--model ds41\|qwen38\|glm\|qwen3\|mimo2\|decide`; the ds41 and qwen38 seats are the two binaries above; `vision` adds V4.1's `--mmproj`) | DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next, GLM-5.3-Flash, Qwen3.6-35B-A3B, Qwen3-30B-A3B, MiMo-V2.6-Flash, Clef-Flash, lev |
| `bloomery-serve` | `… --features glm5next,clef,vision,mimo2 --release --bin bloomery-serve` (`--model ds41\|qwen38\|glm\|qwen3\|mimo2\|nemotron\|decide`; the ds41 and qwen38 seats are the two binaries above; `vision` adds V4.1's `--mmproj`) | DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next, GLM-5.3-Flash, Qwen3.6-35B-A3B, Qwen3-30B-A3B, MiMo-V2.6-Flash, Nemotron 3 Nano 4B, Clef-Flash, lev |
| `generate_qwen3moe` | `… --features gpu --release --bin generate_qwen3moe` | Qwen3.8-Flash-Next, Qwen3.6-35B-A3B, Qwen3-30B-A3B |
| `generate_glm5next` | `… --features glm5next --release --bin generate_glm5next` | GLM-5.3-Flash |
| `r8conv` | `cargo build --release -p bloomery-model --bin r8conv` | the V4.1 sidecar |
Expand Down
10 changes: 8 additions & 2 deletions docs/serving.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,12 @@ Where it ends today: one model a server, sm_86+ GPUs, and the seats' own default

## The seats

The nemotron seat serves Nemotron 3 Nano 4B (`--model nemotron -m PATH`) whole on one card, one slot: `--parallel`
above 1, a placement of more than one card or with a tier card, and a draft lever are refused by name. The body has
no prompt batch, so a prompt runs one decode step per token, at the decode rate. A Mamba-2 state cannot step back: a
request keeps what the slot holds only when its prompt extends the whole held sequence, and any other prompt resets
the slot and is read again. The seat runs no prompt cache; a snapshot or a resume is refused by name.

The decide seat answers SystemOne requests, one at a time. Clef-Flash also reads images (Image input, below); lev reads text states only. It serves two decision models: Clef-Flash (a joint schema head) and lev (`--hf ggml-org/lev-GGUF:Q4_K_M`, a label readout; its file's `qwen35.decision.type` picks it, and its request is rendered by the file's own `systemone` template):

```sh
Expand All @@ -95,7 +101,7 @@ bartowski/Cloudflare_clef-flash-GGUF:Q5_K_M` needs no head flag. The earlier `qw
file: `--head joint_head.safetensors`, its `joint_head_config.json` beside it. A `--head` beside a `clef` file names the
head instead of the file's own.

**Concurrent requests.** Every generative seat takes `--parallel N` (unset, one slot; a set
**Concurrent requests.** Every generative seat but nemotron (one slot, above) takes `--parallel N` (unset, one slot; a set
`--ctx-size` with no `--parallel` serves one slot at the whole of it — the flag is one request's context, as
llama-server reads it, and the startup line says so). A count you set is never lowered. The N slots are resident sequences
inside the one model, switched by pointer exchange: nothing parks, and each round advances every busy slot by one
Expand Down Expand Up @@ -213,7 +219,7 @@ binary's `--help`.
- One model a server; one expert tier card at most (`--place bp`).
- Concurrent streams run as one pass on a whole-card Qwen3-30B or Qwen3.6, on V4.1, MiMo-V2.6-Flash, GLM-5.3 and Qwen3.8; a placed
Qwen3-30B or Qwen3.6 and a sampled request on a drafting load step in turn. A new request's prompt runs whole while the other
streams wait. The decide seat serves one request at a time; DSpark drafts never rejoin — `--parallel > 1` with
streams wait. The decide seat serves one request at a time, and the nemotron seat one slot; DSpark drafts never rejoin — `--parallel > 1` with
`BLOOMERY_DRAFT=dspark` is refused by name.
- With adaptive residency on, a request's tokens follow the placement its passes ran on, and the placement follows
the history of passes (the requests before it and the streams beside it): the same history gives the same tokens
Expand Down
Loading