[juliankang4]

Part of #59 (plan and PR order in #190), E6: the embeddings endpoints, served by an EmbeddingGemma 2 model that bloomery-serve holds beside its chat model.

Builds on #299, now merged. The seven commits are listed under "What the PR adds".

What and why

A server started with --embd-model FILE or --embd-hf <repo>[:<quant>] loads an embedding model beside its chat model in the same process and answers llama-server's embeddings routes from it: POST /v1/embeddings in the /v1 shape, and POST /embedding and /embeddings in llama-server's own array. EmbeddingGemma 2 is the architecture it serves. The encoder is #299's; this PR picks its card, takes its bytes out of the chat plan and serves it.

Release notes: Added: /v1/embeddings, /embedding and /embeddings over an EmbeddingGemma 2 model held beside the chat model (--embd-model FILE or --embd-hf <repo>[:<quant>]), with MRL dimensions and llama-server's embd_normalize.

The maintainer's comment on #190 asked for three things. Items 1 and 2 are in this PR:

• /v1/embeddings answers with the embedding model's name in model (its file name), never the chat model's. A request whose model names the chat model or any other model is a 400 invalid_request_error in the /v1 error shape, and no pass runs.
• /v1/models lists the embedding model as its own data entry after the chat model, with meta.n_embd and meta.n_ctx; models::listing takes a list of models.

My reply on #190 put items 2 and 3 in a follow-up. This PR already changes the listing to take more than one model, so item 2 is here too. Item 3, an embedding_model field in /props, stays in that follow-up PR.

What the PR adds

One contract a commit, seven commits:

1. serve: the prompt reader, the panic text, the base64 encoder and the models listing each have one owner. No answer changes. prompt_ids reads with the tokenizer it is given, so a second vocabulary can read a prompt as /completion does. api::panic_text replaces the worker's and the decision server's copies of the panic message reader. media::encode_base64 is the one base64 encoder; serve_client::base64 re-exports it and the media tests' copy is gone. models::listing takes a list of models.
2. serve: the first engine failure is the one /health, the board and the exit name. A behavior change for the existing engine path too. Before, each failure path wrote the fatal record, killed the board with its own reason and sent its own end, so a second failure overwrote the first. Now Shared::record_fatal adopts only the first failure and Shared::end_on sends that one to the end channel. A worker test holds two failures to it.
3. gpu-gates: the reserves a plan holds beside its tensors have one owner, beside. A move. image_plan's reserve_on, reserve_in, with_reserve, note and refusal move to crates/gpu-gates/src/beside.rs. Beside names each reserve a plan holds on its stage card, one field per user (image, and from commit 5 embedding). The reserve name, the verdict note and the refusal text stay word for word, so by the code no seat's plan or record moves.
4. serve: the embeddings endpoints, over an embedding model beside the engine. crates/serve/src/embed.rs is the seam: an Embed takes one input's ids and returns its pooled vector, and names its tokenizer, name, context and the widths it can be cut to. crates/serve/src/embeddings.rs reads a request as llama-server's handle_embeddings_impl does at llama.cpp 6868a52c:
   • input before content, and the array split rule (an array with an id at its top level is one input, else each element is one).
   • Text is tokenized with the embedding model's vocabulary. The first text element of each input gets its special tokens: BOS first where the GGUF sets add_bos_token, EOS last where it sets add_eos_token (Tokenizer::add_eos, false by default, so no existing model changes).
   • embd_normalize as common_embd_normalize, encoding_format float or base64, and dimensions, which cuts the vector to one of the model's widths before it is normalized.
   • With no embedding model, every route is a 501 not_supported_error.
   • A request checks the server's end before it waits for the model and again before every pass, so once the engine has failed or a stop has begun it gets a 503 and runs no pass.
   • A model failure, a panic, a vector of another width or a non-finite value ends the server as an engine failure does, through commit 2's record_fatal and end_on.
5. gpu-gates: the side seat, an embedding model a generative seat serves beside its own. crates/gpu-gates/src/side_seat.rs:
   • Flags::take reads --embd-model FILE or --embd-hf <repo>[:<quant>], llama-server's -m and -hf behind --embd-. A flag with no value, a flag given twice and both flags at once are refused by name.
   • Flags::read reads the file after the seat's levers and before its plan. Its architecture must be a row of ROWS (crates/gpu-gates/src/arch/embedding.rs); the one row is gemma-embedding2, whose bytes are gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299's card_bytes at the model card's 8,192 tokens.
   • Side::seat picks the card by the image encoder's rule: the smallest card the chat plan leaves off, else the stage card. On the stage card, every plan, whole-fit probe and placement-choice probe of the seat holds the model's bytes as the embedding model reserve through Beside. The embedding_seat record (architecture, card, bytes, reserve, context) is written once, after the placement is chosen.
   • SideEngine opens the model after the chat model loads, on a thread of its own with its own card stream, and implements Embed over a channel to that thread. No pass runs on the chat engine's thread. A panic in the open or in a pass comes back as a named failure.
   • EmbeddingGemma 2's row (side_seat/gemma_embedding2.rs) opens gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299's Encoder on the chosen card and returns Pass::pooled, so the output projection and the mean stay in the encoder. Its widths are the 128, 256 and 512 cuts below the file's embedding_length_out, plus that full width; a file whose output is narrower than 128 is refused at open.
   • The generative seats take the model. The decide and nemotron seats refuse it, and -m refuses a file of an embedding architecture, naming --embd-model.
   • The three serving binaries require gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299's gemma-embedding2 feature; the build recipes, the release build, docs/BUILD.md and docs/contrib/dgx-spark.md build with it. README.md's Use table and docs/serving.md describe the routes, and gate-ds41-bind runs the side seat's unit tests.
6. gpu-gates: EmbeddingGemma 2 beside a chat seat, its vectors, reserve and greedy tokens. The serve gate, just gate-gpu-gemma-embedding2-serve, described under "What I ran".
7. gpu-gates: every gate's file digest is the crate's sha256. gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299 moved sha256 to the crate root. Four gate binaries and one shared taps file still carried their own copies; they now call the crate's. No answer changes.

Choices #190 left open, and our defaults. The maintainer can change any of these here:

• The flags are --embd-model and --embd-hf only (section 7, question 2 of model: EmbeddingGemma and EmbeddingGemma 2 on a side seat #190). An embedding model's image projector waits for E7, so there is no --embd-mmproj yet.
• An input's limit is the model card's 8,192 tokens, read from one place, hparams::CARD_CTX; the GGUF says 262,144 (question 3). A longer input is a 400 exceed_context_size_error, with no flag to raise it.
• /embedding and /embeddings ignore model, as llama-server's native routes do; only /v1/embeddings checks it.
• The model is called through Embed on its own thread, not through the hidden-row media call that item 4 of model: EmbeddingGemma and EmbeddingGemma 2 on a side seat #190's section 3 named. The pooling still belongs to the encoder, as model: EmbeddingGemma and EmbeddingGemma 2 on a side seat #190 planned.
• The reserve is a field of Beside rather than a name argument to reserve_on, so each plan states both reserves in one value.

Code shape

• No second copy. The base64 encoder, the panic text, the models listing, the fatal record and the file digest each have one owner. The plan reserve has one owner, beside, with two users. The mean over positions is gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299's Pass::pooled; the serve crate has no pooling of its own.
• Model code holds only architecture facts. The serve side has no branch per model. EmbeddingGemma 2's own facts (its bytes, its context, its widths) sit in its row and in arch/embedding.rs.
• Dead code is gone. serve_client::base64's body, the media tests' encoder, image_plan's reserve functions, the two copies of the panic reader and the five file-digest copies in the gate sources are removed.
• No silent failure. A vector of the wrong width or with a non-finite value ends the server by name. A p below -1, a norm f32 cannot scale by, a width the model does not cut to and an output narrower than 128 are named errors.
• Comments state what is true now.
• No relaxed gate. The serve gate's bands start from the encoder gate's pooled floor 1e-2, which carries a PIN(2026-10-15): line in crates/refset/src/arch/gemma_embedding2/mod.rs. tools/defaults-twins.tsv gains three rows for the gate's pinned levers (a twin for BLOOMERY_DRAFT=off, dated reasons for the card budget and BLOOMERY_RESIDENCY=off), and tools/recipes.py's exact ds41 selection set gains the new gate with a PIN line, because the gate builds bloomery-serve.
• cargo fmt is clean (under "What I ran"), and the Mac lint step counts the same 59 warnings as main.

Duplication found or removed (path:line, before this PR's commits): crates/gpu-gates/src/serve_client.rs:260 (base64) and crates/serve/src/media.rs:708 (the tests' encode), now crates/serve/src/media.rs:629 (encode_base64); crates/serve/src/worker.rs:406 and crates/serve/src/decide.rs:615 (the panic message reader), now crates/serve/src/api.rs:517 (panic_text); the file digests in crates/gpu-gates/src/bin/gate_clef_hidden.rs, gate_qwen35_media.rs, gate_qwen3_serve.rs and shared/qwen3moe_taps.rs, now crates/gpu-gates/src/lib.rs:143 (sha256).

What I ran

The branch sits on main 38c41f4e, where #299 merged, as head b75dff1a03dda54c4d4e606b3e8691e12a54a1e7. Its seven commits are the ones the checks below ran on at eabeae22, except two deleted doc comment lines (the stale sha256 sentences above open_set in two gates) and the side seat's row naming its architecture through model::arch::GEMMA_EMBEDDING2 instead of a second spelling of it; before #299 merged they sat on main a688accb2c2b43fc4407984f4210db3978753c00, with #299 at 3e8da8dc1df636744c9e150417198787d47ca309 below it. The rebase kept this PR's code. Only the lines beside main's new Bolt and nemotron arms moved (the nemotron seat's bind call, the new nemotron serve recipe's feature set, the ds41 expected set in tools/recipes.py), and tools/gate-paths.tsv and tools/unsafe-ratchet.txt were regenerated. At eabeae22 I ran again the host checks, just contrib-check --tier t0, the Mac static tier, the PTX comparison and the gates just affected selects (last item below). The serve gate, the CUDA build, the FAIL-first edits and the record schemas below are from the earlier head e1624025ef3a77fb3ca0f3d2677731aa2d6f297b on base d66ef090ccc31aa426074f2eaa52ae7c3f17955e, the same code before the rebase. Each says so. Machine for everything but the static tier: a DGX Spark in the project's dev container (CUDA 13.3.1), nvidia-smi:

NVIDIA GB10, 595.84, 12.1

A check that is red here and on main a688accb alike says so.

Host checks, at eabeae22. cargo fmt --all -- --check ended rc 0. cargo build -p bloomery-serve --bin bloomery-serve && cargo test --release -p bloomery-serve -p bloomery-vision -p bloomery-refset ended rc 0, with 232 passed; 0 failed for the serve library tests.

Static tier, on a Mac, at eabeae22. ./tools/mac-static.sh upstream/main:

mac-static: fmt-check rc 0 in 11 s
mac-static: lint rc 1 in 39 s, ^warning: 59, above the ratchet 41
mac-static: combos rc 0 in 78 s, 50 shapes run, 0 red, 7 skipped (scoped)
mac-static: check-recipes rc 1 in 198 s
mac-static: red steps: lint (rc 1) check-recipes (rc 1)

Both red steps are red on main a688accb on the same Mac. The lint step counts 59 warnings there too, from the same warning source files, and check-recipes shows the same FAIL cold run: the base side runs in a temp archive, away from this tree's git and ptx-compare: self-test FAIL (33 cases, 1 failed). I did not look for the cause. Every other check-* step is rc 0.

just contrib-check --tier t0, at eabeae22 and on a688accb: rc 1 on both, with DONE tier=t0 total=84 red=1 and the one red step check-recipes. Its gpu-gate self-test fails on the same line on both: FAIL --v41-queue runs under BLOOMERY_BOX_READONLY=1 and syncs nothing: rc 1 (want 0). The other 83 steps are green. I did not run a higher tier.

CUDA build (head e1624025). cargo oxide build --arch sm_121 -- -p bloomery-gpu-gates --features gpu,deepseek41,glm5next,vision,mimo2,clef,gemma-embedding2 --release --bins finished in 3m 17s with Cargo build succeeded, so the serving binaries and the serve gate binary build with the new feature. The PTX comparison below builds the three serving binaries again at eabeae22.

The serve gate (head e1624025). BLOOMERY_BOX_LOCAL=1 just gate-gpu-gemma-embedding2-serve runs Qwen3-30B-A3B-Instruct-2507 Q4_K_M as the chat model at a 12 GiB card budget, which leaves some experts on the host, with one visible card so the embedding model sits on the stage card. It checks:

• the plans: at the same budget the side seat's plan puts fewer experts on the card (plan_expert_cost); with the reserve added to the budget it gives the plain split back (plan_same_split); the embedding_seat record names gpu, gpu-gemma-embedding2, gpu-gates: the EmbeddingGemma 2 encoder from the common kernels, with its end-to-end gate #299's card_bytes at 8,192 tokens (reserve_record); the cards term of Plan::pool_need counts the reserve (pool_cards_count_reserve);
• the vectors: every text case of the server reference set through /v1/embeddings, raw and normalized, and the shortest case's 128, 256 and 512 cuts, against mainline llama-server's vectors. The bands are derived from the encoder gate's pooled band and the CPU twin, not measured serving distances;
• the chat model: 512 greedy tokens the same with and without the embedding model, and the same while an embedding request on the longest reference input runs, which must see at least two decode steps (concurrent_greedy_same);
• the routes: the listing, the 501 with no embedding model, the 400s for the chat model's name, an unknown name, dimensions 129 and an 8,193-token input, and -m given the embedding file.

The gate ran 70 s and exited 0: 188 checks PASS and none FAIL, 171 of them the vector comparisons. The concurrent check saw 2 decode steps during one 1734-token embedding. I did not run the gate again at eabeae22.

FAIL-first (head e1624025). Three groups of edits that break what a check holds. In each, the edited tree failed and the restored file passed.

The serve crate's width, finite-value and lock tests against edits that remove those checks. Each edit failed the one test that guards it (rc 101) and the restored file passed (rc 0):

host-mutant short-no-width mutant=101 restored=0 clean=0
host-mutant nan-no-finite mutant=101 restored=0 clean=0
host-mutant no-locked-open mutant=101 restored=0 clean=0
host mutation checks ok=True count=3

Two route tests, one for model on the native routes and one for the model type on /v1/embeddings: ROUTE-MUTANT model-on-native mutant=101 restored=0 and ROUTE-MUTANT oai-model-type mutant=101 restored=0.

The serve gate's clauses against 13 edits to the plan, the reserve, the vector path and the guards. Twelve edits made the gate exit 1 on the clause that holds the property, and the restored file passed (exit 0). The thirteenth, vector-shift, stayed inside the gate's bands: 172 comparisons were accepted, so the gate does not claim a detected fault for it. The log ends GPU-MUTANTS aggregate ok=True count=13.

The gates just affected selects against base a688accb2c2b43fc4407984f4210db3978753c00: still running on the card when this was posted; the result goes into this description when it ends.

PTX (at eabeae22). This stack adds one kernel entry, head_norm_neox_append_256_whole in crates/gpu/src/rope_neox.rs, and it comes from #299. This PR adds no kernel. just ptx-compare refuses sm_121 (the scan's blk/SM column is sm_86's occupancy arithmetic), so I ran its steps by hand on the card: cargo oxide build --arch sm_121 of each binary at base and head, PTX_SCAN_ARCH=sm_121 tools/ptx-scan.sh <bin>, then python3 tools/recipes.py ptx-compare on each pair. The comparison reads each scan's stderr and stdout joined, in that order.

ptx-compare: bin=bloomery-serve base=a688accb2c2b head=eabeae22 method=decl1 arch=sm_121 jit-cc=12.1
  base entries  347
  head entries  348
  added         1: head_norm_neox_append_256_whole
  removed       none
  changed       none
ptx-compare: added-only (rc 0)

bloomery-serve-ds41 gives the same result (347 base entries, 348 head entries, the same one added) and bloomery-serve-qwen38 gives 264 and 265 with the same one added. All three are rc 0.

Record schemas (head e1624025). The --records-schema output of bloomery-serve (with a Qwen3.8-Flash-Next file), bloomery-serve-ds41 and bloomery-serve-qwen38 was compared with the checked-in snapshots, which gain the embedding_seat kind. All three are byte-equal and the check ends records-refresh: PASS. I did not run the full just records-refresh, which also rewrites snapshots this PR does not touch.

What I ran into

None of this blocks this pull request.

• An input past the context. llama-server checks its physical batch (n_ubatch) first and answers an input past it with a 500; past the context it answers exceed_context_size_error. This server has no batch limit there, so an input past the model's context is a 400 exceed_context_size_error naming both lengths.
• Object inputs. llama-server reads {"prompt_string": …} and a content array of parts. Here an object input is a 400 naming the types it takes.
• embd_normalize. Mainline computes a p-norm with a negative p for any value below -1, and its scale can come to zero or infinity for a large p. Here a value below -1, and a norm f32 cannot scale by, are a 400.
• Two side models on one card. The image encoder and the embedding model each take the smallest card the plan leaves off. With --mmproj and --embd-model on a machine with one such card, both land there and nothing counts the two together; the second open fails by name if the card is short. docs/serving.md says so.

Hardware (if the change is for a machine)

• The static tier on a Mac (Apple silicon), rustc 1.100.0-nightly (e457a7b0d 2026-08-27), just 1.58.0.
• The host checks, contrib-check, the PTX comparison, the build, the record schemas, the FAIL-first edits, the serve gate and the affected gates on a DGX Spark (NVIDIA GB10, 595.84, 12.1), in the project's dev container (CUDA 13.3.1).
• The models: ggml-org/embeddinggemma-2-GGUF embeddinggemma-2-Q8_0.gguf (sha256 2188ac1d..., the file the reference sets were dumped from) and Qwen3-30B-A3B-Instruct-2507 Q4_K_M.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

[github-actions]

Verdict: CLEAN: No reachable behavior defect or proven quality defect survives review.

Checked

Duplicates: 150 new or moved Rust item headers checked against existing owners. Dryer reported zero pairs in serve/refset. Manual searches covered gpu-gates; the reserve, digest, panic-message and base64 moves reuse common owners.

Complexity: New serve-gate run (72) combines setup, plan assertions, serving and refusal checks; those are its stage boundaries. concurrent (24) coordinates requests and observations; vectors (16) walks the comparison matrix; embeddings answer (20) validates and renders requests. Existing run growth adds optional side-model handling: drive 26→30, qwen38 127→131, ds41 51→54, glm 69→72, mimo2 35→38, qwen3 52→55. API routing adds two arms, 27→29.

Silent failure: 15 added Result/default sites followed. Missing architecture and invalid context reach named errors. Zero reserve means an off-plan seat. Disconnected reply channels have no remaining consumer. Missing decode metrics cannot make the concurrency clause pass.

Tests: Static mutation checks covered normalization arithmetic, special-token insertion, input splitting, context boundaries, model selection, dimensions, response shapes, base64 padding, reserve accounting, thread ownership, first-failure retention and record registration. Dropping the width/finite guards or the shutdown recheck makes their assertions fail. The PR reports FAIL-first runs for width, NaN, shutdown and model guards; its vector-shift mutant remains inside the admitted bands. Tests were not executed in this review.

Gates: Existing encoder bands retain their values and dated PIN when moved. Reserve tests move with their implementation. New serving bands derive from the existing encoder floor and CPU twin.

Adversarial paths: Empty and over-context inputs fail before inference. A request waiting behind a failed or stopping server gets 503 before its pass. Placement probes and loaders carry the same stage-card reserve. Excessive normalization powers produce a named error when their scale is invalid.

Performance: No decode kernel or steady-step dispatch work changes. The new embedding worker copies 4 × input-token-count bytes per pass; this is separate from chat token dispatch.

Escape catalog: No introduced C1–C4 defect found. The changed records-refresh pipeline enables pipefail.

Notes

Dryer did not search gpu-gates; it was searched manually. The PR supplies the NVIDIA GB10 card line, commit hashes, model names and serve-gate command, but summarizes the gate result without its command output. Final affected-gate results are also pending in the description. GPU evidence is recorded as incomplete.

_{Card check: needs-gpu. This pull request changes crates/gpu-gates/Cargo.toml, crates/gpu-gates/src/arch/embedding.rs, crates/gpu-gates/src/beside.rs … and carries no gate run on a card (CONTRIBUTING's evidence list); a maintainer adds gpu-verified after one.}

_{🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)}

[juliankang4]

The review is done and the contribution check passed, so I'm merging this.