Skip to content

fix(server): declare the H100 class NV-Embed-v2 needs #817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
krisztian-gajdar merged 1 commit into from
Oct 19, 2026
Merged

fix(server): declare the H100 class NV-Embed-v2 needs #817

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions packages/sie_server/models/nvidia__NV-Embed-v2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,15 @@ profiles:
# The rendered prefix is passed as the model's `instruction`, so
# is_mask_instruction excludes it from the pooling mean.
query_template: "Instruct: {instruction}\nQuery: {text}"
# NV-Embed-v2 needs an 80 GB card. The checkpoint's code builds its Mistral
# body from a float32 text config, so the weights take about 28 GB on the
# GPU, and a forward pass under the checkpoint's float16 autocast adds about
# 14 GB. Measured through sie-server on one H100 80GB: 29,065 MiB after
# load, 42,993 MiB after a one-sentence query, 67,113 MiB for 32 documents
# of 642 tokens, and 75,041 MiB for one 21,122-token document; a
# 41,602-token document did not raise the peak. It does not load on a
# 24 GB L4 and runs out of memory on its first request on an A100 40GB.
# The hardware name records the tested pairing; callers and operators
# still select the machine profile.
h100:
extends: default
1 change: 1 addition & 0 deletions tools/ci/tests/test_compatibility_matrix.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ def test_profile_variants_carry_their_profile_and_width():
assert (variant["model"], variant["profile"], variant["gpu_class"]) == ("Qwen/Qwen3.8-27B-FP8", "h100-256k", "h100")
assert cells["cuda13-sglang-cu130", "zai-org/GLM-5.3-Flash"]["tp_width"] == 8
assert cells["cuda12-sglang", "Qwen/Qwen3-0.6B"]["profile"] == "default"
assert cells["cuda12-default", "nvidia/NV-Embed-v2:h100"]["gpu_class"] == "h100"


def run(lane, route, image, result="passed", **extra):
Expand Down
Loading