[krisztian-gajdar]

nvidia/NV-Embed-v2 runs out of memory on an L4 and on an A100 40GB in the published 0.10.1 cuda12-default image, and serves on an H100 80GB. The catalog gives no hint that it needs a large card. The compatibility matrix reads a profile's GPU class from hardware names in the profile name, and the bare model's default profile has none, so the model shows as unspecified.

Why it needs 80 GB. The checkpoint's code builds its Mistral body from text_config, whose torch_dtype is float32. The weights therefore occupy about 28 GB on the GPU even though the checkpoint is stored in float16. Each forward pass under the checkpoint's float16 autocast adds about 14 GB.

Measurements. These were taken through sie-server on one H100 80GB, with torch 2.9.1, transformers 4.57.6 and sentence-transformers 5.7.0, the versions in the 0.10.1 image. Values are nvidia-smi memory used.

Step	MiB
After load	29,065
One-sentence query	42,993
32 documents of 642 tokens	67,113
One 21,122-token document	75,041
One 41,602-token document	75,041 (no higher)

This matches the published-image runs:

• L4: out of memory while loading.
• A100 40GB: out of memory on the first request.
• H100: serves.

Change. The PR adds an h100 profile that extends default and records these numbers beside it. The matrix and GPU smoke tests can then place the shape on its class. nvidia/NV-Embed-v2:h100 appears with GPU class h100, and the matrix test asserts it. The served shape and the default profile are unchanged. As with the other hardware-named profiles, the name records the tested pairing and does not select a machine.

Limit. The bare nvidia/NV-Embed-v2 route keeps an unspecified GPU class, because a profile named default cannot carry a hardware hint (gpu_class("default") is None by design). Giving a bare route its own class needs an explicit profile field. That is out of scope here.

Validation.

• Before the YAML change, the new matrix assertion fails with KeyError: ('cuda12-default', 'nvidia/NV-Embed-v2:h100'). After it, the assertion passes.
• test_compatibility_matrix.py, test_bundle_coverage.py, the packages/sie_server/tests/config suite, the loader tests and tools/test_yaml_compatibility.py pass.
• Ruff and ruff format pass on the changed test.

Summary by CodeRabbit

• New Features
  • Added an H100 machine profile for NV-Embed-v2, with measured memory requirements documented.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Team
• Run ID: 3c45d592-e83f-4006-81f4-524ebe9d5e40

📥 Commits

Reviewing files that changed from the base of the PR and between ad4af56 and 16162ff.

📒 Files selected for processing: 2

• packages/sie_server/models/nvidia__NV-Embed-v2.yaml
• tools/ci/tests/test_compatibility_matrix.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

📝 Walkthrough

The NV-Embed-v2 model configuration now includes an H100 profile that inherits from default. The compatibility matrix test checks that the H100 variant is classified as GPU class h100.

Changes: NV-Embed-v2 H100 Profile

Layer / File(s)	Summary
Add and validate the H100 profile packages/sie_server/models/nvidia__NV-Embed-v2.yaml, tools/ci/tests/test_compatibility_matrix.py	Adds an H100 profile that inherits from default. Comments record measured GPU memory behavior. The compatibility matrix test asserts that the H100 variant uses GPU class h100.

Suggested reviewers: svonava

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 16162

The change adds an H100-specific compatibility profile while leaving the default profile unchanged. No actionable merge risk is evident from the supplied change context.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: declaring the H100 GPU class required by NV-Embed-v2.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Fix all pre-merge checks with AI

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}