Repository navigation
Description
Summary
load_model_from_vllm always loads models with quantization="awq", whatever model was requested. For a checkpoint that is not AWQ-quantized this is expected to fail, and the loader then silently loads TheBloke/Mixtral-7B-Instruct-v0.1-AWQ instead. A user who passes model_id="<some model>" with use_gpu=True can end up extracting with a different model than the one they asked for. The only sign is a printed [WARN] line.
Not yet reproduced. Found by reading the code while reviewing #444. No GPU was available to confirm vLLM's error on a non-AWQ checkpoint.
Where
extract-module/laiser/llm_models/model_loader.py
Lines 101 to 132 in 25c1314
| def load_model_from_vllm(model_id: str = None, token: str = None, dtype: str = None, quantization: str = None): | |
| if not VLLM_AVAILABLE: | |
| raise ImportError("vLLM is not installed. Cannot load model using vLLM backend.") | |
| model_id = model_id or DEFAULT_VLLM_MODEL_ID | |
| dtype = dtype or "float16" | |
| quantization = quantization or "awq" | |
| if quantization is None: | |
| # Auto-detect quantization based on model name | |
| model_lower = model_id.lower() | |
| if "awq" in model_lower: | |
| quantization = "awq" | |
| elif "gptq" in model_lower: | |
| quantization = "gptq" | |
| try: | |
| llm_args = {"model": model_id, "dtype": dtype, "quantization": quantization} | |
| llm = LLM(**llm_args) | |
| quant_info = f" with {quantization} quantization" if quantization else "" | |
| print(f"[INFO] Successfully loaded vLLM model: {model_id} with dtype: {dtype}{quant_info}") | |
| except Exception as e: | |
| print(f"[WARN] Failed to load model '{model_id}' with dtype '{dtype}': {e}") | |
| print(f"[INFO] Falling back to default model: {DEFAULT_VLLM_MODEL_ID}") | |
| llm = LLM(model=DEFAULT_VLLM_MODEL_ID, dtype=dtype, quantization=quantization) | |
| print( | |
| f"[INFO] Loaded fallback model: {DEFAULT_VLLM_MODEL_ID} with dtype: {dtype} and quantization: {quantization}" | |
| ) | |
| return llm |
dtype = dtype or "float16"
quantization = quantization or "awq"
if quantization is None: # never true: quantization was just set to "awq"
# Auto-detect quantization based on model name
...
try:
llm_args = {"model": model_id, "dtype": dtype, "quantization": quantization}
llm = LLM(**llm_args)
...
except Exception as e:
print(f"[WARN] Failed to load model '{model_id}' with dtype '{dtype}': {e}")
print(f"[INFO] Falling back to default model: {DEFAULT_VLLM_MODEL_ID}")
llm = LLM(model=DEFAULT_VLLM_MODEL_ID, dtype=dtype, quantization=quantization)LLMRouter._initialize_vllm calls it with only model_id and hf_token, so quantization is always None on entry and always becomes "awq":
extract-module/laiser/llm_models/llm_router.py
Lines 188 to 189 in 25c1314
| def _initialize_vllm(self): | |
| self.llm = load_model_from_vllm(self.model_id, self.hf_token) |
Effects
- Non-AWQ models are expected to fail on vLLM. vLLM is told to apply AWQ to a checkpoint that has no AWQ quantization config.
- The failure becomes a silent model swap, not an error. The
except Exceptionretries with Mixtral, which is AWQ-quantized, so the retry succeeds and extraction proceeds on the wrong model. The router's transformers fallback (llm_router.py#L162-L168) is reached only if the Mixtral load fails too. - The auto-detect branch is dead code.
if quantization is Nonecan never be true after the line above it, so the name-based AWQ/GPTQ detection never runs. - Any failure triggers the swap. The retry catches
Exception, so errors unrelated to the requested model, such as running out of GPU memory, also end in Mixtral being loaded.
Also in the same function:
tokenis accepted but never used, so anhf_tokenpassed to the extractor does not reach vLLM. Gated or private models cannot authenticate through this path.dtypedefaults tofloat16for every model.
Suggested fix
- Default
quantizationtoNone, so the existing name-based detection runs, and otherwise let vLLM read the quantization method from the checkpoint's own config. - Do not substitute a different model when loading fails. Raise instead, so the router's transformers fallback runs, or, if that also fails, construction raises
LAiSERError(docs: bring #438 documentation and CPU fixes to main #443). If a fallback model is kept at all, make it opt-in and log it at warning level. - Pass the Hugging Face token through to vLLM.
- Update
docs/providers.md, which currently describes the fallback order as implemented (fix: deterministic decoding by default; paper updates for JOSS review #424 #444).
To verify (needs a CUDA machine)
- Install
laiser[gpu]and runSkillExtractorRefactored(model_id="Qwen/Qwen2.5-0.5B-Instruct", use_gpu=True). Check whether the log shows[INFO] Loaded fallback model: TheBloke/Mixtral-7B-Instruct-v0.1-AWQ. - Repeat with an AWQ checkpoint to confirm that path still loads the requested model.
Activity