Skip to content

vLLM loader forces AWQ quantization on every model and silently loads Mixtral instead #445

Open

Description

@phanindra-max

Summary

load_model_from_vllm always loads models with quantization="awq", whatever model was requested. For a checkpoint that is not AWQ-quantized this is expected to fail, and the loader then silently loads TheBloke/Mixtral-7B-Instruct-v0.1-AWQ instead. A user who passes model_id="<some model>" with use_gpu=True can end up extracting with a different model than the one they asked for. The only sign is a printed [WARN] line.

Not yet reproduced. Found by reading the code while reviewing #444. No GPU was available to confirm vLLM's error on a non-AWQ checkpoint.

Where

def load_model_from_vllm(model_id: str = None, token: str = None, dtype: str = None, quantization: str = None):
if not VLLM_AVAILABLE:
raise ImportError("vLLM is not installed. Cannot load model using vLLM backend.")
model_id = model_id or DEFAULT_VLLM_MODEL_ID
dtype = dtype or "float16"
quantization = quantization or "awq"
if quantization is None:
# Auto-detect quantization based on model name
model_lower = model_id.lower()
if "awq" in model_lower:
quantization = "awq"
elif "gptq" in model_lower:
quantization = "gptq"
try:
llm_args = {"model": model_id, "dtype": dtype, "quantization": quantization}
llm = LLM(**llm_args)
quant_info = f" with {quantization} quantization" if quantization else ""
print(f"[INFO] Successfully loaded vLLM model: {model_id} with dtype: {dtype}{quant_info}")
except Exception as e:
print(f"[WARN] Failed to load model '{model_id}' with dtype '{dtype}': {e}")
print(f"[INFO] Falling back to default model: {DEFAULT_VLLM_MODEL_ID}")
llm = LLM(model=DEFAULT_VLLM_MODEL_ID, dtype=dtype, quantization=quantization)
print(
f"[INFO] Loaded fallback model: {DEFAULT_VLLM_MODEL_ID} with dtype: {dtype} and quantization: {quantization}"
)
return llm

dtype = dtype or "float16"
quantization = quantization or "awq"

if quantization is None:  # never true: quantization was just set to "awq"
    # Auto-detect quantization based on model name
    ...

try:
    llm_args = {"model": model_id, "dtype": dtype, "quantization": quantization}
    llm = LLM(**llm_args)
    ...
except Exception as e:
    print(f"[WARN] Failed to load model '{model_id}' with dtype '{dtype}': {e}")
    print(f"[INFO] Falling back to default model: {DEFAULT_VLLM_MODEL_ID}")
    llm = LLM(model=DEFAULT_VLLM_MODEL_ID, dtype=dtype, quantization=quantization)

LLMRouter._initialize_vllm calls it with only model_id and hf_token, so quantization is always None on entry and always becomes "awq":

def _initialize_vllm(self):
self.llm = load_model_from_vllm(self.model_id, self.hf_token)

Effects

  1. Non-AWQ models are expected to fail on vLLM. vLLM is told to apply AWQ to a checkpoint that has no AWQ quantization config.
  2. The failure becomes a silent model swap, not an error. The except Exception retries with Mixtral, which is AWQ-quantized, so the retry succeeds and extraction proceeds on the wrong model. The router's transformers fallback (llm_router.py#L162-L168) is reached only if the Mixtral load fails too.
  3. The auto-detect branch is dead code. if quantization is None can never be true after the line above it, so the name-based AWQ/GPTQ detection never runs.
  4. Any failure triggers the swap. The retry catches Exception, so errors unrelated to the requested model, such as running out of GPU memory, also end in Mixtral being loaded.

Also in the same function:

  • token is accepted but never used, so an hf_token passed to the extractor does not reach vLLM. Gated or private models cannot authenticate through this path.
  • dtype defaults to float16 for every model.

Suggested fix

To verify (needs a CUDA machine)

  • Install laiser[gpu] and run SkillExtractorRefactored(model_id="Qwen/Qwen2.5-0.5B-Instruct", use_gpu=True). Check whether the log shows [INFO] Loaded fallback model: TheBloke/Mixtral-7B-Instruct-v0.1-AWQ.
  • Repeat with an AWQ checkpoint to confirm that path still loads the requested model.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Backend Dev.Building underlying LAiSER softwarebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions