FP8 Model - fraseque/llama-3.2-1B-FP8-Neuron, online serving output doesn't look correct

[debroy-rh]

Load Model Query:
python3.12 -m vllm.entrypoints.openai.api_server --max-model-len 4096 --max-num-seqs 1 --no-enable-prefix-caching --port 8009 --tensor-parallel-size 2 --model fraseque/llama-3.2-1B-FP8-Neuron --additional-config '{ "override_neuron_config": { "async_mode": true } }'

v1/completions endpoint call:
curl http://localhost:8009/v1/completions -H "Content-Type: application/json" -d '{
"model": "fraseque/llama-3.2-1B-FP8-Neuron",
"prompt": "My name is"
}'
{"id":"cmpl-96b91c64240240d69fbb6928bf0ded77","object":"text_completion","created":1768319906,"model":"fraseque/llama-3.2-1B-FP8-Neuron","choices":[{"index":0,"text":"rowingobao arbitrary-count Conservamentelve roiularesskin�endencyseyjunctionintonInstant","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":4,"total_tokens":20,"completion_tokens":16,"prompt_tokens_details":null},"kv_transfer_params":null}

[elaineyz]

Hi @debroy-rh, thanks for reporting this issue. Can you share your Neuron SDK dependency version so we can reproduce? E.g. please share the outputs of pip list | grep neuron and apt list | grep neuron

[debroy-rh]

Hi @elaineyz , The FP8 models I’ve been testing are loading successfully, but inference is not behaving correctly. This appears to be an issue related to the quantized models.
I also tested RedHatAI/Qwen3-1.7B-FP8-dynamic, and observed the same behavior. The issue persists when using the vllm-neuron 0.3.0 plugin as well.

pip list | grep neuron
libneuronxla 2.2.14584.0+06ac23d1
neuronx-cc 2.22.12471.0+b4a00d10
neuronx-distributed 0.16.25997+f431c02e
neuronx-distributed-inference 0.7.15063+bafa28d5
torch-neuronx 2.9.0.2.11.19912+e48cd891
vllm-neuron 0.3.0

[aws-yishanm]

I have reproduced the issue and am looking into potential solutions.

[aws-yishanm]

Hi @debroy-rh, I wanted to update you on the progress of this issue.

It appears that this model initially comes from this tutorial written by a solutions architect on Huggingface: https://huggingface.co/fraseque/llama-3.2-1B-FP8-Neuron.

The tutorial uses different flags as well as an older v0 version of vLLM. I tested the tutorial on our v0 Deep Learning Container (DLC): public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.9.1-neuronx-py311-sdk2.26.1-ubuntu22.04. This worked as expected with normal generation. This container was released in Neuron SDK 2.26.1

When applying the equivalent flags to our newest vLLM v1 DLC: public.ecr.aws/neuron/pytorch-inference-vllm-neuronx:0.13.0-neuronx-py312-sdk2.27.1-ubuntu24.04 I ran into the same issues you were experience of gibberish outputs. This container was released in Neuron SDK 2.27.1.

To debug further, I transferred the compiled artifacts saved from the v0 DLC to the v1 DLC and reran the model skipping compilation entirely by reusing the cached artifacts. The model produced correct output in this case.

This tells me that this is likely to be an issue with the newest 2.27.1 compiler, not an issue with vllm-neuron==0.3.0 as it was able to properly serve the model when given the right compiled artifacts.

I will file a ticket internally to our compiler team to have them look at the issue.

[debroy-rh]

hi @aws-yishanm , I was also facing this issue with other quantized model like - RedHatAI/Qwen3-1.7B-FP8-dynamic. You can also test it with other FP8 models if that helps.