OpenAI interface - Home Assistant is crashing model on the second prompt

[DrazorV]

Getting this error always on the second prompt from home assistant to openarc

-- Boot dd3d3c10b9dd428c913f1f4007148098 --
Feb 06 10:09:56 openarc systemd[1]: Started openarc.service - OpenArc Inference Server (venv).
Feb 06 10:09:56 openarc openarc[550]: Configuration saved to: /root/OpenArc/openarc_config.json
Feb 06 10:09:56 openarc openarc[550]: Starting OpenArc server on 0.0.0.0:8000
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO - Launching  0.0.0.0:8000
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO - --------------------------------
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO - OpenArc endpoints:
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /openarc/load           Load a model
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /openarc/unload         Unload a model
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - GET    /openarc/status         Get model status
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO - --------------------------------
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO - OpenAI compatible endpoints:
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - GET    /v1/models
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /v1/chat/completions
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /v1/audio/transcriptions: Whisper only
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /v1/audio/speech: Kokoro only
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /v1/embeddings
Feb 06 10:09:56 openarc openarc[550]: 2026-02-06 10:09:56,656 - INFO -   - POST   /v1/rerank
Feb 06 10:10:02 openarc openarc[550]: 2026-02-06 10:10:02,560 - INFO - Started server process [550]
Feb 06 10:10:02 openarc openarc[550]: 2026-02-06 10:10:02,560 - INFO - Waiting for application startup.
Feb 06 10:10:02 openarc openarc[550]: 2026-02-06 10:10:02,561 - INFO - qwen3-8b loading...
Feb 06 10:10:02 openarc openarc[550]: 2026-02-06 10:10:02,561 - INFO - ModelType.LLM on GPU.1 with {'GPU_ENABLE_LARGE_ALLOCATIONS': True, 'GPU_ENABLE_SDPA_OPTIMIZATION': False}
Feb 06 10:10:04 openarc openarc[550]: WARNING: Resizable BAR not detected for device 0000:31:00.0
Feb 06 10:10:19 openarc openarc[550]: 2026-02-06 10:10:19,678 - INFO - qwen3-8b loaded successfully
Feb 06 10:10:19 openarc openarc[550]: 2026-02-06 10:10:19,679 - INFO - Startup: loaded 'qwen3-8b'
Feb 06 10:10:19 openarc openarc[550]: 2026-02-06 10:10:19,679 - INFO - [qwen3-8b LLM Worker] Started, waiting for packets...
Feb 06 10:10:19 openarc openarc[550]: 2026-02-06 10:10:19,679 - INFO - Application startup complete.
Feb 06 10:10:19 openarc openarc[550]: 2026-02-06 10:10:19,679 - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Feb 06 10:11:38 openarc openarc[550]: 2026-02-06 10:11:38,381 - INFO - Request received: GET /v1/models from 127.0.0.1
Feb 06 10:11:38 openarc openarc[550]: 2026-02-06 10:11:38,382 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 10:12:29 openarc openarc[550]: 2026-02-06 10:12:29,405 - INFO - Request received: GET /openarc/status from 127.0.0.1
Feb 06 10:12:29 openarc openarc[550]: 2026-02-06 10:12:29,406 - INFO - Request completed: GET /openarc/status status=200 duration=0.000s
Feb 06 10:13:25 openarc openarc[550]: 2026-02-06 10:13:25,207 - INFO - Request received: GET /openarc/status from 127.0.0.1
Feb 06 10:13:25 openarc openarc[550]: 2026-02-06 10:13:25,208 - INFO - Request completed: GET /openarc/status status=200 duration=0.001s
Feb 06 10:18:34 openarc openarc[550]: 2026-02-06 10:18:34,956 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 10:18:34 openarc openarc[550]: 2026-02-06 10:18:34,957 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 10:18:36 openarc openarc[550]: 2026-02-06 10:18:36,631 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 10:18:36 openarc openarc[550]: 2026-02-06 10:18:36,632 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,728 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,734 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.006s
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,800 - ERROR - [DEBUG] draft_model_loaded: False
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,800 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,800 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 06 10:18:46 openarc openarc[550]: 2026-02-06 10:18:46,800 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 06 10:18:53 openarc openarc[550]: 2026-02-06 10:18:53,275 - INFO - [qwen3-8b LLM Worker] Metrics: {'load_time (s)': 16.81, 'ttft (s)': 2.98, 'tpot (ms)': 26.37698, 'prefill_throughput (tokens/s)': 2854.96, 'decode_throughput (tokens/s)': 37.91185, 'decode_duration (s)': 6.46583, 'input_token': 8508, 'new_token': 133, 'total_token': 8641, 'stream': True, 'stream_chunk_tokens': 1}
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,956 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,960 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.005s
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,974 - ERROR - [DEBUG] draft_model_loaded: False
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,974 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,974 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 06 10:18:56 openarc openarc[550]: 2026-02-06 10:18:56,974 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 06 10:18:57 openarc openarc[550]: 2026-02-06 10:18:57,011 - ERROR - LLM inference failed!
Feb 06 10:18:57 openarc openarc[550]: Traceback (most recent call last):
Feb 06 10:18:57 openarc openarc[550]:   File "/root/OpenArc/src/server/worker_registry.py", line 87, in infer_llm
Feb 06 10:18:57 openarc openarc[550]:     async for item in llm_instance.generate_type(packet.gen_config):
Feb 06 10:18:57 openarc openarc[550]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 200, in generate_stream
Feb 06 10:18:57 openarc openarc[550]:     result = await gen_task
Feb 06 10:18:57 openarc openarc[550]:              ^^^^^^^^^^^^^^
Feb 06 10:18:57 openarc openarc[550]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 179, in _run_generation
Feb 06 10:18:57 openarc openarc[550]:     return await asyncio.to_thread(
Feb 06 10:18:57 openarc openarc[550]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 10:18:57 openarc openarc[550]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
Feb 06 10:18:57 openarc openarc[550]:     return await loop.run_in_executor(None, func_call)
Feb 06 10:18:57 openarc openarc[550]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 10:18:57 openarc openarc[550]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Feb 06 10:18:57 openarc openarc[550]:     result = self.fn(*self.args, **self.kwargs)
Feb 06 10:18:57 openarc openarc[550]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 10:18:57 openarc openarc[550]: RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
Feb 06 10:18:57 openarc openarc[550]: Check 'intermediates_memories.size() > sequential_gws_subseq_mapping_idx' failed at src/plugins/intel_gpu/src/graph/impls/ocl_v2/sdpa/paged_attention_opt.cpp:1745:
Feb 06 10:18:57 openarc openarc[550]: [GPU] Unexpected number of intermediates buffers for Paged Attention for mixed stage
Feb 06 10:18:57 openarc openarc[550]: 2026-02-06 10:18:57,012 - ERROR - [qwen3-8b LLM Worker] Inference failed, triggering model unload...
Feb 06 10:18:57 openarc openarc[550]: 2026-02-06 10:18:57,226 - INFO - [qwen3-8b] unloaded successfully

[SearchSavior]

@DrazorV thanks for the issue.

ModelType.LLM on GPU.1 with {'GPU_ENABLE_LARGE_ALLOCATIONS': True, 'GPU_ENABLE_SDPA_OPTIMIZATION': False}

hints are often not compatible with each other, and usually openvino sets these automatically at model compile time when openvino.core() accesses your device properties underneath openvino.genai abstraction.

I haven't needed these settings before. Try removing them and see what happens, lmk how it goes

[DrazorV]

@DrazorV thanks for the issue.

ModelType.LLM on GPU.1 with {'GPU_ENABLE_LARGE_ALLOCATIONS': True, 'GPU_ENABLE_SDPA_OPTIMIZATION': False}

hints are often not compatible with each other, and usually openvino sets these automatically at model compile time when openvino.core() accesses your device properties underneath openvino.genai abstraction.

I haven't needed these settings before. Try removing them and see what happens, lmk how it goes

I actually added those because I tried to make it work, here is another example without them:

-- Boot efaa89a44d444b11aa4454b147f57776 --
Feb 06 13:01:06 openarc systemd[1]: Started openarc.service - OpenArc Inference Server (venv).
Feb 06 13:01:07 openarc openarc[235]: Configuration saved to: /root/OpenArc/openarc_config.json
Feb 06 13:01:07 openarc openarc[235]: Starting OpenArc server on 0.0.0.0:8000
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,505 - INFO - Launching  0.0.0.0:8000
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO - --------------------------------
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO - OpenArc endpoints:
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /openarc/load           Load a model
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /openarc/unload         Unload a model
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - GET    /openarc/status         Get model status
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO - --------------------------------
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO - OpenAI compatible endpoints:
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - GET    /v1/models
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /v1/chat/completions
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /v1/audio/transcriptions: Whisper only
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /v1/audio/speech: Kokoro only
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /v1/embeddings
Feb 06 13:01:07 openarc openarc[235]: 2026-02-06 13:01:07,506 - INFO -   - POST   /v1/rerank
Feb 06 13:01:17 openarc openarc[235]: 2026-02-06 13:01:17,544 - INFO - Started server process [235]
Feb 06 13:01:17 openarc openarc[235]: 2026-02-06 13:01:17,544 - INFO - Waiting for application startup.
Feb 06 13:01:17 openarc openarc[235]: 2026-02-06 13:01:17,545 - INFO - Application startup complete.
Feb 06 13:01:17 openarc openarc[235]: 2026-02-06 13:01:17,545 - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Feb 06 13:01:47 openarc openarc[235]: 2026-02-06 13:01:47,129 - INFO - Request received: GET /openarc/status from 127.0.0.1
Feb 06 13:01:47 openarc openarc[235]: 2026-02-06 13:01:47,129 - INFO - Request completed: GET /openarc/status status=403 duration=0.000s
Feb 06 13:02:09 openarc openarc[235]: 2026-02-06 13:02:09,744 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 13:02:09 openarc openarc[235]: 2026-02-06 13:02:09,747 - INFO - Request completed: GET /v1/models status=200 duration=0.003s
Feb 06 13:02:16 openarc openarc[235]: 2026-02-06 13:02:16,039 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 13:02:16 openarc openarc[235]: 2026-02-06 13:02:16,044 - INFO - Request completed: GET /v1/models status=200 duration=0.005s
Feb 06 13:02:21 openarc openarc[235]: 2026-02-06 13:02:21,128 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 13:02:21 openarc openarc[235]: 2026-02-06 13:02:21,130 - INFO - Request completed: GET /v1/models status=200 duration=0.002s
Feb 06 13:02:29 openarc openarc[235]: 2026-02-06 13:02:29,541 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 13:02:29 openarc openarc[235]: 2026-02-06 13:02:29,543 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 13:03:24 openarc openarc[235]: 2026-02-06 13:03:24,189 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 13:03:24 openarc openarc[235]: 2026-02-06 13:03:24,192 - INFO - Request completed: GET /v1/models status=200 duration=0.003s
-- Boot 19a2557d41de4942b3f08d9e0896b17d --
Feb 06 16:42:59 openarc systemd[1]: Started openarc.service - OpenArc Inference Server (venv).
Feb 06 16:42:59 openarc openarc[235]: Configuration saved to: /root/OpenArc/openarc_config.json
Feb 06 16:42:59 openarc openarc[235]: Starting OpenArc server on 0.0.0.0:8000
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO - Launching  0.0.0.0:8000
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO - --------------------------------
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO - OpenArc endpoints:
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /openarc/load           Load a model
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /openarc/unload         Unload a model
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - GET    /openarc/status         Get model status
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO - --------------------------------
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO - OpenAI compatible endpoints:
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - GET    /v1/models
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /v1/chat/completions
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /v1/audio/transcriptions: Whisper only
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /v1/audio/speech: Kokoro only
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /v1/embeddings
Feb 06 16:42:59 openarc openarc[235]: 2026-02-06 16:42:59,733 - INFO -   - POST   /v1/rerank
Feb 06 16:43:06 openarc openarc[235]: 2026-02-06 16:43:06,305 - INFO - Started server process [235]
Feb 06 16:43:06 openarc openarc[235]: 2026-02-06 16:43:06,305 - INFO - Waiting for application startup.
Feb 06 16:43:06 openarc openarc[235]: 2026-02-06 16:43:06,305 - INFO - Application startup complete.
Feb 06 16:43:06 openarc openarc[235]: 2026-02-06 16:43:06,306 - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Feb 06 16:45:19 openarc openarc[235]: 2026-02-06 16:45:19,048 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 06 16:45:19 openarc openarc[235]: 2026-02-06 16:45:19,048 - INFO - Request completed: POST /openarc/load status=403 duration=0.001s
Feb 06 16:45:30 openarc openarc[235]: 2026-02-06 16:45:30,258 - INFO - Request received: GET /openarc/status from 127.0.0.1
Feb 06 16:45:30 openarc openarc[235]: 2026-02-06 16:45:30,259 - INFO - Request completed: GET /openarc/status status=403 duration=0.001s
Feb 06 16:46:00 openarc openarc[235]: 2026-02-06 16:46:00,382 - INFO - Request received: GET /openarc/status from 127.0.0.1
Feb 06 16:46:00 openarc openarc[235]: 2026-02-06 16:46:00,382 - INFO - Request completed: GET /openarc/status status=200 duration=0.000s
Feb 06 16:46:02 openarc openarc[235]: 2026-02-06 16:46:02,824 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 06 16:46:02 openarc openarc[235]: 2026-02-06 16:46:02,829 - INFO - qwen3-8b-test loading...
Feb 06 16:46:02 openarc openarc[235]: 2026-02-06 16:46:02,829 - INFO - ModelType.LLM on GPU.1 with {}
Feb 06 16:46:04 openarc openarc[235]: WARNING: Resizable BAR not detected for device 0000:31:00.0
Feb 06 16:46:26 openarc openarc[235]: 2026-02-06 16:46:26,062 - INFO - qwen3-8b-test loaded successfully
Feb 06 16:46:26 openarc openarc[235]: 2026-02-06 16:46:26,062 - INFO - [qwen3-8b-test LLM Worker] Started, waiting for packets...
Feb 06 16:46:26 openarc openarc[235]: 2026-02-06 16:46:26,063 - INFO - Request completed: POST /openarc/load status=200 duration=23.239s
Feb 06 16:48:55 openarc openarc[235]: 2026-02-06 16:48:55,492 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 16:48:55 openarc openarc[235]: 2026-02-06 16:48:55,492 - INFO - Request completed: GET /v1/models status=200 duration=0.000s
Feb 06 16:49:02 openarc openarc[235]: 2026-02-06 16:49:02,910 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 16:49:02 openarc openarc[235]: 2026-02-06 16:49:02,911 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 16:49:09 openarc openarc[235]: 2026-02-06 16:49:09,975 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 06 16:49:09 openarc openarc[235]: 2026-02-06 16:49:09,976 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,364 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,376 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.012s
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,407 - ERROR - [DEBUG] draft_model_loaded: False
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,407 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,407 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 06 16:49:14 openarc openarc[235]: 2026-02-06 16:49:14,407 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 06 16:49:20 openarc openarc[235]: 2026-02-06 16:49:20,530 - INFO - [qwen3-8b-test LLM Worker] Metrics: {'load_time (s)': 22.86, 'ttft (s)': 2.77, 'tpot (ms)': 25.41918, 'prefill_throughput (tokens/s)': 3075.7, 'decode_throughput (tokens/s)': 39.34037, 'decode_duration (s)': 6.12251, 'input_token': 8507, 'new_token': 133, 'total_token': 8640, 'stream': True, 'stream_chunk_tokens': 1}
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,103 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,104 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.001s
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,118 - ERROR - [DEBUG] draft_model_loaded: False
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,118 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,118 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,118 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,142 - ERROR - LLM inference failed!
Feb 06 16:49:24 openarc openarc[235]: Traceback (most recent call last):
Feb 06 16:49:24 openarc openarc[235]:   File "/root/OpenArc/src/server/worker_registry.py", line 87, in infer_llm
Feb 06 16:49:24 openarc openarc[235]:     async for item in llm_instance.generate_type(packet.gen_config):
Feb 06 16:49:24 openarc openarc[235]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 200, in generate_stream
Feb 06 16:49:24 openarc openarc[235]:     result = await gen_task
Feb 06 16:49:24 openarc openarc[235]:              ^^^^^^^^^^^^^^
Feb 06 16:49:24 openarc openarc[235]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 179, in _run_generation
Feb 06 16:49:24 openarc openarc[235]:     return await asyncio.to_thread(
Feb 06 16:49:24 openarc openarc[235]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 16:49:24 openarc openarc[235]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
Feb 06 16:49:24 openarc openarc[235]:     return await loop.run_in_executor(None, func_call)
Feb 06 16:49:24 openarc openarc[235]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 16:49:24 openarc openarc[235]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Feb 06 16:49:24 openarc openarc[235]:     result = self.fn(*self.args, **self.kwargs)
Feb 06 16:49:24 openarc openarc[235]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 06 16:49:24 openarc openarc[235]: RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
Feb 06 16:49:24 openarc openarc[235]: Check 'intermediates_memories.size() > sequential_gws_subseq_mapping_idx' failed at src/plugins/intel_gpu/src/graph/impls/ocl_v2/sdpa/paged_attention_opt.cpp:1745:
Feb 06 16:49:24 openarc openarc[235]: [GPU] Unexpected number of intermediates buffers for Paged Attention for mixed stage
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,143 - ERROR - [qwen3-8b-test LLM Worker] Inference failed, triggering model unload...
Feb 06 16:49:24 openarc openarc[235]: 2026-02-06 16:49:24,366 - INFO - [qwen3-8b-test] unloaded successfully

[DrazorV]

Also here is a bench if it helps (I am using TB4 egpu and re-bar is not enabled, I am still waiting for my oculink-pcie module to arrive, then hopefully I will have re-bar)

root@openarc:~# openarc bench qwen3-8b-test --p 512 --n 128 --r 5
working...

input tokens: [512]
max tokens:   [128]
runs: 5

  benching... (5/5) 



Qwen3-8B-int4-ov

┏━━━━━┳━━━━━┳━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ run ┃   p ┃   n ┃ ttft(s) ┃ tpot(ms) ┃ prefill(t/s) ┃ decode(t/s) ┃ duration(s) ┃
┡━━━━━╇━━━━━╇━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│   1 │ 512 │ 128 │    0.17 │    21.83 │       3063.3 │        45.8 │        2.94 │
│   2 │ 512 │ 128 │    0.12 │    21.10 │       4354.2 │        47.4 │        2.80 │
│   3 │ 512 │ 128 │    0.12 │    20.87 │       4337.2 │        47.9 │        2.77 │
│   4 │ 512 │ 128 │    0.12 │    21.08 │       4114.1 │        47.4 │        2.80 │
│   5 │ 512 │ 128 │    0.12 │    47.23 │       4274.1 │        21.2 │        0.17 │
└─────┴─────┴─────┴─────────┴──────────┴──────────────┴─────────────┴─────────────┘
Total: 5 runs

[SearchSavior]

I'll investigate tonight if 2.0.3 caused a regression somewhere... these errors surface from openvino runtime, so it might be an upstream issue not controlled by our codebase. Quick test of 2.0.2 would help me a ton. Also try rolling back openvino to latest release instead of nightly.

Can you share openarc tool device-properties?

[DrazorV]

How do I change versions? Do I checkout to 2.0.2 branch and restart the server or do I need to run some build commands too?
Also how do I manage the openvivo version?

Below are the props:

root@openarc:~# openarc tool device-props
Querying device data for all devices...
WARNING: Resizable BAR not detected for device 0000:31:00.0

Available Devices (3):
╭─ Device: CPU ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ AVAILABLE_DEVICES:                                                                                                                                     │
│ RANGE_FOR_ASYNC_INFER_REQUESTS: 1, 1, 1                                                                                                                │
│ RANGE_FOR_STREAMS: 1, 8                                                                                                                                │
│ EXECUTION_DEVICES: CPU                                                                                                                                 │
│ FULL_DEVICE_NAME: 12th Gen Intel(R) Core(TM) i9-12900H                                                                                                 │
│ OPTIMIZATION_CAPABILITIES: FP32, INT8, BIN, EXPORT_IMPORT                                                                                              │
│ DEVICE_TYPE: Type.INTEGRATED                                                                                                                           │
│ DEVICE_ARCHITECTURE: intel64                                                                                                                           │
│ NUM_STREAMS: 1                                                                                                                                         │
│ INFERENCE_NUM_THREADS: 0                                                                                                                               │
│ PERF_COUNT: False                                                                                                                                      │
│ INFERENCE_PRECISION_HINT: <Type: 'float32'>                                                                                                            │
│ PERFORMANCE_HINT: PerformanceMode.LATENCY                                                                                                              │
│ EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE                                                                                                         │
│ PERFORMANCE_HINT_NUM_REQUESTS: 0                                                                                                                       │
│ ENABLE_CPU_PINNING: True                                                                                                                               │
│ ENABLE_CPU_RESERVATION: False                                                                                                                          │
│ SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE                                                                                                      │
│ MODEL_DISTRIBUTION_POLICY: set()                                                                                                                       │
│ ENABLE_HYPER_THREADING: True                                                                                                                           │
│ DEVICE_ID:                                                                                                                                             │
│ CPU_DENORMALS_OPTIMIZATION: False                                                                                                                      │
│ LOG_LEVEL: Level.NO                                                                                                                                    │
│ CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0                                                                                                             │
│ ENABLE_TENSOR_PARALLEL: False                                                                                                                          │
│ TBB_PARTITIONER: TbbPartitioner.???                                                                                                                    │
│ DYNAMIC_QUANTIZATION_GROUP_SIZE: 32                                                                                                                    │
│ KV_CACHE_PRECISION: <Type: 'uint8_t'>                                                                                                                  │
│ KEY_CACHE_PRECISION: <Type: 'uint8_t'>                                                                                                                 │
│ VALUE_CACHE_PRECISION: <Type: 'uint8_t'>                                                                                                               │
│ KEY_CACHE_GROUP_SIZE: 0                                                                                                                                │
│ VALUE_CACHE_GROUP_SIZE: 0                                                                                                                              │
│ ENABLE_WEIGHTLESS: False                                                                                                                               │
│ WEIGHTS_PATH:                                                                                                                                          │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Device: GPU.0 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ AVAILABLE_DEVICES: 0, 1                                                                                                                                │
│ RANGE_FOR_ASYNC_INFER_REQUESTS: 1, 2, 1                                                                                                                │
│ RANGE_FOR_STREAMS: 1, 2                                                                                                                                │
│ OPTIMAL_BATCH_SIZE: 1                                                                                                                                  │
│ MAX_BATCH_SIZE: 1                                                                                                                                      │
│ DEVICE_ARCHITECTURE: GPU: vendor=0x8086 arch=v12.3.0                                                                                                   │
│ FULL_DEVICE_NAME: Intel(R) Iris(R) Xe Graphics (iGPU)                                                                                                  │
│ DEVICE_UUID: 8680a6460c0000000002000000000000                                                                                                          │
│ DEVICE_LUID: e06f5c33dd620000                                                                                                                          │
│ DEVICE_TYPE: Type.INTEGRATED                                                                                                                           │
│ DEVICE_GOPS: {<Type: 'float16'>: 4454.400390625, <Type: 'float32'>: 2227.2001953125, <Type: 'int8_t'>: 8908.80078125, <Type: 'uint8_t'>:               │
│ 8908.80078125}                                                                                                                                         │
│ OPTIMIZATION_CAPABILITIES: FP32, BIN, FP16, INT8, GPU_USM_MEMORY, EXPORT_IMPORT                                                                        │
│ DEVICE_PCI_INFO: {domain: 0 bus: 0 device: 0x2 function: 0}                                                                                            │
│ GPU_DEVICE_ID: 0x46a6                                                                                                                                  │
│ GPU_DEVICE_TOTAL_MEM_SIZE: 30895841280                                                                                                                 │
│ GPU_DEVICE_MAX_ALLOC_MEM_SIZE: 4294959104                                                                                                              │
│ GPU_UARCH_VERSION: 12.3.0                                                                                                                              │
│ GPU_EXECUTION_UNITS_COUNT: 96                                                                                                                          │
│ GPU_MEMORY_STATISTICS: {'cl_mem': 0, 'unknown': 0, 'usm_device': 0, 'usm_host': 0, 'usm_shared': 0}                                                    │
│ PERF_COUNT: False                                                                                                                                      │
│ MODEL_PRIORITY: Priority.MEDIUM                                                                                                                        │
│ GPU_HOST_TASK_PRIORITY: Priority.MEDIUM                                                                                                                │
│ GPU_QUEUE_PRIORITY: Priority.MEDIUM                                                                                                                    │
│ GPU_QUEUE_THROTTLE: Priority.MEDIUM                                                                                                                    │
│ GPU_ENABLE_SDPA_OPTIMIZATION: True                                                                                                                     │
│ GPU_ENABLE_LORA_OPERATION: True                                                                                                                        │
│ GPU_ENABLE_LARGE_ALLOCATIONS: False                                                                                                                    │
│ GPU_ENABLE_LOOP_UNROLLING: True                                                                                                                        │
│ GPU_DISABLE_WINOGRAD_CONVOLUTION: False                                                                                                                │
│ CACHE_DIR:                                                                                                                                             │
│ CACHE_MODE: CacheMode.OPTIMIZE_SPEED                                                                                                                   │
│ PERFORMANCE_HINT: PerformanceMode.LATENCY                                                                                                              │
│ EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE                                                                                                         │
│ COMPILATION_NUM_THREADS: 8                                                                                                                             │
│ NUM_STREAMS: 1                                                                                                                                         │
│ PERFORMANCE_HINT_NUM_REQUESTS: 0                                                                                                                       │
│ INFERENCE_PRECISION_HINT: <Type: 'float16'>                                                                                                            │
│ ENABLE_CPU_PINNING: False                                                                                                                              │
│ ENABLE_CPU_RESERVATION: False                                                                                                                          │
│ DEVICE_ID: 0                                                                                                                                           │
│ DYNAMIC_QUANTIZATION_GROUP_SIZE: 0                                                                                                                     │
│ ACTIVATIONS_SCALE_FACTOR: -1.0                                                                                                                         │
│ WEIGHTS_PATH:                                                                                                                                          │
│ CACHE_ENCRYPTION_CALLBACKS: UNSUPPORTED TYPE                                                                                                           │
│ KV_CACHE_PRECISION: <Type: 'dynamic'>                                                                                                                  │
│ MODEL_PTR: None                                                                                                                                        │
│ CONFIG_FILE:                                                                                                                                           │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Device: GPU.1 ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ AVAILABLE_DEVICES: 0, 1                                                                                                                                │
│ RANGE_FOR_ASYNC_INFER_REQUESTS: 1, 2, 1                                                                                                                │
│ RANGE_FOR_STREAMS: 1, 2                                                                                                                                │
│ OPTIMAL_BATCH_SIZE: 1                                                                                                                                  │
│ MAX_BATCH_SIZE: 1                                                                                                                                      │
│ DEVICE_ARCHITECTURE: GPU: vendor=0x8086 arch=v12.55.8                                                                                                  │
│ FULL_DEVICE_NAME: Intel(R) Arc(TM) A770 Graphics (dGPU)                                                                                                │
│ DEVICE_UUID: 8680a056080000003100000000000000                                                                                                          │
│ DEVICE_LUID: c06d5c33dd620000                                                                                                                          │
│ DEVICE_TYPE: Type.DISCRETE                                                                                                                             │
│ DEVICE_GOPS: {<Type: 'float16'>: 157286.40625, <Type: 'float32'>: 19660.80078125, <Type: 'int8_t'>: 314572.8125, <Type: 'uint8_t'>: 314572.8125}       │
│ OPTIMIZATION_CAPABILITIES: FP32, BIN, FP16, INT8, GPU_HW_MATMUL, GPU_USM_MEMORY, EXPORT_IMPORT                                                         │
│ DEVICE_PCI_INFO: {domain: 0 bus: 49 device: 0x0 function: 0}                                                                                           │
│ GPU_DEVICE_ID: 0x56a0                                                                                                                                  │
│ GPU_DEVICE_TOTAL_MEM_SIZE: 16225243136                                                                                                                 │
│ GPU_DEVICE_MAX_ALLOC_MEM_SIZE: 4294959104                                                                                                              │
│ GPU_UARCH_VERSION: 12.55.8                                                                                                                             │
│ GPU_EXECUTION_UNITS_COUNT: 512                                                                                                                         │
│ GPU_MEMORY_STATISTICS: {'cl_mem': 0, 'unknown': 0, 'usm_device': 0, 'usm_host': 0, 'usm_shared': 0}                                                    │
│ PERF_COUNT: False                                                                                                                                      │
│ MODEL_PRIORITY: Priority.MEDIUM                                                                                                                        │
│ GPU_HOST_TASK_PRIORITY: Priority.MEDIUM                                                                                                                │
│ GPU_QUEUE_PRIORITY: Priority.MEDIUM                                                                                                                    │
│ GPU_QUEUE_THROTTLE: Priority.MEDIUM                                                                                                                    │
│ GPU_ENABLE_SDPA_OPTIMIZATION: True                                                                                                                     │
│ GPU_ENABLE_LORA_OPERATION: True                                                                                                                        │
│ GPU_ENABLE_LARGE_ALLOCATIONS: False                                                                                                                    │
│ GPU_ENABLE_LOOP_UNROLLING: True                                                                                                                        │
│ GPU_DISABLE_WINOGRAD_CONVOLUTION: False                                                                                                                │
│ CACHE_DIR:                                                                                                                                             │
│ CACHE_MODE: CacheMode.OPTIMIZE_SPEED                                                                                                                   │
│ PERFORMANCE_HINT: PerformanceMode.LATENCY                                                                                                              │
│ EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE                                                                                                         │
│ COMPILATION_NUM_THREADS: 8                                                                                                                             │
│ NUM_STREAMS: 1                                                                                                                                         │
│ PERFORMANCE_HINT_NUM_REQUESTS: 0                                                                                                                       │
│ INFERENCE_PRECISION_HINT: <Type: 'float16'>                                                                                                            │
│ ENABLE_CPU_PINNING: False                                                                                                                              │
│ ENABLE_CPU_RESERVATION: False                                                                                                                          │
│ DEVICE_ID: 1                                                                                                                                           │
│ DYNAMIC_QUANTIZATION_GROUP_SIZE: 0                                                                                                                     │
│ ACTIVATIONS_SCALE_FACTOR: -1.0                                                                                                                         │
│ WEIGHTS_PATH:                                                                                                                                          │
│ CACHE_ENCRYPTION_CALLBACKS: UNSUPPORTED TYPE                                                                                                           │
│ KV_CACHE_PRECISION: <Type: 'dynamic'>                                                                                                                  │
│ MODEL_PTR: None                                                                                                                                        │
│ CONFIG_FILE:                                                                                                                                           │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Found 3 device(s)
root@openarc:~# 

[SearchSavior]

@DrazorV OK, so don't worry about rolling back OpenArc. After review, none of the relevant code has changed. I just uploaded a fresh export, please try this one if the following suggestions still dont help. https://huggingface.co/Echo9Zulu/NousCoder-14B-int4_sym-ov

Can you try a few things.

1. Set device to CPU
2. Set device to GPU.0 (to test igpu. may give us some empirical evidence that missing REBAR is a factor with A770 if the model you are using had no issues in the export to openvino format.

Waiting to try after you get occulink may actually be path of least resistance. Having REBAR squared away kills so many strange problems with Arc.

[DrazorV]

Machine resources:

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  q8c                                                                                                                                        │
│ model_path  /root/OpenArc/models/Qwen3-8B-int4-ov                                                                                                      │
│ device      CPU                                                                                                                                        │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  q8i                                                                                                                                        │
│ model_path  /root/OpenArc/models/Qwen3-8B-int4-ov                                                                                                      │
│ device      GPU.0                                                                                                                                      │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  n14c                                                                                                                                       │
│ model_path  /root/OpenArc/models/NousCoder-14B-int4_sym-ov                                                                                             │
│ device      CPU                                                                                                                                        │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  n14i                                                                                                                                       │
│ model_path  /root/OpenArc/models/NousCoder-14B-int4_sym-ov                                                                                             │
│ device      GPU.0                                                                                                                                      │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  n14d                                                                                                                                       │
│ model_path  /root/OpenArc/models/NousCoder-14B-int4_sym-ov                                                                                             │
│ device      GPU.1                                                                                                                                      │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ model_name  q8d                                                                                                                                        │
│ model_path  /root/OpenArc/models/Qwen3-8B-int4-ov                                                                                                      │
│ device      GPU.1                                                                                                                                      │
│ engine      ovgenai                                                                                                                                    │
│ model_type  llm                                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

CPU -QWEN (no response after 5 minutes, maybe didn't finish processing?):

Feb 08 16:26:51 openarc openarc[673]: Starting OpenArc server on 0.0.0.0:8000
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO - Launching  0.0.0.0:8000
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO - --------------------------------
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO - OpenArc endpoints:
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /openarc/load           Load a model
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /openarc/unload         Unload a model
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - GET    /openarc/status         Get model status
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO - --------------------------------
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO - OpenAI compatible endpoints:
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - GET    /v1/models
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /v1/chat/completions
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /v1/audio/transcriptions: Whisper only
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /v1/audio/speech: Kokoro only
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /v1/embeddings
Feb 08 16:26:51 openarc openarc[673]: 2026-02-08 16:26:51,827 - INFO -   - POST   /v1/rerank
Feb 08 16:26:58 openarc openarc[673]: 2026-02-08 16:26:58,477 - INFO - Started server process [673]
Feb 08 16:26:58 openarc openarc[673]: 2026-02-08 16:26:58,477 - INFO - Waiting for application startup.
Feb 08 16:26:58 openarc openarc[673]: 2026-02-08 16:26:58,478 - INFO - Application startup complete.
Feb 08 16:26:58 openarc openarc[673]: 2026-02-08 16:26:58,478 - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Feb 08 16:27:30 openarc openarc[673]: 2026-02-08 16:27:30,436 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 08 16:27:30 openarc openarc[673]: 2026-02-08 16:27:30,440 - INFO - q8c loading...
Feb 08 16:27:30 openarc openarc[673]: 2026-02-08 16:27:30,440 - INFO - ModelType.LLM on CPU with {}
Feb 08 16:27:34 openarc openarc[673]: 2026-02-08 16:27:34,929 - INFO - q8c loaded successfully
Feb 08 16:27:34 openarc openarc[673]: 2026-02-08 16:27:34,930 - INFO - [q8c LLM Worker] Started, waiting for packets...
Feb 08 16:27:34 openarc openarc[673]: 2026-02-08 16:27:34,930 - INFO - Request completed: POST /openarc/load status=200 duration=4.494s
Feb 08 16:27:36 openarc openarc[673]: 2026-02-08 16:27:36,300 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:27:36 openarc openarc[673]: 2026-02-08 16:27:36,304 - INFO - Request completed: GET /v1/models status=200 duration=0.004s
Feb 08 16:27:50 openarc openarc[673]: 2026-02-08 16:27:50,914 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:27:50 openarc openarc[673]: 2026-02-08 16:27:50,917 - INFO - Request completed: GET /v1/models status=200 duration=0.003s
Feb 08 16:27:54 openarc openarc[673]: 2026-02-08 16:27:54,930 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:27:54 openarc openarc[673]: 2026-02-08 16:27:54,931 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,718 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,721 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.003s
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,756 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,757 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,757 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:27:58 openarc openarc[673]: 2026-02-08 16:27:58,757 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:32:52 openarc openarc[673]: 2026-02-08 16:32:52,759 - INFO - [q8c LLM Worker] Metrics: {'load_time (s)': 4.17, 'ttft (s)': 268.06, 'tpot (ms)': 196.48427, 'prefill_throughput (tokens/s)': 31.73, 'decode_throughput (tokens/s)': 5.08947, 'decode_duration (s)': 294.00178, 'input_token': 8507, 'new_token': 133, 'total_token': 8640, 'stream': True, 'stream_chunk_tokens': 1}
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,225 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,226 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.001s
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,240 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,240 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,240 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:33:57 openarc openarc[673]: 2026-02-08 16:33:57,240 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:34:35 openarc openarc[673]: 2026-02-08 16:34:35,549 - INFO - [q8c LLM Worker] Metrics: {'load_time (s)': 4.17, 'ttft (s)': 4.75, 'tpot (ms)': 181.41614, 'prefill_throughput (tokens/s)': 1797.59, 'decode_throughput (tokens/s)': 5.51219, 'decode_duration (s)': 38.30856, 'input_token': 8531, 'new_token': 186, 'total_token': 8717, 'stream': True, 'stream_chunk_tokens': 1}
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,417 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,418 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.001s
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,430 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,430 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,430 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:34:48 openarc openarc[673]: 2026-02-08 16:34:48,430 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,764 - INFO - [q8c LLM Worker] Metrics: {'load_time (s)': 4.17, 'ttft (s)': 5.04, 'tpot (ms)': 185.42438, 'prefill_throughput (tokens/s)': 1697.08, 'decode_throughput (tokens/s)': 5.39303, 'decode_duration (s)': 29.33298, 'input_token': 8556, 'new_token': 132, 'total_token': 8688, 'stream': True, 'stream_chunk_tokens': 1}
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,783 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,784 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.001s
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,799 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,799 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,799 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:35:17 openarc openarc[673]: 2026-02-08 16:35:17,799 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:37:32 openarc openarc[673]: 2026-02-08 16:37:32,941 - INFO - [q8c LLM Worker] Metrics: {'load_time (s)': 4.17, 'ttft (s)': 82.09, 'tpot (ms)': 211.36469, 'prefill_throughput (tokens/s)': 127.28, 'decode_throughput (tokens/s)': 4.73116, 'decode_duration (s)': 135.14161, 'input_token': 10448, 'new_token': 252, 'total_token': 10700, 'stream': True, 'stream_chunk_tokens': 1}
root@openarc:~# 

(The lights were in fact on, LMAO)

iGPU - QWEN:

Feb 08 15:51:12 openarc openarc[817]: 2026-02-08 15:51:12,216 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 15:51:12 openarc openarc[817]: 2026-02-08 15:51:12,217 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 15:51:15 openarc openarc[817]: 2026-02-08 15:51:15,807 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 15:51:15 openarc openarc[817]: 2026-02-08 15:51:15,808 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 15:51:21 openarc openarc[817]: 2026-02-08 15:51:21,767 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 15:51:21 openarc openarc[817]: 2026-02-08 15:51:21,770 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.003s
Feb 08 15:51:22 openarc openarc[817]: 2026-02-08 15:51:22,019 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 15:51:22 openarc openarc[817]: 2026-02-08 15:51:22,019 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 15:51:22 openarc openarc[817]: 2026-02-08 15:51:22,020 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 15:51:22 openarc openarc[817]: 2026-02-08 15:51:22,020 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 15:51:50 openarc openarc[817]: 2026-02-08 15:51:50,538 - ERROR - LLM inference failed!
Feb 08 15:51:50 openarc openarc[817]: Traceback (most recent call last):
Feb 08 15:51:50 openarc openarc[817]:   File "/root/OpenArc/src/server/worker_registry.py", line 87, in infer_llm
Feb 08 15:51:50 openarc openarc[817]:     async for item in llm_instance.generate_type(packet.gen_config):
Feb 08 15:51:50 openarc openarc[817]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 200, in generate_stream
Feb 08 15:51:50 openarc openarc[817]:     result = await gen_task
Feb 08 15:51:50 openarc openarc[817]:              ^^^^^^^^^^^^^^
Feb 08 15:51:50 openarc openarc[817]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 179, in _run_generation
Feb 08 15:51:50 openarc openarc[817]:     return await asyncio.to_thread(
Feb 08 15:51:50 openarc openarc[817]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 15:51:50 openarc openarc[817]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
Feb 08 15:51:50 openarc openarc[817]:     return await loop.run_in_executor(None, func_call)
Feb 08 15:51:50 openarc openarc[817]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 15:51:50 openarc openarc[817]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Feb 08 15:51:50 openarc openarc[817]:     result = self.fn(*self.args, **self.kwargs)
Feb 08 15:51:50 openarc openarc[817]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 15:51:50 openarc openarc[817]: RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
Feb 08 15:51:50 openarc openarc[817]: Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_event.cpp:71:
Feb 08 15:51:50 openarc openarc[817]: [GPU] clWaitForEvents, error code: -14 CL_EXEC_STATUS_ERROR_FOR_EVENTS_IN_WAIT_LIST
Feb 08 15:51:50 openarc openarc[817]: 2026-02-08 15:51:50,542 - ERROR - [q8i LLM Worker] Inference failed, triggering model unload...
Feb 08 15:51:53 openarc openarc[817]: terminate called after throwing an instance of 'ov::Exception'
Feb 08 15:51:53 openarc openarc[817]:   what():  Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:376:
Feb 08 15:51:53 openarc openarc[817]: [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
Feb 08 15:51:53 openarc systemd[1]: openarc.service: Main process exited, code=killed, status=6/ABRT
Feb 08 15:51:53 openarc systemd[1]: openarc.service: Failed with result 'signal'.
Feb 08 15:51:53 openarc systemd[1]: openarc.service: Consumed 24min 30.588s CPU time.
Feb 08 15:51:55 openarc systemd[1]: openarc.service: Scheduled restart job, restart counter is at 1.
Feb 08 15:51:55 openarc systemd[1]: Started openarc.service - OpenArc Inference Server (venv).
Feb 08 15:51:55 openarc openarc[920]: Configuration saved to: /root/OpenArc/openarc_config.json
Feb 08 15:51:55 openarc openarc[920]: Starting OpenArc server on 0.0.0.0:8000
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO - Launching  0.0.0.0:8000
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO - --------------------------------
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO - OpenArc endpoints:
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /openarc/load           Load a model
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /openarc/unload         Unload a model
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - GET    /openarc/status         Get model status
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO - --------------------------------
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO - OpenAI compatible endpoints:
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - GET    /v1/models
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /v1/chat/completions
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /v1/audio/transcriptions: Whisper only
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /v1/audio/speech: Kokoro only
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /v1/embeddings
Feb 08 15:51:55 openarc openarc[920]: 2026-02-08 15:51:55,760 - INFO -   - POST   /v1/rerank
Feb 08 15:52:02 openarc openarc[920]: 2026-02-08 15:52:02,225 - INFO - Started server process [920]
Feb 08 15:52:02 openarc openarc[920]: 2026-02-08 15:52:02,225 - INFO - Waiting for application startup.
Feb 08 15:52:02 openarc openarc[920]: 2026-02-08 15:52:02,226 - INFO - Application startup complete.
Feb 08 15:52:02 openarc openarc[920]: 2026-02-08 15:52:02,226 - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

dGPU - QWEN:

Feb 08 15:59:04 openarc openarc[920]: 2026-02-08 15:59:04,302 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 08 15:59:04 openarc openarc[920]: 2026-02-08 15:59:04,304 - INFO - q8d loading...
Feb 08 15:59:04 openarc openarc[920]: 2026-02-08 15:59:04,307 - INFO - ModelType.LLM on GPU.1 with {}
Feb 08 15:59:07 openarc openarc[920]: WARNING: Resizable BAR not detected for device 0000:31:00.0
Feb 08 15:59:33 openarc openarc[920]: 2026-02-08 15:59:33,304 - INFO - q8d loaded successfully
Feb 08 15:59:33 openarc openarc[920]: 2026-02-08 15:59:33,327 - INFO - [q8d LLM Worker] Started, waiting for packets...
Feb 08 15:59:33 openarc openarc[920]: 2026-02-08 15:59:33,332 - INFO - Request completed: POST /openarc/load status=200 duration=29.028s
Feb 08 15:59:42 openarc openarc[920]: 2026-02-08 15:59:42,074 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 08 15:59:42 openarc openarc[920]: 2026-02-08 15:59:42,098 - INFO - Load failed! model_name 'q8d' already exists
Feb 08 15:59:42 openarc openarc[920]: 2026-02-08 15:59:42,100 - INFO - Request completed: POST /openarc/load status=400 duration=0.026s
Feb 08 15:59:48 openarc openarc[920]: 2026-02-08 15:59:48,134 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 15:59:48 openarc openarc[920]: 2026-02-08 15:59:48,136 - INFO - Request completed: GET /v1/models status=200 duration=0.002s
Feb 08 15:59:52 openarc openarc[920]: 2026-02-08 15:59:52,373 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 15:59:52 openarc openarc[920]: 2026-02-08 15:59:52,373 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 16:00:29 openarc openarc[920]: 2026-02-08 16:00:29,901 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:00:29 openarc openarc[920]: 2026-02-08 16:00:29,902 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 16:00:33 openarc openarc[920]: 2026-02-08 16:00:33,932 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:00:33 openarc openarc[920]: 2026-02-08 16:00:33,941 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.009s
Feb 08 16:00:34 openarc openarc[920]: 2026-02-08 16:00:34,138 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:00:34 openarc openarc[920]: 2026-02-08 16:00:34,138 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:00:34 openarc openarc[920]: 2026-02-08 16:00:34,139 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:00:34 openarc openarc[920]: 2026-02-08 16:00:34,139 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:00:41 openarc openarc[920]: 2026-02-08 16:00:41,204 - INFO - [q8d LLM Worker] Metrics: {'load_time (s)': 28.4, 'ttft (s)': 2.83, 'tpot (ms)': 31.96734, 'prefill_throughput (tokens/s)': 3001.45, 'decode_throughput (tokens/s)': 31.28193, 'decode_duration (s)': 7.05892, 'input_token': 8507, 'new_token': 133, 'total_token': 8640, 'stream': True, 'stream_chunk_tokens': 1}
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,488 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,490 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.002s
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,521 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,521 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,521 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,521 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,572 - ERROR - LLM inference failed!
Feb 08 16:00:51 openarc openarc[920]: Traceback (most recent call last):
Feb 08 16:00:51 openarc openarc[920]:   File "/root/OpenArc/src/server/worker_registry.py", line 87, in infer_llm
Feb 08 16:00:51 openarc openarc[920]:     async for item in llm_instance.generate_type(packet.gen_config):
Feb 08 16:00:51 openarc openarc[920]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 200, in generate_stream
Feb 08 16:00:51 openarc openarc[920]:     result = await gen_task
Feb 08 16:00:51 openarc openarc[920]:              ^^^^^^^^^^^^^^
Feb 08 16:00:51 openarc openarc[920]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 179, in _run_generation
Feb 08 16:00:51 openarc openarc[920]:     return await asyncio.to_thread(
Feb 08 16:00:51 openarc penarc[920]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:00:51 openarc openarc[920]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
Feb 08 16:00:51 openarc openarc[920]:     return await loop.run_in_executor(None, func_call)
Feb 08 16:00:51 openarc openarc[920]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:00:51 openarc openarc[920]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Feb 08 16:00:51 openarc openarc[920]:     result = self.fn(*self.args, **self.kwargs)
Feb 08 16:00:51 openarc openarc[920]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:00:51 openarc openarc[920]: RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
Feb 08 16:00:51 openarc openarc[920]: Check 'intermediates_memories.size() > sequential_gws_subseq_mapping_idx' failed at src/plugins/intel_gpu/src/graph/impls/ocl_v2/sdpa/paged_attention_opt.cpp:1745:
Feb 08 16:00:51 openarc openarc[920]: [GPU] Unexpected number of intermediates buffers for Paged Attention for mixed stage
Feb 08 16:00:51 openarc openarc[920]: 2026-02-08 16:00:51,579 - ERROR - [q8d LLM Worker] Inference failed, triggering model unload...
Feb 08 16:00:54 openarc openarc[920]: 2026-02-08 16:00:54,015 - INFO - [q8d] unloaded successfully

CPU - NousCoder:

Feb 08 16:11:43 openarc openarc[920]: 2026-02-08 16:11:43,717 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 08 16:11:43 openarc openarc[920]: 2026-02-08 16:11:43,718 - INFO - n14c loading...
Feb 08 16:11:43 openarc openarc[920]: 2026-02-08 16:11:43,719 - INFO - ModelType.LLM on CPU with {}
Feb 08 16:11:47 openarc openarc[920]: 2026-02-08 16:11:47,824 - INFO - n14c loaded successfully
Feb 08 16:11:47 openarc openarc[920]: 2026-02-08 16:11:47,826 - INFO - [n14c LLM Worker] Started, waiting for packets...
Feb 08 16:11:47 openarc openarc[920]: 2026-02-08 16:11:47,827 - INFO - Request completed: POST /openarc/load status=200 duration=4.110s
Feb 08 16:11:51 openarc openarc[920]: 2026-02-08 16:11:51,032 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:11:51 openarc openarc[920]: 2026-02-08 16:11:51,033 - INFO - Request completed: GET /v1/models status=200 duration=0.002s
Feb 08 16:11:56 openarc openarc[920]: 2026-02-08 16:11:56,029 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:11:56 openarc openarc[920]: 2026-02-08 16:11:56,029 - INFO - Request completed: GET /v1/models status=200 duration=0.000s
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,871 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,878 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.007s
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,899 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,899 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,899 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:12:02 openarc openarc[920]: 2026-02-08 16:12:02,899 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0

iGPU - NousCoder:

root@openarc:~# openarc load n14i
loading n14i
...working
Request failed: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

────────────────────────────────────────────────────────────
All models failed to load! (0/1)
   ✗ Failed: n14i

dGPU - NousCoder:

Feb 08 16:22:37 openarc openarc[576]: 2026-02-08 16:22:37,568 - INFO - Request received: POST /openarc/load from 127.0.0.1
Feb 08 16:22:37 openarc openarc[576]: 2026-02-08 16:22:37,571 - INFO - n14d loading...
Feb 08 16:22:37 openarc openarc[576]: 2026-02-08 16:22:37,571 - INFO - ModelType.LLM on GPU.1 with {}
Feb 08 16:22:39 openarc openarc[576]: WARNING: Resizable BAR not detected for device 0000:31:00.0
Feb 08 16:23:07 openarc openarc[576]: 2026-02-08 16:23:07,522 - INFO - n14d loaded successfully
Feb 08 16:23:07 openarc openarc[576]: 2026-02-08 16:23:07,557 - INFO - [n14d LLM Worker] Started, waiting for packets...
Feb 08 16:23:07 openarc openarc[576]: 2026-02-08 16:23:07,566 - INFO - Request completed: POST /openarc/load status=200 duration=29.996s
Feb 08 16:23:14 openarc openarc[576]: 2026-02-08 16:23:14,214 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:23:14 openarc openarc[576]: 2026-02-08 16:23:14,234 - INFO - Request completed: GET /v1/models status=200 duration=0.021s
Feb 08 16:23:21 openarc openarc[576]: 2026-02-08 16:23:21,758 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:23:21 openarc openarc[576]: 2026-02-08 16:23:21,761 - INFO - Request completed: GET /v1/models status=200 duration=0.003s
Feb 08 16:23:22 openarc openarc[576]: 2026-02-08 16:23:22,699 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:23:22 openarc openarc[576]: 2026-02-08 16:23:22,701 - INFO - Request completed: GET /v1/models status=200 duration=0.003s
Feb 08 16:23:27 openarc openarc[576]: 2026-02-08 16:23:27,099 - INFO - Request received: GET /v1/models from 172.20.20.200
Feb 08 16:23:27 openarc openarc[576]: 2026-02-08 16:23:27,100 - INFO - Request completed: GET /v1/models status=200 duration=0.001s
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,449 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,499 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.050s
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,795 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,795 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,795 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:23:30 openarc openarc[576]: 2026-02-08 16:23:30,795 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:23:39 openarc openarc[576]: 2026-02-08 16:23:39,114 - INFO - [n14d LLM Worker] Metrics: {'load_time (s)': 29.43, 'ttft (s)': 4.1, 'tpot (ms)': 37.29065, 'prefill_throughput (tokens/s)': 2076.44, 'decode_throughput (tokens/s)': 26.81637, 'decode_duration (s)': 8.31339, 'input_token': 8507, 'new_token': 114, 'total_token': 8621, 'stream': True, 'stream_chunk_tokens': 1}
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,290 - INFO - Request received: POST /v1/chat/completions from 172.20.20.200
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,292 - INFO - Request completed: POST /v1/chat/completions status=200 duration=0.001s
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,309 - ERROR - [DEBUG] draft_model_loaded: False
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,309 - ERROR - [DEBUG] self.model_num_assistant_tokens: None
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,309 - ERROR - [DEBUG] generation_kwargs.num_assistant_tokens: 0
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,309 - ERROR - [DEBUG] generation_kwargs.assistant_confidence_threshold: 0.0
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,416 - ERROR - LLM inference failed!
Feb 08 16:24:07 openarc openarc[576]: Traceback (most recent call last):
Feb 08 16:24:07 openarc openarc[576]:   File "/root/OpenArc/src/server/worker_registry.py", line 87, in infer_llm
Feb 08 16:24:07 openarc openarc[576]:     async for item in llm_instance.generate_type(packet.gen_config):
Feb 08 16:24:07 openarc openarc[576]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 200, in generate_stream
Feb 08 16:24:07 openarc openarc[576]:     result = await gen_task
Feb 08 16:24:07 openarc openarc[576]:              ^^^^^^^^^^^^^^
Feb 08 16:24:07 openarc openarc[576]:   File "/root/OpenArc/src/engine/ov_genai/llm.py", line 179, in _run_generation
Feb 08 16:24:07 openarc openarc[576]:     return await asyncio.to_thread(
Feb 08 16:24:07 openarc openarc[576]:            ^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:24:07 openarc openarc[576]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/asyncio/threads.py", line 25, in to_thread
Feb 08 16:24:07 openarc openarc[576]:     return await loop.run_in_executor(None, func_call)
Feb 08 16:24:07 openarc openarc[576]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:24:07 openarc openarc[576]:   File "/root/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Feb 08 16:24:07 openarc openarc[576]:     result = self.fn(*self.args, **self.kwargs)
Feb 08 16:24:07 openarc openarc[576]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 08 16:24:07 openarc openarc[576]: RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:224:
Feb 08 16:24:07 openarc openarc[576]: Check 'intermediates_memories.size() > sequential_gws_subseq_mapping_idx' failed at src/plugins/intel_gpu/src/graph/impls/ocl_v2/sdpa/paged_attention_opt.cpp:1745:
Feb 08 16:24:07 openarc openarc[576]: [GPU] Unexpected number of intermediates buffers for Paged Attention for mixed stage
Feb 08 16:24:07 openarc openarc[576]: 2026-02-08 16:24:07,437 - ERROR - [n14d LLM Worker] Inference failed, triggering model unload...
Feb 08 16:24:10 openarc openarc[576]: 2026-02-08 16:24:10,040 - INFO - [n14d] unloaded successfully

[SearchSavior]

@DrazorV Failure on iGPU and eGPU tells me maybe your dependencies are having an issue, but I'm not sure.

To continue testing this I recommend not using OpenArc; here is the smallest snippet needed to reproduce in case this ends up being an upstream issue.

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline(model_path, "CPU")
print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

What version of oneapi do you have installed? Since i cannot reproduce there must be something going on with your GPU dependencies. One time I used the offline installer script in place of APT when I was having issues which might have been related to oneapi.

https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?packages=oneapi-toolkit&oneapi-toolkit-os=linux&oneapi-lin=offline

You are right, CPU with 14B was likely taking a while. I shared nouscoder since it was a recent export; this one should be much faster to test.

What kernel version are you running? Is the docker image you are using from the dockerfile in OpenArc?

[SearchSavior]

@DrazorV Hey man, any update- did you end up getting that occulink?

[DrazorV]

@DrazorV Hey man, any update- did you end up getting that occulink?

Hey, I received the occulink yesterday and I did some testing, it still gives me some errors but it is different than before, I have not posted anything yet because I have not yet collected the logs and I am not sure I have found exactly what is happening but I was planning to do next week, it is a local holiday here and it is hard to find some time to spend on this now. I will keep you updated since I am very invested on the project. The performance difference between OpenArc and llama.cpp is night and day.