RuntimeError: flashinfer-cubin version (0.7.0) does not match flashinfer version (0.6.18.post1)

[Lolmc0587]

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

Can't run any models. Already delete old venv and start install again from Free Desktop app.

How did you install FreeToken

pip / uv wheel

FreeToken version

freetoken version 0.1.3+gcc1f5c2c9

OS

Arch Linux

OS details

No response

GPU and driver

RTX 5050 Laptop, 8gb VRAM

CPU and system RAM

i7-13650HX, 32GB Ram

Checkpoint

nvidia/Qwen3.6-35B-A3B-NVFP4

Command

Full log

[20:27:13] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Qwen3.6-35B-A3B-NVFP4 --port 1919 --moe-backend hybrid --max-running-requests 4 --memory-ratio 0.88 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[20:27:22] stdout/WARN  [2026-09-23|20:27:22] WARNING  --moe-backend is deprecated; use --moe-strategy
[20:27:22] stdout/INFO  [2026-09-23|20:27:22] INFO     Parsed arguments:
[20:27:22] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Qwen3.6-35B-A3B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_strategy='hybrid', quant_backend=None, ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.88, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, mm=MultimodalConfig(disabled_encoders=frozenset(), embed_cache_device='cpu', encoder_weights='host', image_min_tokens=None, image_max_tokens=None, processor_kwargs={}), max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=57188', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.6-35B-A3B-NVFP4', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, allowed_media_domains='', allowed_local_media_path='', cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=(), gpu_assigned=None)
[20:27:22] stdout/INFO  [2026-09-23|20:27:22|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
[20:27:22] stderr/INFO  INFO:     Started server process [57188]
[20:27:22] stderr/INFO  INFO:     Waiting for application startup.
[20:27:22] stderr/INFO  INFO:     Application startup complete.
[20:27:22] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[20:27:24] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1168: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[20:27:24] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[20:27:24] stdout/INFO  [2026-09-23|20:27:24|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[20:27:25] stdout/INFO  [2026-09-23|20:27:25|core|rank=0] INFO     Auto-selected attention backend: fi
[20:27:25] stdout/INFO  [2026-09-23|20:27:25|core|rank=0] INFO     Resolved config: moe_strategy='hybrid', attention_backend='fi', cache_type='hybrid_radix', page_size=1
[20:27:25] stdout/INFO  [2026-09-23|20:27:25|core|rank=0] INFO     Free memory before loading model: 7.21 GiB
[20:27:25] stderr/INFO  Process freetoken-TP0-scheduler:
[20:27:25] stdout/ERROR  [2026-09-23|20:27:25|FrontendAPI] ERROR    Backend supervisor: RuntimeError: flashinfer-cubin version (0.7.0) does not match flashinfer version (0.6.18.post1). Please install the same version of both packages. Set FLASHINFER_DISABLE_VERSION_CHECK=1 to bypass this check.
[20:27:25] stderr/INFO  Traceback (most recent call last):
[20:27:25] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[20:27:25] stderr/INFO      self.run()
[20:27:25] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[20:27:25] stderr/INFO      self._target(*self._args, **self._kwargs)
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[20:27:25] stderr/INFO      scheduler = Scheduler(args)
[20:27:25] stderr/INFO                  ^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 68, in __init__
[20:27:25] stderr/INFO      self.engine = Engine(config)
[20:27:25] stderr/INFO                    ^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 359, in __init__
[20:27:25] stderr/INFO      self.model = create_model(config.model_config)
[20:27:25] stderr/INFO                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/__init__.py", line 18, in create_model
[20:27:25] stderr/INFO      return get_model_class(model_config.architectures[0], model_config)
[20:27:25] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/register.py", line 337, in get_model_class
[20:27:25] stderr/INFO      return model_cls(model_config)
[20:27:25] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/qwen3_5_moe/model.py", line 124, in __init__
[20:27:25] stderr/INFO      super().__init__(config)
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/qwen3_5_moe/model.py", line 102, in __init__
[20:27:25] stderr/INFO      self.model = Qwen3_5Model(config)
[20:27:25] stderr/INFO                   ^^^^^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/qwen3_5_moe/model.py", line 85, in __init__
[20:27:25] stderr/INFO      Qwen3_5DecoderLayer(config, layer_id, prefix=f"{prefix}.layers.{layer_id}")
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/models/qwen3_5_moe/model.py", line 59, in __init__
[20:27:25] stderr/INFO      self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
[20:27:25] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/layers/norm.py", line 40, in __init__
[20:27:25] stderr/INFO      from sgl_kernel import fused_add_rmsnorm, rmsnorm
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/sgl_kernel/__init__.py", line 35, in <module>
[20:27:25] stderr/INFO      from sgl_kernel.elementwise import (
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/sgl_kernel/elementwise.py", line 7, in <module>
[20:27:25] stderr/INFO      import flashinfer.norm as _flashinfer_norm
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/__init__.py", line 25, in <module>
[20:27:25] stderr/INFO      from . import jit as jit
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/jit/__init__.py", line 22, in <module>
[20:27:25] stderr/INFO      from . import cubin_loader
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/jit/cubin_loader.py", line 30, in <module>
[20:27:25] stderr/INFO      from .core import logger
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/jit/core.py", line 15, in <module>
[20:27:25] stderr/INFO      from . import env as jit_env
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/jit/env.py", line 110, in <module>
[20:27:25] stderr/INFO      FLASHINFER_CUBIN_DIR: pathlib.Path = _get_cubin_dir()
[20:27:25] stderr/INFO                                           ^^^^^^^^^^^^^^^^
[20:27:25] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/flashinfer/jit/env.py", line 97, in _get_cubin_dir
[20:27:25] stderr/INFO      raise RuntimeError(
[20:27:25] stderr/INFO  RuntimeError: flashinfer-cubin version (0.7.0) does not match flashinfer version (0.6.18.post1). Please install the same version of both packages. Set FLASHINFER_DISABLE_VERSION_CHECK=1 to bypass this check.
[20:27:26] health/ERROR  RuntimeError: flashinfer-cubin version (0.7.0) does not match flashinfer version (0.6.18.post1). Please install the same version of both packages. Set FLASHINFER_DISABLE_VERSION_CHECK=1 to bypass this check.
[20:27:26] stderr/INFO  INFO:     Shutting down
[20:27:26] stderr/INFO  INFO:     Waiting for application shutdown.
[20:27:26] stderr/INFO  INFO:     Application shutdown complete.
[20:27:26] stderr/INFO  INFO:     Finished server process [57188]
[20:27:26] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[20:27:26] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '

Anything else

No response

[Joeppie]

I have the precise same issue, same model, my GPU is a RTX 4080 mobile, however.
RuntimeError: flashinfer-cubin version (0.7.0) does not match flashinfer version (0.6.18.post1). Please install the same version of both packages. Set FLASHINFER_DISABLE_VERSION_CHECK=1 to bypass this check.

It looks like something weird is up with the versio of flashinfer-cubin I am trying to make a local workaround and will post update;

flashinfer-cubin (0.6.13)
Available versions: 0.6.13, 0.6.12, 0.6.11.post3, 0.6.11.post2, 0.6.11.post1, 0.6.11, 0.6.10.post1, 0.6.10, 0.6.9, 0.6.8.post1, 0.6.8, 0.6.7.post3, 0.6.7.post2, 0.6.7.post1, 0.6.7, 0.6.6, 0.6.5, 0.6.4, 0.6.3, 0.6.2, 0.6.1, 0.6.0, 0.5.3, 0.5.2, 0.5.1, 0.5.0, 0.4.1, 0.4.0
  INSTALLED: 0.7.0
  LATEST:    0.6.13

I can´t get it to work, not even when downloading fresh, or building from source. I have never used uv, but that also doesn´t work on my CachyOS machine. So I am out of a working inference. Arch Linux as well (CachyOS, but that is irrelvant.) Same issue with App Image.

It looks like flashinfer.ai made a new release yesterday. This is now being picked up and erroring, because freetoken does not request the specific version it relies on.

Anyone trying to fix thisd locally, I just managed to get it to work locally, but probably Jason-fxz' fix is better, summary:

The current install.sh is pulling incompatible FlashInfer versions on a fresh install. I fixed it by pinning the entire FlashInfer stack to 0.6.18.post1.

1. Change pyproject.toml

Replace:

fi = ["flashinfer-python[cu13]>=0.6,<0.7"]

with:

fi = ["flashinfer-python[cu13]==0.6.18.post1"]

2. Change install.sh

Add:

FLASHINFER_VERSION="0.6.18.post1"

and change:

INSTALL_WHEELS=(
  "${WHEEL}[accel]"
  flashinfer-cubin
  flashinfer-jit-cache
  "$KERNEL_CACHE_WHEEL"
)

to:

INSTALL_WHEELS=(
  "${WHEEL}[accel]"
  "flashinfer-cubin==${FLASHINFER_VERSION}"
  "flashinfer-jit-cache==${FLASHINFER_VERSION}+cu130"
  "$KERNEL_CACHE_WHEEL"
)

3. Do a clean build

If you have already built FreeToken, remove the old virtualenv and wheel so a stale Python-version-specific wheel isn't reused:

rm -rf .venv dist
uv venv .venv --python 3.12
uv sync --extra accel
./install.sh

The Python version matters because install.sh creates the runtime environment with Python 3.12. Building the wheel with Python 3.13 can produce a cp313 native extension that then gets installed into the Python 3.12 runtime.

4. Verify the installed FlashInfer versions

~/.freetoken/venv/bin/python -c '
import importlib.metadata as m
for p in ("freetoken", "flashinfer-python", "flashinfer-cubin", "flashinfer-jit-cache"):
    print(f"{p}: {m.version(p)}")
'

Expected:

freetoken: 0.1.3
flashinfer-python: 0.6.18.post1
flashinfer-cubin: 0.6.18.post1
flashinfer-jit-cache: 0.6.18.post1+cu130

The important fix is that all three FlashInfer components are pinned to the same 0.6.18.post1 release. The previous setup only constrained flashinfer-python, while flashinfer-cubin and flashinfer-jit-cache were unpinned and therefore resolved to 0.7.0.

[jason-fxz]

Thanks for the report. Root cause: install.sh installed flashinfer-cubin / flashinfer-jit-cache without a version, so after flashinfer 0.7.0 was released they resolved to 0.7.0 while flashinfer-python stayed at 0.6.18.post1, and flashinfer refuses to import with mismatched versions. Every fresh install since 0.7.0 was affected, independent of GPU or distro.

quick workaround:

uv pip install --python ~/.freetoken/venv \
    --index-strategy unsafe-best-match \
    --extra-index-url https://flashinfer.ai/whl --extra-index-url https://flashinfer.ai/whl/cu130 \
    "flashinfer-cubin==0.6.18.post1" "flashinfer-jit-cache==0.6.18.post1"

Desktop will update soon to fix this problem.

[jason-fxz]

Fixed in Desktop v0.2.0-beta