Research: FreeToken is faster than llama.cpp

[alphmeric]

Research Stage

• Background Research (Let's try to avoid reinventing the wheel)
• Hypothesis Formed (How do you think this will work and it's effect?)
• Strategy / Implementation Forming
• Analysis of results
• Debrief / Documentation (So people in the future can learn from us)

Previous existing literature and research

https://www.youtube.com/watch?v=Mlo16hrA5wQ

Hypothesis

Now the FreeToken is available, llama.cpp is deprecated for it slowness.

Implementation

Analyze FreeToken how it works and improve the llama.cpp speed.

Analysis

No response

Relevant log output

[vbooka1]

I think OP meant this software https://github.com/FlashML-org/FreeToken

If I'm right then it is not relevant because that software is written in Python.

[EJainDev]

Although the software itself is in Python, giving it a higher overhead, llama.cpp could definitely employ some of the present techniques to improve performance.

[netroy]

llama.cpp is deprecated for it slowness

ah, the classic emotional manipulation techniques that open-source maintainers love 👏🏽

[nikwen]

llama.cpp is deprecated

Definitely not deprecated.

[am17an]

The prefill optimization they mentioned in the paper is already implemented in a PR here #21067. The decode optimization is interesting, I will probably do a PoC for that in sometime, but even those you can find PRs under "moe-cache" which many people sloppily implement.

However note that the paper uses CUDA + unpinned memory. So if you want to run a 122B model you need at least ~64GB pinned ram for this to work. RAM is not the cheap resource it used to be -- buying a 128GB unified mem machine like the strix halo might be cheaper than buying 128GB DDR5.

[brrreaker]

Don't be limited by your imagination on this. Yes, it requires sysram, lots of it, but think of all the workstation/server gear that has tons of memory, but can only use one gpu, with the current llama.cpp those can't be used anywhere near their potential. There are many of these systems laying around at companies, completely unused, this method can pull them out of "e-waste". And even on my desktop, I could spare 35GB of sysram, using freetoken I can run qwen 35b at 3000-4000t/s PP and 40-50t/s TG with a 5070Ti, and it instantly replaced the 9B model in my workflow. Unfortunately based on the github activity, freetoken project itself already seems to be abandonware, and it barely has been a few days.

[dam9000]

I can confirm I get more than 2x token generation on FreeToken vs llama.cpp with Qwen3.8-flash-next Q4 on rtx 5090 (32GB vram) and 128GB sys ram. llama.cpp: 22 t/s FreeToken: 53 t/s. gpu/cpu utilisation in llama.cpp is 20%/1200%, with freetoken is 100%/100%. So there's even room for improvement if cpu would be loaded with work too. Please consider adopting and implementing the methods used in FreeToken in llama.cpp too!

[jan-johansson-mr]

Don't be limited by your imagination on this. Yes, it requires sysram, lots of it, but think of all the workstation/server gear that has tons of memory, but can only use one gpu, with the current llama.cpp those can't be used anywhere near their potential. There are many of these systems laying around at companies, completely unused, this method can pull them out of "e-waste". And even on my desktop, I could spare 35GB of sysram, using freetoken I can run qwen 35b at 3000-4000t/s PP and 40-50t/s TG with a 5070Ti, and it instantly replaced the 9B model in my workflow. Unfortunately based on the github activity, freetoken project itself already seems to be abandonware, and it barely has been a few days.

Yep, this is a research project. I doubt that the authors want to do a full fledged mature solution as llama.cpp. The authors has proven their point that it is possible to improve MoE inference.

[jan-johansson-mr]

I can confirm I get more than 2x token generation on FreeToken vs llama.cpp with Qwen3.8-flash-next Q4 on rtx 5090 (32GB vram) and 128GB sys ram. llama.cpp: 22 t/s FreeToken: 53 t/s. gpu/cpu utilisation in llama.cpp is 20%/1200%, with freetoken is 100%/100%. So there's even room for improvement if cpu would be loaded with work too. Please consider adopting and implementing the methods used in FreeToken in llama.cpp too!

That would be fantastic.