How does LiteRT NPU acceleration compare with CPU int4 inference on modern Qualcomm chips?

[PrinceBad]

When running local language models like Qwen 2.5 1.5B or Phi-4 mini on Android devices, what are the primary throughput and power efficiency trade-offs between utilizing the Hexagon NPU via LiteRT delegate versus CPU int4 multithreaded GEMM execution?

[PrinceBad]

Technical Analysis: NPU (Hexagon via LiteRT) vs. CPU int4 Multithreading

1. Tokens Per Second (Prefill & Decode):

   • Hexagon NPU: Provides massive parallel compute for matrix-vector multiplications during decode phase, achieving up to 2.5x - 4x lower latency per token (18-28 tok/sec on Snapdragon 8 Gen 2/3) compared to CPU.
   • Kryo/Cortex CPU: Bound by memory bandwidth and thermal throttling; decode hovers around 6-11 tok/sec.

2. Thermal & Energy Footprint:

   • LiteRT with NPU offload operates within a strict DSP power budget (~2.5W - 3.8W), avoiding thermal runaway during sustained reasoning runs.
   • Multithreaded CPU saturation causes aggressive thermal throttling after 40-60 seconds of continuous generation, dropping performance by 35-50%.

3. Memory Footprint:

   • Both utilize int4 / int8 weight quantization schemes, but NPU delegates require zero-copy DMA buffers, which reduces Android GC pressure significantly.