LIVE

Llama.cpp update brings up to 6.5x faster prompt processing on Arm CPUs

🤖 AI Models 2 min read 👁 5

A new NEON kernel for llama.cpp dramatically accelerates prompt processing for low-bit quantized models on Arm architecture.

Llama.cpp update brings up to 6.5x faster prompt processing on Arm CPUs🤖AI Models

Photo: Hacker News Front Page

Key points

  • New NEON kernel adds missing Arm inner kernel for low-bit quantized models in llama.cpp
  • Achieved up to 6.5x faster prompt processing speeds on Neoverse N1 hardware
  • Token generation speeds and pre-optimized formats remain unchanged

Developers working with local language models on Arm-based hardware have received a significant performance boost following the release of a specialized NEON kernel for llama.cpp. Previously, low-bit quantized models suffered from sluggish prompt reading speeds on Arm processors compared to x86 systems. This bottleneck stemmed from a missing inner kernel for formats such as Q3_K, Q2_K, and the IQ family, forcing the system to repeatedly unpack weights for every dot product.

The newly introduced patch implements the necessary Arm inner kernel utilizing the dot-product instruction (SDOT). Benchmarks conducted across three generations of Arm server cores—using a 512-token prompt with the Qwen2.5 1.5B model—revealed performance gains of up to 6.5x, specifically recorded with the IQ3_M format on a Neoverse N1 processor. Generation speed remains unaffected, and formats that already possessed optimized Arm kernels, such as Q4_K_M and Q8_0, see no changes.

Comprising 164 added lines across two files for llama.cpp v0.9.0, the update is structured to allow easy integration through scripts or patched files. However, the code has not yet been formally submitted to the main repository, as project contribution rules require human developers to fully comprehend and take ownership of AI-assisted submissions before merging.

Source Hacker News Front Page — github.com/hardwiredagent/silicon-forge-arm-neon-study ↗

Contact the newsroom

Got a tip, a proposal or a partnership question? Write to us and we will reply shortly.