Llama.cpp update brings up to 6.5x faster prompt processing on Arm CPUs
A new NEON kernel for llama.cpp dramatically accelerates prompt processing for low-bit quantized models on Arm architecture.
Photo: Hacker News Front Page
Key points
- New NEON kernel adds missing Arm inner kernel for low-bit quantized models in llama.cpp
- Achieved up to 6.5x faster prompt processing speeds on Neoverse N1 hardware
- Token generation speeds and pre-optimized formats remain unchanged
Developers working with local language models on Arm-based hardware have received a significant performance boost following the release of a specialized NEON kernel for llama.cpp. Previously, low-bit quantized models suffered from sluggish prompt reading speeds on Arm processors compared to x86 systems. This bottleneck stemmed from a missing inner kernel for formats such as Q3_K, Q2_K, and the IQ family, forcing the system to repeatedly unpack weights for every dot product.
The newly introduced patch implements the necessary Arm inner kernel utilizing the dot-product instruction (SDOT). Benchmarks conducted across three generations of Arm server cores—using a 512-token prompt with the Qwen2.5 1.5B model—revealed performance gains of up to 6.5x, specifically recorded with the IQ3_M format on a Neoverse N1 processor. Generation speed remains unaffected, and formats that already possessed optimized Arm kernels, such as Q4_K_M and Q8_0, see no changes.
Comprising 164 added lines across two files for llama.cpp v0.9.0, the update is structured to allow easy integration through scripts or patched files. However, the code has not yet been formally submitted to the main repository, as project contribution rules require human developers to fully comprehend and take ownership of AI-assisted submissions before merging.
Source Hacker News Front Page — github.com/hardwiredagent/silicon-forge-arm-neon-study ↗
Read next
🧠AI AnalysisPedestal AI Secures Seed Funding to Automate Logistics With Intelligent Agents
Pedestal AI has launched publicly with $4.7 million in seed backing to deploy agentic artificial intelligence aimed at streamlining logistics and supply chain operations.
🏢Big TechUnited Launch Alliance Investigates Vulcan Rocket Failure
United Launch Alliance is investigating a mission failure of its Vulcan rocket, which suffered from low booster performance shortly after liftoff. The incident raises que…
🌐Work & SocietyUS Southwest Hydropower Output Hits Record Lows Amid Persistent Drought
Hydropower generation in the US Southwest dropped 28% below the 10-year average this summer as record-low water levels in the Colorado River Basin threaten regional energ…
🏢Big TechYouTube Updates View Metrics to Include TV Co-Viewing Estimates
YouTube is incorporating estimated co-viewing data into its public view counts to better reflect group consumption on television sets.
Apple Services Experience Widespread Outage
Apple users are reporting significant disruptions across iCloud, the App Store, and other core services, despite initial discrepancies in the company's status reporting.
⚖️Regulation & PolicyOpenAI Partners with Chilean Ministry to Advance Public Sector AI
OpenAI has signed a non-binding agreement with Chile's science ministry to integrate artificial intelligence into public administration and support local startups.