Huawei Releases MACaQ Quantizer to Speed Up Local AI Model Inference

Huawei has launched MACaQ, a new internal quantizer aimed at improving local AI inference across multiple modalities including text, images, video, and audio. The tool is compatible with popular frameworks such as llama.cpp, vLLM, Diffusers, and PyTorch, allowing developers to compress large-scale architectures efficiently.

According to the project details, MACaQ can process a 122-billion-parameter model in 64 minutes using 32.5 GB of device memory. The developers claim the tool achieves lower perplexity than MR-GPTQ and DuQuant++, or matches the Llama-3-8B perplexity of competing methods while requiring up to 11 times less quantization time. Additionally, the release includes MACaQ-R, a refinement tool designed to optimize existing GGUF models for specific workloads without breaking compatibility with standard llama.cpp installations.

The software supports various quantization formats, including 2-, 3-, and 4-bit GGUF files, alongside W4A4 dense and grouped matrix multiplication kernels available via CANN pull requests. Licensed under the Apache License 2.0, the repository also provides guidance for handling text Mixture of Experts (MoE) models and speech generators.

What to watch next: Monitor adoption rates among developers working with local inference frameworks and whether the Ascend inference runtime integrations see broader community utilization.