ParoQuant

ParoQuant is a quantization method designed to improve the efficiency and accuracy of large language model (LLM) inference, particularly for reasoning tasks that generate long chains of thought. The tool addresses the compounding quantization errors that occur in LLMs when each generated token feeds back into the model, which can significantly degrade performance on benchmarks requiring extended reasoning. ParoQuant introduces a novel, hardware-friendly transform called scaled pairwise rotation, which aims to suppress outlier channels in LLM weights more effectively than existing methods.

The core innovation behind ParoQuant is its use of a small, carefully chosen subset of pairwise (Givens) rotations, rather than full rotations, to transform weight matrices before quantization. By focusing on pairs of channels with the largest magnitude differences and ensuring these pairs are non-overlapping, the method enables all rotations to be executed fully in parallel on the GPU. This approach is designed to be both expressive and efficient, eliminating the need for dense matrix multiplications that slow down other learnable transforms, while still adapting to the weight distribution of each layer. The process involves dividing weights into 128-channel groups, applying per-channel scaling, and then a series of independent pairwise rotations, which can be stacked and combined with additional scaling for greater effectiveness.

ParoQuant supports deployment on NVIDIA GPUs via vLLM and Transformers, as well as on Apple Silicon through MLX. Installation can be performed using pip with the appropriate extras, and it can also be run in a Docker container. The tool provides a command-line interface for chat-based inference with supported models and offers an OpenAI-compatible API server, as well as a built-in agent with tool-calling capabilities. Details about supported models and further technical documentation are available through its Hugging Face collection and GitHub repository.

Developed as a research project and presented in an ICLR 2026 paper, ParoQuant targets users working with LLMs who require efficient quantization for inference, especially in scenarios demanding robust reasoning performance.