[2604.02525] AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

Computer Science > Machine Learning arXiv:2604.02525 (cs) [Submitted on 2 Apr 2026] Title:AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation Authors:Seonggon Kim, Alireza Khodamoradi, Kristof Denolf, Eunhyeok Park View a PDF of the paper titled AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation, by Seonggon Kim and 3 other authors View PDF HTML (experimental) Abstract:Low-precision training (LPT) commonly employs Hadamard transforms to suppress outliers and mitigate quantization error in large language models (LLMs). However, prior methods apply a fixed transform uniformly, despite substantial variation in outlier structures across tensors. Through the first systematic study of outlier patterns across weights, activations, and gradients of LLMs, we show that this strategy is fundamentally flawed: the effectiveness of Hadamard-based suppression depends on how the transform's smoothing direction aligns with the outlier structure of each operand -- a property that varies substantially across layers and computation paths. We characterize these patterns into three types: Row-wise, Column-wise, and None. Each pair requires a tailored transform direction or outlier handling strategy to minimize quantization error. Based on this insight, we propose AdaHOP (Adaptive Hadamard transform with Outlier-Pattern-aware strategy), which assigns each matrix multiplication its optimal strategy: Inner Hadamard Transform (IHT) wh...