![[2502.11034] AdaGC: Improving Training Stability for Large Language Model Pretraining](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2502.11034] AdaGC: Improving Training Stability for Large Language Model Pretraining
Summary
The paper presents AdaGC, a novel adaptive gradient clipping method aimed at enhancing training stability in large language model pretraining by addressing loss spikes caused by various factors.
Why It Matters
Training stability is crucial for the performance of large language models. Loss spikes can hinder model accuracy and efficiency. AdaGC provides a solution that is optimizer-agnostic and reduces communication costs, making it a significant advancement for researchers and practitioners in machine learning.
Key Takeaways
- AdaGC mitigates loss spikes during large language model training.
- The method is optimizer-agnostic and integrates seamlessly with existing optimizers.
- Empirical results show significant improvements in accuracy and stability over previous methods.
- AdaGC reduces communication costs in distributed training environments.
- The approach addresses the confluence of multiple factors causing training instability.
Computer Science > Machine Learning arXiv:2502.11034 (cs) [Submitted on 16 Feb 2025 (v1), last revised 22 Feb 2026 (this version, v2)] Title:AdaGC: Improving Training Stability for Large Language Model Pretraining Authors:Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen View a PDF of the paper titled AdaGC: Improving Training Stability for Large Language Model Pretraining, by Guoxia Wang and 7 other authors View PDF HTML (experimental) Abstract:Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause of loss spikes by investigating individual factors, we observe that, in practice, such spikes are typically triggered by the confluence of heterogeneous factors. Empirically, loss spikes may arise from a combination of data outliers, hardware or transient computational faults, numerical precision issues, and hyperparameter settings. Regardless of the underlying cause, these spikes manifest as unstable optimizer updates, as abnormal gradients contaminate both first- and second-moment states. In this paper, we propose a principled gradient-centric remedy: AdaGC, an adaptive per-tensor gradient clipping scheme that mitigates such contamination by bounding gradient norms relative to a tensor-wise exponential moving average of their historical clipped values. AdaGC is optimizer-agnostic, introduces negligible memory overhead, and reduces communic...
