![[2411.16085] Cautious Optimizers: Improving Training with One Line of Code](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2411.16085] Cautious Optimizers: Improving Training with One Line of Code
Summary
This article presents a new approach to optimizing training in machine learning by introducing a simple one-line modification to existing momentum-based optimizers, enhancing both speed and stability.
Why It Matters
The research addresses a long-standing challenge in machine learning optimization, particularly for transformer models. By proposing a straightforward modification, it offers a practical solution that could significantly improve training efficiency across various applications, making it relevant for researchers and practitioners in the field.
Key Takeaways
- Introduces a one-line modification to momentum-based optimizers.
- Enhances training speed and stability for large language models.
- Maintains convergence guarantees under Lyapunov analysis.
- Reveals a new family of optimizers, expanding optimization techniques.
- Empirical results show consistent improvements with minimal tuning.
Computer Science > Machine Learning arXiv:2411.16085 (cs) [Submitted on 25 Nov 2024 (v1), last revised 15 Feb 2026 (this version, v4)] Title:Cautious Optimizers: Improving Training with One Line of Code Authors:Kaizhao Liang, Lizhang Chen, Bo Liu, Qiang Liu View a PDF of the paper titled Cautious Optimizers: Improving Training with One Line of Code, by Kaizhao Liang and 3 other authors View PDF HTML (experimental) Abstract:AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a \textbf{one-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at this https URL. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Discrete Mathematics (cs.DM) Cite as: arXiv:2411.16085 [cs.LG] (or arXiv:2411.160...
