![[2602.15322] On Surprising Effectiveness of Masking Updates in Adaptive Optimizers](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.15322] On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
Summary
This paper explores the effectiveness of randomly masking updates in adaptive optimizers for training large language models, introducing a new method called Momentum-aligned gradient masking (Magma) that shows significant performance improvements.
Why It Matters
As machine learning models, particularly large language models, become increasingly complex, optimizing their training processes is crucial. This research challenges traditional adaptive optimizers by demonstrating that masking updates can enhance performance, potentially leading to more efficient training methodologies.
Key Takeaways
- Randomly masking parameter updates can outperform traditional adaptive optimizers.
- The new method, Magma, provides consistent performance gains with minimal computational overhead.
- Magma reduces perplexity significantly compared to established optimizers like Adam.
- The study introduces a curvature-dependent geometric regularization that smooths optimization trajectories.
- This research could influence future approaches to training large language models.
Computer Science > Machine Learning arXiv:2602.15322 (cs) [Submitted on 17 Feb 2026] Title:On Surprising Effectiveness of Masking Updates in Adaptive Optimizers Authors:Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, Eugene Ie View a PDF of the paper titled On Surprising Effectiveness of Masking Updates in Adaptive Optimizers, by Taejong Joo and 4 other authors View PDF HTML (experimental) Abstract:Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming recent state-of-the-art optimizers. Our analysis reveals that the random masking induces a curvature-dependent geometric regularization that smooths the optimization trajectory. Motivated by this finding, we introduce Momentum-aligned gradient masking (Magma), which modulates the masked updates using momentum-gradient alignment. Extensive LLM pre-training experiments show that Magma is a simple drop-in replacement for adaptive optimizers with consistent gains and negligible computational overhead. Notably, for the 1B model size, Magma reduces perplexity by over 19\% and 9\% compared to Adam and Muon, respectively. Comments: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI) Cite as: arXiv:2602.15322 [cs.LG] (or arXiv:2602.15322v1 [cs.LG] for this version) https://d...