![[2604.01563] Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2604.01563] Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training
About this article
Abstract page for arXiv paper 2604.01563: Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training
Computer Science > Artificial Intelligence arXiv:2604.01563 (cs) [Submitted on 2 Apr 2026] Title:Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training Authors:Abdelrahman Abouzeid (Georgia Institute of Technology) View a PDF of the paper titled Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training, by Abdelrahman Abouzeid (Georgia Institute of Technology) View PDF HTML (experimental) Abstract:In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this assumption can fail: Dynamic Erf (Derf; Chen & Liu, 2025) suffers a large negative interaction with Muon (Jordan, 2024), with its gap to RMSNorm growing from +0.31 nats under AdamW to +0.97 under Muon, approximately three times larger. Dynamic Tanh (DyT; Zhu et al., 2025), included as a bounded-normalizer control, shows no such penalty. Our evidence points to two failure modes of erf under Muon's faster spectral-norm growth: saturation (lossy compression) and scale blindness (discarding activation magnitude). An EMA-blend that reintroduces running scale estimates recovers ~84% of the gap. Separately, reducing Derf's alpha from its published default (0.5 to 0.3) recovers ~80% by keeping erf in its near-linear regime, where it approximately preserves relative scale; this setting is not the published default of Chen & Liu (2025). Usin...
