![[2502.05795] The Curse of Depth in Large Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2502.05795] The Curse of Depth in Large Language Models
Summary
This paper introduces the 'Curse of Depth' in Large Language Models (LLMs), revealing that many deep layers are ineffective due to Pre-Layer Normalization. It proposes LayerNorm Scaling (LNS) to enhance training performance across various model sizes.
Why It Matters
Understanding the limitations of depth in LLMs is crucial for improving their training efficiency and effectiveness. The proposed LayerNorm Scaling addresses a significant issue that could lead to better-performing models, impacting various applications in AI and machine learning.
Key Takeaways
- The 'Curse of Depth' highlights inefficiencies in LLM layers.
- Pre-Layer Normalization contributes to ineffective deep layers.
- LayerNorm Scaling (LNS) improves training performance significantly.
- The findings apply across a range of model sizes from 130M to 7B.
- LNS enhances both pre-training and supervised fine-tuning outcomes.
Computer Science > Machine Learning arXiv:2502.05795 (cs) [Submitted on 9 Feb 2025 (v1), last revised 22 Feb 2026 (this version, v5)] Title:The Curse of Depth in Large Language Models Authors:Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu View a PDF of the paper titled The Curse of Depth in Large Language Models, by Wenfang Sun and 5 other authors View PDF HTML (experimental) Abstract:In this paper, we introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling (LNS), which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transf...
