[2604.02459] On the Geometric Structure of Layer Updates in Deep Language Models

Computer Science > Machine Learning arXiv:2604.02459 (cs) [Submitted on 2 Apr 2026] Title:On the Geometric Structure of Layer Updates in Deep Language Models Authors:Jun-Sik Yoo View a PDF of the paper titled On the Geometric Structure of Layer Updates in Deep Language Models, by Jun-Sik Yoo View PDF HTML (experimental) Abstract:We study the geometric structure of layer updates in deep language models. Rather than analyzing what information is encoded in intermediate representations, we ask how representations change from one layer to the next. We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes. Across multiple architectures, including Transformers and state-space models, we find that the full layer update is almost perfectly aligned with the tokenwise component, while the residual exhibits substantially weaker alignment, larger angular deviation, and significantly lower projection onto the dominant tokenwise subspace. This indicates that the residual is not merely a small correction, but a geometrically distinct component of the transformation. This geometric separation has functional consequences: approximation error under the restricted tokenwise model is strongly associated with output perturbation, with Spearman correlations often exceeding 0.7 and reaching up to 0.95 in larger models. Together, these results suggest that most layerwise updates behave lik...