[2604.01978] Homogenized Transformers

Mathematics > Probability arXiv:2604.01978 (math) [Submitted on 2 Apr 2026] Title:Homogenized Transformers Authors:Hugo Koubbi, Borjan Geshkovski, Philippe Rigollet View a PDF of the paper titled Homogenized Transformers, by Hugo Koubbi and 2 other authors View PDF Abstract:We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth as a time variable, the residual stream defines a discrete-time interacting particle system on the unit sphere. We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivial homogenized limit. Depending on the scaling, the limit is either deterministic or stochastic with common noise; in the mean-field regime, the latter leads to a stochastic nonlinear Fokker--Planck equation for the conditional law of a representative token. In the Gaussian setting, the limiting drift vanishes, making the homogenized dynamics explicit enough to study representation collapse. This yields quantitative trade-offs between dimension, context length, and temperature, and identifies regimes in which clustering can be mitigated. Subjects: Probability (math.PR); Machine Learning (cs.LG); Machine Learning (stat.ML) Cite as: arXiv:2604.01978 [math.PR]   (or arXiv:2604.01978v1 [math.PR] for this version)   https://doi.org/10.48550/arXiv.2604.01978 Focus to learn more arXiv-is...