![[2510.11789] Minimax Rates for Learning Pairwise Interactions in Attention-Style Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.11789] Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
Summary
This paper examines the convergence rates for learning pairwise interactions in attention-style models, demonstrating a minimax rate that is independent of certain model parameters, thereby highlighting the statistical efficiency of these models.
Why It Matters
Understanding the minimax rates for learning in attention-style models is crucial for improving training efficiency and performance in machine learning applications. This research provides theoretical insights that can guide practitioners in optimizing model training and understanding the capabilities of attention mechanisms.
Key Takeaways
- The minimax rate for learning pairwise interactions is proven to be $M^{-\frac{2\beta}{2\beta+1}}$.
- This rate is independent of embedding dimension, number of tokens, and rank of the weight matrix under specific conditions.
- The findings emphasize the statistical efficiency of attention-style models in machine learning.
- The results provide a theoretical framework for understanding attention mechanisms.
- Guidance on training attention models is enhanced through these insights.
Statistics > Machine Learning arXiv:2510.11789 (stat) [Submitted on 13 Oct 2025 (v1), last revised 25 Feb 2026 (this version, v2)] Title:Minimax Rates for Learning Pairwise Interactions in Attention-Style Models Authors:Shai Zucker, Xiong Wang, Fei Lu, Inbar Seroussi View a PDF of the paper titled Minimax Rates for Learning Pairwise Interactions in Attention-Style Models, by Shai Zucker and 3 other authors View PDF HTML (experimental) Abstract:We study the convergence rate of learning pairwise interactions in single-layer attention-style models, where tokens interact through a weight matrix and a nonlinear activation function. We prove that the minimax rate is $M^{-\frac{2\beta}{2\beta+1}}$, where $M$ is the sample size and $\beta$ is the Hölder smoothness of the activation function. Importantly, this rate is independent of the embedding dimension $d$, the number of tokens $N$, and the rank $r$ of the weight matrix, provided that $rd \le (M/\log M)^{\frac{1}{2\beta+1}}$. These results highlight a fundamental statistical efficiency of attention-style models, even when the weight matrix and activation are not separately identifiable, and provide a theoretical understanding of attention mechanisms and guidance on training. Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR); Statistics Theory (math.ST) Cite as: arXiv:2510.11789 [stat.ML] (or arXiv:2510.11789v2 [stat.ML] for this version) https://doi.org/10.48550/arXiv.2510.11789 Focus to ...
