![[2602.17898] Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17898] Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors
Summary
This paper explores the limitations of attention-based regression models, particularly the phenomenon of the Pearson correlation coefficient (PCC) plateau during training, and proposes a novel approach to enhance performance.
Why It Matters
Understanding the PCC plateau is crucial for improving the effectiveness of attention-based models in machine learning. This research addresses fundamental optimization challenges and introduces a new method, ECA, which could lead to better model performance in various applications, particularly those involving homogeneous data.
Key Takeaways
- The PCC plateau phenomenon limits the effectiveness of attention-based regressors during training.
- Lowering MSE can paradoxically suppress PCC gradient improvements.
- The Extrapolative Correlation Attention (ECA) method effectively breaks the PCC plateau.
- Data homogeneity exacerbates optimization challenges in regression models.
- Theoretical insights provided can guide future developments in model architecture.
Computer Science > Machine Learning arXiv:2602.17898 (cs) [Submitted on 19 Feb 2026] Title:Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors Authors:Jingquan Yan, Yuwei Miao, Peiran Yu, Junzhou Huang View a PDF of the paper titled Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors, by Jingquan Yan and 3 other authors View PDF HTML (experimental) Abstract:Attention-based regression models are often trained by jointly optimizing Mean Squared Error (MSE) loss and Pearson correlation coefficient (PCC) loss, emphasizing the magnitude of errors and the order or shape of targets, respectively. A common but poorly understood phenomenon during training is the PCC plateau: PCC stops improving early in training, even as MSE continues to decrease. We provide the first rigorous theoretical analysis of this behavior, revealing fundamental limitations in both optimization dynamics and model capacity. First, in regard to the flattened PCC curve, we uncover a critical conflict where lowering MSE (magnitude matching) can paradoxically suppress the PCC gradient (shape matching). This issue is exacerbated by the softmax attention mechanism, particularly when the data to be aggregated is highly homogeneous. Second, we identify a limitation in the model capacity: we derived a PCC improvement limit for any convex aggregator (including the softmax attention), showing that the convex hull of ...
