![[2602.20461] Nonparametric Teaching of Attention Learners](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20461] Nonparametric Teaching of Attention Learners
Summary
This article presents a novel teaching paradigm called Attention Neural Teaching (AtteNT) that enhances the efficiency of attention learners, such as transformers, by optimizing the learning process through nonparametric methods.
Why It Matters
As attention mechanisms become central to machine learning models, improving their training efficiency without sacrificing accuracy is crucial. This research offers a new framework that could significantly reduce training times for large models, making advanced AI more accessible and efficient.
Key Takeaways
- Introduces Attention Neural Teaching (AtteNT) to optimize training of attention learners.
- Demonstrates a reduction in training time by 13.01% for LLMs and 20.58% for ViTs.
- Maintains or enhances model accuracy across various downstream tasks.
- Utilizes nonparametric teaching methods to improve learning efficiency.
- Provides a theoretical framework for better understanding attention mechanisms.
Computer Science > Machine Learning arXiv:2602.20461 (cs) [Submitted on 24 Feb 2026] Title:Nonparametric Teaching of Attention Learners Authors:Chen Zhang, Jianghui Wang, Bingyang Cheng, Zhongtao Chen, Wendong XU, Cong Wang, Marco Canini, Francesco Orabona, Yik Chung WU, Ngai Wong View a PDF of the paper titled Nonparametric Teaching of Attention Learners, by Chen Zhang and 9 other authors View PDF HTML (experimental) Abstract:Attention learners, neural networks built on the attention mechanism, e.g., transformers, excel at learning the implicit relationships that relate sequences to their corresponding properties, e.g., mapping a given sequence of tokens to the probability of the next token. However, the learning process tends to be costly. To address this, we present a novel paradigm named Attention Neural Teaching (AtteNT) that reinterprets the learning process through a nonparametric teaching perspective. Specifically, the latter provides a theoretical framework for teaching mappings that are implicitly defined (i.e., nonparametric) via example selection. Such an implicit mapping is embodied through a dense set of sequence-property pairs, with the AtteNT teacher selecting a subset to accelerate convergence in attention learner training. By analytically investigating the role of attention on parameter-based gradient descent during training, and recasting the evolution of attention learners, shaped by parameter updates, through functional gradient descent in nonparametri...