![[2602.22271] Support Tokens, Stability Margins, and a New Foundation for Robust LLMs](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.22271] Support Tokens, Stability Margins, and a New Foundation for Robust LLMs
Summary
This article presents a novel probabilistic framework for understanding causal self-attention in LLMs, introducing concepts like support tokens and stability margins to enhance model robustness without sacrificing accuracy.
Why It Matters
As large language models (LLMs) become increasingly integral to AI applications, understanding their underlying mechanics is crucial. This research offers a new perspective that could lead to more stable and reliable models, addressing challenges in LLM training and deployment.
Key Takeaways
- Introduces a probabilistic framework for causal self-attention in LLMs.
- Reveals the concept of support tokens and stability margins for improved model robustness.
- Proposes a Bayesian framework requiring minimal modifications to existing LLM training methods.
- Demonstrates that the new approach enhances out-of-sample accuracy.
- Offers theoretical insights into the dynamics of LLM decoding.
Computer Science > Machine Learning arXiv:2602.22271 (cs) [Submitted on 25 Feb 2026] Title:Support Tokens, Stability Margins, and a New Foundation for Robust LLMs Authors:Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta, Karthik Sethuraman, Tejas Dharamsi View a PDF of the paper titled Support Tokens, Stability Margins, and a New Foundation for Robust LLMs, by Deepak Agarwal and 4 other authors View PDF HTML (experimental) Abstract:Self-attention is usually described as a flexible, content-adaptive way to mix a token with information from its past. We re-interpret causal self-attention transformers, the backbone of modern foundation models, within a probabilistic framework, much like how classical PCA is extended to probabilistic PCA. However, this re-formulation reveals a surprising and deeper structural insight: due to a change-of-variables phenomenon, a barrier constraint emerges on the self-attention parameters. This induces a highly structured geometry on the token space, providing theoretical insights into the dynamics of LLM decoding. This reveals a boundary where attention becomes ill-conditioned, leading to a margin interpretation similar to classical support vector machines. Just like support vectors, this naturally gives rise to the concept of support tokens. Furthermore, we show that LLMs can be interpreted as a stochastic process over the power set of the token space, providing a rigorous probabilistic framework for sequence modeling. We propose a Ba...
