![[2602.17363] 2Mamba2Furious: Linear in Complexity, Competitive in Accuracy](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.17363] 2Mamba2Furious: Linear in Complexity, Competitive in Accuracy
Summary
The paper presents 2Mamba, a linear attention transformer variant that achieves competitive accuracy compared to softmax attention while being more memory efficient, particularly for long context lengths.
Why It Matters
As the demand for efficient machine learning models grows, particularly in natural language processing, this research addresses the trade-off between computational efficiency and accuracy. By improving linear attention mechanisms, it offers a promising alternative that could enhance performance in various applications.
Key Takeaways
- 2Mamba is a new linear attention model that rivals softmax attention in accuracy.
- The model is designed to be more memory efficient, making it suitable for long context lengths.
- Improvements include a simplified structure and enhanced hidden state order.
- The research provides code for reproducibility and further experimentation.
- This work contributes to the ongoing evolution of attention mechanisms in machine learning.
Computer Science > Machine Learning arXiv:2602.17363 (cs) [Submitted on 19 Feb 2026] Title:2Mamba2Furious: Linear in Complexity, Competitive in Accuracy Authors:Gabriel Mongaras, Eric C. Larson View a PDF of the paper titled 2Mamba2Furious: Linear in Complexity, Competitive in Accuracy, by Gabriel Mongaras and 1 other authors View PDF HTML (experimental) Abstract:Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments Subjects: Machine Learning (cs.LG) ACM classes: I.2; I.2.6 Cite as: arXiv:2602.17363 [cs.LG] (or arXiv:2602.17363v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2602.17363 Focus to learn more arXiv-issued DOI via DataCite ...
