![[2602.12618] Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.12618] Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
Summary
This paper presents Attention-Driven Self-Compression (ADSC), a novel method for reducing vision tokens in Multimodal Large Language Models (MLLMs) while maintaining performance and efficiency.
Why It Matters
As MLLMs become increasingly complex, their computational demands grow. ADSC offers a solution by leveraging the model's own attention mechanisms for efficient token reduction, making it relevant for researchers and practitioners aiming to optimize model performance without sacrificing accuracy.
Key Takeaways
- ADSC reduces computational costs by 53.7% and memory usage by 56.7%.
- The method maintains 98.2% of original model performance.
- ADSC is compatible with FlashAttention and does not require auxiliary modules.
- It outperforms traditional pruning methods in both efficiency and accuracy.
- The approach is robust under high compression ratios compared to heuristic techniques.
Computer Science > Computer Vision and Pattern Recognition arXiv:2602.12618 (cs) [Submitted on 13 Feb 2026] Title:Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models Authors:Omer Faruk Deniz, Ruiyu Mao, Ruochen Li, Yapeng Tian, Latifur Khan View a PDF of the paper titled Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models, by Omer Faruk Deniz and 4 other authors View PDF HTML (experimental) Abstract:Multimodal Large Language Models (MLLMs) incur significant computational cost from processing numerous vision tokens through all LLM layers. Prior pruning methods operate either before the LLM, limiting generality due to diverse encoder-projector designs or within the LLM using heuristics that are incompatible with FlashAttention. We take a different approach: rather than identifying unimportant tokens, we treat the LLM itself as the optimal guide for compression. Observing that deeper layers naturally transmit vision-to-text information, we introduce Attention-Driven Self-Compression (ADSC), a simple, broadly applicable method that progressively reduces vision tokens using only the LLM's attention mechanism. Our method applies uniform token downsampling at selected layers, forming bottlenecks that encourage the model to reorganize and compress information into the remaining tokens. It requires no score computation, auxiliary modules, or attention modification, and re...
