[2603.03806] Separators in Enhancing Autoregressive Pretraining for Vision Mamba

Computer Science > Computer Vision and Pattern Recognition arXiv:2603.03806 (cs) [Submitted on 4 Mar 2026] Title:Separators in Enhancing Autoregressive Pretraining for Vision Mamba Authors:Hanpeng Liu, Zidan Wang, Shuoxi Zhang, Kaiyuan Gao, Kun He View a PDF of the paper titled Separators in Enhancing Autoregressive Pretraining for Vision Mamba, by Hanpeng Liu and 4 other authors View PDF HTML (experimental) Abstract:The state space model Mamba has recently emerged as a promising paradigm in computer vision, attracting significant attention due to its efficient processing of long sequence tasks. Mamba's inherent causal mechanism renders it particularly suitable for autoregressive pretraining. However, current autoregressive pretraining methods are constrained to short sequence tasks, failing to fully exploit Mamba's prowess in handling extended sequences. To address this limitation, we introduce an innovative autoregressive pretraining method for Vision Mamba that substantially extends the input sequence length. We introduce new \textbf{S}epara\textbf{T}ors for \textbf{A}uto\textbf{R}egressive pretraining to demarcate and differentiate between different images, known as \textbf{STAR}. Specifically, we insert identical separators before each image to demarcate its inception. This strategy enables us to quadruple the input sequence length of Vision Mamba while preserving the original dimensions of the dataset images. Employing this long sequence pretraining technique, our ST...