[2509.22630] StateX: Enhancing RNN Recall via Post-training State Expansion

Computer Science > Computation and Language arXiv:2509.22630 (cs) [Submitted on 26 Sep 2025 (v1), last revised 7 Apr 2026 (this version, v2)] Title:StateX: Enhancing RNN Recall via Post-training State Expansion Authors:Xingyu Shen, Yingfa Chen, Zhen Leng Thai, Xu Han, Zhiyuan Liu, Maosong Sun View a PDF of the paper titled StateX: Enhancing RNN Recall via Post-training State Expansion, by Xingyu Shen and 4 other authors View PDF HTML (experimental) Abstract:Recurrent neural networks (RNNs), such as linear attention and state-space models, have gained popularity due to their constant per-token complexity when processing long contexts. However, these recurrent models struggle with tasks that require accurate recall of contextual information from long contexts, because all contextual information is compressed into a fixed-size recurrent state. Previous studies have shown that recall ability is positively correlated with the recurrent state size, yet directly training RNNs with large recurrent states results in high training costs. In this paper, we introduce StateX, a post-training framework that efficiently expands the states of pre-trained RNNs. For two popular classes of RNNs, linear attention and state-space models, we design post-training architectural modifications in StateX, to scale up the state size with no or negligible increase in model parameters. Experiments on models with up to 1.3B parameters demonstrate that StateX efficiently enhances the recall and in-contex...