![[2601.09495] Parallelizable memory recurrent units](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2601.09495] Parallelizable memory recurrent units
Summary
The paper introduces memory recurrent units (MRUs), a new family of RNNs that combine persistent memory with parallelizable computations, addressing limitations of current models in handling long-term dependencies efficiently.
Why It Matters
As machine learning models evolve, the ability to efficiently process sequences while maintaining memory is crucial. This research presents a significant advancement in recurrent neural networks, potentially enhancing applications in natural language processing and time-series analysis.
Key Takeaways
- MRUs leverage multistability for persistent memory, improving on traditional RNNs.
- The bistable memory recurrent unit (BMRU) is introduced as a proof-of-concept.
- MRUs can be combined with state-space models for enhanced performance.
- This approach addresses the inefficiencies of Transformers in sequence generation.
- The research highlights the importance of parallel processing in modern neural networks.
Computer Science > Machine Learning arXiv:2601.09495 (cs) [Submitted on 14 Jan 2026 (v1), last revised 16 Feb 2026 (this version, v2)] Title:Parallelizable memory recurrent units Authors:Florent De Geeter, Gaspard Lambrechts, Damien Ernst, Guillaume Drion View a PDF of the paper titled Parallelizable memory recurrent units, by Florent De Geeter and 2 other authors View PDF HTML (experimental) Abstract:With the emergence of massively parallel processing units, parallelization has become a desirable property for new sequence models. The ability to parallelize the processing of sequences with respect to the sequence length during training is one of the main factors behind the uprising of the Transformer architecture. However, Transformers lack efficiency at sequence generation, as they need to reprocess all past timesteps at every generation step. Recently, state-space models (SSMs) emerged as a more efficient alternative. These new kinds of recurrent neural networks (RNNs) keep the efficient update of the RNNs while gaining parallelization by getting rid of nonlinear dynamics (or recurrence). SSMs can reach state-of-the art performance through the efficient training of potentially very large networks, but still suffer from limited representation capabilities. In particular, SSMs cannot exhibit persistent memory, or the capacity of retaining information for an infinite duration, because of their monostability. In this paper, we introduce a new family of RNNs, the memory recur...