[2506.01153] Weight-Space Linear Recurrent Neural Networks

Computer Science > Machine Learning arXiv:2506.01153 (cs) [Submitted on 1 Jun 2025 (v1), last revised 2 Mar 2026 (this version, v3)] Title:Weight-Space Linear Recurrent Neural Networks Authors:Roussel Desmond Nzoyem, Nawid Keshtmand, Enrique Crespo Fernandez, Idriss Tsayem, Raul Santos-Rodriguez, David A.W. Barton, Tom Deakin View a PDF of the paper titled Weight-Space Linear Recurrent Neural Networks, by Roussel Desmond Nzoyem and 6 other authors View PDF HTML (experimental) Abstract:We introduce WARP (Weight-space Adaptive Recurrent Prediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruc...