![[2602.18333] On the "Induction Bias" in Sequence Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18333] On the "Induction Bias" in Sequence Models
Summary
This paper examines the 'induction bias' in sequence models, focusing on the limitations of transformer-based models in state tracking compared to recurrent neural networks (RNNs).
Why It Matters
Understanding the induction bias in sequence models is crucial for improving the performance of AI systems, particularly in tasks requiring state tracking. This research highlights the limitations of transformers, which are widely used in NLP, and suggests that RNNs may offer better data efficiency and generalization across sequence lengths.
Key Takeaways
- Transformers require significantly more training data as state-space size and sequence length increase.
- RNNs demonstrate effective weight sharing across sequence lengths, enhancing their learning efficiency.
- Transformers struggle with state tracking even in in-distribution scenarios, indicating a fundamental challenge.
- The study emphasizes the need for improved understanding and techniques to address the limitations of transformers.
- Findings suggest that RNNs may be more suitable for certain applications requiring robust state tracking.
Computer Science > Machine Learning arXiv:2602.18333 (cs) [Submitted on 20 Feb 2026] Title:On the "Induction Bias" in Sequence Models Authors:M.Reza Ebrahimi, Michaël Defferrard, Sunny Panchal, Roland Memisevic View a PDF of the paper titled On the "Induction Bias" in Sequence Models, by M.Reza Ebrahimi and 3 other authors View PDF Abstract:Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length t...