![[2512.16902] In-Context Algebra](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2512.16902] In-Context Algebra
Summary
The paper 'In-Context Algebra' explores how transformers can solve arithmetic problems using variable tokens whose meanings are context-dependent, achieving high accuracy and demonstrating symbolic reasoning capabilities.
Why It Matters
This research is significant as it reveals the potential of transformer models to develop advanced reasoning strategies in dynamic contexts, which could enhance their application in various fields such as natural language processing and artificial intelligence.
Key Takeaways
- Transformers can accurately solve arithmetic problems with context-dependent variables.
- The study identifies three key reasoning mechanisms learned by models: commutative copying, identity element recognition, and closure-based cancellation.
- The findings suggest that task structure influences the reasoning strategies developed by transformer models.
Computer Science > Computation and Language arXiv:2512.16902 (cs) [Submitted on 18 Dec 2025 (v1), last revised 25 Feb 2026 (this version, v2)] Title:In-Context Algebra Authors:Eric Todd, Jannik Brinkmann, Rohit Gandikota, David Bau View a PDF of the paper titled In-Context Algebra, by Eric Todd and 3 other authors View PDF HTML (experimental) Abstract:We investigate the mechanisms that arise when transformers are trained to solve arithmetic on sequences where tokens are variables whose meaning is determined only through their interactions in-context. While prior work has studied transformers in settings where the answer relies on fixed parametric or geometric information encoded in token embeddings, we devise a new in-context reasoning task where the assignment of tokens to specific algebraic elements varies from one sequence to another. Despite this challenging setup, transformers achieve near-perfect accuracy on the task and even generalize to unseen groups. We develop targeted data distributions to create causal tests of a set of hypothesized mechanisms, and we isolate three mechanisms models consistently learn: commutative copying where a dedicated head copies answers, identity element recognition that distinguishes identity-containing facts, and closure-based cancellation that tracks group membership to constrain valid answers. Our findings show that the kinds of reasoning strategies learned by transformers are dependent on the task structure and that models can devel...
