![[2506.23875] Chain of Thought in Order: Discovering Learning-Friendly Orders for Arithmetic](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2506.23875] Chain of Thought in Order: Discovering Learning-Friendly Orders for Arithmetic
Summary
This article explores the importance of ordering in the chain of thought for Transformers in arithmetic tasks, proposing a method to identify learning-friendly sequences that enhance performance.
Why It Matters
Understanding the optimal ordering of reasoning steps in machine learning models can significantly improve their performance on complex tasks. This research addresses a gap in the study of Transformers, potentially leading to advancements in educational applications and AI systems that require arithmetic reasoning.
Key Takeaways
- The order of reasoning steps in Transformers can affect task difficulty.
- A novel two-stage hierarchical approach is proposed for reordering input tokens.
- Experiments demonstrate the identification of effective learning-friendly sequences from a vast search space.
Computer Science > Machine Learning arXiv:2506.23875 (cs) [Submitted on 30 Jun 2025 (v1), last revised 14 Feb 2026 (this version, v2)] Title:Chain of Thought in Order: Discovering Learning-Friendly Orders for Arithmetic Authors:Yuta Sato, Kazuhiko Kawamoto, Hiroshi Kera View a PDF of the paper titled Chain of Thought in Order: Discovering Learning-Friendly Orders for Arithmetic, by Yuta Sato and 2 other authors View PDF HTML (experimental) Abstract:The chain of thought, i.e., step-by-step reasoning, is one of the fundamental mechanisms of Transformers. While the design of intermediate reasoning steps has been extensively studied and shown to critically influence performance on mathematical, multi-step reasoning tasks, the ordering of these steps has received little attention, despite its significant effect on the difficulty of reasoning. This study addresses a novel task of unraveling the chain of thought -- reordering decoder input tokens into a learning-friendly sequence for Transformers, for learning arithmetic tasks. The proposed pipeline first trains a Transformer on a mixture of target sequences arranged in different orders and then identifies benign orders as those with fast loss drops in the early stage. As the search space grows factorially in sequence length, we propose a two-stage hierarchical approach for inter- and intra-block reordering. Experiments on seven order-sensitive arithmetic tasks show that our method identifies a learning-friendly order out of a fe...
