[2510.03223] Attention-Aligned Reasoning for Large Language Models

Computer Science > Computation and Language arXiv:2510.03223 (cs) [Submitted on 3 Oct 2025 (v1), last revised 27 Mar 2026 (this version, v2)] Title:Attention-Aligned Reasoning for Large Language Models Authors:Hongxiang Zhang, Yuan Tian, Tianyi Zhang View a PDF of the paper titled Attention-Aligned Reasoning for Large Language Models, by Hongxiang Zhang and Yuan Tian and Tianyi Zhang View PDF HTML (experimental) Abstract:Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends. Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI) Cite as: arXiv:2510.03223 [cs.CL]   (or arXiv:2510.03223v2 [cs.CL] for this version)   https://doi.org/10.48550/arXiv.2510....