[2603.03031] Step-Level Sparse Autoencoder for Reasoning Process Interpretation

Computer Science > Machine Learning arXiv:2603.03031 (cs) [Submitted on 3 Mar 2026] Title:Step-Level Sparse Autoencoder for Reasoning Process Interpretation Authors:Xuan Yang, Jiayu Liu, Yuhang Lai, Hao Xu, Zhenya Huang, Ning Miao View a PDF of the paper titled Step-Level Sparse Autoencoder for Reasoning Process Interpretation, by Xuan Yang and 5 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) have achieved strong complex reasoning capabilities through Chain-of-Thought (CoT) reasoning. However, their reasoning patterns remain too complicated to analyze. While Sparse Autoencoders (SAEs) have emerged as a powerful tool for interpretability, existing approaches predominantly operate at the token level, creating a granularity mismatch when capturing more critical step-level information, such as reasoning direction and semantic transitions. In this work, we propose step-level sparse autoencoder (SSAE), which serves as an analytical tool to disentangle different aspects of LLMs' reasoning steps into sparse features. Specifically, by precisely controlling the sparsity of a step feature conditioned on its context, we form an information bottleneck in step reconstruction, which splits incremental information from background information and disentangles it into several sparsely activated dimensions. Experiments on multiple base models and reasoning tasks show the effectiveness of the extracted features. By linear probing, we can easily predict surfac...