[2604.06834] On the Step Length Confounding in LLM Reasoning Data Selection

Computer Science > Computation and Language arXiv:2604.06834 (cs) [Submitted on 8 Apr 2026] Title:On the Step Length Confounding in LLM Reasoning Data Selection Authors:Bing Wang, Rui Miao, Chen Shen, Shaotian Yan, Kaiyuan Liu, Ximing Li, Xiaosong Yuan, Sinan Fan, Jun Zhang, Jieping Ye View a PDF of the paper titled On the Step Length Confounding in LLM Reasoning Data Selection, by Bing Wang and 9 other authors View PDF HTML (experimental) Abstract:Large reasoning models have recently demonstrated strong performance on complex tasks that require long chain-of-thought reasoning, through supervised fine-tuning on large-scale and high-quality datasets. To construct such datasets, existing pipelines generate long reasoning data from more capable Large Language Models (LLMs) and apply manually heuristic or naturalness-based selection methods to filter high-quality samples. Despite the proven effectiveness of naturalness-based data selection, which ranks data by the average log probability assigned by LLMs, our analysis shows that, when applied to LLM reasoning datasets, it systematically prefers samples with longer reasoning steps (i.e., more tokens per step) rather than higher-quality ones, a phenomenon we term step length confounding. Through quantitative analysis, we attribute this phenomenon to low-probability first tokens in reasoning steps; longer steps dilute their influence, thereby inflating the average log probabilities. To address this issue, we propose two variant m...