[2505.18116] NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning

Computer Science > Machine Learning arXiv:2505.18116 (cs) [Submitted on 23 May 2025 (v1), last revised 1 Mar 2026 (this version, v3)] Title:NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning Authors:Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Lifan Yuan, Yin Cui, Haotian Ye, Tsung-Yi Lin, Ming-Yu Liu, Jun Zhu, Haoxiang Wang View a PDF of the paper titled NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning, by Huayu Chen and 10 other authors View PDF HTML (experimental) Abstract:Reinforcement Learning (RL) has played a central role in the recent surge of LLMs' math abilities by enabling self-improvement through binary verifier signals. In contrast, Supervised Learning (SL) is rarely considered for such verification-driven training, largely due to its heavy reliance on reference answers and inability to reflect on mistakes. In this work, we challenge the prevailing notion that self-improvement is exclusive to RL and propose Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers. In online training, instead of throwing away self-generated negative answers, NFT constructs an implicit negative policy to model them. This implicit policy is parameterized with the same positive LLM we target to optimize on positive data, enabling direct policy optimization on all LLMs' generations. We conduct experiments on 7B and 32B ...