[2512.20760] Generalization of RLVR Using Causal Reasoning as a Testbed

Computer Science > Machine Learning arXiv:2512.20760 (cs) [Submitted on 23 Dec 2025 (v1), last revised 4 Mar 2026 (this version, v2)] Title:Generalization of RLVR Using Causal Reasoning as a Testbed Authors:Brian Lu, Hongyu Zhao, Shuo Sun, Hao Peng, Rui Ding, Hongyuan Mei View a PDF of the paper titled Generalization of RLVR Using Causal Reasoning as a Testbed, by Brian Lu and 5 other authors View PDF HTML (experimental) Abstract:Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for post-training large language models (LLMs) on complex reasoning tasks. Yet, the conditions under which RLVR yields robust generalization remain underexplored. This paper provides an empirical study of RLVR generalization in the setting of probabilistic inference over causal graphical models. This setting offers two natural axes along which to examine generalization: (i) the level of the probabilistic query -- associational, interventional, or counterfactual -- and (ii) the structural complexity of the query, measured by the size of its relevant subgraph. We construct a dataset of causal graphs and queries spanning these difficulty axes and fine-tune Qwen-2.5-Instruct models using RLVR or supervised fine-tuning (SFT). We vary both the model scale (3B-32B) and the query level included in training. We find that RLVR yields stronger within-level and across-level generalization than SFT, but only for specific combinations of model size and training query level...