[2509.19080] World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation

Computer Science > Robotics arXiv:2509.19080 (cs) [Submitted on 23 Sep 2025 (v1), last revised 19 Mar 2026 (this version, v2)] Title:World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation Authors:Zhennan Jiang, Kai Liu, Yuxin Qin, Shuai Tian, Yupeng Zheng, Mingcai Zhou, Chao Yu, Haoran Li, Dongbin Zhao View a PDF of the paper titled World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation, by Zhennan Jiang and 8 other authors View PDF HTML (experimental) Abstract:Robotic manipulation policies are commonly initialized through imitation learning, but their performance is limited by the scarcity and narrow coverage of expert data. Reinforcement learning can refine polices to alleviate this limitation, yet real-robot training is costly and unsafe, while training in simulators suffers from the sim-to-real gap. Recent advances in generative models have demonstrated remarkable capabilities in real-world simulation, with diffusion models in particular excelling at generation. This raises the question of how diffusion model-based world models can be combined to enhance pre-trained policies in robotic manipulation. In this work, we propose World4RL, a framework that employs diffusion-based world models as high-fidelity simulators to refine pre-trained policies entirely in imagined environments for robotic manipulation. Unlike prior works that primarily employ world models for plan...