[2603.21621] Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective

Computer Science > Machine Learning arXiv:2603.21621 (cs) [Submitted on 23 Mar 2026] Title:Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective Authors:Yuehu Gong, Zeyuan Wang, Yulin Chen, Yanwei Fu View a PDF of the paper titled Proximal Policy Optimization in Path Space: A Schr\"odinger Bridge Perspective, by Yuehu Gong and 3 other authors View PDF HTML (experimental) Abstract:On-policy reinforcement learning with generative policies is promising but remains underexplored. A central challenge is that proximal policy optimization (PPO) is traditionally formulated in terms of action-space probability ratios, whereas diffusion- and flow-based policies are more naturally represented as trajectory-level generative processes. In this work, we propose GSB-PPO, a path-space formulation of generative PPO inspired by the Generalized Schrödinger Bridge (GSB). Our framework lifts PPO-style proximal updates from terminal actions to full generation trajectories, yielding a unified view of on-policy optimization for generative policies. Within this framework, we develop two concrete objectives: a clipping-based objective, GSB-PPO-Clip, and a penalty-based objective, GSB-PPO-Penalty. Experimental results show that while both objectives are compatible with on-policy training, the penalty formulation consistently delivers better stability and performance than the clipping counterpart. Overall, our results highlight path-space proximal regularization as an effective...