![[2509.25424] Polychromic Objectives for Reinforcement Learning](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2509.25424] Polychromic Objectives for Reinforcement Learning
Summary
The paper introduces polychromic objectives for reinforcement learning, enhancing policy diversity and exploration in pretrained models, leading to improved performance across various tasks.
Why It Matters
This research addresses a critical challenge in reinforcement learning where models often converge to suboptimal outputs, limiting their effectiveness. By promoting diverse strategies, the proposed method enhances the adaptability and robustness of AI systems, which is crucial for real-world applications.
Key Takeaways
- Polychromic objectives enhance exploration in reinforcement learning.
- The method improves success rates across diverse tasks and environments.
- Adaptations to proximal policy optimization (PPO) are proposed for better performance.
- The approach maintains a diverse repertoire of strategies, crucial for complex tasks.
- Experimental results demonstrate significant improvements in generalization and coverage.
Computer Science > Machine Learning arXiv:2509.25424 (cs) [Submitted on 29 Sep 2025 (v1), last revised 23 Feb 2026 (this version, v2)] Title:Polychromic Objectives for Reinforcement Learning Authors:Jubayer Ibn Hamid, Ifdita Hasan Orney, Ellen Xu, Chelsea Finn, Dorsa Sadigh View a PDF of the paper titled Polychromic Objectives for Reinforcement Learning, by Jubayer Ibn Hamid and 4 other authors View PDF HTML (experimental) Abstract:Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks. These pretrained policies, trained on large datasets, produce generations with a broad range of promising but unrefined behaviors. Often, a critical failure mode of RLFT arises when policies lose this diversity and collapse into a handful of easily exploitable outputs. This convergence hinders exploration, which is essential for expanding the capabilities of the pretrained policy and for amplifying the benefits of test-time compute scaling. To address this, we introduce an objective for policy gradient methods that explicitly enforces the exploration and refinement of diverse generations, which we call a polychromic objective. We then show how proximal policy optimization (PPO) can be adapted to optimize this objective. Our method (1) employs vine sampling to collect on-policy rollouts and (2) modifies the advantage function to reflect the advantage under our new objective. Experiments on BabyAI, Minigrid, and Algorithmic Crea...
