[2410.19450] Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration

Computer Science > Artificial Intelligence arXiv:2410.19450 (cs) [Submitted on 25 Oct 2024 (v1), last revised 27 Feb 2026 (this version, v2)] Title:Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration Authors:Hai Zhong, Xun Wang, Zhuoran Li, Longbo Huang View a PDF of the paper titled Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration, by Hai Zhong and 3 other authors View PDF HTML (experimental) Abstract:Offline-to-Online Reinforcement Learning has emerged as a powerful paradigm, leveraging offline data for initialization and online fine-tuning to enhance both sample efficiency and performance. However, most existing research has focused on single-agent settings, with limited exploration of the multi-agent extension, i.e., Offline-to-Online Multi-Agent Reinforcement Learning (O2O MARL). In O2O MARL, two critical challenges become more prominent as the number of agents increases: (i) the risk of unlearning pre-trained Q-values due to distributional shifts during the transition from offline-to-online phases, and (ii) the difficulty of efficient exploration in the large joint state-action space. To tackle these challenges, we propose a novel O2O MARL framework called Offline Value Function Memory with Sequential Exploration (OVMSE). First, we introduce the Offline Value Function Memory (OVM) mechanism to compute target Q-values, preserving knowledge g...