[2603.24083] Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning

Computer Science > Robotics arXiv:2603.24083 (cs) [Submitted on 25 Mar 2026] Title:Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning Authors:Aditya Narendra, Mukhammadrizo Maribjonov, Dmitry Makarov, Dmitry Yudin, Aleksandr Panov View a PDF of the paper titled Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning, by Aditya Narendra and 4 other authors View PDF HTML (experimental) Abstract:This paper introduces Knowledge Graph based Massively Multi-task Model-based Policy Optimization (KG-M3PO), a framework for multi-task robotic manipulation in partially observable settings that unifies Perception, Knowledge, and Policy. The method augments egocentric vision with an online 3D scene graph that grounds open-vocabulary detections into a metric, relational representation. A dynamic-relation mechanism updates spatial, containment, and affordance edges at every step, and a graph neural encoder is trained end-to-end through the RL objective so that relational features are shaped directly by control performance. Multiple observation modalities (visual, proprioceptive, linguistic, and graph-based) are encoded into a shared latent space, upon which the RL agent operates to drive the control loop. The policy conditions on lightweight graph queries alongside visual and proprioceptive inputs, yielding a compact, semantically informed state for decision making. Experiments on a suite of manipulation tasks with occlusions, distractors, and layout shif...