[2409.03166] Continual Robot Skill and Task Learning via Dialogue

Computer Science > Robotics arXiv:2409.03166 (cs) [Submitted on 5 Sep 2024 (v1), last revised 28 Mar 2026 (this version, v3)] Title:Continual Robot Skill and Task Learning via Dialogue Authors:Weiwei Gu, Suresh Kondepudi, Anmol Gupta, Lixiao Huang, Nakul Gopalan View a PDF of the paper titled Continual Robot Skill and Task Learning via Dialogue, by Weiwei Gu and 4 other authors View PDF HTML (experimental) Abstract:Interactive robot learning is a challenging problem as the robot is present with human users who expect the robot to learn novel skills to solve novel tasks perpetually with sample efficiency. In this work we present a framework for robots to continually learn tasks and visuo-motor skills and query for novel skills via dialog interactions with human users. Our robot agent maintains a skill library, and uses an existing LLM to perform grounded dialog interactions to query unknown skills from real human users. We developed a novel visual-motor control policy Action Chunking Transformer with Low Rank Adaptation (ACT-LoRA) that can continually learn novel skills using only a few demonstrations which is critical in human-robot interaction scenarios. The paper has twin goals: Firstly to demonstrate better continual learning in simulation; and secondly, to demonstrate the use of our dialog based learning framework in a realistic human-robot interaction use case. Our ACT-LoRA policy consistently outperforms a GMM-LoRA baseline on multiple continual learning simulation b...