[2505.08548] From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation

Computer Science > Robotics arXiv:2505.08548 (cs) [Submitted on 13 May 2025 (v1), last revised 5 Apr 2026 (this version, v3)] Title:From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation Authors:Yifu Yuan, Haiqin Cui, Yibin Chen, Zibin Dong, Fei Ni, Longxin Kou, Jinyi Liu, Pengyi Li, Yan Zheng, Jianye Hao View a PDF of the paper titled From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation, by Yifu Yuan and 9 other authors View PDF HTML (experimental) Abstract:Achieving generalization in robotic manipulation remains a critical challenge, particularly for unseen scenarios and novel tasks. Current Vision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning, providing fine-grained guidance for robotic manipulation. Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD's capabilities in both "seeing" and "doing," achieving outstanding performance across 8 benchmarks for general spatial reasoning and embodied reference abilitie...