![[2505.11409] Visual Planning: Let's Think Only with Images](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2505.11409] Visual Planning: Let's Think Only with Images
Summary
The paper introduces 'Visual Planning', a new paradigm that utilizes images for reasoning in spatial tasks, enhancing planning capabilities beyond traditional text-based methods.
Why It Matters
As machine learning models increasingly integrate multimodal data, this research highlights the potential of visual reasoning in AI. By proposing a framework that prioritizes visual inputs, it opens new pathways for tasks requiring spatial understanding, which could lead to more intuitive AI applications.
Key Takeaways
- Visual Planning enables reasoning through images, improving performance in spatial tasks.
- The proposed Visual Planning via Reinforcement Learning (VPRL) framework shows significant advancements over text-only reasoning.
- This approach can complement existing language-based models, enhancing their capabilities in 'vision-first' scenarios.
- The study demonstrates practical applications in visual navigation tasks like FrozenLake and Maze.
- Visual Planning could lead to more intuitive AI interactions by mimicking human visual reasoning.
Computer Science > Machine Learning arXiv:2505.11409 (cs) [Submitted on 16 May 2025 (v1), last revised 20 Feb 2026 (this version, v3)] Title:Visual Planning: Let's Think Only with Images Authors:Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, Ivan Vulić View a PDF of the paper titled Visual Planning: Let's Think Only with Images, by Yi Xu and 6 other authors View PDF HTML (experimental) Abstract:Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have substantially enhanced machine reasoning across diverse tasks. However, these models predominantly rely on pure text as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations for these "vision-first" tasks, as a supplementary channel to language-based reasoning. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We introduce a novel reinforcement learning framework, Visual Planning via Reinforcement Learning (VPRL), empowered by GRPO for post-training large vision models, leading to substantial improvem...
