[2501.16609] CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation

Computer Science > Artificial Intelligence arXiv:2501.16609 (cs) [Submitted on 28 Jan 2025 (v1), last revised 27 Feb 2026 (this version, v4)] Title:CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation Authors:Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, Graham Neubig View a PDF of the paper titled CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation, by Faria Huq and 6 other authors View PDF HTML (experimental) Abstract:While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform...