![[2502.13069] Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2502.13069] Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering
Summary
The paper introduces Ambig-SWE, a framework for evaluating AI agents' ability to handle underspecified instructions in software engineering, highlighting the importance of interactive clarification.
Why It Matters
As AI agents increasingly automate tasks, their ability to accurately interpret vague instructions is crucial. This research addresses the risks associated with underspecificity, emphasizing the need for interactive models that can ask clarifying questions, thereby improving performance and safety in software engineering tasks.
Key Takeaways
- Ambig-SWE evaluates AI agents' performance with underspecified instructions.
- Interactive models can significantly improve task outcomes by asking clarifying questions.
- Current AI models struggle with distinguishing well-specified from underspecified instructions.
- Effective interaction can enhance performance by up to 74% in ambiguous scenarios.
- The study identifies critical gaps in how AI handles missing information in software engineering.
Computer Science > Artificial Intelligence arXiv:2502.13069 (cs) [Submitted on 18 Feb 2025 (v1), last revised 21 Feb 2026 (this version, v3)] Title:Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering Authors:Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, Graham Neubig View a PDF of the paper titled Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering, by Sanidhya Vijayvargiya and 4 other authors View PDF HTML (experimental) Abstract:AI agents are increasingly being deployed to automate tasks, often based on underspecified user instructions. Making unwarranted assumptions to compensate for the missing information and failing to ask clarifying questions can lead to suboptimal outcomes, safety risks due to tool misuse, and wasted computational resources. In this work, we study the ability of LLM agents to handle underspecified instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance across three key steps: (a) detecting underspecificity, (b) asking targeted clarification questions, and (c) leveraging the interaction to improve performance in underspecified scenarios. We introduce Ambig-SWE, an underspecified variant of SWE-Bench Verified, specifically designed to evaluate agent behavior under ambiguity and interaction. Our findings reveal that models struggle to distinguish between well-specified and underspecified instructions. Howe...