[2603.04370] $τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

Computer Science > Artificial Intelligence arXiv:2603.04370 (cs) [Submitted on 4 Mar 2026] Title:$τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge Authors:Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik Narasimhan, Victor Barres View a PDF of the paper titled $\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge, by Quan Shi and 4 other authors View PDF HTML (experimental) Abstract:Conversational agents are increasingly deployed in knowledge-intensive settings, where correct behavior depends on retrieving and applying domain-specific knowledge from large, proprietary, and unstructured corpora during live interactions with users. Yet most existing benchmarks evaluate retrieval or tool use independently of each other, creating a gap in realistic, fully agentic evaluation over unstructured data in long-horizon interactions. We introduce $\tau$-Knowledge, an extension of $\tau$-Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs to produce verifiable, policy-compliant state changes. Our new domain, $\tau$-Banking, models realistic fintech customer support workflows in which agents must navigate roughly 700 interconnected knowledge documents while executing tool-mediated account updates. Across embedding-based retrieval and terminal-based search, even frontier models with high reasoning budgets achieve only $\sim$25.5% pass^1, with reliabilit...