[2506.05334] Search Arena: Analyzing Search-Augmented LLMs

Computer Science > Computation and Language arXiv:2506.05334 (cs) [Submitted on 5 Jun 2025 (v1), last revised 2 Mar 2026 (this version, v2)] Title:Search Arena: Analyzing Search-Augmented LLMs Authors:Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anastasios N. Angelopoulos, Trevor Darrell, Narges Norouzi, Joseph E. Gonzalez View a PDF of the paper titled Search Arena: Analyzing Search-Augmented LLMs, by Mihran Miroyan and 10 other authors View PDF HTML (experimental) Abstract:Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static e...