[2605.07986] Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios

Computer Science > Human-Computer Interaction arXiv:2605.07986 (cs) [Submitted on 8 May 2026] Title:Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios Authors:Yee-Yin Choong, Kristen Greene, Alice Qian, Meryem Marasli, Ziqi Yang, Sophia Chen, Laura Dabbish, Anand Rao, Hong Shen View a PDF of the paper titled Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios, by Yee-Yin Choong and 8 other authors View PDF HTML (experimental) Abstract:AI measurement science has a wide variety of methodologies and measurements for comparing AI systems, resulting in what often appear to be "apples-to-oranges" comparisons across AI evaluations. To move toward "apples-to-apples" comparisons in real-world AI evaluations, this work advocates for methodological transparency in evaluation scenarios, operational grounding, and human-centered design (HCD) principles. We propose a repeatable process for transforming high-level use cases to detailed scenarios by eliciting use cases from subject matter experts (SMEs) via a structured AI Use Case Worksheet with six key elements: use case, sector, user (direct and indirect), intended outcomes, expected impacts (positive and negative), and KPIs and metrics. We demonstrate utility of the worksheet and process in the U.S. financial services sector. This paper reports on example high-level AI use cases identified by financial services sector SMEs: cyber defense enablement,...