![[2602.21297] Robust AI Evaluation through Maximal Lotteries](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21297] Robust AI Evaluation through Maximal Lotteries
Summary
The paper proposes a new method for evaluating AI models using robust lotteries, addressing limitations of traditional pairwise comparison methods in ranking models based on user preferences.
Why It Matters
This research highlights the challenges of evaluating AI systems in a way that reflects diverse user preferences. By introducing robust lotteries, the authors aim to improve the reliability of model evaluations, which is crucial for developing AI systems that cater to a broad range of applications and user needs.
Key Takeaways
- Traditional pairwise comparisons can misrepresent model performance due to preference heterogeneity.
- Maximal lotteries provide a theoretical framework for better aggregating preferences without imposing assumptions.
- Robust lotteries optimize for worst-case performance, enhancing reliability across diverse user preferences.
- The approach aims to create a pluralistic set of top-performing models rather than a single ranking.
- This research contributes to the development of AI systems that better serve varied human needs.
Computer Science > Machine Learning arXiv:2602.21297 (cs) [Submitted on 24 Feb 2026] Title:Robust AI Evaluation through Maximal Lotteries Authors:Hadi Khalaf, Serena L. Wang, Daniel Halpern, Itai Shapira, Flavio du Pin Calmon, Ariel D. Procaccia View a PDF of the paper titled Robust AI Evaluation through Maximal Lotteries, by Hadi Khalaf and 5 other authors View PDF HTML (experimental) Abstract:The standard way to evaluate language models on subjective tasks is through pairwise comparisons: an annotator chooses the "better" of two responses to a prompt. Leaderboards aggregate these comparisons into a single Bradley-Terry (BT) ranking, forcing heterogeneous preferences into a total order and violating basic social-choice desiderata. In contrast, social choice theory provides an alternative approach called maximal lotteries, which aggregates pairwise preferences without imposing any assumptions on their structure. However, we show that maximal lotteries are highly sensitive to preference heterogeneity and can favor models that severely underperform on specific tasks or user subpopulations. We introduce robust lotteries that optimize worst-case performance under plausible shifts in the preference data. On large-scale preference datasets, robust lotteries provide more reliable win rate guarantees across the annotator distribution and recover a stable set of top-performing models. By moving from rankings to pluralistic sets of winners, robust lotteries offer a principled step t...
