[2603.04409] Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework

Computer Science > Computation and Language arXiv:2603.04409 (cs) [Submitted on 3 Feb 2026] Title:Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework Authors:Nora Petrova, Andrew Gordon, Enzo Blindow View a PDF of the paper titled Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework, by Nora Petrova and 1 other authors View PDF HTML (experimental) Abstract:The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson (BTD) model, with post-stratification to census data, and our analysis reveals three key insights. \textbf{(1)} We establish a clear performance hierarchy where \texttt{google/gemini-2.5-pro} ranks first overall, with a 95.6\% posterior probability of being the top-ranked model. \textbf{(2)} We uncover significant preference heterogeneity, with user age em...