[2603.00590] Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs

Computer Science > Artificial Intelligence arXiv:2603.00590 (cs) [Submitted on 28 Feb 2026] Title:Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs Authors:Yiran Zhao, Lu Zhou, Xiaogang Xu, Zhe Liu, Jiafei Wu, Liming Fang View a PDF of the paper titled Fair in Mind, Fair in Action? A Synchronous Benchmark for Understanding and Generation in UMLLMs, by Yiran Zhao and 5 other authors View PDF HTML (experimental) Abstract:As artificial intelligence (AI) is increasingly deployed across domains, ensuring fairness has become a core challenge. However, the field faces a "Tower of Babel'' dilemma: fairness metrics abound, yet their underlying philosophical assumptions often conflict, hindering unified paradigms-particularly in unified Multimodal Large Language Models (UMLLMs), where biases propagate systemically across tasks. To address this, we introduce the IRIS Benchmark, to our knowledge the first benchmark designed to synchronously evaluate the fairness of both understanding and generation tasks in UMLLMs. Enabled by our demographic classifier, ARES, and four supporting large-scale datasets, the benchmark is designed to normalize and aggregate arbitrary metrics into a high-dimensional "fairness space'', integrating 60 granular metrics across three dimensions-Ideal Fairness, Real-world Fidelity, and Bias Inertia & Steerability (IRIS). Through this benchmark, our evaluation of leading UMLLMs uncovers systemic phenomena such as the ...