[2507.05890] Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators

Computer Science > Computation and Language arXiv:2507.05890 (cs) [Submitted on 8 Jul 2025 (v1), last revised 3 Mar 2026 (this version, v3)] Title:Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators Authors:Sungjib Lim, Woojung Song, Eun-Ju Lee, Yohan Jo View a PDF of the paper titled Psychometric Item Validation Using Virtual Respondents with Trait-Response Mediators, by Sungjib Lim and 3 other authors View PDF HTML (experimental) Abstract:As psychometric surveys are increasingly used to assess the traits of large language models (LLMs), the need for scalable survey item generation suited for LLMs has also grown. A critical challenge here is ensuring the construct validity of generated items, i.e., whether they truly measure the intended trait. Traditionally, this requires costly, large-scale human data collection. To make it efficient, we present a framework for virtual respondent simulation using LLMs. Our central idea is to account for mediators: factors through which the same trait can give rise to varying responses to a survey item. By simulating respondents with diverse mediators, we identify survey items that robustly measure intended traits. Experiments on three psychological trait theories (Big5, Schwartz, VIA) show that our mediator generation methods and simulation framework effectively identify high-validity items. LLMs demonstrate the ability to generate plausible mediators from trait definitions and to simulate respondent beh...