[2509.24186] Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks

Computer Science > Computation and Language arXiv:2509.24186 (cs) [Submitted on 29 Sep 2025 (v1), last revised 6 Apr 2026 (this version, v2)] Title:Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks Authors:Zhimeng Luo, Lixin Wu, Adam Frisch, Daqing He View a PDF of the paper titled Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks, by Zhimeng Luo and 3 other authors View PDF HTML (experimental) Abstract:Accuracy-based evaluation of Large Language Models (LLMs) measures benchmark-specific performance rather than underlying medical competency: it treats all questions as equally informative, conflates model ability with item characteristics, and thereby produces rankings that vary with benchmark choice. To address this, we introduce MedIRT, a psychometric evaluation framework grounded in Item Response Theory (IRT) that (1) jointly models latent competency and item-level difficulty and discrimination, and (2) includes benchmark integrity validation to ensure items within each topic measure a single, coherent underlying ability. We prospectively evaluate 71 diverse LLMs on a USMLE-aligned benchmark across 11 medical topics. As internal validation, MedIRT correctly predicts held-out LLM responses on unseen questions with 83.3% accuracy. As external validation, IRT-based rankings outperform accuracy-based rankings across 6 independent external medical benchmarks -- including expert preferences, holistic cli...