![[2602.18182] Capabilities Ain't All You Need: Measuring Propensities in AI](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18182] Capabilities Ain't All You Need: Measuring Propensities in AI
Summary
The paper introduces a framework for measuring AI propensities, emphasizing the importance of behavioral tendencies alongside capabilities in AI evaluation.
Why It Matters
As AI systems become more integrated into various applications, understanding their propensities is crucial for predicting performance and ensuring safety. This research provides a new method to evaluate AI behavior, potentially leading to more reliable AI systems.
Key Takeaways
- Traditional AI evaluation focuses mainly on capabilities, neglecting behavioral propensities.
- The new framework uses a bilogistic model to measure AI propensities effectively.
- Combining propensity measurements with capability evaluations enhances predictive power.
- The framework has been tested on various LLM models, demonstrating its applicability.
- Understanding propensities can lead to better safety outcomes in AI applications.
Computer Science > Machine Learning arXiv:2602.18182 (cs) [Submitted on 20 Feb 2026] Title:Capabilities Ain't All You Need: Measuring Propensities in AI Authors:Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi, Hugo Save, Siddhesh Milind Pawar, Behzad Mehrbakhsh, Pablo Antonio Moreno Casares, Ben Slater, Paolo Bova, Peter Romero, Zachary R. Tyler, Jonathan Prunty, Luning Sun, Jose Hernandez-Orallo View a PDF of the paper titled Capabilities Ain't All You Need: Measuring Propensities in AI, by Daniel Romero-Alvarado and 13 other authors View PDF HTML (experimental) Abstract:AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the tendencies of models to exhibit particular behaviours - play a central role in determining both performance and safety outcomes. However, traditional IRT describes a model's success on a task as a monotonic function of model capabilities and task demands, an approach unsuited to propensities, where both excess and deficiency can be problematic. Here, we introduce the first formal framework for measuring AI propensities by using a bilogistic formulation for model success, which attributes high success probability when the model's propensity is within an "ideal band". Further, we estimate the limits of the ideal band using LLMs equipped with newly developed task-agnostic rubrics. Applying our framework to six fa...
