![[2504.13359] Cost-of-Pass: An Economic Framework for Evaluating Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2504.13359] Cost-of-Pass: An Economic Framework for Evaluating Language Models
Summary
The paper presents an economic framework for evaluating language models by analyzing the tradeoff between performance and inference costs, introducing the concept of cost-of-pass.
Why It Matters
As AI systems become integral to various industries, understanding their economic efficiency is crucial for organizations. This framework aids in assessing the value generated by language models, guiding deployment decisions and resource allocation in AI development.
Key Takeaways
- Cost-of-pass measures the expected monetary cost of generating correct solutions with language models.
- Lightweight models are cost-effective for basic tasks, while larger models excel in knowledge-intensive applications.
- Tracking cost-of-pass reveals significant advancements in model efficiency, particularly for complex quantitative tasks.
Computer Science > Artificial Intelligence arXiv:2504.13359 (cs) [Submitted on 17 Apr 2025 (v1), last revised 26 Feb 2026 (this version, v2)] Title:Cost-of-Pass: An Economic Framework for Evaluating Language Models Authors:Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou View a PDF of the paper titled Cost-of-Pass: An Economic Framework for Evaluating Language Models, by Mehmet Hamza Erol and 4 other authors View PDF HTML (experimental) Abstract:Widespread adoption of AI systems hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics accounting for both performance and costs. Building on production theory, we develop an economically grounded framework to evaluate language models' productivity by combining accuracy and inference cost. We formalize cost-of-pass: the expected monetary cost of generating a correct solution. We then define the frontier cost-of-pass: the minimum cost-of-pass achievable across available models or the human-expert(s), using the approx. cost of hiring an expert. Our analysis reveals distinct economic insights. First, lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive ones, and reasoning models for complex quantitative problems, despite higher per-token costs. Second, tracking the frontier cost-of-pass over the past year reveals significant progress, particularly for complex quant. tasks where the...
