![[2602.16666] Towards a Science of AI Agent Reliability](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.16666] Towards a Science of AI Agent Reliability
Summary
This paper explores the reliability of AI agents, proposing twelve metrics to evaluate their performance across dimensions like consistency and safety, revealing persistent limitations despite advancements in capability.
Why It Matters
As AI agents are increasingly integrated into critical tasks, understanding their reliability is essential. This research addresses gaps in current evaluation methods, providing a framework that can enhance the safety and effectiveness of AI applications in real-world scenarios.
Key Takeaways
- Current evaluations of AI agents often overlook critical reliability factors.
- The paper introduces twelve metrics to assess AI agent reliability comprehensively.
- Findings indicate that improvements in AI capabilities have not significantly enhanced reliability.
Computer Science > Artificial Intelligence arXiv:2602.16666 (cs) [Submitted on 18 Feb 2026] Title:Towards a Science of AI Agent Reliability Authors:Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, Arvind Narayanan View a PDF of the paper titled Towards a Science of AI Agent Reliability, by Stephan Rabanser and 5 other authors View PDF HTML (experimental) Abstract:AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 14 agentic models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail. Subjects: Artificial Intelligence (cs.AI); C...
