![[2602.18986] Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18986] Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight
Summary
This paper presents a Bayesian framework for assessing automation risk in high-automation AI systems, focusing on failure propagation and optimal oversight to mitigate potential harms.
Why It Matters
As organizations increasingly rely on automated AI systems across various sectors, understanding and quantifying the risks associated with automation is crucial. This framework provides a structured approach to evaluate how failures can lead to significant harm, thus offering a pathway for better risk governance and oversight in AI deployments.
Key Takeaways
- Introduces a Bayesian risk decomposition model for AI systems.
- Isolates the conditional probability of failure propagation into harm.
- Provides theoretical foundations for risk governance tools in AI.
- Illustrates the framework with a case study of the Knight Capital incident.
- Emphasizes the need for empirical validation across deployment domains.
Computer Science > Artificial Intelligence arXiv:2602.18986 (cs) [Submitted on 22 Feb 2026] Title:Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight Authors:Vishal Srivastava, Tanmay Sah View a PDF of the paper titled Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight, by Vishal Srivastava and 1 other authors View PDF HTML (experimental) Abstract:Organizations across finance, healthcare, transportation, content moderation, and critical infrastructure are rapidly deploying highly automated AI systems, yet they lack principled methods to quantify how increasing automation amplifies harm when failures occur. We propose a parsimonious Bayesian risk decomposition expressing expected loss as the product of three terms: the probability of system failure, the conditional probability that a failure propagates into harm given the automation level, and the expected severity of harm. This framework isolates a critical quantity -- the conditional probability that failures propagate into harm -- which captures execution and oversight risk rather than model accuracy alone. We develop complete theoretical foundations: formal proofs of the decomposition, a harm propagation equivalence theorem linking the harm propagation probability to observable execution controls, risk elasticity measures, efficient frontier analysis for automation policy, ...