[2604.03257] Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation

Computer Science > Computation and Language arXiv:2604.03257 (cs) [Submitted on 11 Mar 2026] Title:Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation Authors:Minghe Shen, Ananth Balashankar, Adam Fisch, David Madras, Miguel Rodrigues View a PDF of the paper titled Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation, by Minghe Shen and 4 other authors View PDF HTML (experimental) Abstract:The ability to rigorously estimate the failure rates of large language models (LLMs) is a prerequisite for their safe deployment. Currently, however, practitioners often face a tradeoff between expensive human gold standards and potentially severely-biased automatic annotation schemes such as "LLM-as-a-Judge" labeling. In this paper, we propose a new, practical, and efficient approach to LLM failure rate estimation based on constrained maximum-likelihood estimation (MLE). Our method integrates three distinct signal sources: (i) a small, high-quality human-labeled calibration set, (ii) a large corpus of LLM-judge annotations, and, most importantly, (iii) additional side information via domain-specific constraints derived from known bounds on judge performance statistics. We validate our approach through a comprehensive empirical study, benchmarking it against state-of-the-art baselines like Prediction-Powered Inference (PPI). Across diverse experimental regimes -- spanning varying judge accuracies, calibration set sizes, and LLM...