[2412.15176] Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure

Computer Science > Machine Learning arXiv:2412.15176 (cs) [Submitted on 19 Dec 2024 (v1), last revised 27 Feb 2026 (this version, v2)] Title:Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure Authors:Lukas Aichberger, Kajetan Schweighofer, Sepp Hochreiter View a PDF of the paper titled Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure, by Lukas Aichberger and 2 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generated text. To this end, reliable uncertainty estimation is essential. Leading uncertainty estimation methods generate and analyze multiple output sequences, which is computationally expensive and impractical at scale. In this work, we inspect the theoretical foundations of these methods and explore new directions to enhance computational efficiency. Building on the framework of proper scoring rules, we find that the negative log-likelihood of the most likely output sequence constitutes a theoretically principled uncertainty measure. To approximate this alternative measure, we propose G-NLL, obtained using a single output sequence from greedy decoding. This approach streamlines uncertainty estimation while preserving theoretical rigor. Empirical results demonstrate that G-NLL achieves state-of-the-art performance across various scenarios. Our work lays the theoreti...