[2604.06723] Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision

Computer Science > Software Engineering arXiv:2604.06723 (cs) [Submitted on 8 Apr 2026] Title:Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision Authors:Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude View a PDF of the paper titled Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision, by Hong Yi Lin and 4 other authors View PDF HTML (experimental) Abstract:In today's AI-assisted software engineering landscape, developers increasingly depend on LLMs that are highly capable, yet inherently imperfect. The tendency of these models to produce incorrect outputs can reduce developer productivity. To this end, a canonical mitigation method is to provide calibrated confidence scores that faithfully reflect their likelihood of correctness at the instance-level. Such information allows users to make immediate decisions regarding output acceptance, abstain error-prone outputs, and better align their expectations with the model's capabilities. Since post-trained LLMs do not inherently produce well-calibrated confidence scores, researchers have developed post-hoc calibration methods, with global Platt-scaling of sequence-level confidence scores proving effective in many generative software engineering tasks but remaining unreliable or unexplored for automated code revision (ACR) tasks such as program repair, vulnerability repair, and code refinement. We hypothesise that the coarse-grained n...