[2601.02627] Improved Evidence Extraction and Metrics for Document Inconsistency Detection with LLMs

Computer Science > Computation and Language arXiv:2601.02627 (cs) [Submitted on 6 Jan 2026 (v1), last revised 8 Apr 2026 (this version, v2)] Title:Improved Evidence Extraction and Metrics for Document Inconsistency Detection with LLMs Authors:Nelvin Tan, Yaowen Zhang, James Asikin Cheung, Fusheng Liu, Yu-Ching Shih, Dong Yang View a PDF of the paper titled Improved Evidence Extraction and Metrics for Document Inconsistency Detection with LLMs, by Nelvin Tan and 5 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) are becoming useful in many domains due to their impressive abilities that arise from large training datasets and large model sizes. However, research on LLM-based approaches to document inconsistency detection is relatively limited. We address this gap by investigating evidence extraction capabilties of LLMs for document inconsistency detection. To this end, we introduce new comprehensive evidence-extraction metrics and a redact-and-retry framework with constrained filtering that substantially improves evidence extraction performance over other prompting methods. We support our approach with strong experimental results and release a new semi-synthetic dataset for evaluating evidence extraction. Comments: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI) Cite as: arXiv:2601.02627 [cs.CL]   (or arXiv:2601.02627v2 [cs.CL] for this version)   https://doi.org/10.48550/arXiv.2601.02627 Focus to learn more arXiv-is...