[2603.03321] DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following

Computer Science > Computation and Language arXiv:2603.03321 (cs) [Submitted on 10 Feb 2026] Title:DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following Authors:Nardine Basta, Dali Kaafar View a PDF of the paper titled DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following, by Nardine Basta and 1 other authors View PDF HTML (experimental) Abstract:Evaluating instruction following in Large Language Models requires decomposing instructions into verifiable requirements and assessing satisfaction--tasks currently dependent on manual annotation and uniform criteria that do not align with human judgment patterns. We present DIALEVAL, a type-theoretic framework using dual LLM agents to automate instruction decomposition into typed predicates and implement type-specific satisfaction semantics. The framework enforces formal atomicity and independence constraints during automated extraction, then applies differentiated evaluation criteria--semantic equivalence for content predicates, exact precision for numerical predicates--mirroring empirically observed human assessment patterns. Extended to multi-turn dialogues through history-aware satisfaction functions, DIALEVAL enables evaluation in conversational contexts where single-turn methods fail. Validation demonstrates 90.38% accuracy (26.45% error reduction over baselines) and substantially stronger correlation with human judgment for complex instructions. Comments: Subjects: Computation and ...