[2603.28295] Evaluating LLMs for Answering Student Questions in Introductory Programming Courses

Computer Science > Artificial Intelligence arXiv:2603.28295 (cs) [Submitted on 30 Mar 2026] Title:Evaluating LLMs for Answering Student Questions in Introductory Programming Courses Authors:Thomas Van Mullem, Bart Mesuere, Peter Dawyndt View a PDF of the paper titled Evaluating LLMs for Answering Student Questions in Introductory Programming Courses, by Thomas Van Mullem and 2 other authors View PDF HTML (experimental) Abstract:The rapid emergence of Large Language Models (LLMs) presents both opportunities and challenges for programming education. While students increasingly use generative AI tools, direct access often hinders the learning process by providing complete solutions rather than pedagogical hints. Concurrently, educators face significant workload and scalability challenges when providing timely, personalized feedback. This study investigates the capabilities of LLMs to safely and effectively assist educators in answering student questions within a CS1 programming course. To achieve this, we established a rigorous, reproducible evaluation process by curating a benchmark dataset of 170 authentic student questions from a learning management system, paired with ground-truth responses authored by subject matter experts. Because traditional text-matching metrics are insufficient for evaluating open-ended educational responses, we developed and validated a custom LLM-as-a-Judge metric optimized for assessing pedagogical accuracy. Our findings demonstrate that models, ...