[2604.07304] Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems

Computer Science > Software Engineering arXiv:2604.07304 (cs) [Submitted on 8 Apr 2026] Title:Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems Authors:Eduard Frankford, Erik Cikalleshi, Ruth Breu View a PDF of the paper titled Chatbot-Based Assessment of Code Understanding in Automated Programming Assessment Systems, by Eduard Frankford and 1 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) challenge conventional automated programming assessment because students can now produce functionally correct code without demonstrating corresponding understanding. This paper makes two contributions. First, it reports a saturation-based scoping review of conversational assessment approaches in programming education. The review identifies three dominant architectural families: rule-based or template-driven systems, LLM-based systems, and hybrid systems. Across the literature, conversational agents appear promising for scalable feedback and deeper probing of code understanding, but important limitations remain around hallucinations, over-reliance, privacy, integrity, and deployment constraints. Second, the paper synthesizes these findings into a Hybrid Socratic Framework for integrating conversational verification into Automated Programming Assessment Systems (APASs). The framework combines deterministic code analysis with a dual-agent conversational layer, knowledge tracking, scaffolded questioning, and guardrail...