[2511.08225] Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback

Computer Science > Computation and Language arXiv:2511.08225 (cs) [Submitted on 11 Nov 2025 (v1), last revised 1 Apr 2026 (this version, v2)] Title:Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback Authors:Yishan Du, Conrad Borchers, Mutlu Cukurova View a PDF of the paper titled Benchmarking Educational LLMs with Analytics: A Case Study on Gender Bias in Feedback, by Yishan Du and 2 other authors View PDF HTML (experimental) Abstract:As teachers increasingly turn to GenAI in their educational practice, we need robust methods to benchmark large language models (LLMs) for pedagogical purposes. This article presents an embedding-based benchmarking framework to detect bias in LLMs in the context of formative feedback. Using 600 authentic student essays from the AES 2.0 corpus, we constructed controlled counterfactuals along two dimensions: (i) implicit cues via lexicon-based swaps of gendered terms within essays, and (ii) explicit cues via gendered author background in the prompt. We investigated six representative LLMs (i.e. GPT-5 mini, GPT-4o mini, DeepSeek-R1, DeepSeek-R1-Qwen, Gemini 2.5 Pro, Llama-3-8B). We first quantified the response divergence with cosine and Euclidean distances over sentence embeddings, then assessed significance via permutation tests, and finally, visualised structure using dimensionality reduction. In all models, implicit manipulations reliably induced larger semantic shifts for male-female counterfactuals than f...