[2605.07830] CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Computer Science > Cryptography and Security arXiv:2605.07830 (cs) [Submitted on 8 May 2026] Title:CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios Authors:Taein Lim, Seongyong Ju, Munhyeok Kim, Hyunjun Kim, Hoki Kim View a PDF of the paper titled CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios, by Taein Lim and 4 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, each agent exhibits an attack-selection bias, disproportionately concentrating its efforts on a narrow subset of attack families regardless of prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evaluates five agents on three targets and four prompt conditions with ten attack families. We identify explicit bias across agents, with different dominant attack families and varying entropy levels in their attack-family allocation distributions. Such bias is better characterized as a trait of the agents, rather than a factor associated with the attack success rate. Furthermore, our experiments reveal a bias momentum effect, where agents resist explicit steering toward attack families that conflict with their bias. This forced distribution shift does not yield measurable improvem...