[2505.17568] JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models

Computer Science > Cryptography and Security arXiv:2505.17568 (cs) [Submitted on 23 May 2025 (v1), last revised 28 Feb 2026 (this version, v3)] Title:JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models Authors:Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Zeren Luo, Jingyi Zheng, Wenhan Dong, Xinlei He, Xuechao Wang, Yingjie Xue, Shengmin Xu, Xinyi Huang View a PDF of the paper titled JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models, by Zifan Peng and 11 other authors View PDF HTML (experimental) Abstract:Large Audio Language Models (LALMs) have made significant progress. While increasingly deployed in real-world applications, LALMs face growing safety risks from jailbreak attacks that bypass safety alignment. However, there remains a lack of an adversarial audio dataset and a unified framework specifically designed to evaluate and compare jailbreak attacks against them. To address this gap, we introduce JALMBench, a comprehensive benchmark that assesses LALM safety against jailbreak attacks, comprising 11,316 text samples and 245,355 audio samples (>1,000 hours). JALMBench supports 12 mainstream LALMs, 8 attack methods (4 text-transferred and 4 audio-originated), and 5 defenses. We conduct in-depth analysis on attack efficiency, topic sensitivity, voice diversity, and model architecture. Additionally, we explore mitigation strategies for the attacks at both the prompt and response levels. Our systematic evaluation reveals th...