[2506.11680] Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information

Computer Science > Computers and Society arXiv:2506.11680 (cs) [Submitted on 13 Jun 2025] Title:Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information Authors:Xiao Zhan, Juan Carlos Carrillo, William Seymour, Jose Such View a PDF of the paper titled Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information, by Xiao Zhan and 3 other authors View PDF HTML (experimental) Abstract:LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM application remains unexplored: an LLM-based CAI that is deliberately designed to extract personal information from users. In this paper, we report on the malicious LLM-based CAIs that we created based on system prompts that used different strategies to encourage disclosures of personal information from users. We systematically investigate CAIs' ability to extract personal information from users during conversations by conducting a randomized-controlled trial with 502 participants. We assess the effectiveness of different malicious and benign CAIs to extract personal information from participants, and we analyze participants' perceptions after their interactions with the...