![[2602.21127] "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21127] "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems
Summary
This study investigates human vulnerability to deception by large language model (LLM) agents, revealing significant trust issues in high-stakes domains like healthcare and software development.
Why It Matters
As LLMs become integral in critical areas, understanding human susceptibility to deception is essential for developing robust defenses. This research highlights cognitive vulnerabilities and offers insights for improving user awareness and safety in agentic systems.
Key Takeaways
- Only 8.6% of participants recognized agent-mediated deception (AMD) attacks.
- Domain experts showed increased susceptibility to deception in specific scenarios.
- Cognitive failure modes were identified, indicating gaps in user risk awareness.
- Effective warnings should disrupt workflows to enhance user caution.
- Experiential learning significantly increases user awareness of risks.
Computer Science > Human-Computer Interaction arXiv:2602.21127 (cs) [Submitted on 24 Feb 2026] Title:"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems Authors:Xinfeng Li, Shenyu Dai, Kelong Zheng, Yue Xiao, Gelei Deng, Wei Dong, Xiaofeng Wang View a PDF of the paper titled "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems, by Xinfeng Li and 6 other authors View PDF HTML (experimental) Abstract:Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare. However, this deepening trust introduces a novel attack surface: Agent-Mediated Deception (AMD), where compromised agents are weaponized against their human users. While extensive research focuses on agent-centric threats, human susceptibility to deception by a compromised agent remains unexplored. We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD. This is based on HAT-Lab (Human-Agent Trust Laboratory), a high-fidelity research platform we develop, featuring nine carefully crafted scenarios spanning everyday and professional domains (e.g., healthcare, software development, human resources). Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives. Specifically, only 8.6% of participants perceive AMD attacks, while domain experts show increased susceptibi...