![[2510.22620] Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.22620] Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
Summary
This article evaluates the security of large language models (LLMs) used in AI agents, introducing a framework for identifying vulnerabilities and proposing a benchmark for assessment.
Why It Matters
As AI agents become more prevalent, understanding their security is crucial. This research addresses gaps in existing frameworks, providing a systematic approach to evaluate LLM vulnerabilities and their implications for AI agent security. The findings can guide developers in enhancing AI safety.
Key Takeaways
- Introduces threat snapshots to identify LLM vulnerabilities in AI agents.
- Develops the $b^3$ benchmark based on 194,331 adversarial attacks.
- Finds that enhanced reasoning capabilities improve security, while model size does not correlate with it.
- Releases benchmark and evaluation code for broader adoption by LLM providers.
- Encourages prioritization of backbone security improvements by model developers.
Computer Science > Cryptography and Security arXiv:2510.22620 (cs) [Submitted on 26 Oct 2025 (v1), last revised 24 Feb 2026 (this version, v2)] Title:Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents Authors:Julia Bazinska, Max Mathys, Francesco Casucci, Mateo Rojas-Carulla, Xander Davies, Alexandra Souly, Niklas Pfister View a PDF of the paper titled Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents, by Julia Bazinska and 6 other authors View PDF HTML (experimental) Abstract:AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security. The non-deterministic sequential nature of AI agents complicates security modeling, while the integration of traditional software with AI components entangles novel LLM vulnerabilities with conventional security risks. Existing frameworks only partially address these challenges as they either capture specific vulnerabilities only or require modeling of complete agents. To address these limitations, we introduce threat snapshots: a framework that isolates specific states in an agent's execution flow where LLM vulnerabilities manifest, enabling the systematic identification and categorization of security risks that propagate from the LLM to the agent level. We apply this framework to construct the $b^3$ benchmark, a security benchmark based on 194,331 unique crowdsourced ...