[2603.28998] Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

Computer Science > Cryptography and Security arXiv:2603.28998 (cs) [Submitted on 30 Mar 2026] Title:Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems Authors:Yicheng Cai, Mitchell John DeStefano, Guodong Dong, Pulkit Handa, Peng Liu, Tejas Singhal, Peiyu Tseng, Winston Jen White View a PDF of the paper titled Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems, by Yicheng Cai and 7 other authors View PDF HTML (experimental) Abstract:As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to achieve more autonomous SOCs (security operation centers) and reduce manual effort. In particular, the AI and cybersecurity communities have recently developed several benchmarks for evaluating the red team capabilities of multi-agent AI systems. However, because the operations in SOCs are dominated by blue team operations, the capabilities of AI systems & agents to achieve more autonomous SOCs cannot be evaluated without a benchmark focused on blue team operations. To our best knowledge, no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature. Existing blu...