[2603.20572] LJ-Bench: Ontology-Based Benchmark for U.S. Crime

Computer Science > Machine Learning arXiv:2603.20572 (cs) [Submitted on 21 Mar 2026] Title:LJ-Bench: Ontology-Based Benchmark for U.S. Crime Authors:Hung Yun Tseng, Wuzhen Li, Blerina Gkotse, Grigorios Chrysos View a PDF of the paper titled LJ-Bench: Ontology-Based Benchmark for U.S. Crime, by Hung Yun Tseng and 3 other authors View PDF Abstract:The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories: LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs. The LJ-...