[2603.11687] SemBench: A Universal Semantic Framework for LLM Evaluation

Computer Science > Computation and Language arXiv:2603.11687 (cs) [Submitted on 12 Mar 2026 (v1), last revised 26 Mar 2026 (this version, v2)] Title:SemBench: A Universal Semantic Framework for LLM Evaluation Authors:Mikel Zubillaga, Naiara Perez, Oscar Sainz, German Rigau View a PDF of the paper titled SemBench: A Universal Semantic Framework for LLM Evaluation, by Mikel Zubillaga and 3 other authors View PDF HTML (experimental) Abstract:Recent progress in Natural Language Processing (NLP) has been driven by the emergence of Large Language Models (LLMs), which exhibit remarkable generative and reasoning capabilities. However, despite their success, evaluating the true semantic understanding of these models remains a persistent challenge. Traditional benchmarks such as Word-in-Context (WiC) effectively probe this capability, but their creation is resource-intensive and often limited to high-resource languages. In this paper, we introduce SemBench, a framework for automatically generating synthetic benchmarks that assess the semantic competence of LLMs using only dictionary sense definitions and a sentence encoder. This approach eliminates the need for curated example sentences, making it both scalable and language-independent. We evaluate SemBench in three languages (English, Spanish, and Basque) spanning different levels of linguistic resources, and across a wide range of LLMs. Our results show that rankings derived from SemBench strongly correlate with those obtained fro...