[2511.21448] The Phish, The Spam, and The Valid: Generating Feature-Rich Emails for Benchmarking LLMs

Computer Science > Cryptography and Security arXiv:2511.21448 (cs) [Submitted on 26 Nov 2025 (v1), last revised 20 Mar 2026 (this version, v5)] Title:The Phish, The Spam, and The Valid: Generating Feature-Rich Emails for Benchmarking LLMs Authors:Rebeka Toth, Tamas Bisztray, Nils Gruschka View a PDF of the paper titled The Phish, The Spam, and The Valid: Generating Feature-Rich Emails for Benchmarking LLMs, by Rebeka Toth and 2 other authors View PDF HTML (experimental) Abstract:In this paper, we introduce a metadata-enriched generation framework (PhishFuzzer) that seeds real emails into Large Language Models (LLMs) to produce 23,100 diverse, structurally consistent email variants across controlled entity and length dimensions. Unlike prior corpora, our dataset features strict three-class labels (Phishing, Spam, Valid), provides full URL and attachment metadata, and annotates each email with attacker intent. Using this dataset, we benchmark two state-of-the-art LLMs (Qwen-2.5-72B and Gemini-3.1-Pro) under both Basic (body, subject) and Full (+URL, sender, attachment) settings. By applying formal confidence metrics (Task Success Rate and Confidence Index), we analyze model reliability, robustness against linguistic fuzzing, and the impact of structural metadata on detection accuracy. Our fully open-source framework and dataset provide a rigorous foundation for evaluating next-generation email security systems. To support open science, we make the PhishFuzzer Dataset, the ge...