![[2602.14778] A Geometric Analysis of Small-sized Language Model Hallucinations](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.14778] A Geometric Analysis of Small-sized Language Model Hallucinations
Summary
This paper explores hallucinations in small-sized language models (LLMs) through a geometric lens, demonstrating that genuine responses cluster tightly in embedding space and introducing a label-efficient method for response classification.
Why It Matters
Understanding hallucinations in LLMs is crucial for improving their reliability, especially in applications requiring multi-step reasoning. This research offers a novel geometric perspective that enhances traditional evaluation methods and proposes efficient classification techniques, contributing to the ongoing discourse in AI safety and model robustness.
Key Takeaways
- Hallucinations in LLMs can be analyzed geometrically, revealing clustering patterns.
- Genuine responses exhibit tighter clustering in embedding space compared to hallucinated ones.
- A new label-efficient method allows classification of responses with minimal annotations, achieving high F1 scores.
- This approach complements existing knowledge-centric evaluation paradigms.
- The findings pave the way for further research into mitigating hallucinations in AI models.
Computer Science > Computation and Language arXiv:2602.14778 (cs) [Submitted on 16 Feb 2026] Title:A Geometric Analysis of Small-sized Language Model Hallucinations Authors:Emanuele Ricco, Elia Onofri, Lorenzo Cima, Stefano Cresci, Roberto Di Pietro View a PDF of the paper titled A Geometric Analysis of Small-sized Language Model Hallucinations, by Emanuele Ricco and 4 other authors View PDF HTML (experimental) Abstract:Hallucinations -- fluent but factually incorrect responses -- pose a major challenge to the reliability of language models, especially in multi-step or agentic settings. This work investigates hallucinations in small-sized LLMs through a geometric perspective, starting from the hypothesis that when models generate multiple responses to the same prompt, genuine ones exhibit tighter clustering in the embedding space, we prove this hypothesis and, leveraging this geometrical insight, we also show that it is possible to achieve a consistent level of separability. This latter result is used to introduce a label-efficient propagation method that classifies large collections of responses from just 30-50 annotations, achieving F1 scores above 90%. Our findings, framing hallucinations from a geometric perspective in the embedding space, complement traditional knowledge-centric and single-response evaluation paradigms, paving the way for further research. Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computers and Society (cs.CY) Cite a...
