![[2602.22345] Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.22345] Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory
Summary
This paper explores the reliability and efficiency of large language models (LLMs) using Random Matrix Theory. It introduces EigenTrack for real-time detection of model failures and RMT-KD for compressing networks while maintaining performance.
Why It Matters
As LLMs become more prevalent, understanding their internal dynamics is crucial for improving reliability and efficiency. This research offers innovative methods to address hallucinations and optimize model performance, which are significant challenges in AI development.
Key Takeaways
- EigenTrack provides a method for real-time detection of hallucinations in LLMs.
- RMT-KD enables efficient compression of deep networks while preserving accuracy.
- Spectral analysis offers insights into model behavior and representation dynamics.
- The study addresses critical issues of reliability and computational efficiency in AI.
- Understanding eigenvalue dynamics can enhance model interpretability.
Computer Science > Machine Learning arXiv:2602.22345 (cs) [Submitted on 25 Feb 2026] Title:Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory Authors:Davide Ettori View a PDF of the paper titled Structure and Redundancy in Large Language Models: A Spectral Study via Random Matrix Theory, by Davide Ettori View PDF HTML (experimental) Abstract:This thesis addresses two persistent and closely related challenges in modern deep learning, reliability and efficiency, through a unified framework grounded in Spectral Geometry and Random Matrix Theory (RMT). As deep networks and large language models continue to scale, their internal behavior becomes increasingly opaque, leading to hallucinations, fragile generalization under distribution shift, and growing computational and energy demands. By analyzing the eigenvalue dynamics of hidden activations across layers and inputs, this work shows that spectral statistics provide a compact, stable, and interpretable lens on model behavior, capable of separating structured, causal representations from noise-dominated variability. Within this framework, the first contribution, EigenTrack, introduces a real-time method for detecting hallucinations and out-of-distribution behavior in large language and vision-language models. EigenTrack transforms streaming activations into spectral descriptors such as entropy, variance, and deviations from the Marchenko-Pastur baseline, and models their temporal evolut...
