[2506.15963] On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy

Computer Science > Machine Learning arXiv:2506.15963 (cs) [Submitted on 19 Jun 2025 (v1), last revised 4 Mar 2026 (this version, v2)] Title:On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy Authors:Jingyi Cui, Qi Zhang, Yifei Wang, Yisen Wang View a PDF of the paper titled On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy, by Jingyi Cui and 3 other authors View PDF HTML (experimental) Abstract:Sparse autoencoders (SAEs) have recently emerged as a powerful tool for interpreting the features learned by large language models (LLMs). By reconstructing features with sparsely activated networks, SAEs aim to recover complex superposed polysemantic features into interpretable monosemantic ones. Despite their wide applications, it remains unclear under what conditions SAEs can fully recover the ground truth monosemantic features from the superposed polysemantic ones. In this paper, we provide the first theoretical analysis with a closed-form solution for SAEs, revealing that they generally fail to fully recover the ground truth monosemantic features unless the ground truth features are extremely sparse. To improve the feature recovery of SAEs in general cases, we propose a reweighting strategy targeting at enhancing the reconstruction of the ground truth monosemantic features instead of the observed polysemantic ones. We further establish a theoretical weight selection principle for our proposed weighted SAE (WSAE)...