[2603.04198] Stable and Steerable Sparse Autoencoders with Weight Regularization

Statistics > Machine Learning arXiv:2603.04198 (stat) [Submitted on 4 Mar 2026] Title:Stable and Steerable Sparse Autoencoders with Weight Regularization Authors:Piotr Jedryszek, Oliver M. Crook View a PDF of the paper titled Stable and Steerable Sparse Autoencoders with Weight Regularization, by Piotr Jedryszek and 1 other authors View PDF HTML (experimental) Abstract:Sparse autoencoders (SAEs) are widely used to extract human-interpretable features from neural network activations, but their learned features can vary substantially across random seeds and training choices. To improve stability, we studied weight regularization by adding L1 or L2 penalties on encoder and decoder weights, and evaluate how regularization interacts with common SAE training defaults. On MNIST, we observe that L2 weight regularization produces a core of highly aligned features and, when combined with tied initialization and unit-norm decoder constraints, it dramatically increases cross-seed feature consistency. For TopK SAEs trained on language model activations (Pythia-70M-deduped), adding a small L2 weight penalty increased the fraction of features shared across three random seeds and roughly doubles steering success rates, while leaving the mean of automated interpretability scores essentially unchanged. Finally, in the regularized setting, activation steering success becomes better predicted by auto-interpretability scores, suggesting that regularization can align text-based feature explanat...