[2507.04219] Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs

Computer Science > Machine Learning arXiv:2507.04219 (cs) [Submitted on 6 Jul 2025 (v1), last revised 1 Mar 2026 (this version, v4)] Title:Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs Authors:Yan Scholten, Sophie Xhonneux, Leo Schwinn, Stephan Günnemann View a PDF of the paper titled Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs, by Yan Scholten and 3 other authors View PDF Abstract:Current unlearning methods for LLMs optimize on the private information they seek to remove by incorporating it into their fine-tuning data. We argue this not only risks reinforcing exposure to sensitive data, but also fundamentally contradicts the principle of minimizing its use. As a remedy, we propose a novel unlearning method-Partial Model Collapse (PMC), which does not require unlearning targets in the unlearning objective. Our approach is inspired by recent observations that training generative models on their own generations leads to distribution collapse, effectively removing information from model outputs. Our central insight is that model collapse can be leveraged for machine unlearning by deliberately triggering it for data we aim to remove. We theoretically analyze that our approach converges to the desired outcome, i.e. the model unlearns the data targeted for removal. We empirically demonstrate that PMC overcomes four key limitations of existing unlearning methods that explicitly optimize on unlearning targets, and more effect...