![[2602.20162] Talking to Yourself: Defying Forgetting in Large Language Models](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.20162] Talking to Yourself: Defying Forgetting in Large Language Models
Summary
The paper introduces SA-SFT, a self-augmentation method for fine-tuning large language models (LLMs) that mitigates catastrophic forgetting while enhancing in-domain performance.
Why It Matters
Catastrophic forgetting in LLMs is a significant challenge that affects their general knowledge and reasoning. The proposed SA-SFT method offers a novel approach to maintain model performance during task-specific fine-tuning, which is crucial for the development of robust AI systems.
Key Takeaways
- SA-SFT generates self-dialogues to counteract catastrophic forgetting in LLMs.
- The method improves in-domain performance without requiring external data.
- Empirical results show SA-SFT outperforms traditional methods in 40 out of 50 evaluation scenarios.
- Theoretical analysis links forgetting to style-induced parameter drift.
- Self-augmentation is presented as a simple yet effective adaptation mechanism.
Computer Science > Computation and Language arXiv:2602.20162 (cs) [Submitted on 23 Jan 2026] Title:Talking to Yourself: Defying Forgetting in Large Language Models Authors:Yutao Sun, Mingshuai Chen, Tiancheng Zhao, Phillip Miao, Zilun Zhang, Haozhan Shen, Ruizhe Zhu, Jianwei Yin View a PDF of the paper titled Talking to Yourself: Defying Forgetting in Large Language Models, by Yutao Sun and 7 other authors View PDF HTML (experimental) Abstract:Catastrophic forgetting remains a major challenge when fine-tuning large language models (LLMs) on narrow, task-specific data, often degrading their general knowledge and reasoning abilities. We propose SA-SFT, a lightweight self-augmentation routine in which an LLM generates self-dialogues prior to fine-tuning, and the resulting self-authored data are mixed with task data without modifying optimization or training schedules. Despite requiring no external data or additional tuning, SA-SFT consistently mitigates catastrophic forgetting while improving in-domain performance. Across 50 evaluation scenarios, it maintains performance comparable to the original model and achieves the best results in 40 cases, outperforming common baselines such as layer freezing and external data mixing. Guided by these empirical findings, we further present a theoretical analysis suggesting that forgetting can partly stem from style-induced parameter drift, and that self-alignment through self-generated data provides an effective means to counteract this ...