[2603.04964] Replaying pre-training data improves fine-tuning

Computer Science > Computation and Language arXiv:2603.04964 (cs) [Submitted on 5 Mar 2026] Title:Replaying pre-training data improves fine-tuning Authors:Suhas Kotha, Percy Liang View a PDF of the paper titled Replaying pre-training data improves fine-tuning, by Suhas Kotha and 1 other authors View PDF HTML (experimental) Abstract:To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited amount of target data. Typically, generic data is only mixed in during fine-tuning to prevent catastrophic forgetting of the generic domain. We surprisingly find that replaying the generic data during fine-tuning can actually improve performance on the (less related) target task. Concretely, in a controlled pre-training environment with 4M target tokens, 4B total tokens, and 150M parameter models, generic replay increases target data efficiency by up to $1.87\times$ for fine-tuning and $2.06\times$ for mid-training. We further analyze data schedules that introduce target data during pre-training and find that replay helps more when there is less target data present in pre-training. We demonstrate the success of replay in practice for fine-tuning 8B parameter models, improving agentic web navigation success by $4.5\%$ and Basque question-answering accuracy by $2\%$. Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG) Cite as: arXiv:2603.04964 [cs.CL]   (...