![[2602.10531] From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.10531] From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources
Summary
This paper explores the dynamics of iterative training on contaminated data sources, demonstrating that model performance can improve despite the presence of synthetic samples if fresh information from the true distribution is included.
Why It Matters
Understanding how to mitigate model collapse during iterative training is crucial for improving the performance of generative models. This research provides insights into balancing synthetic and true data to enhance model robustness, which is particularly relevant in machine learning applications where data contamination is common.
Key Takeaways
- Model collapse can be mitigated by including true target distribution data.
- The interplay between sample size and mixture weights is critical for performance.
- Training in a contamination-agnostic manner can lead to recovery of the true distribution.
- Simulation studies validate the findings across various model classes.
- Fresh information from true distributions is essential for iterative training success.
Statistics > Machine Learning arXiv:2602.10531 (stat) [Submitted on 11 Feb 2026 (v1), last revised 17 Feb 2026 (this version, v2)] Title:From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources Authors:Soham Bakshi, Sunrit Chakraborty View a PDF of the paper titled From Collapse to Improvement: Statistical Perspectives on the Evolutionary Dynamics of Iterative Training on Contaminated Sources, by Soham Bakshi and 1 other authors View PDF HTML (experimental) Abstract:The problem of model collapse has presented new challenges in iterative training of generative models, where such training with synthetic data leads to an overall degradation of performance. This paper looks at the problem from a statistical viewpoint, illustrating that one can actually hope for improvement when models are trained on data contaminated with synthetic samples, as long as there is some amount of fresh information from the true target distribution. In particular, we consider iterative training on samples sourced from a mixture of the true target and synthetic distributions. We analyze the entire iterative evolution in a next-token prediction language model, capturing how the interplay between the mixture weights and the sample size controls the overall long-term performance. With non-trivial mixture weight of the true distribution, even if it decays over time, simply training the model in a contamination-agnostic manner with a...
