![[2510.03313] Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2510.03313] Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining
Summary
The paper introduces a new dimensionless data-quality parameter for language model pretraining, establishing a quality-aware scaling law that predicts loss based on model size, data volume, and data quality.
Why It Matters
Understanding the impact of data quality on language model performance is crucial for optimizing resource allocation in AI development. This research provides a formalized approach to balancing data curation efforts with model scaling, which can lead to more efficient training processes and better-performing models.
Key Takeaways
- Introduces a dimensionless data-quality parameter Q for language models.
- Proposes a quality-aware scaling law extending the Chinchilla framework.
- Demonstrates that higher-quality data can reduce model size and compute requirements.
- Establishes a generalizable law for data quality in language model training.
- Offers practical estimators for data quality, aiding in data curation decisions.
Computer Science > Machine Learning arXiv:2510.03313 (cs) [Submitted on 30 Sep 2025 (v1), last revised 23 Feb 2026 (this version, v2)] Title:Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining Authors:Anirudh Subramanyam, Yuxin Chen, Robert L. Grossman View a PDF of the paper titled Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining, by Anirudh Subramanyam and 2 other authors View PDF HTML (experimental) Abstract:Scaling laws for language model training traditionally characterize how performance scales with model size and dataset volume. Prior work has explored architecture variants and data treatments such as dataset filtering and noise injection in language model pretraining; however, these studies have not formalized data quality within a principled scaling law. We introduce a dimensionless data-quality parameter Q, and propose a quality-aware scaling law extending the Chinchilla framework to predict loss as a joint function of model size, data volume, and data quality. The law is motivated by an effective-sample-size and information-theoretic view of noisy or redundant corpora, and it admits two practical estimators for Q: (i) a corruption rate proxy and (ii) a deficiency measure. Through synthetic experiments in neural machine translation and autoregressive modeling -- where we systematically control data quality via multiple levels of noise injection variation -- we show that loss scales predict...
