[2603.08022] Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization

Computer Science > Machine Learning arXiv:2603.08022 (cs) [Submitted on 9 Mar 2026 (v1), last revised 6 May 2026 (this version, v2)] Title:Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization Authors:Jingwei Li, Xinran Gu, Jingzhao Zhang View a PDF of the paper titled Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization, by Jingwei Li and 2 other authors View PDF HTML (experimental) Abstract:A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target mode...