[2604.02766] Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs

Computer Science > Machine Learning arXiv:2604.02766 (cs) [Submitted on 3 Apr 2026] Title:Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs Authors:Giyeong Oh, Junghyun Lee, Jaehyun Park, Youngjae Yu, Wonho Bae, Junhyug Noh View a PDF of the paper titled Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs, by Giyeong Oh and 4 other authors View PDF HTML (experimental) Abstract:Modern LLMs inherit strong priors from web-scale pretraining, which can limit the headroom of post-training data-selection strategies. While Active Preference Learning (APL) seeks to optimize query efficiency in online Direct Preference Optimization (DPO), the inherent richness of on-policy candidate pools often renders simple Random sampling a surprisingly formidable baseline. We evaluate uncertainty-based APL against Random across harmlessness, helpfulness, and instruction-following settings, utilizing both reward models and LLM-as-a-judge proxies. We find that APL yields negligible improvements in proxy win-rates compared to Random. Crucially, we observe a dissociation where win-rate improves even as general capability -- measured by standard benchmarks -- degrades. APL fails to mitigate this capability collapse or reduce variance significantly better than random sampling. Our findings suggest that in the regime of strong pre-trained priors, the computational overhead of active selection is difficult to justify against the ``cheap diversity'' provided by s...