[2508.07638] Data Selection for LLM Alignment Using Fine-Grained Preferences

Computer Science > Machine Learning arXiv:2508.07638 (cs) [Submitted on 11 Aug 2025 (v1), last revised 2 Mar 2026 (this version, v2)] Title:Data Selection for LLM Alignment Using Fine-Grained Preferences Authors:Jia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu, Yi-Xuan Jin, Lan-Zhe Guo, Yu-Feng Li View a PDF of the paper titled Data Selection for LLM Alignment Using Fine-Grained Preferences, by Jia Zhang and 6 other authors View PDF HTML (experimental) Abstract:Large language models (LLMs) alignment aims to ensure that the behavior of LLMs meets human preferences. While collecting data from multiple fine-grained, aspect-specific preferences becomes more and more feasible, existing alignment methods typically work on a single preference and thus struggle with conflicts inherent in such aggregated datasets. As one early attempt, in this paper, we propose a data-centric approach to align LLMs through the effective use of fine-grained preferences. Specifically, we formulate the problem as a direct fine-grained preference optimization and introduce preference divergence (PD) that quantifies inter-aspect preference conflicts. Instead of directly tackling the consequent complicated optimization, we recast it as a data selection problem and propose a simple yet effective strategy, which identifies a subset of data corresponding to the most negative PD values, for efficient training. We theoretically analyze the loss-bound optimality of our selection strategy and conduct extensive empiric...