[2604.08723] Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Computer Science > Computation and Language arXiv:2604.08723 (cs) [Submitted on 9 Apr 2026] Title:Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? Authors:Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai, Supriyo Chakraborty, Shixiong Zhang, Sambit Sahu, William Campbell View a PDF of the paper titled Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?, by Chia-Hsuan Lee and 8 other authors View PDF HTML (experimental) Abstract:Preference optimization methods such as DPO and KTO are widely used for aligning language models, yet little is understood about what properties of preference data drive downstream reasoning gains. We ask: what aspects of a preference pair improve a reasoning model's performance on general reasoning tasks? We investigate two distinct notions of quality delta in preference data: generator-level delta, arising from the differences in capability between models that generate chosen and rejected reasoning traces, and sample-level delta, arising from differences in judged quality differences within an individual preference pair. To study generator-level delta, we vary the generator's scale and model family, and to study sample-level delta, we employ an LLM-as-a-judge to rate the quality of generated traces along multiple reasoning-quality dimensions. We find that increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks and filtering data by sample-lev...