Preference Tuning LLMs with Direct Preference Optimization Methods

Back to Articles Preference Tuning LLMs with Direct Preference Optimization Methods Published January 18, 2024 Update on GitHub Upvote 78 +72 Kashif Rasul kashif Follow Edward Beeching edbeeching Follow Lewis Tunstall lewtun Follow Leandro von Werra lvwerra Follow Omar Sanseviero osanseviero Follow Addendum After consulting with the authors of the IPO paper, we discovered that the implementation of IPO in TRL was incorrect; in particular, the loss over the log-likelihoods of the completions needs to be averaged instead of summed. We have added a fix in this PR and re-run the experiments. The results are now consistent with the paper, with IPO on par with DPO and performing better than KTO in the paired preference setting. We have updated the post to reflect these new results. TL;DR We evaluate three promising methods to align language models without reinforcement learning (or preference tuning) on a number of models and hyperparameter settings. In particular we train using different hyperparameters and evaluate on: Direct Preference Optimization (DPO) Identity Preference Optimisation (IPO) Kahneman-Tversky Optimisation (KTO) Introduction In this post, we perform an empirical evaluation of three promising LLM alignment algorithms: Direct Preference Optimization (DPO), Identity Preference Optimisation (IPO) and Kahneman-Tversky Optimisation (KTO). We conducted our experiments on two high quality 7b LLMs that have undergone a supervised fine-tuning step, but no preference ali...