[2604.04497] One Model for All: Multi-Objective Controllable Language Models

Computer Science > Machine Learning arXiv:2604.04497 (cs) [Submitted on 6 Apr 2026] Title:One Model for All: Multi-Objective Controllable Language Models Authors:Qiang He, Yucheng Yang, Tianyi Zhou, Meng Fang, Mykola Pechenizkiy, Setareh Maghsudi View a PDF of the paper titled One Model for All: Multi-Objective Controllable Language Models, by Qiang He and Yucheng Yang and Tianyi Zhou and Meng Fang and Mykola Pechenizkiy and Setareh Maghsudi View PDF HTML (experimental) Abstract:Aligning large language models (LLMs) with human preferences is critical for enhancing LLMs' safety, helpfulness, humor, faithfulness, etc. Current reinforcement learning from human feedback (RLHF) mainly focuses on a fixed reward learned from average human ratings, which may weaken the adaptability and controllability of varying preferences. However, creating personalized LLMs requires aligning LLMs with individual human preferences, which is non-trivial due to the scarce data per user and the diversity of user preferences in multi-objective trade-offs, varying from emphasizing empathy in certain contexts to demanding efficiency and precision in others. Can we train one LLM to produce personalized outputs across different user preferences on the Pareto front? In this paper, we introduce Multi-Objective Control (MOC), which trains a single LLM to directly generate responses in the preference-defined regions of the Pareto front. Our approach introduces multi-objective optimization (MOO) principles i...