[2509.15888] Distribution-Aligned Decoding for Efficient LLM Task Adaptation

Computer Science > Computation and Language arXiv:2509.15888 (cs) [Submitted on 19 Sep 2025 (v1), last revised 28 Feb 2026 (this version, v5)] Title:Distribution-Aligned Decoding for Efficient LLM Task Adaptation Authors:Senkang Hu, Xudong Han, Jinqi Jiang, Yihang Tao, Zihan Fang, Yong Dai, Sam Tak Wu Kwong, Yuguang Fang View a PDF of the paper titled Distribution-Aligned Decoding for Efficient LLM Task Adaptation, by Senkang Hu and 7 other authors View PDF HTML (experimental) Abstract:Adapting billion-parameter language models to a downstream task is still costly, even with parameter-efficient fine-tuning (PEFT). We re-cast task adaptation as output-distribution alignment: the objective is to steer the output distribution toward the task distribution directly during decoding rather than indirectly through weight updates. Building on this view, we introduce Steering Vector Decoding (SVDecode), a lightweight, PEFT-compatible, and theoretically grounded method. We start with a short warm-start fine-tune and extract a task-aware steering vector from the Kullback-Leibler (KL) divergence gradient between the output distribution of the warm-started and pre-trained models. This steering vector is then used to guide the decoding process to steer the model's output distribution towards the task distribution. We theoretically prove that SVDecode is first-order equivalent to the gradient step of full fine-tuning and derive a globally optimal solution for the strength of the steering ...