[2603.28263] Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

Computer Science > Computation and Language arXiv:2603.28263 (cs) [Submitted on 30 Mar 2026] Title:Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights Authors:Eneko Valero, Maria Ribalta i Albado, Oscar Sainz, Naiara Perez, German Rigau View a PDF of the paper titled Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights, by Eneko Valero and 4 other authors View PDF HTML (experimental) Abstract:Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior ...