[2508.14285] Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning

Computer Science > Machine Learning arXiv:2508.14285 (cs) [Submitted on 19 Aug 2025 (v1), last revised 1 Apr 2026 (this version, v3)] Title:Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning Authors:Liyi Zhang, Jake Snell, Thomas L. Griffiths View a PDF of the paper titled Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning, by Liyi Zhang and 2 other authors View PDF HTML (experimental) Abstract:Fine-tuning large language models (LLMs) with low-rank adaptation (LoRA) is a cost-effective way to incorporate information from a specific dataset. However, when a problem requires incorporating information from multiple datasets - as in few shot learning - generalization across datasets can be limited, driving up training costs. As a consequence, other approaches such as in-context learning are typically used in this setting. To address this challenge, we introduce an efficient method for adapting the weights of LLMs to multiple distributions, Amortized Bayesian Meta-Learning for LoRA (ABMLL). This method builds on amortized Bayesian meta-learning for smaller models, adapting this approach to LLMs by reframing where local and global variables are defined in LoRA and using a new hyperparameter to balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. ABMLL supports effective generalization across datasets and scales to large models such as Llama3-8B and ...