![[2602.21222] Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21222] Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases
Summary
This paper presents a novel framework for dynamic LoRA adapter composition using similarity retrieval in vector databases, enabling efficient zero-shot generalization across various NLP tasks.
Why It Matters
The research addresses the challenge of composing multiple specialized adapters for unseen tasks in large language models, enhancing multitask learning efficiency without retraining. This is crucial for advancing AI applications in natural language processing, where adaptability and efficiency are paramount.
Key Takeaways
- Introduces a framework for dynamic LoRA adapter composition leveraging similarity retrieval.
- Demonstrates improved performance in zero-shot tasks compared to traditional methods.
- Utilizes a task-aware vector database constructed from 22 diverse datasets.
- Evaluates multiple merging methods, with Linear merging achieving significant performance gains.
- Enables efficient multitask learning without full model retraining.
Computer Science > Computation and Language arXiv:2602.21222 (cs) [Submitted on 1 Feb 2026] Title:Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases Authors:Riya Adsul, Balachandra Devarangadi Sunil, Isha Nalawade, Sudharshan Govindan View a PDF of the paper titled Task-Aware LoRA Adapter Composition via Similarity Retrieval in Vector Databases, by Riya Adsul and 3 other authors View PDF HTML (experimental) Abstract:Parameter efficient fine tuning methods like LoRA have enabled task specific adaptation of large language models, but efficiently composing multiple specialized adapters for unseen tasks remains challenging. We present a novel framework for dynamic LoRA adapter composition that leverages similarity retrieval in vector databases to enable zero-shot generalization across diverse NLP tasks. Our approach constructs a task-aware vector database by embedding training examples from 22 datasets spanning commonsense reasoning, question answering, natural language inference, and sentiment analysis. At inference time, we retrieve the most similar training examples, compute task similarity distributions via nucleus sampling, and dynamically merge relevant LoRA adapters using retrieval weighted fusion strategies. We evaluated four merging methods Linear, Concatenation, TIES, and Magnitude Prune demonstrating that our dataset centric retrieval approach often matches or exceeds the performance of individually fine-tuned task-specific adapters. No...