[2601.11471] Low-Rank Key Value Attention

Computer Science > Machine Learning arXiv:2601.11471 (cs) [Submitted on 16 Jan 2026 (v1), last revised 7 Apr 2026 (this version, v3)] Title:Low-Rank Key Value Attention Authors:James O'Neill, Robert Clancy, Mariia Matskevichus, Fergal Reid View a PDF of the paper titled Low-Rank Key Value Attention, by James O'Neill and 3 other authors View PDF HTML (experimental) Abstract:The key-value (KV) cache is a primary memory bottleneck in Transformers. We propose Low-Rank Key-Value (LRKV) attention, which reduces KV cache memory by exploiting redundancy across attention heads, while being compute efficient. Each layer uses a shared full-rank KV projection augmented with low-rank, head-specific residuals, providing a continuous trade-off between complete sharing and full independence. After pretraining models of size 128M to 6.3B parameters, LRKV consistently achieves the lowest test loss among standard MHA, MQA/GQA, and MLA while using only 45-53\% of MHA's KV cache. LRKV reaches equivalent baseline quality 18-25\% faster (measured in training steps). After supervised midtraining, LRKV achieves the highest downstream task performance across ARC-Easy, ARC-Challenge, MMLU, GSM8K, and HumanEval benchmarks. Subjects: Machine Learning (cs.LG) Cite as: arXiv:2601.11471 [cs.LG]   (or arXiv:2601.11471v3 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2601.11471 Focus to learn more arXiv-issued DOI via DataCite Submission history From: James O'Neill [view email] [v1] Fri, 16 Jan...