![[2511.04485] Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2511.04485] Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
Summary
The paper introduces Q3R, a novel Quadratic Reweighted Rank Regularizer designed to enhance low-rank training in deep learning models, achieving efficient performance with minimal computational overhead.
Why It Matters
As deep learning models grow in size, efficient training methods like low-rank optimization become crucial. Q3R addresses challenges in maintaining low-rank structures while optimizing performance, making it relevant for researchers and practitioners focused on model efficiency and performance.
Key Takeaways
- Q3R effectively maintains low-rank structures during training.
- It achieves predictive performance comparable to dense models.
- The method shows minimal accuracy drops even with significant parameter reductions.
- Q3R is compatible with existing architectures, enhancing its applicability.
- Demonstrated efficacy across vision and language tasks.
Computer Science > Machine Learning arXiv:2511.04485 (cs) [Submitted on 6 Nov 2025 (v1), last revised 18 Feb 2026 (this version, v2)] Title:Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training Authors:Ipsita Ghosh, Ethan Nguyen, Christian Kümmerle View a PDF of the paper titled Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training, by Ipsita Ghosh and 2 other authors View PDF HTML (experimental) Abstract:Parameter-efficient training based on low-rank optimization has become a highly successful tool for fine-tuning large deep learning models. However, these methods often fail for low-rank pre-training, where simultaneously maintaining low-rank weight structure and optimizing the task objective remains challenging. We propose the $\textit{Quadratic Reweighted Rank Regularizer}$ ($\texttt{Q3R}$), which leads to a novel low-rank-inducing training strategy inspired by the Iteratively Reweighted Least Squares (IRLS) framework. $\texttt{Q3R}$ is based on a quadratic regularizer term that majorizes a smoothed log-determinant rank surrogate. Unlike other low-rank training techniques, $\texttt{Q3R}$ can train weight matrices to prescribed low target ranks while achieving predictive performance comparable to dense models, with small computational overhead and full compatibility with existing architectures. For example, we demonstrate a $\texttt{Q3R}$-regularized ViT-Tiny experiment where truncating the model to $60\%$ and $80\%$ of its param...
