GaLore: Advancing Large Model Training on Consumer-grade Hardware

Back to Articles GaLore: Advancing Large Model Training on Consumer-grade Hardware Published March 20, 2024 Update on GitHub Upvote 32 +26 Titus von Koeller Titus-von-Koeller Follow Jiawei Zhao jiaweizhao Follow guest Matthew Douglas mdouglas Follow guest Yaowei Zheng hiyouga Follow guest Younes B ybelkada Follow Zachary Mueller muellerzr Follow Amy Roberts amyeroberts Follow Sourab Mangrulkar smangrul Follow Benjamin Bossan BenjaminB Follow The integration of GaLore into the training of large language models (LLMs) marks a significant advancement in the field of deep learning, particularly in terms of memory efficiency and the democratization of AI research. By allowing for the training of billion-parameter models on consumer-grade hardware, reducing memory footprint in optimizer states, and leveraging advanced projection matrix techniques, GaLore opens new horizons for researchers and practitioners with limited access to high-end computational resources. Scaling LLMs with Consumer-Grade Hardware The capability of GaLore to facilitate the training of models with up to 7 billion parameters, such as those based on the Llama architecture, on consumer GPUs like the NVIDIA RTX 4090, is groundbreaking. This is achieved by significantly reducing the memory requirements traditionally associated with optimizer states and gradients during the training process. The approach leverages the inherent low-rank structure of gradients in deep neural networks, applying a projection that red...