LoRA training scripts of the world, unite!

Back to Articles LoRA training scripts of the world, unite! Published January 2, 2024 Update on GitHub Upvote 76 +70 Linoy Tsaban linoyts Follow Apolinário from multimodal AI art multimodalart Follow A community derived guide to some of the SOTA practices for SD-XL Dreambooth LoRA fine tuning TL;DR We combined the Pivotal Tuning technique used on Replicate's SDXL Cog trainer with the Prodigy optimizer used in the Kohya trainer (plus a bunch of other optimizations) to achieve very good results on training Dreambooth LoRAs for SDXL. Check out the training script on diffusers🧨. Try it out on Colab. If you want to skip the technical talk, you can use all the techniques in this blog and train on Hugging Face Spaces with a simple UI and curated parameters (that you can meddle with). Overview Stable Diffusion XL (SDXL) models fine-tuned with LoRA dreambooth achieve incredible results at capturing new concepts using only a handful of images, while simultaneously maintaining the aesthetic and image quality of SDXL and requiring relatively little compute and resources. Check out some of the awesome SDXL LoRAs here.In this blog, we'll review some of the popular practices and techniques to make your LoRA finetunes go brrr, and show how you can run or train yours now with diffusers! Recap: LoRA (Low-Rank Adaptation) is a fine-tuning technique for Stable Diffusion models that makes slight adjustments to the crucial cross-attention layers where images and prompts intersect. It achieves q...