How we sped up transformer inference 100x for 🤗 API customers

Back to Articles How we sped up transformer inference 100x for 🤗 API customers Published January 18, 2021 Update on GitHub Upvote - Nicolas Patry Narsil Follow 🤗 Transformers has become the default library for data scientists all around the world to explore state of the art NLP models and build new NLP features. With over 5,000 pre-trained and fine-tuned models available, in over 250 languages, it is a rich playground, easily accessible whichever framework you are working in. While experimenting with models in 🤗 Transformers is easy, deploying these large models into production with maximum performance, and managing them into an architecture that scales with usage is a hard engineering challenge for any Machine Learning Engineer. This 100x performance gain and built-in scalability is why subscribers of our hosted Accelerated Inference API chose to build their NLP features on top of it. To get to the last 10x of performance boost, the optimizations need to be low-level, specific to the model, and to the target hardware. This post shares some of our approaches squeezing every drop of compute juice for our customers. 🍋 Getting to the first 10x speedup The first leg of the optimization journey is the most accessible, all about using the best combination of techniques offered by the Hugging Face libraries, independent of the target hardware. We use the most efficient methods built into Hugging Face model pipelines to reduce the amount of computation during each forward pass. Th...