Hyperparameter Search with Transformers and Ray Tune

Back to Articles Hyperparameter Search with Transformers and Ray Tune Published November 2, 2020 Update on GitHub Upvote 4 system system Follow A guest blog post by Richard Liaw from the Anyscale team With cutting edge research implementations, thousands of trained models easily accessible, the Hugging Face transformers library has become critical to the success and growth of natural language processing today. For any machine learning model to achieve good performance, users often need to implement some form of parameter tuning. Yet, nearly everyone (1, 2) either ends up disregarding hyperparameter tuning or opting to do a simplistic grid search with a small search space. However, simple experiments are able to show the benefit of using an advanced tuning technique. Below is a recent experiment run on a BERT model from Hugging Face transformers on the RTE dataset. Genetic optimization techniques like PBT can provide large performance improvements compared to standard hyperparameter optimization techniques. Algorithm Best Val Acc. Best Test Acc. Total GPU min Total $ cost Grid Search 74% 65.4% 45 min $2.30 Bayesian Optimization +Early Stop 77% 66.9% 104 min $5.30 Population-based Training 78% 70.5% 48 min $2.45 If you’re leveraging Transformers, you’ll want to have a way to easily access powerful hyperparameter tuning solutions without giving up the customizability of the Transformers framework. In the Transformers 3.1 release, Hugging Face Transformers and Ray Tune teamed ...