Fine-Tune W2V2-Bert for low-resource ASR with 🤗 Transformers

Back to Articles Fine-Tune W2V2-Bert for low-resource ASR with 🤗 Transformers Published January 19, 2024 Update on GitHub Upvote 44 +38 Yoach Lacombe ylacombe Follow New (01/2024): This blog post is strongly inspired by "Fine-tuning XLS-R on Multi-Lingual ASR" and "Fine-tuning MMS Adapter Models for Multi-Lingual ASR". Introduction Last month, MetaAI released Wav2Vec2-BERT, as a building block of their Seamless Communication, a family of AI translation models. Wav2Vec2-BERT is the result of a series of improvements based on an original model: Wav2Vec2, a pre-trained model for Automatic Speech Recognition (ASR) released in September 2020 by Alexei Baevski, Michael Auli, and Alex Conneau. With as little as 10 minutes of labeled audio data, Wav2Vec2 could be fine-tuned to achieve 5% word-error rate performance on the LibriSpeech dataset, demonstrating for the first time low-resource transfer learning for ASR. Following a series of multilingual improvements (XLSR, XLS-R and MMS), Wav2Vec2-BERT is a 580M-parameters versatile audio model that has been pre-trained on 4.5M hours of unlabeled audio data covering more than 143 languages. For comparison, XLS-R used almost half a million hours of audio data in 128 languages and MMS checkpoints were pre-trained on more than half a million hours of audio in over 1,400 languages. Boosting to millions of hours enables Wav2Vec2-BERT to achieve even more competitive results in speech-related tasks, whatever the language. To use it for ASR, ...