mmBERT: ModernBERT goes Multilingual

Back to Articles mmBERT: ModernBERT goes Multilingual Published September 9, 2025 Update on GitHub Upvote 133 +127 Marc Marone mmarone Follow jhu-clsp Orion Weller orionweller Follow jhu-clsp William Fleshman will-fleshman Follow jhu-clsp Eugene Yang eugene-yang Follow jhu-clsp Dawn Lawrie dlawrie Follow jhu-clsp Ben Van Durme vandurme Follow jhu-clsp TL;DR This blog post introduces mmBERT, a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual models, being the first to improve upon XLM-R, while also developing new strategies for effectively learning low-resource languages. mmBERT builds upon ModernBERT for a blazingly fast architecture, and adds novel components to enable efficient multilingual learning. If you are interested in trying out the models yourself, some example boilerplate is available at the end of this blogpost! Training Data Figure 1: the training data is progressively annealed to include more languages and more uniform sampling throughout training. mmBERT was trained on a carefully curated multilingual dataset totaling over 3T tokens across three distinct training phases. The foundation of our training data consists of three primary open-source and high-quality web crawls that enable both multilingual coverage and data quality: DCLM and Filtered DCLM provides the highest quality English content available, serving as the...