mmBERT

mmBERT is a multilingual encoder released in 2025 by researchers at Johns Hopkins University. It applies the ModernBERT architecture to multilingual pre-training over roughly 1,800 languages, with a large shared vocabulary (the Gemma 2 tokenizer, about 256k entries) so that non-Latin scripts are tokenised into meaningful pieces rather than byte fragments. It is published in small and base sizes.

For classification, a multilingual encoder matters because a model can only reason about text its tokenizer can represent. An English BPE vocabulary shreds Hindi or Korean into near-meaningless units, and accuracy collapses.

mmBERT in Laya

The laya-multilingual checkpoint (bundled as the multilingual subfolder of convaiinnovations/laya) uses mmBERT-base: 22 layers, 256k vocabulary, 322M parameters in total with the decision head. It defaults to a 1,024-token context window with head_max_len 256, and the encoder supports up to 8k.

Measured on the model card's shared benchmark (T4 GPU):

• XNLI outside English: 0.731 vs 0.521 for the English checkpoint.
• MASSIVE intent outside English: 0.451 vs 0.306.
• Usable languages (over 3x random): 45 of 51 with routing, against 23 of 51 for English alone.
• It is faster: 32.8 ms for one question, 72.3 ms for ten, versus 39.5 ms and 158.6 ms.

It is slightly weaker on English (MASSIVE 0.657 vs 0.783), which is why Laya's router sends English Latin text to the ModernBERT checkpoint and everything else to mmBERT. See mmBERT and multilingual classification.