mmBERT: the multilingual encoder behind Laya's global checkpoint

mmBERT is a massively multilingual encoder-only model from Johns Hopkins University, released in 2025. It uses the ModernBERT architecture, was pretrained on about 3 trillion tokens covering more than 1,800 languages, and is positioned by its authors as a modern successor to XLM-R. Laya's multilingual checkpoint is built on mmBERT-base.

For most of the last five years, the default multilingual encoder was XLM-R (Conneau et al., 2019), a RoBERTa-style model trained on about 100 languages. It was strong, but architecturally it predates everything that ModernBERT later brought to English encoders: rotary position embeddings, efficient attention, unpadding and long context.

mmBERT, described by Marone, Weller, Fleshman, Yang, Lawrie and Van Durme (2025), closes that gap. It is an encoder-only model built on the ModernBERT architecture, pretrained on about 3 trillion tokens of multilingual text covering more than 1,800 languages, and explicitly positioned by its authors as a modern successor to XLM-R.

Three training ideas distinguish mmBERT, per the paper:

• Annealed language learning (ALL). Instead of training on every language from the start, the number of languages grows in stages: about 60 languages, then 110, then all 1,833 in the final decay phase. High-resource languages build a strong shared representation first; low-resource languages are added late, when a small amount of data can still produce a large improvement.
• Inverse temperature sampling. Language sampling starts close to the natural data distribution (dominated by high-resource languages) and moves toward uniform, so low-resource languages get relatively more exposure as training progresses.
• An inverse mask-ratio schedule. The fraction of masked tokens decreases over training, from harder denoising early to finer-grained learning later.

Architecturally it inherits ModernBERT's choices and pairs them with a large multilingual vocabulary. The Laya model card describes the base configuration it uses as 22 layers with a 256k-token vocabulary, and the Laya package notes that the checkpoint uses the Gemma tokenizer family. A 256k vocabulary means text in most scripts is split into sensible subwords rather than bytes or unknown tokens, which is the fundamental thing an English-only tokenizer cannot provide.

One detail explains a result that surprises many people: Laya's multilingual checkpoint is faster than the English one. The model card reports:

Parameter counts are misleading here. A large share of a 256k-vocabulary model's parameters sit in the embedding table, and looking up an embedding costs almost nothing. The compute that dominates latency is in the transformer layers, and a base-size model has narrower layers than ModernBERT-large. The result is about 2.2x throughput on batched questions despite supporting far more languages.

mmBERT is not a free upgrade for English. Laya's two base checkpoints, measured on the same 17,416-question benchmark:

On English, the larger ModernBERT-based checkpoint wins by several points. On other languages, the mmBERT-based checkpoint wins by 15 to 21 points. Six of 51 tested languages remain below the "3x random" usability bar even with mmBERT. This is the classic trade-off Conneau et al. called the "curse of multilinguality": a fixed model capacity spread over more languages gives up some per-language quality.

The multilingual checkpoint is also over-confident as shipped: the card reports mean ECE of 0.314, falling to 0.106 after per-bucket temperature refitting.

Two further cautions apply to any multilingual encoder, mmBERT included:

• Pretraining coverage is not task coverage. mmBERT saw 1,833 languages, but for most of them the data was small and arrived only in the final phase. A language being "in" the model does not mean a downstream classifier will be accurate on it. Measure on labelled examples in each language you serve.
• Benchmarks are uneven across languages. MASSIVE and XNLI cover a few dozen languages between them. For languages outside that set, including many African and South Asian languages, there is little public evidence either way, so plan an evaluation before you rely on automated decisions.

laya-multilingual wraps mmBERT-base with the same decision head as the English checkpoint: a type embedding, two transformer layers, an option-marker scorer and an act head, for 322M parameters total. Differences in configuration:

The larger context and option budget matter: multilingual text often needs more tokens per word than English, and option descriptions written in the user's language consume budget too.

Rather than asking users to choose, Laya's Router picks between the two automatically. Script detection sends any non-Latin text to the multilingual checkpoint; a stopword and diacritic heuristic catches Latin-script languages such as French, Portuguese or Romanian. The combined result is the better of the two columns in each row above: 0.783 on English MASSIVE and 0.451 on the other languages. See language routing.

Automatic routing is right for mixed traffic. Pin "model": "multilingual" when:

• your traffic is predominantly non-English and short messages often lack enough words for the router to identify the language (the router falls back to English on undecided Latin text);
• you mix languages within one state, for example a Spanish customer message inside an English ticket template;
• you need the longer 1,024-token context for long states;
• you value the latency gain more than the few points the English checkpoint gains on English text.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"body": "Mein Konto wurde zweimal belastet. Bitte erstatten Sie den doppelten Betrag."},
    "questions": {
      "team": {
        "type": "choice",
        "instructions": "Which team should handle the email in body?",
        "criteria": {
          "billing": "invoices, payments, refunds",
          "technical": "bugs, outages, integrations",
          "sales": "pricing, demos, new purchases",
          "other": "none of the above"
        }
      },
      "refund_requested": {"type": "noul", "instructions": "Does the sender ask for money back?"}
    }
  }'

You do not need to set model. The German text is Latin script, so the router uses its language heuristic, and the routing.reason will say why it chose the multilingual checkpoint (for example, that the language looks like German rather than English). Note that the option labels and instructions can stay in English; only the state needs to be in the user's language. Two questions read the state twice, billed per input token. Sign up or read the docs.