ModernBERT: a modern encoder for fast classification

ModernBERT is an encoder-only transformer released in December 2024 by Answer.AI and LightOn. It brings recent architecture improvements to the BERT design, was trained on 2 trillion tokens of English text and code, reads up to 8,192 tokens natively, and comes in base (149M) and large (395M) sizes under the Apache-2.0 licence.

BERT (Devlin et al., 2018) made encoder-only transformers the default for classification, retrieval and tagging. For years the architecture barely changed while decoder-only LLMs picked up a long list of improvements. ModernBERT, described in Warner et al. (2024), "Smarter, Better, Faster, Longer", applies those improvements back to an encoder.

The authors describe it as a Pareto improvement over older encoders: better downstream quality at the same or lower inference cost. It was trained on 2 trillion tokens of English text and code, with a native context length of 8,192 tokens, and released in two sizes:

Both are released under Apache-2.0, which is why a derivative such as Laya can be released under the same licence.

The main architectural and training changes, as described in the paper:

• Rotary positional embeddings (RoPE) instead of learned absolute positions. RoPE encodes relative position and extends to longer contexts more gracefully.
• GeGLU activations in the feed-forward layers instead of GeLU, following results from decoder models.
• Alternating local and global attention. Most layers use a sliding-window (local) attention; every few layers use full global attention. Long inputs therefore cost much less than full attention in every layer.
• Unpadding. Padding tokens are removed and sequences in a batch are packed, so compute is not wasted on padding. Combined with FlashAttention, this is a large part of the speed gain on variable-length inputs.
• No bias terms in most linear layers and an extra normalisation after embeddings, choices borrowed from recent LLM practice to stabilise training.
• A modern BPE tokenizer with a vocabulary of roughly 50k tokens that handles code well.
• Much more and more varied data: 2T tokens including code, versus BERT's roughly 3.3B-word corpus.

The result is an encoder designed for inference on common GPUs, with a context window long enough for whole emails and tickets.

Good at: English classification, retrieval (single-vector and multi-vector), code-related tasks, and long English documents. Its efficiency makes it a strong default backbone when you need many predictions per second.

Not good at: languages other than English. ModernBERT's training data and tokenizer are English- and code-centric. Text in non-Latin scripts is broken into tokens the model has barely seen. The Laya model card shows how severe this is in practice. The English checkpoint, on 20-option MASSIVE intent classification, scores:

Worse than the accuracy is the confidence: on Khmer the English checkpoint reports 0.952 confidence at 0.000 accuracy, and ECE reaches 0.855 on Hindi. The model does not know it cannot read the input. That is why Laya pairs ModernBERT with an mmBERT-based multilingual checkpoint and a script-detecting router. See mmBERT and language routing.

ModernBERT is also not a generator and not a reasoner. It produces vectors, and what you build on those vectors determines what the system can do.

Two of Laya's three checkpoints use ModernBERT-large as their backbone:

On top of the 395M-parameter encoder, Laya adds a decision head trained from scratch:

• a learned type embedding added to every token, telling the model whether this is a choice, score or noul question;
• two extra transformer layers that refine the encoder's output for the decision task;
• an option-marker scorer, a small MLP that turns the vector at each option's [MASK] position into one logit;
• an act head that reads the pooled vector plus summary features of the answer distribution.

The encoder is fully fine-tuned, not frozen. Training uses RLCD, a reinforcement learning scheme with a strictly proper scoring-rule reward; see RLCD.

Two implementation details from the package source are worth knowing if you self-host. First, Laya disables ModernBERT's automatic torch.compile (reference_compile = False), because compilation is a net loss for the small batches Laya runs and can hang on some platforms. Second, the English checkpoint uses only 512 of ModernBERT's 8,192-token context, split into 192 tokens for the question and options and the remainder for the state. The long-context capability is available in the backbone but not used by default.

Laya's design needs three things from its backbone, and ModernBERT supplies all of them:

1. Bidirectional attention, so each option marker can attend to the full state and to the competing options. See option-marker scoring.
2. A masked-token pretraining objective, so the representation at a [MASK] position is already trained to summarise surrounding context. Laya's option markers reuse that token.
3. Fast inference at small batch sizes, so several questions per call can be answered well inside an agent's latency budget. The card reports 39.5 ms for one question and 158.6 ms for ten on a T4.

The trade-off is language coverage, which is handled by routing rather than by stretching ModernBERT beyond what it was trained for.

Laya Studio routes English text to the ModernBERT-large checkpoint automatically. You can also pin it explicitly with "model": "english":

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "english",
    "state": {"post": "Great write-up, but the benchmark table is missing the batch size."},
    "questions": {
      "toxic": {"type": "noul", "instructions": "Is post toxic: rude, disrespectful or likely to make someone leave the discussion?"},
      "severity": {
        "type": "score",
        "instructions": "How severe is any rule-breaking in post?",
        "criteria": [
          "no rule-breaking: ordinary on-topic post",
          "mild: rude tone or off-topic, no target",
          "clear violation: insults, harassment or spam aimed at someone",
          "severe: threats, hate speech or calls for violence"
        ]
      }
    }
  }'

The routing block in the response records "model": "english" and the reason (explicit model='english' when pinned, English Latin text when auto-routed). Pin the English checkpoint only when you know the traffic is English: forcing it onto other scripts reproduces the collapse shown above. Two questions cost two credits; see /pricing and /docs.