ModernBERT

ModernBERT is an encoder-only transformer family released in December 2024 by Answer.AI and LightOn as a modernised BERT. It keeps the bidirectional masked-language-model recipe but updates the architecture and data: rotary positional embeddings, alternating local and global attention, unpadding and FlashAttention-friendly kernels, GeGLU layers, a native context of 8,192 tokens and pre-training on about two trillion tokens including code. It comes in base and large sizes.

The practical result is an encoder that is faster and longer-context than classic BERT/RoBERTa while being at least as strong on classification and retrieval benchmarks, which made it a natural backbone for new task heads.

ModernBERT in Laya

The root checkpoint, convaiinnovations/laya, uses ModernBERT-large: 395M parameters in the backbone, fully fine-tuned, plus a decision head trained from scratch for 421M total. The typed-decisions checkpoint uses the same backbone. Configuration details that affect what you can send:

• The English checkpoint runs at 512 tokens per question, of which up to 192 (head_max_len) go to the question and options, leaving roughly 320 for the state. The typed-decisions checkpoint defaults to 1,024.
• Its tokenizer is an English BPE vocabulary (about 50k entries), which is why it cannot read non-Latin scripts and why language routing sends such text to the mmBERT checkpoint.
• The runtime disables ModernBERT's automatic torch.compile because it is a loss at Laya's small batch sizes.

Measured single-question latency on a T4 is 39.5 ms. See ModernBERT and encoder vs decoder models.