Glossary
Encoder model
An encoder model reads a whole text at once, looking both forwards and backwards, to understand it rather than to write new text; BERT-family models are encoders.
Swiss-hosted inference. Nothing you send is ever stored.Swiss data residency
What Encoder model means
An encoder model is a transformer stack in which every token can attend to every other token in the input, left and right. Its output is one contextual vector per input token, not generated text. BERT, RoBERTa, DeBERTa, ModernBERT and mmBERT are encoders, usually pre-trained as masked language models.
Because attention is bidirectional, an encoder sees the whole input at once. That is the right inductive bias for understanding tasks: classification, extraction, retrieval embeddings, natural language inference. Encoders are also much smaller than frontier LLMs for comparable understanding accuracy on these tasks, and their cost is one forward pass regardless of the answer.
The trade-off is that an encoder alone cannot generate open-ended text. It needs a task head on top.
Encoders in Laya
Laya is an encoder plus a decision head. The English checkpoint uses ModernBERT-large (395M parameters, 421M with the head, 512-token context); the multilingual one uses mmBERT-base (322M total, 1,024-token context by default). The head adds a question-type embedding, two extra transformer layers, an option-marker scorer and an act head.
The input is one sequence per question: the question type and instructions, then each option introduced by a [MASK] token, then the state. Bidirectional attention lets each option's marker see both the question and the full state before it is scored, which is what makes request-time options work without retraining.
See encoder vs decoder models and ModernBERT.
Related terms
How Encoder model connects to the rest of the vocabulary.