Masked language model (MLM)

A masked language model (MLM) is trained by corrupting a fraction of input tokens, most often replacing them with a special [MASK] token, and asking the model to reconstruct them. Because the missing token can depend on words both before and after it, the objective requires bidirectional attention, and it is the standard pre-training recipe for encoder models such as BERT, RoBERTa, ModernBERT and mmBERT.

MLM pre-training produces contextual representations that transfer well to classification, extraction and retrieval with a small amount of task-specific training. Its limits are the flip side: an MLM is not a fluent left-to-right generator, and on its own it does not know about your labels.

A related trick, used in prompt-based classification (PET, and "cloze" formulations), is to phrase a task so the answer is a masked word and read the model's prediction at that position.

How Laya uses the MLM heritage

Laya borrows the [MASK] position but not the vocabulary prediction. In each question's sequence, every option is preceded by a [MASK] token, and the decision head reads the hidden state at that position, the option marker. Instead of predicting a word, a small scorer turns that vector into one logit for the option, and a softmax over the question's markers gives the answer distribution.

Pre-training taught the encoder that a [MASK] position should gather evidence from the whole context; Laya's fine-tuning with RLCD teaches it to gather evidence about whether this option fits the state and the question. The runtime strips any literal mask token that appears in your instructions, options or state, so user text cannot inject extra markers.

See option-marker scoring.