[MASK] token

The [MASK] token is a reserved vocabulary entry in BERT-style masked language models. During pre-training, some input tokens are replaced by [MASK] and the model learns to predict what was there, using context on both sides. After pre-training, the hidden state at a [MASK] position is effectively a summary of what the surrounding text implies should go there.

Classic fine-tuning ignores [MASK] and classifies from the [CLS] vector instead. Cloze-style prompting (PET and similar) puts a [MASK] in a template and reads the vocabulary prediction at that spot.

How Laya uses [MASK]

Laya places one [MASK] immediately before each answer option:

… [SEP] [MASK] refund: money returned … [MASK] technical_help: a bug … [SEP] state [SEP]

The decision head reads the hidden state at each of these positions, the option markers, and scores it into a single logit. A softmax over a question's markers gives its answer distribution. Because the options are just text in the sequence, you can define new labels at request time without retraining.

Two implementation details worth knowing:

• Any literal mask token that appears in your instructions, options or state is replaced with a space before tokenisation, so user content cannot create spurious markers.
• Markers that would fall beyond the sequence limit are dropped, and the runtime rejects the question with an error if its options no longer fit head_max_len, rather than silently answering over fewer options.

Laya does not use the [MASK] prediction over the vocabulary at all; only the position's hidden state matters. See option-marker scoring.