Tokenizer

A tokenizer converts raw text into a sequence of integer token ids from a fixed vocabulary, and back. Modern transformers use subword schemes such as byte-pair encoding (BPE), WordPiece or SentencePiece/Unigram: frequent words become single tokens, rare words are split into pieces, and anything else falls back to characters or bytes.

The tokenizer decides two things that matter in production:

1. Length. Limits such as a 512-token context window are counted in tokens, not words or characters. English prose averages somewhat more than one token per word; languages under-represented in the vocabulary can take several times more.
2. What the model can read. If a script is poorly covered, text is broken into fragments that carry little meaning, and no amount of fine-tuning on top recovers it.

Tokenizers in Laya

Laya's two backbones have very different vocabularies. The English checkpoint uses ModernBERT's English BPE (about 50k entries). The multilingual checkpoint uses mmBERT's 256k-entry vocabulary built for many scripts. The consequence is measured on the model card: on non-Latin scripts the English checkpoint collapses (Khmer 0.000 accuracy at 0.952 confidence; MASSIVE Hindi 0.100 with 20 options, where random is 0.050).

That is why Laya routes by script detection before the forward pass rather than trusting confidence to catch the problem.

Tokens also set Laya's budgets: instructions and options share head_max_len (192 English, 256 multilingual), and each option's text is truncated to 48 tokens. The API's usage.input_tokens counts real tokens across all questions in the call. See language routing.