Script detection

Script detection determines which writing system a text uses: Latin, Cyrillic, Greek, Arabic, Devanagari, Hangul, Han, Thai and so on. It is different from language identification. Many languages share a script (English, French and Vietnamese are all Latin), and a few languages are written in more than one. Script detection is also much easier: every character's Unicode code point falls in a known block, so counting letters per block gives an exact answer.

Why Laya routes on script first

The English Laya checkpoint uses ModernBERT-large with an English BPE tokenizer. On text in a script that tokenizer was not built for, the model does not degrade gracefully, it collapses. The maintainers' MASSIVE results at 20 options (random is 0.050): Hindi 0.100, Korean 0.103, Swahili 0.103, Tamil 0.113. Worse, it stays confident while wrong: ECE 0.855 on Hindi, and 0.000 accuracy at 0.952 confidence on Khmer. Confidence gating cannot catch this, so the input has to go to the multilingual checkpoint before the forward pass.

How it is implemented

Laya's Router uses a dependency-free, pure-Python detector that counts letters across about two dozen Unicode ranges (Greek, Cyrillic, Armenian, Hebrew, Arabic, the Indic scripts, Thai, Lao, Tibetan, Myanmar, Georgian, Ethiopic, Khmer, Hangul, Kana and Han). The model card reports it runs in under 0.5 ms. Any non-Latin script sends the request to the multilingual checkpoint; since 0.3.7, scripts the detector has no range for also no longer fall through to English.

For Latin-script text the Router then applies a separate, best-effort stopword and diacritic heuristic to decide whether the text is English. That part is a guess; the script part is exact. If you already know the language, pass it explicitly.

See language routing and the language routing guide.