MASSIVE

MASSIVE (Multilingual Amazon SLURP for Slot-filling, Intent classification and Virtual-assistant Evaluation) is a dataset released by Amazon in 2022. It localises the English SLURP virtual-assistant corpus into 51 languages, giving about one million utterances labelled with 60 intents across 18 domains (alarms, calendar, music, weather, IoT and so on) plus slot annotations. Because every language shares the same items, it is a strong test of cross-lingual intent classification.

How Laya uses it

The Laya maintainers use MASSIVE intent classification as a headline multilingual benchmark, framed as a 20-option choice question, where random guessing scores 0.050. The model card reports:

and, per language for the English checkpoint alone: Hindi 0.100, Korean 0.103, Swahili 0.103, Tamil 0.113, while Latin-script French (0.487) and Spanish (0.480) held up much better. On Hindi the English checkpoint's ECE was 0.855, meaning it was confident and wrong.

This is the evidence behind Laya's script detection and language routing: with routing, 45 of the 51 languages tested are usable (above 3x random), against 23 of 51 for the English checkpoint alone. MASSIVE was also used to measure the 0.3.7 routing fix for accent-stripped Romance text, where Italian utterances of six or more words routed correctly went from 39% to 80%.

Reading the numbers

These are single-question, zero-shot results on short assistant commands. Longer support tickets or emails carry more context and may behave differently. See multilingual classification and multilingual intake.