Banking77

Banking77 is an intent classification dataset released by PolyAI (Casanueva et al., 2020). It contains 13,083 online-banking customer queries labelled with 77 fine-grained intents, such as card_arrival, card_delivery_estimate, lost_or_stolen_card and compromised_card. Many intents are near neighbours, so the dataset tests fine discrimination more than general understanding.

Laya's weakest reported benchmark

The model card is open that Banking77 is where Laya does worst:

The cause is a fixed token budget. Laya places every option of a question, each with its own option marker, inside a head segment of head_max_len tokens: 192 on the English checkpoint, 256 on the multilingual one. With 77 options at 256 tokens, each label gets about (256 − 16) // 77, roughly 3–4 tokens. Labels like card_arrival and card_delivery_estimate become hard to tell apart. Jev supports up to 255 options and is currently the better fit for very large label sets out of the box.

Workarounds

• Coarse-to-fine: ask a choice among a handful of groups (cards, transfers, top-ups, account), then a second choice among the intents in the chosen group. Each call stays in the range where Laya is strong.
• Embedding shortlist: embed the text and the labels, keep the top 20 or so, and ask Laya to choose among those. The Laya package ships an opt-in helper for this.
• Self-hosting: raise head_max_len (and max_len) so each label keeps enough tokens.

See intent classification, the intent detection use case and Laya vs Jev.