Zero-shot classification

Zero-shot classification means applying a model to a label set it has not been trained on for that task, with no labelled examples, only the label names or descriptions. Common approaches are prompting an LLM, reformulating the task as natural language inference ("this text is about {label}": entailed or not), or comparing embeddings of the text and the labels.

Zero-shot is valuable for prototyping and for long-tail tasks with no training data. Its accuracy is typically below a model fine-tuned on the exact task, and its probabilities are often poorly calibrated for the new labels.

Zero-shot in Laya

Laya's architecture supports zero-shot use by construction: options are text at option markers, so any new schema works without a new output layer. On public benchmarks where the task resembles its training, results are strong: AG News (4 labels) 0.950 and DAIR Emotion (6 labels) 0.595, each ahead of Jev's published numbers.

The model card is equally clear about where zero-shot falls short:

• Base checkpoints are near chance on the typed-decisions benchmark zero-shot: 0.362 English and 0.352 multilingual, against 0.318 random and 0.461 majority-class. The 0.766 figure belongs to the checkpoint fine-tuned on that benchmark's training split. In the card's words, Laya is "a fast base to specialise, not a zero-shot decision engine."
• High-cardinality label sets (Banking77, 77 labels: 0.425) and ordinal scores (SST-5: 0.372) are weak spots.

Practical guidance: prototype zero-shot, measure on a few hundred labelled examples, and fine-tune if accuracy is not there. See zero-shot vs fine-tuned and Laya vs zero-shot NLI.