Decoder model

A decoder model (decoder-only transformer) applies causal, left-to-right attention: token t can attend to tokens before it but not after. Trained on next-token prediction, it becomes an autoregressive generator. GPT, Llama, Mistral, Qwen and Claude-style models are decoder-only.

The design is ideal for generation and in-context learning, and it scales very well. For classification it has two structural disadvantages compared with an encoder:

1. One-directional context. The representation of an early token never sees later tokens, so a decoder classifying a text must rely on its final positions or generate an answer after reading everything.
2. Generation to answer. To return a label, a decoder usually writes it out, which means output tokens, parsing, and the possibility of labels you did not offer.

Decoders can be adapted with a classification head on the last hidden state, but most production uses of LLMs as classifiers still prompt them and parse text.

Decoders and Laya

Laya deliberately does not use a decoder. Its backbone is ModernBERT (English) or mmBERT (multilingual), both bidirectional encoders, with a decision head that scores every option at its option marker in one pass. That is why Laya reports zero output tokens and returns a probability distribution rather than a generated label.

The honest trade: a large decoder LLM can handle instructions and reasoning Laya cannot. Use a decoder for System 2 work and a decision model for the fast, fixed-schema calls. See encoder vs decoder models and Laya vs LLM classifiers.