Encoder vs decoder models: which architecture should make your decisions?

Encoder models, such as BERT and ModernBERT, read the whole input in both directions at once and turn it into a representation, which suits classification, retrieval and scoring. Decoder models, such as the GPT family, predict the next token from left to right and generate text. For labelling, an encoder answers in one pass; a decoder must generate its answer.

All modern language models descend from the transformer of Vaswani et al. (2017), which had two halves: an encoder that builds a representation of the input and a decoder that generates output one token at a time. Later work split them apart.

• Encoder-only models (BERT, RoBERTa, DeBERTa, ModernBERT, mmBERT) keep only the encoder. Every token attends to every other token in both directions. They are pretrained with masked language modelling: hide some tokens, predict them from the context on both sides.
• Decoder-only models (the GPT family and most current LLMs) keep only the decoder. Each token attends only to tokens before it (causal attention). They are pretrained to predict the next token, which makes them natural text generators.
• Encoder-decoder models (T5, BART) keep both. The encoder reads the input bidirectionally; the decoder generates an output sequence while attending to it. They are common for translation and summarisation.

A classifier needs to understand the whole input before committing to an answer. In an encoder, the representation of every token already includes information from the entire sequence, left and right. A classification head can read one or more of those vectors and produce a distribution over labels in a single forward pass.

A decoder can classify too, but it has to do it by generating. The usual pattern is to prompt for a label and read the first generated token (or several). Because attention is causal, the model's representation of the input can only use context that came before each position, and the label is produced after the model has read the input. Decoders compensate with scale and with instruction tuning, which is why a large LLM can be a good zero-shot classifier. But the mechanics are indirect: the answer is a string that has to be parsed, and the "probability" is a token probability that depends on how the label is spelled.

The practical difference is in how work scales.

Encoder: one forward pass over the input. Cost grows with input length and model size. There is no output length. Laya's model card reports 39.5 ms for one question on a T4 GPU with a 421M-parameter encoder, and 158.6 ms for ten questions batched.

Decoder: one forward pass over the input (the prefill), then one additional forward pass per generated token. Cost grows with input length, model size and output length. A label of three tokens costs at least three decode steps; a label with a short justification costs dozens. Hosted LLM latency also includes queueing and network overhead that a small self-hosted encoder avoids.

encoder latency  ≈ prefill(input)
decoder latency  ≈ prefill(input) + n_output_tokens × decode_step

For a single ticket this may not matter. For an agent that makes several decisions per turn, or a pipeline classifying millions of emails, it dominates. See latency budgets for agents and the cost of using an LLM as a classifier.

An honest comparison has to say where encoders lose.

• World knowledge and reasoning. A multi-billion-parameter LLM knows far more than a 400M-parameter encoder and can reason through a multi-step policy. If the decision genuinely requires reasoning (reading a contract and applying a clause), a small encoder will not substitute for it.
• Very large label spaces. Laya scores options at marker tokens that share a fixed budget, and its accuracy falls off with dozens of options: 0.425 on 77-label Banking77 against 0.870 published for Jev. An LLM that can read a long label list, or a fine-tuned classifier with a fixed head, does better here.
• Zero-shot on unfamiliar schemas. The model card reports that Laya's base checkpoints are near chance on the multi-field typed-decisions benchmark zero-shot (0.362 vs 0.461 majority class). Instruction-tuned LLMs are generally stronger zero-shot on complex, unfamiliar tasks.
• Explanations. If you need a written rationale, you need a generator.

The two are complementary. A common production pattern is a fast encoder for the high-volume, well-defined decisions, with escalation to an LLM or a human when the encoder's calibrated confidence is low. See act and escalate routing.

Laya is an encoder-only decision model. The English checkpoint uses ModernBERT-large (395M parameters, fully fine-tuned) and the multilingual one uses mmBERT-base. On top sits a decision head trained from scratch: two transformer layers, an option-marker scorer and an act/escalate head, for 421M parameters in total on the English checkpoint.

The input is built so that the question, the options and the state are all in one sequence:

[CLS] <type> question: <instructions> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP]

Because attention is bidirectional, each option's [MASK] marker can attend to the whole state and to the other options, and the state tokens can attend to the options. The scorer reads one logit per marker. This is what lets the label set change per request without retraining, something a classic fine-tuned BERT with a fixed classification head cannot do. The details are in option-marker scoring.

Every question in a request is batched into the same forward pass, and nothing is generated. That is what "non-autoregressive" means here; see non-autoregressive models.

Laya Studio runs the Laya encoders as a hosted API. A request looks like any other /v1/systemone call; the difference from an LLM call is that the response has no generated text and output_tokens is always zero.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"request": "Write a SQL query that returns the top 10 customers by revenue last quarter."},
    "questions": {
      "domain": {
        "type": "choice",
        "instructions": "What domain does request belong to?",
        "criteria": {
          "code": "software engineering, programming, debugging",
          "math_or_logic": "mathematics, logic puzzles, proofs",
          "writing": "creative writing, essays, emails",
          "factual_lookup": "facts, definitions, trivia",
          "data_analysis": "statistics, SQL, data manipulation, metrics",
          "chitchat": "casual conversation, greetings"
        }
      },
      "needs_tools": {
        "type": "noul",
        "instructions": "Does answering request require external tools, search or private data?"
      }
    }
  }'

The answers come back as typed fields (choice, probabilities, confidence, noul) with a routing block naming the checkpoint that answered. Two questions read the state twice, billed per input token. Create a key or read the API docs.