Non-autoregressive models: how AI can decide in a single pass

A non-autoregressive model produces its whole output in parallel, in a single forward pass, instead of generating one token at a time with each token waiting on the one before. For decisions, that means scoring every allowed answer at once and returning a probability for each. No text is generated, so there is nothing to parse.

Compared with a chat-style LLM, this changes three things:

• Speed. There is no decoding loop, so latency depends on the input and the number of questions, not on how long the answer is.
• Cost. Nothing is generated, so there are no output tokens. Laya reports output_tokens: 0 on every call.
• Format. The answer is always one of the options you supplied, never malformed JSON or an invented label.

The sections below explain the mechanics, the latency numbers and the limits.

A decoder-only language model (GPT-style, Claude, Llama and most chat models) is autoregressive. It models the probability of a sequence as a product of next-token probabilities:

P(y₁, …, yₙ | x) = ∏ P(yₜ | x, y₁, …, yₜ₋₁)

At inference time that means a loop. Read the prompt (the "prefill"), sample token 1, append it, run the model again to get token 2, and so on until a stop token. Prefill can be parallelised across the prompt, but decoding cannot: each step depends on the previous output. Caching (the KV cache) makes each step cheaper, but the number of sequential steps still equals the number of output tokens.

For free-form text that is the right design. For a classification answer such as {"department": "billing", "confidence": 0.9}, it is overhead. The model spends a dozen or more sequential steps spelling out braces, keys and quotes, each one a chance to produce malformed output, and the "confidence" it writes is a string it generated, not a quantity it computed.

See encoder vs decoder models for the architectural background.

A non-autoregressive model produces all parts of its output in parallel, conditioned on the input but not on its own earlier outputs. The term became common in machine translation. Gu et al. (2017), "Non-Autoregressive Neural Machine Translation", proposed generating all target tokens at once to cut decoding latency, at some cost in quality, because the output tokens are predicted independently of each other.

For decisions the independence problem mostly goes away. A classification answer is not a sequence whose tokens must agree with each other. It is a distribution over a small, known set of options. Once the options are fixed, "generating" the answer means computing one score per option and normalising. A single pass over the input can do that.

That is the design of both System 1 decision APIs discussed on this site:

• Laya (open-source, Apache-2.0, by Convai Innovations) describes itself as a "non-autoregressive System 1 decision model". Its model card says "every question in a call is answered in one single forward pass", and the source code confirms it: all questions are collated into one batch and run through the encoder together.
• TypeSafe Jev is described by its maker as using "a new model architecture, parallel sampler" with parallel sampling that "generates all outputs in a single query". TypeSafe has not publicly documented Jev's architecture, parameter count or backbone beyond that.

Laya's architecture is small and inspectable, so it makes a good worked example. From the model card and common.py in the laya Python package (v0.3.7):

Backbone. A bidirectional encoder. The English checkpoint uses ModernBERT-large (395M parameters, fully fine-tuned); the multilingual checkpoint uses mmBERT-base (307M, 22 layers, 256k-token vocabulary). Bidirectional means every token attends to every other token, both left and right. That is what lets one pass see the question, all options and the state together. See ModernBERT and mmBERT.

Input layout. Each question becomes one sequence:

[CLS] <type> question: <instructions> [SEP] [MASK] opt0 [MASK] opt1 ... [MASK] optK [SEP] <state> [SEP]

Every option is preceded by its own [MASK] token. For a noul question the two options are rendered as false: … and true: …. For a score question they become level 0: …, level 1: …. A choice option renders as key: description.

Decision head. On top of the encoder sit 2 transformer layers, a learned embedding for the question type (choice, score or noul), and a small scorer network. The hidden state at each [MASK] position goes through the scorer to produce one logit per option. A softmax over that question's options gives the probability distribution. A separate act/escalate head reads the pooled state, but the model card says it "carries no usable signal yet", so do not use it.

Batching. Each question is a row in the batch. A call with ten questions runs one batched forward pass of ten rows, not ten sequential calls.

This design is called option-marker scoring. Its main practical property is that the answer space is defined at request time: "new schemas need no retraining". You can change your label set in the next request.

Output. The runtime divides each question's logits by a fitted temperature for its (type, option-count) bucket (2, 3-5, 6-10, 11+), clamped to [0.5, 5.0], then returns:

{
  "type": "choice",
  "choice": "billing",
  "probabilities": {"billing": 0.91, "technical": 0.06, "sales": 0.03},
  "confidence": 0.67,
  "action": {"act_probability": 1.0}
}

confidence is one minus the normalised Shannon entropy of the distribution, 1 − H(p)/log k. A score answer's score is the expected level, Σ i · pᵢ, so it is a float such as 1.84 on a 0–2 scale. A noul answer returns P(true) directly.

Because there is no decode loop, latency is set by input length and batch size, not by the number of output tokens. The Laya model card reports these figures on a Tesla T4, measured in-process with byte-identical questions:

The card summarises this as "103–332 questions/sec batched on a single T4". Two details are worth noticing:

1. The smaller multilingual checkpoint is faster, despite its 256k vocabulary. The card explains that "the 768-dim / 22-layer encoder is cheaper per token than 1024-dim / 28-layer, and the gap widens with batch size."
2. Per-question cost falls with batch size. On the multilingual checkpoint, 50 questions take about ten times as long as one, not fifty times. Batching questions about the same state is the main lever for throughput.

On CPU the picture changes. The card lists 193–464 ms per request for a preloaded router on CPU, and the package warns that CPU inference is "roughly 10-15x slower". Laya's BENCHMARKS.md also records a case where untuned torch threading on a busy host gave a 9,396 ms p50, which fell to 783 ms after pinning inter-op threads to 1. Non-autoregressive does not make a model fast on any hardware. It removes the decode loop, and the forward pass still has to be served well.

For comparison, the independent benchmarks cited in the Laya card measured TypeSafe Jev at 236–276 ms p50 end to end (a hosted API called over the internet), and the DMB benchmark found Jev's latency "flat from 2 to 255 options". The Laya and Jev figures are not like-for-like: Laya's are model-side GPU timings and Jev's include network. A hosted Laya Studio call also includes network time.

A Laya response always reports "usage": {"input_tokens": N, "output_tokens": 0}. That is not rounding: the model emits no tokens. input_tokens is the number of non-padding tokens across all the question sequences in the batch.

Two consequences follow:

• Cost scales with input, not output. TypeSafe prices Jev the same way, charging input tokens at "$0.042 / MTok" and describing output tokens as "FREE (too cheap to meter)". Jev's own API examples do show nonzero output_tokens values, so the two services report usage differently even though the field names match. Laya Studio bills the same unit, input tokens only (1 credit = 1 input token), at a list price 30% below Jev's; see /pricing.
• Asking more questions about the same input is cheap in time. Each question re-encodes the state in its own row, so input tokens grow with question count, but the wall-clock cost grows much more slowly because the rows run in parallel on the GPU.

Removing the decode loop has costs. These are the ones that show up in practice.

1. A fixed option budget

All options for a question share one slice of the sequence, head_max_len: 192 tokens on the English checkpoint, 256 on the multilingual and typed-decisions checkpoints. Each option is truncated to 48 tokens, and when the budget is tight every option is cut further. The model card works the numbers for Banking77: a 77-option question gets roughly (256 − 16) // 77 ≈ 3 tokens per label. At that point many labels are indistinguishable, and accuracy falls to 0.425 against Jev's published 0.870. The BENCHMARKS.md file notes that both base checkpoints score exactly 0.425, "which is what you would expect from a budget ceiling rather than a capability gap".

Mitigations, from the model card and package:

• Keep choice questions under about 20 options.
• Raise head_max_len (for example to 512) and max_len when self-hosting.
• Split large label sets into a coarse-to-fine hierarchy, or use the package's opt-in embedding shortlist (predict_shortlist, default k = 20), which ranks labels by cosine similarity and asks only about the top k.

Jev documents a limit of 255 options per Choice. The DMB benchmark confirmed 100% accuracy on its synthetic code-word task up to 255 options and a 400 Too many choices. error at 256.

2. A fixed state budget

The state gets what is left after the question header and options. On the English checkpoint that is about 320 tokens, on the 1,024-token checkpoints about 768. Longer inputs are truncated. If your tickets have long quoted email threads, clean them first; the package ships an email cleaner that strips quoted replies and signatures. Jev's documented context is much larger: "64k tokens per request; 32k tokens for state plus the longest question".

3. No reasoning steps

An autoregressive model can, in effect, compute on intermediate tokens (chain of thought). A single pass cannot. Multi-hop questions, arithmetic, counting and date comparison are weak spots. TypeSafe lists the same categories as known Jev failure modes. Do those parts in code.

4. Label-wording sensitivity

Because each option is scored from its own text, the wording and order of options matter. Laya's BENCHMARKS.md reports that on 20-option MASSIVE intent, 15% of English-checkpoint answers change when the options are permuted, against 13% measured for Jev. The model card also documents a case where a noul answer follows its false:/true: labels instead of the input. The workaround is to ask a two-option choice with neutral keys (A/B) and your yes/no wording in the descriptions.

5. It cannot write

A non-autoregressive decision model has no way to produce text it was not given. That is what makes it safe to parse, and it is also why you still need a generative model for replies, summaries and extraction of free-form values.

Single-pass scoring is not new. Several older approaches also classify in one pass, and it helps to know how a decision model differs:

The comparison pages go into each alternative: Laya vs fine-tuned BERT, Laya vs zero-shot NLI and Laya vs embeddings + kNN.

The distinguishing features of the decision-model approach are the typed question schema (so one call can ask a choice, a score and several true/false questions), options as input (so the label set can change per request), and training against proper scoring rules (so the probabilities are meant to be honest; see RLCD and proper scoring rules).

The open-source Laya package includes a Jev-compatible HTTP server, so the same request works against a local server, Laya Studio, or Jev's endpoint.

Hosted (Laya Studio):

import os, requests

resp = requests.post(
    "https://api.laya.studio/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['LAYA_API_KEY']}"},
    json={
        "state": "The dashboard has returned 502 errors since 09:10 and our checkout is down.",
        "questions": {
            "category": {"type": "choice", "instructions": "What is this about?",
                         "criteria": ["billing", "outage", "feature_request", "other"]},
            "severity": {"type": "score", "instructions": "How severe is the impact?",
                         "criteria": ["cosmetic", "degraded", "blocking"]},
            "outage": {"type": "noul", "instructions": "Does the message report a service outage?"},
        },
    },
    timeout=5,
)
out = resp.json()
print(out["answers"]["category"]["choice"], out["usage"])  # usage.output_tokens == 0

Self-hosted (open-source package):

pip install "laya[serve]"
LAYA_DEVICE=cuda LAYA_PRELOAD=1 LAYA_API_KEY=change-me laya-serve   # listens on 0.0.0.0:8000

Without LAYA_API_KEY, the self-hosted server accepts unauthenticated requests on all interfaces, so set it or bind to localhost.

In-process:

from laya import Router

router = Router(preload=True, device="cuda")
result = router.predict(state, questions)   # one forward pass on the routed checkpoint
print(result["routing"]["reason"])

Whichever you choose, the mechanics are the same: one pass, one distribution per question, no generated tokens. Laya Studio is an independent hosted service powered by the open-source Laya model, and is not affiliated with Convai Innovations or TypeSafe. Sign up for an API key with 5 free runs, or read the docs.