Throughput

Throughput measures volume: how many requests, items or tokens a system completes per second. It is distinct from latency, which measures how long one request takes. The two trade off: larger batches usually increase throughput while increasing the time each request waits.

For generative models throughput is often quoted in output tokens per second. For a decision model that produces no output tokens, the natural unit is decisions (questions) per second.

What drives throughput

• Batch size: more items per forward pass amortises fixed overhead. See batching.
• Sequence length: attention cost grows with tokens per sequence, so long states lower throughput.
• Model size: fewer parameters means less compute per token.
• Hardware and precision: GPUs with fp16 or bf16 mixed precision are much faster than fp32 on CPU. See GPU inference.

Laya throughput

The model card reports 103–332 questions per second batched on a single Tesla T4. The spread reflects the two checkpoints: at 50 questions per call, the multilingual checkpoint (mmBERT-base, 322M parameters) takes 337 ms, about 6.8 ms per question, while the English checkpoint (ModernBERT-large, 421M) takes 771 ms. The multilingual checkpoint is roughly 2.2x faster despite a longer default context, because its encoder is smaller.

Because Laya is non-autoregressive, throughput does not depend on answer length: a 20-option choice and a yes/no question cost one marker-scored forward pass each. It does depend on how many option tokens and state tokens each question carries.

Hosted vs self-hosted

On Laya Studio you pay per input token (1 credit = 1 input token); see pricing for plan limits. If you self-host the Apache-2.0 weights with laya-serve, throughput is whatever your hardware delivers, and the T4 numbers above are a reasonable baseline. The LLM-as-classifier cost guide compares throughput economics with generative models.