Inference latency

Inference latency is the wall-clock time between submitting an input to a trained model and receiving its prediction. It is usually reported per request and summarised with percentiles such as p50 and p95.

End-to-end latency for a hosted model has several parts: network round trip, queueing, tokenisation, the forward pass itself, post-processing, and serialisation. For generative models the forward pass repeats once per output token, so latency grows with output length. For a non-autoregressive model there is exactly one forward pass regardless of the answer.

Laya's model-side latency

The model card reports these figures on a single Tesla T4 GPU with checkpoints preloaded:

On CPU with checkpoints preloaded, the card reports 193–464 ms per request. A cold checkpoint load costs seconds (median 7.4 s on CPU, 10.3 s on T4), which is why production servers preload. For comparison, TypeSafe Jev has been independently measured at 236–276 ms p50 by third parties.

These are model-side numbers. Calls to Laya Studio's hosted API at https://api.laya.studio/v1/systemone also include network time between your service and ours, so measure from your own region before setting budgets.

Why it matters

In an agent loop, a decision call often sits on the critical path of every step: route the request, check a guardrail, decide whether to escalate. At tens of milliseconds you can ask several questions per step. At hundreds of milliseconds or more each, you start dropping checks to save time. The latency budgets for agents guide works through the arithmetic.