p50 / p95 latency

p50 (the median) is the response time that half of requests beat. p95 is the time that 95% of requests beat, so 1 in 20 requests is slower. p99 extends the same idea further into the tail.

Percentiles are preferred over the mean because latency distributions are skewed. A handful of slow requests (cold starts, garbage collection, queueing behind a large batch, network retries) can pull the mean well above what a typical user sees, while hiding how bad the worst cases are.

Why the tail matters for agents

An agent step that makes several sequential calls experiences roughly the sum of their latencies, and a user-facing workflow with many steps will hit each dependency's tail often. If one step calls five services in sequence, the chance that at least one lands in its own slowest 5% is about 23%. Budget with p95 or p99, not p50. See latency budgets for agents.

Laya numbers and how to read them

The Laya model card reports single-question latency of 32.8 ms (multilingual) and 39.5 ms (English) on a T4 with checkpoints preloaded; these are model-side measurements, not percentiles of a public API. The card's reference comparison, TypeSafe Jev, has been independently measured at 236–276 ms p50 by AbdelStark and nibzard.

Factors that widen Laya's tail in practice:

• more questions per call (10 questions: 72.3 ms multilingual, 158.6 ms English)
• long states near the context window limit
• a cold checkpoint load (seconds) if the serving process has not preloaded it
• network distance to api.laya.studio

Measure p50 and p95 from your own infrastructure with your own question sets before committing to a timeout. The inference latency entry has the full benchmark table.