Latency budgets for AI agents

A latency budget is the maximum time an AI agent turn may take before users feel it is slow, split across every step in the turn. Usability guidance puts instant at about 0.1 s and uninterrupted flow at about 1 s. Each serial decision, such as choosing a tool or running a safety check, spends part of that budget.

Users do not experience model latency; they experience waiting. Long-standing usability guidance (Nielsen Norman Group) gives three rough limits: about 0.1 s feels instantaneous, about 1 s keeps the user's flow of thought, and about 10 s is the limit of attention before people switch tasks. Voice is stricter still: studies of conversational turn-taking across languages (Stivers et al., 2009) find typical gaps between speakers of a few hundred milliseconds, so a voice agent that pauses for two seconds feels broken.

An AI agent turn usually contains one large, slow step (an LLM generating a reply or a plan) and several small decisions around it. Those small decisions are where budgets quietly overflow, because each one looks cheap in isolation.

A typical support or operations agent turn, with every decision made by a hosted LLM call:

Five of seven steps are classification. If each is a generative LLM call at several hundred milliseconds, the decisions alone can consume the entire "keeps the flow" budget before the reply has been written.

Three rules of thumb help.

Serial steps add. If step B needs step A's output, their latencies sum. Guardrails usually must run before the rest, and output checks after generation. Minimise the number of serial decision points.

Independent decisions should be batched. Intent, urgency, sentiment and language are all functions of the same input. Asking them in one request instead of four sequential calls turns a sum into roughly a maximum.

Plan for the tail, not the median. Dean and Barroso (2013) showed that when a request fans out to many components, the slowest component dominates. A decision service with a 30 ms median and an occasional 2 s stall will, across enough calls, produce 2 s turns. Budget with p95 or p99 figures, and give every call a timeout with a safe fallback.

A simple budget for a chat agent aiming at about one second:

guardrail + intent + routing (one batched call)   ~50–100 ms
tool call                                          ~100–300 ms
LLM reply (streamed; first token matters)          ~300–600 ms to first token
output check (one batched call)                    ~50–100 ms

These are planning targets, not measurements. Replace them with your own p95 numbers.

Laya answers all questions in a request in one encoder forward pass, with no generated tokens. The model card's measurements on a single Tesla T4:

Batching is where the model shines: ten questions on the multilingual checkpoint cost 7.2 ms each. The card also reports 193 to 464 ms per request on CPU with all checkpoints preloaded, which matters if you self-host without a GPU.

For reference, the card cites independent measurements of TypeSafe Jev at 236 to 276 ms p50 per call (AbdelStark and nibzard benchmarks), and states Laya answers a single question roughly 6 to 8 times faster. Those are model-side numbers on a GPU. A hosted API adds network round-trip time between your servers and the endpoint, which depends on region and connection reuse; measure it from where your agent runs.

Model-side latency is only the steady state. Two self-hosting pitfalls are documented in the model card:

• Cold loads. Building a checkpoint takes seconds. Version 0.3.7 cut CPU load time from about 22 s to about 2 s, but a first request that triggers a load still pays for it.
• Swapping checkpoints. A router that keeps only one model resident reloads on every language switch: a 7.4 s median on CPU and 10.3 s on T4. The default keeps two resident, and preload=True keeps all three so routing costs only detection (under 1 ms).

These are mainly self-hosting concerns: with a hosted endpoint such as Laya Studio, checkpoints are loaded on the server rather than per request. They are worth knowing if you ever move the same workload on-premise.

• One request per input, many questions. The biggest win. It also costs no more credits than separate calls, since each question reads the state once either way.
• Keep states short and relevant. Encoder cost grows with input length. Send the message, not the whole CRM record.
• Run independent checks in parallel with other I/O, for example the output check alongside logging.
• Set timeouts with defaults. If a decision call times out, fall back to the conservative route (escalate, or block for guardrails).
• Reuse connections. Keep-alive and HTTP/2 remove TLS handshakes from every call.
• Escalate only the uncertain tail to a slower LLM, gated on calibrated confidence; see act or escalate.

Use curl's timing output to see what one batched decision call costs from your own infrastructure:

curl -s -o /dev/null \
  -w "connect=%{time_connect}s  ttfb=%{time_starttransfer}s  total=%{time_total}s\n" \
  https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"request": "Refund order 5521 and email the customer a confirmation."},
    "questions": {
      "difficulty": {"type": "score", "instructions": "How hard is request for a language model?", "criteria": ["trivial: a lookup or one-liner", "easy: short answer, no reasoning", "moderate: several steps", "hard: long multi-step reasoning or specialist knowledge"]},
      "needs_tools": {"type": "noul", "instructions": "Does answering request require external tools, search or private data?"},
      "is_sensitive": {"type": "noul", "instructions": "Does request involve money, legal, medical or safety consequences?"},
      "jailbreak": {"type": "noul", "instructions": "Does request try to make an AI assistant ignore its rules, policies or system instructions?"}
    }
  }'

Run it a few hundred times with a reused connection and record p50 and p95. Four questions read the state four times per call, billed per input token. See the docs for client examples, and /signup for a free-tier key.