Agent tool routing and model-tier selection with a decision model

Agent tool routing is the decision an AI agent makes before each step: which tool to call, which model to send the request to, or whether to hand off. Laya makes that decision as a typed question with a probability in one fast forward pass, so expensive models only run on the steps that need them.

A production agent does not only answer questions. Before each step it decides what to do: call a SQL tool or a search tool, answer directly, send the request to a small model or a frontier one, ask the user a clarifying question, or stop and hand off. Most teams make these decisions with the same large model that does the reasoning, either through function calling or a "router prompt".

That works, but it puts a multi-second, token-billed call in front of every step, including the trivial ones ("hi", "thanks", "what is your refund policy"). It also makes the routing decision hard to audit: the model emits a tool call, but there is no probability attached, so you cannot tell a confident choice from a coin flip, and you cannot set a policy such as "use the cheap tier only when we are sure the request is easy".

Routing decisions have exactly the properties a decision model for agents is designed for: small, fixed answer spaces, high volume, a tight latency budget, and a need for honest uncertainty. The laya package ships a preset for this (router_questions(), with difficulty, domain, needs_tools and is_sensitive), and the example on this page extends it with a tool-selection question.

The comparison that matters is between routing with the model you are trying to avoid calling and routing with something an order of magnitude cheaper.

• Latency. The model card reports 32.8 ms for a single question on the multilingual checkpoint and 39.5 ms on the English checkpoint on one T4 GPU. Five questions in one call took 84.5 ms on the English checkpoint and 40.1 ms on the multilingual one, because all questions share a single forward pass (batching). Network time to the hosted API is extra. For comparison, TypeSafe Jev, a closed decision API with the same wire protocol, has been measured by third parties at 236–276 ms p50.
• Typed output. The router returns one of your labels with a probability for each. A non-autoregressive model does not generate text, so it cannot emit a malformed tool call or a tool that does not exist. It can still pick the wrong tool, which is why the probability matters.
• Calibrated confidence. A router that says "cheap tier, 0.95" and one that says "cheap tier, 0.40" should be treated differently. With calibrated probabilities you can send the second case to the stronger model and keep the savings on the first.
• Cost. Laya Studio bills input tokens (1 credit = 1 input token), and each of the router's five questions reads the step once. See pricing.

The goal is not to replace the reasoning model. It is the System 1 / System 2 split: a fast, calibrated classifier decides what to do; the slow, generative model does it when needed.

The example combines the package preset with one custom question.

• difficulty (score, 4 levels) drives model-tier selection. Each level describes the work required ("several steps", "long multi-step reasoning"), which is observable from the request, rather than an opinion of how "smart" the model must be.
• domain (choice, 6 options) lets you send code to a code-tuned model and chit-chat to the cheapest one.
• tool (choice) names your actual tools. Describe each by what data it reaches ("internal tables: revenue, refunds, usage, orders"), because that is what distinguishes them. Always include none so direct answers are possible.
• needs_tools and is_sensitive (noul) are cross-checks. If tool = none but needs_tools is high, the answers disagree and the step should escalate.

Design rules that matter for routing:

• Keep tool lists short. Options share a prompt budget (192 tokens on the English checkpoint, 256 on multilingual), and the model card shows accuracy falling off on 77-option Banking77. If you have 40 tools, route in two steps: tool family first, then the tool within the family.
• Put context in the state, not the instructions. Pass the latest user message, the turn count and plan tier as JSON. The English checkpoint reads about 320 tokens of state; truncate long histories to the last turn or two.
• Watch noul on English. The card documents noul answers that follow the false:/true: labels rather than the text (issue #156). If needs_tools looks stuck, rephrase it as a two-option choice with neutral keys.

Observability of agent traces

The laya-typed-decisions checkpoint was fine-tuned on four workflows, one of which is agent_trace_observability, with exactly the question ids action, needs_review, outcome, risk and urgency. The model card reports 0.730 accuracy on that workflow. It suits after-the-fact review of a completed agent trace rather than pre-step routing. The router only selects that checkpoint when the ids match exactly and auto task detection is enabled, or when you pass "model": "typed-decisions" explicitly. See typed decisions.

Gate on confidence, not on action.act_probability.

The model card is explicit that act_probability carries no usable signal yet: it reads close to 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions, issue #185). confidence reached an AUROC of 0.77 on the same items. For choice and score questions confidence is one minus normalized entropy, so a split between two tools shows up as low confidence even when one is slightly ahead. For noul questions it is max(p, 1 - p).

Fit temperatures on a few hundred labelled routing decisions before choosing thresholds: the checkpoints ship over-confident, and the card reports mean ECE dropping from 0.466 to 0.081 on the English checkpoint after per-(type, option count) temperature scaling.

A starting policy, to be tuned against your own error costs:

def plan_step(a):
    diff, tool = a["difficulty"], a["tool"]
    sensitive = a["is_sensitive"]["noul"] >= 0.5
    needs_tools = a["needs_tools"]["noul"] >= 0.5
    # act_probability is logged but not used until issue #185 is fixed
    if tool["confidence"] < 0.4 or (tool["choice"] == "none" and needs_tools):
        return {"tier": "frontier", "tool": None, "reason": "uncertain tool choice"}
    tier = "small" if diff["score"] < 1.5 and diff["confidence"] >= 0.5 else "frontier"
    if sensitive:
        tier = "frontier"
    return {"tier": tier, "tool": None if tool["choice"] == "none" else tool["choice"]}

The asymmetry is the point. Wrongly sending a hard request to a small model costs a bad answer; wrongly sending an easy request to a frontier model costs money. Set the "small tier" threshold strictly and let uncertain steps fall through to the stronger path. Track the fraction of steps that fall through; that is your real saving. See act/escalate routing.

Call POST https://api.laya.studio/v1/systemone with the step's state and questions. Omit model to let Laya route between the English and multilingual checkpoints automatically.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer lsk_live_..." \
  -H "Content-Type: application/json" \
  -d '{"state": {"request": "Which region grew refunds fastest last quarter?"},
       "questions": {"tool": {"type": "choice", "instructions": "Which tool should the agent call first?",
         "criteria": ["sql_warehouse", "web_search", "ticket_lookup", "none"]}}}'

import os, requests

LAYA = "https://api.laya.studio/v1/systemone"
HEADERS = {"Authorization": "Bearer " + os.environ["LAYA_API_KEY"]}

def route(request_text: str, questions: dict) -> dict:
    r = requests.post(LAYA, headers=HEADERS, timeout=3,
                      json={"state": {"request": request_text}, "questions": questions})
    r.raise_for_status()
    body = r.json()
    return {"answers": body["answers"], "routing": body.get("routing")}

type Answer = { type: string; choice?: string; score?: number; noul?: number;
  confidence: number; action: { act_probability: number } };

export async function routeStep(request: string, questions: object): Promise<Record<string, Answer>> {
  const res = await fetch("https://api.laya.studio/v1/systemone", {
    method: "POST",
    headers: { Authorization: "Bearer " + process.env.LAYA_API_KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ state: { request }, questions }),
    signal: AbortSignal.timeout(3000),
  });
  if (!res.ok) throw new Error("laya " + res.status);
  return (await res.json()).answers;
}

Set a short timeout and a fallback: if the router call fails, take the conservative path (the stronger model). Log the full answer object per step so you can later label routing decisions and refit thresholds. Get an API key or read the docs.

• It routes; it does not reason. Laya can say a request looks like "data_analysis, moderate, needs sql_warehouse". It cannot write the SQL or verify the result.
• Zero-shot accuracy on your tools is unknown until you measure it. The model card reports the base checkpoints near chance on the typed-decisions benchmark zero-shot (0.362 vs a 0.461 majority baseline); the 0.766 figure belongs to the fine-tuned checkpoint on its own four workflows. Collect a labelled set of real agent steps before trusting the router.
• Difficulty is a score question, and score is the weakest primitive (the card cites 0.372 on SST-5). Use difficulty to pick a tier, with a conservative default, not to make irreversible decisions.
• Large tool catalogues degrade accuracy because options share a fixed token budget. Route hierarchically above roughly 20 tools.
• Short context. Roughly 320 tokens of state on the English checkpoint and 768 on the multilingual and typed-decisions checkpoints. Summaries of long conversations must be done elsewhere.
• The typed-decisions checkpoint is narrow. It was fine-tuned on synthetic workflows; do not send it unrelated schemas.