Support ticket triage with a calibrated decision model

Automated support ticket triage means a model reads each incoming ticket and decides who should handle it and how fast, before a person opens it. Laya answers intent, urgency, frustration, refund and churn-risk questions in one call, each with a confidence score, so confident tickets route themselves and uncertain ones go to an agent.

A support queue is not one classification problem, it is five or six of them stacked on every ticket. Before an agent writes a single word, someone has to decide which team owns the ticket, whether it is urgent, whether the customer is angry enough to need a senior person, whether a refund is being requested and whether the account is at risk of churning. In most helpdesks those decisions are made by the first human who opens the ticket, which means the most expensive resource in the building spends its first minutes doing sorting work.

The usual automation paths each have a cost. Keyword rules ("refund", "cancel", "urgent") are cheap but brittle: "I don't want a refund, I want it to work" matches the wrong rule, and a polite customer threatening to leave never says "cancel". A fine-tuned classifier per field works well but needs labelled data and a retraining cycle for every new category. Sending each ticket to a large language model works on day one, but you pay generation latency and per-token cost on every ticket, you have to parse free text back into fields, and you get no reliable signal for when the answer is a guess.

What triage actually needs is a fast function from ticket text to a fixed set of typed fields, each with a probability you can threshold. That is the shape of a decision model. Laya takes a state (the ticket, as text or JSON) and a dictionary of typed questions, and returns a typed answer per question with a probability distribution and a confidence score. It never generates text, so the output is always one of the labels you defined.

Latency. Laya is non-autoregressive: every option is scored at its own option marker in one forward pass of a bidirectional encoder. The model card reports 39.5 ms for one question on the English checkpoint and 32.8 ms on the multilingual checkpoint on a single T4 GPU, and all questions in a call are answered together (10 questions in 72.3 ms on the multilingual checkpoint). Over the hosted API you add network time, but you are still far below the budget of a generative model that has to emit tokens and close a JSON object. For comparison, the card cites third-party measurements of TypeSafe Jev at 236 to 276 ms p50.

Cost. Laya Studio bills the input tokens the model reads (1 credit = 1 input token), 30% below Jev's list price, and nothing for output because it generates none. Each question reads the ticket once, so five triage questions cost about five times the ticket's tokens. See pricing for credit packs and the 5 free runs.

Calibrated confidence. Laya is trained with RLCD, reinforcement learning against strictly proper scoring rules, so the reward is maximised only by reporting honest probabilities. That gives you a number worth thresholding, which is the whole point of triage automation: you want to auto-route what the model is sure about and escalate the rest.

Nothing to hallucinate. The answer space is the set of criteria you send. The model cannot invent a "billing_dispute" team that does not exist or return a malformed field. It can still be wrong, which is why the confidence number matters, but it cannot be wrong in a shape your code did not anticipate. See hallucination-free decisions and the broader Laya vs LLM classifiers comparison.

The example on this page is the triage_questions() preset that ships in the open-source laya package, with one change: a criteria description on churn_risk. Each field maps to one of the three question types:

• intent is a choice question. Six labels, each with a one-line description. Descriptions matter: the model reads refund: money returned or a duplicate charge reversed as text next to the ticket, so write them the way you would brief a new agent. Always include an other bucket so a ticket that fits nothing has somewhere honest to go.
• frustration is a score question. Levels are ordered, index 0 first. The answer includes score, the expected level (sum of level index times probability), so a ticket split between "clearly annoyed" and "very angry" reads as roughly 2.5 rather than snapping to one bucket.
• is_urgent, refund_requested and churn_risk are noul questions, yes/no with a probability. criteria is optional; when omitted the options render as "no, the statement does not hold" and "yes, the statement holds".

Practical rules that hold up in production:

1. One decision per question. Do not ask "is this an urgent refund?". Ask urgency and refund separately and combine them in code.
2. Keep choice lists short. All options share a fixed head budget (head_max_len, 192 tokens on the English checkpoint), so 6 to 12 well-described labels work far better than 60 terse ones. If your taxonomy is large, route coarse first, then ask a second, narrower question.
3. Reference the field. Instructions such as "in message" point the model at the part of a JSON state that matters.
4. Watch noul label-following. The model card documents that on the English checkpoint a noul question can follow its false/true labels rather than the state (issue #156). If a noul answer looks stuck, re-ask it as a two-option choice with neutral keys, for example {"A": "yes, the customer asks for money back", "B": "no refund is requested"}.

If your helpdesk matches the shape of the customer_service typed-decisions workflow, whose question ids are exactly action, category, churn_risk, needs_human and urgency, you can also send "model": "typed-decisions" to use the checkpoint fine-tuned on that workflow (0.764 accuracy on its customer-service split, per the model card). Validate it against your own labels first: it was trained on synthetic data with its own phrasing.

Every answer carries two numbers: confidence and action.act_probability. Be precise about what each means.

• For choice and score questions, confidence is one minus the normalised entropy of the option distribution: 1 - H(p) / log(k). It is not the top probability. A six-way intent split 0.70 / 0.10 / 0.05 / 0.05 / 0.05 / 0.05 has a top probability of 0.70 but a confidence of roughly 0.45, because the leftover mass is spread out.
• For noul questions, confidence is max(p, 1 - p), so a noul of 0.08 has confidence 0.92.
• action.act_probability is the output of a separate act/escalate head. The model card states plainly that it carries no usable signal yet (issue #185): it reads close to 1.0 for almost every input, and its raw logits ran against correctness (AUROC 0.30 on 396 labelled decisions) while confidence reached 0.77 on the same items.

So gate on confidence. Log act_probability, and keep it as a secondary condition in your policy so you can switch it on when a future checkpoint makes it meaningful. A reasonable starting policy:

These are starting points, not recommendations for your data. The card is explicit that the checkpoints ship over-confident: refitting one temperature per question type and option count moved mean ECE from 0.466 to 0.081 on the English checkpoint. Label 300 to 500 recent tickets, run them through the API, plot a reliability diagram and choose thresholds that give the precision your team can live with. Route everything below threshold to a human; that is human-in-the-loop done on purpose rather than by default.

The hosted endpoint speaks the same /v1/systemone wire protocol as TypeSafe Jev. Send the ticket as state and the questions as questions; omit model and the router picks the English or multilingual checkpoint from the text.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer lsk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"subject": "Charged twice for September",
              "message": "Two identical charges this month. Please reverse one before Friday."},
    "questions": {
      "intent": {"type": "choice", "instructions": "What does the customer want in `message`?",
                 "criteria": {"refund": "money returned or a duplicate charge reversed",
                              "technical_help": "a bug, outage or integration problem",
                              "other": "none of the other options fits"}},
      "is_urgent": {"type": "noul", "instructions": "Does `message` communicate time pressure or a deadline?"}
    }
  }'

Python, with the escalation policy from the previous section:

import os
import requests

API = "https://api.laya.studio/v1/systemone"
HEADERS = {"Authorization": f"Bearer {os.environ['LAYA_API_KEY']}"}

def triage(ticket: dict, questions: dict) -> dict:
    r = requests.post(API, headers=HEADERS,
                      json={"state": ticket, "questions": questions}, timeout=5)
    r.raise_for_status()
    a = r.json()["answers"]
    intent = a["intent"]
    return {
        "queue": intent["choice"] if intent["confidence"] >= 0.55 else "general",
        "urgent": a["is_urgent"]["noul"] >= 0.8,
        "senior": a["frustration"]["score"] >= 2.3,
        "notify_owner": a["churn_risk"]["noul"] >= 0.7,
        "act_probability": intent["action"]["act_probability"],  # log it, do not gate on it yet
    }

TypeScript, for a webhook handler:

type Answer = {
  type: 'choice' | 'score' | 'noul';
  choice?: string;
  score?: number;
  noul?: number;
  probabilities?: Record<string, number>;
  confidence: number;
  action: { act_probability: number };
};

export async function triage(ticket: unknown, questions: Record<string, unknown>) {
  const res = await fetch('https://api.laya.studio/v1/systemone', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.LAYA_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ state: ticket, questions }),
  });
  if (!res.ok) throw new Error(`laya ${res.status}: ${await res.text()}`);
  const { answers } = (await res.json()) as { answers: Record<string, Answer> };
  return answers;
}

A malformed question returns a 422 naming the question and the problem. Get a key at /signup and see the full request and response reference in the docs.

• Zero-shot is a starting point, not a guarantee. On the model card's typed-decisions benchmark the base checkpoints score near chance zero-shot (0.362 English against a 0.461 majority-class baseline); the 0.766 figure belongs to the checkpoint fine-tuned on that benchmark. Measure on your own tickets before you automate anything.
• Context is finite. The English checkpoint reads 512 tokens per question, of which roughly 320 are left for the state after the question and options. Long threads get truncated, so send the latest customer message and a short summary rather than the full history.
• Score is the weakest primitive. Ordinal questions scored 0.372 on SST-5 in the card's benchmarks. Use frustration as a routing hint, not as a metric you report upward.
• Non-English tickets need the multilingual checkpoint. The English checkpoint collapses on non-Latin scripts while staying confident. Leave model unset so the router detects the script, or see multilingual intake.
• act_probability is not an escalation signal yet. Gate on confidence until that changes.