The real cost of using an LLM as a classifier

The cost of an LLM classifier is more than its token price. Each call pays for the full prompt with the label list, the generated answer, retries when the output fails to parse, and the latency of generation. At volume these costs multiply per decision, which is why one-pass decision models are often cheaper for routine labelling.

An LLM classifier takes an afternoon to build. Write a prompt that lists the labels, paste in the text, ask for JSON, parse the result. No training data, no model hosting, and instruction-tuned models are genuinely good zero-shot classifiers on many tasks (Brown et al., 2020). For a prototype or a low-volume internal tool, that is often the right call.

The trouble starts when the prototype becomes a pipeline. Volume multiplies every per-call cost, latency compounds inside agent loops, and the operational burden of parsing, retrying and re-validating grows. None of these costs show up in the demo.

Every call pays for several things, most of which are independent of how hard the decision is:

Put rough numbers on it with a formula rather than a price list, because prices change:

tokens_per_decision = (system + labels + examples + text) + output × output_price_ratio
monthly_cost        = volume × questions_per_item × tokens_per_decision × price_per_token × (1 + retry_rate)

A worked token count: a 400-token system prompt, 150 tokens of labels, 300 tokens of few-shot examples and a 250-token email is 1,100 input tokens, of which only 23% is the email. Add 30 output tokens for a JSON label. If you ask four separate questions about each email with separate prompts, you send the email four times and pay the fixed overhead four times. At a million emails a month, that is on the order of 4.4 billion input tokens before a single retry.

Prompt caching and batching discounts, where your provider offers them, cut the fixed overhead. Combining questions into one prompt reduces calls but makes each output longer and each parse more fragile.

Generative decoding adds a forward pass per output token after the prompt is processed. For a short label, that is a few steps; for a JSON object with a rationale, dozens. Hosted APIs add network time and queueing.

For context, the Laya model card cites independent measurements of TypeSafe Jev, a hosted decision API, at 236 to 276 ms p50 per call, and reports Laya at 32.8 ms for a single question on a T4 GPU (multilingual checkpoint). Hosted LLM calls have their own latency profile, which varies with model size, output length and provider load; measure yours at p50 and p95. Inside an agent that makes several decisions per turn, those milliseconds add up to seconds; see latency budgets for agents.

Parsing and validation. A generated label is a string. It can be misspelled, capitalised differently, wrapped in prose or replaced by a label that is not in your list. Structured-output features reduce this but do not remove the need to validate. Every validation failure is a retry or a fallback path you have to build and maintain.

Order and phrasing sensitivity. Zheng et al. (2023) showed that LLMs choosing among options have measurable position biases: moving the correct answer to a different slot changes accuracy. Label order and wording become hidden hyperparameters.

Probabilities. Token log-probabilities are not always exposed, depend on how each label tokenises, and verbalised confidence ("confidence: 0.9") is just more generated text. Kadavath et al. (2022) found models can be reasonably calibrated in some formats, but getting usable probabilities is extra engineering. Without them, you cannot threshold.

Model churn. Hosted models are updated and retired. Each change means re-running your evaluation set and possibly re-tuning prompts.

A decision model such as Laya removes several rows of the cost table by construction:

On Laya Studio, billing is per input token: 1 credit = 1 input token, the tokens the model actually reads (at most the model's context per question). Each question reads the state once, so a four-question request costs about four times the state's tokens. See /pricing for current plans and the 5 free runs.

Being honest about the other side:

• Reasoning. If the decision requires reading a policy and applying it step by step, an LLM does something a 421M-parameter encoder cannot.
• Large label spaces. Laya's accuracy falls with dozens of options (0.425 on 77-label Banking77 vs 0.870 published for Jev). An LLM reading a long label list can do better.
• Zero-shot on idiosyncratic tasks. Laya's base checkpoints are near chance on the multi-field typed-decisions benchmark without fine-tuning.
• Explanations. If an auditor needs a rationale, you need text.

The economical architecture is often both: a fast decision model for every item, and an LLM only for the items where the fast model's calibrated confidence is low. See act or escalate.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"from": "ap@contoso.example", "subject": "Overdue: invoice 8812", "body": "Our records show invoice 8812 unpaid for 45 days. Please remit this week to avoid a hold on your account."},
    "questions": {
      "category": {
        "type": "choice",
        "instructions": "Which team should handle the email in body?",
        "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages, integrations", "sales": "pricing, demos, new purchases", "security": "phishing, scams, account compromise", "other": "none of the above"}
      },
      "is_phishing": {
        "type": "noul",
        "instructions": "Is this email a phishing or scam attempt to steal money, credentials, or personal data?",
        "criteria": {"true": "phishing, scam, or fraud", "false": "a legitimate email"}
      },
      "urgency": {"type": "score", "instructions": "How urgent is the request in body?", "criteria": ["no time pressure", "needs attention soon", "blocking issue or hard deadline"]},
      "needs_reply": {"type": "noul", "instructions": "Does the sender expect a reply?"}
    }
  }'

Four questions, one call, billed per input token, no output tokens and no parsing. Compare accuracy on your own labelled sample before switching: the right answer depends on your task. Sign up and see the docs.