Aspect-based sentiment analysis with calibrated confidence

Aspect-based sentiment analysis finds how a customer feels about each part of their experience, such as the product, billing or support, instead of giving the whole review one score. Laya answers one typed question per aspect in a single API call and attaches a probability to every label, so you can act on each topic separately.

Most feedback pipelines reduce a review to one number: positive, negative, or a 1–5 rating. The example review on this page is a three-star rating that contains praise for the product, a billing error, a slow support response and a renewal risk. A single polarity label throws away every one of those facts, and a product team reading an average of such labels learns almost nothing.

Aspect-based sentiment asks separate questions per aspect: how does the customer feel about the product, about billing, about support? It is the version of sentiment analysis that actually drives decisions, but it multiplies the number of classifications per review. With an LLM, that means either one long structured-output call per review (slow, expensive, and fragile to parse) or several calls. With a fine-tuned classifier, it means training and maintaining one head per aspect.

The other half of the problem is aggregation. Dashboards average predictions across thousands of reviews. If the classifier's probabilities are not calibrated, a systematic over-confidence on one class silently skews the aggregate. Averaging calibrated probabilities is one of the few ways to get a sentiment trend you can defend.

• Several aspects, one pass. Laya answers every question in a request in a single forward pass. The model card reports 84.5 ms for five questions on the English checkpoint and 40.1 ms on the multilingual one on a T4 GPU, against 39.5 ms and 32.8 ms for one question. Network time to the hosted API is extra.
• Typed answers. Each aspect returns exactly one of your labels plus a probability per label. There is no JSON for an LLM to half-produce and nothing for it to invent. The label can still be wrong; it cannot be outside your schema.
• Calibration that survives aggregation. Laya is trained with RLCD: exploration noise on the logits and a reward from a strictly proper scoring rule, under which reporting honest probabilities maximizes expected reward. After per-bucket temperature scaling the model card reports ECE of 0.081 on the English checkpoint. Summing calibrated probabilities gives an unbiased count estimate; summing argmax labels does not.
• Emotion as well as polarity. On DAIR Emotion (six labels: sadness, joy, love, anger, fear, surprise) the model card reports 0.595 accuracy for routed Laya against a published 0.480 for TypeSafe Jev, and notes Jev assigned zero probability to the true label on 16% of examples. Six-way emotion is hard for everyone; 0.595 is a useful signal, not ground truth.
• Cost. 1 credit = 1 input token. Each question reads the review once, so the five-question example costs about five times the review's tokens. See pricing.

The example uses only choice questions, and that is deliberate.

Polarity uses neutral keys. The obvious design would be a noul such as "is the review positive?". The model card documents that on the English checkpoint, noul can follow its false:/true: option labels instead of the text and return a confident "no" for clearly positive input (issue #156). Its recommended workaround is a choice question with neutral keys (A, B) and the yes/no wording in the descriptions. The example applies that to every polarity and yes/no question.

Every aspect has a "not discussed" option. Without it, a review that never mentions billing is forced into positive or negative. That is the most common source of garbage in aspect sentiment.

Mixed is an explicit label. For overall sentiment, "mixed: clear positives and clear negatives" is a real category, not a failure to decide. If you leave it out, mixed reviews show up as low-confidence positives or negatives, which is harder to report.

Avoid a five-level score for sentiment. It is tempting to model "very negative … very positive" as a score question. The model card flags score as the weakest primitive and cites 0.372 on SST-5, the five-class sentiment benchmark. If you need intensity, use three well-separated levels or a choice question, and check the confusion matrix.

Emotion as a list of labels. Criteria can be a plain list when the label names are self-explanatory. The six labels above match the DAIR Emotion set, which is the only emotion benchmark the model card reports.

Keep aspect questions to the aspects your team acts on. Each one re-reads the review, so each adds its input tokens to the bill.

Sentiment pipelines use confidence in two ways: to decide which individual reviews need a human, and to weight aggregates.

Per-review gating. Use confidence. For choice questions it is one minus the normalized entropy of the probabilities, so it falls when mass is spread across labels. Do not gate on action.act_probability: the model card and issue #185 report that it reads close to 1.0 for almost every input and that its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions, against 0.77 for confidence). Log it so you can switch it on when a checkpoint fixes it.

A starting policy to tune on your own labels:

Aggregation. For dashboards, sum probabilities rather than counting argmax labels: the expected number of negative billing mentions is the sum of probabilities.B across reviews. This is only honest if the probabilities are calibrated, so fit temperatures first. The model card reports the English checkpoint's mean ECE falling from 0.466 to 0.081 after one temperature per (question type, option count), and the multilingual checkpoint from 0.314 to 0.106. Check the result with a reliability diagram.

def negative_billing_estimate(results):
    # expected count of reviews negative about billing
    return sum(r["billing_sentiment"]["probabilities"]["B"] for r in results)

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"state": {"text": "Faster dashboard, but I was double charged and support took four days."},
       "questions": {"billing_sentiment": {"type": "choice",
         "instructions": "How does text describe billing, pricing or charges?",
         "criteria": {"A": "positive about billing or price", "B": "negative about billing or price",
                      "C": "billing and price are not discussed"}}}}'

import os, requests

def score_review(text: str, questions: dict) -> dict:
    r = requests.post(
        "https://api.laya.studio/v1/systemone",
        headers={"Authorization": "Bearer " + os.environ["LAYA_API_KEY"]},
        json={"state": {"text": text}, "questions": questions},
        timeout=5,
    )
    r.raise_for_status()
    return r.json()["answers"]

export async function scoreReview(text: string, questions: object) {
  const res = await fetch("https://api.laya.studio/v1/systemone", {
    method: "POST",
    headers: { Authorization: "Bearer " + process.env.LAYA_API_KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ state: { text }, questions }),
  });
  if (!res.ok) throw new Error("laya " + res.status);
  const { answers, routing } = await res.json();
  return { answers, checkpoint: routing?.model };
}

Reviews in other languages need no extra code: leave model unset and the router sends non-English text to the multilingual checkpoint. For backfills of historical reviews, send requests concurrently rather than one at a time; each request is billed per question either way. Sign up for a key or see the docs.

• Fine-grained intensity is weak. Five-level sentiment is where the model card reports its weakest number (SST-5, 0.372). Prefer coarse, well-described categories.
• Emotion is hard. 0.595 on six-label DAIR Emotion beats the published Jev figure but still means roughly four in ten top labels are wrong. Use emotion for trends and routing hints, not for decisions about individuals.
• Sarcasm and domain slang are classic failure modes for any classifier. Label a sample of your own reviews that contain them and measure.
• Zero-shot needs checking. The base checkpoints were not fine-tuned on your aspects. Evaluate on a labelled set before automating, and fine-tune if accuracy is not good enough.
• Context length. The English checkpoint reads about 320 tokens of state; long reviews or survey answers are truncated. The multilingual checkpoint reads about 768.
• Language. The English checkpoint can be confidently wrong on non-Latin scripts; the model card shows 0.000 accuracy at 0.952 confidence on Khmer. Let the router pick the checkpoint rather than forcing english.