Temperature scaling: the one-parameter fix for over-confidence

Temperature scaling is a post-hoc calibration method that divides a classifier's logits by a single fitted number, the temperature T, before the softmax. A temperature above 1 softens the probabilities, correcting over-confidence. It leaves the ranking of options, and so every predicted label, unchanged, and is fitted on held-out labelled data.

Guo et al. (2017) observed that modern neural networks, despite being more accurate than older ones, are systematically over-confident: when they say 90%, they are right less than 90% of the time. Larger models, batch normalisation and training long past the point of lowest validation loss all make it worse. Desai and Durrett (2020) found pre-trained transformers better calibrated than earlier models in-domain, but still miscalibrated out of domain.

Laya is no exception. Its model card states that the checkpoints "ship over-confident", with mean expected calibration error of 0.466 for the English checkpoint and 0.314 for the multilingual one before correction.

A classifier produces logits z_1 … z_k and turns them into probabilities with a softmax. Temperature scaling inserts one scalar T > 0:

p_i = exp(z_i / T) / Σ_j exp(z_j / T)

• T = 1 leaves the model unchanged.
• T > 1 softens the distribution: the top probability falls, the others rise. This corrects over-confidence.
• T < 1 sharpens it. This corrects under-confidence.

Because dividing every logit by the same positive number preserves their order, the argmax never changes. Accuracy is untouched; only the probabilities move.

T is fitted on a held-out labelled set by minimising negative log-likelihood (the log score, a strictly proper rule; see proper scoring rules). With one parameter, a few hundred examples are enough, and overfitting is hard.

Temperature scaling's limitation is that one number cannot fix miscalibration that differs across classes or input types. That is the reason Laya uses several temperatures rather than one.

Laya's checkpoints ship fitted temperatures, and the runtime selects one per answer by bucket: the question type and the number of options.

bucket = "<type>:<size>"   where size ∈ { "2", "3-5", "6-10", "11+" }
e.g. "choice:3-5", "score:3-5", "noul:2", "choice:11+"

If a bucket has no fitted temperature, the runtime falls back to one temperature per question type. Bucketing makes sense because a two-way noul and a twelve-way choice have very different entropy ranges and failure modes.

The card reports the effect of refitting one temperature per (question type, option count): mean ECE falls from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual one. On the Laya vs Jev comparison, Laya's 0.081 ECE is explicitly "post-temperature"; the base checkpoint's raw ECE on typed-decisions is 0.213 against Jev's 0.144.

The Laya runtime clamps every temperature to [0.5, 5.0], and warns when a checkpoint ships a value outside that range. The source explains why with a real case: the shipped choice:11+ temperature was 0.1006, which multiplies logits by about ten. As the code comment puts it, "a 0.24 top probability is published as 0.99, so a caller gating on confidence is told a coin flip is a certainty."

A fitted temperature that low usually means the fit was dominated by a quirk of the calibration set, not by genuine under-confidence. Clamping trades a little theoretical optimality for protection against that failure. The practical consequence: probabilities for 11+-option questions are clamped rather than fitted, and the runtime tells you to "treat confidence from the affected entries as uncalibrated." Combined with Laya's known weakness on large option sets, questions with more than ten options deserve extra scepticism.

A few errors come up repeatedly when teams calibrate classifiers:

• Fitting on the data you evaluate on. Always split: fit T on one half, report ECE and log loss on the other. With one parameter the optimism is small, but per-bucket fits multiply the parameters.
• Mixing populations. A single temperature fitted on pooled English and Portuguese traffic can be wrong for both. Fit per checkpoint (routing.model), and per language if volumes allow.
• Judging calibration by accuracy. Temperature scaling cannot change accuracy, so an unchanged accuracy is expected, not a failure. Judge it by log loss, Brier score and ECE.
• Forgetting to refit. New options, reworded instructions or a shift in traffic all move the logits. Refit when the schema changes.
• Sparse buckets. A bucket with thirty examples will give a noisy temperature. Pool it with its neighbour or keep the shipped value.

The shipped temperatures were fitted on the model authors' data. Yours differs, so fit a second temperature on your own labelled sample. You do not need logits: Laya Studio returns probabilities, and applying an extra temperature T to log-probabilities is equivalent to raising each probability to the power 1/T and renormalising:

p'_i = p_i^(1/T) / Σ_j p_j^(1/T)

This composes exactly with the temperature the server already applied. A minimal fitter, one T per bucket:

import math

def rescale(probs, T):
    w = [max(p, 1e-4) ** (1.0 / T) for p in probs]   # responses are rounded to 4 decimals
    s = sum(w)
    return [x / s for x in w]

def nll(items, T):
    # items: list of (prob_list, true_index)
    return -sum(math.log(rescale(p, T)[y]) for p, y in items) / len(items)

def fit_temperature(items):
    grid = [0.5 + 0.05 * i for i in range(91)]        # 0.5 .. 5.0, same range as the runtime clamp
    return min(grid, key=lambda T: nll(items, T))

Collect the items by sending your labelled set through the API:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"message": "Can I switch my annual plan to monthly billing?"},
    "questions": {
      "intent": {
        "type": "choice",
        "instructions": "What does the customer want in message?",
        "criteria": {"refund": "money returned", "technical_help": "a bug or outage", "billing_question": "invoice, plan or payment method", "cancellation": "cancel or downgrade", "other": "none of these"}
      }
    }
  }'

Then store answers.intent.probabilities in option order with the true label index, fit T on half the data, and check ECE on the other half. Recompute confidence after rescaling (1 − H(p')/log k), since the server's value reflects the unscaled distribution. Each question costs one credit; see /docs and /signup.