Expected calibration error: measuring whether 80% means 80%

Expected calibration error (ECE) measures how far a classifier's confidence is from its actual accuracy. Predictions are grouped into confidence bins, commonly fifteen equal-width bins; in each bin the gap between average confidence and accuracy is weighted by the bin's share of predictions, and the gaps are summed. Zero means perfectly calibrated on that data.

A classifier is calibrated if, among all the predictions it makes with confidence c, the fraction that are correct is c. Of the tickets routed with 80% confidence, about 80% should be routed correctly. Calibration is separate from accuracy: a model can be highly accurate and badly calibrated (always 99% sure, right 90% of the time) or poorly accurate and well calibrated (60% sure, right 60% of the time).

Calibration matters whenever a probability is used as a probability: to set a threshold, to combine with a cost, to decide when to escalate. See calibrated probabilities.

ECE, popularised by Naeini et al. (2015) and Guo et al. (2017), estimates the average gap between confidence and accuracy by binning predictions:

1. Split [0, 1] into M bins B_1 … B_M by confidence.
2. For each bin: acc(B_m)  = fraction of predictions in the bin that are correct
                 conf(B_m) = mean confidence of predictions in the bin
3. ECE = Σ_m |B_m| / n · |acc(B_m) − conf(B_m)|

It is a weighted average of the per-bin gaps, weighted by how many predictions fall in each bin. Zero means perfectly calibrated on this sample; the maximum is 1. A reliability diagram plots acc(B_m) against conf(B_m); a calibrated model lies on the diagonal, and an over-confident one sits below it.

Laya's package implements exactly this in ece_score: 15 equal-width bins, with the first bin closed at 0 so that zero-confidence predictions are counted.

Ten predictions, three bins for readability:

ECE = 0.2 · 0.05 + 0.3 · 0.37 + 0.5 · 0.14 = 0.010 + 0.111 + 0.070 = 0.191

The model is over-confident in the middle and top bins: it claims 0.70 and 0.94 but delivers 0.33 and 0.80. A temperature above 1 would pull those confidences down; see temperature scaling.

Reading a reliability diagram

A reliability diagram makes the same information visual. Plot one point per bin, with mean confidence on the x-axis and accuracy on the y-axis, and draw the diagonal. Points below the diagonal mean over-confidence; points above it mean under-confidence. Adding a histogram of how many predictions fall in each bin shows which gaps matter: a large gap in a bin with 1% of predictions contributes little to ECE, while a small gap in the bin holding most predictions can dominate it. For a decision system, the bins near your operating threshold deserve the closest look, because that is where calibration errors turn into wrong automated actions.

ECE is useful but has well-documented weaknesses (Nixon et al., 2019; Kumar et al., 2019):

• It depends on binning. The number of bins and whether they are equal-width or equal-mass change the value. Compare ECE only when computed the same way. Equal-width bins with most predictions near 1.0 leave most bins nearly empty.
• It is biased with small samples. With few predictions per bin, noise inflates gaps. Report the sample size, and prefer several hundred items or more.
• It is not a proper scoring rule. A model that always predicts the base rate can have near-zero ECE and be useless. Always report ECE alongside a proper score such as Brier or log loss; see proper scoring rules.
• Top-label ECE ignores the rest of the distribution. The standard version uses only the top probability. Classwise variants check every class but need more data.
• "Confidence" must be defined. Different systems report different confidence signals. Compute ECE on the quantity you will actually threshold.
• Averages hide subgroups. A good overall ECE can conceal a language or question type that is badly miscalibrated.

The model card reports several ECE numbers. Reading them carefully:

Two honest conclusions. First, Laya's probabilities become well calibrated after temperature fitting, and the card is explicit that the headline 0.081 depends on it. Second, calibration depends on reaching the right checkpoint: the 0.855 figure is why language routing exists.

A note on definitions: Laya's API returns two things you could call confidence: the top probability in probabilities, and a confidence field that is 1 − H(p)/log k for choice and score questions and max(p, 1 − p) for noul. The two have different scales. For comparability with published ECE numbers, compute ECE on the top probability; to evaluate a gate, compute accuracy against coverage on whichever field you gate on.

Collect a labelled sample and send each item through the API:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"review": "Arrived late and the box was damaged, but the product itself works fine."},
    "questions": {
      "sentiment": {
        "type": "choice",
        "instructions": "What is the overall sentiment of review?",
        "criteria": {"positive": "mostly satisfied", "mixed": "both praise and complaints", "negative": "mostly dissatisfied"}
      }
    }
  }'

Then compute ECE with the same binning Laya's package uses:

import numpy as np

def ece(conf, correct, bins=15):
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    edges = np.linspace(0, 1, bins + 1)
    total = 0.0
    for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
        sel = ((conf >= lo) if i == 0 else (conf > lo)) & (conf <= hi)
        if sel.any():
            total += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())
    return total

# for each labelled item:
#   probs = resp["answers"]["sentiment"]["probabilities"]
#   top = max(probs, key=probs.get)
#   conf.append(probs[top]); correct.append(top == gold_label)

Report ECE with the Brier score and the sample size, per question and per language. If ECE is high, fit a temperature and re-measure on held-out data. Billing is per input token. Sign up for 5 free runs and see the docs.