Reliability diagram

A reliability diagram is the standard picture of calibration. You sort predictions into bins by their stated confidence, then for each bin plot the average confidence on the x-axis against the fraction that were actually correct on the y-axis.

• A perfectly calibrated model lies on the diagonal: among answers stated at 0.8, 80% are right.
• Points below the diagonal mean over-confidence: the model says 0.9 and is right 70% of the time.
• Points above the diagonal mean under-confidence.

Bar charts of the per-bin gap, plus a histogram of how many predictions fall in each bin, are usually drawn together, because a large gap in a nearly empty bin matters little.

Relationship to ECE

Expected calibration error is the reliability diagram collapsed into one number: the bin-size-weighted average of |confidence − accuracy|. Laya's reference implementation uses 15 equal-width bins. The diagram is more informative than the number, because it shows where the model is miscalibrated. A model can have a modest ECE while being badly over-confident in the 0.95–1.0 bin, which is exactly the bin automation thresholds live in.

Using one with Laya

The model card reports that the checkpoints ship over-confident: mean ECE is 0.466 for the English checkpoint and 0.314 for the multilingual one before refitting, falling to 0.081 and 0.106 after fitting one temperature per (question type, option count). A reliability diagram of your own traffic is how you confirm that the refit worked for your data.

A practical recipe:

1. Collect a few hundred labelled answers per question.
2. Plot the reliability diagram of the raw confidence or top-choice probability.
3. Fit temperature scaling and plot again.
4. Pick escalation thresholds from the calibrated curve.

The calibrated probabilities explainer shows worked examples.