Calibration

Calibration is the agreement between the probability a model reports and how often it is actually right. A perfectly calibrated classifier that says "0.8" on a thousand inputs is correct on about eight hundred of them. Calibration is separate from accuracy: a model can be accurate but over-confident (it says 0.99 when it is right 90% of the time), or poorly accurate but honest about it.

Calibration matters whenever a probability drives an action. If you auto-resolve every ticket above 0.9 and escalate the rest, the threshold only means something if 0.9 really means nine in ten. Uncalibrated scores force you to tune thresholds by trial and error for every new question.

It is usually measured with expected calibration error, the Brier score or a reliability diagram, and repaired after training with temperature scaling.

Calibration in Laya

Laya is trained with RLCD: its reward is a strictly proper scoring rule, so reporting honest probabilities is the reward-maximising strategy. That is a training objective, not a guarantee. The model card is explicit that the checkpoints ship over-confident: refitting one temperature per (question type, option count) moves mean ECE from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual one. It also documents a failure calibration cannot fix: on non-Latin scripts the English checkpoint is confidently wrong (Khmer: 0.000 accuracy at 0.952 confidence), which is why language routing runs before the forward pass.

Practical advice: measure calibration on a few hundred labelled examples from your own traffic before you gate automation on confidence. See calibrated probabilities for the full walkthrough.