Expected calibration error (ECE)

Expected calibration error (ECE) is the most common single-number summary of calibration. You sort predictions into confidence bins, compute the mean confidence and the actual accuracy in each bin, and take the size-weighted average of the absolute gaps:

ECE = Σ_b (n_b / N) · |accuracy_b − confidence_b|

An ECE of 0 means every bin is perfectly honest; an ECE of 0.2 means the model's confidence is off by twenty percentage points on average.

Caveats

ECE depends on the number of bins and on which confidence you bin (top-1 probability, or something else). It ignores how the probability mass is spread across the non-winning options, and a model can score a low ECE by predicting the base rate for everything. Pair it with a proper score such as the Brier score or log loss, and look at a reliability diagram.

ECE in Laya

The Laya package ships an ece_score function that uses 15 equal-width bins over [0, 1]. The published numbers:

• After per-bucket temperature scaling, mean ECE drops from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual one.
• Against TypeSafe Jev's published 0.246, routed Laya reports 0.081 post-temperature. Before temperature fitting, the raw figure on the typed-decisions benchmark is higher than Jev's (0.213 vs 0.144), and the model card says so.
• The English checkpoint on Hindi text reaches an ECE of 0.855: confidently wrong, which is why routing by script comes first.

When you evaluate Laya on your own data, compute ECE on at least a few hundred labelled answers per question. Details in expected calibration error and Laya vs Jev.