Proper scoring rule

A scoring rule assigns a number to a probabilistic forecast once the outcome is known. It is proper if reporting your true belief maximises your expected score, and strictly proper if the true belief is the unique maximiser. Under a strictly proper rule there is no gain from hedging toward 0.5 or exaggerating toward 0 or 1.

Classic strictly proper rules are the log score, the spherical score, the quadratic rule (whose negative form is the Brier score) and, for ordered outcomes, the ranked probability score. Accuracy is not proper: it only rewards the argmax, so it gives a model no reason to get the probabilities right. Gneiting and Raftery (2007) is the standard reference.

Why Laya is trained on one

Laya's training method, RLCD, treats the model as a policy that reports a distribution and pays it a reward computed by a strictly proper rule. The reward in the package's proper_reward function is:

• log score (with log-probability floored at -9.21, about ln 0.0001), plus
• 0.5 times the spherical score, minus
• 1.0 times the ranked probability score, applied only to score questions, whose levels are ordered.

A positive combination of strictly proper rules is still strictly proper, so the expected reward is maximised only by honest probabilities. That is the theoretical basis for Laya's calibration claims.

It is not a guarantee in practice: finite data, a finite network and exploration noise still leave the checkpoints over-confident until temperature scaling is applied. Read more in proper scoring rules.