Brier score

The Brier score measures the accuracy of probabilistic predictions as a mean squared error. For a binary event with predicted probability p and outcome y (0 or 1), the score is (p − y)². For a K-class prediction it is the sum of squared differences between the predicted probability vector and the one-hot outcome, averaged over examples. Lower is better; 0 is perfect.

Why it is useful

The Brier score is a proper scoring rule: in expectation it is minimised only by reporting your true belief. Unlike accuracy it rewards getting the probabilities right, and unlike expected calibration error it cannot be gamed by a model that always predicts the base rate. It decomposes into calibration (reliability), resolution and uncertainty terms, so it captures both "are the numbers honest" and "does the model discriminate".

Compared with the log score, the Brier score is bounded and penalises a confident mistake far less harshly. Many evaluation reports show both.

Brier scores for Laya

On the typed-decisions benchmark (400 cases, 2,000 decisions across four workflows), the model card reports:

The fine-tuned checkpoint's Brier score is the strongest evidence that its probabilities are both sharp and honest on that task. The base checkpoints' higher Brier reflects that they are near chance on that benchmark zero-shot.

Laya is not trained on the Brier score directly; its RLCD reward uses the log, spherical and ranked probability scores. See proper scoring rules for how these rules relate.