Proper scoring rules: rewarding a model for honest probabilities

A proper scoring rule grades a probabilistic forecast against the actual outcome so that reporting your true beliefs earns the best expected score; it is strictly proper when that honest report is the only best choice. Log loss, the Brier score, the spherical score and the ranked probability score are standard examples. Accuracy is not one.

A classifier that returns probabilities is making a forecast: "70% billing, 20% technical, 10% other." Once the true label is known, we want a single number that says how good that forecast was. A scoring rule S(q, y) does exactly that: it takes the reported distribution q and the observed outcome y and returns a score (higher is better, or a loss, lower is better, depending on convention).

Accuracy is the scoring rule most people reach for, and it is a bad one for probabilities. Accuracy only looks at the argmax. A forecast of 0.51 and a forecast of 0.99 on the correct label score identically, and so do 0.49 and 0.01 on the wrong one. A model optimised only for accuracy has no reason to make its probabilities mean anything.

Suppose the true probability of each outcome is p. If you report q, your expected score is:

E_{y ~ p}[ S(q, y) ] = Σ_y p(y) · S(q, y)

A scoring rule is proper if reporting q = p maximises that expectation, and strictly proper if q = p is the unique maximiser. In words: under a strictly proper rule, any distortion of your true belief, whether toward over-confidence or under-confidence, lowers your expected score. Honesty is the optimal strategy.

This is the key result formalised by Gneiting and Raftery (2007). It is why proper scoring rules are the standard for evaluating weather forecasts, and why they are a principled training objective for any model whose probabilities will be used downstream.

Let q be the reported distribution over k outcomes and y the true outcome (or a one-hot vector t).

Log score is the same thing as negative cross-entropy, the usual training loss for classifiers. Its unboundedness is both its strength and its weakness. A forecast that puts probability zero on what happened scores minus infinity, which is the correct verdict on a forecaster who was certain and wrong, but it also means one bad example can dominate an average.

Brier score, introduced for weather forecasting in 1950, is the mean squared error between q and the one-hot outcome. It is bounded, so it is robust to occasional confident mistakes, and it decomposes neatly into calibration and refinement terms.

Spherical score normalises the probability on the true outcome by the Euclidean norm of the forecast. It is bounded and strictly proper, and it gives a different gradient profile than log score, which makes it a useful complement in a mixture.

RPS applies to ordinal outcomes. It compares the cumulative distribution of the forecast with the cumulative distribution of the truth. If the truth is level 1 on a 0 to 3 scale, putting mass on level 2 costs less than putting it on level 3. Log score cannot express that: it only looks at q(y).

Here are four forecasts for a three-way outcome. The truth is option A.

Accuracy cannot distinguish rows 1 and 2, or rows 3 and 4. Both proper rules can. Notice how differently they treat row 4: the log score is roughly seven and a half times worse than row 3, while Brier is only about three times worse. Which behaviour you want depends on how costly confident mistakes are for you.

The model card for Laya gives a real example of why this matters: on DAIR Emotion, Jev "assigned zero probability to the true label on 16% of examples." Under a log score, each of those is an unbounded penalty. Under accuracy, they are ordinary errors. A model trained against a proper rule learns not to do that.

Laya is trained with RLCD (reinforcement learning for calibrated decisions), where the reward for each reported distribution is a mixture of strictly proper rules. From the package source (proper_reward in common.py):

reward = log_score + 0.5 · spherical_score                       (choice and noul)
reward = log_score + 0.5 · spherical_score − 1.0 · RPS           (score questions)

A few implementation details are worth knowing:

• The log term is floored at −9.21, which is log(0.0001). That caps how much a single catastrophic example can contribute to a batch, a practical guard against the unboundedness discussed above. The floor only bites below 0.01% probability on the truth, so it preserves the incentive everywhere that matters.
• Targets can be soft. The reward is written as Σ t_i log q_i, so the target t can be a teacher's distribution rather than a one-hot label. Positive combinations of strictly proper rules remain strictly proper.
• RPS is only added for score questions, normalised by (k − 1), so ordinal questions are rewarded for putting mass near the true level.

Because the combined reward is strictly proper, the expected-reward-maximising policy reports its true belief. The model card phrases it as: "Expected reward is maximised only by reporting honest probabilities." The RL mechanics are covered in RLCD.

A strictly proper objective makes honesty optimal in expectation. It does not make a finite model perfectly calibrated on your data. Three gaps remain, and the model card is candid about them:

1. Over-confidence at shipping time. Refitting one temperature per (question type, option count) moves mean ECE from 0.466 to 0.081 on the English checkpoint and from 0.314 to 0.106 on the multilingual one. That is a large post-hoc correction, and you should do it on your own data. See temperature scaling.
2. Distribution shift. A model that is honest on its training distribution can be confidently wrong elsewhere. The English checkpoint scores 0.000 accuracy at 0.952 confidence on Khmer, which is why Laya routes by language. See language routing.
3. Soft-target fit. On the typed-decisions benchmark, the fine-tuned checkpoint wins on argmax accuracy (0.766 vs 0.727) but Jev achieves higher soft accuracy against the teacher distributions (0.580 vs 0.471).

Measure the result with a proper score such as Brier alongside a calibration metric such as expected calibration error. ECE on its own is not a proper scoring rule and can be gamed.

The practical use of all this is evaluation. Send a labelled sample through Laya Studio and score the returned distributions with a proper rule:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Your March invoice lists the Pro plan twice. Can you correct it?",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": { "billing": "invoices, payments, refunds", "technical": "bugs, outages", "sales": "pricing, new contracts", "other": "everything else" }
      }
    }
  }'

Then, for each labelled item, compute the scores from answers.department.probabilities:

import math

def log_score(probs, truth):
    return math.log(max(probs[truth], 1e-12))

def brier_loss(probs, truth):
    return sum((p - (1.0 if k == truth else 0.0)) ** 2 for k, p in probs.items())

# probs = response["answers"]["department"]["probabilities"]
# accumulate log_score / brier_loss over your labelled set and compare checkpoints or temperatures

Averaged over a few hundred labelled items, these numbers tell you whether the probabilities are good enough to threshold, and whether a temperature fitted on your data improves them. Billing is per input token (1 credit = 1 input token), so a few hundred short items cost very little. Start at /signup and see /docs for batching guidance.