Ranked probability score (RPS)

The ranked probability score (RPS) is a strictly proper scoring rule for ordered categories such as severity levels or star ratings. Instead of comparing probabilities option by option, it compares cumulative distributions:

RPS = (1 / (K − 1)) · Σ_k (CDF_forecast(k) − CDF_outcome(k))²

Lower is better. The point of using cumulative sums is that distance matters. If the true level is 3 of 4, putting your mass on level 2 costs less than putting it on level 0. A plain log score or Brier score would treat both misses identically, because they ignore order.

RPS originates in weather forecasting (Epstein, 1969) for things like precipitation categories, and it is the natural choice whenever the answer is a rung on a ladder rather than a label in a bag.

RPS in Laya

Laya's score question is exactly an ordered ladder: you pass a list of level descriptions, index 0 first, and the model returns a probability per level plus an expected level. During RLCD training, the reward for score questions subtracts 1.0 × RPS, normalised by K − 1 as above, on top of the log and spherical terms. Choice and noul questions do not get the RPS term, because their options have no order.

In practice this is why a score answer tends to spread probability over adjacent levels when it is unsure, rather than splitting between the two extremes. That spread is also why the returned score (the expectation over levels) can be fractional, for example 1.84 on a 0–2 scale.

The model card is candid that score is still the weakest primitive (SST-5 accuracy 0.372). See choice, score and noul.