Reward shaping

Reward shaping is the practice of designing or augmenting the reward in reinforcement learning so a policy learns faster or learns the behaviour you actually want. Classic shaping adds intermediate rewards on the way to a sparse goal. The danger is well known: a shaped reward can change the optimal policy, and the agent learns to exploit the shaping term instead of solving the task ("reward hacking"). Potential-based shaping (Ng, Harada and Russell, 1999) is a form proven not to change the optimal policy.

Shaping with proper scoring rules

For a model that reports probabilities, there is a clean way to shape without hacking: build the reward only from strictly proper scoring rules. Each strictly proper rule is maximised in expectation by the true distribution, and a positive-weighted sum of proper rules is still proper. So you can combine rules for their different behaviour without introducing an incentive to misreport.

Laya's shaped reward

RLCD uses exactly this construction:

• Log score: sharp, heavily penalises putting near-zero probability on the true answer. Its log term is floored at −9.21 so a single catastrophic example cannot dominate an update.
• Spherical score, weight 0.5: bounded, gentler on mistakes, which stabilises learning.
• Ranked probability score, weight 1.0, subtracted only for ordinal score questions: it compares cumulative distributions, so predicting level 2 when the truth is level 3 costs less than predicting level 0.

The RPS term encodes a property the log and spherical scores ignore, ordinal distance, without rewarding dishonest probabilities. Note one caveat: the log floor makes the combined reward very slightly less than strictly proper for extreme reports, a deliberate trade for stable training.

See proper scoring rules and the RLCD explainer.