RLCD

RLCD, Reinforcement Learning for Calibrated Decisions, is the training method behind Laya. Its aim is a model whose probabilities can be taken at face value, not just one that picks the right label.

The idea

Treat the model as a policy whose "action" is to report a probability distribution over the options of a question. Score that report against the true outcome with a strictly proper scoring rule. By definition, the expected score of a strictly proper rule is maximised only when the reported distribution equals the true one, so the reward-maximising policy is the honest one. See proper scoring rule.

The mechanics, as published

• Reward: the log score plus 0.5 times the spherical score. For ordinal score questions, the ranked probability score is subtracted (weight 1.0), so being one level off costs less than being three levels off. The log term is floored at −9.21 to bound the penalty for a near-zero probability on the true answer.
• Exploration: zero-mean Gaussian noise is added to the option logits.
• Update: REINFORCE with a group-mean baseline, in the style of GRPO.
• Multi-turn: conversations use TD(λ) targets with λ = 1.0 over prefix slices, so earlier turns learn from how the conversation ended.

What it does and does not guarantee

A proper-scoring reward makes calibration the objective; it does not make a trained checkpoint perfectly calibrated on your data. The model card reports the checkpoints ship over-confident (mean ECE 0.466 English, 0.314 multilingual) and that fitting temperature scaling per question type and option count brings that to 0.081 and 0.106. Treat RLCD as a strong starting point, then calibrate.

Read the RLCD explainer and proper scoring rules for the derivations.