Reinforcement learning

Reinforcement learning (RL) is a family of methods in which an agent (a policy) takes actions, receives a scalar reward, and is updated to make high-reward actions more likely. Unlike supervised learning, RL does not need the correct action for every input; it needs a way to score the actions the policy actually took.

Core ingredients:

• Policy: a distribution over actions given a state.
• Reward: a number saying how good an outcome was.
• Exploration: trying actions other than the current best, so the policy can discover better ones.
• Credit assignment: deciding which earlier actions caused a later reward, often with temporal-difference (TD) methods.
• Variance reduction: subtracting a baseline from the reward so updates are less noisy.

Policy-gradient methods such as REINFORCE update parameters in the direction of reward-weighted log-probability of the sampled actions. GRPO-style variants use the mean reward of a group of samples for the same input as the baseline, avoiding a separate value network.

RL in Laya

Laya's training, RLCD, is an unusual application of RL. The "action" is not a label but a reported probability distribution over a question's options. The reward is a strictly proper scoring rule (log plus spherical, minus ranked probability score for ordinal questions), so the highest expected reward goes to honest probabilities. Exploration adds zero-mean Gaussian noise to the logits; updates are REINFORCE with a group-mean baseline; multi-turn conversations use TD(λ = 1.0) over prefix slices.

Why RL instead of plain cross-entropy? Cross-entropy is itself the log score, so the two are related; the published RLCD recipe adds further proper-scoring terms and the exploration and multi-turn machinery on top. See the RLCD explainer and reward shaping.