Gaussian exploration noise

Gaussian exploration noise is a standard way to make a reinforcement learning policy explore. During training, instead of always emitting its current best output, the policy's output is perturbed by noise drawn from a normal distribution with mean zero and some standard deviation σ. The rewards of these perturbed outputs tell the learner which direction to move. It is common in continuous-control algorithms (for example Gaussian policies in policy-gradient methods, or the action noise in DDPG and TD3), where outputs are real-valued and discrete sampling does not apply.

Zero mean matters: the noise explores around the current policy without biasing it in any direction. σ controls the trade-off between exploring widely and learning from outputs close to the current behaviour; it is often annealed over training.

How Laya uses it

In RLCD, Laya's policy output is not a discrete label but a probability distribution over a question's options, produced by a softmax of the option logits. Exploration adds zero-mean Gaussian noise to the logits. The perturbed logits give a slightly different reported distribution, which is scored with a strictly proper scoring rule. REINFORCE with a group-mean baseline then pushes the policy toward the perturbations that scored better than the group average.

Perturbing logits rather than sampling a single label has two advantages for calibration training:

• every perturbation still yields a full distribution, so the reward can judge the probabilities, not just the argmax
• the exploration is smooth: small σ explores small changes in confidence, which is exactly the quantity RLCD is trying to get right

Exploration noise is a training-time device only. At inference, Laya computes its answer deterministically: the same state and questions give the same probabilities. See the RLCD explainer and calibrated probabilities.