RLCD: reinforcement learning for calibrated decisions

RLCD (reinforcement learning for calibrated decisions) is the training recipe Convai Innovations used for Laya. The model is treated as a policy that reports a probability distribution over the options, and its reward is a strictly proper scoring rule computed from the true outcome, so reporting honest probabilities is the reward-maximising behaviour.

A decision model is only as useful as the numbers it reports. If a router says "90% billing", downstream code will act on that 90%: auto-route above a threshold, escalate below it, or combine it with a cost. That only works if 90% means roughly nine in ten are right.

Standard training does not guarantee this. Modern neural networks trained with cross-entropy are frequently over-confident, a finding documented at length by Guo et al. (2017). Large language models add their own problem: the probability of a generated label token depends on phrasing, tokenisation and sampling settings, and verbalised confidence ("I am 95% sure") is a separate text output with no guarantee of meaning.

RLCD (reinforcement learning for calibrated decisions) is Convai Innovations' name for the training recipe used for Laya. Its central idea is simple: make the reward itself a strictly proper scoring rule, so that the policy with the highest expected reward is the one that reports its true beliefs.

In RLCD the "action" is not a label. It is a probability distribution over the options of a question. For a choice question with k options, the policy observes the state and the question, and reports q = (q_1, ..., q_k). After the true outcome is revealed, the reward is:

R(q, y) = strictly proper score of q given outcome y

Framing the report as the action has a direct consequence. Under a strictly proper rule, for any belief p the expected reward E_{y~p}[R(q, y)] is maximised uniquely at q = p. A policy that shades its report toward the favourite (over-confidence) or toward uniform (hedging) earns less on average. The theory is in proper scoring rules.

The package's proper_reward function combines three strictly proper rules:

R = log q(y) + 0.5 · spherical(q, y)                     for choice and noul
R = log q(y) + 0.5 · spherical(q, y) − 1.0 · RPS(q, y)   for score (ordinal) questions

spherical(q, y) = q(y) / ‖q‖₂
RPS(q, y)       = Σ_j (Q_j − T_j)² / (k − 1)    with Q, T the cumulative forecast and target

Details from the source:

• The log term is clamped at −9.21 (log 0.0001), which bounds the damage one example can do to a batch.
• The target can be a soft distribution, not just one-hot, so a teacher's probabilities can be used as targets. The model card's typed-decisions table reports a "teacher self-agreement ceiling", which indicates teacher-labelled data in at least that benchmark.
• Masks let a single batch mix questions with different numbers of options.

Reinforcement learning needs exploration: the policy has to try reports other than its current best guess to learn which ones score better. The model card describes RLCD's mechanism:

1. Exploration noise. Zero-mean Gaussian noise is added to the option logits, producing perturbed reports q̃ around the policy's current distribution.
2. Reward. Each perturbed report is scored with the proper reward above.
3. Baseline. Rewards within a group of samples for the same item are compared with the group mean. This is the group-relative baseline popularised by GRPO (Shao et al., 2024), and it reduces the variance of the gradient without a learned value network.
4. Update. The policy is updated with REINFORCE (Williams, 1992): perturbations that scored above the group mean are made more likely, those below less likely.

In plain terms: "try slightly different probability reports, keep the ones that a proper scoring rule liked better." Because the scoring rule rewards honesty, the policy is pushed toward reports that match the empirical frequencies of outcomes.

Some decisions evolve over a conversation. Whether a customer will churn is clearer at turn eight than at turn two. RLCD handles multi-turn trajectories by training on prefix slices of the conversation and using temporal-difference targets (Sutton, 1988).

The package's td_lambda_targets walks each episode backwards:

G ← y_final
for step j from last to first:
    if j is not the last step:  G ← (1 − λ) · p_true(step j+1) + λ · G
    target(step j) = (1 − G, G)

With λ = 1.0, which the model card states is what Laya uses, every prefix is trained toward the final outcome: a Monte Carlo target. Smaller λ would bootstrap from the model's own prediction at the next step. The effect is that an early turn is rewarded for the probability it assigns to the eventual outcome, which is the right target for "how likely is this conversation to end in escalation?"

Alongside the option scorer, Laya's decision model has an act head: a small network that reads the pooled representation plus four summary features of the answer distribution (top probability, margin between top two, normalised entropy, and option count) and outputs whether to act or escalate. It is sized from an act_costs configuration, which suggests it was trained with per-action costs.

The model card is explicit that this output does not work yet: "action.act_probability carries no usable signal yet. It reads 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate on confidence instead, which reaches an AUROC of 0.77 on the same items." See act and escalate routing for how to build that gate yourself.

RLCD makes honesty the optimal policy in expectation. The measured results show that this is necessary but not sufficient:

The card summarises: the model "ships over-confident". The training objective points the right way, but the finished checkpoints still need post-hoc temperature scaling on your data before their probabilities should be trusted for thresholds. The card also does not publish an ablation comparing RLCD against plain supervised cross-entropy training with the same data, so treat RLCD as a principled design choice rather than a measured advantage over supervised training.

Laya Studio serves the RLCD-trained checkpoints. The fields that come out of the reward design are probabilities and confidence. A good first exercise is to collect a labelled sample and check calibration before relying on a threshold:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": [
      {"role": "user", "content": "Your app logged me out again."},
      {"role": "agent", "content": "Sorry about that. Can you try clearing the cache?"},
      {"role": "user", "content": "I did. Honestly I am looking at other tools now."}
    ],
    "questions": {
      "churn_risk": {
        "type": "noul",
        "instructions": "Does the conversation suggest the customer may leave for a competitor or cancel?"
      },
      "frustration": {
        "type": "score",
        "instructions": "How frustrated is the customer by the last turn?",
        "criteria": ["calm", "concerned but civil", "clearly annoyed", "very angry"]
      }
    }
  }'

A list state is serialised as JSON, so the model sees the whole conversation. The response returns answers.churn_risk.noul and answers.frustration.probabilities, plus action.act_probability, which you should ignore for now. Log the probabilities against outcomes, compute expected calibration error per question type, and fit temperatures if needed. Two questions read the state twice, billed per input token. Start with 5 free runs at /signup, and see the docs for response fields.