Calibrated probabilities: when can you trust an AI model's confidence?

Calibrated probabilities are confidence scores that match how often a model is actually right. If a calibrated model gives 100 answers at 80% confidence, about 80 of them are correct. Calibration is what makes a confidence threshold meaningful, so software can act on sure answers automatically and send unsure ones to a person.

A classifier outputs a probability distribution over options. It is calibrated if, for every probability level p, the answers it gives with confidence p are correct a fraction p of the time:

P(correct | reported confidence = p) = p

Calibration is a property of groups of predictions, not of one answer. A single answer at 0.9 can be wrong, and a calibrated model will be wrong on about 10% of its 0.9 answers. TypeSafe's documentation puts it the same way: "Calibration is measured across groups of predictions; it does not guarantee that an individual answer is correct."

Calibration is also separate from accuracy:

Model C is more useful than model D even though they are equally accurate. C tells you which of its answers to distrust; D does not. For an automated system, which is the setting System 1 decision models are built for, that difference decides whether a confidence threshold does anything at all.

Guo et al. (2017), "On Calibration of Modern Neural Networks", showed that modern deep networks are often badly over-confident even when accurate, and that a one-parameter fix, temperature scaling, removes much of the error. Both points apply directly to decision models today.

Reliability diagram

Sort predictions by confidence into bins (say 0.0–0.1, 0.1–0.2, …). For each bin, plot mean confidence on the x-axis and actual accuracy on the y-axis. A calibrated model sits on the diagonal. Points below the diagonal mean over-confidence (claims more than it delivers); points above mean under-confidence.

Expected calibration error (ECE)

ECE compresses the reliability diagram into one number: the weighted average gap between confidence and accuracy across bins.

ECE = Σ_b (|B_b| / n) · |acc(B_b) − conf(B_b)|

where B_b is the set of predictions in bin b, and n is the total number of predictions. ECE is 0 for perfect calibration; larger is worse. The open-source laya package implements it with 15 equal-width bins (ece_score in common.py). The details matter when comparing numbers across papers: bin count, whether you use the top-label probability or a class-wise version, and sample size all change the result. See expected calibration error for the caveats.

Brier score and log loss

ECE only checks the top answer's confidence. Proper scoring rules judge the whole distribution:

• Brier score (Brier, 1950): mean squared error between the predicted distribution and the one-hot truth. Lower is better.
• Log loss / negative log-likelihood (NLL): −log of the probability assigned to the true outcome. It penalises confident mistakes heavily, and assigning exactly 0 to the true outcome is infinitely bad.

A scoring rule is strictly proper when the only way to maximise the expected score is to report your true belief (Gneiting & Raftery, 2007). That property is the basis of how Laya and Jev are trained. See proper scoring rules.

Both Laya and TypeSafe Jev name their training method RLCD, Reinforcement Learning for Calibrated Decisions. TypeSafe coined the term in its Jev launch post. The Laya model card describes its own implementation in detail:

• The policy outputs a probability distribution over the options.
• Exploration adds zero-mean Gaussian noise to the logits.
• The reward is a strictly proper scoring rule: log score plus spherical score, plus a ranked probability score (RPS) for ordinal score questions.
• Updates use REINFORCE with a group-mean baseline (GRPO-style).
• Multi-turn conversations use TD(λ = 1.0) over prefix slices.

In the package source (proper_reward in common.py), the reward is the log score + 0.5 × the spherical score, minus 1.0 × RPS for score questions, with the log term floored at about −9.21 so one catastrophic answer cannot dominate a batch.

The card's summary of the logic: "Expected reward is maximised only by reporting honest probabilities." That is true of the objective. It does not guarantee the trained model ends up calibrated on your data, as the next section shows. See RLCD for more on the training method.

Here is what has actually been measured. Every figure comes from the source named in its row.

Laya (from its model cards and BENCHMARKS.md)

The card is explicit: the checkpoints "ship over-confident". The multilingual checkpoint ships with temperature = [1.0, 1.0, 1.0], meaning no fitted temperatures at all. The fix is refitting one temperature per (question type, option count) on held-out data, and the card instructs: "Do this on your own data before trusting the probabilities."

On the typed-decisions benchmark (2,000 decisions), the fine-tuned laya-typed-decisions checkpoint has ECE 0.213 and Brier 0.062, against Jev 1.13.0's published ECE 0.144 and Brier 0.148. So Jev is better calibrated on that benchmark by ECE, and Laya has the lower Brier score. The card also notes that this checkpoint's temperatures were fitted on training data (issue #186), so its confidence should be treated as uncalibrated until refit.

The language failure case

The most instructive number in the Laya documentation is a failure. Given Khmer text, the English checkpoint scores 0.000 accuracy at 0.952 confidence. On Hebrew, Armenian and Bengali it is near random while reporting 0.89–0.96 confidence, and its mean confidence "never drops below 0.885 at any accuracy level". A model reading text it cannot tokenise meaningfully still produces a sharp distribution. No confidence threshold can catch this, because the confidence is not low.

Laya's answer is to route before the forward pass: script detection (under 0.5 ms, pure Python) sends non-Latin and non-English text to the multilingual checkpoint, where macro ECE across 51 MASSIVE languages is 0.387, against 0.733 for the English checkpoint. Laya Studio does this routing automatically. See language routing.

TypeSafe Jev (third-party measurements)

TypeSafe describes Jev's outputs as calibrated but does not publish an ECE figure itself. Independent measurements:

• DMB (nibzard): on forced-uncertainty items, Jev "admits" uncertainty on 49.7% of them, while every LLM tested did so on 97.3–100%. Jev's ECE was 0.246, "the worst calibration error measured", against 0.039–0.122 for the LLMs.
• jev-benchmarks (AbdelStark): on DAIR Emotion (6 labels), Jev had Brier 0.846, NLL 5.588, and assigned zero probability to the true label on 16% of examples. On AG News and Banking77 it had a clear accuracy and Brier advantage over the GLiNER baseline in that pilot.

These are small, task-specific studies. The DMB and jev-benchmarks authors both flag the limitations of their setups. Treat them as evidence that no vendor's calibration should be taken on trust, including Laya's.

Temperature scaling divides a model's logits by a scalar T before the softmax:

pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

T > 1 flattens the distribution (less confident); T < 1 sharpens it. The argmax does not change, so accuracy is unaffected. Only the confidence moves. You fit T by minimising NLL on a held-out labelled set.

Laya's runtime applies temperatures per (question type, option-count bucket), with buckets 2, 3-5, 6-10 and 11+, because a 2-option noul and a 15-option choice are miscalibrated in different ways. The package also clamps temperatures to [0.5, 5.0]. The source comment explains why: a shipped choice:11+ temperature of 0.1006 multiplied logits roughly tenfold, so "a 0.24 top probability is published as 0.99". A temperature that sharpens that much is not honest calibration. The clamp refuses it.

When you call a hosted API you do not control the model's internal temperatures, but you can recalibrate on your side. Every answer includes the full probabilities map, so you can apply a temperature to the log-probabilities:

import numpy as np
from scipy.optimize import minimize_scalar

def ece(conf, correct, bins=15):
    """Expected calibration error with equal-width bins (same scheme as laya.ece_score)."""
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    edges = np.linspace(0, 1, bins + 1)
    total = 0.0
    for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
        sel = ((conf >= lo) if i == 0 else (conf > lo)) & (conf <= hi)
        if sel.any():
            total += sel.mean() * abs(conf[sel].mean() - correct[sel].mean())
    return total

def rescale(probs, T):
    """Apply temperature T to a probability vector via its log-probabilities."""
    z = np.log(np.clip(probs, 1e-12, 1.0)) / T
    z -= z.max()
    p = np.exp(z)
    return p / p.sum()

def fit_temperature(prob_rows, labels):
    """prob_rows: list of probability vectors (options in a fixed order); labels: true index."""
    def nll(T):
        return -np.mean([np.log(max(rescale(p, T)[y], 1e-12)) for p, y in zip(prob_rows, labels)])
    return minimize_scalar(nll, bounds=(0.5, 5.0), method="bounded").x

# logged: answers from /v1/systemone for one choice question, plus the label a person assigned
keys = ["billing", "technical", "account", "other"]
rows = [np.array([a["probabilities"][k] for k in keys]) for a in logged_answers]
ys = [keys.index(lbl) for lbl in logged_labels]

T = fit_temperature(rows, ys)
before = ece([r.max() for r in rows], [r.argmax() == y for r, y in zip(rows, ys)])
after_rows = [rescale(r, T) for r in rows]
after = ece([r.max() for r in after_rows], [r.argmax() == y for r, y in zip(after_rows, ys)])
print(f"T={T:.2f}  ECE {before:.3f} -> {after:.3f}")

Practical notes:

• Fit per question. A temperature for your queue choice question will not suit your is_urgent noul.
• Use held-out data. Fitting and evaluating on the same items overstates the improvement; the typed-decisions checkpoint's issue #186 is an example.
• A few hundred labelled answers per question is a reasonable starting point. With very few items, per-bin ECE is noisy.
• Refit after changes. New options, reworded instructions, a model upgrade or a traffic shift can all move calibration.
• The ECE above uses top-label probability. Laya's confidence field is a different statistic (next section), so compute ECE on probabilities, not on confidence.

More detail: temperature scaling.

Decision APIs return both a probability distribution and a single confidence. They are not the same thing.

Laya's confidence is normalised entropy: 1 − H(p) / log(k), where H is the Shannon entropy of the distribution and k the number of options. It is 1.0 when all mass is on one option and 0.0 when the distribution is uniform. It measures how peaked the distribution is, not the probability of the top answer. For a noul, Laya reports confidence = max(p, 1 − p).

Jev's confidence is also "derived from the probabilities". TypeSafe's docs demonstrate it for three options with (3 × largest probability − 1) / 2, a rescaling of the top probability so that uniform = 0 and certain = 1. Jev returns confidence on Choice and Score answers only; "Noul answers don't carry one."

A worked example with three options:

(Laya values computed from the entropy formula; Jev-style values from the formula in TypeSafe's confidence docs.)

The two services return the same field name on the same wire format with different formulas, so a threshold tuned on one does not transfer to the other. If you migrate between them, re-derive thresholds from logged data. You can also skip both and threshold on the top probability after your own temperature scaling, which is directly interpretable when calibrated.

The Laya card also reports that confidence separates right from wrong answers usefully (AUROC 0.77 on 396 labelled decisions), while the action.act_probability field does not (AUROC 0.30). Use confidence or the probabilities, not act_probability.

The point of calibration is to decide how much work a model can do without supervision. The standard method is selective prediction: auto-handle answers above a threshold, send the rest to review.

1. Log answers with labels. Run the model on a representative sample and have people label it.
2. Calibrate (fit temperatures per question) on part of the sample.
3. Sweep thresholds on the rest: for each threshold t, compute coverage (share of items with confidence ≥ t) and error on those items.
4. Pick the lowest threshold that meets your error budget. If you can tolerate 2% errors on auto-routed tickets, choose the t where the error on covered items is ≤ 2%.
5. Scale thresholds with stakes. A read-only action can run at a lower threshold than an irreversible one.

def coverage_at_error(conf, correct, budget=0.02):
    """Largest coverage whose error rate on covered items stays within budget."""
    order = np.argsort(-np.asarray(conf))
    c = np.asarray(correct, float)[order]
    best_t, best_cov = None, 0.0
    for i in range(1, len(c) + 1):
        err = 1.0 - c[:i].mean()
        if err <= budget:
            best_t, best_cov = np.asarray(conf)[order][i - 1], i / len(c)
    return best_t, best_cov

The jev-benchmarks pilot reports exactly this metric ("coverage at ≤5% error") and shows why it matters. On AG News, Jev could auto-handle 83% of items within a 5% error budget. On DAIR Emotion, where its distributions put zero mass on the truth 16% of the time, coverage was 0%: no threshold was safe.

In TypeScript, a gate on a Laya Studio response looks like this:

type ChoiceAnswer = { type: 'choice'; choice: string; probabilities: Record<string, number>; confidence: number };

function decide(a: ChoiceAnswer, threshold: number, temperature = 1): { act: boolean; label: string } {
  const entries = Object.entries(a.probabilities);
  const logits = entries.map(([, p]) => Math.log(Math.max(p, 1e-12)) / temperature);
  const m = Math.max(...logits);
  const exps = logits.map((z) => Math.exp(z - m));
  const sum = exps.reduce((s, x) => s + x, 0);
  const top = Math.max(...exps) / sum;
  return { act: top >= threshold, label: a.choice };
}

For the routing side of this pattern, see act / escalate routing and decision models for AI agents.

Before you trust any model's probabilities in production:

Laya Studio is an independent hosted API powered by the open-source Laya model. It returns the full probability map on every answer so you can do all of the above on your own data. It is not affiliated with Convai Innovations or TypeSafe. Get an API key, read the docs, or see /pricing. For the vendor comparison, see Laya vs Jev.