Entropy-based confidence

Entropy-based confidence summarises a whole probability distribution in one number by measuring how concentrated it is. Shannon entropy H(p) = −Σ p_i log p_i is 0 when all mass is on one option and log k when mass is spread evenly over k options. Normalising and flipping gives a score in [0, 1]:

confidence = 1 − H(p) / log k

Compared with using the top probability alone, this penalises distributions where the runner-up options are all non-trivial, and it is comparable across questions with different numbers of options: a uniform distribution scores 0 whether there are 3 options or 12.

How Laya computes confidence

For choice and score questions, Laya returns exactly this normalised-entropy confidence (confidence_from_probs in the package), computed after temperature scaling. For a noul question, which has only two options, the field is simpler: max(p, 1 − p), the probability of whichever side won. Note the different scales: a noul confidence can never fall below 0.5, while an entropy confidence of 0.5 on a four-way choice already implies a fairly peaked distribution. Set thresholds per question type, not globally.

This field is the one to gate automation on. The model card reports that on 396 labelled decisions confidence reached an AUROC of 0.77 for separating right from wrong answers, while action.act_probability currently carries no usable signal (AUROC 0.30, issue #185). The decision head computes a normalised entropy internally too, as one of the features fed to the act head.

Entropy confidence only means something if the underlying probabilities are calibrated; see calibration and act/escalate routing.