Human-in-the-loop

Human-in-the-loop (HITL) describes systems where automated decisions and human judgement are combined deliberately: the model handles the routine bulk, people handle what the model should not decide alone, and human corrections flow back as labelled data.

A well-designed HITL loop has three parts:

1. A routing rule that decides which items reach a person. Usually a mix of content rules ("any refund over a limit") and uncertainty rules ("confidence below a threshold"). See escalation and abstention.
2. A review surface that shows the reviewer the input, the model's answer and its probabilities, so they can accept or correct quickly.
3. A feedback path that stores corrections as labels for evaluation, threshold re-fitting and fine-tuning.

HITL with Laya

Laya fits this pattern because its outputs are already structured for review. Each answer has a typed value, the full probability distribution over the options you defined, and a confidence. A reviewer sees "billing 0.62, technical 0.31" rather than a paragraph of generated reasoning, and their correction is a clean label for the same schema.

Practical guidance grounded in what the model card reports:

• Route on confidence, not action.act_probability; the latter carries no usable signal yet.
• The checkpoints ship over-confident. Use the first few hundred human-reviewed items to fit per-question-type temperature scaling, then re-check calibration with a reliability diagram.
• The base checkpoints are close to chance on some specialised schemas zero-shot; the typed-decisions checkpoint went to 0.766 accuracy only after fine-tuning. Human labels are what make that possible for your own workflow.

Start with a low automation rate and widen it as measured error on auto-handled items stays within budget. The act/escalate routing guide and zero-shot vs fine-tuned cover both halves.