Lead scoring from free text with calibrated confidence

AI lead scoring from text means a model reads what a prospect wrote in a form, email or call note and judges how well they fit and how ready they are to buy. Laya returns fit, buying stage and persona as typed answers with confidence scores, which you can combine with the firmographic lead score you already use.

Classic lead scoring adds points for company size, industry, job title and page visits. Those signals are useful, but they are proxies. The strongest evidence of intent is usually a sentence the prospect wrote: "our renewal is in Q1", "budget is approved", "I am a student working on a thesis". Point-based systems cannot read those sentences, so sales teams either read every inbound message by hand or accept a score that treats a procurement request and a job application from the same company identically.

The workload has a specific shape. Volume is moderate to high (every form fill, chat opener and reply to outbound), the answer space is small and stable (fit tier, stage, persona, a few yes/no flags), and the cost of a wrong answer is asymmetric: missing a ready-to-buy lead is expensive, while sending a researcher to an SDR is cheap. That is a textbook fit for a System 1 decision model: fast, typed, and honest about uncertainty so that borderline leads go to a human.

What Laya does not do is replace your firmographic model. It scores the text signal only. It does not know that Northwind has 300 employees unless you put that in the state, and it has no access to your win-rate history. The practical design is to treat Laya's outputs as features, or as a separate "text intent" score that you blend with the model you already trust.

You can prompt a general LLM to "rate this lead from 1 to 10". It will produce a number, but the number has no defined meaning, drifts between prompt versions, and comes wrapped in text you have to parse. Laya is built for the opposite contract:

• Typed output. A score question returns an expected level and a probability for every level. A choice question returns one of your labels, never a paraphrase. There is no generated text, so there is nothing to parse and no invented field. The answer can still be wrong, but it is always one of the options you defined.
• Calibrated probabilities. Laya is trained with RLCD, reinforcement learning against proper scoring rules, so its probabilities are meant to be read as probabilities. After temperature scaling on your own data, a 0.8 should be right about 80% of the time, which is what lets you set a routing threshold.
• Latency. The model card reports 39.5 ms for one question on the English checkpoint and 32.8 ms on the multilingual one, on a single T4 GPU, with every question in a call answered in one forward pass. Network time to the hosted API comes on top, but scoring a lead synchronously inside a form submission is realistic, which it rarely is with a multi-second LLM call.
• Predictable cost. On Laya Studio, 1 credit is 1 input token, 30% below Jev's list price. Each question reads the lead once, so the five-question example costs about five times the lead's tokens, and a short lead costs less than a long one. See pricing for the 5 free runs and plans.

If you are comparing approaches in more depth, read Laya vs LLM classifiers and the cost of using an LLM as a classifier.

The example request asks five questions about one state. Each question type is chosen for a reason.

fit is a score question because fit is ordinal: "strong" is more than "good", which is more than "weak". Score questions return an expected value (for example 2.6 on a 0–3 scale) and a full distribution, so you can distinguish a confident "good" from a split between "weak" and "strong". Write each level as a concrete description of evidence, not an adjective. "A concrete high-volume workload and an explicit replacement project" is scoreable; "very interested" is not.

stage and persona are choice questions because the options are unordered categories. Descriptions matter: the model scores each option at its own option marker, and the description is most of what it sees. Include an explicit escape option (not_a_buyer, other) so the model is not forced to pick a plausible-sounding wrong label for a vendor pitch.

budget_mentioned and has_timeline are noul questions (yes/no with a probability). Phrase them as checkable statements about the text ("does message say..."), not judgements ("is this a good lead?").

A few rules that hold across lead-scoring schemas:

• Keep choice questions under roughly 20 options. Options share a fixed prompt budget (head_max_len, 192 tokens on the English checkpoint), so long label lists leave each option only a few tokens.
• Put structured fields you already know (company size, title, source) into the state as JSON. The model reads them, and a title such as "Head of Customer Operations" helps the persona question.
• The model card documents that noul answers on the English checkpoint can follow the false:/true: labels instead of the text (issue #156). If a flag looks stuck on your data, ask it as a two-option choice with neutral keys:

{
  "budget_mentioned": {
    "type": "choice",
    "instructions": "Does message say that budget exists or has been approved?",
    "criteria": { "A": "yes, budget is mentioned as available", "B": "no budget is mentioned" }
  }
}

Every answer carries a confidence field and an action.act_probability field. Use them differently.

confidence is the signal to gate on. For choice and score questions it is one minus the normalized entropy of the answer distribution (entropy-based confidence), so it drops when probability is spread across options even if the top option is still ahead. For noul questions it is max(p, 1 - p). On the model card's own labelled test, confidence separated right from wrong answers with an AUROC of 0.77.

act_probability is not usable yet. The model card and issue #185 state that it reads close to 1.0 for almost every input and that its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Log it, but do not route on it. Write your policy so it can be added as a second condition when a checkpoint fixes it.

Calibrate before you trust the numbers. The checkpoints ship over-confident. Refitting one temperature per (question type, option count) moved mean ECE from 0.466 to 0.081 on the English checkpoint in the model card. Label a few hundred historical leads, fit temperatures, and pick thresholds from a reliability diagram, not from intuition.

A starting policy to tune, not a recommendation:

def route_lead(a):
    stage, fit = a["stage"], a["fit"]
    # act_probability is logged only; see issue #185
    if stage["choice"] == "ready_to_buy" and stage["confidence"] >= 0.6:
        return "ae_now"
    if stage["choice"] == "not_a_buyer" and stage["confidence"] >= 0.7:
        return "auto_reply"
    if fit["score"] >= 2.2 and fit["confidence"] >= 0.5:
        return "sdr_priority"
    return "human_review"

Measure the share routed to human review. If it is too high, you are either asking questions the model cannot answer from the text, or your thresholds are stricter than your error budget requires. More on this pattern in act/escalate routing.

Laya Studio exposes the /v1/systemone protocol. Send the state and questions; omit model and the router picks the English or multilingual checkpoint from the text.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer lsk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"job_title": "Head of Customer Operations", "message": "Budget is approved; replacing our rules engine before Q1 renewal."},
    "questions": {
      "stage": {"type": "choice", "instructions": "What buying stage does message indicate?",
                "criteria": {"researching": "no project yet", "evaluating": "comparing vendors or asking for a pilot",
                             "ready_to_buy": "asks for pricing or contract", "not_a_buyer": "job seeker, pitch or spam"}},
      "has_timeline": {"type": "noul", "instructions": "Does message mention a deadline or target quarter?"}
    }
  }'

import os, requests

def score_lead(state: dict, questions: dict) -> dict:
    r = requests.post(
        "https://api.laya.studio/v1/systemone",
        headers={"Authorization": "Bearer " + os.environ["LAYA_API_KEY"]},
        json={"state": state, "questions": questions},
        timeout=5,
    )
    r.raise_for_status()
    return r.json()["answers"]

export async function scoreLead(state: unknown, questions: Record<string, unknown>) {
  const res = await fetch("https://api.laya.studio/v1/systemone", {
    method: "POST",
    headers: {
      Authorization: "Bearer " + process.env.LAYA_API_KEY,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ state, questions }),
  });
  if (!res.ok) throw new Error("laya " + res.status);
  return (await res.json()).answers;
}

The response has answers keyed by your question ids (with choice, score or noul, probabilities, confidence and action), a usage block, and a routing block that records which checkpoint answered and why. Store the full probabilities alongside the lead so you can refit thresholds later. Create a key or read the API docs.

• Text only. Laya reads what is in the state. It does not know your ideal customer profile, deal history or product-usage data. Use its answers as features next to your firmographic model, not as the whole score.
• Zero-shot is a starting point. The model card reports that the base checkpoints are near chance on the typed-decisions benchmark zero-shot (0.362 against a 0.461 majority-class baseline), and that the 0.766 accuracy belongs to a checkpoint fine-tuned on that benchmark. Your lead questions are not that benchmark. Evaluate on a labelled sample of your own leads before automating anything.
• Score questions are the weakest primitive. The card cites 0.372 on SST-5. Keep fit to four clearly separated levels and check its confusion matrix.
• Short context. The English checkpoint reads 512 tokens per question, roughly 320 of them for the state. Send the form message and key fields, not a whole CRM history. The multilingual checkpoint reads 1,024.
• Non-English leads must go to the multilingual checkpoint. The English checkpoint degrades sharply off English, and on non-Latin scripts it can be confidently wrong. Leaving model unset lets the router handle this; see multilingual intake.
• Bias in text. A model that reads prose can reward fluent writers. Audit outcomes by region and language before letting a score gate human attention.