Zero-shot vs fine-tuned: when is a general model good enough?

A zero-shot classifier labels text using only a description of the labels, with no task-specific training examples. A fine-tuned classifier has been trained on hundreds to thousands of labelled examples of your task. Zero-shot is faster to start and easier to change; fine-tuning usually wins on accuracy for domain-specific or idiosyncratic tasks.

Text classifiers sit on a spectrum defined by how much task-specific supervision they get:

• Zero-shot. The model sees only the input and a description of the labels. Examples: NLI-based classifiers that test "This text is about {label}" as an entailment hypothesis (Yin et al., 2019); instruction-tuned LLMs prompted with a label list; encoders like Laya that read options supplied in the request.
• Few-shot. The model sees a handful of labelled examples per class, either in the prompt (in-context learning, Brown et al., 2020) or through light training such as SetFit (Tunstall et al., 2022), which fine-tunes a sentence encoder on as few as eight examples per class.
• Fine-tuned. The model is trained on hundreds to thousands of labelled examples of the exact task, usually starting from a pretrained encoder (Howard and Ruder, 2018, established the recipe).

Zero-shot works when the task resembles something the model has seen and the labels are self-explanatory. "Is this email about billing, technical support or sales?" is close to what text on the internet already encodes. Topic classification and coarse sentiment often fall in this category.

It struggles when the labels encode your organisation's policy rather than common meaning. What counts as "needs_human" in your support desk, or "true_positive" for your security team, is a convention that no amount of pretraining can guess. The model has to infer the boundary from the label wording alone, and small wording differences move it.

The second failure mode is multi-field workflows where the right answer to one question depends on business rules that interact. These look like classification but behave more like policy application.

The Laya model card publishes results on the typed-decisions benchmark: 400 cases and 2,000 decisions across four synthetic workflows (customer service, invoice processing, security incidents and agent-trace observability).

Zero-shot, both base checkpoints land barely above random and well below simply predicting the most common answer per question. Fine-tuned on the same workflows, the same architecture clears the teacher's own self-agreement ceiling. The card's conclusion is blunt: "Laya is a fast base to specialise, not a zero-shot decision engine" for this kind of task.

On simpler, more conventional tasks the picture differs. With routing, the card reports 0.950 on four-label AG News and 0.595 on six-label DAIR Emotion. The card does not say whether those datasets were part of training, so treat them as evidence of what the model can do on conventional tasks, not as clean zero-shot results.

Laya does not fit neatly into one regime. Its options are supplied per request, which makes it zero-shot at the level of the schema: no retraining for a new label set. But the model was trained on many decision tasks in this format, so it has learned how to read a question and options, much as an instruction-tuned LLM has learned to follow instructions.

One consequence is that option wording does much of the work a training set would otherwise do. Compare two versions of the same option:

{ "needs_human": "needs a person" }
{ "needs_human": "the customer explicitly asks for a person, threatens legal action, or disputes a charge above the refund limit" }

The second encodes your policy in text the model can read against the state. It is the cheapest improvement available before collecting labels, and it is worth iterating on with a labelled evaluation set before concluding that zero-shot has failed. Keep descriptions short enough to fit the option budget (48 tokens per option at most).

That design has a useful property: the same format serves both regimes. You can start zero-shot with the base checkpoint, measure, and if accuracy is not enough, fine-tune on your labelled data without changing the request format. That is what laya-typed-decisions is: the base architecture, fine-tuned, served behind the same API.

1. Build a labelled evaluation set first. Two hundred to five hundred real examples per question, labelled by the people who own the decision. You need it whichever regime you choose.
2. Compute the baselines. Random and majority-class accuracy per question. A model that does not beat majority class is not helping.
3. Try zero-shot. Run the evaluation set through a general model. Iterate on option wording; descriptions matter a lot for prompted-schema models.
4. Check calibration as well as accuracy. A zero-shot model at 85% accuracy with honest probabilities may be more useful behind a confidence gate than a 90% model that is always sure.
5. Fine-tune if the gap is large and stable. If zero-shot stays far below the accuracy you need after reasonable prompt iteration, labelled data is the fix. Few-shot methods are a cheap middle step.
6. Keep the evaluation set after you ship. Re-run it when the schema changes or traffic shifts.

Laya Studio serves the base checkpoints and the fine-tuned typed-decisions checkpoint on one endpoint, so the comparison in step 3 is one parameter:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "typed-decisions",
    "state": {"invoice": "INV-2291 from Acme Supplies, 40 units at 12.50, PO-7781 lists 40 units at 11.00."},
    "questions": {
      "matches_order": {"type": "noul", "instructions": "Does invoice match its purchase order?"},
      "disposition": {
        "type": "choice",
        "instructions": "What should happen to invoice?",
        "criteria": {"approve": "pay as billed", "hold": "hold for review", "reject": "return to supplier"}
      }
    }
  }'

Run the same labelled set with "model": "english" and with "model": "typed-decisions", then compare accuracy, Brier score and calibration per question. The typed-decisions checkpoint was fine-tuned on four specific workflows; expect gains only where your questions resemble them. Billing is per input token and each question reads the item once, so a 300-item, two-question comparison across two checkpoints costs about 1,200 times an item's tokens. See /pricing for the 5 free runs and /docs for details.