Option-marker scoring: how Laya reads a question and its answers together

Option-marker scoring is the technique Laya uses to classify with labels supplied at request time. The options are written into the input, each preceded by a [MASK] marker; the encoder reads the question, all options and the text together, and a scorer reads one logit per marker. A softmax over those logits gives the answer probabilities.

A classic fine-tuned classifier has a fixed output layer: one neuron per label, learned during training. That is efficient, but it ties the model to one label set. Add a department, rename a category or start a new workflow and you need new labelled data, a new training run and a new deployment.

Several techniques let the label set change at request time:

Option-marker scoring aims for the best cell in each column: one pass, labels supplied per request, and full attention between the labels, the question and the text.

For every question, Laya builds one token sequence in this format (from build_sequence in the package source):

[CLS] <type> question: <instructions> [SEP] [MASK] option0 [MASK] option1 ... [MASK] optionK [SEP] <state> [SEP]

Concretely, for a choice question:

[CLS] choice question: Which department should handle this request? [SEP]
[MASK] billing: invoices, payments, refunds
[MASK] technical: bugs, outages, system errors
[MASK] other: everything else [SEP]
{"subject": "Duplicate charge", "body": "We were billed twice for March."} [SEP]

Options are rendered according to the question type:

• choice: label: description, or just label when no description is given;
• score: level 0: <description>, level 1: <description>, and so on;
• noul: always two options, false: <description> then true: <description>, with default wording if you omit criteria.

Object and list states are serialised to JSON. Any literal [MASK] text inside your instructions, options or state is replaced with a space so it cannot be confused with a marker.

The model then does four things:

1. Encode. The encoder (ModernBERT-large or mmBERT-base) runs over the whole sequence with bidirectional attention. Every option marker attends to the instructions, to the other options and to the state.
2. Condition on question type. A learned type embedding (one vector each for choice, score and noul) is added to every token, and two further transformer layers refine the representation.
3. Read the markers. The vector at each [MASK] position is gathered and passed through a small scorer (LayerNorm, Linear, GELU, Linear) that outputs a single logit.
4. Normalise. The logits for that question's options are divided by a calibration temperature and softmaxed into the returned probabilities.

logit_i = scorer( h[marker_i] )
p_i     = exp(logit_i / T) / Σ_j exp(logit_j / T)

The [MASK] token is a deliberate choice. In masked-language-model pretraining, the vector at a [MASK] position is trained to summarise its surroundings in order to predict the hidden token. Reusing it as an option slot gives the scorer a representation that is already context-aware.

Because every option sits in the same sequence, the model compares options directly. "Billing" and "other" are scored with knowledge of each other, which NLI-style zero-shot classification (one pass per label, each label judged alone) cannot do.

Each question gets its own sequence, but all sequences in a request are padded into one batch and run through the model together. The model card reports that ten questions cost 158.6 ms on the English checkpoint and 72.3 ms on the multilingual one on a T4, against 39.5 ms and 32.8 ms for one. Per-question cost falls as you batch more.

The state is repeated in each question's sequence, so input tokens scale with the number of questions. That is visible in the usage.input_tokens field of the response.

The sequence has two budgets:

Inside the head budget, the rules in the source are:

• each option is truncated to 48 tokens plus its marker;
• if the options leave fewer than 16 tokens for the instructions, every option is cut to max(4, (head_max_len − 16) // k) tokens, marker included;
• the instructions get whatever remains, with a floor of 8 tokens;
• the state fills what is left of max_len, truncated from the end.

Work through a 77-option question on the multilingual checkpoint: (256 − 16) // 77 = 3, so each option gets the floor of 4 tokens: one marker and three tokens of label text. The instructions are cut to 8 tokens. Many banking intents are indistinguishable in three tokens ("card payment not recognised" vs "card payment wrong exchange rate"). This is the mechanism behind the model card's Banking77 result: 0.425 for Laya against 0.870 published for Jev.

The card suggests two fixes: raise head_max_len (for example to 512) and max_len when self-hosting, or split the decision into a coarse question followed by a fine one. On a hosted API, the second is the practical option. If the markers would not fit in the sequence at all, Laya rejects the question with an error rather than silently dropping options.

• Keep option counts modest. Under about 20 per question, and fewer on the English checkpoint.
• Front-load the distinguishing words. Truncation cuts from the end of each option.
• Keep descriptions short and parallel. "invoices, payments, refunds" beats a full sentence.
• Put the decisive part of the state first. State truncation keeps the beginning.
• Test for order sensitivity. Options are scored in the order you give them. Shuffle on a validation set and check that answers are stable.
• Use neutral keys when noul sticks. The model card documents noul sometimes following its false:/true: labels; a two-option choice with keys like A/B avoids that.

Because options are part of the input, you can change them on every call. This example uses a coarse-to-fine split for a large intent space:

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer $LAYA_STUDIO_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"message": "My card payment in euros was charged at a strange exchange rate."},
    "questions": {
      "area": {
        "type": "choice",
        "instructions": "Which area is message about?",
        "criteria": {
          "cards": "card payments, card delivery, lost or stolen cards",
          "transfers": "bank transfers, top-ups, direct debits",
          "account": "identity checks, account access, personal details",
          "fees_and_rates": "fees, charges, exchange rates"
        }
      }
    }
  }'

Then send a second request with only the fine-grained options under the chosen area. Two small questions typically beat one enormous one on this architecture, and each is billed only the input tokens it reads. See /docs for request limits and /signup for a key.