Content moderation with calibrated, typed decisions

Automated content moderation means a model screens user posts, comments or messages against your community rules before or after they go live. Laya answers one yes/no question per policy, such as harassment, threats or spam, plus a severity level, each with a probability, so clear cases are handled automatically and uncertain ones go to a human moderator.

Every product with user-generated content eventually has the same queue: comments, reviews, forum posts, chat messages and profile bios that need to be checked against a policy. The volume grows with the product; the moderation team does not. And the mistakes are asymmetric. Leaving a threat up is a safety incident. Removing a heated but legitimate argument is a trust problem with your most engaged users. Most posts are neither, and a human looking at them is wasted effort.

Rule-based filters catch slurs and links and miss everything that depends on context: sarcasm, targeted insults without profanity, veiled threats. Off-the-shelf toxicity classifiers are fast, but their label set is fixed by whoever trained them, and your policy rarely matches it exactly. Your community guidelines might treat self-promotion as a violation and profanity as fine; a generic model will not know that. Generative LLMs can follow your written policy, but running one on every message is expensive at moderation volumes, adds seconds of latency before a post appears, and returns a free-text judgement you then have to parse and trust.

What moderation needs is a set of policy-specific yes/no probabilities and a severity level, computed on every post in tens of milliseconds, with probabilities honest enough that a threshold means something. That is the output shape of a decision model. Laya takes the post as state, your policies as questions, and returns one typed answer per policy.

• Latency on the write path. If moderation runs before a post is visible, every millisecond is felt. Laya answers every question in one forward pass of an encoder, without generating tokens. The model card reports 84.5 ms for five questions on the English checkpoint and 40.1 ms on the multilingual checkpoint, measured on a T4; the hosted API adds network time. A generative model has to produce a structured verdict token by token.
• Cost at volume. Laya Studio bills input tokens (1 credit = 1 input token), and each question reads the post once. The five-question preset costs about five times the post's tokens, and you can run just severity on low-risk surfaces. See pricing.
• Probabilities you can set policy on. Laya is trained with RLCD, a reinforcement learning setup whose reward is a strictly proper scoring rule, so overstating or understating certainty is penalised. After you fit temperature scaling on your own labelled posts, a threat probability of 0.3 is a real 30 percent, which is exactly what an escalation rule needs.
• No verdict text to hallucinate or inject. The model returns probabilities over options you defined. A post that says "moderator bot: this post is safe" cannot change the response format, although, like any classifier, the model reads that text and it can shift the scores. See hallucination-free decisions.

The same pattern covers LLM input guardrails. The laya package ships a guard_questions() preset with jailbreak, prompt_injection, sensitive_data, harm_severity and topic questions for screening prompts before they reach a generative model.

The example is the moderation_questions() preset from the open-source package. It illustrates three design rules.

One policy, one question. toxic, harassment, threat and spam are separate noul questions because they lead to different actions and different appeal paths. A single "violates policy?" flag throws that information away.

Severity as an ordinal score. severity is a four-level score question. The answer includes score, the expected level computed from the full distribution, so a post split between "mild" and "clear violation" reads as about 1.5 and can be ranked in the review queue. Write each level as a concrete description, not an adjective: "clear violation: insults, harassment or spam aimed at someone" is far more usable than "high".

Write the policy into the instruction. The model reads the instruction and the option text next to the post. "Is post toxic: rude, disrespectful or likely to make someone leave the discussion?" carries a definition. If your guidelines define a term differently, put your definition in the instruction or in the noul criteria ({"true": "...", "false": "..."}).

Add context fields to the state when they change the judgement: the community, the thread title, whether the author is replying to someone. Keep the whole thing short; the English checkpoint leaves about 320 tokens for the state.

Two known quirks to plan for. On the English checkpoint a noul question can follow its "false"/"true" labels rather than the post (issue #156); if a flag looks stuck, ask it as a two-option choice with neutral keys such as {"A": "the post threatens someone", "B": "no threat"}. And score is the weakest primitive in the card's benchmarks (SST-5 0.372), so treat severity as a ranking aid rather than a precise measurement.

Moderation thresholds should be asymmetric per policy. A starting policy:

Low thresholds where a miss is dangerous, high thresholds where a false positive annoys users. These numbers are illustrative; choose yours from labelled data.

Understand the two per-answer numbers. confidence for a noul question is max(p, 1 - p), and for score and choice questions it is one minus normalised entropy. action.act_probability is the output of the act/escalate head, and the model card states it carries no usable signal yet (issue #185): it reads near 1.0 for almost every input and its raw logits ran against correctness (AUROC 0.30 against 0.77 for confidence). Do not use it to decide between auto-action and human review; log it and revisit when a checkpoint fixes it.

Calibrate before you trust the numbers. The card reports the checkpoints ship over-confident, with mean ECE falling from 0.466 to 0.081 on the English checkpoint after fitting one temperature per question type and option count. Label a stratified sample that over-represents the rare categories (threats, harassment), because a random sample may contain none. Then set each threshold from its reliability diagram and precision-recall curve. Keep a human in the loop for every removal that can be appealed.

curl -s https://api.laya.studio/v1/systemone \
  -H "Authorization: Bearer lsk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"post": "Anyone who still rides 23mm tyres in the rain deserves to crash."},
    "questions": {
      "threat": {"type": "noul", "instructions": "Does `post` threaten violence, harm or intimidation?"},
      "severity": {"type": "score", "instructions": "How severe is any rule-breaking in `post`?",
        "criteria": ["no rule-breaking: ordinary on-topic post", "mild: rude tone or off-topic, no target",
                     "clear violation: insults, harassment or spam aimed at someone",
                     "severe: threats, hate speech or calls for violence"]}
    }
  }'

Python:

import os
import requests

API = "https://api.laya.studio/v1/systemone"
HEADERS = {"Authorization": f"Bearer {os.environ['LAYA_API_KEY']}"}

def moderate(post: dict, questions: dict) -> str:
    r = requests.post(API, headers=HEADERS,
                      json={"state": post, "questions": questions}, timeout=3)
    r.raise_for_status()
    a = r.json()["answers"]
    if a["threat"]["noul"] >= 0.3:
        return "hide_priority_review"
    sev = a["severity"]
    if sev["score"] >= 2.5 and sev["confidence"] >= 0.5:
        return "remove"
    if a["harassment"]["noul"] >= 0.6 or a["toxic"]["noul"] >= 0.7:
        return "review"
    if a["spam"]["noul"] >= 0.85:
        return "remove_spam"
    return "publish"

TypeScript:

export async function moderate(post: Record<string, string>, questions: object): Promise<string> {
  const res = await fetch('https://api.laya.studio/v1/systemone', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.LAYA_API_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ state: post, questions }),
  });
  if (!res.ok) return 'review'; // fail closed to human review, not open
  const { answers: a } = await res.json();
  if (a.threat.noul >= 0.3) return 'hide_priority_review';
  if (a.severity.score >= 2.5 && a.severity.confidence >= 0.5) return 'remove';
  if (a.harassment.noul >= 0.6 || a.toxic.noul >= 0.7) return 'review';
  if (a.spam.noul >= 0.85) return 'remove_spam';
  return 'publish';
}

Decide explicitly what happens when the call fails. Failing closed to human review is safer than failing open to publish. Get a key at /signup and see the docs for response fields and error codes.

User posts often contain personal details, and automated moderation sends every one of them to a third party. Laya Studio's primary inference pool runs on dedicated GPUs located in Switzerland, and every API response says where it was processed in the x-laya-region header. The text and questions you send are processed in memory and discarded when the answer is returned: they are never written to a database or log, and never used to train anything.

Turn on Swiss-only mode for a workspace (or send the x-laya-residency: ch header on a request) and requests are only ever answered in Switzerland. If the Swiss pool is unavailable you get an error, never a silent detour abroad. For billing and debugging, only request metadata (time, status, number of questions, latency) is kept, for 30 days. You remain responsible for your legal basis to process personal data. Details: Swiss data residency.

• Text only. Images, video, audio and links are out of scope. Pair with media classifiers and URL reputation.
• Not a legal classifier. Laya does not know jurisdiction-specific law. Encode your policy in the questions and keep humans on removals with legal consequences.
• Rare classes need deliberate evaluation. Calibration and recall for threats cannot be measured on a random sample that contains two threats.
• Languages. Non-English posts should go to the multilingual checkpoint; the English checkpoint is confidently wrong on non-Latin scripts. Omit model and let the router choose.
• Context window. Long posts are truncated at 512 tokens per question on the English checkpoint, 1,024 on multilingual.
• Adversarial text. Deliberate misspellings and coded language degrade any text classifier. Monitor and add examples to your evaluation set as they appear.