Batching

Batching means grouping several inputs into one tensor and running them through the model in a single forward pass. Accelerators such as GPUs are highly parallel, so processing 10 items together usually costs far less than 10 separate passes. The trade-off is that every item in a batch waits for the slowest one, and very large batches increase per-request latency.

Serving systems use two kinds:

• Static batching: the client sends several items in one request.
• Dynamic batching: the server collects requests arriving within a short window and runs them together.

Batching in Laya

Laya batches at the level of questions. Every question in a /v1/systemone call is turned into its own sequence (question header, option markers, then the shared state) and all of them are padded into one batch and answered in a single forward pass. You do not need to do anything to get this: send five questions and they are answered together.

The model card's T4 measurements show the effect clearly:

Per-question cost drops sharply, and the card reports 103–332 questions per second batched on a single T4.

Practical advice

• Put every question you need about one state into one call rather than several calls. Billing is per input token and each question reads the state either way, so this does not cost more, and it saves round trips.
• Keep the question count per call proportional to your latency budget: 50 questions is fine offline, less so inside an interactive agent step.
• Long states are repeated in every question's sequence, so trim boilerplate first.

See throughput and GPU inference for the serving side.