GPU inference

GPU inference is running a trained neural network's forward pass on a graphics processing unit. Transformer inference is dominated by large matrix multiplications, which GPUs execute in parallel across thousands of cores. Modern GPUs also have tensor cores that run fp16 or bf16 arithmetic much faster than fp32, so inference is usually done in mixed precision.

Key practical concerns:

• Memory: the weights, activations and batch must fit in VRAM.
• Precision: fp16 and bf16 halve memory and speed up math; bf16 requires newer hardware.
• Warm vs cold: loading weights onto the device takes seconds, so servers load once and keep the model resident.
• Utilisation: small single requests leave a GPU mostly idle; batching fills it.

How Laya runs on GPU

The Laya runtime picks CUDA automatically when available, then Apple MPS, then CPU. On CUDA it runs the forward pass under autocast in the checkpoint's configured precision, falling back to fp16 on GPUs older than compute capability 8 (which lack fast bf16). On CPU and MPS it uses fp32. If the model cannot be placed on the GPU, or a forward pass runs out of GPU memory, it falls back to CPU and warns that inference will be much slower.

The model card's figures, all on one Tesla T4:

• single question: 39.5 ms (English), 32.8 ms (multilingual)
• 103–332 questions per second batched
• preloaded CPU for comparison: 193–464 ms per request
• cold checkpoint reload: 10.3 s median on T4, 7.4 s on CPU

A T4 is a modest, widely available inference GPU, so these numbers are a conservative reference rather than a best case.

On Laya Studio you do not manage devices, precision or checkpoint loading yourself. If you self-host, set LAYA_DEVICE=cuda and LAYA_PRELOAD=1 for laya-serve. See throughput and latency budgets for agents.