Inference Optimisation

Training is a one-off cost; inference is paid on every request. For any product at scale, inference efficiency determines whether the unit economics work at all.

Weights are compressed to lower precision, requests are batched across users, repeated prefixes are cached, and smaller student models take the easy traffic while a larger model handles the rest.

Cost per token has fallen steeply for several years. Dedicated inference accelerators are now a credible alternative to general-purpose GPUs for steady-state workloads.