The set of techniques — quantisation, batching, caching, distillation and specialised silicon — that reduce the cost and latency of running a trained model.
Training is a one-off cost; inference is paid on every request. For any product at scale, inference efficiency determines whether the unit economics work at all.
Weights are compressed to lower precision, requests are batched across users, repeated prefixes are cached, and smaller student models take the easy traffic while a larger model handles the rest.
Cost per token has fallen steeply for several years. Dedicated inference accelerators are now a credible alternative to general-purpose GPUs for steady-state workloads.
Accelerators, high-bandwidth memory and serving frameworks.
A serving layer that sends each request to the cheapest model meeting a declared quality threshold rather than defaulting to the largest available model.
Source: Huburb demo record · no external source attached
A chip designed for serving rather than training reaches volume manufacture with a supporting compiler stack.
Source: Huburb demo record · no external source attached