Models & Training

Inference

Running a trained model to generate output. The day-to-day cost in any production LLM system, distinct from training cost.

Last updated: April 26, 2026

A glass prism lattice on dark volcanic ground with one crisp amber beam entering and one leaving

What Is Inference?

Inference is the act of running an already-trained model on a new input to generate output. For LLMs this is the API call you make every time the model produces a response. Production inference cost dominates total LLM spend by a wide margin: training costs are large but one-time, inference costs scale linearly with usage and recur forever. Three things drive inference cost: model size (larger models cost more per token), input length (more tokens to process), and output length (more tokens to generate). Optimizing inference cost is what cost-engineering an LLM system means.

Running inference in production

Calling a model once is easy. Running it for real users every day is where the habits below start to matter. None of them depend on a particular provider.

A close view of a glass prism lattice with a single amber beam passing cleanly through it

Latency is a product decision

Decide which requests a person is waiting on and which nobody is watching. For the first kind, stream the answer so text appears while the model is still writing, and send easy steps to a smaller model. For the second kind, move the work off the request path entirely and let it finish in the background. Most complaints about slow AI features come from treating every request as if a person were staring at it.

Batch what nobody is waiting for

Reports, document sweeps and nightly clean ups do not need an instant answer. Group them, queue them and let them run at a calmer pace, ideally through a provider batch option if one exists. Batching trades speed for lower cost and fewer rate limit collisions. Keep user facing traffic on its own path so a large batch never slows the chat window.

Cache the parts that repeat

Many requests start with the same long instructions or the same reference text. Put the stable material first and the changing question last so the provider can reuse the shared prefix. Separately, store the final answer for questions that repeat exactly, keyed by the input and the prompt version. When the prompt changes, the old cache entries must stop being served, so the version belongs in the key.

Control cost before it controls you

Set a ceiling on output length for every call, and cap the number of retries and tool loops so a confused agent cannot run all night. Give each feature its own budget and an alert when it drifts. Send routine work to the cheapest model that passes your checks and keep the large model for the cases that need it. Review the most expensive requests by hand now and then, because they usually point at a prompt that grew without anyone noticing.

What to log on every call

Record the model name, the prompt version, how many tokens went in and out, how long the call took, whether the cache was used, how many retries happened and what the outcome was. Add a request identifier that ties the call to the user action behind it. Keep secrets and personal details out of the log unless you have a clear reason to store them. With those fields you can answer why a request was slow, expensive or wrong without guessing.

If you can only do one thing this week, start logging those fields. Every other improvement above is easier once you can see what each call really costs.

When Should You Use Inference?

Track inference cost from day one. The single most useful metric: cost per user interaction. If that number is increasing without commensurate value increase, you have a problem.